LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development
Year: 2023

Abstract:
In this work, we conduct a detailed analysis on the performance of legal-oriented pretrained language models (PLMs). We examine the interplay between their original objective, acquired knowledge, and legal language understanding capacities which we define as the upstream, probing, and downstream performance, respectively. We consider not only the models’ size but also the pre-training corpora used as important dimensions in our study. To this end, we release a multinational English legal corpus (LeXFiles) and a legal knowledge probing benchmark (LegalLAMA) to facilitate training and detailed analysis of legal-oriented PLMs. We release two new legal PLMs trained on LeXFiles and evaluate them alongside others on LegalLAMA and LexGLUE. We find that probing performance strongly correlates with upstream performance in related legal topics. On the other hand, downstream performance is mainly driven by the model’s size and prior legal knowledge which can be estimated by upstream and probing performance. Based on these findings, we can conclude that both dimensions are important for those seeking the development of domain-specific PLMs.

heading: 1 Introduction
text: Following closely the advances in the development of NLP technologies, the legal NLP literature is flourishing with the release of many new resources, including large legal corpora (Henderson* et al., 2022), datasets (Chalkidis et al., 2021a; Koreeda and Manning, 2021; Zheng et al., 2021; Chalkidis et al., 2022a; Habernal et al., 2022), and pre-trained legal-oriented language models (PLMs) (Chalkidis et al., 2020; Zheng et al., 2021; Xiao et al., 2021). Benchmark suites (Chalkidis et al., 2022a; Hwang et al., 2022; Niklaus et al., 2023) to evaluate the performance of PLMs in a more systematic way
∗ Equal contribution.
have been also developed, showcasing the superiority of legal-oriented PLMs over generic ones on downstream legal NLP tasks.
Despite this impressive progress, there is still not a thorough study on (a) how PLMs trained under different settings (pre-training corpora, size of the model) perform across different legal subcorpora, and (b) what sort of knowledge such models have acquired from pre-training, and (c) how important is domain (legal) specificity vs general (cross-domain) legal knowledge. Furthermore, often times, legal NLP relies on datasets without drawing clear lines and comparisons between the various legal systems they may reflect. A legal system may be defined as a set of rules adopted and enforced at a given governance level, which may be national, regional or international (Friedman and Hayden, 2017), e.g., UK, EU, US, CoE, etc.
We define the upstream evaluation as the task PLMs are explicitly designed to do: Masked Language Modelling (MLM) (Devlin et al., 2019). We then probe for specific legal concepts that are legalsystem specific, in a similar fashion as Petroni et al. (2019) did using the “LAnguage Models Analysis” (LAMA) framework. Finally, we assess the PLMs performance in LexGLUE (Chalkidis et al., 2022a) downstream tasks. More importantly, we explore how the aforementioned factors (upstream, and probing performance) interplay and relate to downstream performance. Our contributions are:
(a) We release LeXFiles, a new diverse English legal corpus including 11 sub-corpora that cover legislation and case law from 6 primarily English-speaking legal systems (EU, CoE, Canada, US, UK, India). The corpus comprises approx. 6 million documents which sum up to approx. 19 billion tokens.
(b) We release 2 new legal-oriented PLMs, dubbed LexLMs, warm-started from the RoBERTa (Liu et al., 2019) models, and further pre-trained on the LeXFiles for 1M additional steps.
15513
(c) We release LegalLAMA, a diverse probing benchmark suite comprising 8 sub-tasks that aims to assess the acquaintance of legal knowledge that PLMs acquired in pre-training.
(d) We evaluate 7 PLMs on both LeXFiles and LegalLAMA, analyzing their performance out of the box per LeXFiles sub-corpus and LegalLAMA tasks. We also fine-tune and evaluate these models in selected LexGLUE tasks, and examine the interplay between MLM, probing, and downstream performance.

heading: 2 LeXFiles Corpus
text: The LeXFiles is a new diverse English multinational legal corpus that we created including 11 distinct sub-corpora (Table 1) that cover legislation and case law from 6 primarily English-speaking legal systems (EU, CoE, Canada, US, UK, India). The corpus contains approx. 19 billion tokens. In comparison, the Pile of Law corpus released by Henderson* et al. (2022) comprises 32 billion in total, where the majority (26/30) of sub-corpora come from the United States of America (USA), hence the corpus as a whole is biased towards the US legal system in general, and the federal or state jurisdiction in particular, to a significant extent. The LeXFiles’s sub-corpora are: (a) EU Legislation. We release 93.7K EU laws
(regulations, decisions, directives) published in EUR-Lex, the website of the EU Publication Office.1
(b) EU Case Law. We release 29.8K EU court decisions, mainly issued from the Court of
1https://eur-lex.europa.eu/
Justice (CJEU), published in EUR-Lex.1
(c) UK Legislation. We release 52.5 UK laws published in UK.LEGISLATION.GOV.UK, the official website of the UK National Archives.2
(d) UK Case Law. We release 47K UK court decisions published in the British and Irish Legal Information Institute (BAILII) database.3
(e) US Legislation. We re-distribute 518 US state statutes (legislation) originally published by Henderson* et al. (2022).
(f) US Case Law. We release 4.6M US decisions (opinions) published by Court Listener,4 a web database hosted by the Free Law Project.5
(g) US Contracts. We release 622K US contracts (agreements) obtained from US Securities and Exchange Commission (SEC) filings, which are publicly available from the SEC-EDGAR6
database. (h) Canadian Legislation. We release 6K Cana-
dian laws (acts, regulations) published in the official legislation portal of Canada.7
(i) Canadian Case Law. We re-distribute 13.5K Canadian decisions (opinions) originally published by Henderson* et al. (2022).
(j) ECtHR Case Law. We release 12.5K decisions ruled by the European Court of Human rights
2https://www.legislation.gov.uk/ 3https://www.bailii.org/ 4https://www.courtlistener.com/ 5We release decisions published from 1965 on-wards (cf. post Civil Rights Act), as a hard threshold for cases that possibly rely on out-dated and discriminatory law standards. The rest of the sub-corpora include more recent documents.
6https://www.sec.gov/edgar 7https://laws-lois.justice.gc.ca/eng/
(ECtHR) published in HUDOC,8 the database of ECtHR.
(k) Indian Case Law. We include 34.8K Indian Supreme Court cases originally published by Malik et al. (2021).
The LeXFiles is pre-split into training and test subsets to provide a fair ground for comparing the performance of PLMs that have not been trained in the training set. We use the training subset of the LeXFiles corpus to train 2 new transformer-based languages models, dubbed LexLMs (Section 4.1), and evaluate their MLM performance across many other already available PLMs (Section 4.2).
3 LegalLAMA Benchmark
LAnguage Model Analysis (LAMA) (Petroni et al., 2019) is a probing task that is designed to assess specific capabilities of PLMs. The general framework of LAMA is to let PLMs predict a target token behind a [MASK] given its context, e.g., “Paris is the capital of [MASK]”, where the answer is ‘France’. LegalLAMA is a new probing benchmark suite inspired by this framework. It includes 8 sub-tasks that aim to assess the acquaintance of legal knowledge that PLMs acquired in the pretraining phase in a zero-shot fashion. Such tasks cannot be resolved by laypersons or even law professionals that are not experts in the specific fields of law in many cases.9 The acquaintance of legal knowledge can be interpreted as some form of primitive understanding of the law, for specific aspects in very controlled (limited) settings -limited legal concepts under a specific jurisdiction-. As Sahlgren and Carlsson (2021) mentioned:
“Rather than asking whether a language model understands or not, we should ask to what extent, and in which way, a model understands.”
We further extend the LAMA framework by allowing PLMs to predict multi-token targets. Take for example the “Drug Trafficking” offence under the “Drug-Related” crimes of the US legislation. Using the RoBERTa tokenizer, this term is split into two tokens, that is “Drug” and “Trafficking”. We replace thus the “drug trafficking” phrase with two [MASK] tokens, and then ask the model to predict these tokens simultaneously.
8https://hudoc.echr.coe.int/eng 9In Appendix A, we present a discussion on the Legal-
LAMA tasks’ level of difficulty.
We evaluate the overall performance of PLMs using the macro-averaged Mean Reciprocal Rank (MRR) (Voorhees and Tice, 2000) over the set of labels (not the entire vocabulary).10 In the case of multi-token targets, we average the MRR over the predicted tokens.11 Note that LegalLAMA examples come from the test subset of the related LexFiles sub-corpora in order to have a fair comparison between models trained or not on the LexFiles training sets. We provide a concrete example in Figure 1, and describe the tasks in detail:
ECHR Articles (CoE). In this task, we have paragraphs from the court assessment section of ECtHR decisions. We extract those paragraphs from the newly introduced ECHR corpus presented in Section 2. The paragraphs include references to ECHR articles, e.g., “Article [MASK] of the Convention”, where [MASK] is the article number. For example, “The applicant complained under Article [2] of the Convention that the prison authorities had failed to protect her son’s right to life by taking the necessary measures.” Given a paragraph, where the article number is masked, the model has to predict the associated article number given the context. The dataset is composed of 5,072 test instances containing on average 69 tokens and 13 unique article numbers to predict.
10We decided to report only MRR results in the main paper for the sake of clarity. Moreover, MRR avoids penalizing for near-identical outcomes. Detailed results including Precision at 1 (P@1) are available in Appendix C.
11A stricter evaluation would be to consider a multi-token prediction valid only if all the sub-tokens are properly predicted by the PLM. We decided to average the MRR to consider minor variations and errors.
Contractual Section Titles (US). In this task, we have sections from US contracts reusing the dataset of Tuggener et al. (2020). Contractual sections are usually numbered and titled, e.g., "10. [Arbitration]. Any controversy, dispute or claim directly or indirectly arising out of or relating to this Agreement [...]". The section titles reflect the content (subject matter) of the section, and are commonly re-used. Given a section, where the section title is masked, the model has to predict the associated title given the context. The dataset is composed of 1,527 test instances containing on average 85 tokens and 20 unique section titles to predict.
Contract Types (US). In this task, we have introductory paragraphs from US contracts. We extract those paragraphs from the newly introduced corpus of US contracts, presented in Section 2. Introductory paragraphs usually start with the contract title revealing the contract type, e.g., "Service Agreement", and follow with the names of the involved parties, and their roles in this agreement. For example, “This [Purchase] Agreement is entered into this 23rd day of January 2020 by and between A (the "Purchaser") and B (the "Seller").”. Given an introductory paragraph, where the contract type is masked, the model has to predict the associated type given the context. The task is composed of 1,089 test instances containing on average 150 tokens and 15 unique types of contracts to predict.
Crime Charges (US). In this task, we have paragraphs from US court judgments (opinions). We extract those paragraphs from the US case law corpus, presented in Section 2. We select a list of criminal offenses (e.g., “Sexual Assault”), categorized into 11 major categories (e.g., Sex-related) from the FindLaw website.12 We filter out paragraphs that refer the specified criminal charges verbatim. For example, “A person commits the crime of [burglary] in the first degree when he or she enters or remains unlawfully in a building with the intent to commit a crime against a person or property therein” Given a paragraph, where a criminal charge is masked, the model has to predict the associated criminal charge given the context. The task is composed of 4,518 test instances containing on average 118 tokens and 59 charges to predict.
Legal Terminology (US). In this task, we have paragraphs from US court judgments (opinions).
12https://www.findlaw.com/criminal/ criminal-charges.html
We extract those paragraphs from the US case law corpus, presented in Section 2. We select a subset of legal terms per legal topic (e.g., finance law, property law, family law) using the legal vocabularies provided by the Legal Information Institute (LII) of the Cornell Law School.13 We filter out paragraphs that use the specified legal terms. For example, “The [marital privilege] against selfincrimination is [...] grounded upon the theory that just as one may not be convicted by his own compelled testimony, so may he not be convicted by the testimony of his spouse.” Given a paragraph, where a legal term is masked, the model has to predict the associated legal term given the context. The task is composed of 5,829 test instances containing on average 308 tokens and 92 legal terms from 7 topics to predict.
Legal Terminology (EU). In this task, we have paragraphs from CJEU judgments (opinions). We extract those paragraphs from the newly introduced EU case law corpus, presented in Section 2. We select a subset of legal terms based on the subject matters provided by the database of the courts (CURIA).14 We filter out paragraphs that use the specified legal terms. For example, “The guiding principle at the basis of EU [data protection] law is that of a self-determined decision of an individual who is capable of making choices about the use and processing of his or her data.” Given a paragraph, where a legal term is masked, the model has to predict the associated legal term given the context. The task is composed of 2,127 test instances containing on average 164 tokens and 42 legal terms from 23 topics to predict.
Legal Terminology (CoE). In this task, we have paragraphs from ECtHR decisions. We extract those paragraphs from the newly introduced ECHR corpus presented in Section 2. We select a subset of legal terms (legal issues) based on the keywords provided by the database of the courts (HUDOC).15 We filter out paragraphs that use the specified legal terms. For example, “The applicants alleged that their relatives’ [right to life] was violated in that they were deliberately killed by village guards.” Given a paragraph, where a legal term is masked, the model has to predict the associated legal term given the context. The task is composed of 6,803
13https://www.law.cornell.edu/ 14https://curia.europa.eu/ 15https://www.echr.coe.int/Documents/HUDOC_
Keywords_ENG.pdf
test instances containing on average 97 tokens and 250 legal terms from 15 articles to predict.
Criminal Code Sections (Canada). In this task, we have paragraphs from the Criminal Court of Canada’s decisions containing Section Numbers of the Criminal Code of Canada (CCC)16. For example, “Section [680] of the Criminal Code provides that a bail review is to be conducted by a panel of this court where directed by the Chief Justice.” Given a paragraph, where a criminal code’s section is masked, the model has to predict the associated section number, paragraph, and sub-paragraph (if any) given the context. The task is composed of 321 test instances containing on average 72 tokens and 144 different section numbers to predict.
In Appendix D, we present the full list of vocabulary (masked terms) grouped in categories (clusters) -when applicable- per LegalLAMA sub-task.

heading: 4 Experiments

heading: 4.1 Pre-trained Language Models
text: We consider 7 large language models to assess their performance with respect to the upstream (MLM), probing, and downstream evaluation:
RoBERTa (Base/Large) are the original RoBERTa models (Liu et al., 2019) trained for 64k steps with very large batches on generic corpora; thus do not have any clear legal prior (knowledge).
LegalBERT (Base) is a legal-oriented BERT model (Devlin et al., 2019) released by Chalkidis et al. (2020) trained for 1M steps on legal corpora from EU, UK, CoE, and USA.
CaseLawBERT (Base) is another legal-oriented BERT released by Zheng et al. (2021). CaseLawBERT (which we will refer to as CL-BERT henceforth) is trained from scratch for 2M steps on the Harvard Law case corpus, which comprises 3.4M legal decisions from US federal and state courts.
16https://laws-lois.justice.gc.ca/eng/acts/ c-46/index.html
PoL-BERT (Large) is a legal-oriented RoBERTa model released by Henderson* et al. (2022) trained from scratch for 2M steps on the Pile of Law, a corpus consisting of approx. 256GB of English, mainly US, language legal and administrative text.
LexLM (Base/Large) are our newly released RoBERTa models. We follow a series of bestpractices in language model development:
(a) We warm-start (initialize) our models from the original RoBERTa checkpoints (base or large) of Liu et al. (2019).
(b) We train a new tokenizer of 50k BPEs, but we reuse the original embeddings for all lexically overlapping tokens (Pfeiffer et al., 2021).
(c) We continue pre-training our models on the diverse LeXFiles (Section 2) corpus for additional 1M steps with batches of 512 samples, and a 20/30% masking rate (Wettig et al., 2023), for base/large models, respectively.
(d) We use a sentence sampler with exponential smoothing of the sub-corpora sampling rate following Conneau et al. (2019) since there is a disparate proportion of tokens across subcorpora (Table 1) and we aim to preserve percorpus capacity (avoid overfitting).
(e) We consider mixed cased models, similar to all recently developed large PLMs.
Additional details on LexLM models pre-training can be found in Appendix B.

heading: 4.2 Upstream Evaluation
text: In Table 3, we present the upstream (MLM) performance for all PLMs across the LeXFiles subcorpora. The performance is measured in terms of accuracy, i.e. Precision@1 of the masked token to be predicted. The accuracy is thus averaged over all the masked tokens for each task. We also provide the average across all tasks, per model. We observe that results vary across models trained in very different settings (model’s capacity, pre-
training corpora), while the results also vary across legal sub-corpora.
We want to remind the reader that the upstream evaluation offers a rough idea of a model’s capabilities since it relies on random masked sub-words, in which case many of those can be generic and thus highly predictable (e.g. preposition “of”). This phenomenon further motivates the construction of the LegalLAMA benchmark, in which case only “legal knowledge sensitive” words have been masked.
Type of Documents. In terms of differences across sub-corpora, we observe that the performance on legislation is better compared to case law in 3/4 legal systems, where we have both (EU, UK, US, Canada), with US contractual language being the most predictable for the models which have been trained on it (LexLMs, LegalBERT).
Comparison of PLMs. Overall, the large LexLM model outperforms the rest, being 3% more accurate on average compared to the 2nd best models (base versions of LexLM, and LegalBERT). Such results are expected since LexLMs have been trained in a diverse corpus, similarly to LegalBERT, compared to CL-BERT, and PoL-BERT, which have been trained on US corpora. Overspecialization harms the two US-centric models in a great extend since they are outperformed even from the generic RoBERTa models.
We also observe that LegalBERT outperforms the similarly-sized LexLM in specific sub-corpora (Both EU, UK legislation, ECtHR case law, and US
Contracts) that were included in its training. We hypothesize that these results are related to the pretraining data diversity, since LexLMs have been trained in a more diverse corpus including many more documents from different legal systems with a sampling smoothing to preserve capacity per subcorpus. The larger LexLM model has the capacity to cover all sub-corpora to a greater detail.
In general, larger models pre-trained on the same corpora (RoBERTas, LexLMs) perform better compared to smaller ones, but in-domain pre-training is a much more important factor for upstream performance, e.g., LegalBERT outperforms RoBERTa-L.

heading: 4.3 Probing Evaluation
text: In Table 4, we present the results across all examined PLMs on LegalLAMA. We analyze the results from two core perspectives: the prior knowledge and the probing task.
Prior Knowledge. The pre-training corpus has a significant impact on the probing performance. RoBERTa models, having little to no legal prior, were expected to achieve worst performance on all probing tasks. Surprisingly, CL-BERT and PoLBERT achieve on-par or sometimes worst performance than RoBERTa (Base & Large) in most tasks. Being trained on the “Harvard Law Case” corpus (CL-BERT) and the Pile of Law (PoL-BERT), we would have expected better performance than a model without legal prior. Their pre-training corpora might be lacking diversity, which might cause their poor performance even on Legal-US probing
tasks. LegalBERT (Base), being trained on UK, EU and USA data illustrates important improvement over models without legal prior (RoBERTa) or having only US legal prior (CaseLaw and PoLBERT). LexLM models, being trained on the new LeXFiles dataset, show performance improvement over LegalBERT across all tasks, especially on the task of predicting Section Numbers of the Criminal Code of Canada. Regarding the size of the model, we are able to compare the cased versions of RoBERTa Base/Large and LexLM Base/Large. As expected, the larger versions offer better performance than the smaller ones on every task.
Probing Tasks. We characterize the difficulty of the tasks by their semantic level, the output space (the number of labels to predict from), and the label complexity (how many tokens per label). We expose the tasks’ different characteristics in Table 4. Given the best-performing model (LexLM-L), we can see that Crime Charges and Legal Terminology (US and EU) are the hardest tasks to solve. Looking at Table 4, we can see that these three tasks are characterized by a higher label complexity (>2).
We further demonstrate the label complexity impact in Figure 2. The output space does not seem to have a correlation with the models’ performance, since the selected Legal Terminology Topic Clusters (US) has only 7 possible labels, whereas the Criminal Code Section (Canada) has 144 possible labels. Finally, Crime Charges, being the hardest task to solve, has on average 118 tokens as input and 59 possible labels with moderate complexity, similar to the Terminology tasks (EU and CoE). This suggests that the difficulty of the task is not only driven by the labels’ complexity but may rather lie in the lack of contextualization. Take for example the following sentence:
“This case involves perhaps the first prosecution under New York’s new [computer crime] statute, Penal Law article 156, which went into effect on November 1, 1986, just days before the incidents charged herein.”
The only contextual hint the PLMs have to predict the correct tokens ([computer crime]) is the utterance “Penal Law article 156, which went into effect on November 1, 1986”. This is the opposite task of predicting article numbers given a context, which is much more difficult than predicting the actual context because the output space is larger.17

heading: 4.4 Downstream Evaluation
text: For downstream evaluation, we conduct experiments for 6 legal classification tasks, 5 part of LexGLUE (Chalkidis et al., 2022a), covering US contracts, US, EU, and ECHR law.
ECtHR (Task B) (Chalkidis et al., 2021b) is a multi-label topic classification task, where given
17The actual tokens predicted by the best-performing examined PLM were “sexual” and “abuse”.
the facts of an ECtHR case, the model has to predict the alleged violated ECHR article among 10 such articles (e.g., “Art 3. - Prohibition of Torture”, “Art. 6 - Right to Fair Trial”).
LEDGAR (Tuggener et al., 2020) is a single-label multi-class topic classification task, where given a contractual paragraph, the model has to predict one of the correct topic among 100 topics (e.g., “Limitation of Liability”, “Arbitration”).
ContractNLI (Koreeda and Manning, 2021) is a contract-based Natural Language Inference (NLI) task, where given an Non-Disclosure Agreement (NDA) and one out 17 templated hypotheses (e.g., “The Party may share some Confidential Information with some third-parties.”), the model has to predict if the hypothesis is (entailed, contradicted, or is neutral) to the terms of the NDA.
SCOTUS (Chalkidis et al., 2022a) is a single-label multi-class topic classification task, where given a Supreme Court of US (SCOTUS) opinion, the model has to predict the relevant area among 14 issue areas (e.g., “Civil Rights”, “Judicial Power”).
CaseHOLD (Zheng et al., 2021) is a multiple choice QA classification task, where given a paragraph from a US legal opinion where a legal rule (holding) is masked, the model has to predict the applicable rule among 5 alternatives (the correct one and 2 irrelevant presented in other cases).
EURLEX (Chalkidis et al., 2021a) is a multi-label topic classification task, where given an EU law, the model has to predict the correct EUROVOC concept among hundred concepts (e.g., “Environmental Policy”, “International Trade”).
We fine-tune all examined PLMs (Section 4.1)
for a single epoch with a learning rate of 1e−5 leading to a small number of updates. We are interested to examine how fast each model convergence based on its prior knowledge; in other words, what can a model learn in a single pass over training data? Finetuning models for many epochs over large datasets will eventually lead to a full re-parameterization of the models, in which case the importance of prior knowledge will diminish compromise the goal of our study (Figure 3).18
For all tasks, we use standard N-way classifiers with a classification head (Devlin et al., 2019). For ECtHR, and SCOTUS, involving long documents, we warm-start Longformer (Beltagy et al., 2020) models from each PLM’s parameters to encode up to 2048 tokens. We evaluate classification performance with micro-F1 (µF1) and macro-F1 (mF1) across tasks following Chalkidis et al. (2022a).
Results In Table 5, we present the test results across all tasks/datasets. We analyze the results from two perspectives: model’s capacity (size), and prior legal knowledge abducted via pre-training.
18In most tasks, models fully converge after approx. 5 epochs with improved performance, and the relative differences between generic and legal-oriented models are diminished (Chalkidis et al., 2022a).
Model’s capacity (size) strongly correlates with the overall downstream performance. Across all tasks, there are 2/6 exceptions (CNLI and CaseHOLD) where LegalBERT outperforms larger PLMs. Both tasks are using sentence pairs, a setup used in BERT’s pre-training, but not in RoBERTa, which may bring LegalBERT, a BERT-based model, in a better initial condition co-considering the minimal updates steps, compared to all large models following the RoBERTa pre-training setup, which do no use pairs of sentences or optimized based on a sentence-level objective (NSP).
Legal Knowledge also plays an important role following the model’s capacity (size). We observe that LexLM-B trained in the diverse LeXFiles corpus outperforms the equally-sized RoBERTa-B model in 5/6 tasks, while LegalBERT and CL-BERT outperform it only in 3 out of 6 tasks. In this case, the results are mixed, i.e., acquaintance of legal knowledge as expressed by upstream (Section 4.2) and probing (Section 4.3) performance does not correlate with downstream performance.
In the case of large-sized models, LexLM-L outperform RoBERTa-L across all tasks, while PoLBERT trained on the US-biased Pile of Law corpus is outperformed by RoBERTa-L in 5 out of 6 tasks. Given the results with respect to upstream and probing performance, RoBERTa-L has a better legal prior; so in these regards, acquaintance of legal knowledge fully correlates with downstream performance in the large models’ regime.

heading: 5 Release of Resources
text: We release our code base to assure reproducibility and let others extend our study by experimenting with other PLMs, or develop new ones.19 The new LexLM models (Section 4.1), the LeXFiles corpus 20 (Section 2), and the LegalLAMA benchmark 21 (Section 4.3) are available on Hugging Face Hub (Lhoest et al., 2021).22

heading: 6 Conclusions and Future Work
text: In this work, we introduced a multinational English legal corpus (LeXFiles) and a legal knowledge probing benchmark (LegalLAMA) to facilitate training and detailed analysis of legal-oriented
19https://github.com/coastalcph/lexlms 20https://huggingface.co/datasets/lexlms/lex_
files 21https://huggingface.co/datasets/lexlms/ legal_lama 22https://huggingface.co/lexlms
PLMs. We also released two new legal PLMs and evaluate them alongside others on LegalLAMA and LexGLUE. Based on our analysis (Section 4), we make the following general observations:
(a) The use of diverse legal corpora leads to better overall upstream performance (Section 4.2). (b) We find that probing performance strongly correlates with upstream performance in related legal topics (Section 4.3). (c) For both upstream, and probing performance, the selection of pre-training corpora has a much larger effect compared to model’s capacity (Sections 4.2-4.3). Nonetheless, larger models pre-trained on similar corpora have better overall performance. (d) Downstream performance is mainly driven by the model’s capacity and prior legal knowledge which can be estimated by upstream and probing performance (Section 4.4).
In future work, we plan to further analyze the learning dynamics of legal language models by comparing their representations with representations derived from legal knowledge bases. Given the availability of the new resources, the development of instruction-following (Wei et al., 2021) fine-tuned legal-oriented GPT-like (Ouyang et al., 2022) models is also an anticipated direction.
Limitations
Diversity of Corpora While the newly introduced LeXFiles corpus is significantly more diverse compared to the Pile of Law corpus of Henderson* et al. (2022), it is still an English-only corpus covering only 6 legal systems (EU, UK, CoE, US, India, Canada). Despite, the fact that we can train better models (LexLMs) and evaluate these models across these corpora, in future work, we should extend our analysis to cover even more languages and legal systems, and a higher granularity in the labeling of legal fields within these systems. Not only will this help support the inclusion of other legal traditions but also adding more linguistic and cultural diversity will help us better understand the robustness of existing methods.
Similarly, the newly introduced LegalLAMA benchmark consists of 8 sub-tasks targeting EU, ECHR, US, and Canadian jurisdictions in a very controlled setting; where examples were automatically extracted. While on this benchmark, legaloriented PLMs has demonstrated a significant degree of “understanding" of legal language and legal
topics, this benchmark should be further expanded with more sub-tasks to evaluate the acquaintance of legal knowledge across more legal systems and topics, and possibly cleansed from both very easy and unsolvable examples.
Model Considerations In this work, we consider encoder-only (BERT-like) models up to approx. 350M parameters, while recent work on the development of Large Language Models (LLMs) (Kaplan et al., 2020; Brown et al., 2020; Hoffmann et al., 2022; Chowdhery et al., 2022) is mainly targeting billion-parameter-sized models (10-100Bs of parameters) that usually follow a decoder-only, e.g., GPT (Radford and Narasimhan, 2018), or encoder-decoder, e.g., T5 (Raffel et al., 2020), architecture. Moreover, new paradigms of training PLMs have been introduced, such as instructionbased finetuning (Wei et al., 2021), and alignment via Reinforcement Learning from Human Feedback (RLHF) (Stiennon et al., 2020; Ouyang et al., 2022). Latest GPT models (Ouyang et al., 2022) have recently shown significant zero-shot progress on law-related tasks such as bar examination question answering (Katz et al., 2023). Thus, future work should follow the most recent advances by pre-training much larger auto-regressive GTP-like models that seem to lead to emergent zero-shot and few-shot capabilities.
Evaluation Considerations In Section 3, we present how we account for and evaluate multitoken expressions (terms) on the LegalLAMA benchmark; we are open to ideas on how we should possibly improve the current approach to provide a fairer and more robust evaluation framework across all models. Similarly, in Section 4.4, we fine-tune all examined PLMs for a single epoch to avoid extreme over-reparameterization and better estimate how model’s knowledge affects convergence and performance. Nonetheless, there are possibly better approaches to control for these aspects, e.g., Adapter-based (Rücklé et al., 2021) finetuning, or other approaches, such as LoRA (Hu et al., 2022).
Beyond Performance While we consider a multi-facet analysis, we do not cover other interesting dimensions that should also be explored, especially since law is a very sensitive application domain; for instance trustworthiness-related topics, such as model interpretability (Chalkidis et al., 2021b; Malik et al., 2021), and fairness (Chalkidis et al., 2022b). Future work can build from the
results reported herein to explore these important topics.
Ethics Statement
The scope of this work is to examine the performance of legal-oriented PLMs from a multi-facet perspective and broaden the discussion to help practitioners build assisting technology for legal professionals and laypersons. We believe that this is an important application field, where research should be conducted (Tsarapatsanis and Aletras, 2021) to improve legal services and democratize law, while also highlighting (informing the audience on) the various multi-aspect shortcomings seeking a responsible and ethical (fair) deployment of legaloriented technologies.
In this direction, we introduce new resources covering various legal systems to build new models that better represent law and better assess their capabilities. All newly developed and published resources are based on publicly available data, most of them scattered on several web portals.

heading: Acknowledgments
text: This work was partly funded by the Innovation Fund Denmark (IFD, https: //innovationsfonden.dk/en) and the Fonds de recherche du Québec – Nature et technologies (FRQNT, https://frq.gouv.qc.ca/ nature-et-technologies/).

heading: A LegalLAMA Discussion
text: The LegalLAMA tasks cannot be resolved by laypersons or even law professionals that are not experts in the specific fields of law in many cases. Another consideration that often goes unspecified is that expertise is legal system-specific (e.g. US law differs widely from EU law), as do the distinctions between the academic and the practical knowledge of law (including potential sub-distinctions between different types of legal practitioners, e.g. litigation experts, contract drafting experts, due dilligence experts, etc.). Lastly, it is also important to note that legal systems can be clustered according to similarities or differences. Specifically:
• For task ‘ECHR Articles’, both laypersons and lawyers who are not experts in human rights law (particularly ECHR) would perform at random chance level, since they lack knowledge of the ECHR in an article level. Providing the titles of the articles (Table 6), we can expect improved performance in case of rich context. Generally, the same can be said for the related task ‘Legal Terminology (CoE)’. Legal terminology is very particular to individual legal systems, and predicting the place of legal concepts within the ECHR would require a very high level of specialization. • For task ‘Contractual Section Titles (US)’,
structural knowledge of US contracts would be necessary for the performance of this task with a high degree of accuracy. This is due to the fact that contracts often have some structural similarities, but also particular characteristics depending on the type of contract (e.g. employment, sale, credit). Laypersons would perform this task at random chance
level. Practicing lawyers with contract drafting expertise would potentially have the highest performance in this task. Non-US lawyers with no contract drafting expertise would perform slightly higher than random chance level. The same considerations apply to the task ‘Contract Types (US)’. • For tasks ‘Crime Charges (US)’ and ‘Crim-
inal Code Sections (Canada)’, both laypersons and lawyers who are not experts in criminal law (particularly US law and Canadian law) would perform at random chance level, since the legal concepts are very specific (e.g. manslaughter). Improved performance could be seen in cases where the masked terms are specifically defined. • For tasks ‘Legal Terminology (US)’ and ‘Le-
gal Terminology (EU)’, the same discussion as above is applicable. Legal terminology is system-specific. There may be similar terms, but in the absence of knowledge relating to how such similarities may be interpreted, a non-expert lawyer would not perform such a task with a very high accuracy level.
A.1 ECtHR Articles
We hereby provide details on the 13 ECtHR articles;

heading: B LexLM Pre-training Details
text: For the newly released, LexLM models (LexLMs), we followed a series of best-practices in language model development literature:
(a) We warm-start (initialize) our models from the original RoBERTa checkpoints (base or large) of Liu et al. (2019). Model recycling
is a standard process followed by many (Wei et al., 2021; Ouyang et al., 2022) to benefit from starting from an available “well-trained” PLM, instead from scratch (random).
(b) We train a new tokenizer of 50k BPEs based on the training subsets of LeXFiles to better cover legal language across all covered legal systems. Although, we reuse the original RoBERTa embeddings for all lexically overlapping tokens (Pfeiffer et al., 2021), i.e., we warm-start word embeddings for tokens that already exist in the original RoBERTa vocabulary, and use random ones for the rest.
(c) We continue pre-training our models on the diverse LeXFiles (Section 2) corpus for additional 1M steps with batches of 512 samples. We do initial warm-up steps for the first 5% of the total training steps with a linearly increasing learning rate up to 1e−4, and then follow a cosine decay scheduling, following recent trends. For half of the warm-up phase (2.5%), the Transformer encoder is frozen, and only the embeddings, shared between input and output (MLM), are updated. We also use an increased 20/30% masking rate, where also 100% of the predictions are based on masked tokens, compared to Devlin et al. (2019)23 for base/large models respectively, based on the findings of Wettig et al. (2023).
(d) For both training the tokenizer and the LexLM models, we use a sentence sampler with exponential smoothing of the sub-corpora sampling rate following Conneau et al. (2019) and
23Devlin et al. –and many other follow-up work– used a 15% masking ratio, and a recipe of 80/10/10% of predictions made across masked/randomly-replaced/original tokens.
Raffel et al. (2020), since there is a disparate proportion of tokens across sub-corpora (Table 1) and we aim to preserve per-corpus capacity, i.e., avoid overfitting to the majority (approx. 94% of the total number or tokens) US-origin texts.
(e) We consider mixed cased models, similar to all recently developed large PLMs (Liu et al., 2019; Raffel et al., 2020; Brown et al., 2020).
We make LexLM models (base/large) publicly available alongside all intermediate checkpoints every 50k training steps on Hugging Face Hub.24

heading: C Detailed Legal-LAMA results per tasks
text: Table 7 contains the same results as in Table 4 with the addition of Precision@1 scores (P@1). The reason why we decided to only present MRR results in the main paper is that the difference between MRR and P@1 does not change the ranking of the models, and P@1 does not account for minor variations in predictions.
For each task, we display detailed results per predicted terms for each model. Table 8 contains results on the 13 article numbers from the ECHR task. Table 9 contains results on the 20 clause types from the Contract Section task. Table 10 contains results on the 16 types of contracts from the Contract Section task. Table 11 contains results on the 11 topics from the Crime Charges (US) task. Each topic contains multiple labels. Table 12 contains results on the 7 topics from the Terminology (US) task. Each topic contains multiple labels. Table 13
24https://huggingface.co/lexlms
contains results on the 23 topics from the Terminology (EU) task. Each topic contains multiple labels. Table 14 contains results on the 12 articles from the Terminology (CoE) task. Each article contains multiple labels. Table 15 contains results on the 43 sections from the Criminal Code Sections (Canada) task.

heading: D LegalLAMA Tasks’ Vocabulary
text: In Tables 8, 9, 10, 13, and 15 we present the labels’ list for the ‘ECHR Articles’, ‘Contract Sections’, ‘Contract Types’, ‘Terminology (EU)’ and ’Criminal Code Sections (Canada)’ sub-tasks and the label-wise performance. In Tables 16, 17, and 18, we present the labels’ list for the ‘Terminology (CoE)’, ‘Crimes Charges (US)’, and ‘Terminology (US)’ sub-tasks grouped in clusters.

heading: 839 0.17 0.25 0.07 0.20 0.33 0.37 0.33 0.36 0.33 0.35 1.00 1.00 1.00 1.00

heading: 784 0.20 0.29 0.30 0.49 0.50 0.52 0.38 0.46 0.50 0.52 1.00 1.00 1.00 1.00

heading: 718.2 0.20 0.30 0.17 0.31 0.52 0.59 0.52 0.57 0.59 0.64 0.76 0.85 0.87 0.92

heading: 718.1 0.17 0.26 0.08 0.24 0.67 0.67 0.67 0.67 0.33 0.48 1.00 1.00 1.00 1.00

heading: 715.1 0.12 0.25 0.12 0.22 0.33 0.50 0.33 0.45 0.50 0.56 1.00 1.00 1.00 1.00

heading: 687 0.20 0.30 0.30 0.49 0.62 0.64 0.38 0.51 0.50 0.53 0.88 0.94 0.75 0.83

heading: 686 0.21 0.28 0.28 0.36 0.57 0.65 0.65 0.68 0.43 0.55 0.68 0.79 0.94 0.96

heading: 684 0.35 0.43 0.60 0.72 0.25 0.51 0.25 0.34 0.25 0.27 1.00 1.00 1.00 1.00

heading: 683 0.11 0.26 0.18 0.30 0.48 0.52 0.52 0.57 0.48 0.54 0.81 0.88 0.90 0.94

heading: 676 0.14 0.29 0.14 0.27 0.50 0.62 0.57 0.66 0.36 0.55 0.93 0.94 1.00 1.00

heading: 672.78 0.27 0.34 0.34 0.43 0.42 0.46 0.50 0.55 0.42 0.48 0.83 0.92 1.00 1.00

heading: 672.73 0.25 0.30 0.25 0.32 0.33 0.34 0.33 0.34 0.33 0.38 0.67 0.71 1.00 1.00

heading: 650 0.64 0.70 0.72 0.77 0.78 0.80 0.69 0.74 0.75 0.76 0.97 0.99 0.97 0.99

heading: 601 0.58 0.62 0.58 0.64 0.86 0.89 0.79 0.81 0.29 0.49 0.86 0.93 0.86 0.93

heading: 591 0.13 0.31 0.25 0.36 0.61 0.71 0.52 0.58 0.51 0.60 0.87 0.93 0.87 0.92

heading: 530 0.00 0.01 0.00 0.01 0.00 0.01 0.00 0.02 0.00 0.03 1.00 1.00 1.00 1.00

heading: 495 0.32 0.40 0.32 0.46 0.60 0.66 0.56 0.61 0.60 0.65 0.77 0.87 0.87 0.92

heading: 467.1 0.29 0.41 0.57 0.75 0.67 0.76 0.33 0.64 0.58 0.70 1.00 1.00 1.00 1.00

heading: 465 0.50 0.63 0.75 0.76 0.50 0.63 0.38 0.54 0.75 0.75 1.00 1.00 1.00 1.00

heading: 462.37 0.40 0.49 0.40 0.52 0.65 0.69 0.67 0.70 0.65 0.69 0.78 0.80 0.81 0.87

heading: 380 0.24 0.35 0.24 0.36 0.39 0.47 0.47 0.53 0.35 0.48 0.78 0.80 0.71 0.73

heading: 279 0.83 0.86 0.92 0.92 0.75 0.81 0.83 0.88 0.83 0.86 1.00 1.00 0.92 0.96

heading: 268 0.45 0.54 0.25 0.41 0.35 0.44 0.35 0.44 0.40 0.49 0.50 0.65 0.75 0.86

heading: 267.8 0.47 0.54 0.56 0.62 0.66 0.69 0.56 0.63 0.60 0.66 0.83 0.87 0.83 0.88

heading: 267.5 0.67 0.78 0.75 0.85 0.83 0.90 0.67 0.76 0.50 0.62 1.00 1.00 1.00 1.00

heading: 267.12 0.33 0.53 0.33 0.44 0.67 0.77 0.75 0.84 0.75 0.80 1.00 1.00 1.00 1.00

heading: 264 0.67 0.67 0.50 0.56 0.50 0.59 0.42 0.51 0.25 0.38 0.92 0.96 1.00 1.00

heading: 254 0.50 0.61 0.65 0.73 0.50 0.58 0.40 0.52 0.50 0.59 0.75 0.85 0.85 0.92

heading: 249 0.40 0.45 0.33 0.41 0.60 0.68 0.66 0.74 0.53 0.66 0.87 0.91 0.88 0.90

heading: 231 0.25 0.29 0.38 0.54 0.62 0.65 0.44 0.51 0.56 0.59 0.94 0.94 1.00 1.00

heading: 163.1 0.33 0.40 0.44 0.57 0.67 0.68 0.33 0.51 0.33 0.46 1.00 1.00 1.00 1.00

heading: 163 0.50 0.52 0.50 0.51 0.50 0.51 0.50 0.51 0.50 0.52 0.50 0.75 1.00 1.00

heading: 152 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.50 0.75 1.00 1.00 1.00 1.00

heading: 151 0.59 0.61 0.89 0.91 0.62 0.64 0.04 0.34 0.02 0.32 0.91 0.92 0.91 0.92

heading: 145 0.25 0.38 0.25 0.40 0.44 0.51 0.38 0.50 0.50 0.55 0.62 0.71 0.88 0.90

heading: 122 0.17 0.19 0.11 0.15 0.17 0.18 0.12 0.15 0.12 0.16 0.50 0.67 0.83 0.86

heading: 95 0.48 0.54 0.52 0.56 0.52 0.55 0.46 0.52 0.45 0.49 0.79 0.84 0.80 0.85

heading: 88.23 0.25 0.34 0.33 0.38 0.33 0.38 0.33 0.39 0.33 0.42 0.33 0.40 0.33 0.38

heading: 87 0.75 0.78 0.50 0.62 0.75 0.79 0.50 0.65 0.75 0.82 0.75 0.83 0.75 0.80

heading: 86 0.38 0.53 0.38 0.50 0.50 0.62 0.50 0.54 0.50 0.53 0.50 0.71 0.50 0.66

heading: 85 0.46 0.51 0.31 0.42 0.30 0.41 0.29 0.37 0.30 0.39 0.40 0.52 0.57 0.69

heading: 21 0.23 0.41 0.37 0.47 0.46 0.56 0.43 0.56 0.44 0.55 0.94 0.96 0.97 0.99

heading: 16 0.00 0.08 0.00 0.04 0.00 0.04 0.00 0.10 0.00 0.08 0.50 0.62 1.00 1.00
text: ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Left blank.
3 A2. Did you discuss any potential risks of your work? Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Sections 2-3
3 B1. Did you cite the creators of artifacts you used? Section 2
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? Section 4
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: In future work, we plan to further analyze the learning dynamics of legal language models by comparing their representations with representations derived from legal knowledge bases. Given the availability of the new resources, the development of instruction-following (Wei et al. , 2021) fine-tuned legal-oriented GPT-like (Ouyang et al. , 2022) models is also an anticipated direction. Limitations
Diversity of Corpora While the newly introduced LeXFiles corpus is significantly more diverse compared to the Pile of Law corpus of Henderson* et al. (2022), it is still an English-only corpus covering only 6 legal systems (EU, UK, CoE, US, India, Canada). Despite, the fact that we can train better models (LexLMs) and evaluate these models across these corpora, in future work, we should extend our analysis to cover even more languages and legal systems, and a higher granularity in the labeling of legal fields within these systems. Not only will this help support the inclusion of other legal traditions but also adding more linguistic and cultural diversity will help us better understand the robustness of existing methods. Similarly, the newly introduced LegalLAMA benchmark consists of 8 sub-tasks targeting EU, ECHR, US, and Canadian jurisdictions in a very controlled setting; where examples were automatically extracted. While on this benchmark, legaloriented PLMs has demonstrated a significant degree of “understanding" of legal language and legal
topics, this benchmark should be further expanded with more sub-tasks to evaluate the acquaintance of legal knowledge across more legal systems and topics, and possibly cleansed from both very easy and unsolvable examples. Model Considerations In this work, we consider encoder-only (BERT-like) models up to approx. 350M parameters, while recent work on the development of Large Language Models (LLMs) (Kaplan et al. , 2020; Brown et al. , 2020; Hoffmann et al. , 2022; Chowdhery et al. , 2022) is mainly targeting billion-parameter-sized models (10-100Bs of parameters) that usually follow a decoder-only, e. g. , GPT (Radford and Narasimhan, 2018), or encoder-decoder, e. g. , T5 (Raffel et al. , 2020), architecture. Moreover, new paradigms of training PLMs have been introduced, such as instructionbased finetuning (Wei et al. , 2021), and alignment via Reinforcement Learning from Human Feedback (RLHF) (Stiennon et al. , 2020; Ouyang et al. , 2022). Latest GPT models (Ouyang et al. , 2022) have recently shown significant zero-shot progress on law-related tasks such as bar examination question answering (Katz et al. , 2023). Thus, future work should follow the most recent advances by pre-training much larger auto-regressive GTP-like models that seem to lead to emergent zero-shot and few-shot capabilities. Evaluation Considerations In Section 3, we present how we account for and evaluate multitoken expressions (terms) on the LegalLAMA benchmark; we are open to ideas on how we should possibly improve the current approach to provide a fairer and more robust evaluation framework across all models. Similarly, in Section 4. 4, we fine-tune all examined PLMs for a single epoch to avoid extreme over-reparameterization and better estimate how model’s knowledge affects convergence and performance. Nonetheless, there are possibly better approaches to control for these aspects, e. g. , Adapter-based (Rücklé et al. , 2021) finetuning, or other approaches, such as LoRA (Hu et al. , 2022). Beyond Performance While we consider a multi-facet analysis, we do not cover other interesting dimensions that should also be explored, especially since law is a very sensitive application domain; for instance trustworthiness-related topics, such as model interpretability (Chalkidis et al. , 2021b; Malik et al. , 2021), and fairness (Chalkidis et al. , 2022b). Future work can build from the
results reported herein to explore these important topics