MultiLegalPile: A 689GB Multilingual Legal Corpus (2024)

Abstract: Large, high-quality datasets are crucial for training Large Language Models (LLMs). However, so far, few datasets are available for specialized critical domains such as law and the available ones are often small and only in English. To fill this gap, we curate and release MULTILEGALPILE, a 689GB corpus in 24 languages from 17 jurisdictions. MULTILEGALPILE includes diverse legal data sources and allows for pretraining NLP models under fair use, with most of the dataset licensed very permissively. We pretrain two RoBERTa models and one Longformer multilingually, and 24 monolingual models on each of the languagespecific subsets and evaluate them on LEXTREME. Additionally, we evaluate the English and multilingual models on LexGLUE. Our multilingual models set a new SotA on LEXTREME and our English models on LexGLUE. We release the dataset, trained models, and all code under the most open licenses possible.

heading: 1 Introduction
text: Recent years have seen LLMs achieving remarkable progress, as demonstrated by their performance on various benchmarks such as SuperGLUE (Wang et al., 2019), MMLU (Hendrycks et al., 2021), and several human Exams (OpenAI, 2023), including U.S. bar exams for admission to practice law (Katz et al., 2023). These models are typically trained on increasingly large corpora, such as the Pile (Gao et al., 2020a), C4 (Raffel et al., 2020), and mC4 (Xue et al., 2021). However, public corpora available for training these models are predominantly in English, and often constitute web text with unclear licensing. This even led to lawsuits against LLM producers1, highlighting this critical issue. Furthermore, there is a scarcity of largescale, domain-specific pretraining corpora, which
1 https://www.theverge.com/2022/11/8/23446821/
microsoft-openai-github-copilot-class-action-lawsuit-ai-copyright-violation-training-data
constitutes a significant gap in the current body of resources available for the training of LLMs. We find that only one in every thousand document in mC4 contains legal text. Similarly, LLMs are predominantly English, especially considering domain-specific models, e.g., ones specialized in biomedical, legal, or financial texts.
Legal texts, often produced by public instruments (e.g., state governments, international organizations), are typically available under public licenses, offering a rich resource for domain-specific pretraining. Given this context, we curate a massive, openly available, corpus of multilingual law text spanning across numerous jurisdictions (legal systems), predominantly under permissive licenses.
Further on, we continue pretraining XLM-R models (Conneau and Lample, 2019) on our corpus and evaluated these models on the recently introduced LEXTREME (Niklaus et al., 2023) and LexGLUE (Chalkidis et al., 2022) benchmarks. Given the often extensive nature of legal text, we also pretrained a Longformer model (Beltagy et al.,
15077
2020) for comparison with hierarchical models (Chalkidis et al., 2019; Niklaus et al., 2021, 2022).
Our multilingual models set a new state-of-theart (SotA) on LEXTREME overall. Our legal Longformer outperforms all other models in four LEXTREME datasets and reaches the highest dataset aggregate score. Our monolingual models outperform their base model XLM-R in 21 out of 24 languages, even reaching language specific SotA in five. On LexGLUE our English models reach SotA in five out of seven tasks with the large model achieving the highest aggregate score.
In the spirit of open science, we provide the dataset under a CC BY-NC-SA 4.0 license, with some subsets licensed more permissively. Dataset creation scripts, models, and pretraining code are public under Apache 2.0 licenses. This opensource approach encourages further research and advancements in the field of legal text analysis and understanding using LLMs.
Contributions
The contributions of this paper are three-fold: First, we curate and release a large multilingual legal text corpus, dubbed MULTILEGALPILE,2 covering 24 languages and 17 legal systems (jurisdictions). Second, we release two multilingual and 24 monolingual legal PLMs, termed LEGALXLMS, initiated from XLM-R (Conneau and Lample, 2019) and further pretrained on the MULTILEGALPILE. We also pretrain a Longformer (Beltagy et al., 2020) based on our multilingual base-size model on context lengths of up to 4096 tokens. Third, we benchmark the newly released models on LEXTREME and LexGLUE, reaching SotA for base- and large-size models and increasing performance drastically in Greek legal code. Our Longformer model achieves SotA in four tasks and the highest dataset aggregate score. Our monolingual models set languagespecific SotA in five languages.

heading: 2 Related Work
text: In this section, we briefly discuss prior general and domain-specific pretraining corpora. See Appendix B for a more detailed discussion of related works.

heading: 2.1 General Pretraining Corpora
text: The One Billion Word Language Model Benchmark (LM1B) (Chelba et al., 2014), Wikipedia, and derived datasets like WikiText (Merity et al., 2016)
2Link will be released upon acceptance.
and BookCorpus (Zhu et al., 2015) have been crucial in developing language models such as GPT-2 (Radford et al., 2019), BERT (Devlin et al., 2019), and RoBERTa (Liu et al., 2019). Large-scale datasets like the Colossal Clean Crawled Corpus (C4) (Raffel et al., 2020), OpenWebText (Gokaslan and Cohen, 2019), The Pile (Gao et al., 2020b), and Glot500 (ImaniGooghari et al., 2023) have further advanced the field, contributing to the training of models like T5, MegatronBERT (Shoeybi et al., 2020), GPT-3 (Brown et al., 2020), and Glot500-m. Although general pretraining datasets are large and widely available, we find that mC4 only contains around 0.1% legal text (see Section 3.1), exemplifying the need for datasets specifically tailored to the legal domain.

heading: 2.2 Domain Specific Corpora
text: Pretraining on domain-specific text like medicine, law, or science can boost Language Model (LM) performance on related tasks (Beltagy et al., 2019; Gu et al., 2021; Chalkidis et al., 2020; Niklaus and Giofré, 2022). In the scientific field, SciBERT was pretrained on a mix of computer science and biomedical papers (Beltagy et al., 2019). Similarly, models like PubMedBERT (Gu et al., 2021) and BioBERT (Lee et al., 2020) were pretrained on biomedical datasets. ClinicalBERT utilized the Medical Information Mart for Intensive Care III (MIMIC-III) dataset, encompassing 2 million clinical notes, demonstrating superior performance on medical NLP tasks (Huang et al., 2019). In the legal realm, LegalBERT was pretrained on 12 GB of English legal texts, achieving high performance on domain-specific tasks (Chalkidis et al., 2020). CaseLaw-BERT utilized the English Harvard Law case corpus from 1965 to 2021 (Zheng et al., 2021). Recently, LexFiles was released, with 11 sub-corpora covering six English-speaking legal systems and 19B tokens (Chalkidis* et al., 2023). It was used to train new legal English PLMs, showing enhanced results in legal tasks. Though efforts to
pretrain legal LMs exist in languages like Italian, Romanian, and Spanish (Licari and Comandè, 2022; Masala et al., 2021; Gutiérrez-Fandiño et al., 2021), English remains predominant, emphasizing the need for multilingual legal corpora. Table 1 compares previous domain-specific corpora, all in English and all legal corpora less than 1/4 of MULTILEGALPILE’s size.

heading: 3 MULTILEGALPILE

heading: 3.1 Construction
text: We transformed all datasets into xz compressed JSON Lines (JSONL) format. The combination of XZ compression and JSONL is ideal for streaming large datasets due to reduced file size and efficient decompression and reading.
Filtering mC4 We used the vast multilingual web crawl corpus, mC4 (Xue et al., 2021), as our base dataset. To effectively isolate legal content, we used regular expressions to identify documents with legal references, such as “Art. 5” or “§ 8” . We found that detecting legal citations, such as references to laws and rulings, served as a reliable indicator of legal-specific documents in the corpus.
To ensure the accuracy of our filtering, we engaged legal experts to aid in identifying citations to laws and rulings across different jurisdictions and languages. We manually reviewed the precision of the retrieved documents for five languages, namely German, English, Spanish, French, and Italian, as shown in Table 2. The proficiency levels of the evaluators included native German, fluent English and Spanish, intermediate French, and basic Italian.
Subsequent to the initial review, we performed a second round of precision evaluation, during which we refined our regex expressions based on our findings from the first iteration. This iterative process not only enhanced the precision of the legal content detection, but also resulted in a reduction of the corpus size from 133GB to 106GB. Although the overall volume of data was reduced, this process significantly improved the quality and specificity of the corpus by focusing on legal content with a higher degree of precision. A major reason for using regexes instead of an machine learning
based classifier was speed. Already when utilizing regexes, filtering through such a huge corpus like mC4 (27TB in total, of which 10.4TB are in English) took several days on our hardware. An ML model based on Bag-of-Words, Word vectors or even contextualized embeddings would a) need an annotated dataset and b) likely be much slower.
We find that on average, only one in every thousand pages in mC4 contains legal content. We show a precise overview of language-specific percentages of legal text in mC4 in Figure 4.
Compiling Native MULTILEGALPILE To compile the corpus, we scraped several sources containing legal language materials. Our search was conducted in a loose manner, meaning that when we found a suitable source with legal text data, we included it in our corpus. It is important to note that we do not claim completeness, as we were unable to perform quality analysis for all available languages. For a detailed overview of sources used for the Native MULTILEGALPILE corpus, please refer to Table 9. Most sources offered direct data download links. For inconsistently formatted data, we converted them to a unified format like jsonl. The post-processing steps involved performing various tasks depending on the initial data format. For example, in the case of CASS3, we extracted the textual data from XML tags.
Curating Eurlex Resources To curate the Eurlex resources, we utilized the eurlex R package (Ovádek, 2021) to generate SPARQL queries and
3https://echanges.dila.gouv.fr/OPENDATA/CASS
download the data. Subsequently, we converted the data into a format more amenable to handling large datasets using Python.
Integrating Pile of Law Henderson et al. (2022) released a large corpus of diverse legal text in English mainly originating from the US. We integrated the latest version with additional data (from January 8, 2023) into our corpus.

heading: 3.2 Description
text: MULTILEGALPILE consists of four subsets: a) Native Multi Legal Pile (112 GB), b) Eurlex Resources (179 GB), c) Legal MC4 (106 GB) and d) Pile of Law (Henderson et al., 2022) (292 GB). Figure 3 details the distribution of languages. Due to Pile of Law integration, English dominates, comprising over half the words. Figure 2 shows text type distribution. Caselaw comprises over half the corpus, due to the good public access to court rulings especially in common law countries. Even in civil law countries, where legislation is crucial, caselaw often outnumbers legislation, as seen in the Swiss case in Table 9. Publicly available contracts are scarce, contributing less than 10% to the corpus, despite potentially making up most existing legal texts (from the private sector). Note that most contracts in our corpus originate from the US or EU international treaties. Table 9 in Appendix E provides additional information on MULTILEGALPILE, including sources and licenses.

heading: 3.3 Licenses and Usage of MULTILEGALPILE
text: The Creative Commons AttributionNonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) license applied to the MULTILEGALPILE corpus depends on the upstream licenses of the data subsets described above.
First, our Native Multi Legal Pile consists of data sources with different licenses. They range from restrictive licenses such as CC BY-NC-SA 4.0 up to the most liberal Creative Commons Zero (CC0) license, which, in essence, releases the data into the public domain. Many sources, however, do not explicitly state the license used for the available data. We assume that such data sources allow pretraining usage, since the creators are usually public agencies such as courts and administrations. Such legislation and caselaw is usually not protected by copyright law. Table 9 provides an overview of the license or copyright situation for each of the 29 sources in the Native Multi Legal Pile. Second,
the Eurlex Resources is CC BY 4.0 licensed by the European Union4, thus posing no legal issues for pretraining. Third, the Legal mC4 corpus was created by filtering multilingual C4 (Xue et al., 2021) for legal content as described above. As mC45 is licensed under ODC-BY, we also release the filtered Legal mC4 corpus under the same license. Finally, the Pile of Law (Henderson et al., 2022) is published under CC BY-NC-SA 4.0 and the dataset is unaltered, thus preserving the license.
Usage of the MULTILEGALPILE corpus is presumably possible for pretraining of NLP models. In general, we assume that the fair use doctrine allows employing the data for legal NLP models because the results are rather transformative (Henderson et al., 2023). Nevertheless, copyright issues in generative AI remain an unresolved problem for the moment. Several court cases are currently pending, such as Getty Images suing Stability AI for intellectual property infringement (Sag, 2023).

heading: 4 Pretraining Legal Models
text: As part of this study, we release 2 new multilingual legal-oriented PLMs, dubbed Legal-XLMRs, trained on the newly introduced MULTILEGALPILE corpus (Section 3). For the newly released Legal-XLM-Rs we followed a series of bestpractices in the LM development literature:
(a) We warm-start (initialize) our models from the original XLM-R checkpoints (base or large) of Conneau and Lample (2019). Model recycling is a standard process followed by many (Wei et al., 2021; Ouyang et al., 2022) to benefit from starting from an available “well-trained” PLM, rather from scratch (random). XLM-R was trained on 2.5TB of cleaned CommonCrawl data in 100 languages. (b) We train a new tokenizer of 128K BPEs on the training subsets of MULTILEGALPILE to better cover legal language across all available legal systems and languages. However, we reuse the original XLM-R embeddings for all lexically overlapping tokens (Pfeiffer et al., 2021), i.e., we warmstart word embeddings for tokens that already exist in the original XLM-R vocabulary, and use random ones for the rest. Similarly to Liang et al. (2023) who increased the vocabulary size from around 2.5K tokens per language (250K for 100 languages) to around 10K (1M for 100 languages),
4https://eur-lex.europa.eu/content/legal-notice/ legal-notice.html
5https://huggingface.co/datasets/mc4
we increased to around 5K (128K for 24 languages), thus roughly doubling compared to XLM-R. (c) We continue pretraining our models on the diverse MULTILEGALPILE corpus with batches of 512 samples for an additional 1M/500K steps for the base/large model. We do initial warm-up steps for the first 5% of the total training steps with a linearly increasing learning rate up to 1e−4, and then follow a cosine decay scheduling, following recent trends. For half of the warm-up phase (2.5%), the Transformer encoder is frozen, and only the embeddings, shared between input and output (MLM), are updated. We also use an increased 20/30% masking rate for base/large models respectively, where 100% of token predictions are based on masked tokens, compared to Devlin et al. (2019)6, based on the findings of Wettig et al. (2023). (d) For both training the tokenizer and our legal models, we use a sentence sampler with exponential smoothing of the sub-corpora sampling rate following Conneau and Lample (2019) and Raffel et al. (2020), since there is a disparate proportion of tokens across sub-corpora and languages (Figures 1 and 3) and we aim to preserve per-corpus and language capacity, i.e., avoid overfitting to the majority (approx. 50% of the total number of tokens) US-origin English texts. (e) We consider mixed cased models, i.e., both
6Devlin et al. – and many other follow-up work – used a 15% masking ratio, and a recipe of 80/10/10% of predictions made across masked/randomly-replaced/original tokens.
upper- and lowercase letters covered, similar to recently developed large PLMs (Conneau and Lample, 2019; Raffel et al., 2020; Brown et al., 2020).
To better account for long contexts often found in legal documents, we continue training the basesize multilingual model on long contexts (4096 tokens) with windowed attention (128 tokens window size) (Beltagy et al., 2020) for 50K steps, dubbing it Legal-XLM-LF-base. We use the standard 15% masking probability and increase the learning rate to 3e−5 before decaying but otherwise use the same settings as for training the small-context models.
In addition to the multilingual models, we also train 24 monolingual models on each of the language-specific subsets of the corpus. Except for choosing a smaller vocab size of 32K tokens, we use the same settings as for the multilingual models. Due to resource constraints, we only train base-size models and stop training at 200K steps. Due to limited data available in some low-resource languages, these models sometimes do multiple passes over the data. Because of plenty of data and to achieve a better comparison on LexGLUE, we continued training the English model for 1M steps and also trained a large-size model for 500K steps. See Table 7 in appendix C for an overview.
We make all our models publicly available alongside all intermediate checkpoints (every 50K/10K training steps for RoBERTa/Longformer models) on the Hugging Face Hub.7
7Link will be released upon acceptance.

heading: 5 Evaluating on Legal Benchmarks
text: In the absence of established legal benchmarks for generative tasks and our focus on pretraining encoder-only models, we select two established legal benchmarks involving challenging text classification and named entity recognition tasks involving long documents: LEXTREME and LexGLUE.

heading: 5.1 Benchmark Description
text: Below, we briefly describe each dataset and refer the reader to the original works for more details.
LEXTREME (Niklaus et al., 2023) is a multilingual legal benchmark. It includes five single label text classification datasets, three multi label text classification datasets and four Named Entity Recognition (NER) datasets.
Brazilian Court Decisions (BCD) (LageFreitas et al., 2022) is from the State Supreme Court of Alagoas (Brazil) and involves predicting case outcomes and judges’ unanimity on decisions. German Argument Mining (GAM) (Urchs et al., 2021) contains 200 German court decisions for classifying sentences according to their argumentative function. Greek Legal Code (GLC) (Papaloukas et al., 2021) tackles topic classification of Greek legislation documents. Tasks involve predicting topic categories at volume, chapter, and subject levels. Swiss Judgment Prediction (SJP) (Niklaus et al., 2021) focuses on predicting the judgment outcome from 85K cases from the Swiss Federal Supreme Court. Online Terms of Service (OTS) (Drawzeski et al., 2021) contains 100 contracts for detecting unfair clauses with the tasks of classifying sentence unfairness levels and identifying clause topics. COVID19 Emergency Event (C19) (Tziafas et al., 2021): consists of legal documents from several European countries related to COVID-19 measures where models identify the type of measure described in a sentence. MultiEURLEX (MEU) (Chalkidis et al., 2021a) is a corpus of 65K EU laws annotated with EUROVOC taxonomy labels. Task involves identifying labels for each document. Greek Legal NER (GLN) (Angelidis et al., 2018) is a dataset for NER in Greek legal documents. LegalNERo (LNR) (Pais et al., 2021) tackles NER in Romanian legal documents. LeNER BR (LNB) (Luz de Araujo et al., 2018) addresses NER in Brazilian legal documents. MAPA (MAP) (Baisa et al., 2016) is a multilingual corpus based on EUR-Lex for NER annotated at a coarse-grained and fine-grained level.
LexGLUE (Chalkidis et al., 2022) is a legal benchmark covering two single-label and, four multi-label text classification datasets, and a multiple choice question answering dataset.
ECtHR Tasks A & B (Chalkidis et al., 2019, 2021b) contain approx. 11K cases from the European Court of Human Rights (ECtHR) public database. Based on case facts, Task A predicts violated articles and Task B allegedly violated articles of the European Convention of Human Rights (ECHR). SCOTUS (Spaeth et al., 2020) combines information from US Supreme Court (SCOTUS) opinions with the Supreme Court DataBase (SCDB). The task is to classify court opinions into 14 issue areas. EUR-LEX (Chalkidis et al., 2021a) contains 65K EU laws from the EUR-Lex portal, annotated with EuroVoc concepts. The task is to predict EuroVoc labels for a given document. LEDGAR (Tuggener et al., 2020) contains approx. 850K contract provisions from the US Securities and Exchange Commission (SEC) filings. The task is to classify contract provisions into categories. UNFAIR-ToS (Lippi et al., 2019) contains 50 Terms of Service (ToS) from online platforms, annotated with types of unfair contractual terms. The task is to predict unfair types for a given sentence. CaseHOLD (Zheng et al., 2021) contains approx. 53K multiple choice questions about holdings of US court cases. The task is to identify the correct holding statement out of five choices.

heading: 5.2 Experimental Setup
text: To ensure comparability, we followed the experimental setups described in the original papers (Niklaus et al., 2023; Chalkidis et al., 2022) using hierarchical transformers for datasets where the sequence length of most documents exceeds the maximum sequence length of the model (Aletras et al., 2016; Niklaus et al., 2022). The hyperparameters used for running experiments on each dataset are provided in Table 8 in the appendix. We follow previous work (Niklaus et al., 2023; Chalkidis et al., 2022) and do not tune hyperparameters.
All scores are macro-F1 scores, equally weighing each class for fairness in unbalanced datasets. To obtain Table 6, we follow Chalkidis et al. (2022), running five repetitions with different random seeds (1-5) and report test scores from the best-performing seed on the development data. For values in Tables 4 and 5, we follow the procedure in Niklaus et al. (2023), taking the harmonic mean of the results of 3 random seeds (1-3). We calculate
the dataset aggregate in Table 4 by successively taking the harmonic mean of (i) the languages in the configurations (e.g., de,fr,it in SJP), (ii) configurations within datasets (e.g., OTS-UL, OTS-CT in OTS), and (iii) datasets in LEXTREME (BCD, GAM). The language aggregate score in Table 5 is computed similarly: by taking the harmonic mean of (i) configurations within datasets, (ii) datasets for each language (e.g., MAP, MEU for lv), and (iii) languages in LEXTREME (bg,cs). We show an overview of the models evaluated in Table 3.
Note that most LLMs are predominantly trained on English and Chinese with the exception of mT5
(Xue et al., 2021) and BLOOM (Scao et al., 2022) (more than 95% of LLaMA’s pretraining corpus is English (Touvron et al., 2023)). Because LEXTREME and LexGLUE consist of NLU tasks, we compare to encoder-only LMs only.

heading: 5.3 Evaluation on LEXTREME
text: We evaluate our models on LEXTREME (Niklaus et al., 2023) and show results across datasets in Table 4 and across languages in Table 5.
We notice that our Legal-XLM-R-base model is on par with XLM-R large even though it only contains 33% of the parameters (184M vs 560M).
All our models outperform XLM-R large on the dataset aggregate score. Our base model sets a new SotA on MAPA (MAP), the large model on CoViD 19 emergency event (C19) and the Longformer on Brazilian court decisions (BCD), German argument mining (GAM), Greek legal code (GLC) and Swiss judgment prediction (SJP). Surprisingly, the legal models slightly underperform in three NER tasks (GLN, LNR, and LNB). Sensitivity to hyperparameter choice could be a reason for this underperformance (we used the same hyperparameters for all models without tuning due to limited compute resources). We see the largest improvements over prior art in BCD (72.4 vs. 63.5) and in GLC (70.2 vs 62.8). Maybe these tasks are particularly hard, and therefore legal in-domain pretraining helps more. For BCD especially, the large amount of Brazilian caselaw in the pretraining corpus may offer an additional explanation.
The monolingual models underperform their base model XLM-R base only in Italian, Polish, and Romanian. In some languages the monolingual model even outperforms XLM-R base clearly (Estonian, Croatian, Hungarian, Latvian, Maltese, Dutch, Slovenian, and Swedish), and in five of them even set the new SotA for the language, sometimes clearly outperforming all other models (the Dutch model even outperforms its closest competitor mDeBERTa-v2 by 11.2 macro F1 and its base model XLM-R by almost 20 macro F1). These languages are all in the lower end of the data availability in the MULTILEGALPILE with the richest language (Dutch) containing only 810M words (see Figure 3). Pretraining a monolingual model on indomain data may therefore be worth it, especially in low-resource languages.
Even though our legal Longformer model performs best on the dataset level, it performs much worse on the language level, possibly due to its lower scores in the most multilingual tasks MEU, MAP and C19 (24, 24 and 6 languages, respectively). Our legal base and large models achieve SotA in some languages, and are in aggregate almost as robust across languages as XLM-R.
Computing the final LEXTREME scores (harmonic mean of dataset aggregate and language aggregate scores), we find that the Legal-XLM-Rlarge is the new SotA on LEXTREME with a score of 59.5 vs 59.4 for Legal-XLM-R-base and 59.3 for XLM-R large. The legal Longformer’s LEXTREME score (56.5) is not competitive due to its low language aggregate score.

heading: 5.4 Evaluation on LexGLUE
text: We evaluate our English and multilingual models on LexGLUE (Chalkidis et al., 2022) and compare with baselines (see Table 6). Our models excel on the ECtHR, SCOTUS, EUR-LEX, and CaseHOLD tasks, setting new SotA. In the other two tasks, our models match general-purpose models such as RoBERTa. A reason for slight underperformance of the legal models in the LEDGAR and especially the Unfair ToS tasks may be the relatively low availability of contracts in the MULTILEGALPILE.

heading: 6 Conclusions and Future Work
text: Conclusions Due to a general lack of multilingual pretraining data especially in specialized domains such as law, we curate a large-scale highquality corpus in 24 languages from 17 jurisdictions. We continue pretraining XLM-R checkpoints on our data, achieving a new SotA for base and large models on the LEXTREME benchmark and vastly outperforming previous methods in Greek legal code. We turn our XLM-R base model into a Longformer and continue pretraining on long documents. It reaches a new SotA in four LEXTREME datasets and reaches the overall highest dataset aggregate score. Monolingual models achieve huge gains over their base model XLM-R in some languages and even set language specific SotA in five languages outperforming other models by as much as 11 macro F1. On LexGLUE our English models reach SotA in five out of seven tasks with the large model achieving the highest aggregate score. To conclude, following best practices in continued pretraining on our comprehensive multilingual legal corpus establishes new state-of-the-art across multiple datasets and languages, significantly enhancing performance in legal text analysis.
Future Work We focused on the 24 EU languages, but in the future, we would like to expand the corpus in terms of languages and jurisdictions covered. Especially in China there exist many accessible sources suitable to extend the corpus. Additionally, we would like to find out whether our findings on in-domain pretraining hold for multibillion generative models. Finally, a detailed examination of the contents of the MULTILEGALPILE could provide valuable insights into its structure and efficacy in enhancing legal language models.
Ethics Statement
This study focuses on evaluating legal-specific LMs from multiple aspects, expanding the dialogue to aid in creating support technologies for both legal professionals and the general public. This area represents a vital field for research, as emphasized by Tsarapatsanis and Aletras (2021), aiming to enhance legal services and make legal knowledge more accessible. The study also aims to shed light on the multifaceted limitations that need addressing to ensure the responsible and ethical application of legal-oriented technologies.
In pursuit of these goals, we introduce novel resources encompassing a range of legal systems. These resources are designed to construct new models that more accurately reflect legal nuances and evaluate their effectiveness more precisely. All resources created and shared in this work are derived from data that is publicly accessible, often distributed across various online platforms.
Limitations
We did not perform deduplication, thus data from legal mC4 might be present in other parts. However, Muennighoff et al. (2023) suggest that data duplication does not degrade performance during pretraining for up to four epochs. Overlap between other subsets is highly unlikely, since they originate from completely different jurisdictions.
Due to limited compute, we were not able to pretrain a large generative model and leave this to future work.

heading: A Use of AI assistants
text: We used ChatGPT and Grammarly for improving the grammar and style of our writing.

heading: B Additional Related Work
text: B.1 General Pretraining Corpora
The use of pretrained Language Models (PLMs) has become increasingly popular in NLP tasks, particularly with the advent of models such as BERT (Devlin et al., 2019) that can be finetuned for specific applications. One key factor in the success of pretraining is the availability of large and diverse text corpora, which can help the model learn the nuances of natural language. In the following, we discuss large-scale general-purpose text corpora used for pretraining.
One of the earliest widely-used datasets is the One Billion Word Language Model Benchmark (LM1B) (Chelba et al., 2014). It was created by extracting one billion words from web pages to evaluate novel language modeling techniques. It has been used, among others, to evaluate GPT-2 (Radford et al., 2019).
Wikipedia is a commonly used multilingual dataset for pretraining language models, and has been used to pretrain BERT (Devlin et al., 2019), MegatronBERT (Shoeybi et al., 2020), T5 (Raffel et al., 2020), and GPT-3 (Brown et al., 2020), among others.
Based on Wikipedia, Merity et al. (2016) created WikiText by selecting articles fitting the Good or Featured article criteria. The dataset contains 103M words and has two versions: WikiText2 and the larger WikiText103. It has been used to pretrain models like MegatronBERT (Shoeybi et al., 2020) and GPT-2 (Radford et al., 2019).
The BookCorpus (Zhu et al., 2015), also known as the Toronto Books Corpus, is an English dataset used for pretraining BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and T5 (Raffel et al., 2020). It consists of almost 1B words from over 11K books collected from the web.
The Common Crawl corpus is a publicly available multilingual dataset of scraped web pages, regularly updated with new "snapshots". It has been used to pretrain GPT-3 (Brown et al., 2020) as well as XLM-R (Conneau et al., 2020). One
significant drawback of Common Crawl is the presence of uncleaned data, which includes a considerable amount of “gibberish or boiler-plate text like menus, error messages, or duplicate text” (Raffel et al., 2020). As a result, utilizing the Common Crawl dataset necessitates additional post-filtering and cleaning procedures. To address this issue, Raffel et al. (Raffel et al., 2020) performed several cleaning steps on the April 2019 snapshot of Common Crawl, resulting in the creation of the Colossal Clean Crawled Corpus (C4), comprising 750 GB of English-language text. It was used for pretraining models such as T5 (Raffel et al., 2020) and Switch Transformer (Fedus et al., 2022).
OpenWebText (Gokaslan and Cohen, 2019) openly replicates OpenAI’s closed English WebText dataset (Radford et al., 2019), used to pretrain GPT-2 (Radford et al., 2019). WebText comprises over 8M documents with a combined text size of 40 GB. To ensure data uniqueness, any documents sourced from Wikipedia were excluded from WebText, as they are commonly utilized in other datasets. OpenWebText, on the other hand, consists of 38 GB of text data from 8M documents and was used for pretraining RoBERTa (Liu et al., 2019) and MegatronBERT (Shoeybi et al., 2020).
News articles are also a common source for pretraining corpora. The RealNews dataset (Zellers et al., 2019) is a large corpus extracted from Common Crawl, containing news articles from December 2016 to March 2019 (training) and April 2019 (evaluation), totaling 120 GB. It was used for pretraining MegatronBERT (Shoeybi et al., 2020). For pretraining RoBERTa, Liu et al. (2019) used an English subset of RealNews8, comprising 63M English news articles crawled from September 2016 to February 2019.
The rise of LLMs brought about the creation of ever larger training datasets. The Pile (Gao et al., 2020b) combines 22 distinct, well-curated datasets, such as Wikipedia (English), OpenWebText2 (Gokaslan and Cohen, 2019), OpenSubtitles (Tiedemann, 2016) etc., encompassing 825 GB of data. Besides general-purpose textual datasets, it also contains domain-specific datasets, such as ArXiv (Science), FreeLaw (Legal), PubMed Abstracts (Biomedicine), and GitHub data (to improve code-related task performance (Gao et al., 2020b)). GPT-2 (Radford et al., 2019) and GPT-3 (Brown
8https://commoncrawl.org/2016/10/news-datasetavailable
et al., 2020) were evaluated on this dataset. Touvron et al. (2023) compiled a substantial dataset from various publicly available sources, including CommonCrawl, C4, Github, Wikipedia, etc., totaling 1.4T tokens. They trained the 13Bparameter LLaMA model using this dataset, surpassing the performance of the 175B-parameter GPT-3 on most benchmark tasks. However, the dataset itself is not publicly available. To address this, a collaborative effort resulted in the creation of the RedPajama-Data-1T9 dataset, replicating LLaMA’s dataset with a similar size of 1.2T tokens.
Some of the afore-mentioned datasets, such as Common Crawl, are used to pretrain multilingual versions of BERT, DistilBERT, RoBERTa etc. These models were pretrained on datasets that cover approximately 100 languages, thereby neglecting low-resource languages. ImaniGooghari et al. (2023) addressed this by compiling Glot500, a 700 GB dataset covering 500 diverse languages, with a focus on low-resource ones. The Glot500-m model, pretrained on this dataset, outperformed the XLM-RoBERTa base model on six out of seven tasks.
B.2 Domain Specific Corpora
While pretraining on general-purpose text like Wikipedia and news articles shows promise, evidence suggests that pretraining on domain-specific text can enhance language model performance on related tasks (Beltagy et al., 2019; Gu et al., 2021; Chalkidis et al., 2020; Niklaus and Giofré, 2022). Domain-specific text corpora include texts specific to fields like medicine, law, or science.
Several studies have examined pretraining on scientific text corpora. Beltagy et al. (2019) pretrained SciBERT, a BERT-based model, on a random subset of 1.14M papers sourced from Semantic Scholar. This collection comprises 18% of computer science papers and 82% of papers from the broader biomedical field. Similarly, PubMed and PubMedCentral are common sources for biomedical datasets. Gu et al. (2021) trained PubMedBERT using PubMed abstracts and PubMedCentral articles; BioBERT (Lee et al., 2020) was pretrained similarly. Johnson et al. (2016) compiled the Medical Information Mart for Intensive Care III (MIMIC-III) dataset, a large single-center database of critical care patients. “a large, single-center database comprising infor-
9https://github.com/togethercomputer/RedPajama-Data
mation relating to patients admitted to critical care units at a large tertiary care hospital”. Huang et al. (2019) used over 2 million de-identified clinical notes from this dataset to pretrain ClinicalBERT. These models outperformed general-purpose models on biomedical NLP tasks.
In the legal domain, similar strategies are observed. Chalkidis et al. (2020) collected 12 GB of diverse English legal texts, including legislation, court cases, and contracts. They pretrained LegalBERT on this dataset, showing SotA performance, especially in tasks requiring domain knowledge. Another study by Zheng et al. (2021) used the entire English Harvard Law case corpus (1965-2021) comprising 37 GB of text to pretrain CaseLawBERT.
Recently, Chalkidis* et al. (2023) released LexFiles, an English legal corpus with 11 sub-corpora covering legislation and case law from six Englishspeaking legal systems (EU, Council of Europe, Canada, US, UK, India). The corpus contains approx. 6M documents or approx. 19B tokens. They trained two new legal English PLMs, showing improved performance in legal probing and classification tasks.
Efforts to pretrain legal language models also exist for Italian (Licari and Comandè, 2022), Romanian (Masala et al., 2021), and Spanish (GutiérrezFandiño et al., 2021). However, English dominates, underscoring the importance of compiling multilingual legal corpora.

heading: C Training Details
text: For finetuning the pretrained models on the evaluation benchmarks we used the following NVIDIA GPUs: 24GB RTX3090, 32GB V100 and 80GB A100. We used v3-8 TPUs for pretraining. All our experiments were run on Linux machines (Debian).

heading: D Hyperparameter Details
text: .

heading: E Dataset Details

heading: Future Work
text: Future Work We focused on the 24 EU languages, but in the future, we would like to expand the corpus in terms of languages and jurisdictions covered. Especially in China there exist many accessible sources suitable to extend the corpus. Additionally, we would like to find out whether our findings on in-domain pretraining hold for multibillion generative models. Finally, a detailed examination of the contents of the MULTILEGALPILE could provide valuable insights into its structure and efficacy in enhancing legal language models