LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain
Authors: Joel Niklaus
Veton Matoshi
Pooja Rani
Andrea Galassi
Matthias Stürmer
Ilias Chalkidis
Venue: EMNLP 2023 Findings (2023)
Keywords: legal
nlp
benchmark
multilingual
multitask
TL;DR: LEXTREME: A Novel Multilingual Multitask Benchmark for the Legal Domain

Abstract: Lately, propelled by phenomenal advances around the transformer architecture, the legal NLP field has enjoyed spectacular growth. To measure progress, well-curated and challenging benchmarks are crucial. Previous efforts have produced numerous benchmarks for general NLP models, typically based on news or Wikipedia. However, these may not fit specific domains such as law, with its unique lexicons and intricate sentence structures. Even though there is a rising need to build NLP systems for languages other than English, many benchmarks are available only in English and no multilingual benchmark exists in the legal NLP field. We survey the legal NLP literature and select 11 datasets covering 24 languages, creating LEXTREME. To fairly compare models, we propose two aggregate scores, i.e., dataset aggregate score and language aggregate score. Our results show that even the best baseline only achieves modest results, and also ChatGPT struggles with many tasks. This indicates that LEXTREME remains a challenging task with ample room for improvement. To facilitate easy use for researchers and practitioners, we release LEXTREME on huggingface along with a public leaderboard and the necessary code to evaluate models. We also provide a public Weights and Biases project containing all runs for transparency.

LEXTREME: Multi-Lingual and Multi-Task Benchmark for the Legal Domain
Joel Niklaus 1,2,6+
Veton Matoshi 2+
Pooja Rani 3
Andrea Galassi 4
Matthias Stürmer 1,2
Ilias Chalkidis 5
University of Bern
Bern University of Applied Sciences University of Zurich
UUniversity of Bologna
UUniversity of Copenhagen
"Stanford University
Abstract
LEXTREME Score (mF1)
Lately, propelled by phenomenal advances around the transformer architecture, the legal NLP field has enjoyed spectacular growth. To measure progress, well-curated and challeng- ing benchmarks are crucial. Previous efforts have produced numerous benchmarks for gen- eral NLP models, typically based on news Wikipedia. However, these may not fit specific domains such as law, with unique lexicons and intricate sentence structures. Even though there rising need to build NLP systems for languages other than English, many bench- marks are available only in English and no mul- tilingual benchmark exists in the legal NLP field. We survey the legal NLP literature and select 11 datasets covering 24 languages, cre- ating LEXTREME. fairly compare models, we propose aggregate scores, i.e., dataset aggregate score and language aggregate score. Our results show that even the best baseline only achieves modest results, and also Chat- GPT struggles with many tasks. This indicates that LEXTREME remains challenging task with ample room for improvement. To facili- tate easy use for researchers and practitioners, release LEXTREME on huggingface along with public leaderboard and the necessary code to evaluate models. also provide public Weights and Biases project containing all runs for transparency.
XLM-R 41.3
Language Aggregate Score (mFF1)
mDeBERTo 38.9
GEEN
30
DiStIBERT 36.7
28
20
MiniLM 1g.0
XLM-R 257
15
Dataset Aggregate Score (mF1)
Figure 1: Overview of multilingual models LEX- TREME. The bubble size and text inside indicate the parameter count. The bold number helow model name indicates the LEXTREME score (harmonic mean of the language agg- score and the dataset agg- score).
with many resources, such as large legal cor- pora (Henderson et al., 2022), task-specific datasets (Shen al., 2022; Christen et al., 2023; Brugger et al., 2023; Niklaus et al., 2023a), and pre-trained legal-oriented language models (Chalkidis et al., 2020; Zheng et al., 2021; Xiao et al., 2021; Niklaus and Giofre, 2023; Hua et al., 2022: Chalkidis et al.. 2023). Greco and Tagarelli (2023) offer compre- hensive survey the topic.
1 Introduction
Specifically, the emergence of pre-trained Lan- guage Models (PLMs) has led to significant perfor- mance boosts on popular benchmarks like GLUE (Wang et al., 2019b) or SuperGLUEE (Wang et al., 2019a), emphasizing the need for more challeng- ing benchmarks to measure progress. Legal bench- mark suites have also been developed to systemat- ically evaluate the performance of PLMs, show- casing the superiority of legal-oriented models over generic ones on downstream tasks such as legal document classification or question answer- ing (Chalkidis et al., 2022a; Hwang et al., 2022).
In the last decade, Natural Language Processing (NLP) has gained relevance Legal Artificial In- telligence, transitioning from symbolic to subsym- bolic techniques (Villata et al., 2022). Such a shift is motivated partially by the nature of legal re- sources, which appear primarily in textual for- mat (legislation, legal proceedings, contracts, etc.). Following the advancements in NLP technologies, the legal NLP literature (Zhong et al., 2020; Ale- tras et al., 2022; Katz et al., 2023) flourishing
Equal contribution.

---

Even though these PLMs are shown be ef- fective for numerous downstream tasks, they are general-purpose models that are trained on broad- domain resources, such as Wikipedia or News, and therefore, can be insufficient to address tasks spe- cific to the legal domain (Chalkidis et al., 2020; Hua et al., 2022; Niklaus and Giofre, 2023). In- deed, the legal domain is strongly characterized both by its lexicon and by specific knowledge typ- ically not available outside of specialized domain resources. Laypeople even sometimes call the lan- guage used in legal documents "legalese" or "legal jargon", emphasizing its complexity. Moreover, the length of a legal document usually exceeds the length of Wikipedia or news article, and in some tasks the relationships between its entities may span across the entire document. Therefore, it is nec- essary develop specialized Legal PLMs trained on extensive collections of legal documents and evaluate them on standardized legal benchmarks. While new PLMs capable of handling long docu- ments have been developed in the last years, they are predominantly trained for the general domain and on English data only.
between the model size and performance on LEX- TREME. Surprisingly. the low end. DistilBERT (Sanh et al., 2019) strongly outperforms MiniLM (Wang et al., 2020) (36.7 vs 19.0 LEXTREME score) while only having marginally more parame- ters (1 (135M vs 118M).
For easy evaluation of future models, we release the aggregate dataset on the huggingface hub along with a public leaderboard and the necessary code to run experiments on GitHub. Knowing that our work can not encompass "Everything in the Whole Wide Legal World" (Raji et al., 2021), we design LEXTREME as a living benchmark and pro- vide detailed guidelines on our repository and en- courage the community to contribute high-quality multilingual legal datasets.? Finally, we integrated LEXTREME together with the popular English le- gal benchmark LexGLUE (Chalkidis et al., 2022a) into HELM (Liang et al., 2022) (an effort to eval- uate language models holistically using a large number of datasets from diverse tasks) to ease the adoption of curated legal benchmarks also for the evaluation of large language models such as GPT- 3 (Brown al., 2020), PALM (Chowdhery et al., 2022) LLaMA (Touvron et al., 2023).
The rising need to build NLP systems for lan- guages other than English, the lack of textual re- sources for such languages, and the widespread use of code-switching in many cultures (Torres Ca- coullos, 2020) is pushing researchers to train mod- els on massively multilingual data (Conneau et al.. 2020). Nonetheless, to the best of our knowledge, no multilingual legal language model has been pro- posed so far. Consequently, there is a need for standardized multilingual benchmarks that can be used to evaluate existing models and assess whether more research efforts should be directed toward the development of domain-specinn models. This is particularly important for legal NLP where inher- ently multinational (European Union, Council of Europe) or multilingual (Canada, Switzerland) lc- gal systems are prevalent.
Contributions of this paper are two-fold: 1. We review the legal NLP literature to find relevant legal datasets and compile multi- lingual legal benchmark of 11 datasets in 24 languages from language groups.
2. We evaluate several baselines on LEXTREME. to provide a reference point for researchers and practitioners to compare to.
Related Work
2.1 Benchmarks
Benchmarking is an established method to enable easy and systematic comparison of approaches. GLUE (Wang et al., 2019b) is one of the first benchmarks to evaluate general-purpose neural lan- guage models. It is a set of supervised sentence understanding predictive tasks in English that were created through aggregation and curation of sev- eral existing datasets. However, it became quickly obsolete due to advanced contextual language mod- els, such as BERT (Devlin al., 2019), which
In this work, we propose a challenging multi- lingual benchmark for the legal domain, named LEXTREME. We survey the literature from 2010 to 2022 and select 11 relevant NLU datasets cov- cring 24 languages in 8 subdivisions (Germanic, Romance, Slavic, Baltic, Greek, Celtic, Finnic, and Hungarian) from two language families (Indo- European and Uralic). We evaluate five widely used multilingual encoder-based language models as shown in Figure 1 and observe a correlation
lextreme httt::///heeiinfse...oooattsss/s
hhttps://gi cwm ctmm//oooNNKKLLa//LLXXXEEE
'Since the release of this call in February 2023, already eleven new tasks have been contributed and integrated.

---

Name
Source
Domain
Tasks
Datasets
Languages
Agg. Score
GLUE
(Wang el al., 2019b)
Misc. Texts
9
English
Yes
SUPERGLUE
(Wang et al., 2019a)
Misc. Texts
8
8
English
Yes
MMLU
(Hendrycks et al., 2021)
Misc. Texts
57
English
Yes
CLUE
(Xu et 2020)
Mise. Texts
Chinese
Yes
XTREME
(Hu et al., 2020)
Misc. Texts
9
40
Yes
BLUE
(Peng et al., 2019)
Biomedical Texts
5
10
English
Yes
CBLUE
(Zhang et al., 2022)
Biomedical Texts
Chinese
Yes
LegalBench
(Guha et al., 2022)
Legal Texts
English
No
LexGLUB
(Chalkidis et al., 2022a)
Legal Texts
English
Yes
FairLex
(Chalkidis et al., 2022b)
Legal Texts
4
5
No
LBOX
(Hwang et al., 2022)
Legal Texts
5
5
Korean
Yes
LEXTREME
(our work)
I Legal Texts
18
11
24
Yes
SUPERB
(Yang al., 2021) (Tsai et al., 2022)
Speech
10
10
English English
SUPERB-S SG
Speech
5
No
TAPE
(Rao et al., 2019)
Proteins
5
n/a
No
Table 1: Characteristics of popular existing NLP benchmarks.
excelled on most tasks. Subsequently, its updated version, named SUPERGLUE (Wang et al., 201 9a) was proposed, incorporating new predictive tasks that are solvable by humans but are difficult for machines. Both benchmarks proposed evalua- tion score computed an aggregation of the scores obtained by the same model each task. They are also agnostic regarding the pre-training of the model, and do not provide a specific corpus for it. Inspired by these works, numerous benchmarks have been proposed over the years. We describe some well-known ones in Table 1.
ready designed for cross- loossllnngual tasks, others were extended by translating part of the data using hu- man professionals and automatic methods.
2.2 Benchmarks for the Legal Domain
LEXGLUE (Chalkidis al., 2022a) is the first benchmark for the legal domain and covers six predictive tasks over five datasets made of textual documents in English from the US, EU, and Coun- cil of Europe. While some tasks may not require specific legal knowledge to be solved, others would probably need, or at least benefit from, information regarding the EU US legislation the specific topic. Among the main limitations of their bench- mark, Chalkidis et al. highlight its monolingual nature and remark that "there an increasing need for developing models for other languages". parallel, Chalkidis et al. (2022b) released FairLex multilingual benchmark for the evaluation of fair- ness in legal NLP tasks. With a similar aim, Hwang et al. (2022) released the LBOX benchmark, cring two classification tasks, two legal judgment prediction tasks, and one Korean summarization task. Motivated by LEXGLUE and LBOX, we propose a benchmark to encourage multilingual models, diverse tasks, and datasets for the legal domain. Guha et al. (2022) proposed the LEGAL- BENCH initiative that aims to establish open and collaborative legal reasoning benchmark for few-shot evaluation of LLMs where legal practi- tioners and other domain experts can contribute by submitting tasks. At its creation, the authors have already added 44 lightweight tasks. While most tasks require legal reasoning based on the common
The MMLU benchmark is specifically designed to evaluate the knowledge acquired during the pre- training phase of the model by featuring only zero- shot and few-shot learning tasks (Hendrycks et al., 2021). Similarly, SUPERB (Yang et al., 2021) and SUPERB-SS SG (Tsai al., 2022) were proposed for speech data, unifying well-known datasets. How- ever, they mainly vary tasks, e.g. SUPERB B- SG includes both predictive and generative tasks, which makes different from the other benchmarks discussed in this section. Additionally, SUPER B- SG includes diverse tasks, such as speech transla- tion and cross-lingual automatic speech recogni- tion, which require knowledge of languages other than English. Neither of the two (SUPERB or SUPERB SG) proposes an aggregated score. XTREME (Hu al., 2020) is benchmark specifically designed to evaluate the ability of cross- lingual generalization of models. It includes six cross-lingual predictive tasks over ten datasets of miscellancous texts, covering a total of 40 lan- guages. While some original datasets were al-

---

law system (mostly prevalent in the UK and former colonies), there also clause classification task. For a more comprehensive overview of the many tasks related to automated legal text analysis, we recommend reading the works of Chalkidis et al.
Task
Examples
# Labels
BCD-J
3234 / 404 / 405
3/3/3 3
BCD-U
1715 211 / 204
2/2/2
GAM
19271 2726 /0778
4/4/4
GLC-V
28536 / 9511 /9516
47 1474 (447
GLC-C
28536 / 9511 /9516
386/ 377 374
(2022a) and Zhong et al. (2020).
GLC-S
28536 951 I /9516
2143 1679 1685
59709 8208 17357
2/2/2
2.3 Legal Language Models
OTS-UI.
2074 / 191 / /417
3/3/3
OTS-CT
19942 / 1690 / 4297
9/8/9
Several works have proposed legal language mod- els (models specifically trained for the legal do- main) for several languages other than English. For example, legal language models for En- glish (Chalkidis al., 2020; Yin and Habernal, 2022: Chalkidis et al., 2023), French (Douka et al., 2021), Romanian (Masala et al., 2021), Ital- ian (Tagarelli and Simeri, 2022; Licari and Co- mandé, 2022), Chinese (Xiao et al., 2021), Ara- bic (AI-Qurishi et al., 2022), Korean (Hwang et al., 2022), and Portuguese (Ciurlino, 2021). Re- cently, pre-trained multilingual legal language mod- els (Niklaus et al., 2023b; Rasiah et al., 2023) have been released. Unfortunately, these models were not available at the time of submission, so we do not present results as part of this work.
C19
3312 / 418/414
8/8/8
MEU-1
817239 / 115000
21/21
MRU-2
817239 112500 115000
127 / 126 / 127
MEU-3
817239
/ 115000
500 / 454 465
GLN
17699 / 49909 4017
17/ 17
LNR
7552 966 907
11/9/11
LNB
7828 / 1177 1390
13/13/1
MAP-C
27823 / 3354/ 10590
13/11/11
MAP-F
27823 / 3354 10590
26/2//34
Table 2: Dataset and task overview. # Examples and # Labels show values for train, validation, and test splits.
about the legal text, similar to LEXGLUE (Chalkidis et al., 2022a)
13: The current tasks are set in European lan- guage, per the scope of our present work, but we aim to incorporate broader range of languages in future iterations of LEXTREME 14: The dataset is annotated by humans directly or indirectly (e.g., judgementt labels are extracted with regexes)
3 LEXTREME Datasets and Tasks
3.1 Dataset and Task Selection
To find relevant datasets for the LEXTREME benchmark explore the literature of NLP and the legal domain, identifying relevant venues such as ACL, EACL, NAACL, EMNLP, LREC, ICAIL, and the NLLP workshop. We search the literature on these venues for the years 2010 to 2022. We search for some common keywords (case insensi- tive) that are related to the legal domain, e.g., crim- inal, judicial, judgment, jurisdictions, law, legal, legislation, and dataset, c.g., dataset, and corpus vie their union. These keywords help to select 108 potentially relevant papers. Then, we formulate several criteria select the datasets. Finally, three authors analyze the candidate papers and perform the selection. We handled the disagreement be- tween authors based on mutual discussion and the majority voting mechanism.
Exclusion criteria:
El: The dataset is not publicly available
E2: The dataset does contain a public license or does not allow data redistribution
E3: The dataset contains labels generated with MI.
systems
E4: It is not a peer-reviewed paper
After applying the above criteria, we select 11 datasets from 108 papers. We provide the list of all these datasets in our repository.
3.2 LEXTREME Datasets
In the following, we briefly describe the selected datasets. Table 2 provides more information about the number of examples and label classes per split for each task. For a detailed overview of the juris- dictions well as the number of languages cov- ered by each dataset, see Table 3. Each dataset can have several configurations (tasks), which are the basis of our analyses, i.e., the pre-trained mod- cls have always been fine-tuned on a single task. LEXTREME consists of three task types: Single
Inclusion criteria:
11: It is about legal text (e.g., patents are not con- sidered part of the legal text)
12: It performs legal tasks (e.g., judgment pre- diction) or NLU tasks on legal text in order to have datasets that understand or reason

---

Dataset
Jurisdiction
Languages
documents cover 47 main thematic topics which are called volumes. Each of them is divided into the- matic sub categories which are called chapters and subsequently, each chapter breaks down to subjects. Therefore, the dataset is used to perform three dif- ferent SLTC tasks along volume level GLC-VV, chapter level (GLC-C), and subject level (GLC-S). The input to the models is the entire document, and the output is one of the several topic categories.
BCD
BR
pt
GAM
DE
de
GLC
GR
el
SJP
CH
de, fr, it
OTS
EU
de, en, it, pl
C19
BE, FR, HU.HHUUTTT NL, PL.
en, fr, hu, it, nb,
UK
nl. pl
MEU
EU
24 EU langs
GLN
GR
cl
LNR
RO
ro
LNB
BR
pt
MAP
EU
24 EU langs
Swiss Judgment Prediction (SJP). Niklaus al. (2021, 2022), focus on predicting the judgment outcome of 85K cases from the Swiss Federal Supreme Court (FSCS). The input to the models is the appeal description, and the output is whether the appeal is approved or dismissed (SLTC task).
Table 3: Overview of datasets, the jurisdiction, and the languages. The 24 EU languages are: bg, es, da, de, el, en, es, et, fi, fr, ga, hu, it, It, lv, mt, nl, pt, TO, sk, SV.
Label Text Classification (SLTC), Multi Label Text Classification (MLTC), and Named Entity Recog- nition (NER). We use the existing train, validation, and test splits if present. Otherwise, we split the data randomly ourselves (80% train, 10% valida- tion, and 10% test).
Online Terms of Service (OTS). While multi- lingualism's 's benefits (e.g., cultural diversity) in the EU legal world are well-known (Commission, 2005), creating an official version of every legal act 24 languages raises interpretative challenges. Drawzeski et al. (2021) attempt to automatically detect unfair clauses in terms of service documents, We use their dataset of 100 contracts to perform a SLTC and MLTC task. For the SLTC task (OTS- UL), model inputs are sentences, and outputs are classifications into three unfaimess levels: clearly fair, potentially unfair and clearly unfair. The MLTC task (OTS-CT) involves identifying sen- tences based on nine clause topics.
Brazilian Court Decisions (BCD). Legal sys- tems are often huge and complex, and the informa- tion is scattered across various sources. Thus, pre- dicting case outcomes from multiple vast volumes of litigation is a difficult task. Lage-Freitas et al. (2022) propose an approach to predict Brazilian legal decisions to support legal practitioners. We use their dataset from the State Supreme Court of Alagoas (Brazil). The input to the models is always the case description. We perform two SLTC tasks: In the BCD-J subset models predict the approval or dismissal of the case or appeal with the three labels no, partial, yes, and the BCD-U models predict the judges' unanimity on the decision alongside two labels, namely unanimity, not-unanimity.
COVID19 Emergency Event (C19). The COVID-19 pandemic showed various exceptional measures governments worldwide have taken to contain the virus. Tziafas et al. (2021), presented a dataset, also known EXCEPTIUS, that contains legal documents with sentence-level annotation from several European countries automatically identify the measures. We use their dataset perform the MLTC task of identifying the type of sentence. The input to the
German Argument Mining (GAM). Identify- ing arguments in court decisions is vital and chal- lenging for legal practitioners. Urchs. et al. (2021) assembled a dataset of 200 German court decisions for sentence classification based on argumentative function. We utilize this dataset for SLTC task. Model input is sentence, and output is categorized as conclusion, definition, subsumption, or other.
measure described models are the sentences, and the output is neither or at least one of the measurement types.
MultEEURLEX (MEU). Multilingual transfer learning has gained significant attention recently due to its increasing applications in NLP tasks. Chalkidis al. (2021a) explored the cross-lingual transfer for legal NLP and presented corpus of 65K EU laws annotated with multiple labels from the EUROVOC taxonomy. We perform a MLTC task to identify labels (given in the taxonomy) for
Greek Legal Code (GLC). Legal documents can cover a wide variety of topics, which makes accu- rate topic classification all the more important. Pa- paloukas et al. (2021) compiled dataset for topic classification of Greek legislation documents. The

---

each document. Since the taxonomy exists on mul- tiple levels, we prepare configurations according to three levels (MEU-I, MEU-2, MEU-3).
sophisticated prompting techniques because of pro- hibitively long inputs in many tasks.
4.2 Monolingual
Greek Legal NER (GLN). Identifying various named entities from natural language text plays an important role for Natural Language Understand- ing (NLU). Angelidis et al. (2018) compiled an annotated dataset for NER in Greek legal docu- ments. The source material are 254 daily issues of the Greek Government Gazette over the period 2000-2017. In all NER tasks of LEXTREME the input the models is the list of tokens, and the output an entity label for each token.
In addition to the multilingual models, we also fine-tuned available monolingual models on the language specific subsets. We chose monolingual models only if a certain language was represented in at least three datasets." We made a distinction between general purpose models, i.e., models that have been pre-trained on generic data aka Native- BERTs, and legal models, i.e., models that have been trained (primarily) on legal data aka NativeLe- galBE.RTs. list of the monolingual models can be found in the appendix in Table 8.
LegalNERo (LNR). Similar to GLN, Pais et al. (2021) manually annotated Romanian legal docu- ments for various named entities. The dataset is derived from 370 documents from the larger MAR- CELL Romanian legislative subcorpus."
4.3 Hierarchical Variants
A significant part of the datasets consists of very long documents, the best examples being all vari- ants of MultiEURLEX; we provide detailed using different tokenizers on all datasets in our online repository. However, the models we evaluated were all pre-trained with maximum sequence length of only 512 tokens. Directly applying pre-trained language models on lengthy legal documents may necessitate substantial truncation, severely restrict- ing the models. To overcome this limitation, we hierarchical versions of pre-training models for datasets containing long documents.
LeNER BR (LNB). Luz de Araujo et al. (2018) compiled dataset for NER for Brazilian legal doc- uments. 66 legal documents from several Brazilian Courts and four legislation documents were col- lected, resulting in a total of 70 documents anno- tated for named entities.
MAPA (MAP). de Gibert al. (2022) built multilingual corpus based on EUR-Lex (Baisa et al., 2016) for NER annotated at a coarse-grained (MAP-C) and fine-grained (MAP-F) level.
The hicrarchical variants used here are broadly equivalent to those in (Chalkidis et al., 2021b; Niklaus et al.. 2022). First, we divide each doc- ument into chunks of 128 tokens each. Second, we use the model to be evaluated to encode each of these paragraphs in parallel and obtain the [CLS] embedding of each chunk which can be used as a context- unawwre chunk representation. In order make them context-aware, i.e. aware of the sur- rounding chunks, the chunk representations are fed into a 2-layered transformer encoder. Finally, max- pooling over the context-aware paragraph represen- tations is deployed, which results in document representation that is fed to classification layer. Unfortunately, to the best of our knowledge mod- els capable of handling longer context out the box, such as Longformer (Beltagy et al., 2020) and SLED (Ivgi et al., 2023) are not available multilin- gually and predominantly trained on English data only.
4 Models Considered
Since our benchmark only contains NLU tasks, we consider encoder-only models for simplicity. Due to resource constraints, we did not fine-tune models larger than IB parameters.
4.1 Multilingual
We considered the five multilingual models listed in Table 4, trained on at least 100 languages each (more details are in Appendix B). For XLM-R we considered both the base and large version. Fur- thermore, used ChatGPT (gpt-3.5-turbo) for zero-shot evaluation of the text classification tasks with less than 50 labels.5 To be fair across tasks we did not consider few-shot evaluation or more
'hups://marcell http://mmarrelllppjjee..oc
'We excluded MultiEurlex because only contains nu- meric labels and not textual ones, and because the inputs are very long in 24 languages rendering a valid comparison with reasonable costs impossible.
"Which is why we did not include Norwegian pre-trained models, even though Norwegian is covered in C19.

---

Model
Source
Parameters
Vocab
# Steps
Batch Size
Corpus
# Langs
MiniLM
Wang et al. (2020)
118M
250K
IM
256
2.5TB CC100
100
DistilBERT
Sanh et al. (2019)
135M
120K
n/a
4000
Wikipedia
104
mDeBERTa-v3
He et al. (2020, 2021)
278M
128K
500K
8192
2.5TB CC100
100
XLM RuzelLgge
Conneau al. (2020)
278M/560M
250K
1.5M
8192
2.5TB CC100
100
Table 4: Multilingual models. All models support a maximum sequence length of 512 tokens. The third column shows the total number of parameters, including the embedding layer.
5 Experimental Setup
fine- grainer analysis. However, we believe that by taking the following three measures, an aggregate score can provide more benefits than drawbacks, encouraging the community to evaluate multilin- gual legal models on curated benchmark, thus easing comparisons.
Multilingual models were fine-tuned on all lan- guages of specific datasets. Monolingual models used only the given model's language subset.
Some datasets are highly imbalanced, one of the best examples being BCD-U with a proportion of the minority class of about 2%. Therefore, we applied random oversampling on all SLTC datasets, except for GLC, since all its subsets have too many labels, which would have led to drastic increase in the data size and thus in the computational costs for fine-tuning. For each run, we used the same hyperparameters, as described in Section A.3. As described in Section 4.3, some tasks contain very long documents, requiring the usage of hier- archical variants to process sequence lenghts of 1024 to 4096 tokens. Based on the distribution of the sequence length per example for each task (cf. Appendix H), we decided on suitable sequence lengths for each task before fine-tuning. list of suitable sequence lengths are in A.I.
We (a) evaluate all datasets with the same score (macro-FIl) making aggregation more intuitive and easier to interpret, (b) aggregating the Fl scores again using the harmonic mean, since scores are already rates and obtained using the harmonic mean over precision and recall, following Tatiana and Valentin (2021), and (c) basing our final aggre- gate score on two intermediate aggregate scores the dataset aggregate and language aggregate score thus weighing datasets and languages equally promoting model fairness, following Tatiana and Valentin (2021) and Chalkidis al. (2022a).
The final LEXTREME score is computed using the harmonic mean of the dataset and the language aggregate score. We calculate the dataset aggregate by successively taking the harmonic mean of (i) the languages in the configurations (e.g.. fe,fr, in SJP), (ii) configurations within datasets (e.g., OTS-UL, OTS-CT in OTS), and (iii) datasets in LEXTREME (BCD, GAM). The language aggre- gate score computed similarly: by taking the har- monic mean of (i) configurations within datasets, (ii) datasets for each language (e.g.. MAP, MEU for Iv), and (iii) languages in LEXTREME (bg.cs).
5.1 Evaluation Metrics.
We the macro-F1 score for all datasets en- sure comparability across the entire benchmark, since can be computed for both text classification and NER tasks. Mathew's Correlation Coefficient (MCC) (Matthews, 1975) a suitable score for evaluating text classification tasks but its applica- bility to NER tasks is unclear. For brevity, we do not display additional scores, but more detailed (such precision and recall, and scores per seed) and additional scores (such as MCC) can be found online on our Weights and Biases project.?
We do not address the dimension of the jurisdic- tion, which we consider beyond the scope of this work.
Results
5.2 Aggregate Score
In this section, we discuss baseline evaluations. Scores and standard deviations for validation and test datasets across seeds on our Weights and Biases project can be found in Table 11, 12, 13, 14. Comparisons with prior results on each dataset can be drawn from the tables provdided in section G in the appendix. Aggregated results by dataset and language are in Tables and 6.
We acknowledge that the datasets included in LEX- TREME diverse and hard to compare due to variations in the number of samples and task com- plexity (Raji et al., 2021). This is why we always report the scores for each dataset subset, enabling a
chttps://wandee ai http:/d....llleeeeeeeeeeeeeeeeee http///eteeeeepperreeeeeeeee

---

Model
BCD
GAM
GLC
SJP
OTS
C19
MEU
GLN
LNR
LNB
MAP
Agg.
MiniLM
$2.0
12.3
67.7
21.8
4.5
12.2
43.5
46.4
86.1
52.9
19.9
DistiBBERT
53.7
69.5
53.4
66.8
52.4
21.2
23.2
38.1
48.0
78.7
53.0
43.2
mDeBERTa v3
59.1
71.3
26.5
69.1
63.7
26.4
24.7
44.8
46.7
87.3
58.6
44.1
XLM-R Base
62.6
71.9
42.1
69.3
64.6
18.4
11.4
46.4
45.6
87.3
53.2
36.8
XLM-Riage Riage
58.0
73.1
71.7
68.9
73.8
20.6
27.9
45.1
55.4
88,4
55.1
48,5
Table 5: Dataset aggregate scores for multilingual models. The best scores in bold.
Model
bg
da
de
en
68
et
ga
br
bm
Iv
ml
II
p
41
SY
Age:
MinLM
20.9
20.4
19.8
27.6
8.2
21.2
19.9
40.2
11.9
20.3
203
14.9
20.5
15.2
30.5
25.9
20.2
12.4
20.5
18.1
DisHBBRT
32.7
1940
mDeBERIa v3
34.5
34,4
33.6
49.9
33.5
41.0
36.6
34.6
33.9
49.4
24.7
35.6
44.5
24,9
35.0
33,4
24.5
29.2
463
39.5
35.6
24,8
33.9
34,8
XLM-Raue
19.9
19.4
18.8
33.3
25.
27.6
19.4
18.8
18.6
27.3
44.9
116
134
31.0
18.7
18.9
16.0
15.2
21.
303
24.1
18.9
11.7
194
19.7
XI MIM-RRap
38.5
37.9
38.0
51.5
44.1
44.7
19.7
36.9
35.3
42.1
48.6
2X.1
22.7
48.0
37.4
37.9
34.1
19.3
327
48.9
42.
37.1
280
37.0
36.0
NatissLeguBBERT
NaiveBFRT
33.3
362
Table 6: Language aggregate scores for multilingual models. The best scores in bold. For each language, also list the best-perforning monolingual legal model under NativeL.egaBBERT and the best-performing monolingual non-legal model under NativeBERT. Missing values indicate that no suitable models were found.
Larger models are better For both, we see a clear trend that larger models perform better. How- ever, when looking at the individual datasets and languages, the scores XLM-R Base underperforms MEU (especially MEU-3; see Table and Table 12) leading low dataset aggregate score due to the harmonic mean. Additionally, low performance on MEU-3 has a large impact on its language aggregate score, since it affects all 24 languages.
Task
XIMMRRargy
ChatGPT
BCD-J
58.1
52.1
BCD-U
70.4
48.2
erratic. Note that
GAM
73.0
35.5
GLC-V
58.2
32.9
SJP
60.9
51.2
OTS-UL
79.8
15.1
OTS-CT
64.5
12.7
C19
27.7
23.6
Table 7: Results with ChatGPT on the validation sets performed June 15, 2023. Best results are bold.
Differing model variance across datasets We observe significant variations across datasets such GLC, OTS or C19, with differences as large 52 (in OTS) between the worst-performing MiniLM and the best-performing XLM-R large. MiniLM seems struggle greatly with these three datasets, while even achieving the best performance on GAM. On other datasets, such as GAM, SJP. and MAP the models are very close together (less than 6 points between best and worst model). Even though XLM-RLarg takes the top spot aggre- gate, it only has the best performance in six out of cleven datasets.
show strong performance with only few excep- tions (on Bulgarian, Spanish, Polish, and Slovak). In 13 of 19 available languages they reach the top performance, leading to the top language aggre- gate score, The few available models pre-trained on legal data (Nativel.egalBERR) slightly outperform multilingual models the same size.
ChatGPT underperforms We show compar- ison of ChatGPT with the best performing mul- tilingual model XI-M-Raage in Table To save costs, we limited the evaluation size to 1000 sam- ples for ChatGPT. We use the validation set instead of the test set to be careful not to leak test data into ChatGPT, possibly affecting future evaluation. Chalkidis (2023) showed that ChatGPT still out- performed by supervised approaches LeX. Similarly, we find that much smaller supervised models clearly outperform ChatGPT in all of tested tasks, with very large gaps in GAM and OTS.
Less variability across languages In contrast to inconsistent results the datasets, XLM-R1age outperforms the other multilingual models most languages (21 out 24). Additionally, we note that model variability within a language is similar to the variability within dataset, however, we don't see extreme cases such as GLC, OTS, or C19.
Monolingual models are strong Monolingual ceneral-purpose models (NativeBERT Table 6)

---

7 Conclusions and Future Work
argue that language models do not really learn "meaning". Bender al. (2021) further expand the discussion by addressing the risks related to these technologies and proposing mitigation methods. Koch et al. (2021) evaluate the use of datasets in- side scientific communities and highlight that many machine learning sub-communities focus very few datasets and that often these dataset are "bor- rowed" from other communities. Raji al. (2021) offer a detailed exploration of the limits of popu- lar "general" benchmarks, such as GLUE (Wang et al., 2019b) and ImageNET (Deng ct al., 2009). Their analysis covers aspects: limited task design, de-contextualized data and performance reporting, inappropriate community use.
Conclusions We survey the literature and select 11 datasets out of 108 papers with rigorous crite- ria to compile the first multilingual benchmark for legal NLP. By open-sourcing both the dataset and the code, invite researchers and practitioners evaluate any future multilingual models on our benchmark. We provide baselines for five popu- lar multilingual encoder-based language models of different sizes. We hope that this benchmark will foster the creation of novel multilingual legal models and therefore contribute to the progress of natural legal language processing. We imagine this work as a living benchmark and invite the commu- nity extend it with new suitable datasets.
The first problem concerns the fact that typically tasks are not chosen considering proper theories and selecting what would be needed to prove gener- ality. Instead, they are limited to what is considered interesting by the community, what is available, or other similar criteria. These considerations hold also for our work. Therefore, we cannot claim that our benchmark can be used to assess the "gener- ality" of model proving that it "understands natural legal language".
Future Work In future work, we plan to extend this benchmark with other NLU tasks and also gen- eration tasks such summarization, simplifica- tion, or translation. Additionally, deeper analysis of the differences in the behavior of monolingual general-purpose models versus models trained on legal data could provide useful insights for the de- velopment of new models. Another relevant aspect that deserves further studies is the impact of the jurisdiction and whether the jurisdiction informa- tion is predominantly learned as part of the LLM is instead learned during fine-tuning. Finally, extending datasets in more languages and evaluat- ing other models such as mT5 (Xue al., 2021) can be other promising directions.
The second point addresses the fact that any task. data, or metric are limited to their context, there- fore "data benchmarks are closed and inherently subjective, localized constructions". In particular, the content of the data can be too different from real data and the format of the tasks can be too homogeneous compared to human activities. More- over, any dataset inherently contains biases. We tackle this limitation by deciding to include only tasks and data that are based on real world scenar- ios, in an effort to minimize the difference between the performance of a model on our benchmark and its performance on a real world problem.
Acknowledgements
Joel Niklaus is funded by the Swiss National Re- search Programme "Digital Transformation" (NRP- 77) grant number 187477. Pooja Rani is funded by the Swiss National Science Foundation with Projects No. 200021_199000 funded by the European Commission's ' NextGen- eration EU Programme, PNRR- M4C2 Investi- mento 1.3, Partenariato Esteso PE00000013 FAIR Future Artificial Intelligence Research Spoke 8 Pervasive AL. Ilias Chalkidis is funded by the Novo Nordisk Foundation (grant NNF A006666))
Andrea Galassi is
The last aspect regards the negative conse- quences that benchmarks can have. The compet- itive testing may encourage misbehavior and the aggregated performance evaluation does create a mirage of cross-domain comparability. The pres- ence of popular benchmarks can influence a sci- entific community up to the point of steering to- wards techniques that perform well on that specific benchmark, in disfavor of those that do not. Fi- nally, benchmarks can be misused in marketing promote commercial products while hiding their flaws. These behaviours obviously cannot be fore- casted in advance, but we hope that this analysis of
Limitations
It important not exceed the enthusiasm for language models and the ambitions of benchmarks: many recent works have addressed the limits of these tools and analyzed the consequences of their misuse. For example, Bender and Koller (2020)

---

the shortcomings of our work will be sufficient to prevent misuses of our benchmark and will also in- spire research directions for complementary future works. For what specifically concerns aggregated evaluations, they provide an intuitive but impre- cise understanding of the performance of a model. While we do not deny their potential downsides, we believe that their responsible use is beneficial, especially when compared to the evaluation of model on only an arbitrarily selected set of datasets. Therefore, we opted to provide an aggregated eval- uation and to weigh languages and tasks equally to make it as robust and fair as possible.
and Aletras, 2021) to improve legal services and democratize law, while also highlight (inform the audience on) the various multi-aspect shortcomings seeking a responsible and ethical (fair) deployment of legal-oriented technologies.
Nonetheless, irresponsible use (deployment) of such technology is plausible risk, as in any other application (e.g., online content moderation) and domain (e.g., medical). We believe that similar technologies should only be deployed to assist hu- man experts (e.g., legal scholars in research, or legal professionals in forecasting or assessing legal case complexity) with notices on their limitations. All datasets included in LEXTREME, are pub- licly available and have been previously published. We referenced the original work and encourage LEXTREME users to do believe this work should only be referenced, in addition to citing the original work, when experi- menting with multiple LEXTREME datasets and using the LEXTREME evaluation infrastructure. Otherwise, only the original work should be cited.
While Raji et al. and Koch et al. argument against the misrepresentations and the misuses of benchmarks and datasets, they do not argue against their usefulness. On the contrary, they consider the creation and adoption of novel benchmarks a sign of healthy scientific community.
well. In fact, we
Finally, we want to remark that for many datasets the task of outcome prediction is based not on the document provided by the parties, but on the docu- ment provided by the judge along with its decision. For example, Semo et al. (2022) provide a more realistic setup of judgment prediction than other datasets, using actual complaints as inputs. How- ever, due to very limited access to the complaint documents, especially multilingually, creating such datasets is extremely challenging. Thus, most re- cent works used text from court decisions prox- ies. However, predicting the judgment outcome based on text written by the court itself can still be a hard task (as evidenced by results on these datasets). Moreover, it may still require legal reasoning capa- bilities from models because of the need to pick out the correct information. Additionally, we believe that these tasks can also be interesting conduct post hoc analyses of decisions.
References
Muhammad Al-Qurishi, Sarah AlQaseemi, and Riad Souissi. 2022. Aralegal-bert A pretrained language model for arabic legal text. In NLLP.
Nikolaos Aletras, Ilias Chalkidis, Lestie Barrett, Cätälina Goantä, and Daniel Pecotiuc-Piettoo editors. 2022. Proceedings of the Natural Legal Language Processing Workshop 2022. Association for Compu- tational Linguistics, Abu Dhabi, United Arab Emi- rates (Hybrid).
losif Angelidis, Ilias Chalkidis, and Manolis Koubarakis. 2018. Named entity recognition, linking and genera- tion for greek legislation. In JURIX, volume 313 of Frontiers in Artificial Intelligence and Applications. pages 1-10. IOS Press.
Vit Baisa, Jan Michelfeit, Marek Medved', and Milaš Jakubfček. 2016. European Union language sources Sketch Engine. In Proceedings of the Tenth International Conference on Language Re- sources and Evaluation (LREC'16), pages 2799- 2803, Portorož, Slovenia. European Language Re- sources Association (ELRA).
Ethics Statement
The scope of this work is to release a unified multi- lingual legal NLP benchmark to accelerate the development and evaluation of multilingual legal language models. A transparent multilingual and multinational benchmark for NLP in the legal do- main might serve orientation for scholars and industry researchers by broadening the discussion and helping practitioners to build assisting tech- nology for legal professionals and laypersons. We believe that this is an important application field, where research should be conducted (Tsarapatsanis
James Barry, Joachim Wagner, Lauren Cassidy, Alan Cowap, Teresa Lynn, Abigail Walsh, M'iche' al J. "O Meachair, and Jennifer Foster. 2021. gabert - an irish language model. In International Conference Language Resources and Evaluation.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long document transformer. CoRR. abs/2004.05150..

---

Emily M. Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrols: Can language models be too big? In FAceT 21: 2021 ACM Conference on Fairness, Accountabiliyy and Transparency, Virtual Event / Toronto, Canada, March 3-10, 2021, pages 610-623, ACM.
American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 226-241, Online. Association for Computa- tional Linguistics.
Ilias Chalkidis, Nicolas Garneau, Catalina Goanta, Daniel Katz, and Anders Søgaard. 2023. LeXFiles and LegalI.AM4 Facilitating English multinational legal language model development. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 15513-15535, Toronto, Canada, Association for Computational Linguistics.
Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: on meaning. form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, ACI 2020, Online, July 5-10, 2020, pages 5185-5198. Association for Computational Linguis- tics.
Iiis Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Damiel Katz, and Nikolaos Aletras. 2022a. LexGLUE: benchmark dataset for legal language understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties VVolume 1: Long Papers), pages 4310-4330, Dublin, Ireland. Association for Computational Linguisties.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Fric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πlya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. arXiv:2005. 14165 [cs]. ArXiv: 2405.14165.
Ilias Chalkidis, Tommaso Pusini, Sheng Zhang, Letizia Tomada, Sebastian Felix Schwemer, and Anders Sø- gaard. 2022h. eairlex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing. arXiv:2203.07226 [es]. ArXiv: 2203.07228.
Branden Chan, Stefan Schweter, and Timo Möller. 2020. German's 's next language model. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6788 6796, Barcelona. Spain (On- line). International Committee on Computational Lin- guistics.
Tobias Brugger, Matthias Stürmer, and Joel Niklaus. 2023. Multilegalsbd: A multilingual legal sentence boundary detection dataset. In Proceedings of the Nineteenth International Conference on Artificial In- telligence and Law, ICAIL '23, page 42-51, New York, NY, USA. Association for Computing Machin- ery.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hycontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi. David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai. Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slay Petrov, and Noah Fiedel. 2022. PaLM: Scaling Language Modeling with Pathways. Axiv:2204.023.1 [cs]. ArXiy: 2204.02311.
Taas Chalkidis. 2023. Chatgpt may pass the bar exam soon, but has long way to go for the lexglue bench- mark. CoRR, abs/2304.1 12202.
Iliss Chalkidis, Manos Fergadiotis, and Ion Androut- sopoulos. a MultiEURLEX and multi-label legal document classification dataset for zero-shot cross-fingual transfer. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 6974-6996, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.
multi-lingual
Tlis Chalkidis, Manos Fergadiotis, Prodromos Malaka- siotis, Nikolaos Aletras, and lon Androutsopoulos. 2020. LEGAL-BERTT The muppets straight out of law school. Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 2898- 2904, Online. Association for Computational Lin- guistics.
Hass Chalkidis. Manos Fergadiotis, Dimitrios Tsarapat- sanis, Nikolaos Aletras, Ion Androutsopoulos, and Prodromos Malakasiotis. 202 Paragraph-level rationale extraction through regularization: A case study on European court of human rights cases. In Proceedings of the 2021 Conference of the North
Ramona Christen, Anastassia Shaitarova, Matthias Stürmer, and Joel Niklaus. 2023. Resolving Legalese: A Multilingual Exploration of Negation Scope Reso- lution in Legal Documents.

---

Victor Hugo Ciurlino. 2021. Bertbr: a pretrained lan- guage model for law texts. Master's thesis, Universi- dade de Brasília.
JuriBERT; A masked- -laguugge model adaptation for French legal text. In Proceedings of the Natural Le- gal Language Processing Workshop 2021, pages 95- 101, Punta Cana, Dominican Republic. Association for Computational Linguistics.
European Commission. 2005. A new framework strat- egy for multilingualism.
Kasper Drawzeski, Andrea Galassi, Agnieszka Jablonowska, Francesca Lagioia, Marco Lippi. Hans Wolfgang Micklitz, Giovanmi Sartor, Giacomo Tagiuri, and Paolo Torroni. 2021. corpus for mul- tilingual analysis of online terms of service. In Proceedings of the Natural Legal Language Pro- cessing Workshop 2021, pages 1-8. Dataset URL:
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised gross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451. Online. Association for Computational Lin- guistics.
Stefan Dumitrescu, Andrei-Marius Avram, and Sampo Pyysalo. 2020. The birth of Romanian BERT. In Findings of the Association for Computational Lin- guisties: EMNLP 2020, pages 4324 4328, Online. Association for Computational Linguisties.
Ona de Gibert, A Gareía-Pablos, Montse Cuadros, and Maite Melero. 2022. Spanish datasets for sensitive entity detection in the legal domain. In Proceedings of the Thirteenth International Con- ference Language Resources and Evaluation 22), Marseille, France, june. European Lan- guage Resource Association (ELRA). Dataset URL: ptps://tinurl..cmmm/fff
Candida Maria Greco and Andrea Tagarelli. 2023. Bringing order into the realm transformer-baser language models for artificial intelligence and law. Artificial Intelligence and Law, To be published.
Neel Guha, Daniel E. Ho, Julian Nyarko, and Christo- pher Ré. 2022. Legalbench: Prototyping col- laborative benchmark for legal reasoning. CoRR, abs/2209.061
Javier de la Rosa, Eduardo G. Ponferrada, Manu Romero, Paulo Villegas, Pablo González de Prado Salas, and María Grandury. 2022. BERTIN: efficient pre-training of spanish language model using perplexity sampling, Proces, del Leng. Natural, 68:13-23.
Asier Guttérrez-Fanddio Jordi Armengol-Estap'e Aitor Gonzalez Agirre, and Marta Villegas. 2021a. Span- ish legalese language model and corpora. ArXiv, abs/2 12201.
Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. Bertje: A dutch bert model. ArXiv, abs/191..09552.
Asier Mutiérrez-Fandiin Jordi Armengol-Estapp Mare Pamies, Joan Llop-Palao. Joaquín Silveira- Ocampo, Casimiro Pio Carrino, Aitor Gonzalez-Agirre, Carme Armentano-Oller Carlos Bodriguez-Penagor and Marta Villegas. 2021b. MarlA: Spanish language models. ArXiv, abs/2107.07253.
Pieter Delobelle. Thomas Winters, and Bettina Berendt. 2020. RobBERT: a Dutch RoBERTa-based Lan- guage Model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3255-3265, Online. Association for Computational Linguisties.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTaV3: Improving DeBERTa us- ing ELECTRA-Style Pre-Training with Gradient- Disentangled Embedding Sharing. CoRR, pages 1- 17.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Patter Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248-255. IEEE Computer Soci- cty.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding- enhanced bert with disentangled attention. ArXiv, abs/2006.03654.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minncapolis, Minnesota. Association for Computational Linguistics.
Peter Henderson, Mark S. Krass, Lucia Zheng, Neel Guha, Christopher D. Manning, Dan Jurafsky, and Daniel E. Ho. 2022. Pile of Law: Learning Responsi- ble Data Filtering from the Law and a 256GB Open- Source Legal Dataset. LxXiv:2277.00222 [cs].
Dan Hendrycks, Collin Burns Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2021. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Stella Douka, Hadi Abdine, Michalis Vazirgiannis, Ra- jaa El Hamdani, and David Restrepo Amariles. 2021.

---

Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi- task benchmark for evaluating cross-lingual generali- sation. In ICML, volume 119 of Proceedings of Ma- chine Learning Research, pages 441 1-4421. PMLR.
Hudson, Eric Zelikman, Esin Durmus, Faisal Lad- hak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shihani Sanurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Ieard. Tianyi Zhang, Vishrav Chaudhary, William Wang. Xuechen Li. Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022. Holistic Eval- uation of Language Models. ArXiv:2211.099 [cs].
Wenyue Hua, Yuchen Zhang. Zhe Chen, Josie Li, and Melanie Weber. 2022. LegalRelectra: Mixed-domain Language Modeling for Long-range Legal Text Com- prehension. wcXiv:2212.08200 [cs].
Rasmus Hvingelby, Amalie Brogaard Pauli, Maria Bar- rett, Christina Rosted, Lasse Malm Lidegaard, and Anders Søgaard. 2020. Dane: A named entity re- source for danish. In International Conference on Language Resources and Evaluation.
Daniele Licari and Giovanni Comandé. 2022. ITALIAN-LEGAL- A pre-trained transformer language model for italian law. In EKAW (Compan- ion), volume 3256 of CEUR Workshop Proceedings. CEUR-WS.org.
Wonseok Hwang, Dongjun Lee, Kyoungyeon Cho, Hanuhl Lee, and Minjoon Seo. 2022. A multi-task benchmark for korean legal language understanding and judgement prediction. In siith Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretrain- ing Approach. CoRR, abs/2111.09543011
Maor Ivgi, Uri Shaham, and Jonathan Berant, 2023. Er- ficient long-tess text understanding with short-text mod- els. Transactions of the Association for Computa- tional Linguistics, 1 11:284-299.
Pedro Henrique Luz de Araujo, Teófilo E de Cam- pos, Renato RR de Oliveira, Matheus Stauffer, Samuel Couto, and Paulo Bermejo. 2018. Lener- br: a dataset for named entity recognition in brazil- ian legal text. In International Conference on Computational Processing of the Portuguese Lan- guage, pages 3 313-323. Springer. Dataset URL: https://huggingface.eo hww.r/ggaa.ccodatttss/l/eeeerree
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 3651 3651-3657, Florence, Italy, Association for Computational Linguistics.
Martin Malmsten, Love Börjeson, and Chris Haf- fenden. 2020. Playing with words at the national library of sweden making a swedish bert. ArXiv, abs/2007.01 00.658......
Daniel Martin Katz, Dirk Hartung, Lauritz Gerlach, Abhik Jana, and Michael James Bommarito. 2023. Natural Language Processing in the Legal Domain.
Louis Martin, Benjamin Muller, Pedro Javier Or- tiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoit Sagot. 2020. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 7203- 7219, Online. Association for Computational Lin- guistics.
Bernard Koch, Emily Denton, Alex Hanna, and Jacob G. Foster. 2021. Reduced, reused and recycled: The life of a dataset machine learning research. In NewrIPS Datasets and Benchmarks
John Koutsikakis, Ilias Chalkidis, Prodromos Malaka- siotis, and Ion Androutsopoulos. 2020. Greek-bert: The greeks visiting sesame street. In 1/th Hellenic Conference on Artificial Intelligence, SETN 2020, page 110-117, New York, NY, USA. Association for Computing Machinery.
Mihai Masala, Radu Cristian Alexandru Iacob. Ana Sabina Uban, Marina Cidota, Horia Velicu, Traian Rebedea, and Marius Popescu. 2021. ju- rBERT: A Romanian BERT model for legal judge- ment prediction. In Proceedings of the Natural Legal Language Processing Workshop 2021, pages 86-94, Punta Cana, Dominican Republic. Association for Computational Linguistics.
André Lage-Freitas, Héctor Allende-Cid, Ori- valdo Santana, and Lívia Lage. 2022.
Predicting brazilian court decisions. PeerJ Computer Science, 8:e904. Dataset URL:
court-decisions.
B. W. Matthews. 1975. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA) Profein Struc- ture, 405(2):442-451
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang. Christian Cosgrove, Christopher D. Man- ning, Christopher Ré, Diana Acosta-Navas, Drew A.
Dávid Márk Nemeskey. 2020. Natural Language Pro- cessing Methods for Language Modeling. Ph.D. the- sis, Eötvös Loránd University.

---

Joel Niklaus, Hass Chalkidis, and Matthias Stürmer. 2021. Suiss-judgmeett prediction: prediientprediiitonn: A multilingual le- gal judgment prediction benchmark. In Proceedings of the Natural Legal Language Processing Workshop 2021, pages 19-35, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John F. Canny, Pieter Abbeel, and Yun S. Song. 2019. Evaluating protein transfer learn- ing with TAPE. In NeurIPS, pages 9686-9698.
Vishvaksenan Rasiah, Ronja Stern, Veton Matoshi, Matthias Stürmer. Ilias Chalkidis, Daniel E. Ho, and Joel Niklaus. 2023. Scale: Scaling up the complexity for advanced language model evaluation.
Joel Niklaus and Daniele Giofre. 2023. Can we pre- train SotA legal language model on a budget from scratch? In Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Process- ing (SustaiNLP), pages 182. Toronto, Canada (Hybrid). Association for Computational Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108.
Joel Niklaus, Robin Mamié, Matthias Stürmer, Daniel Brunner, and Marcel Gygli. 2023a. Automatic Anonymization of Swiss Federal Supreme Court Rul- ings. ArXiv:3310.0463 [cs].
Gil Semo, Dor Bernsohn, Ben Hagag, Gila Hayat, and Joel Niklaus. 2022. ClassActionPredictionn A chal- lenging benchmark for legal judgment prediction of class action cases in the US. In Proceedings of the Natural Legal Language Processing Workshop 2022, pages 3 51-46, Abu Dhabi, United Arab Tamirates (Hy- brid). Association for Computational Linguistics.
Joel Niklaus, Veton Matoshi, Mathhis Stürmer, Ilias Chalkidis, and Daniel E. Ho. 2023b. Multilegalpile: A 689gb multilingual legal corpus.
Zejiang Shen, Kyle Lo, Lauren Yu, Nathan Dahlberg. Margo Schlanger, and Doug Downey, 2022. Multi- lexsum: Real-world summaries of civil rights law- suits at multiple granularities. In Thirty-sixth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track.
Joel Niklaus, Matthias Stürmer, and Ilias Chalkidis. 2022. An empirical study on cross-X transfer for legal judgment prediction. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Pupers), pages 22-46, Online only. Association for Computational Linguistics.
Jakub Sido. Ondfej Pražák, Pavel Pfibáñ, Jan Pašek, Michal Seják, and Miloslav Konopik. 2021. Czert Czech BERT-like model for language representa- tion. In Proceedings of the International Conference on Recent Advances in Natural Language Process- ing (RANI.P 2021), pages 1326-1338, Held Online. INCOMA Ltd.
Vasile Pais, Maria Mitrofan, Carol Luca Gasan, Vlad Coneschi, and Alexandru lanov. 2021. Named entity recognition in the Romanian legal domain. In Pro- ceedings of the Natural Legal Language Processing Workshop 2021, pages 9-18, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2020. Bertimbau: Pretrained hert models for brazil- ian portuguese. In Intelligent Systems: 9th Brazilian Conference, BRACIS 2020. Rio Grande, Bruzil, Octo- ber 20-23, 2020, Proceedings, Part I, page 403 Berlin, Heidelberg. Springer-Verlag.
Christos Papaloukas, Mīas Chalkidis, Konstantinos Athi- naios, Despina Pantazi, and Manolis Koubarakis. 2021. guutti-rranulr legal topic classification on Greek legislation. In Proceedings of the Natural Legal Language Processing Workshop 2021, pages 63-75, Punta Cana, Dominican Republic. Associa- tion for Computational Linguistics.
Andrea Tagarelli and Andrea Simeri. 2022. Lamberta: Law article mining based on bert architecture for the italian civil code. In ICRDI..
Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Trans- fer learning biomedical natural language process- ing: An evaluation of BERT and elmo on ten bench- marking datasets. In BioNLP@ ACLL pages 58 65. Association for Computational Linguistics.
Hasan Tanvir, Claudia Kittask, Sandra Eiche, and Kairit Sirts. 2021. EStBERT: A pretrained language- specific BERT for Estonian. In Proceedings of the 23rd Nordic Conference on Computational Lin- guistics (NoDaLiDa), pages 11-19, Reykjavik, Ice- land (Online). Linköping University Electronic Press, Sweden.
Matús Pikuliak, Stefan Grivalsky, Martin Konopka, Miroslav Blsták, Martin Tamajka, Viktor Bachraty. Marián Simko, Pavol Balázik, Michal Trnka, and Filip Uhlárik. 2021. Slovakbert: Slovak masked lan- guage model. CoRR, abs/2109.1525..
Shavrina Tatiana and Malykh Valentin. 2021. How not to lie with benchmark: Rearranging NLP leader- boards. In I (Still) Can' 't Believe It's I's Not Better Work- shop at NewrIPS 2021, volume abs/21 12.01142..
Inioluwa Deborah Raji, Emily Denton, Emiil M. Ben- der, Alex Hanna, and Amandalynne Paullada. 2021. Al and the everything in the whole wide world bench- mark. In Ne Datasets and Benchmarks.
Rena Torres Cacoullos. 2020. Code-switching strate- gies: Prosody and syntax. Frontiers in Psychology. 11.

---

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models.
Wenhui Wang. Furu Wei, Li Dong, Hangbo Bao, Nan Yang. and Ming Zhou. 2020. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS 20, Red Hook, NY, USA. Curran Associates Inc.
Hsiang-Sheng Tsai, Heng-Jui Chang, Wen-Chin Huang, Zili Huang, Kushal Lakhotia. Shu-Wer Yang, Shuyan Dong, Andy T. Liu, Cheng-I Lai, Jiatong Shi, Xu- ankai Chang, Phil Hall, Hsuan-Jui Chen, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung-yi Lee. 2022. SUPERB- SG: enhanced speech processing universal performance benchmark for se- mantic and generative capabilities. In ACL. (T), pages 1179-8492.00 Association for Computational Linguis- tics.
Chaojun Xiao, Xueyu Hu. Zhiyuan Liu, Cunchao Tu, and Maosong Sun. 2021. Lawformer: A pre-trained language model for chinese legal long documents, Al Open, 2479-84.
Liang Xu, Hai Hu, Xuanwei Zhang. Lu Li, Chenjic Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng. Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang. Kyle Richardson, and Zhenzhong Lan. 2020. CLUE: A Chinese language understanding evalu- ation benchmark. In COLING, pages 4762-4772, Barcelona, Spain (Online). International Committee Computational Linguistics.
Dimitrios Tsraapatsani and Nikolaos Aletras. 2021. On the ethical limits of natural language processing on legal text. In Findings of the Association for Com- putational Linguistics: ACI-IJCNIPP 2021, pages 3590-3599, Online. Association for Computational Linguistics.
Georgios Tziafas, Eugenie de Saint-Phalle, Wietse de Vries, Clara Egger, and Tommaso Caselli. 2021. multilinguua approach to identify and classify exceptional measures against covid-19. In Pro- ceedings of the Natural Legal Language Process- ing Workshop 2021, pages 46-62. Dataset URL: http::/tinyurl.co/yyyyyyt
Linting Xue, Noah Constant, Adam Roberts, Mi- hir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A mas- sively multilingual pre-trained text-to-text trans- former. arXiv:2010. 11934 [cs]. ArXiv: 2010.11934.
Shu-Wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai. Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang. Guan- -ing-TTig Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko-tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang- Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung-yi Lee. 2021. SUPERB: speech processing universal performance benchmark. In Interspeech. pages 1194 1198. ISCA.
Stefanie Urchs., Jelena Mitrović., and Michael Gran- itzer. 2021. Design and implementation of german legal decision corpora. In Proceedings of the 13th International Conference on Agents and Artificial Intelligence- Volume 2: ICAART,, pages 515-521. SciTePress.
Serena Villata. Michat Araszkiewicz, Kevin D. Ash- ley. Trevor J. M. Bench-Capon, L. Karl Branting, Jack G. Conrad, and Adam Zachary Wyner. 2022. Thirty years of artificial intelligence and law: the third decade. Artificial Intelligence and Law, 30:561- 591.
Ying Yin and Ivan Habernal. 2022, Privacy-preserving models for legal natural language processing. In Pro- ceedings of the Natural Legal Language Processing Workshop 2022, pages 172-183, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computa- tional Linguistics.
Antti Virtanen, Jenna Kanerva. Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: Bert for finnish. ArXiv, abs/ 12200776..
Ningyu Zhang, Mosha Chen, Zhen Bi, Xiaozhuan Liang, Lei Li. Xin Shang. Kangping Yin, Chuanqi Tan, Jian Xu, Fei Huang, Luo Si, Yuan Ni, Guotong Xie, Zhi- fang Sui, Baobao Chang, Hui Zong. Zheng Yuan, Linfeng Li, Jun Yan. Hongying Zan, Kunli Zhang, Buzhou Tang, and Qingcai Chen, 2022. CBL.UE; A Chinese biomedical language understanding evalua- tion benchmark. In ACL, pages 7888-7915, Dublin, Ireland. Association for Computational Linguistics.
Alex Wang. Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stick- ier benchmark for general-purpose language under- standing systems. In NeurlPS, pages 3261-3275.
Lucia Zheng, Neel Guha. Brandon R. Anderson, Peter Henderson, and Daniel Ho. 2021. When does pre- training help? assessing self-supervised learning for law and the caschold dataset of 53,000+ legal hold- ings. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, ICAIL
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. ICLR (Poster). OpenReview.net.

---

'21. page 159-168. New York, NY, USA. Association for Computing Machinery.
B Model Descriptions
MiniLM. MiniLM (Wang et al., 2020) is the result of novel task-agnostic compression tech- nique, also called distillation, in which a compact model the so-called student is trained reproduce the behaviour of a larger pre-trained model the so-called teacher. This is achieved by deep self-attention distillation, i.c. only the self-attention module of the last Transformer layer of the teacher, which stores a lot of contextual information (Jawahar et al., 2019), is distilled. The student is trained by closely imitating the teacher's final Transformer layer's self-attention behavior. To aid the learner in developing better imitation, (Wang et al., 2020) also introduce the self-attention value-relation transfer addition to the self-attention distributions. The addition of a teacher assistant results in further improvements. For the training of multilingual MiniLM, XLM- RBASE was used.
Haoxi Zhong, Chaojun Xiso, Cunchao Tu, Tianyang Zhang. Zhiyuan Liu, and Maosong Sun. 2020. How does NLP benefit legal system: A summary legal artificial intelligence. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8--3-0,,, Online, Association for Computational Linguistics.
A Experiment Details
A.1 Maximum Sequence Lengths
Brazilian Court Decisions: 1024 (128 8)
CoVID19: 256
German Argument Mining: 256
Greek Legal Code: 4096 (if speed important:
2048) (128 x 32/ 16)
Greek Legal NER: 512 (max for non-hierarchical)
LegaINERo: 512 (max for non-hierarchical)
LeNER: 512 (max for non-hierarchical)
MAPA: 512 (max for non non-hierarehical)
MuliEURLEX:: 4096 (or for maximum perfor- mance 8192) (128 x 32/ 64) Online Terms of Service: 256
DistiBBERT DistilBERT (Sanh et al., 2019) is a more compressed version of BERT (Devlin et al., 2019) using teacher-student learning, similar to MiniLM. DistilBERT is distilled from BERT, thus both share similar overall architecture. The pooler and token-type embeddings are eliminated, and the number of layers is decreased by factor of DiSTiBBERT. DistilBERT is distilled in very large batches while utilizing gradient accumulation and dynamic masking, but without the next sentence prediction objective. DistilBERT was trained on the same corpus as the original BERT.
Swiss Judgment Prediction: 2048 (or for maximum performance on fr: 4096) (128 x 16 32)
A.2
Total compute
We used total of 689 GPU days.
A.3 Hyperparameters
We used learning rate le-5 for all models and datasets without tuning. We ran all experiments with random seeds (1-3). We always used batch size 64. In case the GPU memory was insuffi- cient, we additionally used gradient accumulation. We trained using early stopping the validation loss with early-stopping patience of 5 epochs. Because MultiEURLEX is very large and the ex- periment very long, we just train for 1 epoch and evaluated after every 1000th step when finetuning multilingual models on the entire dataset. For finetuning the monolingual models on language- specific subsets of MultiEURLEX, we evaluated on the basis of epochs. We used AMP mixed precision training and evaluation to reduce costs. Mixed precision was not used in combination with microssoft/mdeebrtae-3333
mDEBERTa He et al. (2020) suggest a new model architecture called DeBERTa (Decoding- enhanced BERT with disentangled attention), which employs two novel methods to improve the BERT and RoBERTa models. The first is the disen- tangled attention mechanism, in which each word represented by two vectors that encode its con- tent and position, respectively, and the attention weights between words are calculated using disen- tangled matrices on their respective contents and relative positions. To predict the masked tokens during pre-training, an enhanced mask decoder is utilized, which incorporates absolute positions in the decoding layer. Additionally, the generaliza- tion of models is enhanced through fine-tuning us- ing a new virtual adversarial training technique. He et al. (2021) introduce mDEBERTa-v3 by fur- ther improving the efficiency of pre-training by
because it led to er- rors. For the experiments we used the following NVIDIA GPUs: 24GB RTX3090, 32GB V100 and 80GB A100.

---

replacing Masked-Language Modeling (MLM) in DeBERTa with the task of replaced token detec- tion (RTD) where the model is trained to predict whether a token in the corrupted input is either original or replaced by agenerator. Further im- provements are achieved via embedding sharing (GDES).
XLM-RoBERTa XLM-R (Conneau et al., 2020) is multilingual language model which has the same pretraining objectives as RoBERTa (Liu et al., 2019), such as dynamic masking, but not next sen- tence prediction. It is pre-trained on a large corpus comprising 100 languages. The authors report sig- nificant performance gain over multilingual BERT (mBERT) in variety of tasks with results com- petitive with otate-of-the-art monolingual models (Conneau et al., 2020).
Monolingual Models Overview

---

Model
Language
Source
Params
Vocab
Specs
general
hurfnoose/rooeeta bulgaria
bg
52K
200K steps
(Sido 2021)
109M
31K
50K steps
MaltehhViunish--beer hotto
da
(Hvingelby al., 2020)
111M
32K
IM steps 1280
dbmdz/bert-basee sese-bbase--eee german-cased
de
110M
31K
1.5M steps
deepset/gbert-bbse
de
(Chan et al., 2020)
110M
3IK
30K steps BS 1024
alpaucb/bert-ta base-gress greek- greek-uncased-x1
(Koutsikakis ct al., 2020)
113M
35K
IM steps / BS 256
roberta-base
cn
(Liu ct al., 2019)
125M
50K
500K steps BS 8K
bertin-proecc//errinn- httpepppjjcaeeeeti--beet-------e bubertu-buse-spniish
es
(de Rosa et al., 2022)
125M
50K
250K steps BS 2048
PlanTL-GOB- GOB-ES/roberts SS/roberta-base-bee
(Gutiérrzz Fauiérrez-Fandiio et al., 2021b)
125M
50K
10K steps BS 2048
(Tanvir et al., 2021)
124M
50K
600K steps
TurkuNLPPeer--ee finnish cased-v]
2019)
steps 1120
camembert-base
(Martin 2020)
32K
100K steps
dbmáz/hert.i bwse-french-eeeeem fwench-eurppeammccaee
111M
32K
3M steps 128
DCU-NLPhert base-brish -ree--rrs---c--vv
ga
(Barry al., 2021) (Nemeskey, 2020)
109M
30K
100K steps BS 128
SZTAKI-H DETAKI-HLThhalee HLTThhbert--baeeee baaeettt---eeeeeeee
hu
111M
32K
600K steps BS 384
Musi htw.c///wnbbeetooommmmmeee
111M
32K
dbmdz/hert base italian-casee cased
110M
31K
2:3M steps
GroNeP//ert- Pewe-P/ttttttteeeeeeeee bas---uchueuuueeeeeeed
(de Vries et al,, 2019) (Delohelle al., 2020)
109M
30K
850K steps
mdelahelle/mmheett www.eeeeeeeeeteeteeeeeeaeaaa
2-duucc--use
117M
40K
16K BS 8192
dkleczek/bert-Be base polss-s-ooiiss--eeeed----
pl
132M
60K
100K steps 256
neuralmind/berr-bag porteee-ooouuuueccaaa
pt
(Souza et al., 2020)
30K
steps
duminesccustefam/eer www.rruuu-e---annn-inteo un
(Dumitrescuu al., 2020)
steps
gemolata/slovakbert
(Pikuliak al., 2021)
125M
50K
300K steps BS 512
KB/bert-base-sweet -eeessseddsisseeeeee
(Malmsten 2020)
125M
50K
100K steps BS 128
legal
slucia/customm-B Ahucia/customm-egalleen
(Zhng al., 2021)
32K
steps
olpauch/legal base uncased
(Chalkidis 2020)
31K
steps
Coutiemez-Pandiike 2021a)
126M
52K
BS 2048
dlicari/ltalian- BuRR readerbench/juBBERTThse
(Licari and Comandé 2022)
111M
32K
8.4M steps BS
(Masala et al., 2021)
111M
33K
Table 8: Monolingual models. BS is short for batch size. For a detailed overview of the pretraining corpora, we refer to the publications. For some models we were not able to find publicationss/pees..

---

D
Dataset Splits

---

/66616666 560
176 / 563
/01/111111 550
1457 166//666666 /
1529 174 F
893 /98 4408
8066 248 115
1391 /163 /15
1398 1187 /531
1297 99 490
1665 /
1711 /
/62//6////// 550
/413/ 173 548
167 153
937 /93 /42
1391 1614/14111 /640
1OS 3990
115 /55
166 125
1453 175 139
Table 9: Overview of number of examples each language- specific subset of multilingua tasks. The order of the values is train validation /tstt.
MAP-F
1411
1464
1455
1383
1390
1411
1383
1088
1480
1395
1411 / 166 560
176 563
164 550
166 1558
174 5884
806 /424 /155
163 /516
187 /331
99 490
1383 166 /
111//11555 /55
162 /50
1413 173 1/33 548
1353/ 167 /67//177/55 /53
/91/11 164 530
100 390
1775 /
165 526
1453 175 539
8931 198 4408
117//93/2 097//9/442 /442
1171
MAP-C
937
1464
1455
1457
1529
1.391
1.99
1297
1390
1411
1391
1086
1480
1395
5000 5000
5000 5000
5000 5000
5000 5000
5000 5000
SOOD S0X0
S0O0 5000
500D 5000
5000 5000
5000 5000
2500 5000
5000 5000
5000
5000
5000 5000
5000
5000 5000
5000 50000
S000 50000
5000 5000
5000 5000
5000 5000
42490 5000 5000
5000
5000
5000
MEU-3
15986
23187
55000
55000
55000
55000
52785
23126
42497
55000
7944
22664
55000
23188
23208
17521
55000
23197
52370
15921
22971
23184
5000
5000
5000 5000
5000 5000
5000 5000
SOOD 5000
S0O0 5000
23126 500D 5000
5000 5000
5000 5000
2500 5000
5000 5000
5000
5000
5000
5000
5000
5000 50000
50000
5000
5000
S000 5000
5000 5000
5000
5000
5000
5000
5000
5000
5000
S000
5000
5000
MEU-2
15986
23187
55000
55000
55000
55000
52785
42497
55000
7944
22664
55000
23188
23208
17521
55000
23197
52370
15921
22971
23184
42490 5000 5000 42490
5000
5000
55000 5000 5000
55000 5000 5000
55000 5000 5000
55000 SOOD 5000
52785 S00D 5000
23126 5000 5000
42497 5000 5000
55000 5000 5000
2500 5000
22664 5000 5000
5000
5000
23208 5000 5000
5000
55000 5000 5000
23197 5000 5000
5000
15921 5000 5000
22971 5000 5000
23184 5000 5000
15986 5000
23187 5000
55000 5000
23188 5000
17521 5000
52370 S000
MEU-1
7944
1416/ 178 1178
648 F81 F81
75 10/10/10
742/93 /93
221 /28 F28
/ 18/18
75 10//00
C19
742
1027
1038
4806 432/ 1057
540/ 507 109 5278 439 11755
404
415
OTS-C7
4480
5378
103
103
517 50/ 102
O OTS-UE
491 42/
5261 49
35458 /4705 9725
21179/ 3095 6682
3072/ 408 812
Language SIP
bg
CS
da
de
el
en
65
0
fr
ga
hr
hu
It
It
Iv
mt
mb mL pl
p1 T sk
sl SV

---

24 16/13 13
30/ 17 16
26 14/11
78/14/14
31/17/20
28 18/17/18
26/13/15
25 14
24/19 19//1/16
19 /
33/17/18
20/15/14
25/15/16
28/19/21
31/15/21
27/15/15
12/12/14
24/14/18
26/16/18
25/16/18
23/ 15 15
Table 10: Overview of the number of labels for each language- Sanngage--spciitiee specific subset of multilingual tasks. The order of the values is train validation test.
MAP-F
32 92/199
25
MAP-C
11/11/8
11/11/9
11/10/11
11/9/
11/11/11
11/9/9
11/8/11
11/11/11
11/11/10
11/11
13/11/11
11/10/10
11/10/11
I1/
11/11/11
/1/11/11
10/9/10
11/10/11
11/11/11
11/11/11
11/11/10
481 /454 465
/454 465
/454/ 465
/444//5//// 465
466
0000/454// /054/4400 465
/ / 465
40///54//400 /55/4655
465
454 465
500 4444 465
1/37/466 466
486 4544 465
500/ /444 /454/465
465
486 /444 /444//65
465
/544/54//40 /454/465
466
/544 /54/465
465
500 /454 465
/454/ 466
/444//4//// 1/54/465
465
/454////// /465
/5/4//4/// 1454/466 /66
465
/93 454 465
500/454
500/454:
486/4541
493/4544
469 4437
486/454
485/4544
497/454,
086/454-
MEU-3
497
486/458
486
127 / 126 127 500
/ 126 127 500/
500/
497
486
493
486
485
127/1261 126 127 486
127/126// 127 497
481
485/
485
486
127 126 127
126 127
127/1261 127 127
127 127
127 /27/126/ 126 126//222 127
127 127/126/ 126 17//126/11 127
127 126 127
127
127 126 127
127
127
127
126/ 127
127
12//126/122 127
1262/122 127
127
127/126/ 127
127/ 126 127
/2/1122/12 127
127 126 127
MEU-2
127/
126/
127
/27/126/
127/
127 127/
127/
127 126
127 126
/
127
127
127
21/21/22
21/21/22
21/21/21
21/21/21
21/21/21
21/21/21
21/21/21
21/21/21
21/21/21
21/21/21
21/21/21
21/21/21
21/21/21
21/21/21
21/21/21
21/21/21
21/21/21
21/21/21
21/21/21
21/21/21
21/21/21
21/21/21
21/21/21
MEU-1
6/6/5
8/817
4/1/1
7/7/6
7/5/6
2/212
7/5/3
OTS-CT C19
9/719
9/8/9
9/8/9
91819
OTS-LL
3/3/3
3/3/3
3/3/3
3/3/3
2/2/2
2/2/2
2/2/2
SJP
Language
bg
da
de
el
en
es
fi
ga
hr
hu
it
It
ly
mt
nb
pl
pt
ro
sk
sl

---

E
Detailed Multilingual Results

---

49.3 49.3
48. (43.22
51.1 (22.7)
558 (13.)
39.9 31
45.7711.00
(1.8(12/4
48.7041-1)
MAP-
MAP-F
558
MAP-C
7941
76. (8.2)
734
77. (42.0)
Table 11: Arithmetic mean of macro-FI and standard deviation over all seeds for multilingua models from the validation set. The best scores are in bold.
MAP-C
62.7
65.69477
58.0
6401225
61.3
Table 12: Arithmetic mean of macro-Fi and the standard deviation over all seeds for multilingual models from the test set The best scores are in bold.
(40.1)
93.0 2300266
96-2 (4039)
96.8 (6.....)))
(40.3
$7.4 37/401100
LNB
98.2
LNB
861
78.7
87.3
88.4
62. (464)
60 9 (880)
78.2 (43.)
(a1.))
(20.7)
15.0 (30.8)
(4.9)
LNR
LNK
16.5
85.8
48.
064414
(24.5
63.74400
64. (42)4
63.04..0
42.4
(11.9)
GLN
GLN
46.4
45.5
(.33)
13.8 (806)
13.3 (808)
5.6.2003)
12.0 (01.)
(200)
14.2 (20.8)
MEU-3
MEU-3
23.5 (41.))
37.3 3)
32.2 (+1.))
47.6 47.6 (90.4)
21.7 (.0.5)
29.8
44.4 (20.5
MEU-2
MEU-2
38.6
34.1
(50.3)
(60. 1)
69 2 (e0.1)
(200
65.6 (30.1) (30.1)
68.1 (21.2)
MEU-
64.8
MEU-1
68 2
71.2
63.6
65.0
5816151
414465.
434
#11 (+79))
538 (1.0.)
6.
459 (3110)
OTSCT CD
C19
393
407
542
(53()14)
496
52- (=50)
629 (E6
OTS-CT
(A3
46.2 (28%)
55.9
66.9 (G0.5)
OTS-L
7101c5.0)
669.63.
22
849
00X- tIL.
(+38)
836
61.
(00)))
6 (e00)
69.6 (=00))
700
67.6111
66.7 66.711111
68.3 (s100)
SJP
71.2
SJP
69.0
68.3
5.1
339912200
(223
15.7 (412.))
389 (33.7))
5.2(1.1)))
(5123)))))
CLC-
GLCS
33:2
39.1
(200)))))))
69.1 (200))
73.9 (20.0)
39-1 (41-))
20.0 (213))
(678)
20.7 (110)
74.7 (21b.9)
GLCC
GLCC
84.92..5
856 656 (402)
58.2 (550.)
73.3 (0.99 81.7 (...0...)
86.1 (10.4)
58.5 (130.3)
OLC-V
GLC-V
85.0
85.2
(#0.))
70.5 (1.7))
72.5 (2.99
(1.4)
71.3 (20.3)
72.0 (2.4)
CAM
72.1
GAM
69.5
23.1
634 (=12.3)
61.6 (211.2)
BCD-U
(560))
60.0 60.0 (.9.)
7044 (s37)
BCD-D
56.7 (a7.))
53.2
55.
634
58.
73
52.1 (64.5)
67.5 (:2.))
58 (69.3)
(67.b
(63.6)
Mean BCD-J
Mean BCDJ
52N
49-4
50.3
67.2
65.1
55.1
$1.0
50.3
59.0
63.6
46.4
535
55.6
5K.9
MhotMM
X1
XLM-Ruu
MiniLM
DimIBERT
XLM--a-
XLM RuM

---

F
Detailed Monolingual Results

---

52-1e-54)
372667
m2
(00200)
61133
(2.011
3021813
35
S13624
MAP-F
MAREC
624 (221)
502 (603)
(02
999 2300
182 (22000
00
84. 3588
26006
713/2100
05X (218))
743
75.
066 (600)
LNB
Table 13: Arithmetic mean of macro-Fi and the standard deviation over all seeds for momolingual models from the validation set.
55. (607)
LNR
GLN
37665
27.3 GU0.0
2500110
272600
1511899)
2101007)
2:3013
186629
257
20-1540
263810
262010
130/200
2786080
2596033
219tm
103603
MRDS
MED2
467
11
49
48.
513008
41.7/22
208 00 5130005
66 457648
45.512
35-
51.5
(5577
0151200
005122
000000
211 12009
MBN-44
67.4
68.
162 (1.))
362 0556)
21
292 (550
CD
381
20-4
O1SCT
(43.9)
AKT (63.5
42.
7631455 (651)
714 (4160)
O1S-I
907
934
590
709111
715tl3
08
702
37.1550
SIP
GLCS
GLCC
GLEA
GAM
Moa RCDJ
52.3
0.3
5722
58.3
6.8
3088
57.1
12.3
50.0
65.8
59.9
47.5
63.1
55.0
dumine www.inooottiiiiieoooe hent-- Sww. www.eeeeettnnennne... www.enantttttaaaaa
aodinv wwweehheninnnne www.c----------eooo sparish
deceaale ou be pobih pwl.b-atcacco stcanche
sou disirdeen has portLEDD cood
inmooeemeee but bripiiu
Par CON Aerttt how foee
UWB- AIR Cocn B--oo Mase-cad
aeriaaccmmmmme legileen
peoii Weotaaaoooeee
Werrd--ber ber boe
smmmmm bose
DCL-M ACL--LL

---

12.1 6o0.2
47762.0
04
166688
(00280)
3.763
41771-
33
MAP-
MAREC
02. (831))
043 (278) (518
S40/600
2641155
5150004)
00
4230113
000
(229022)
5155115
420
560
03.
522
872 (AR
LNB
431 6511
Table 14: Arithmetic mean of macro-Fi and the standard deviation over all seeds for monolingual models from the test set.
LNR
GLN
250500
0912000
13161
219:6000
2511131
173650
23768
25-400)
250
253
25AG03
226604
2436033
1831000
MIDS
MED2
43.10220
47
457000
4521204
403000
57.5222
424
422043
6.5
47.3
45.
651 1:00
65.4 123.00
65
66 15000
65912200
65712111
65:55
60:20
674151.5
659
8151500
MBN-4
06.1
137 (1
360 360
(13660
127 (2.7)
C9
O1SCT
409122.
60 (64M
700 L830)
6TS (467)
O1SII
769
853
087111
0926141
7026144
574
SIP
GLC4
GLOC
GLEA
72012114
(5.1
GAM
645 (A63) P0R
Mon RCDJ
66.7
53.6
46.7
42.9
412
29.3
67.1
57.6
48.6
38.4
50.0
50.2
dumine www.inooottiiiiiieooea hent-- Sww. www.eeoenttnnennneme wow.eaatttataaaa
dececale ou be pobih pol.b-ttcacco stcanche
sou disiedeen has portuaDD cood
unmooeemeee but bripiiu
Par CON Aarttt how foee
Smnihher Shac french www.checneeeeee.............e..e caneerraan
UWB- AIR Cocn B-loo Mase-tad
aeriaacammmm legaleen
peoii Weotaaaoooeee
Werrd-/ber boe
smmmmm bose
DCL-M pell-Attthheiie

---

G
Original Paper Results
this section, we present an overview of scores for each configuration of the LEXTREME dataset as provided in the original papers. When certain configurations were not available, no scores were obtained. It should be noted that different papers provide varying scores, making direct comparisons with our results challenging. Additionally, the variability in the training and evaluation procedure used across different papers may impact the resulting scores, which is an important factor to consider. To gain a better understanding of the training and evaluation procedure please refer to the cited references. The LEXTREME scores are calculated by taking the arithmetic mean of each seed (three in total).
Store
Methed
Traislang Toilang mampenin nurooaa iecreel
sirrppccic
niceomed sécre.fi pecciéee rmall
seneery
(2102)
Radincied
(BL.STM)
Lap--hstuc (202))
Conmurtonal heard Netweke (CON)
Lags Intus a- (2003)
Encuse me
631
23.0
614
720
Lape-rntue mal (2002)
alrene Grsden Bourtag (iCatoom
F!
sy (2007)
Navo (DN3)
Eign Fusks (2007)
Long Shxt-Tra Merwey (ISTM)
T10
630
610
710
Lags trsta a- (2122)
rankes tnioe
250
320
360
610
Laps (2102)
aspprt reactracto
F1
953
Intoran
MMating al Mili Mr 19-856
11.
535
164
93.
91.
913
400
Intevas
conabbkaalhn hooe par
68.5
04.
Table 15: BCD-J. The best scores are bold.
Samme
Methed
Trsislang Teilanz nxcepriin sucriaa marrsd]
sérnepericn sime-mll wirreffi
periiden
swall aarrrry
Te (2003)
Bifonieed
Then Moxor (82 (TMSTM)
Fag Tedus al (2007)
cine Neaul browods
96
$8.0
(800
96.0
Lag thota a- (2122)
Exten noe
513
55.
77/
1933
Lig--ratta aal 22122)
circne Grddan Boanay (XUltoot
Triluc (2003)
Candda Naw (DNI)
Lag Trits al (200))
Long Shen Mervey (ISTM)
9901
180
660
96A
Lagr Trstuce (2003)
frasd
F
Lag-Prrita
agper veeue raadio
11
31.5
573
512
96.4
55.0
208
167
86.
837
13.7
Inturas
Table 16: BCD-U. The best in bold.

---

Seence
Methed
Tratdant
TelLan surnegniii mavod mardl
sinsprakie nknemal nkooli
pritió
melll
arceriy
latdo (222))
XL L.M.iadtati pewetttttttttttootooooo Cn1
35.3
02.5
Iadio a.4 (222))
NJ So C11
41
1930
723
543
063
ladeo
3533
badoo (222))
343
153
05
900
Tikds (00N)
985
364
Tiide au (207)
955
P1
TLI
Tidds aut (071)
sedillyeod pedilloooopprrgo purrpca
185
WI
egpur
Tikde (300)
clotf A.A.AIR pns ois C14
550
905
Tikde (271)
futtoe 30 M. RUATRD prnceialien on Cis
533
960
batte ast (220)
cmm-bbe chattianne M1.4kt-tttn pearazdd
71.
213
26
Taltu au (220)
chanitaatie MLMittttinn prorandd CIs
255
Isuts ax (221)
www.brchuttttee MLSk-ielltt pronatad CTS
15.5
135
189
Tiide aul (221)
MLM--teiit peananed
423
370
200
543
lozces
casveher-taa hast
D
31.5
U5
137
220
343
223
losxes
cheucaun carpcc-crrprraaaadd
263
loscse
300
750
603
Intene
183
41
523
BEKT
33
31
44
135
163
Iatene
41
750
16.5
lanere
Fibradbrlggi wagiggilteee herp-hh--eieee uncoeed
965
223
264
121
433
412
513
lazees
podlside aediseelattnneree
02
43
100
03
101
556
34
16
0.3
160
38
03
00
910
64.
akccalsarees lepalben
562
Table 17: C19. The best scores are in bold.
Metiod
Traislang
incroprniiee
Ucta a-d. 123211
110
22.11
17.0
X3.5
1hcls a (703/1
aillie regrosionn
10.0
48.0
Usta a.d. 122211
B-abi suppon rradine
67.1
78.0
77.5
thote a (M37))
340
674
760
Usda ddd.(222) (2321)
Ungranniapprt vodtr machace
500
160
05.
14.0
lecrrew
(beadohun haut jommicaast
de
66A
764
12A
913
913
932
913
deguetiglen has
de
T1A
20.
75.1
82.
828
425
82.
782
sto maveebrrtb sllv taee
863
15.3
756
7955
704
789
73.1
81.
810
11.
01.0
Table 18: GAM. The best scores are in bold.
Methal
Trainlang
Teelang
sumn-rrcll mamm-f micr-ppccbiie mirneraa micnel
Papancica alll (2101)
BAGRL-ATT
$131
116
$1.0
Papaboica ctal (2521)
-WAA
7131
310
13.0
Papaboica eta.. (2021)
BIGRU-MAX MAX
75.0
180
150
Pypatoice at (220))
$311
17.0
Papanxice all. (2t2))
GRIEK-LIGAL BRGAL---GGALRRRTT
3111
112
140
Papaoodios
800
100
Papsootoo
Suppot
Bag BOW)
ROSERTS
distlhet wnd.tingualloooomm asel
63.1
805
80.5
katrin:
51.
91.
332
31.2
11.2
72.2
katrrn::
rilzssed/hhuniiggnaa... Mww...M....2... L12-41384
42.5
41.
304
66.1
8611
86.1
olperttfher howe bese-gewk wwwelggg-eeeeeeoeee rncosedev stre reberstiioo
765
765
85.3
153
153
853
113
619
60.
13.
13.1
13.1
123
stre-mtuttt-brr:
17.3
349
347
515
145
$4.5
865
Table 19: GLC-C. The best scores are in bold.
Methol
Trainl
nacra-eeiiion surmevolll
aicropreciin aimorrolll nimrof
Prpaioiio at (7101)
BXCRI -ATT
75
15,
15.3
Papabolia etal. (2021)
BAGRL-LWAN
65.0
550
6500
Papatolis et.. (2021)
BIGRU MAX
63
536
630
Papatolis et (2021)
GREEK BERT
7500
390
100
Phpabnica cta. (2021)
GREEK-LBGG BEGAL-BERTT
$1/1
B1A
Papainica
330
118
Pypsooice
Support
diothenn www.iinggdd..ooooo
345
644
ALL
15.1
157
407
167
162
ALT
Eiltosn/idiiiigaall MimLM L12-004
7.3
52
2822
25.2
25.2
26
latrene
Hipeachfaast.o gieek uncsed-vi site veberta base
668
655
63.8
803
103
103
803
Inttene
15.
18.6
15.4
42.2
42.
43.3
423
kntrene
Str.-mberta-ams
39.5
415
363
33A
35A
55.6
536
Table 20: GL.CS. The best scores are in bold.

---

macrspeniiin
sieneprekion
Peperoice at (2101)
BEGRRU-ATT
31.11
$60
Pepakoiin et.. (2021)
BIGRU-LMAN
540
140
140
Pepalotia etd. (2021)
MAX
540
140
140
Papaootio ot (2221)
GREEK REKT
8111
15.0
15.0
Papaksica eta. 12501)
UREEK LEGAL-SERT
5331
195
150
Papaioica etd. (2021)
MBEK?
360
160
160
Pepanvice aaal. (220))
Suppon Vicao Madiinss Weog
810
150
Papanxice (210))
730
310
Papabxica (2001)
150
565
reberta
564
530
553
620
830
630
620
Table 21: GLC-V. The best are in bold.
matre-grechie
mice-precaion
aecarsy
Angelids BILST-LBILSMM-LL
71A
920
lesteme
Almo zoberta base
6
75.7
73.8
74,6
91.5
91.3
97.4
9710
73.1
76.3
74.1
97.8
91.3
97.4
91.1
Table 22: GLN. The best scores are in bold.
Saam
Medsel
Trandang Imilasy nanggmmii inamoral isrrll
tempriciee Hommal skrill BYIES
Calliaa ned aniakligge MARCCLL www.eSioooH......A.
Peresl (30))
Citcla net canokligge MARCCLL war calccll AingsSBLLLSSSE..... gastoo
345
net
Aww.AnneBLLLAAAAA... pastaco sikk
Cabila
Tal 03.e-11330 (303)
Cilcia ned can.AAige--Bnn LITSACE paoctcca
151
Par (301)
Lpa
133
MARCTLLL pupi
(303)
153
Far (303)
MARCTELL wusdechec 20 CRT 2 giotsrre dfo
534
Pareaa
konts
233
353
253
12.3
360
18.0
564
kosts
235
360
konss
152
135
345
17A
721
965
konze
15.
153
352
113
R22
913
563
Table 23: LNR. The best scores are in bold.
Suunce
MeDhol
Triilay Tislase esscopreciin
nucirnca raciel nittoprciie
skntonal sknedl
gislãe
SCCETAO
Lanre 1300
LSTT dheg den www.wwwwweeeee...... www.eeieeaeeeeeerr
913
91A
9L.5
54.5
902
511
11.
11A
611
811
541
912
211
15.1
1ia volato
802
Table 24: LNB. The best scores are in bold.

---

Methed
Chicceo et (2001a)
con enboma Rour
11
Chalceo atal::0
Chekozo (202)a)
stx-vrbettate
Chttrz at (222)-)
a1
Ohdccdo cial (212
a1
Cheluco
skk-rthettale
Chatzzo (200)
(hebcio ctal (200
x1
Osicce etal (2021a)
31
Chtice 022
Chllzzo (202la)
Oubide aial (201
OuBilk
Chelcdo (222al
Oatoit (10) drembimntou
Checcio (21).a)
x1
Ostcca ztal: (2021a)
31
Chticee
Mipitos kyas
Calozz (222
trembmattu Mbzin
Cubiia vtal
dembetebw Aukies kyas
-1
663
Oxbi/k (10
dembbtteew Adggos kyys
Chaccco 202
diementttt Aapann
drombmmroo Auppor
Outilb (10)
Clelcto (202la)
strunbataaaa Mapin
Cbtoca at (22)
dremam-bor Adper byan
-1
634
Chalcre
crmmmm-bo Adppo bym
atonbaaa Mipion kgas
Chllcz (222a
dremmmatt hor Mbzen
Clebii a
demnbatelas Adgless kyns
66.
shooraateeee Mlgion kyms
Chalozo atal attial
chr-mamattuu Auppan boan
deesbmnto Auppess
Oubilk atal (100
Chelozo (2t2ia)
ste Mapin
Chalede at
dremmn--bo Adper byan
Chalsck
crr-nam--ao Adbpo byrn
Chaludo
shr-rbattatee I Mipinn
leonzne
211
752
52.8
lestese
317
395
353
410
735
53.4
05
lestese
260
128
161
360
563
27
consrmbnn-tat
38.
103
363
360
136
220
125
decpeciahen haic
42
66
222
22A
41%
724
304
100
111
318
16
200
dansiees scceaferrieee beceeeeeee -nosel-
235
3.0
13.5
452
563
06
200ce
govd eevieabinall
155
12.
11D
135
255
365
13
250
25.3
445
32
heigaties
1g
351
133
Manctiitaraaa hembbens
346
120
19
pec.octhinakeeeteaoo viibus
41
125
122
11.1
307
825
28L1
122
231
20.0
320
700
73.1
12.1
65
alpeadslgaa www.wl--eeeeeeeeee casceaaccoa
43.0
243
356
720
S1A
04
41
193
153
13.
150
345
000
16
30.9
153
KSRT
655
631
011
110
03
100024
123
25.
133
31A
25D
314
10
SPIANNI Moddeeeseeeee
1a
305
577
11
100722
21.7
M1
66
she-vnbata-tauu
47
112
315
07
ste-rrhata-agg
21
41
193
343
142
160
545
000
12
368
717
Table 25: MEU-3, The best scores are in bold.

---

Ssurce
Melled
TruisLang Tealang mUAT-grrciin
matroorval maste-i1
mkkv-pprsii micro-cccall
uiccoff
uccurscy
Niteus (2021)
Fracch
Mitoss (3021)
Teach Henrchicl BEXT
8012
Nillas stal (2021)
Frosc Lotg BERT
8 de
63.0
17.2
Nittess call (11(1)
Carmas Hicnadioi BEXT
de
65.3
77.1
Nillas ctal (2021)
Carmas Long BEKT
de
de
619
76.5
at (2171)
de
617
76.0
Nilluss et (3021)
Jullin BEKT
B
53.
76
Nilln ct (2021)
Itainn Hicraciica BERT
51.1
758
Millas et (2021)
Ialiin Long BERT
39.
77.1
Nittn (1I21)
Mahiingua SStt
d2
58.2
68.6
Nittrs (2011)
Mattlingu
Nillas
Maliinguu
Nillus (2021)
Matilingual Hleaacccca BERT
de
Hicnndiss BERT
Muhhingial
Mattiliguual Long
Stttiligusall
Nillas (2021)
Malilingul Long
lecterue
canonbett-taec
Ircieree
shendshurts www.fffcnnnnerrr.ccooom
Iecieme
123
lestrenee
dhrddhhee base kollan cased
11
63.
36A
50.3
79
lestsuue
despac/ybett Buac
dc
63.5
716
69.
76.7
763
767
76.7
241
633
713
66-7
143
263
14.3
74.3
lesuene
dilcor/ttalan Legal BERT
11
65.1
602
606
903
303
M0S
903
Ircmme
denessttiineeeeeerra T-haee
al1
311
633
71A
690
786
76.
7366
79A
lsuene
mitiiittilliiligaall MisLLM 1384
41
65.4
71.
07.6
76.
76.1
76.1
76.1
62.1
56.9
53.4
784
76.4
78.
71.4
201
tin-mecrtt--ery
Table 26: SJP. The best scores bold.

---

H Histograms
the following, we provide the histograms for the distribution of the sequence length of the input (sentence or entire document) from each dataset. The length is measured by counting the tokens using the tokenizers of the multilingual models, i.c., DistilBERT, MiniLM, mDeBERTa v3, XLM-R basc, XLM-R large. We only display the distribution within the 99th percentile; the rest is grouped together at the end.
Madegram am dataser, tradiaan court deciaions judgment aeciooi
de
addel
Figure 2: Histogram for dataset BCD-J
Helzgra counimity
300
Figure 3: Histogram for dataset BCD-U

---

15togram dataset german, amgumerrrcmmmmmmmmmm mig
3000
dett d
Figure 4: Histogram for dataset GAM
Hatagram detaset g logal cook volume
di Wie sew rut sne
Figure 5: Histogram for dataset GLC-V

---

Histogram dataser wwwckrregeleoooooooooooooo Jegel code chapter
2000
Figure 6: Histogram for dataset GL-C-C
Motogram datasst groek Jaga code subjott
tie sew rut sne Aneducl
Figure 7: Histogram for dataset GLC-S

---

+5tegram for dacaset oM wwg.ert Cp rediition
Figure 8: Histogram for dataset SJP
"istoge am for sussset urline terrs sarvice unlaimes lovel
pud L12-000 aude was se
Figure 9: Histogram for dataset OTS-UL

---

Histogram for dataset on iic.itcemm... www.icccclaceeeeoe www.sccccooooo topecs
mereetiee dett
Figure 10: Histogram for dataset OTS-CT
Mistog am dataser covid1o emergensy evant
Figure 11: Histogram for dataset C19

---

Hottoy am dacaee. nuti_curooJvve..
Viat
Figure 12: Histogram for dataset MEU-1
Histogram for datanet grack
the sew rutl dimms sne Aneduct UnrrStatiiu
Figure 13: Histogram for dataset GLN

---

Histogram for dataset kealinrro
Figure 14: Histogram for dataset LNR
Hotogram eataset
Figure 15: Histogram for dataset LNB

---

Mitoogram for dbtasct mapa, co0rse
meseerirre dett d
Figure 16: Histogram for dataset MAP-C
Motogram dalaset mapa. Fine
Figure 17: Histogram for dataset MAP-F


Reviews:
[{"review_id": "mNRl3T3HHE", "reviewer": "Reviewer_wXJc", "summary": "", "questions": "The paper uses macro F1 for comparison - could micro F1 or label-weighted scores alter the relative model rankings? Did you experiment with other metrics?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank wXJc for their review. We are glad to hear that they acknowledge the gap we highlighted by showing moderate performance of baseline models and underperformance of ChatGPT and that they appreciate LEXTREME's usefulness to track multilingual progress in NLP.\n\nDefining a standardized benchmark is incremental: We want to stress that defining a standardized benchmark is not incremental, but a defining contribution in the field, pushing research forward for the following reasons:\nMuch like a literature review, equally highly regarded and often highly cited, we also followed a systematic process (described in detail in Section 3.1) analyzing and tagging 108 papers, and finally selecting 11 datasets using clearly defined inclusion and exclusion criteria. \nStandardized benchmarks that also present an aggregation of datasets without providing new methods, such as GLUE (presented at ICLR), SuperGLUE (presented at NeurIPS), and LexGLUE (presented at ACL), are widely used across the field and have been instrumental in shaping NLP progress.\nAs evidenced by the acceleration of saturation in standardized NLP benchmarks (https://contextual.ai/plotting-progress-in-ai/), we think that multilinguality and domain-specific subfields of NLP are getting ever more attention since they offer novel challenges for current methods. Our results on LEXTREME show that multiligual legal NLP is still challenging for current models. \n\nNo new methods defined: Since we present a standardized benchmark, by itself a large contribution requiring a lengthy description, we did not present new models, since we think this to be out of scope. The goal of our experimental results is only to provide a reference point. Providing this reference point already required significant effort to provide a fair and rigorous evaluation of the current state of the art (689 GPU days spent in total resulting in over 5 months of experiments on our hardware). As indicated by our submission to the Resource and Evaluation track, the focus of our paper lies in the dataset survey, selection, description, and standardization for easy use and fast adoption by the community. LEXTREME is already getting attention from the community for multilingual evaluation in the legal domain, and has also been integrated into HELM.\n\nNo novelty in evaluation: Meaningful and fair evaluation and aggregation over multiple dataset configurations and languages is challenging, especially with highly imbalanced label distributions as in LEXTREME. Therefore, we put considerable thought into our evaluation method and present a novel method for fair evaluation in such a scenario. As described in more detail in Section 5, we use macro-averaged F1 score (giving each class equal weight) and aggregate it hierarchically using the harmonic mean, thus promoting models that report good performance robustly across configurations and languages, a property highly desirable in multilinguality in our opinion.\n\nCould other metrics alter model rankings?: Yes, different metrics could alter model rankings, but we specifically focused on the macro averaged score to promote fairness across languages and jurisdictions (as described in detail in Section 5). Including multiple metrics can be misleading and confusing, especially in a benchmark like LEXTREME that includes many configurations and languages. However, we have already computed other metrics, such as micro-F1 and MCC scores to better understand results and we are happy to add them to the appendix for completeness.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Below are the main strengths of the paper:\n\n1. The benchmark would be useful to the community to track multi-lingual progress in the NLP domain.\n\n2. The paper highlights moderate performances of the baseline models and the underperformance of ChatGPT highlighting gaps to be filled in this domain. ", "reasons_to_reject": "The main weakness of the paper is:\n\n1. The work does not provide any model, data, or evaluation novelty and is just a collection of other datasets and models that are already used by everyone in the community. While standardizing the benchmark is a good contribution, the work is just incremental in that all the datasets are already available and people can already evaluate models on these datasets.\n\n\n", "questions_to_authors": "The paper uses macro F1 for comparison - could micro F1 or label-weighted scores alter the relative model rankings? Did you experiment with other metrics?", "paper_topic_and_main_contributions": "This paper introduces LEXTREME, a new multilingual benchmark for evaluating natural language processing (NLP) models on legal and legislative text data. The paper chooses popular models and fine-tune them on the the final benchmark and showcase moderate performance indicating more work needs to be done to improve performance on this benchmark. The key contributions are:\n\n1. The paper compiles 11 existing legal NLP datasets covering 24 languages from 8 language families into a unified benchmark.\n\n\n2. The work provides baseline results using 5 popular multilingual encoder models (DistilBERT, MiniLM, mDeBERTa, XLM-R base/large) as well as monolingual models.\n\nOverall, this paper makes a valuable contribution to introducing a standardized multilingual benchmark for the legal domain, which has traditionally relied more on monolingual resources. The benchmarks and baselines could aid the future development and evaluation of multilingual legal NLP models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PEGI6d7JzV", "reviewer": "Reviewer_RRMe", "summary": "", "questions": "A. The results (discussion) section is a bit underwhelming. While interesting observations are made there are little to no possible explanations for the results. \n\nB. Do you have any insights on why native monolingual models are so strong compared to native legal models? Why does it seem that (pre)training on legal data leads to worse models? Is there a significant difference based on task type (i.e NER vs SLTC/MLTC)? \n\nC. How does the jurisdiction influence the results? I think an analysis performed also on jurisdiction is almost mandatory, as different countries (and therefore languages) have widely different jurisdiction. As your work shows, at the current time, the best solution for a particular language is a native (legal or general) model. Maybe a multimodal model trained in the same jurisdiction (e.g. French law system) could better understand and reason over legal texts. \n\nD. I find the last remark in the limitation section very important to the practical value of the benchmark. Already having access to the result of the case and the judge’s reasoning kind of beats the purpose. Furthermore the documents provided by the judge can (and most probably do) contain elements that explain and emphasise the final decision. Finally, in most cases, the documents from the judge are significantly smaller in size (number of tokens) than the documents brought forward by the involved parties. With all these simplifications, the task is still hard for current models.\n\nE. What is your intuition, how does a “better” model look? More data? More quality data? Larger models?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank RRMe for their review. We are glad to hear they appreciate the living benchmark, the public leaderboard, full access to the code for experiments and that we bring together most open datasets in the legal domain in a single accessible space. We appreciate that they acknowledge our new evidence for a lack of good multilingual legal model and the difficulty of the presented tasks as evidenced by the large potential for improvement.\n\nReason to reject 1: limited novelty: \nPlease refer to our response to reviewer wXJc (Defining a standardized benchmark is incremental, No new methods defined, No novelty in evaluation).\n\nReason to reject 2: long documents\nAs we mentioned in Section 4.3 (lines 461-465), to the best of our knowledge, models capable of handling longer context out of the box are not available multilingually and predominantly trained on English data only. This seems to be the case for SLED also: all the available versions (https://huggingface.co/tau) are pre-trained only on English documents.\nWe will expand the introduction (lines 60-79) to further stress this characteristic of legal documents and the current lack of multi-lingual models capable of handling long contexts in literature. We will also include a reference to SLED along the Longformer in Section 4.3.\nWe hope that our work will contribute to stimulating the development of multilingual models capable of handling large documents.\n\nAnswers to questions: \nA. Limited discussion\nDue to limited space, we kept the discussion short, and we rather focused on the description of the benchmark and the systematic process we followed to create it. We are happy to use the additional page to expand the discussion, including the following points.\n\nB. Why are native monolingual models better than native legal models? \nThe results are inconclusive, and we do not see any general trend explaining these phenomena. We studied the papers introducing the native monolingual models and the native legal models. For example, the Romanian legal model had more training data than the native model, but it was out of domain (court decisions) whereas our benchmark data is from legislation in Romania and the EU. On the other hand, the Spanish legal model was trained on approx. 9GB of Spanish legal text and outperformed the native Spanish model (trained on 570GB of Spanish data). We do not have any explanation for this result. \nWe will add additional analysis and suggest this to be explored further in future work.\n\nC. How does the jurisdiction influence the results?\nWe respectfully disagree with the reviewer. We agree that the jurisdiction dimension is an interesting aspect, but we believe that such analysis is beyond the scope of this work and consciously decided to leave this for future work.\nWe believe that studying the effect of the jurisdiction should be done in a single language before it is expanded to multiple languages to properly control for the effect of the language. To achieve this, we would need a dataset that measures the same (or a very similar) task in the same language from different jurisdictions. German text from Switzerland, Austria and Germany, French text from France and Canada or English text from the UK and the US would be potential candidates. Unfortunately, we did not find suitable datasets in the literature. Scaling this experiment up to multilingual datasets and models complicates it significantly. \nWe agree that this would be an exciting direction for future research. In particular, it would be interesting to understand whether the jurisdiction information is predominantly learned as part of the LLM or it is instead learned during fine-tuning. Possibly, this could be analyzed by comparing the embedded representation of some keywords inside the LLMs in the different training setting.\nWe believe that this research direction would require an entire paper of its own focused on these experiments, and therefore should be considered as a future research direction.\nWe will expand our discussion and future works sections to include these considerations.\n\nD. for outcome prediction tasks, documents are written by the court: \nWe agree with the reviewer that the judgment prediction task in its classic form should be based on complaints by the parties and not on the facts or even reasoning written by the court. Due to very limited access to the complaint documents, especially multilingually, creating such datasets is extremely challenging. Thus, most recent work used text from court decisions as proxies. However, predicting the judgment outcome based on text written by the court itself can still be a hard task (as evidenced by results on these datasets). Moreover, it may still require legal reasoning capabilities from models because of the need to pick out the correct information. Additionally, we believe that these tasks can also be interesting to conduct post hoc analyses of decisions (e.g., would a model learned over decisions from multiple judges have come to the same conclusion as this specific judge?). \nWe want to stress that we based our work on existing literature. Unfortunately, to the best of our knowledge, the number of datasets that also include the material provided by the parties is extremely scarce, especially for languages other than English. Nonetheless, since we designed LEXTREME as a living benchmark, it will be possible to include additional dataset of this type if more will be published in the future. \nAs the reviewer noted, we already touched on this limitation. We will use the additional page provided to further expand the discussion and limitation sections with these observations and the comments provided by the reviewer.\n\nE. What is a “better” model?\nUsually, scaling up model size and quality and quality of pretraining data leads to better performance across many benchmarks as demonstrated by mT5 for example (Xue et al., 2021). We see this trend confirmed in LEXTREME (see Figure 1). Surprisingly, monolingual models perform very well on our benchmark (see Table 6). \n\nWe are deeply grateful for their detailed list of typos and improvements for the bibliography. We will incorporate these changes for the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This work brings together, in a single accessible space most of the open datasets in the legal domain. This leads to a new multilingual and multi-task benchmark for the legal domain. Furthermore, this is a living benchmark, with a public leaderboard and full access to code to run experiments. \n\n- As for the results, LEXTREME proves that, at least for the moment, a good multilingual legal model does not exist, as monolingual general and legal models perform better. There seems to be a good margin for improvement on most tasks, as they are pretty challenging.\n", "reasons_to_reject": "- There is nothing actually novel from a research perspective: all datasets are already public, there is no new model or architecture presented. While the effort is definitely present and the contributions are nice to have and definitely useful for the legal domain, I don’t see any novelty worthy of this conference.\n\n- As legal documents are usually long, as the authors also mention, I would have liked to see a more detailed discussion on how recent works on long context can improve the results - e.g. SLED [1] and others that can be applied to any transformer encoder.\n\n[1] - Ivgi, M., Shaham, U., & Berant, J. (2023). Efficient long-text understanding with short-text models. Transactions of the Association for Computational Linguistics, 11, 284-299. ", "questions_to_authors": "A. The results (discussion) section is a bit underwhelming. While interesting observations are made there are little to no possible explanations for the results. \n\nB. Do you have any insights on why native monolingual models are so strong compared to native legal models? Why does it seem that (pre)training on legal data leads to worse models? Is there a significant difference based on task type (i.e NER vs SLTC/MLTC)? \n\nC. How does the jurisdiction influence the results? I think an analysis performed also on jurisdiction is almost mandatory, as different countries (and therefore languages) have widely different jurisdiction. As your work shows, at the current time, the best solution for a particular language is a native (legal or general) model. Maybe a multimodal model trained in the same jurisdiction (e.g. French law system) could better understand and reason over legal texts. \n\nD. I find the last remark in the limitation section very important to the practical value of the benchmark. Already having access to the result of the case and the judge’s reasoning kind of beats the purpose. Furthermore the documents provided by the judge can (and most probably do) contain elements that explain and emphasise the final decision. Finally, in most cases, the documents from the judge are significantly smaller in size (number of tokens) than the documents brought forward by the involved parties. With all these simplifications, the task is still hard for current models.\n\nE. What is your intuition, how does a “better” model look? More data? More quality data? Larger models?\n", "paper_topic_and_main_contributions": "The paper presents a collection of (already) public legal text datasets, in order to build an open multilingual and multi-task benchmark, LEXTREME. The main contributions of this work are bringing together a diverse set of datasets from multiple sources into a single place, providing an integration with the popular huggingface. Finally, the authors evaluate a plethora of models on the entire collection of tasks and datasets. ", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "L437: we provide detailed using -> we provide details using\nL1441: replaced by agenerator -> replaced by a generator \n\nReference at lines 782-784 is missing details.\nReferences at lines 785-792 and 793-798 are the same paper(?).\nReferences at lines 799-805 and 806-810 are the same paper(?).\nReferences at lines 821-828 and 829-835 are the same paper(?).\nReference at lines 875-876 might be better suited as a footnote(?).\nReferences at lines 877-885 and 886-891 are the same paper(?).\nReferences at lines 965-969 and 970-974 are the same paper(?).\nReferences at lines 975-978 and 979-982 are the same paper(?).\nReference at lines 1021-1023 is missing details.\nReference at lines 1061-1063 is missing details.\nReferences at lines 1089-1096 and 1097-1104 are the same paper.\nReferences at lines 1116-1124 and 1125-1133 are the same paper.\nReference at lines 1160-1163 is missing details.\nReferences at lines 1176-1178 and 1208-1210 are the same paper.\nReferences at lines 1269-1274 and 1275-1281 are the same paper.\nReferences at lines 1324-1331 and 1332-1339 are the same paper.\n"}}, {"review_id": "NGolfoPGhX", "reviewer": "Reviewer_CdWt", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank CdWt for their review. We appreciate that they are convinced that LEXTREME will help the legal NLP community significantly and that the paper is well-written and comprehensive.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and very much comprehensive. This paper has a contribution which will help the legal NLP community significantly. ", "reasons_to_reject": "N/A", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, authors introduce a new benchmark for legal NLP namely LEXTREME. It contains 11 datasets across 24 languages. To compare state-of-the-art models authors propose two different aggregate scores i.e., dataset aggregate score and language aggregate score. Authors also release LEXTREME on huggingface along with a public leaderboard and  necessary codes. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces LEXTREME, a new multilingual benchmark for evaluating natural language processing (NLP) models on legal and legislative text data. The paper chooses popular models and fine-tune them on the the final benchmark and showcase moderate performance indicating more work needs to be done to improve performance on this benchmark. The key contributions are:

The paper compiles 11 existing legal NLP datasets covering 24 languages from 8 language families into a unified benchmark.
The work provides baseline results using 5 popular multilingual encoder models (DistilBERT, MiniLM, mDeBERTa, XLM-R base/large) as well as monolingual models.
Overall, this paper makes a valuable contribution to introducing a standardized multilingual benchmark for the legal domain, which has traditionally relied more on monolingual resources. The benchmarks and baselines could aid the future development and evaluation of multilingual legal NLP models.

Reasons To Accept:
- The benchmark would be useful to the community to track multi-lingual progress in the NLP domain.
- The paper highlights moderate performances of the baseline models and the underperformance of ChatGPT highlighting gaps to be filled in this domain.
- This work brings together, in a single accessible space most of the open datasets in the legal domain. This leads to a new multilingual and multi-task benchmark for the legal domain. Furthermore, this is a living benchmark, with a public leaderboard and full access to code to run experiments.


Reasons To Reject:
- The work does not provide any model, data, or evaluation novelty and is just a collection of other datasets and models that are already used by everyone in the community. 
- While standardizing the benchmark is a good contribution, the work is just incremental in that all the datasets are already available and people can already evaluate models on these datasets.
Decision: Accept-Findings