EUROPA: A Legal Multilingual Keyphrase Generation Dataset (2024)

Abstract: Keyphrase generation has primarily been explored within the context of academic research articles, with a particular focus on scientific domains and the English language. In this work, we present EUROPA, a dataset for multilingual keyphrase generation in the legal domain. It is derived from legal judgments from the Court of Justice of the European Union (EU), and contains instances in all 24 EU official languages. We run multilingual models on our corpus and analyze the results, showing room for improvement on a domain-specific multilingual corpus such as the one we present.

heading: 1 Introduction
text: Keyphrases are short phrases that describe a text, and have been shown to be useful for many applications, from document indexing (Medelyan and Witten, 2006) to opinion mining (Berend, 2011). While heavily researched for the STEM (science, technology, engineering, and mathematics) domains, there has been little investigation on its application to law. This is surprising, as keywords can reduce the workload of legal experts by allowing them to get the gist of lengthy documents (Mandal et al., 2017; Sakiyama et al., 2023; Cérat et al., 2023).
The scope of this work is to assess to what extent keyphrases can be automatically generated in the legal domain. Our contributions are the following: (1) We collected and curated EUROPA, the first open benchmark for legal KPG (keyphrase generation), spanning 24 languages and extracted from real-world European judgments.1 (2) We provide in-depth analysis of our corpus, highlighting its differences compared to other existing corpora. (3) We report performances achieved by multilingual generative models on this benchmark and
1The dataset can be downloaded from the Hugging Face hub at https://huggingface.co/datasets/NCube/ europa while evaluation scripts and model outputs are available at https://github.com/rali-udem/europa.
point out areas where performances could be improved.

heading: 2 Related Work
text: While English KPG has been studied fairly extensively in technical and academic domains, our work bridges two less common fields : legal KPG and multilingual KPG. In this section, we first review the literature surrounding keyphrase extraction (KPE) and generation before covering legal KPE and KPG. For a more complete survey on KPG, we refer to Xie et al. (2023).

heading: 2.1 Keyphrase Extraction and Generation
text: Keyphrases may be either present in the input (could be retrieved with purely extractive methods) or absent from it (needing abstractive methods to generate them).
Many extractive methods have been suggested and used over the years, either using unsupervised heuristic rules and pre-trained machine learning (ML) models to detect and extract keyphrases (Liu et al., 2010; Gollapalli and Caragea, 2014; Meng et al., 2017; Yu and Ng, 2018) or using ML/deep learning (DL) classifiers trained on labelled data to learn to identify them (Yang et al., 2018; Chen et al., 2018; Wang et al., 2018; Basaldella et al., 2018; Alzaidy et al., 2019; Sun et al., 2019; Mu et al., 2020; Sahrawat et al., 2020). Extractive methods are however incomplete as, for most domains, a large portion of the target keyphrases are absent from the document, hence an increased interest in generative models in more recent works.
While research on KPG initially used seq-to-seq models in the form of RNNs (Meng et al., 2017, 2019, 2021; Yuan et al., 2020a), modern KPG uses mostly fine-tuning of pre-trained transformer models (Vaswani et al., 2017), such as BART (Lewis et al., 2020), which showed itself to be as efficient as previous, more complex RNN models (Chowdhury et al., 2022). Current English state-of-the-art
12718
results are currently obtained by using BART models which are further pre-trained for scientific KPG (KeyBART by Kulkarni et al. (2022) and SciBART by Wu et al. (2022)). KPG has been conducted on limited domains, such as academic and STEM papers (Hulth, 2003a; Nguyen and Kan, 2007; Kim et al., 2010; Meng et al., 2017; Krapivin et al., 2009) or news (Gallina et al., 2019; Koto et al., 2022), although the need for KPG extends beyond those domains.
Of particular interest is KPG for long documents, which has benefited from a few studies, especially in recent years. Ahmad et al. (2021) proposed a two-step approach of this problem, where they first select salient sentences before using those to generate keyphrases, while Garg et al. (2022) tested multiple ways of including information from the main body (generated summary, citation sentences, random sentences, etc), showing that adding a generated summary was the most efficient strategy. Mahata et al. (2022) proposed a corpus for scientific KPG using the whole document instead of only the title and abstract, which is what is currently used in KPG. They do not however run any baseline on that corpus.

heading: 2.2 Keyphrase Generation in the Context of Legal Domain
text: Legal documents are more complex and longer than those commonly used in NLP, as illustrated by the European Court of Human Rights corpus (Chalkidis et al., 2019). This motivates a high demand from legal professionals for automatic digests of such documents (Bendahman et al., 2023), mostly from the angle of legal summarization. This task, more widespread than legal KPG, was first formalized as an extractive task in which the most salient segments from the document are returned as a summary (Farzindar and Lapalme, 2004a,b; Saravanan and Ravindran, 2010; Polsley et al., 2016; Aumiller et al., 2022). Similarly to KPG tasks, abstractive summarization models based on transformer architecture (Vaswani et al., 2017) became more widespread than extractive ones. However, open legal summarization benchmarks such as BigPatent (Sharma et al., 2019), Multi-LexSum (Shen et al., 2022), and EUR-Lex-Sum (Aumiller et al., 2022), are still scarce.
Legal KPG may be considered as a specialized type of summarization but, to the best of our knowledge, no open benchmark is available for such a
task. Moreover, most existing works have been focusing on keyphrase extraction (Le et al., 2013; Audich et al., 2016; Mandal et al., 2017; Daojian et al., 2019), ignoring absent keyphrases. The very first legal KPG experiments addressing abstractive keyphrases were conducted by Cérat et al. (2023) and Sakiyama et al. (2023), but within a monolingual setting and with no public dataset release. This is a major issue as generative models are particularly data-greedy. Furthermore, languagewise, most of the open legal datasets are in English (Chalkidis et al., 2023) and, when it comes to existing open multilingual legal benchmarks (Chalkidis et al., 2021; Savelka et al., 2021; Aumiller et al., 2022; Niklaus et al., 2023), none are related to KPG. Overall, our contribution aims at fulfilling these gaps by providing an open multilingual dataset for legal keyphrase generation.

heading: 3 The Creation of the EUROPA Dataset
text: Cases in the Court of Justice of the European Union (CJEU), which aims at ensuring the consistent interpretation and application of the EU law across all EU institutions (Court of Justice of the European Union, 2023), can be processed in any of the 24 EU official languages, depending on the Member State involved, making drafting and translation pivotal and complex tasks. Stakeholders and judges communicate in their respective languages and rely on lawyer-linguists for document exchange. Once a judgment is rendered, it is translated into other EU languages to ensure consistency in the judgment text, keyphrases, and their legal implications (Domingues, 2017). Through private correspondence, the CJEU explained that keyphrases, drafted by the Registry and completed by the reporting judge’s cabinet or the advocate general, aim at providing concise case descriptions. Their meticulous construction establishes them as high-quality gold references for KPG evaluation.

heading: 3.1 Data Collection & Processing
text: For the sake of clarity, we define a judgment as a collection of documents which refer to the same ruling provided in up to 24 languages. Each judgment has a unique CELEX identifier that remains the same for every language version available. Therefore, all language versions with the same CELEX ID are semantically and legally equivalent, and can be considered as parallel. Within a judgment, each language version is named an instance, a pair
comprising the target keyphrases and the input text from the judgment, with both being in the same language. For example, the judgment shown in Table 1 has 22 instances spanning 22 languages.
In May 2023, we performed a first pass on the EUR-Lex database2, the main legal EU online database (Bernet and Berteloot, 2006), scraping 304 426 query results corresponding to 19 319 judgments released by the CJEU. These query results (each being a potential instance for our corpus) consist in small snippets (as the one in Figure 1) containing the multi-line title and meta information of the case including the document identifier, but not the judgment text. Then, we made a second pass in order to scrape the plain text of the judgment using the document identifier collected during the first pass. The plain text is available as PDF and/or HTML, but we used the latter for convenience. A total of approximately two weeks was required for collecting the query results snippets and the judgments’ HTML files in all 24 languages.
2https://eur-lex.europa.eu
An overview of the corpus collection and curation process is presented in the following paragraphs (more details in Appendix E).
The structure of a case HTML file generally consists of a mix of keyphrases and meta-information at the top of the document followed by paragraphs that will be merged into the input text. Separating the top of the document from the paragraphs is crucial in order to ensure that the input text is not contaminated by target keyphrases. However, identifying keyphrases from the judgment plain text was infeasible as they are surrounded by quotation marks, symbols, and HTML tags that vary across languages and time. Therefore, the keyphrases were obtained from the small snippet multi-line title as the one in Figure 1.
Still, in such snippet, the keyphrases are mixed with meta information about the case which we need to get rid of as they are redundant and of low value from a KPG standpoint (e.g. sequences such as “Reference for a preliminary ruling” followed by the referring court and the country are so frequent that they would bring noise during training). Using the BeautifulSoup library3, languagespecific regular expressions and domain-aware engineered heuristics, we filtered out the sequence of keyphrases that had the highest number of meaningful phrases. An additional sanity check was also performed to ensure that keyphrases are properly split (e.g. missing whitespace beside a phrase delimiter) and that the number of phrases remains consistent across languages for each judgment. Our approach thus ensures the quality of the target keyphrases that will be used in the KPG task during training and evaluation. For the sake of evaluation consistency with the existing KPG literature, keyphrases are lowercased and separated by semicolons.
Another critical part of corpus preparation is the input text from the judgment HTML files. These raw documents are delicate to process as they begin with a mix of meta-information (e.g. stakeholders identities) and target keyphrases (which must be excluded from input text), followed by numbered paragraphs. Moreover, since the HTML tags are inconsistent and vary across years and languages, BeautifulSoup cannot be used to extract the input text from the judgment. Therefore, we manually designed several language-specific regular expressions to reliably split the documents by
3https://pypi.org/project/beautifulsoup4
matching delimiters in all languages (e.g. “Judgment”, “Sentencia”, “Urteil”). By doing so, only the paragraphs of the judgment are retained as a single input string, while the section containing superficial meta-information and target keyphrases is taken away, thus preventing any data leakage. This was confirmed by a manual examination of 100 random instances, equally distributed across the 24 languages.
After removing instances with empty input or target texts, our final corpus is composed of 17 833 judgments, in 16 languages on average and spanning cases from 1957 to 2023. This amounts to a total of 284 957 instances (input/keyphrases pairs). Expectedly, these instances are unevenly distributed across all 24 languages, with languages from older EU Member States being more represented in the dataset. For instance, French (the most represented language) amounts to 17 461 instances (6.13% of all instances) while Croatian and Irish (a significant outlier) contain 5153 (1.81%) and 92 (0.03%) instances, respectively.

heading: 3.2 Chronological Split
text: It is a common practice to randomly split data in a NLP task (Gorman and Bedrick, 2019). However, training and evaluating a model on data from overlapping time periods with similar distributions fails to assess the actual model’s temporal generalization (Lazaridou et al., 2021). This is why legal NLP generally uses a chronological split of documents instead of combining random shuffle with random split (Chalkidis et al., 2019; Medvedeva et al., 2021; Chalkidis et al., 2021). In our case, we tried both splits with a mBART50 model. On the test set, the performance in terms of F1@M for present keyphrases differs by 14.1 percentage points (11.1 for absent keyphrases) in favour of random split. Results in Table 2 confirm that random split, by ignoring real-world temporal concept drifts, tends to overestimate true performance. This is consistent with Søgaard et al. (2021); Mu et al. (2023). Therefore, we choose a chronological split
for a proper assessment: The training set covers judgments from 1957 to 2010 (131 076 instances), the validation those from 2011 to 2015 (63 373 instances), and the test set the ones from 2016 to 2023 (90 508 instances).4 Full details about documents distribution across these splits are shown in Appendix F.

heading: 4 Dataset Analysis
text: As shown in Table 3, compared to previous works and KPG benchmarks, EUROPA covers more languages and is highly positioned in matters of number of instances, ratio of absent keyphrases, and average input length.
The distribution of keyphrases in our dataset varies across languages due to the fact that the most recent judgments tend to have a higher number of keyphrases attached to them (the average number of keyphrases per language can be found in Appendix F). Consequently, instances in languages from the most recent Member States tend to be biased towards having more keyphrases.
Another consequence of the temporal evolution of the average number of keyphrases coupled with the chronological split, is that EUROPA’s validation and test sets contain more keyphrases per instance on average (8.3 and 10.5 respectively) compared to its training set (5.4 keyphrases on average). This
4The temporal split is available at https://huggingface. co/datasets/NCube/europa and a random split version for those who wish it can be found at https://huggingface. co/datasets/NCube/europa-random-split
creates a discrepancy between the sets that practitioners should be aware of. However, we advocate that a higher number of keyphrases in the test set of EUROPA will be beneficial to the evaluation of the competing models by providing a larger number of potential gold keyphrases, and by assessing the capacity of models to generalize across time when target keyphrases follow different patterns with respect to the past.

heading: 5 Models
text: As the majority of EUROPA’s target keyphrases are absent from input documents, we focus on generative models, as extractive ones such as YAKE Campos et al. (2020) are ill-suited here. Recent state-of-the-art models in English KPG rely heavily on pretrained models, such as KeyBART (Kulkarni et al., 2022) or SciBART (Beltagy et al., 2019), which are pre-trained and fine-tuned on a massive corpus of English scientific documents. However, such models are not available in a multilingual format, nor are they specific to the legal domain. We therefore choose to use mBART50 (Tang et al., 2020), a variant of mBART (Liu et al., 2020) with support for 50 languages instead of 25. We also test mT5 (Xue et al., 2021), which covers 101 languages.
Most models, including mBART50, have a maximum input sequence length typically set to 1024 tokens. While mT5’s input length can be set arbitrarily, it was originally pretrained with a context of 1024. The main caveat is mT5’s memory complex-
ity that increases quadratically with input length, hence a prohibitive computing cost. While there exists some models whose maximum input length reaches up to 16k tokens, such as Longformer (Beltagy et al., 2020), BigBird (Zaheer et al., 2020) or LongT5 (Guo et al., 2022), these remain computationally expensive to run with our current resources and are only suitable for English data. Therefore, similarly to Cérat et al. (2023), we implemented a mBART variant with LSG attention (Condevaux and Harispe, 2023) such that the maximum input length reaches 8192 tokens instead of 1024. Doing so makes the memory complexity increase linearly with respect to the input length, instead of quadratically as it is the case with a traditional attention mechanism. All models are trained with early stopping and a maximum epoch number of 10, except mBART50-8k and mT5-large with only 5 epochs, as their training is more time-consuming. Details about the hyperparameters and training process are provided in Appendices A and D.

heading: 5.1 Evaluation Protocol
text: As has become common practice in recent works on KPG (Meng et al., 2017; Garg et al., 2023; Shen and Le, 2023; Chen and Iwaihara, 2023), model evaluation relies on two F1 measures: F1@k (k = {5, 10}) and F1@M , calculated separately for present and absent keyphrases. F1@M is computed using the entirety of the generated keyphrases while F1@k is computed using the best k generated keyphrases (by truncating if necessary). In both cases, the number of target keyphrases remain untouched. F1@k is calculated using only the top k best scoring keyphrases among the model’s predictions, hence an upper boundary below 1 when-
ever the number of target keyphrases exceeds k, which occurs in most of EUROPA’s instances. This is why F1@M tends to better reflect the model performance as all candidate keyphrases are taken into account without being truncated. For more details about these metrics, we refer to Yuan et al. (2020b). Following the literature, target and predicted keyphrases are lowercased and stemmed (e.g. Meng et al. (2017) applied Porter Stemmer for English) before conducting an exact match. Stemming is a critical step without which a candidate keyphrase could be errouneously considered wrong because of the morphological nature of the language. That is why we applied stemming for all languages for treating them as fairly as possible. For most of them, the Snowball stemmer (Porter, 2001) was used. In addition to F1 measures, we also computed MAP@50 (Mean Average Precision).

heading: 6 Results
text: F1@5, F1@10 and F1@M scores for present and absent keyphrases over all languages are reported in Table 4. Since the number of instances varies across languages, we report average scores that are weighted and unweighted. The former is an average score across all instances without taking language into consideration. Consequently, it can be highly influenced by high-resource languages covering more instances. The latter is an unweighted average among languages’ scores. In other words, all languages have equal importance, thus reflecting the ability of a model to perform equally well in high- and low-resource languages.
Unsurprisignly, the YAKE extractive model performs poorly, finding none of the absent keyphrases.
With a fixed input length of 1024 tokens, the three mT5 variants dramatically underperform mBART50. This is surprising as mT5 covers twice as many languages than mBART50. Also, as mT5large has 59% more parameters than mBART50, it would have been expected to outperform the latter, but the results reveal otherwise. When comparing mBART50 with mBART50-8k, the increase in the maximum context length brings significant improvement across all metrics. While an average gain of around 2% (for F1@M over present phrases) may not seem significant, KPG evaluation often greatly underestimates the true performance of the models, due to the difficulty of correctly evaluating whether a keyphrase is correct or not (Wu et al., 2023). As such, the improvement shown by the mBART50-8k model is significant and reflected in the generated keyphrases, thus emphasizing the benefits in enlarging the maximum input length. Still, Table 5 shows that input context
enlargement does not have uniform effects across languages. For instance, for present keyphrases, some languages get small gains (Greek, Swedish), and some degrade (Lithuanian, Latvian). For highresource languages such as English and Italian, performance improves on present phrases, but seems stagnant on absent ones. For low-resource languages, Croatian has the most dramatic improvement on present phrases while Irish gets one score above 0 with mBART50-8k. However, the performance for these languages still lags behind that of moderate-resource ones. This is understandable as these languages have almost no training instances in our temporal split setting, thus revealing the difficulty of conducting KPG for unseen languages.5

heading: 7 Analysis and Discussion
text: The first striking observation is that mBART models, despite covering less languages compared to mT5s, succeed in outperforming the latter models. One explanation is that mT5 small, base and large variants generated on average significantly fewer phrases per instance (2.1, 2.3 and 2.3, respectively) with respect to mBART50 and mBART50-8k (5.5 and 6.0). Consequently, mT5 models are less likely to achieve high scores. The other noticeable result is that the mBART models even succeed in outperforming mT5s in languages that only mT5s support such as Bulgarian or Greek. This suggests that Conneau et al. (2020)’s corpus used for pretraining mBART50 gave it a capacity to deal with much more languages than stated by Tang et al. (2020).
The improvement from mBART50 to mBART50-8k is significant and emphasizes the importance of larger input length. Less that 14% of all instances fit into a 1024-tokens context window. Around 58% do when that length reaches 8k tokens. Building models that can efficiently generate keyphrases from larger documents is therefore crucial in order to achieve further progress (the length reaches 9160 tokens on average, and 17k at the 90th percentile, with mBART50 tokenizer).6
Moreover, KPG models struggle for keyphrases with more tokens (split by whitespaces). In the case of mBART50-8k, a matched target keyphrase contains on average 3.1 tokens, while an unmatched
5With a random split dataset, F1@M achieved by mBART50 for Croation/Irish reaches 46.9/20.5 for present keyphrases and 13.9/10.2 for absent ones
6We tried a sliding-window-based model that was not conclusive, thus the need to find better ways to capture context.
target phrase contains 5.7 tokens. Unmatching generated keyphrases contain 7.7 tokens on average. Upon manual inspection of generated keyphrases, most models succeed in generating simple phrases made of up to 3 terms, but they indeed struggle for longer noun phrases that refer to very specific or technical concepts such as “Market for antidepressant medicines (citalopram)” in Table 6. Also, although some candidate keyphrases are still relevant from a reader’s standpoint, they are penalized by the exact matching evaluation approach, although stemming is applied. For instance, the output “Concept of ‘restrictions of competition by object’" could be a decent generation for the target “Restriction by object” despite the lack of matching. In order to mitigate this issue, we evaluated the
models again with a semantic matching metric (Wu et al., 2023) allowing us to compare predictions and targets without having to apply any sort of post-processing or stemming (the tool is however only available for English). Results in Table 7 have a correlation of 0.99 with F1@M scores obtained for English for both present and absent keyphrases. This seems consistent with the ranking among models observed previously in Table 4: mBART50based models outperform mT5-based ones, and mBART50-8k is the front-runner.

heading: 8 Conclusion
text: In this paper, we present EUROPA, a novel and open multilingual keyphrase generation dataset in the le-
gal domain. We believe this dataset can help alleviate two current shortcomings of the keyphrase generation task: the lack of data in domains other than STEM; and the lack of multilingual non-English datasets. Our dataset is available at https:// huggingface.co/datasets/NCube/europa. We furthermore provide an analysis of EUROPA with key statistics, thus giving insights on the particularities of our dataset. Finally, we run multiple models in various settings in order to give an initial point of comparison for future works on EUROPA. Our corpus also highlights the need to efficiently capture larger input context, and will be a suitable testbed for models designed to do so.
Limitations
Low Resource Languages: The choice of a chronological split, though enabling a realistic KPG performance assessment with quasi-equal amount of test instances for each language (ex-
cept Irish), results in dramatic differences in available training data across languages. This is particularly true for the latest official EU languages, such as Croatian and Irish. One possible approach for mitigating this issue in future works would be to complement the training data with other legal documents released by other EU institutions. A random split version of our dataset is available for those who would like an identical distribution of languages across training, validation and test sets (https://huggingface.co/datasets/ NCube/europa-random-split).
Computing Cost Scalability: The architecture of most of the models used here is derived from experiments in which the text input length rarely exceeds 1024. This is a major caveat when deploying such models on real-world data with significantly longer documents, such as the legal documents in our dataset which often exceed several thousand tokens. Moreover, dealing with documents whose language is not supported by the tokenizer mechanically increases the number of tokens. This is due to the fact that running and training transformer models with higher maximum input sequences is costly both in terms of time and GPU memory. The computing burden was already emphasized by Aumiller et al. (2022) of legal summarization for EU documents and Sakiyama et al. (2023) for legal KPG in Portuguese.
Ethics Statement
Copyright: The Publications Office of the European Union gave us the written confirmation that cases of the Court of Justice of the European Union could be used for commercial and non-commercial purposes, as stated in the European Commission decision released on 12 December 2011 (European Commission, 2011). Our models are implemented with Wolf et al. (2020)’s transformers library licensed under Apache 2.0 while evaluation is performed with Meng et al. (2017)’s tools available under the MIT license. Therefore, one is granted permission to modify and distribute the licensed material.
Personal Data Protection: According to the CJEU personal data protection policy, the Court anonymizes personal information upon request of a party, referring court/tribunal or upon its own volition. In that case, anonymity is granted throughout the entire procedure according to article 95 of Rules of Procedure of the Court of Justice. It must be em-
phasized that anonymity must be requested at the earliest stage of the proceedings. Once the decision is drafted and released, there is no a posteriori optout option due to the transparency principle with which the CJEU must comply. Moreover, we have a written confirmation from the Publications Office of the EU that we are allowed to share and redistribute our corpus made from documents publicly available on EU platforms.

heading: Acknowledgements
text: We would like to thank the European Union for granting us access to EUR-Lex data (http:// eur-lex.europa.eu, © European Union, 1998- 2023). We are also grateful to Benjamin Cérat and Di Wu for their advice on some coding implementations.

heading: A Hyperparameters
text: While the batch size was set at 16 (with the help of gradient batch accumulation when needed), a grid search was performed for the learning rate with the following values: 1 × 10−6, 5 × 10−6, 1×10−5, 5×10−5, 1×10−4. For both mBART50 models, we settled on a learning rate of 1× 10−5 which we coupled with the AdamW (Loshchilov and Hutter, 2018) optimizer of Pytorch. The same applies to mT5 variants but with a learning rate of 1× 10−4. All other optimizer hyperparameters are left at default values. Once the learning rate was determined, we also added 2500 warming steps and a scheduler with a linear decay. The objective function was cross-entropy loss. For mBART models, we applied mixed precision (FP16) while brainfloat16 (BF16) was used for mT5 models. The maximum number of epochs was set to 10 and the model achieving the lowest loss on the validation set is used for the test set. Patience was set to 5 epochs. In the case of large models (mBART-8k, mT5-large), the maximum number of epochs was
set to 5 due to much longer training epochs. For generation, we used a beam search of 10 and return the best sequence, with a maximum output length of 256 tokens. The hardware consisted in a RTX4090 and a couple of A5000 graphical cards, each with approximately 24 GB of VRAM. Inference only required a single card for all models, including those with the highest number of parameters. Overall, the training for either mBART-8k or mT5-large was performed on 2 GPUs for up to 62 hours overall. For other models, the training lasted for up to 28h on a single GPU (more details in Table 10). The inference on the test set took up to 34 hours, depending on the model. Given the delay in obtaining keyphrases during inference, the results are based on a single run.

heading: B Languages Support per Model
text: mBART50 by Tang et al. (2020) and mT5 by Xue et al. (2021), though presented as multilingual models, do not cover the same languages and both support only partially the official EU languages, as shown in Table 8. Although mT5 (its tokenizer remains the same for all variant sizes) supports twice as many languages than mBART, the vocabulary of both tokenizers is roughly of the same size: 250 112 and 250 054 tokens, respectively. Such large vocabulary size is a computing burden during inference time, especially during decoding.
By design, mBART50 tokenizer adds to input and output sequences a token specifying source and target languages (e.g. [en_XX] for English). As not all EU languages are not covered by mBART50, we manually added to the tokenizer special tokens for all unsupported languages (e.g. [bg_BG] for Bulgarian). Following mBART50 tokenization process, a language token is added in both input and output sequences. Unlike translation tasks, our multilingual KPG task is such that, for each instance, the input document and the target keyphrases are both in the same language. Therefore, the language token remains the same in the input and output within the same instance.
When it comes to mT5, such languages prefixes are not required. However, applying mBART50 language tokens to mT5-small brought around 1 percentage point improvement across F1@k metrics, thus motivating us to deploy mBART50 language prefixes to the rest of mT5 models.

heading: C Number of Parameters per Model

heading: D Training and Inference Times
text: Model Num. of training epochs Total training time Total inference time

heading: E Details About the Data Preprocessing

heading: F Additional Statistics

heading: G Scores per Language for each Model

heading: H Random Split
text: As discussed previously, we perform a temporal split on the data to better reflect the reality of temporal concept shifting as well as new languages being introduced in the EU. However, a random split might be better in some cases, such as if we train a model for short-time use, and are willing to recollect data regularly to insure performance remains as predicted. In short, a random split can only procure guarantee about its performance as long as the real world data remains similar to the training data, while a temporal split give a view of the performance while accounting for changes in the real world data.
Still, to better allow practitioners to use data as they desire, we released a random split (https: //huggingface.co/datasets/NCube/europa-random-split). The training, validation and test sets have respectively 10 003, 3 391 and 4 439 judgments. These figures are the same as in the chronologically split dataset. However, the number of instances per set in the random split setting differs since each set has the same distribution in terms of languages. Consequently, each set contains 159 306, 53 943 and 71 708 instances respectively. The results achieved by mBART50 are shown in Table 15. As expected, results are overall much higher since all training, validation and testing sets have the same distribution in terms of languages and vocabulary. We want to emphasize once again however that due to the fast changing nature of legal NLP, these performances are only valid for a short time after the data collection.

heading: Future Work
text: Finally, we run multiple models in various settings in order to give an initial point of comparison for future works on EUROPA. Our corpus also highlights the need to efficiently capture larger input context, and will be a suitable testbed for models designed to do so. Limitations
Low Resource Languages: The choice of a chronological split, though enabling a realistic KPG performance assessment with quasi-equal amount of test instances for each language (ex-
cept Irish), results in dramatic differences in available training data across languages. This is particularly true for the latest official EU languages, such as Croatian and Irish. One possible approach for mitigating this issue in future works would be to complement the training data with other legal documents released by other EU institutions. A random split version of our dataset is available for those who would like an identical distribution of languages across training, validation and test sets (https://huggingface. co/datasets/ NCube/europa-random-split). Computing Cost Scalability: The architecture of most of the models used here is derived from experiments in which the text input length rarely exceeds 1024. This is a major caveat when deploying such models on real-world data with significantly longer documents, such as the legal documents in our dataset which often exceed several thousand tokens. Moreover, dealing with documents whose language is not supported by the tokenizer mechanically increases the number of tokens. This is due to the fact that running and training transformer models with higher maximum input sequences is costly both in terms of time and GPU memory. The computing burden was already emphasized by Aumiller et al. (2022) of legal summarization for EU documents and Sakiyama et al. (2023) for legal KPG in Portuguese