Title is (Not) All You Need for EuroVoc Multi-Label Classification of European Laws
Authors: Lorenzo Bocchi, Alessio Palmero Aprosio (Università di Trento, 2024)
Topic: In-domain IR and IE

Abstract
Machine Learning and Artificial Intelligence approaches within Public Administration (PA) have grown significantly in recent years. Specifically, new guidelines from various governments recommend employing the EuroVoc thesaurus for the classification of documents issued by the PA. In this paper, we explore some methods to perform document classification in the legal domain, in order to mitigate the length limitation for input texts in BERT models. We first collect data from the European Union, already tagged with the aforementioned taxonomy. Then we reorder the sentences included in the text, with the aim of bringing the most informative part of the document in the first part of the text. Results show that the title and the context are both important, although the order of the text may not. Finally, we release on GitHub both the dataset and the source code used for the experiments.

Introduction
The presence of Machine Learning and Artificial Intelligence techniques has become almost ubiquitous in many fields, from hobbyist projects to industrial and government usage. Also inside the Italian Public Administration, there have been efforts to digitize and modernize the processes for more than a decade. In particular, some documents released by the Italian PA suggest the use of EuroVoc,} a multilingual thesaurus developed and maintained by the Publications Office of the European Union (EU), that covers a wide range of subjects (law, economics, environment, ...) organized hierarchically. Outside Italy, Portuguese AUTHOR and Croatian AUTHOR communities are making efforts to automatically perform tagging of official regulations using EuroVoc. In addition to that, in 2010 the EU organized in Luxembourg the Eurovoc Conference,} in order to facilitate the comprehension and use of the taxonomy. The classification of a document with respect to the EuroVoc taxonomy has previously been addressed by several studies (see Section~sec:related), since at present the classification of the documentation in the PA is carried out manually, a task that can be very expensive in the long run. In this context, we concentrate our work on automatically assigning EuroVoc labels to a document, starting from the existing approaches in document and text classification, that use pretrained large language models followed by a fine-tuning phase on a specific task. Unfortunately, these families of language models have an intrinsic limit regarding the maximum number of words present in a text (usually 512). In the case of documents that can be quite large, like legal ones, it is important to try and make sure that the key information about a text is included in the chosen set of words. The previous research deals with this limit by concatenating the title with the raw text, and then clipping it to the limit. In some countries (such as Italy, see AUTHOR) the title is usually very well formulated and it is very important to correctly classify a document. On the contrary, the text of a law is usually very redundant, and the most representative text is often after a notable sequence of preambles. Given these premises, we investigate how the previous approaches work on European laws and apply different strategies to create a summarized version of a text by reordering the sentences. The results show that in this specific case, both the title and the context are important, and that the best approach in regulations enacted by the European Parliament is to fill the 512-words limit with as much information as possible. The paper is structured as follows: Section~sec:related will expose the related work; Section~sec:dataset describes the data; the approach and the experiments are described in Section~sec:experiments; the results are then discussed in Section~sec:discussion. Finally, both the software and the dataset are available for download, as described in Section~sec:release.

Conclusion
In this paper, we presented some approaches to perform document classification on long documents, by reordering their sentences before the fine-tuning phase. The best results are obtained when all the 512 tokens allowed in the BERT paradigm are filled, possibly including the title of the law. In the future, we want to extend this approach to other languages, trying to understand whether the same reordering algorithm leads to some improvement in the classification task. We will also investigate other summarization approaches, or new architectures that rely on Local, Sparse, and Global attention AUTHOR so that longer texts (up to 16K tokens) can be used to train the model.