PoliToHFI at SemEval-2023 Task 6: Leveraging Entity-Aware and Hierarchical Transformers For Legal Entity Recognition and Court Judgment Prediction
Year: 2023

Abstract:
The use of Natural Language Processing techniques in the legal domain has become established for supporting attorneys and domain experts in content retrieval and decision-making. However, understanding the legal text poses relevant challenges in the recognition of domainspecific entities and the adaptation and explanation of predictive models. This paper addresses the Legal Entity Name Recognition (L-NER) and Court judgment Prediction (CPJ) and Explanation (CJPE) tasks. The L-NER solution explores the use of various transformer-based models, including an entity-aware method attending domain-specific entities. The CJPE proposed method relies on hierarchical BERTbased classifiers combined with local input attribution explainers. We propose a broad comparison of eXplainable AI methodologies along with a novel approach based on NER. For the LNER task, the experimental results remark on the importance of domain-specific pre-training. For CJP our lightweight solution shows performance in line with existing approaches, and our NER-boosted explanations show promising CJPE results in terms of the conciseness of the prediction explanations.

heading: 1 Introduction
text: AI-powered tools for legal AI can analyze vast amounts of documents providing lawyers and judges with relevant insights on case laws (Sansone and Sperlí, 2022). For example, based on the analysis of past court cases legal AI models can predict the outcome of similar cases, thus offering efficient and effective ways of resolving disputes (Medvedeva et al., 2022).
The legal domain poses relevant challenges due to the specificity of the legal language, the variety of scenarios and application contexts, and the temporal evolution of norms and regulations (Chalkidis and Søgaard, 2022; Benedetto et al., 2022).
This paper addresses two notable legal AI tasks,
i.e., Legal Named Entity Recognition and Court Judgment Prediction with Explanation.
Legal Named Entity Recognition The L-NER task aims at annotating portions of legal content with domain-specific entities. Annotations are particularly helpful to support content retrieval and indexing. For example, the identification of relevant legal provisions can help lawyers and judges make informed decisions based on previous cases. Despite NER being a well-known NLP task, the application of existing techniques to legal data is challenged by the inherent complexity and nuances of the language used in the legal domain (Williams, 2005; Tiersma, 2000). We address L-NER by exploring the use of established transformer-based models, established for many NLP tasks (Vaswani et al., 2017) and tailored to the legal domain (Chalkidis et al., 2020). Unlike previous studies, we also explore the use of the entity-aware attention mechanism (Yamada et al., 2020), which allows the transformer to also attend to domain-specific entities. The preliminary results achieved on the development set confirm the potential benefits of leveraging the entity-aware attention mechanism in L-NER.
Court Judgment Prediction with Explanation CJPE focuses on predicting the outcome of a given case. Unlike the traditional Legal judgment Prediction (Cui et al., 2022), it entails predicting the outcome along with a textual explanation consisting of an extract of the case content. The presented approach to CJPE relies on hierarchical transformerbased models fine-tuned on annotated legal judgments. On top of the classification model, a posthoc feature attribution method is exploited to derive sentence-level importance for judgment prediction. Beyond testing multiple models and fine-tuning datasets, we empirically analyze two complementary aspects, i.e., the role of sentence-level tokenization and the use of single- or mixed-type train-
1401
ing data and models. The experiments show that single-type models on average perform best due to the high specificity of the legal vocabulary and syntax. Furthermore, sentence-level tokenization allows transformers to effectively handle long documents, enhancing conciseness of the generated explanations.
This paper is organized as follows:
• In Section 2 we examine the related works and highlight the difference between the present work and existing approaches.
• In Section 3 we provide an overview of the methodologies employed for the three tasks.
• In Section 4 we describe the experimental setup, and the metrics, and we provide an extensive validation and discussion of the results.
• Section 5 draws the conclusion and discusses our future research lines.

heading: 2 Related Works
text: Transformer-based models have exhibited remarkable performance in many legal AI domains such as legal question answering (Hendrycks et al., 2021) and legal document summarization (Jain et al., 2021). This paper addresses three specific research lines in Legal AI: Named Entity Recognition, Court judgment Prediction, and Predictions Explanation.

heading: 2.1 Legal Named Entity Recognition
text: Prior works on Named Entity Recognition have already explored the use of statistical models (e.g., (McCallum and Li, 2003)) and, more recently, of deep neural networks (Li et al., 2020). Pioneering works on Legal NER focus on named entity recognition and resolution on US case law (Dozier et al., 2010) and on the creation of a German NER dataset with fine-grained semantic classes (Leitner et al., 2020). Since deep learning approaches require large-scale annotated datasets and generalpurpose NER models are trained on a different set of entities, publicly available legal NER data sets have recently been made available (e.g., (Au et al., 2022)). Inspired by the recent advances in NER tasks with span representation (Ouchi et al., 2020), in this paper we explore the use of entity-aware attention mechanism (Yamada et al., 2020) to accomplish the L-NER task.

heading: 2.2 Court Judgment Prediction
text: The problem of predicting court case outcomes has received considerable attention in recent years (Cui et al., 2022). Most research efforts on the jurisprudence of the U.S. Supreme Court (Strickson and De La Iglesia, 2020; Kowsrihawat et al., 2018). Other studies analyze the cases of the European Court of Human Rights, utilizing both traditional and machine learning methods (Aletras et al., 2016; Visentin et al., 2019; Quemy and Wrembel, 2020). Still others are related to the judicial system in India (Shaikh et al., 2020; Malik et al., 2021).
Transformer-based models have recently shown to achieve remarkable results (Chalkidis et al., 2019, 2020; Kaur and Bozic, 2019; Medvedeva et al., 2021). To the best of our knowledge, none of them focus on handling long legal documents using attention-based models such as Longformer (Beltagy et al., 2020) for court judgment prediction or hierarchical-version of transformer model (Lu et al., 2021), where the hierarchical mechanism is based on the attention.

heading: 2.3 Predictions Explanation in the Legal Domain
text: The growing adoption of automatic decisionmaking systems raised awareness of its risks. Research on explainable AI has consequently grown, addressing the need to understand model behavior (Adadi and Berrada, 2018). Hence, several XAI approaches have been proposed in the literature ( Lundberg and Lee; Ribeiro et al.; Pastor and Baralis; Sarti et al.; Simonyan et al.; Sundararajan et al.; Ventura et al.; Wallace et al., inter alia) and multiple approaches and analysis have been designed to assess their quality (Atanasova et al., 2020; Attanasio et al., 2022; DeYoung et al., 2020; Jacovi and Goldberg, 2020). The demand for explainability is particularly imperative in the legal domain (Bibal et al., 2021).
A line of work consists in directly adopting interpretable models. To judge the violation of an article of the convention of human rights, Aletras et al. train an SVM with a linear kernel on n-grams and topics to facilitate the model interpretation. Given their outstanding performance, black-box models as deep learning models, especially transformerbased ones, are, however, more widely adopted. As a result, multiple works leverage post-hoc explanation methods to explain the reasons behind individual predictions of black-box models. Górski
et al. adopt the explanation method Grad-CAM, firstly proposed for computer vision, to understand predictions of convolutional neural networks for legal texts. Górski and Ramakrishna compares Grad-CAM (Selvaraju et al., 2017), LIME (Ribeiro et al., 2016), and SHAP (Lundberg and Lee, 2017) explanation methods for a legal text classification problem. The work assesses the plausibility of explanations compared to the judgment of legal professionals. The work of Malik et al. also applies a post-hoc explanation method and evaluates the plausibility of the derived explanation compared to the human rationales. The authors propose an occlusion-based approach leveraging masking individual chunk embedding and estimating their importance via classification probability change. In our work, we assess the quality of three explanation methods, i.e., a sentence-occlusion-based approach, gradient, and gradient-per-input explanation methods. We then propose a novel approach to enhance explanations with Named Entity Recognition importance tagging.

heading: 3 System Overview
text: In this section we separately introduce the approaches to L-NER, CJP (classification only), and CJPE (classification with predictions explanation).

heading: 3.1 The L-NER Task
text: Problem statement The L-NER task (Kalamkar et al., 2022) aims at identifying and classifying the named entities in unstructured legal texts. Specifically, we identify the following entities: the name of the court, the name of petitioners, the name of respondents, the name of the judge, the name of the lawyers, the date of the judgment, any organization involved, geopolitical locations, name of the act or law, sections, subsections and articles, past cases, case number, name of witnesses and other person involved.
Contribution Our main contributions to L-NER is twofold:
• We apply a recently proposed entity-aware attention mechanism, implemented in the LUKE model (Yamada et al., 2020). It produces contextualized representations of both words and entities at the same time.
• We carry out a comprehensive analysis of the performance of transformer-based models for L-NER. In particular, we examine the effect
of using both domain-specific pre-training and fine-tuned models. The purpose is to investigate the impact of pre-training on the quality of the models’ representations, the effectiveness of transfer learning in the legal domain, and the ability of the pre-trained models to capture the nuances of the legal language and domain-specific vocabulary.
Beyond LUKE (Yamada et al., 2020), hereafter we will consider the established BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) models generically fine-tuned for the NER task (namely, BERT-large and RoBERTa-large) or specifically on the legal domain (Legal BERT-base and Legal RoBERTa-base), EURLEX (BERT-base), and ECHR (BERT-base) corpora (Chalkidis et al., 2020). All the pre-trained checkpoints of these models are taken from the Hugging Face hub repository1.

heading: 3.2 The CJP Task
text: Problem statement The task of Court Judgment Prediction (CJP) and Explanation (Malik et al., 2021) aims to predict the decision for a case given all its facts and arguments (CJP). It returns a binary outcome per case.
Contribution The proposed methodology for the CJP task consists of three main stages. Firstly, we employ four distinct transformer-based encoders, namely RoBERTa-base, RoBERTa-large, Legal BERT, and Longformer-4096, to generate alternative document embeddings. All the pre-trained checkpoints of these models are taken from the Hugging Face hub repository1. The two types of train documents, i.e., single and multi, have been analyzed separately. In the multi-type dataset, for each case, multiple petitions have been filed by the appellant leading to multiple decisions. As stated by the author, this dataset is a superset of the singletype dataset (where for each case a single petition have been filed).
As these models, including Longformer, have a limited capacity for token input compared to the average length of legal documents, we investigate which section of the document exerts the most significant influence on classification performance, either the head or the tail, as previously studied by (Malik et al., 2021).
1https://huggingface.co/models latest access: February 2023
The second stage relies on a hierarchical strategy, wherein we segment sentences based on document syntax before encoding them using the best performing encoder identified at the previous step. We use ply library and customize the sentence splitter in order to avoid split occurring in presence of abbreviations or digits. The hierarchical model comprises a stack of attention layers followed by a stack of linear layers. The resulting embeddings for each document are combined through an average pooling operation between sentence representations, and the document representation is subsequently passed to the linear layers to compute the final prediction.
Finally, we generate an ensemble prediction exclusively for the test set by combining different sets of hierarchical models, trained using distinct document types and settings. The ensemble method maximizes the per-class sum of the probabilities returned by the considered models. This approach leverages the models’ diversity to enhance the final prediction’s robustness and accuracy.

heading: 3.3 The CJPE Task
text: Problem statement The task of Court Judgment Prediction and Explanation (CJPE) (Malik et al., 2021) combines the prediction of the final decision (binary outcome) with a prediction explanation consisting of a subset of case sentences that justify the decision. To generate the predictions CJPE re-uses the same approach and derived models used for the CJP sub-task (see Section 3.2).
Contribution We leverage post-hoc input attribution explanation methods to generate predictions explanations. This class of methods can be applied to explain individual predictions of a generic model. Post-hoc feature attribution methods, given a model, a target class, and a prediction, measure how much each token contributed to that outcome. These approaches are typically adopted by providing input tokens as input, and they provide as output the contribution to the prediction of each token. However, this scenario is not suitable for our case. First, the CJPE task aims at deriving explanations at the sentence level to understand which sentences are relevant to the prediction. Hence, we should remap token-level attributions to sentence-based ones. Second, expensive computational methods such as LIME (Ribeiro et al., 2016) may struggle due to the long length of documents when applied at the token level (Malik et al., 2021).
Our CJP already works at the sentence level: the model receives as input the entire document, split into sentences. Hence, we leverage post-hoc feature attribution methods to derive the importance of the input sentences provided as input. Specifically, we adopt Gradient (Simonyan et al., 2013) (also known as Saliency), Integrated Gradient (Sundararajan et al., 2017), and leave-one-out methods. At the implementation level, we use ferret (Attanasio et al., 2023), a XAI library that generates and benchmarks explanations for Transformers models. We extended the Explainer APIs of these approaches from ferret to deal with sentence inputs rather than tokens.
The adopted post-hoc attribution methods leverage only the input document and the model to derive explanations. Hence, no external knowledge of which parts of the inputs should be relevant for human experts is considered. Unfortunately, data on human rationales for the reasons behind decisions are typically unavailable. We propose to leverage NER tagging to enhance explanations, providing the information on which sentence contains legal entities. The intuition is that sentences containing legal entities should be more meaningful for the predicted outcome. Specifically, we assign each sentence’s proportion of legal entities as a relevance score (NER score). We then boost the explanations post-hoc attribution method with the NER scores. Specifically, let e(s) be the sentence attribution score provided by an attribution method for a sentence s, let n(s) be its NER score, and β ∈ R≥0 a boosting parameter. The boosted sentence score is derived as e(s)(1 + βn(s)). The larger the value β, the more importance is given to the presence of legal entities in the sentence.

heading: 4 Experimental Results
text: In this section, we report the outcomes of our empirical investigations on the three aforementioned tasks. Specifically, we participated in the shared task competition for tasks B, C-1, and C-2 (Modi et al., 2023) and evaluated the performance of our proposed models on the respective test sets using the official evaluation metrics.
We rank 11th out of 17 participants to the LNER task, 4th out of 11 participants to CJP, and 5th out of 11 to CJPE. These findings offer valuable insights into the potential of the proposed models for enhancing the accuracy and efficiency of legal NLP applications.

heading: 4.1 Experimental Design
text: Hardware Experiments were run on a machine equipped with Intel® CoreTM i9-10980XE CPU, 2 × Nvidia® RTX A6000 GPU, 128 GB of RAM running Ubuntu 22.04 LTS. We provide detailed information about the models used for the evaluation and the fine-tuning procedure in the official project repository2.
Parameter setting For the L-NER task, the LUKE-based models are trained using a batch size of 256 with a 10−4 learning rate, while the BERTbased models utilize a batch size of 1 and a learning rate of 10−4. Both models are trained for a maximum of 5 epochs with an early stopping criterion. Additionally, a weight decay of 0.01 is applied for both models, and the warmup ratio is set to 0.06.
For the CJP task, we train sentence encoders for a maximum of 15 epochs, using a learning rate of 5 · 10−5, a warmup ratio of 0.06, a weight decay of 0.01, and a batch size of 64. The hierarchical transformer-based architecture has a maximum length of 256 tokens and it is trained for a maximum of 100 epochs, with a learning rate of 5 ·10−5, and a batch size of 256.
2https://github.com/koudounasalkis/ PoliToHFI-SemEval2023-Task6

heading: 4.2 Evaluation Metrics
text: L-NER On the development set, the models are evaluated by using strict, partial, exact, and typematch F1 scores on the combined preamble and judgment sentences:
• Strict: exact boundary surface string match and entity type;
• Exact: Exact match of the entity’s boundaries to the corresponding boundaries in the text, without considering the entity’s type;
• Partial: Surface string match that covers only a portion of the boundary, irrespective of its type;
• Type-match: Some overlap between the tagged entity and the gold entity is required along with entity type match; this score gives an indication of how much overlap exists between ground truth and prediction.
On test data, we report the standard F1score (Kalamkar et al., 2022).
CJP We evaluate the binary court judgment predictions using the macro Precision, Recall and F1 score metrics.
CJPE We quantitatively evaluate explanations by comparing them with experts’ gold annotations. The gold annotations, collected by the task proposers, are the set of sentences considered relevant by legal experts. These ground-truth explanations are unavailable at training/test time and could not be used as input to the model or to tune explanation parameters. To measure the adherence of our explanations with gold ones, we use the ROUGE-L, ROUGE-1, ROUGE-2 (Lin, 2004), BLEU (Papineni et al., 2002), METEOR (Lavie and Agarwal, 2007), Jaccard Similarity, Overlap Maximum, and Overlap Minimum (Malik et al., 2021) metrics.

heading: 4.3 Results
text: L-NER Table 1 compares the performance of several models for L-NER on both the development and test sets. The empirical findings of this study suggest that models fine-tuned on legal documents outperform the ones fine-tuned on general-purpose text. The BERT-base model fine-tuned on the European Court of Human Rights (ECHR) dataset achieved the best performance on the test set with an F1 score of 83.18%, close to the Legal BERTbase outcome (F1 score: 83.09%). A possible reason is that the fine-tuned representation achieved on ECHR is robust enough to well describe other legal datasets as well. In fact, the ECHR dataset contains a large number of legal judgments, capturing a broad range of legal concepts and terms, many of them are also relevant to the Indian legal documents. Notably, the performance gaps between the best performing models are relatively small. While LUKE-base and LUKE-large achieve the best performance on the development set, they exhibit the worst performance on the test data. This could be due to models pre-training on a generalpurpose corpus, which may be not always effective in accomplishing a specific task. This confirms the importance of fine-tuning on domain-specific data.
The results of this study underscore the superior efficacy of models fine-tuned on legal text data for L-NER tasks. In the presence of legal pre-training, a BERT model of the same architecture size shows a +4% improvement over its counterpart. Additionally, Legal RoBERTa-base, which has a smaller architecture than RoBERTa-large, outperform the latter in some of the F1 scores, further demonstrating the importance of pre-training on legal text data.
CJP The preliminary results confirm the higher importance of the document tail compared to other document sections for the CJP task. The set of results obtained using various document types and textual encoders confirm the preliminary findings reported by (Malik et al., 2021).
The experimental results also indicate that the encoders’ performance depends on the document type. Specifically, RoBERTa-large and Legal BERT have shown to be the most effective models for single-type and multi-type documents, respectively.
Based on the preliminary results, we employ the aforementioned encoders to compute the sentence embeddings provided as input to the hierarchical models. Table 2 summarizes the hierarchical model outcome for both single- and multi-type documents. Each hierarchical model is a series of linear layers atop two attention layers and is validated on a set of documents of the same kind as those used in the training. Notice that the test document type is unknown.
The performance of the hierarchical models in the validation phase, i.e., between 76% and 78%, are comparable to that of the best baseline. The main difference is in the higher efficiency of attention-based networks compared to BiGRUs.
It also appears that the number of linear network layers has a weak influence on CPJ performance. Such a finding is confirmed by further tests on the transformer architectures.
It is worth noting that the model performance slightly degrades on the test set, independently of document type and distribution (i.e., roughly 10% drop compared to the development set).
Table 3 shows the outcomes of the model ensembles utilized for the CJP task. Specifically, TopS+M refers to the ensemble comprising the best performing single- and multi-type models, whereas AllS , AllM , and AllS+M are ensembles composed of all single-type, all multi-type, and all models, respectively (see Table 2). These ensembles are evaluated on the test set to ascertain their effectiveness in
handling the diverse document types. Notice that the model ensembles do not lead to any improvements in performance on the test set, indicating that the unknown typology of test documents cannot be effectively managed through ensembling.
CJPE We consider the best performing model on ILDC documents where multiple petitions have been filed by the appellant leading to multiple decisions (multi). As stated by the author, this dataset is a superset of the single-type dataset, therefore, we focus our analyses on models trained to predict multiple decisions for achieving higher generalizability than the single-type setup with almost comparable results.
The test set for the CJPE consists of 50 documents annotated by domain experts with gold labels and explanations (unavailable at test time). The adopted model achieves an F1-score of 45.25%.
We evaluate the quality of our leave-one-out (LOO), gradient (G), and gradientXInput (GxI) explanations and the one boosted with NER (Explainer-NER-β). We compare the explanation’s quality with experts’ gold annotations.
Given the input document, the adopted explainers provide an importance score for each document sentence for the predicted class. We derive the explanation as the set of top-K sentences by importance. Table 4 shows a summary of the results for the submitted explanations. Following (Malik et al., 2021), we first consider 40% of the sentences as explanations. Among LOO, GxI, and G, G explanations achieve higher results for all metrics except minimum overlap. Hence, in the following evaluations, we consider only Gradient explanations. We boosted gradient explanations with NER tagging leveraging LUKE-large model predictions. This choice has been made according to the validation
results in Table 1. We set the value β for the boosting to 3. It corresponds to a modification of the relevant sentences in the explanations of 9% across all documents and 24% of the documents. In this case, we do not observe a significant impact of the NER boosting.
We then study the impact of the length of explanations. We prefer concise explanations to summarize the relevant content better and to ease understanding. We evaluate explanations that represent from 25% to 30% of the documents. We observe that shorter explanations are indeed associated with higher evaluation scores. From the empirical results we further observe that NER boosting is effective for short explanations. When selecting a few sentences, the NER tagging provides insights into the importance of sentences containing legal entities that domain experts might consider as part of the decision justification.

heading: 5 Conclusions and Future Research Directions
text: This paper addressed two established legal AI task, i.e., the Named Entity Recognition and Court judgment Prediction. More specifically,
• It proposed to apply LUKE to address Named Entity Recognition on legal data sources and performed a comprehensive fine-tuning of state-of-the-art language models tailored to the legal domain.
• It extensively fine-tuned state-of-the-art language models to improve their ability to predict the outcome of court cases, proposing an approach based on hierarchical, attentionbased models, and a fine-grained sentence splitting.
• It presented a novel approach based on NER to explain the CJP predictions. By leveraging NER tagging, we enhanced the explanations produced by well-known XAI methodologies by also considering the presence of legal entities within a sentence.
In light of the achieved results, the L-NER models fine-tuned on legal text data outperformed those fine-tuned on general text data. Specifically, the BERT-base model fine-tuned on the European Court of Human Rights (ECHR) dataset performed best. The hierachical approach to CJP achieved results comparable to the baseline methods. For the CJPE task, NER-boosted explanations show promising results in producing concise yet informative explanations.
As future work, we plan to explore the applicability of entity-aware transformers to other legal AI tasks such as relation extraction. Entities could help to better capture the inter-dependencies between different entities and their relationships within the legal text.
Limitations and Ethics Statement
CJP systems intend to assist legal experts by providing useful information and not substituting them. The performance results that we and the other participants of the task obtained show that much research effort should be made to entitle this aim. CJP systems may reflect biases and discriminatory aspects of our society. The explanations behind CJP systems’ predictions could help practitioners reveal, mitigate, and remove biases in such systems. Moreover, explanations allow legal experts to assess the reason behind predictions, assessing if the system decisions are for the right reasons. It is then essential that system explanations are of adequate quality. The preliminary study of the plausibility of explanations we carried out in this work goes in this direction. We envision a more comprehensive assessment of the plausibility and faithfulness of explanations (Jacovi and Goldberg, 2020). Moreover, we encourage evaluating CJP performance also at the subgroup level (Pastor et al., 2021a,b; Goel et al., 2021) to assess which data subgroups experience lower performance. Subgroup-level evaluation has proven beneficial in identifying modeling issues or biases toward specific subgroups for various transformer-based models (Koudounas et al., 2023).

heading: Acknowledgements
text: This work has been partially supported by the “National Centre for HPC, Big Data and Quantum Computing”, CN000013 (approved under the M42C Call for Proposals - Investment 1.4 - Notice “Centri Nazionali” - D.D. No. 3138, 16.12.2021, admitted for funding by MUR Decree No. 1031,17.06.2022), the FAIR - Future Artificial Intelligence Research and received funding from the European Union Next-GenerationEU (PIANO NAZIONALE DI RIPRESA E RESILIENZA (PNRR) – MISSIONE 4 COMPONENTE 2, INVESTIMENTO 1.3 – D.D. 1555 11/10/2022, PE00000013), and SmartData@PoliTO center on Big Data and Data Science. This manuscript reflects only the authors’ views and opinions, neither the European Union nor the European Commission can be considered responsible for them.

heading: Future Work
text: 2022), the FAIR - Future Artificial Intelligence Research and received funding from the European Union Next-GenerationEU (PIANO NAZIONALE DI RIPRESA E RESILIENZA (PNRR) – MISSIONE 4 COMPONENTE 2, INVESTIMENTO 1. 3 – D. D. 1555 11/10/2022, PE00000013), and SmartData@PoliTO center on Big Data and Data Science. This manuscript reflects only the authors’ views and opinions, neither the European Union nor the European Commission can be considered responsible for them