IRIT_IRIS_A at SemEval-2023 Task 6: Legal Rhetorical Role Labeling Supported by Dynamic-Filled Contextualized Sentence Chunks
Year: 2023

Abstract:
This work presents and evaluates an approach to efficiently leverage the context exploitation ability of pre-trained Transformer models as a way of boosting the performance of models tackling the Legal Rhetorical Role Labeling task. The core idea is to feed the model with sentence chunks that are assembled in a way that avoids the insertion of padding tokens and the truncation of sentences and, hence, obtain better sentence embeddings. The achieved results show that our proposal is efficient, despite its simplicity, since models based on it overcome strong baselines by 3.76% in the worst case and by 8.71% in the best case.

heading: 1 Introduction
text: SemEval-2023 Task 6 (LegalEval) focuses on the understanding of texts from the legal domain and it comprises three sub-tasks: Legal Rhetorical Role Labeling, Legal Named Entity Recognition, and Court Judgment Prediction with Explanation (Modi et al., 2023). The Legal Rhetorical Role Labeling (Legal RRL) task aims to assign semantic roles to the sentences from a legal document. Examples of such roles are facts, arguments, and rulings. The exact roles are defined by the documents’ nature and by the following utilization of the labeled sentences. Legal documents are usually long and complex, so the task is relevant because the labeled sentences assist the document interpretation and the extraction of information. The labeled sentences are useful for downstream tasks, such as summarization and judgment prediction. The task is not trivial even for professionals from the legal domain, mainly because of its inherent subjectivity. The Legal RRL sub-task dataset comprises judgment documents written in English from the Indian legal system. The documents’ sentences were manually labeled by students from Indian law universities, whose each sentence was assigned with one of 13 pre-defined rhetorical roles (Modi et al., 2023).
The Legal RRL can be tackled as a sentence classification task, and this approach requires the utilization of an encoder to convert text sentences into numeric vectors. Recent Legal RRL works exploit pre-trained Transformer models to encode sentences. However, they do not fully leverage the context exploitation ability of Transformer models: sentences are usually encoded in isolation and the inter-sentence context is lost (Savelka et al., 2020; Aragy et al., 2021), the inter-sentence context is yielded by a Recurrent Neural Network (RNN) (Li et al., 2021; Bhattacharya et al., 2021) or by using data augmentation (de Lima et al., 2022), or only cover facts instead of the full range of rhetorical roles (de Lima et al., 2023).
This work presents the approach exploited by the IRIT_IRIS team to tackle the SemEval-2023 Task 6, sub-task A. We propose a novel strategy to encode sentences that fully leverages the context exploitation ability of Transformer and hence yields richer sentence embeddings. Instead of encoding sentences in isolation, we exploit sentence chunks that avoid the truncation of core sentences and the utilization of padding tokens. Achieved results lead us to deem that our proposal is efficient as the respective models overcome strong baselines by 3.76% in the worst case and by 8.71% in the best case. A secondary finding is that models based on pre-trained RoBERTa and Longformer models benefit more from the proposed approach than BERT-based models1.

heading: 2 Background
text: To leverage the context exploitation ability of Transformers, we just need to feed the encoder with a sequence of sentences separated by a special token such as the [SEP] one from BERT (Devlin et al., 2019). Because most Transformer models
1The code of our models is available at https: //github.com/alexlimatds/SemEval_2023_ Task_6A
905
have a limit regarding the sequence length (e.g., 512 for BERT) and legal documents are usually long, feeding the encoder with a whole document is not computationally feasible. A workaround is breaking the documents into sentence chunks and feeding the encoder with such chunks as proposed by Cohan et al. (2019). A downside of this approach is that inter-sentence context is restricted to chunks (sentences in a chunk are not aware of sentences in other chunks). Mullenbach et al. (2021) and van den Berg and Markert (2020) rely on overlapped chunks to overcome this: consecutive chunks share some of their edge sentences as illustrated in Figure 1.
Only two works about Legal RRL leverage sentence chunks and Transformers to encode sentences. In Malik et al. (2022) a chunk comprises up to three sentences: the core sentence (i.e., the sentence to be classified) and its left and right neighboring sentences (remark that the first sentence does not have a left neighbor and the last sentence does not have a right neighbor). Nevertheless, this model performs worse than one that encodes sentences in isolation. Kalamkar et al. (2022) exploit chunks without sharing sentences as proposed by Cohan et al. (2019) but the model performs significantly inferior to one that leverages inter-sentence context through RNNs.

heading: 3 Dynamic-Filled Contextualized Sentence Chunks
text: The approaches from Cohan et al. (2019), Mullenbach et al. (2021), and van den Berg and Markert
(2020) differ in the number of shared sentences, number of core sentences, and how the sentences are distributed among chunks.
In Cohan et al. (2019) there are no shared sentences and the number of chunks is dynamically set from the number of sentences in the document and from a hyperparameter delimiting the maximum number of sentences in a chunk. The chunks are filled so that they contain almost equal amounts of sentences. To assure that, sentences can be truncated. To establish chunks with the same number of tokens, which allows batch processing, the chunks are often filled with padding tokens.
Mullenbach et al. (2021) and van den Berg and Markert (2020) propose chunks with a fixed number of core sentences and edge sentences that must be set in advance. The drawback of this approach is that it yields chunks with a small number of content tokens when the concerned sentences are short. A large number of core sentences could avoid this, but it may lead to many truncated core sentences.
Chunks with many padding tokens are contextpoor and truncation of core sentences results in data loss. Such facts may result in poor sentence embeddings that harm the performance of classification models.
Algorithm 1 Assembly logic of DFCSCs Require: {s1, s2, ..., sn} : sentences in a document Require: m : minimum desired number of edge tokens Require: c_len : number of tokens in a chunk 1: chunkList← new List 2: chunk ← new chunk 3: i← 1 4: while i ≤ n do 5: ts ← number of tokens in si 6: tc ← number of tokens in chunk 7: if tc = 0 then 8: Put chunk into chunkList 9: Put si into chunk 10: if c_len− ts < m then 11: Fill the edges of chunk until reach c_len 12: chunk ← new chunk 13: i← i+ 1 14: else 15: if c_len− ts − tc ≥ m then 16: Put si into chunk 17: i← i+ 1 18: else 19: Fill the edges of chunk until reach c_len 20: chunk ← new chunk 21: return chunkList
To overcome the limitations of the three cited approaches, we propose Dynamic-Filled Contextualized Sentence Chunks (DFCSC). These are chunks with a variable number of core and edge sentences, without truncated core sentences, and that usually
do not require padding tokens. These characteristics lead to better leverage of the context exploitation ability of Transformer models. The procedure of filling a DFCSC has two hyperparameters, the number of tokens in a chunk c_len, and the minimum desired number of edge (shared) tokens, m. The goal is to get a chunk with the maximum number of consecutive sentences while maintaining a desired minimum number of shared tokens among successive chunks. A minimum number of edge tokens assures that inter-sentence context is not restricted to a chunk and the more content tokens in a chunk, the better the representations of core sentences. Working oriented to tokens in the chunk edges, instead of sentences, avoids the insertion of padding tokens. To fully leverage the context exploitation ability of a PTLM, c_len must be set equal to the maximum sequence length supported by the model (e.g., 512 for BERT).
Algorithm 1 presents the assembly logic of DFCSCs. A sentence s is put into a chunk if it does not harm m (lines 15 to 16). Harming m means that the remaining capacity of the chunk after inserting s is lesser than m. Otherwise, the remaining capacity in the chunk is filled with edge tokens extracted from the neighboring sentences (lines 18 to 19). Then, an empty chunk is created (line 20) and s is put into it (lines 7 to 9). When the chunk is empty, s is always inserted into the chunk (lines 7 to 9). This is the unique case in which harming m is allowed, and the chunk is filled with edge tokens when it happens (lines 10 to 11). Remark that a core sentence is truncated only when its number of tokens is larger than the chunk supports (c_len).
Algorithm 1 abstracts some details such as the special tokens. In practice, a chunk starts with a special token to mark its beginning (e.g., [CLS]), and sentences and context edges are separated by another special token (e.g., [SEP]). The examples in Figure 2 present the resulting DFCSCs, with special tokens, assembled from a set of five sentences. The first chunk has s1 and s2 as core sentences. Because there are no sentences at left, only the right edge is filled with tokens from neighboring sentences (s3 and s4). The second chunk has only s3 as core sentence. This is because inserting s4 would harm m. The edges are filled with tokens from s1 and s2 at left, and from s4 at right. The third chunk has only s4 as core sentence and it harms m. This happens because the length of s4 is greater than c_len, but since s4 is the first sentence
in the chunk, the insertion is allowed. The fourth chunk contains only s5 as core sentence since this sentence is the remaining one. Because there are no sentences at right, only left edge is filled with tokens from the neighboring sentence (s4).

heading: 4 Experimental Setup

heading: 4.1 Dataset
text: The available dataset (Kalamkar et al., 2022) comprises 277 judgment documents written in English from the Indian legal system and it is split into two sets, train (247 documents and 28,986 sentences) and validation (30 documents and 2,879 sentences). The documents’ sentences were labeled by law students and each sentence is labeled with one of 13 pre-defined rhetorical roles. The Fleiss Kappa score is 0.59, which indicates a moderate agreement among annotators and highlights the inherent subjectivity of the task. The dataset is highly unbalanced. The most and lesser frequent rhetorical roles are ANALYSIS and PRE_NOT_RELIED with respectively 11,609 and 170 sentences. The dataset also presents a high variation in sentences’ length. The shortest and longest training sentences respectively have 1 and 669 tokens. We refer the reader to Kalamkar et al. (2022) for more details about the dataset and its development.

heading: 4.2 Models and Evaluation
text: We employ two sets of models relying on DFCSCs, namely DFCSC-SEP and DFCSC-CLS. 1) DFCSC-SEP: the three models in this set rely on DFCSCs to get sentence embeddings from a pre-trained Transformer model. The first model exploits InCaseLaw (Paul et al., 2022) as a sentence encoder (c_len = 512). The second model exploits RoBERTa-base (Liu et al., 2019) (c_len = 512) while the third one exploits Longformer-base (Beltagy et al., 2020) (c_len = 1, 024). InCaseLaw is a BERT model pre-trained with corpora from the Indian legal system. Longformer is a model that relies on a sparse self-attention mechanism which allows it to deal with longer sentences than the BERT-based models. In all models, core sentences are represented by the hidden states of the respective separator tokens (the token at the right of a core sentence). We set m = 350 for the three models. 2) DFCSC-CLS: as before, but each core sentence is represented by the concatenation of hidden states of the special token marking the beginning of the
chunk ([CLS] or 
) and of the respective separator token ([SEP] or 
).
We also employ SingleSC, Cohan, and Sharing edges as baselines. Each model is presented bellow. SingleSC: single sentence classification models, that is, models that do not rely on chunks. Each sentence is individually fed to the model and it is represented by the hidden state of the respective token marking the beginning of the sentence (
 or [CLS]). The exploited pre-trained Transformers are BERT-base-uncased, RoBERTa-base, and InCaseLaw. Cohan: models that follows the chunk design from (Cohan et al., 2019), that is, chunks without sentence sharing. The exploited pre-trained Transformers are InCaseLaw, RoBERTa, and Longformer. Models based on InCaseLaw and RoBERTa have the following hyperparameter values: 85 as the maximum sentence length, 512 as the chunk length, and 7 as the maximum number of sentences in a chunk. The model based on Longformer has the following hyperparameter values: 90 as the maximum sentence length, 1,024 as the chunk length, and 16 as the maximum number of sentences in a chunk. Each sentence is represented by the hidden state of the following separator token. Sharing edges: models relying on chunks with a fixed number of core sentences and edge sentences. The exploited pre-trained Transformers are InCaseLaw, RoBERTa-base, and Longformer-base. Models based on InCaseLaw and RoBERTa have the following hyperparameter values: 80 as the maximum sentence length, 512 as the chunk length, 2 as the number of core sentences in a chunk, and 2 as the number of edge sentences. The model based on Longformer has the following hyperparameter
values: 85 as the maximum sentence length, 1,152 as the chunk length, 9 as the number of core sentences in a chunk, and 2 as the number of edge sentences. Each core sentence is represented by the hidden state of the following separator token.
Appendix A presents the remaining hyperparameters.
For all models, a fully-connected layer is employed as a classifier. All models are fine-tuned over the train set for four epochs and evaluated over the validation set. For each model, the fine-tuning is repeated four times with a different random seed for each repetition (the same four seeds for all models). The evaluation metric is the weighted F1 score.

heading: 5 Results

heading: 5.1 Experimental Results
text: Table 1 presents the scores achieved by all models. We start our analysis by considering the models employing the same pre-trained Transformer model as a sentence encoder. Firstly, and as expected, we verify that inter-sentence context exploitation is advantageous as all SingleSC models perform lower than their counterparts.
Secondly, we verify that both DFCSC approaches are really effective as they always overcome the baselines: DFCSC-SEP InCaseLaw and DFCSC-CLS InCaseLaw are respectively 4.29% and 3.76% better than Cohan InCaseLaw; DFCSCSEP RoBERTa and DFCSC-CLS RoBERTa are respectively 4.12% and 4.52% better than Cohan RoBERTa; and DFCSC-SEP Longformer and DFCSC-CLS Longformer are respectively 6.32% and 8.71% better than Cohan Longformer.
Thirdly, we realize that DFCSCs provide more gains to RoBERTa and Longformer-based models than to InCaseLaw-based ones. This suggests that the formers exploit contextual data in a better way than the last, and we believe this is due to the pretraining process of such models: InCaseLaw follows the same procedure of BERT whereas Longformer and RoBERTa are pre-trained with another and the same procedure.
DFCSC-SEP Longformer and DFCSC-CLS Longformer achieve the two highest scores among all models. We hypothesize this is because Longformer-based models work with longer chunks that result in richer sentence embeddings since they are yielded from chunks with more context, and in no truncation of core sentences.
Histograms in Figure 3 show the number of padding tokens inserted into the chunks following Cohan and Sharing edges layouts. We can realize that most of such chunks have more padding tokens than content tokens. In our experiments, the DFCSCs do not have padding tokens because the edges of a DFCSC are always filled with content tokens. We deem that this is the main factor of the superior performance of DFCSC-based models: chunks with fewer padding tokens allow the Transformer encoders to yield better sentence embeddings.
The DFCSC-CLS approach appears to be better than DFCSC-SEP. Besides the improvements that the DFCSC-CLS approach provides to RoBERTa and Longformer-based models, the standard deviation scores of DFCSC-CLS models indicate that
they have higher training stability. The performance differences among Cohan and Sharing edges models are remarkable. We believe this is mainly due to a bad tuning of Sharing edges models’ hyperparameters. This highlights another advantage of the DFCSC-based approach: it defines fewer hyperparameters and hence it presents an easier tuning.

heading: 5.2 DFCSC Hyperparameter Analysis
text: Based on the hypothesis that a richer context leads to better sentence embeddings, we deem that a good practice of hyperparameter optimization is by setting c_len with the maximum sequence length supported by the employed sentence encoder (e.g., 512 for BERT) and by the hardware platform.
Regarding the m hyperparameter, it is not obvious whether the larger its value, the larger the model performance. In order to assess the impact of m, we perform experiments on selected models with several values of m whose results are presented in Table 2. From such results, we see that 250 is the best value for both models. This suggests that the absolute amount is more important than the proportion of edges tokens in a chunk.
The assembly logic of DFCSC was not designed to deal with m = 0 and so we employ m = 2 to assure at least one token in each edge. This value leads to the worst results for each model. Although, the achieved results (0.7856 and 0.8009) are still better than the ones achieved by the models based on the Cohan layout (0.7573 and 0.7487
from Table 1). We remark that, despite the value assigned to m, many chunks will have more than 2 edge tokens as can be verified in Figure 4. As consequence, rich context is often employed when generating sentence embeddings.
There are no large differences in performance from the exploited values of m. We believe this is due to the characteristics of the DFCSC approach: core sentences are usually not truncated and hence there is no data loss; context is always provided since the filling of edges is oriented to tokens instead to sentences; and edges are filled with content tokens instead padding tokens. The low sensitivity to m is a good characteristic of the approach since it eases the hyperparameter optimization effort and leads to more predictable models.

heading: 5.3 Shared Task Results
text: The resulting board of the shared task shows 27 participants from which our team took 9th place with a 0.8076 weighted F1 score. The best-achieved score is 0.8593 by the AntContentTech team. We made five submissions and our best model was a DFCSCCLS Longformer fine-tuned for four epochs over the train and development sets (c_len = 1, 280 and m = 350). This model overcomes the hierar-
chical baseline model reported by the authors of the dataset (Kalamkar et al., 2022) that achieves a 0.7900 F1 score.

heading: 6 Conclusions
text: This work presents a novel approach to leverage the context exploitation ability of Transformers and employs it to tackle the Legal RRL subtask of SemEval-2023 Task 6. The proposal relies on DFCSCs as a way of achieving better sentence embeddings from Transformers’ hidden states. Models based on DFCSCs are compared to baselines based on similar approaches and the results show that our proposal is effective: the best DFCSC model achieves a 0.8139 F1 score whereas the best baseline does 0.7573. In the future, we plan to employ DFCSCs over other Legal RRL datasets as a way of reinforcing our findings.

heading: Acknowledgements
text: This work has been supported by the Coordination for the Improvement of Higher Education Personnel - Brasil (CAPES) and was partially supported by the LawBot project (ANR-20-CE380013), granted by ANR the French Agence Nationale de la Recherche.

heading: A Training and Implementation Details
text: Python 3.8.8, PyTorch 1.9.0, and Hugging Face 4.17.0 are the programming language and main libraries utilized by us to develop all models. Hugging Face is mainly used to load pre-trained Transformer models.
Pre-trained Transformer models are utilized with default parameters, except for Longformer ones whose global attention is set to take 
 and 
 tokens into account.
For all models, we employ the Cross-Entropy loss function and Adam optimizer (β1 = 0.9, β2 = 0.999, ϵ = 10−8, weight_decay = 10−3). The learning rate is linearly scheduled from an initial value to zero and there are no warm-up steps.
A single NVIDIA GeForce GTX 1080 Ti GPU with 11GB is used to train all models.
Table 3 presents the training hyperparameters values employed to train the models presented in Table 1. The indicated dropout rate is applied only in the fully-connected layer.

heading: Future Work
text: In the future, we plan to employ DFCSCs over other Legal RRL datasets as a way of reinforcing our findings