CSECU-DSG at SemEval-2023 Task 6: Segmenting Legal Documents into Rhetorical Roles via Fine-tuned Transformer Architecture
Year: 2023

Abstract:
Automated processing of legal documents is essential to manage the enormous volume of legal corpus and to make it easily accessible to a broad spectrum of people. But due to the amorphous and variable nature of legal documents, it is very challenging to directly proceed with complicated processes such as summarization, analysis, and query. Segmenting the documents as per the rhetorical roles can aid and accelerate such procedures. This paper describes our participation in SemEval-2023 task 6: Sub-task A: Rhetorical Roles Prediction. We utilize a finetuned Legal-BERT to address this task. We also conduct an error analysis to illustrate the shortcomings of our deployed approach.

heading: 1 Introduction
text: Analyzing and extracting critical information from legal documents poses a great challenge for researchers. Legal documents can vary substantially in content, format, and structure depending on the jurisdiction, domain, and nature of documents. The corpus of legal documents is expanding rapidly, resulting in the need for automated systems for the effective retrieval of consequential information. Moreover, to ensure that these legal documents are utilized efficiently and justly, they must be accessible to a wide range of people. The task benefiting from the automation of legal documents are case law analysis, summarization, semantic search, and so on. Identifying the rhetorical role of each segment in the legal documents can aid in these tasks. Rhetorical role identification refers to partitioning the document into semantically meaningful segments with each segment having specific roles and purposes. These rhetorical roles can be used to extract significant information for further processing and analysis. But due to legal documents being unstructured, varying from domain to domain, and
The first three authors have equal contributions.
containing domain-specific literature, the identification of rhetorical roles is a challenging task. To address this challenge, SemEval-2023 task 6 (Modi et al., 2023) incorporated Sub-task A: Rhetorical Roles Prediction (RR). The task is to segment a given legal document by predicting the rhetorical role label for each sentence. An instance of a document segmented according to the rhetorical roles is illustrated in Figure 1.
Hachey and Grover (2004) employed a feature set consisting of the location of the sentence in the document, thematic words, sentence length, and cue phrases. They utilized decision tree, naive bayes, winnow algorithm, and support vector machine (SVM) as the classifiers. On the other hand, Saravanan et al. (2008) investigated rule-based learning algorithms (such as SLIPPER) and conditional random fields (CRF) along with features like indicator features, local features, layout features, state transition features, and legal vocabulary features. A multitask learning (MTL) framework incorporating Bi-LSTM and CRF was developed by Malik et al. (2021). They used document rhetorical role label shift as an auxiliary task for segmenting a legal document. They experimented with modeling the label shift using both SBERTShift (Reimers and Gurevych, 2019) and BERT-SC. Moreover, experiments were conducted on whether generalization can be implemented so that model trained on one sort of legal domain works well on different legal domains while classifying rhetorical roles. Savelka et al. (2021) employed RoBERTa (Liu et al., 2019) and SVM as their classification model for investigating this notion. Their findings portray that a model trained on a certain legal domain can work well on other legal domains, in fact sometimes better than training with the same legal domain. Moreover, using multiple domains together to train a model yields good performance.
In this paper, we present our deployed framework to address the challenge of predicting rhetor-
2112
ical roles as elucidated in SemEval-2023 task 6: Subtask A. We employ Legal-BERT (Zheng et al., 2021) and further finetune it to obtain a better performance.
We organize the remaining contents of the paper as follows: we illustrate our proposed framework in Section 2. Section 3 elucidates the experimental specification. We depict an analysis of erroneous predictions of rhetorical roles in Section 4. In Section 5, we conclude this paper with some strategies that we intend to adopt in future.

heading: 2 Proposed Framework
text: In this section, we elucidate our presented framework for the task of rhetorical role predictions. We exploit fine-tuned Legal-BERT (Zheng et al., 2021) for classifying legal sentences into 13 rhetorical roles. Given a legal judgment document, we segment it into separate sentences to cast this challenge as a multi-class sequence classification task. The overview of our proposed framework is depicted in Figure 2.

heading: 2.1 Legal-BERT
text: To better capture the critical context of the legal sentences, we employ Legal-BERT (Zheng et al., 2021) for the rhetorical roles prediction task. LegalBERT is a variant of BERT (Devlin et al., 2019) architecture that is one of the phenomena of harnessing transfer learning in language-related tasks such as sequence classification, question answering, and summarization. It is pre-trained on a corpus of Harvard Law legal documents from 1965 to 2021. It contains 3,446,187 case judgments of the entire federal and state courts in the US. The model was fine-tuned based on the pre-trained ‘bertbase-uncased’ model which contains 12 stacking encoder layers and utilizes the notion of understanding languages bidirectionally to capture the context of the text. Before fine-tuning, we segment the legal documents into separate sentences and then tokenize them using the BertTokenizer. Subsequently, we train the model through several iterations for the legal text classification task and get the final predicted label which is one of the 13 rhetorical roles.

heading: 3 Experiments and Evaluations

heading: 3.1 Dataset
text: The dataset used for this task is the Rhetorical Roles Corpus (Kalamkar et al., 2022) provided in
the SemEval-2023 task 6 (Modi et al., 2023). The corpus comprises 354 English Indian legal judgment documents consisting of 40,305 sentences labeled with 13 rhetorical roles. The roles can be defined as preamble, facts, ruling by lower court, issues, argument by petitioner, argument by respondent, analysis, statute, precedent relied, precedent not relied, ratio of the decision, ruling by present court, and none. The judgment documents were collected from the Supreme Court, High Court, and some district-level courts of India. The documents are provided in JSON format where each sentence is annotated according to the rhetorical roles. Consecutive sentences with the same rhetorical role form a segment. The legal sentences were manually labeled by several selected law students. A brief summary of the dataset is presented in Table 1. We consider the micro F1-score as an evaluation measure for this task.

heading: 3.2 Experimental Setup
text: In Table 2, we illustrate the hyper-parameter setting of our proposed model. We utilized the Google Colaboratory platform for experimental purposes and implemented our model using Pytorch. We finetuned the pre-trained Legal-BERT model proposed by (Zheng et al., 2021) to handle the challenge of legal text classification. After assembling the training data into 32 batches, we train this model in 20 epochs with a learning rate of 1e− 5.

heading: 3.3 Results and Analysis
text: In this section, we compare our system with other participants of the SemEval-2023 task 6. Table 3 depicts the accomplished results of other participants along with our submitted system. We can observe that our model attained a mediocre performance compared to the other top-performing teams. Though our model obtained a 64% F1-score on the test set of the judgment documents, it lacks by a huge margin compared to the top-performing team.

heading: 4 Discussion
text: In this section, we further inspect the performance of our proposed system on the development set by presenting some of the predictions in Table 4.
We can observe that in the first instance, our proposed method successfully predicted the text as a statute by capturing the context of describing an established law. However, in the second instance, the legal sentence is incorrectly predicted as a preamble (PREAMBLE) while it is actually a fact (FAC). This erroneous prediction can occur because of the mentioned names of the alleged parties which typi-
cally occurs in the preamble. In the next example, the text is misclassified as a ruling by the present court (RPC) but the ground truth is a ratio (RATIO). It can happen because the sentence reflects a similar pattern as a final decision which is normally defined in an RPC.
Further, we investigate the performance of our system in Figure 3 by presenting a statistic of correctly predicted labels against the ground truth of the development set. We can observe that in most cases, our system correctly predicted nearly the same number of rhetorical roles as the gold labels of the development set. But in some cases, such as ARG_PETITIONER, RLC, ARG_RESPONDENT, STA, PRE_NOT_RELIED, and RATIO, our system lacks a considerable number of differences. The difference can occur since the dataset is quite imbalanced and very few training samples are provided for some of the rhetorical roles.

heading: 5 Conclusion
text: In this paper, we have described our proposed framework Legal-BERT. In the future, we intend to extend the corpus to include other legal domains with different structure. We also aspire to address the other two subtasks of SemEval-2023 task 6.

heading: Future Work
text: In the future, we intend to extend the corpus to include other legal domains with different structure. We also aspire to address the other two subtasks of SemEval-2023 task 6