YNU-HPCC at SemEval-2023 Task 6: LEGAL-BERT based Hierarchical BiLSTM with CRF for Rhetorical Roles Prediction
Year: 2023

Abstract:
To understand a legal document for real-world applications, SemEval-2023 Task 6 proposes a shared Subtask A, rhetorical roles (RRs) prediction, which requires a system to automatically assign a RR label for each semantical segment in a legal text. In this paper, we propose a LEGAL-BERT based hierarchical BiLSTM model with conditional random field (CRF) for RR prediction, which primarily consists of two parts: word-level and sentence-level encoders. The word-level encoder first adopts a legaldomain pre-trained language model, LEGALBERT, initially word-embedding words in each sentence in a document and a word-level BiLSTM further encoding such sentence representation. The sentence-level encoder then uses an attentive pooling method for sentence embedding and a sentence-level BiLSTM for document modeling. Finally, a CRF is utilized to predict RRs for each sentence. The officially released results show that our method outperformed the baseline systems. Our team won 7th rank out of 27 participants in Subtask A.

heading: 1 Introduction
text: In populous countries, the number of pending legal cases has been rising exponentially. For instance, in India, according to the National Judicial Data Grid1, as of 04 July 2022, the Supreme Court of India had approximately 6 million cases pending, and with the addition of other local courts, there will be far more than 6 million cases pending. It is urgent to require an automatic legal system, which helps practitioners extract accurate and valid information from legal documents, for efficient legal processing. To this end, SemEval-2023 proposes a shared Task 6 (Modi et al., 2023), LegalEval, for understanding legal texts, which mainly comprises of three subtasks including rhetorical roles (RRs) prediction, legal named entity recognition (L-NER), and
1https://njdg.ecourts.gov.in/ hcnjdgnew/?p=main/pend_dashboard
court judgment prediction with explanation (CJPE), respectively.
• Subtask A RR Prediction. Structuring unstructured legal documents into semantically coherent units.
• Subtask B L-NER. Identifying relevant entities in a legal document.
• Subtask C CJPE. Predicting the outcome of a case along with an explanation.
Focusing on the RR prediction task, it requires a system to structure unstructured legal documents into semantical segments that are aligned with RR labels such as preamble, fact, ratio, etc. Such segmentations as fundamental building blocks are crucial for many legal artificial intelligence (AI) applications, e.g., judgment summarizing, judgment outcome prediction, and precedent search. Due to the shared task having preprocessed legal documents by splitting them into several semantical sentences as coherent units, RR prediction in Subtask A could be regarded as a sentence-wise sequence labeling task that predicts a RR label for each sentence in a legal document.
With the increasing growth of deep learning (DL) and advanced pre-trained language models (PLMs) (Devlin et al., 2019), various natural language processing (NLP) tasks have been effectively addressed, such as sentiment analysis (Wang et al., 2018; Zhang et al., 2021b,a), named entity recognition (Zhou and Su, 2002), and text classification (Soni et al., 2022; Cho et al., 2014). For instance, SciBERT-HSLN (Brack et al., 2021) introduced a sequential sentence classification method for RR predictions. To further consider the problem of low shift probability in sentence labels, several works adopted label shift prediction as an auxiliary task for accurate RR prediction (Kalamkar et al., 2022). Although general PLMs, e.g., BERT (Devlin et al., 2019), RoBERT (Liu et al., 2019), and AlBERT
2075
(Lan et al., 2019), have shown a robust ability to effectively model most natural language texts and be deployed in real-world applications, they are difficult to apply directly for RR prediction tasks. The main reason is that legal texts differ from general texts used for existing model training. Most of the legal texts are much longer than the input (512 tokens) of most pre-trained models. Moreover, legal texts contain a lot of specialized terminology, and pre-trained models that are not trained on the legal corpus may not work particularly well.
To address the above problem, we proposed LEGAL-BERT-based (Zheng et al., 2021) hierarchical BiLSTM (bidirectional long short-term memory) with conditional random field (CRF) (Bhattacharya et al., 2023), denoted as HLBERTCRF, for RR prediction. LEGAL-BERT is a legalspecific PLM that further trains the BERT model on legal domain data. HLBERT-CRF primarily consists of two parts: word-level and sentence-level encoder. Word-level encoder based on LEGALBERT is first proposed to encode sentence representations via BiLSTM. Sentence representations are then integrated via attention pooling and further encoded via BiLSTM at the sentence level, resulting in a document representation that contains a set of sentence embeddings. Finally, a CRF-based classifier predicts probabilities over various RR labels. Moreover, we found that using a hard-voting strategy gained more performance. A detailed experiment is conducted on Subtask A of SemEval2023 shared Task 6, revealing our team won the 7th rank out of 27 participant teams.
The remainder of the paper is structured as follows. Section 2 provides a detailed description of the proposed method. Extensive experiments are conducted and analyzed in Section 3, and conclusions are drawn in Section 4.

heading: 2 System Description
text: This section mainly describes the proposed HLBERT-CRF model for RR predictions. As shown in Fig. 1, HLBERT-CRF primarily consists of two parts, including word-level and sentencelevel encoders. The word-level encoder is intented to generate sentence representations; the sentencelevel encoder aims to generate document representation containing sentence embeddings that are utilized for predicting sentence-wise RR labels by the CRF module. Before introducing the proposed model in detail, we first formulate the RR prediction tasks.

heading: 2.1 RR Prediction Task
text: Given a legal document x, which is semantically segmented into Ns sentences x = [x1,x2, . . . ,xNs ]. Each sentence xns = [xns1 , x ns 2 , . . . , x ns Nw
], ns ∈ [1 : Ns] contains Nw tokens split by special tokenizers, such as wordpiece tokenizer (Devlin et al., 2019). The RR predictor tasks regarded as sentence-wise sequence labeling tasks require a system fθ(ŷ|x) to predict the ground-truth sentence labels y ∈ RNs (Ma et al., 2021), where θ is the whole parameters in the system and each dimension in y is the groundtruth RR label index in the RR label list with a
vocabulary size of C.

heading: 2.2 Word-level Encoder
text: The word-level encoder is built from LEGALBERT and word-level BiLSTM. To tokenize each sentence into sequential tokens xns, we adopt a BERT tokenizer compatible with LEGAL-BERT PLM. LEGAL-BERT: LEGAL-BERT will first be used as a word embedding tool to project discrete tokens into a high-dimensional real-valued space, for nsth sentence:
wns = LEGAL-BERT(xns) ∈ RNw×dw , (1)
where dw denote the dimensionality of word embedding.
Due to LEGAL-BERT could feasibly adapt legal domain datasets and generate contextual word embeddings, it facilitates efficient modeling of sentence and document representations in legal texts by hierarchical BiLSTM. BiLSTM: LSTM is a sequential model, which employs three gate units to address the gradient vanishing problem occurring in traditional recurrent neural networks (RNNs). Given a sentence representation wns = [wns1 , w ns 1 , ..., w ns Nw
] as the input of an LSTM layer, LSTM cell calculate tth (t ∈ [1 : Nw]) word representation hnst as,
it = σ(Lineari([hnst−1, w ns t−1])) ft = σ(Linearf ([hnst−1, w ns t−1])) ot = σ(Linearo([hnst−1, w ns t−1]))
C̃t = tanh(Linearc([hnst−1, w ns t−1]))
Ct = ft ⊙ Ct−1 + it ⊙ C̃t hnst = ot ⊙ tanh(Ct)
, (2)
where Linearm(·) presents the one-layer fullyconnected layer, m ∈ {i, f, o, c} stand for input
gate, forget gate, output gate, and cell state, respectively; [·; ·] is the concatenation manner and ⊙ is the element-wise multiplication.
LSTM generally model sequence forward, i.e., left-right, lacking further contextual features for sentence representation. Therefore, we also introduce a backward LSTM to model sentence representation from right to left (Lample et al., 2016). Thus, BiLSTM at word level modeling each sentence is simply formulated as,
hns = BiLSTM(wns) ∈ RNw×dh
= [ −−−−→ LSTM(wns); ←−−−− LSTM(wns)] , (3)
where dh is the output feature dimension and hns = [hns1 , h ns 2 , ..., h ns Nw
] is the nsth sentence representation of the output of the word-level encoder.

heading: 2.3 Sentence-level Encoder
text: Sentence-level encoder model sentence representation h = [h1,h2, . . . ,hNs ] ∈ RNs×Nw×dh to capture document representation h′ = h′1, h′2, . . . , h′Ns ] ∈ RNs×dh′ , which mainly comprises of attentive pooling, sentence-level BiLSTM, and CRF-based sequential label classifier. Attentive pooling: To integrate a sentence representation with sequential tokens into a semantic vector, attentive pooling is introduced. In terms of the nsth sentence representation hns = [hns1 , h ns 2 , ..., h ns Nw
] , its integral representation is denoted as hns and formulated as,
anw = score(hnsnw)
= vT tanh(Linearatt(hnsnw)) ∈ R1
αnw = exp(anw)
Nw∑ i
exp(score(hnsi )) ∈ R1
hns =
Nw∑
i
hnsi ·αi ∈ Rdh
, (4)
where v and Linearatt(·) are a trainable vector parameter and a fully-connected layer in attentive pooling, respectively. The nw represents the nwth token in nsth sentences. Sentence-level BiLSTM: To further model longrange dependency among sentences, we use another BiLSTM applied to integral sentence representations in a document. Based on integral sentence representations ĥ = [h1, h2, ..., hNs ] ∈ RNs×dh , the sentence-level BiLSTM calculates
a document representation with sentence embeddings by
h′ = BiLSTM(ĥ) ∈ RNs×dh′ . (5)
CRF-based classifier: Following the previous work for sequence labeling tasks (Bhattacharya et al., 2023), we employ a CRF-based classifier to predict the RR label for each sentence embedding. CRF-based classifier considers not only the current sentence representation but also the relatedness among RR labels for RR prediction, formulated as,
ŷ = CRF-Classifier(h′) , (6)
where ŷ ∈ RNs is predicted RR labels for each sentence.

heading: 2.4 Hard Voting Strategy
text: To further leverage the performance of RR predictions in Subtask A, we use a hard-voting strategy to assemble multiple HLBERT-CRF models for an ultimate inference. As shown in Fig. 2, given Nv HLBERT-CRF models that are initialized with different random seeds and individually trained on the same training datasets, the final sequential label is selected according to the RR labels with the maximum votes recommended by different models in the statistic (Farooqi et al., 2021).

heading: 3 Experimental Results
text: In this section, extensive experiments were conducted and analyzed.

heading: 3.1 Datasets
text: RR prediction tasks provided a RR corpus, which is collected from Indian Court websites and mixed of Supreme Court judgments, High Courts judgments, and district court judgments. The corpus is split into train, dev, and test datasets (Malik et al., 2021). Only train and dev datasets can be available for any participant and used for system optimization. while test datasets used to evaluate the performance of submitted models are peculiar to task organization. A detailed statistic of the corpus is listed in Table 1.
Each entry contains three parts: ID, annotations, and data. ID is the unique identifier of the document; data is the actual judgment; and annotations include the sentence ID, sentence text, sentence position in the judgment text (marked by start and end), and label of the sentence. It should be noted
that this is a multi-class classification task where each sentence has only one label. There are 13 kinds of labels such as preamble, fact, ratio, arguments, etc.

heading: 3.2 Evaluation Metrics
text: To evaluate the performance of participant systems, the organizer provides weighted micro-F1 score as the official metric.
• F1 score
Precision = TP
TP + FP
Recall = TP
TP + FN
F1 = Precision× Recall Precision + Recall · 2
, (7)
where TP, FP, and FN represent the number of predicted true-positive, false-positive, and false-negative samples, respectively, given a certain RR label as a positive label and others as negative labels.
• Weighted F1 score
weighted-F1 = C∑
c
Ec E F1,c , (8)
where C is the number of RR label types; Ec, F1,c, and E denote the number of samples with respect to label c, F1 score as label c being positive label, and the total number of samples, respectively.

heading: 3.3 Implementation Details
text: Hyper-parameters. For token splitting, the BERT-base-uncased tokenizer is used. To build legal documents as hierarchical structures, we chose the maximum number of sentences in each document of a batch of samples as the fixed document length (Ns). For documents less than such
document length, zeroed sentences are padded at the end of them. Regarding sentence length (Nw), we set it to 128. Then, sentences less or longer than 128 tokens will be zeroed-token padded or truncated. LEGAL-BERT is initialized from well pre-trained checkpoint2. The dimensionalities of word embedding (dw), sentence representation (dh) and document representation (dh′) are identical and set to 768. In terms of optimization, we use Adam as the optimizer and gradient clip strategy for avoid gradient explosion. More detailed hyper-parameter settings can be found in Table 2.
Note that, all parameters are selected in grid search method, monitored on the best Dev performance. The code of this paper is availabled at https://github.com/cy330874054/ 2023_task6_RR_predition/ Baselines: To investigate the effect of the proposed method, we introduce several baseline models. Initially, we replace the LEGAL-BERT (Zheng et al., 2021) with other PLMs, such as XLNet (Yang et al., 2019), RoBERTa (Liu et al., 2019), ALBERT (Lan et al., 2019), and BERT (Devlin et al., 2019). Next, we altered hierarchical structures as baselines to investigate effect of the hierarchical BiLSTM.

heading: 3.4 Result and Analysis
text: Comparative results are reported in Table 3, which is categorized into two groups.
The first group comprises hierarchical BiLSTMCRF models with different PLMs as word embeddings. It can be observed that, with varying embeddings, RR prediction models perform differ-
2https://huggingface.co/docs/ transformers
ently. The main reason is that PLMs may learn different specific knowledge during pre-training phases, and their diverse supervision objectives make the learnt information to different extents. The proposed method that utilizes LEGAL-BERT achieves the best results, because LEGAL-BERT is pre-trained in a large legal corpus and easy to facilitate legal-specific downstream tasks, i.e., legal RR prediction.
In the second group, we conduct several ablation studies to investigate the effect of hierarchical structure. First, we replace the word and sentence-level BiLSTM with Transformer layers as HLBERT-CRF w/ Transformer structure, which reduces weighted-F1 scores. This scenario reveals that BiLSTM is more suitable for building hierarchical structures than Transformer layers. Moreover, with the drops of word or sentence or bothlevel BiLSTM, corresponding performance is simultaneously degraded, further demonstrating the effect of BiLSTM-based hierarchical structure for RR predictions.
Finally, the proposed HLBERT-CRF model won the 7th rank out of 27 participant teams in SemEval2023 shared Task 6 (Subtask A) with a weighted-F1 score of 0.8146. Table 4 reported feedback test performance in two submissions. The proposed model introducing a hard-voting ensemble strategy slightly outperforms the model without hardvoting.
In summary, the priority of the proposed model is three-fold: 1) introducing a legal-specific PLM to initialize word embeddings; 2) regarding RR prediction as a sentence-wise sequence labeling task and employing a hierarchical BiLSTM with CRF as the backbone; 3) adopting hard-voting ensemble strategy for performance improvement.

heading: 4 Conclusions
text: In this paper, we proposed an HLBERT-CRF model for RR prediction. HLBERT-CRF utilizes hierarchical BiLSTM with CRF structure the backbone and introduces legal-specific PLM, LEGAL-BERT, for knowledge transferring. As a result, HLBERTCRF with a hard-voting strategy won the 7th in SemEval-2023 shared Task 6 (Subtask A).
In the future, we will explore more complex legal text-based NLP tasks by devising more efficient structures and explainable modules.

heading: Acknowledgements
text: This work was supported by the National Natural Science Foundation of China (NSFC) under Grant Nos. 61966038 and 62266051, and the Yunnan Postdoctoral Science Foundation under Grant Nos. C615300504048. The authors would like to thank the anonymous reviewers for their constructive comments.

heading: Future Work
text: In the future, we will explore more complex legal text-based NLP tasks by devising more efficient structures and explainable modules