VTCC-NER at SemEval-2023 Task 6: An Ensemble Pre-trained Language Models for Named Entity Recognition
Year: 2023

Abstract:
We propose an ensemble method that combines several pre-trained language models to enhance entity recognition in legal text. Our approach achieved a 90.9873% F1 score on the private test set, ranking 2nd on the leaderboard for SemEval 2023 Task 6, Subtask B Legal Named Entities Extraction. Our code is available for further exploitation at: https://github.com/ tqgminh/SemEval2023_LegalNER_VTCC.

heading: 1 Introduction
text: Named Entity Recognition (NER) is a key problem in natural language processing (NLP) that involves recognizing named entities in text and classifying them into specific types. Recent studies on NER have achieved many promising results. Most of state-of-the-art NER models are based on pretrained language models, which have been trained on large text corpora and are effective for word representation.
However, Domain-Specific NER, such as Legal NER, remains a challenging task. Named entities in legal texts are more different and specific than named entities in general texts. For example, a person’s name in legal texts may refer to a petitioner, respondent or lawyer. Therefore, some pre-trained language models for legal domain have been released for use not only in Legal NER, but also for other NLP tasks in legal texts.
In this work, we propose a model that combines a pre-trained language model and some techniques, such as CRF or dependency parsing, for NER in legal texts. An ensemble method is used to further boost the performance.

heading: 2 Related Works
text: NER in legal texts has been researched for many years and in many languages, but it still remains a challenging task. Dozier et al. (2010) defined
∗* Corresponding author
five legal named entities in US legal texts (judges, attorneys, companies, jurisdictions and courts) and developed a named entity recognition and resolution system based on pattern-based and statistical models. Cardellino et al. (2017) trained a Named Entity Recognizer, Classifier and Linker by mapping the LKIF ontology to the YAGO ontology and developed a structured knowledge representation of legal concepts. Due to variant writing style and vocabulary across different languages and forms, some datasets have been published to create separate models, such as German LER (Leitner et al., 2020), LegalNERo (Păis, et al., 2021), LeNER-Br (Luz de Araujo et al., 2018), and Legal NER (Kalamkar et al., 2022).
Applying pre-trained language models boost the accuracy of NER tasks. BERT (Devlin et al., 2018) is a language model based on Transformer architecture, and was trained on two different tasks: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). RoBERTa (Liu et al., 2019) is a robust version of BERT, and was trained on a much larger dataset and using a more effective training procedure. XLM-RoBERTa (Conneau et al., 2019) is based on RoBERTa architecture and was trained on the large multi-lingual dataset. DeBERTa-V3 (He et al., 2021) replaced MLM with Replaced Token Detection (RTD) and implemented a gradient-disentangled embedding sharing method. DeBERTa-V3 achieves state-of-the-art performance on many downstream NLP tasks.
Regarding the NLP tasks in legal texts including NER, some pre-trained language models have been trained exclusively in legal texts in order to enhance the performance. For instance, Legal-BERT (Chalkidis et al., 2020), based on BERT architecture, was trained on 12GB of diverse English legal texts and achieved better results than BERT with the same number of parameters on some Legal NLP tasks. InLegalBERT and InCaseLegalBERT (Paul et al., 2022) were initialized from Legal-BERT and
415
Input:
were trained on 27GB of Indian legal documents. CRF (Collobert et al., 2011) is popular techniques used for NER. Many studies have stacked them on top of other methods and achieved impressive results, as demonstrated by Lample et al. (2016), Ma and Hovy (2016), Peters et al. (2018), Yang and Zhang (2018), Akbik et al. (2019).

heading: 3 Methodology
text: For the Legal NER Subtask in SemEval 2023 Task 6 (Modi et al., 2023), the architecture of the proposed method is illustrated in the Figure 1. Accordingly, we first execute the task with various benchmark pre-trained language models (PLMs) in this research field such as BERT, RoBERT, InlegalBERT, and so on. The output of aforementioned PLMs are then put into an ensemble module for extracting the final results. The processes of the proposed method are sequentially described as follows:

heading: 3.1 PLMs for legal name entity extraction task
text: We processed the benchmark dataset using two approaches. The first approach use the model based on the spaCy framework, following the work in Kalamkar et al. (2022). The second approach is our custom model using IOB format incorporating with CRF Layer and Dependency parsing as features in the model.

heading: 3.1.1 The first approach
text: With the input dataset, we adopt the pipeline proposed by Kalamkar et al. (2022). This pipeline used a transition-based parser, proposed by Honnibal and Johnson (2015), on the top of a PLM.
For the PLMs, we used RoBERTa, BERT, LegalBERT, InLegalBERT, and XLM-RoBERTa as the backbones of the pipeline, and all experiments were performed using the spaCy framework.

heading: 3.2 The second approach
text: The second approach is our custom model for the task. Specifically, given the input X = x1, x2, ..., xt, each word in the input is assigned a different tag depending on whether it is the beginning (B-y), inside (I-y), or outside (O) of a named entity phrase Y = y1, y2, ..., yt, respectively. For the NER task, we adopted the baseline model of SemEval 2022 (Malmasi et al., 2022). Furthermore, we also incorporated dependency parsing as a feature of the model. The PLMs were fed subword to obtain contextualized embedding information, which are then concatenated with the dependency feature and part-of-speech feature of each word. Subsequentially, the output are fed into a CRF layer to classify the label of each word in the sequence.
For the backbone PLMs, we selected various models such as XLM-RoBERTa, RoBERTa, DeBERTa, BERT, LegalBERT, InLegalBERT, and InCaseLegalBERT.
.

heading: 3.3 Ensemble Module
text: After obtaining results from two aforementioned approaches, we apply an ensemble strategy for each approach in order to improve the performance. Specifically, ensemble strategy combines multiple models through majority voting with weights. In particular, For each token xi, we predict the final label yi based on the majority label f of the models, which is formulated as follows:
yi = f(w1h1(xi), w2h2(xi), . . . , wnhn(xi)) (1)
where hj(xi) denotes the output of token xi by the PLM hj . wj denotes the weight of the PLM hj , which can be set equally (wi = w2 = ... = wn) or dynamically based on the validation process. Specifically, the values of the weights are discussed in the experiment section.

heading: 4 Experiment

heading: 4.1 Dataset
text: Our experiments were conducted on the Legal NER dataset, introduced by Kalamkar et al. (2022), which includes training and dev data. The dataset, collected from Indian court judgments, consists of two parts: preamble and judgment. Each preamble and judgment were split into sentence for training and evaluation. Named entities to be recognized belong to 14 labels.

heading: 4.2 Setting
text: For the first approach, we trained for 30,000 iterations with 4 sentences per batch. The gradient accumulation was set to 3. The optimizer used was Adam, with an initial learning rate of 5e− 5. We selected the checkpoint with the best performance on the dev set for each model. First, we combined five models with RoBERTa, BERT, LegalBERT, InLegalBERT, and XLM-RoBERTa as the backbone, with equal weights. Second, we assigned weights of 4, 2, 1, 1, and 1 to the RoBERTa, BERT, LegalBERT, InLegalBERT, and XLM-RoBERTa models, respectively, based on their performance on the dev set. We assigned higher weights to RoBERTa and BERT since they outperformed the other models, as shown in Table 1. Due to submission limitations, we only submitted the predictions of two highest
models. As shown in Table 3, RoBERTa outperformed BERT, so we assigned a higher weight to RoBERTa in the ensemble.
For the second approach, we set the gradient accumulation to 4 and trained for 50 epochs. The optimizer was Adam, with an initial learning rate of 1e− 4. We saved the checkpoint of the model after each epoch and kept the ten latest checkpoints after training. The checkpoint with the best performance on the dev set was selected for inference. We used the Transkit (Nguyen et al., 2021) to obtain the dependency feature. Ensemble method with equal weights were then applied, similar to the first approach. All experiments were executed on a single 40G Nvidia-A100 GPU.

heading: 4.3 Results
text: Table 1 and Table 2 show the Micro-F1 score of each single and ensemble model on the dev set, based on the two following approaches. In the first approach, the model using BERT as its backbone achieved the highest performance. Our ensemble method helped improve the performance, with the highest score by implementing following the first way. In the second approach, the highest perfor-
mance was achieved by DeBERTa-V3, which incorporated dependency parsing as a feature and added a CRF layer on top (the model named DeBERTaV3+Dep+CRF).
Table 3 and Table 4 show the results of our submission to the competition evaluation on the private test set. Our highest score, 90.9873, was achieved by implementing the second way of the ensemble method with five models in the first approach. The ensemble model with the first approach outperformed the baseline model, which uses RoBERTa and incorporates a transition-based parser. However, ensemble models with the second approach do not improve the performance on the private test set. Furthermore, the models from the first approach achieved better performance than the second approach on the private test set
Table 5 shows the resulting score for the Legal NER task. Our team achieved the 2nd place ranking on the final leaderboard.

heading: 5 Conclusion
text: To summarize, we applied two approaches to solve the named entity recognition of SemEval 2023 Task 6, Subtask B. Specifically, we applied an ensemble method to improve performance in the first approach. In contrast, the model did not perform well on the private test in the second approach. For the future work of this study, we try to optimize the weight for each PLM in the ensemble module in order to improve the performance.

heading: Future Work
text: For the future work of this study, we try to optimize the weight for each PLM in the ensemble module in order to improve the performance