MLlab4CS at SemEval-2023 Task 2: Named Entity Recognition in low-resource language Bangla using Multilingual Language Models
Year: 2023

Abstract:
Extracting of NERs from low-resource languages and recognizing their types is one of the important tasks in the entity extraction domain. Recently many studies have been conducted in this area of research. In our study, we introduce a system for identifying complex entities and recognizing their types from low-resource language Bangla, which was published in SemEval Task 2 MulitCoNER II 2023. For this sequence labeling task, we use a pretrained language model built on a natural language processing framework. Our team name in this competition is MLlab4CS. Our model MuRILCRF Bangla produces a macro average Fscore of 76.27%, which is a comparable result for this competition.

heading: 1 Introduction
text: An interesting and well-studied area of natural language processing (NLP) is the extraction of entities and detecting relationships between entities from unstructured text. Named entity recognition (NER) is a sub-task of information extraction, where entities are extracted from a given text and classified into some pre-defined classes.
NER has many practical use cases in various NLP tasks, including text summarization (Toda and Kataoka, 2005), information retrieval, machine translation (Babych and Hartley, 2003), and question answering (Mollá et al., 2006). A wide range of methodologies have been investigated by researchers to perform NER tasks including rulebased techniques (Krupka and IsoQuest, 2005), supervised feature-based approaches (Liao and Veeramachaneni, 2009), unsupervised methods (Feldman and Rosenfeld, 2006), deep learning-based methods (Li et al., 2020) and transformer-based approaches (Vaswani et al., 2017).
Recent applications of pre-trained language models (PLMs) have shown significant improvement
*These authors contributed equally to this work
in entity extraction and relation extraction tasks. PLMs can learn highly accurate linguistic, semantic, and factual information from a sizable amount of unlabeled data through self-supervised pre-training (Wang et al., 2022).
The second edition of the shared task was conducted in SemEval 2023 and was named Multilingual Complex Named Entity Recognition (MultiCoNER II). The goal of this task was to build a system, which is capable of extracting fine-grained complex entities of different categories, as provided in the data. The complete list of categories is - Facility, OtherLOC, HumanSettlement, Station, VisualWork, MusicalWork, WrittenWork, ArtWork, Software, MusicalGRP, PublicCorp, PrivateCorp, AerospaceManufacturer, SportsGRP, CarManufacturer, ORG, Scientist, Artist, Athlete, Politician, Cleric, SportsManager, OtherPER, Clothing, Vehicle, Food, Drink, OtherPROD, Medication/Vaccine, MedicalProcedure, AnatomicalStructure, Symptom, and Disease (Fetahu et al., 2023b).
The organizers (Fetahu et al., 2023b) invited the participants to submit NER models for each of the 13 languages listed on the website. They provided a different dataset (Fetahu et al., 2023a) for each language. Each dataset was made up of three parts: training, development, and testing. During the evaluation period of this competition, only the tokens were provided in the testing dataset and the named entity tags were provided for the training and development datasets. After the end of the evaluation phase, the organizers released the gold-standard test data and we used it to analyze our system. We participated in the task to build a NER model in the Bangla language track.
We participated in SemEval MultiCoNER II 2023 and developed a neural framework to extract the entities from a Bangla language dataset (Track 11). Note that Bangla is a low-resource Asian language used primarily in Bangladesh and
1388
in some parts of eastern India. Our proposed system performs entity extraction from a low-resource Bengali dataset using MuRIL language model. It uses a transfer learning strategy by fine-tuning MuRIL (Khanuja et al., 2021). Furthermore, we apply a conditional random field (CRF) decoder to implement the entity extraction task. Experimental results show that the proposed system achieves comparable results on this task.

heading: 2 Related Work
text: Named entity recognition (NER), also known as entity extraction, is a fundamental task in natural language processing. Early works in NER systems used handcrafted features (Zhou and Su, 2002), but even state-of-the-art systems developed for one domain did not generally perform well for other domains. These models also struggle with multilingual datasets, and while they may obtain better precision, it comes at the cost of a lower recall and higher cost of work. Since the early 2000s, NER has attracted much attention and has seen various applications in bioinformatics, molecular biology, natural language processing, and information extraction. In recent years, deep-learning methods have been employed in NER tasks (Chiu and Nichols, 2015) and have shown considerable improvement over conventional methods. Deep learning models can perform better than feature-based approaches as NER benefits from non-linear transformations, which help the model to learn complex features of the dataset.
Pre-trained contextual embeddings like BERT, LUKE, and XLM-R have achieved near-human performance in certain NER tasks. These embeddings are generally trained on large datasets like Wikipedia in order to improve the contextual representation of entities. However, these models significantly underperform on unseen entities and noisy datasets (which are more “real-world”-like) Meng et al. (2021). Some models implement knowledge bases or gazetteers, into neural architectures in order to incorporate external information. But, such systems have only reported limited improvements (Rijhwani et al., 2020). Models that use gazetteer features have been shown to suffer feature undertraining, where the model relies heavily on either context or gazetteer. Hence, such models are not reliable as utility of gazetteer is variable.
Recent studies (Lin et al., 2018), (Yang et al., 2016), (Xie et al., 2018) have developed multi-
lingual, multi-task models that achieve state-ofthe-art performance in a cross-lingual setting, with much lower resource requirements than earlier models. In some cases, these models are able to perform at the level of systems that use data augmentation, by using transfer learning. They also provide comparable results in scenarios with significantly fewer cross-lingual resources.
The SemEval 2023 task 2 is a continuation of MultiCoNER 2022 (Malmasi et al., 2022b) which also featured the aforementioned tracks along with a code-mixed track. The 2022 datasets also featured complex entities with limited context (Malmasi et al., 2022a). Methods that augmented external knowledge into transformer models achieved the best results, but they were unable to perform at the same level for Indian languages. NER in multi-lingual domain has benefited from advances in pre-trained multi-lingual transformers, but these systems are bottlenecked by limited entity knowledge across all languages (Fetahu et al., 2021). Another key challenge for pre-trained models is the presence of emerging entities in real-world scenarios like web-queries. Fine-tuning pre-trained embeddings is a significantly effective approach for many downstream NLP tasks including NER. Recent approaches propose to train already finetuned embeddings on specific training data. Shi et al.(Shi and Lee, 2021) proposed to first train on a general multilingual model and then fine-tune it for a specific language. Multi-stage fine-tuning has also shown to accelerate the training process.

heading: 3 Data
text: There are 33 different fine-grained NER entity types in the Bangla dataset (Fetahu et al.,
2023a). The dataset is in standard CoNLL format, which uses BIO (Beginning-Inside-Outside) tagging scheme. The datasets provided were of three types namely, training, development, and testing. The training and development data contained tokens with tags, whereas testing data contained only tokens during the evaluation phase. After the completion of the competition, updated test data containing tokens and tags was released. The dataset statistics have been shown in Table 3. Finegrained entity-type statistics have been presented in Table 2. In Table 1, a snippet of the dataset has been displayed. In Figure 1, the distribution of entity tags was shown for the training, development, and testing data.
From Table 3, we can clearly see that the testing data consists of double the number of instances present in the training data. By checking the frequency counts of the entity tags, we found that they are present in all three types of data in similar proportions. In Figure 1, the distribution of entity tags in terms of frequency buckets has been shown. It shows that only a few of the entity tags are highly frequent.

heading: 4 Methodology
text: In this section, we introduce the problem definition followed by the proposed methodology of our neural system called (MuRIL + CRF + Bangla) MuRILCRF Bangla 1.

heading: 4.1 Problem Definition
text: The primary task of our work may be stated as follows. Given a set of Documents D = (D1,D2, ...,Dn), our objective is to identify entity spans Es as well as the categories of the identified entities from the sentences (x1, x2, x3...xn) of a particular document Di ∈ D. Here, we consider our tasks as a sequence labeling problem such that, given a sequence of tokens in any sentence x = (v1, v2, v3...vn) from a document Di ∈ D, our proposed neural framework aims to learn certain parameters of fθ, which can map an input sequence of word vectors to a sequence of output labels f : (Vi, θ) → yi, where each Vi ∈ Rd denotes an embedding vector for the token Vi and each yi ∈ {B− < type >, I− < type >,O} represents a label indicating whether the token is the beginning, continuation, or end of an entity span.

heading: 4.2 System Description
text: In our system, we use MuRIL Large (Khanuja et al., 2021) model as the underlying language model and fine-tune it using a task-specific loss function. MuRIL is specifically designed to deal with the diverse linguistic characteristics of Indic
1https://github.com/MLlab4CS/MurilCRFBangla. git
languages. Therefore, we chose this model for the NER task in the Bangla language. We take a set of sequence pairs, D = {(x, y)}, and input the word sequences into the tokenizer module. Subsequently, we pass them through MuRIL Large PLM. The output representation produced by the module (xMuRIL = {v1, v2, ..vn}, vi ∈ RdMuRIL) is fed into a linear-chain CRF layer to obtain the conditional probability pθ(y|x):
ϕ(y′, y, ) = exp(W Ty vi + by′,y)
pθ(y|x) = ∏n
i=1 ϕ(yi−1, yi, vi)∑ y′∈γ(x) ∏n i=1 ϕ(y ′ i−1, y ′ i, vi)
(1)
The set of model parameters is referred to as θ, while γ(x) stands for the collection of all feasible label sequences corresponding to x. In the potential function ϕ(y′, y, ), W Ty vi is the emission score and by′,y is the transition score, where W T ∈ Rt×dMuRIL and b ∈ Rt×t are parameters and the subscripts y′ and y are the indices of the matrices. We train our proposed model using negative
log-likelihood loss LNLL(θ) = − log (p)θ(y⋆|x). During inference, the model prediction ŷθ is given by Viterbi decoding.
The overall architecture diagram of our proposed model, MuRILCRF Bangla has been shown in Figure 2.

heading: 5 Results and Analysis
text: We explore the effectiveness of several available multilingual PLMs for our downstream entity extraction task.
– IndicBertBangla: In this model, we fine-tune indic-bert 2 (Kakwani et al., 2020) on our downstream task.
– XLMR-LargeBangla: Here, we apply fine-tuning strategy using xlm-robertalarge 3 (Conneau et al., 2020) for our downstream task.
2https://huggingface.co/ai4bharat/indic-bert 3https://huggingface.co/xlm-roberta-large
– MuRILBangla: Similarly, we fine-tune muril-large-cased 4 for our entity extraction task.
– IndicBertCRF Bangla: Similar to IndicBertBangla, we fine-tune indic-bert follwed by a CRF decoder layer.
– XLMR-LargeCRF Bangla: Here, we fine-tune xlm-roberta-large along with a CRF decoder layer.
– MuRILCRF Bangla: Additionally, we also finetune muril-large-cased with a CRF decoder for our downstream entity extraction task.
We conduct the required training experiments on these six proposed models using Bangla training data. Table 4 shows the performance of our five proposed models on Bangla development data. From Table 4, we observe that MuRILCRF Bangla outperforms all the models in terms of macro average precision, recall, and F-score. So, we fixed MuRILCRF Bangla as our best model to submit the prediction on the test dataset. Table 5 shows the
4https://huggingface.co/google/ muril-large-cased
proposed model’s performance in terms of macro average precision, recall, and F-score. We observe that MuRILCRF Bangla outperforms other models in the macro average precision, recall, and F-score.
Table 6 shows a dataset snippet, where our model captures NERs from the test dataset. Table 7 produces fine-grained entity-wise precision, recall, and F-score and also macro average on precision, recall, and F-score by our best model MuRILCRF Bangla. From Table 7, we clearly see that our model MuRILCRF Bangla performs well for some low-frequency entities like PrivateCORP but fails to do so, for some other low-frequency entities like ArtWork, and AerospaceManufacture.
Throughout our experiments, we utilize the FLAIR (Akbik et al., 2019) framework and train all the neural models for 30 epochs with a batch size of 64, using AdamW (Loshchilov and Hutter, 2019) optimizer with a very small initial learning rate of 5e−5 and a stopping criterion as mentioned in (Conneau et al., 2020). To carry out our experiments, we used NVIDIA A100 and NVIDIA RTX A600 GPUs on the Paperspace cloud.

heading: 6 Conclusion
text: This paper discusses a transformer-based deep neural network for extracting complex named entities from a Bangla (low-resource language) dataset released by SemEval MultiCoNER II 2023 shared task team. Our system, MuRILCRF Bangla, has achieved a macro F-score of 76.27% on the test data. Our future work will include few-shot learning strategies to extract complex NERs from lowresource languages. We also plan to look into datadriven factored modeling techniques for dealing with class imbalance issues.

heading: Future Work
text: Our future work will include few-shot learning strategies to extract complex NERs from lowresource languages. We also plan to look into datadriven factored modeling techniques for dealing with class imbalance issues