LSJSP at SemEval-2023 Task 2: FTBC: A FastText based framework with pre-trained BERT for NER
Year: 2023

Abstract:
This study introduces the system submitted to the SemEval 2022 Task 2: MultiCoNER II (Multilingual Complex Named Entity Recognition) by the LSJSP team. We propose FTBC, a FastTextbased framework with pre-trained Bert for NER tasks with complex entities and over a noisy dataset. Our system achieves an average of 58.27% F1 score (fine-grained) and 75.79% F1 score (coarsegrained) across all languages. FTBC outperforms the baseline BERT-CRF model on all 12 monolingual tracks.

heading: 1 Introduction
text: NER (Named Entity Recognition) is a sequence labeling task that finds spans of text and tags them to a named entity (persons, locations, organizations, and others.). NER finds its applications in various Natural Language Processing (NLP) tasks, including Information Retrieval, Machine Translation, Question Answering, and Automatic Summarization (Jurafsky and Martin, 2021). It also finds application across various domains, including social media, news, e-commerce, and healthcare. Words having multiple meanings increase the complexity of the NER task. In English, the word “Sunny” can refer to both a person and the weather. The problem of ambiguous entities become all the more pronounced for Indian languages. In the Bangla sentence “cañcala caudhurı̄ cañcala bodha karachena" the first “cañcala” refers to a person named cañcala whereas the second “cañcala” (restless) is an adjective.
Pre-trained contextual embeddings such as BERT (Devlin et al., 2019), XLM-R (Conneau et al., 2019), and other transformer-based (Vaswani et al., 2017) models have improved the performance of NER. The embeddings are trained on
*These authors contributed equally to this work
large corpora, improving the named entities’ contextual representations. However, for low-resource languages like Bengali, Hindi, Swedish, and Farsi, the performance of these pre-trained models lag compared to their English counterpart.
MultiCoNER II shared task (Fetahu et al., 2023b) aims at building Named Entity Recognition (NER) systems for 12 languages, including Bangla (BN), German (DE), English (EN), Spanish (ES), Farsi (FA), French (FR), Hindi (HI), Italian (IT), Portuguese (PT), Swedish (SV), Ukrainian (UK), Chinese (ZH). The task has two kinds of tracks, including one multilingual track and 12 monolingual tracks for all the languages. The monolingual track requires the development of NER models for that particular language, whereas the multilingual track requires the development of a single multilingual model that can handle all 12 languages. The sequence of tokens in MultiCoNER II is classified into 30+ classes. Further, including simulated errors, like spelling mistakes, in the test set makes the task more realistic and difficult.
This paper proposes FTBC, a framework comprising FastText (Bojanowski et al., 2017) and BERT for the MultiCoNER II task without using any other external knowledge bases. FTBC achieves an average F1 score of 58.27% on finegrained and 75.79% on coarse-grained categories across all the 12 monolingual tracks. FTBC outperforms the baseline BERT-CRF for all the languages for both fine and coarse-grained categories.

heading: 2 Related Work
text: Named Entity Recognition is a fundamental task in natural language processing (NLP). Most of the earlier works considered NER as a sequence labelling task. Chiu and Nichols (2016) introduced convolutional neural networks for NER, whereas ŽukovGregorič et al. (2018) used recurrent neural networks (RNNs) with CRF for NER. However, these models do not perform very well on complex enti-
1254
ties like movie names (Meng et al., 2021), which are challenging to tag. Fetahu et al. (2021) has extended this work to multilingual code-mixed settings. Pre-trained language models like ELMo (Peters et al., 2018) or transformer-based (Vaswani et al., 2017) models like BERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2019) improve the performance of NER on complex entities.
MultiCoNER I (Malmasi et al., 2022c) was a shared task to build NER systems for complex entities in 11 languages. MultiCoNER I shared challenge was conducted using the dataset curated by Malmasi et al. (2022a). The critical challenge of MultiCoNER I was dealing with complex entities with limited context. MultiCoNER II classifies the tokens into 30+ finer classes. Additionally, simulated errors have been inserted in the dataset to make the task more challenging and realistic.

heading: 3 Dataset Description
text: MultiCoNER II shared task (Fetahu et al., 2023b) (Fetahu et al., 2023a) focussed on building complex Named Entity Recognition (NER) systems for 12 languages. The languages include Bangla (BN), German (DE), English (EN), Spanish (ES), Farsi (FA), French (FR), Hindi (HI), Italian (IT), Portuguese (PT), Swedish (SV), Ukrainian (UK), Chinese (ZH). MultiCoNER II is an extension of MultiCoNER I (Malmasi et al., 2022c) (Malmasi et al., 2022b) where the six broader classes, namely Person, Group, Creative Works, Location, Medical, and Product, are further classified into 30+ finer classes. The fine to the coarse level mapping of the classes is as follows:
• Person (PER): Scientist, Artist, Athlete, Politician, Cleric, SportsManager, OtherPER.
• Group (GRP): MusicalGRP, PublicCORP, PrivateCORP, AerospaceManufacturer, SportsGRP, CarManufacturer, ORG.
• Creative Work (CW): VisualWork, MusicalWork, WrittenWork, ArtWork, Software.
• Location (LOC): Facility, OtherLOC, HumanSettlement, Station.
• Medical (MED): Medication/Vaccine, MedicalProcedure, AnatomicalStructure, Symptom, Disease.
• Product (PROD): Clothing, Vehicle, Food, Drink, OtherPROD.
The train data of each of the 12 languages varies from 9,000+ sentences to 16,500+ sentences, and the validation set ranges between 500+ to 800+ sentences. The test set for each track had at least 18,000+ sentences, with the lowest for Hindi (18,349 sentences). The multilingual track comprises data from each of the monolingual tracks. The dataset consists of simulated errors like typographical and spelling mistakes, making the NER task more challenging and realistic.

heading: 4 Baseline Systems
text: We now describe the baseline systems developed for MultiCoNER II shared task (Fetahu et al., 2023b).

heading: 4.1 BERT
text: We tried using only pre-trained BERT models for all the languages in the monolingual and multilingual tracks initially. We fine-tuned the pre-trained BERT models available in Hugging Face on the provided NER dataset. The output vectors by the BERT model are then fed to either a Linear Layer or a CRF layer to get the predictions.

heading: 4.2 BERT-Linear
text: The input token sequences were passed to the BERT model, which generated vectors of dimension d, equal to the fixed vector dimension of BERT. These vectors then go through a classifier layer made of two fully connected layers, followed by a softmax normalization to get the predicted tags of the input tokens. The predictions have a dimension of m, where m denotes the total number of unique labels provided for the task.

heading: 4.3 BERT-CRF Layer
text: Analysis of the results obtained from BERT with linear layer shows that it was important to consider the sequence for the NER task. Conditional Random Fields(CRF) (Sutton and McCallum, 2012) is a class of discriminative models that helps in recognizing how tags can change considering the position of a particular token in a sentence. CRF often finds its use in machine learning tasks like Named Entity Recognition, Parts of Speech Tagging, Object Identification, etc. Instead of using the BERT-Linear model, which predicts tags one at a time, we employ pre-trained BERT models with an extra CRF layer to predict NER tags jointly. This helps in avoiding mistakes like predicting tags
like I-PER after B-LOC. Let us consider an input sequence:
X = x1, x2, ...xn
and the predicted labels as
Y = y1, y2, ....yn
. We obtain the token embedding of each of the input sequences, xi in dimension d where d is the dimension of BERT embedding. These embeddings are then passed through a dense layer, where the dimensions are transformed from d to m (m denotes the number of labels/tags present in the dataset). The output of the dense layer is the emission score for all the tokens in the input sequence X . These emission scores are then passed on to the CRF layer, where we calculate the score of the predicted label for each input token.
We calculate the score, following the (Lample Guillaume and Chris, 2016) as follows: Score(X,Y ) = ∑i=n i=0 Ayi,yi+1 + ∑i=n i=1 Pi,yi .Aij denote the transition score of the ith label to the jth label in the CRF layer and P ∈ RnXm denotes the emission scores of the input tokens from the dense layer. We intend to maximize the logarithmic probability of the correct label sequence Y for the input sequence X . log p(Y |X) = Score(X,Y )−log ∑
y′∈Yx
eScore(X,y ′ )
where Yx denotes all possible label sequence for the input sequence X . While decoding, we take the label sequence obtaining the maximum score as the predicted label.
y∗ = argmax y′∈Yx Score(X, y ′ )
Table 3 shows the fine-grained F1 score of all the monolingual tracks obtained by BERT-CRF model. We use this result as the baseline result for our system evaluation.

heading: 5 FTBC
text: This section discusses the FTBC system developed for the MultiCoNER II monolingual tracks. The MultiCoNER II dataset contains simulated errors, including typographical and spelling mismatches. Hence, we devised a two-layered system for the task. The tokens are checked for spelling or typographical errors in the first layer using FastText’s nearest neighbor method. Since (Kumar
et al., 2020) showed that for inflectional languages such as Bangla, Hindi, etc., FastText performs better than Glove (Pennington et al., 2014) and Word2Vec (Mikolov et al., 2013); hence, we used FastText for our framework. We observed that FastText’s nearest neighbor could sometimes return garbage values. To avoid those garbage values, we used Levenshtein distance to calculate the distance between the provided input token and the nearest neighbor prediction by the FastText model. If the Levenshtein distance was ≥ 50, we considered the predicted word from the fastText model as our input token. The output of the fastText model is then passed on to the BERT+CRF model to get the predictions on the input tokens.Figure 1 shows a schematic representation of the FTBC system.

heading: 5.1 Models Description
text: We used the pre-trained embeddings released by models from FastText project (Bojanowski et al., 2017) for all the languages. The corrected sequences are passed through the BERT-CRF layer. Table 1 describes the hyperparameters for the models used, whereas Table 2 contains links to HuggingFace pre-trained models used for the FTBC system. F1 scores of each of the monolingual track using FTBC is reported in Table Table 4.

heading: 6 Results
text: In this section, we quantitatively evaluate FTBC against the baseline models. Table 3 shows the performance of FTBC compared to BERT-Linear and BERT-CRF models, whereas Table 4 shows the FTBC system’s performance in all the monolingual tracks along with the final rank on the test set. FTBC outperforms the baseline models in all the tracks.
Analysis of the output revealed that finegrained categories like PrivateCorp, Scientist, and Symptom were misclassified across all the languages. Additionally, categories like AerospaceManufacturer, OtherPER, and OtherLOC have F1 scores of less than 0.3 for all the languages. Using external knowledge bases or gazetteer lists can improve the F1 score of these classes.
Observations also revealed confusion between the categories Politician and Artist and between MusicalGRP and Artist. Analysis shows this confusion is mainly because the train set classifies the same terms into different tags in different sentences. For example, “John” was classified in Spanish training as B-Politician, B-OtherPer, and B-Artist, leading to misclassification by the models. These instances are also found in other languages, including English, German, Italian, Portuguese, and Swedish. Tale 5 reports the sentences having different tags of “John".

heading: 7 Conclusion
text: We developed FTBC, a FastText-based framework with pre-trained Bert for the MultiCoNER II shared
task. Our system outperforms the BERT-CRF baseline models for all 12 monolingual tracks. FTBC performs well for noisy data, but ambiguous entities affect its overall performance. In the future, we would like to enhance the performance of FTBC using external knowledge bases.

heading: Future Work
text: In the future, we would like to enhance the performance of FTBC using external knowledge bases