VBD_NLP at SemEval-2023 Task 2: Named Entity Recognition Systems Enhanced by BabelNet and Wikipedia
Year: 2023

Abstract:
We describe our systems participated in the SemEval-2023 shared task for Named Entity Recognition (NER) in English and Bangla. In order to address the challenges of the task, where a large number of fine-grained named entity types need to be detected with only a small amount of training data, we use a method to augment the training data based on BabelNet concepts and Wikipedia redirections to automatically annotate named entities from Wikipedia articles. We build our NER systems based on the powerful mDeBERTa pretrained language model and trained on the augmented data. Our approach significantly enhances the performance of the fine-grained NER task in both English and Bangla subtracks, outperforming the baseline models. Specifically, our augmented systems achieve macro-f1 scores of 52.64% and 64.31%, representing improvements of 2.38% and 11.33% over the English and Bangla baselines, respectively.

heading: 2 Dataset
text: We present the data statistics for English and Bangla in Table 1.
Compared to the Bangla subset, the English subset has almost twice as many sentences in the training phase. As a result, the English subset has more distinct entities and a larger vocabulary than the Bangla subset (19,466 versus 6,081 in terms of the number of distinct entities; 34,140 versus 22,274 in terms of vocabulary size, respectively). On average, each sentence consists of approximately 13 to 15 tokens, which is similar for both subsets. In addition, more statistics about the fine-grained taxonomy of the train, dev, and test sets of both subtasks are shown in Table 8 in Appendix A.
On the other hand, the test sets for both languages are extremely large in comparison to the train and dev sets. The reason for this, as stated in MULTICONER (Malmasi et al., 2022a), is that the organizers want to evaluate systems’ ability to generalize to unseen and complex entities, as well as their performance on cross-domain adaptation tasks.

heading: 3 Systems Description

heading: 3.1 Baseline
text: We use pretrained language model mDeBERTaV3 (He et al., 2021) as the encoder. A sentence tokenized by the byte-pair-encoding algorithm (Sennrich et al., 2016) is feeding into the encoder to extract the representation of tokens. Then they passed through a pooling layer to get the representation of words. The BiLSTM-CRF architecture (Huang et al., 2015) is used to enhance the feature of words before projecting to n-labels-dimension embedding and predicting the label of the sequence.
The AdamW optimizer (Loshchilov and Hutter, 2017) is used to optimize the objective function. In addition, we split the parameters of model into two groups before feeding into optimizer. The first one contains the parameters of pretrained encoder with the use of a small learning rate. The second one contains the rest of parameters in the model which are tuned by using a larger learning rate.

heading: 3.2 Knowledge Based Enhanced Systems
text: To begin with, we use the provided dataset as seed samples to extend the data using the method based on MultiNERD, which we present in detail in Section 4. The enhanced data is then combined with the original dataset for training the model.

heading: 4 Knowledge Based Enhanced Data

heading: 4.1 Data Augmentation Using Provided Entities
text: In order to expand the training set of the English subset, we retrieved all the entities directly from the sentences to obtain their corresponding Wikipedia articles. As shown in Figure 7, we used BabelNet to retrieve the corresponding articles. For ambiguous entities with multiple possible articles or no associated article, we either discarded them or kept only the first article, respectively. After obtaining the article names, we used the Wikipedia1 package to extract the article texts, which were then segmented into sentences, and only the sentences that contained the corresponding entities were kept.

heading: 4.2 Data Augmentation Using BabelNet
text: Following the approach proposed in (Tedeschi and Navigli, 2022), we manually curated 1,254 synsets to encompass a wide range of high-level concepts
1Version: 1.4.0, https://pypi.org/project/ wikipedia/
based on the task’s taxonomy. Examples of some of these synsets are provided below:
• SCIENTIST: bibliotist, linguist, physicist, etc
• DISEASE: horse disease, seafood poisoning, etc
• PRIVATE CORP: public limited company, public corporation, etc
To expand the initial set of concepts, we utilized BabelNet’s hyponymy (has-instance) relationships to obtain child synsets. For example, for the ’linguist’ concept (bn:00051385n), we retrieved its children synsets, such as Noam Chomsky2 (bn:00000162n) and Leonard Bloomfield3 (bn:00011432n), which inherited the SCIENTIST label.
We followed a similar approach for Bangla, but we did not annotate high-order concepts in Bangla. Instead, we used the English-Bangla mechanisms of BabelNet to obtain children synsets. In total, this process yielded 120k children synsets for English and 113k for Bangla.

heading: 4.3 Wikipedia Redirections
text: Apart from using BabelNet synsets, we also leveraged another resource, namely Wikipedia redirection links, to expand our concept list. For example, the term Apple (corporation) can be redirected to Apple Inc., even though the terms have different surface text, they refer to the same Wikipedia page.

heading: 4.4 Annotation Enhancement
text: Figure 2 illustrates our methodology for improving the quality of our augmented data. This methodology was inspired by the method proposed in (Tedeschi and Navigli, 2022). The augmented data, which is a combination of additional data and the provided data, was divided into n non-overlapping
2https://en.wikipedia.org/wiki/Noam_Chomsky 3https://en.wikipedia.org/wiki/Leonard_
Bloomfield
subsets. The first subset (D0) was used to train a self-annotating model, which is a Transformerbased neural classifier (mBERT + Bi-LSTM + CRF, (Mueller et al., 2020)), to obtain an initial model (M0). The initial model is used as the first version for the self annotating loop, which proceeds as follows:
1. The first subset (D0) is concatenated with another subset (Dj , j ∈ {1, 2, ..., n}) to form a larger set (DL).
2. The self-annotating model (Mk, k ∈ {1, 2, ..., t}, in which t = 0 denotes the initial model, while t represents the number of loops) annotates the larger set (DL). During this annotation process, if the NER class assigned by the self-annotating model for a predicted
entity is different from the one assigned by the data augmentation process (Section 4.4, 4.2, and 4.1), the corresponding sentence is removed to obtain a noise-reduced set (D′L).
3. D′L is then used to combine with the next subset in the queue in the first step of the next loop instead of the first subset (D0).

heading: 4.5 Augmented Data statistics
text: The augmentation method we applied significantly improved the training sets for both English and Bangla, as demonstrated in Table 2. With the aid of additional data, the total number of training instances for English increased from 16,788 to 419,483, and from 9,708 to 22,216 for Bangla. Furthermore, our method increased the vocabulary size by nearly 6.5 times in English and 3 times in Bangla, indicating that it not only enriched the context of existing entities in the training sets, but also provided new contexts for previously unseen entities. These findings highlight the effectiveness of our augmentation method in improving the quality and size of the training sets. Such improvements can have significant implications for the development of better entity recognition models (see Section 5 for more details).

heading: 4.6 Experimental Settings
text: Our experiments were conducted on hardware with a single RTX 2080Ti GPU. The training process was set up with the following hyper-parameters: a batch size of 8, a dropout rate of 0.3, learning rates of 0.001 and 0.00005 for two parameter groups (as mentioned above), and a default of 10 training epochs. The best checkpoint was chosen based on the f1-macro scores on the dev set during training.

heading: 4.7 Results
text: We present our results in Tables 3 and 4. In overall, our augmented systems significantly improve
the baseline systems in all the metrics, which are precision, recall, and F1-macro.

heading: 5 Results and Analyses
text: In particular, the results on English presented in Table 3 show that the precision, recall, and F1-score of the baseline model were 51.64%, 51.29%, and 50.26%, respectively, while these numbers were 57.89%, 51.36%, and 52.64% for the augmented model. The F1-score saw a moderate improvement as a result of the augmentation technique. The precision ratings of the two models, however, did not differ significantly from one another. As a result of having a slightly higher recall score than the baseline model, the augmented model was able to recognize more pertinent instances. The model’s performance may be affected by several factors, including the task’s complexity and the quantity and quality of the training data. In this instance, it seems that the augmentation technique was advantageous for enhancing the model’s capacity to to identify relevant instances, resulting in an improvement in the overall F1-score.
Meanwhile, the augmented model improve all scores by almost 10±2% in all scores for the Bangla results (Table 4). The augmentation technique had a significantly positive impact on all three metrics, greatly enhancing the performance of the model. The precision of the augmented model increased by 10.18%, demonstrating that it was able to recognize more relevant instances while fewer false positive predictions were made. Additionally, the recall score of the augmented model increased by 8.21%, showing that it was able to extract more relevant instances from the dataset. The augmented model’s improved overall performance is evidenced by the F-1 score’s increase.

heading: 5.1 Analyses
text: Table 5, 6, 9, and 10 compare the baseline system’s results to augmented system’s results on the finegrained taxonomy of the dev and test set of English and Bangla languages.
In the Bangla language, the augmented system using additional data (see Section 4) exhibits significantly better performance than the baseline model in most classes. For example, in the dev set (Table 5), the F1-scores of classes such as Scientist are 33.33% and 83.33%, OtherLOC are 47.62% and 95.65%, Food are 61.11% and 87.50%, Software 81.48% and 94.12%, among others. Similarly, in the test set (Table 9), examples include SportManager with scores of 15.93% and 50.64%, Symptom with scores of 62.41% and 81.06%, and PrivateCorp with scores of 50.81% and 78.95%. However, the augmented model has worse performance than the baseline model in some classes, namely Cleric, Drink, Vehicle, AnatomicalStructure, Facility, and
ArtWork.
In contrast to the Bangla language, the Baseline model in English outperforms the Augmented model in a greater number of classes (Table 6 and 10). Specifically, out of a total of 33 classes, the Baseline model performs better than the Augmented model in 14 classes in the dev set and 15 classes in the test set.
There are a number of possible explanations for why the Baseline model outperformed the aug-
mented model in English while the augmented model performed better in Bangla. The quantity and caliber of additional data used for augmentation could be one factor in the explanation. The augmentation method in Bangla was successful in obtaining a larger and more varied set of data, which might have added more context and enhanced the model’s capacity for entity recognition and classification. In contrast, it is possible that the complexity and nuance of the English language prevented
the augmentation of English language data from having as much of an impact on the model’s performance. The structure and characteristics of the languages themselves might also have been important. For example, Bangla language has a more rigid grammatical structure and less variation in word order, which could make it easier for the model to identify and classify entities. In contrast, the English language has a more complex grammatical structure and greater variation in word order, which could make it more difficult for the model to accurately identify and classify entities.
Prediction samples Table 7 shows the varying performance of the baseline and augmented models. In the English subtask, the baseline model performs worse in all cases, but still able to predict some entities. In contrast,the augmented model shows a consistent performance in predicting the correct entity boundaries, albeit with some missed entities, indicating that the augmentation did not significantly improve the performance in this task. In particular, all samples shows a clear weakness of the baseline model in identifying the correct boundaries and classes of the entities. Whereas the augmented
model shows an improvement in recognizing all the entities except the second entity in the first sample.In the Bangla subtask, the baseline model’s performance is significantly worse than in the English subtask, failing to recognize the first entity in both samples. The augmented model shows better performance, correctly identifying both entities in Sample 1. However, in Sample 2, the augmented model shows a concerning behavior, predicting an additional wrong entity.
Overall, the results suggest that the augmentation has a limited impact on improving the baseline model’s performance in the both subtasks, as the two models struggle to identify the correct boundaries of the entities.
Limitations and Future work. For this work, we only evaluated on English and Bangla, a low resource language. We intend to extend our experiments on other languages. Additionally, different NER models can be applied to extract more complex entities such as nested entities using exhaustive methods (Sohrab et al., 2019a,b; Wang et al., 2022). Filtering noisy annotations is also an important task in future work.

heading: 6 Conclusion
text: In this paper, we have introduced our NER systems for the SemEval-2023 Task 2: MultiCoNER II Multilingual Complex Named Entity Recognition. Our NER systems are finetuned on the mDeBERTa pretrained language model and enhanced by augmented data from BabelNet and Wikipedia. Specifically, BabelNet concepts are utilized to annotate named entities from Wikipedia articles. Experimental results show that the augmented data significantly improve baseline systems trained on the limited training data, especially on the lowresource language such as Bangla. For future work, we plan to improve the quality of augmented data by filter noisy annotations as well as extend our systems on other languages.

heading: A Taxonomy statistics

heading: B Test Results Analysis

heading: Future Work
text: For future work, we plan to improve the quality of augmented data by filter noisy annotations as well as extend our systems on other languages