silp_nlp at SemEval-2023 Task 2: Cross-lingual Knowledge Transfer for Mono-lingual Learning
Year: 2023

Abstract:
Our team silp_nlp participated in SemEval2023 Task 2: MultiCoNER II. Our work made systems for 11 mono-lingual tracks. For leveraging the advantage of all track knowledge we chose transformer-based pretrained models, which have strong cross-lingual transferability. Hence our model trained in two stages, the first stage for multi-lingual learning from all tracks and the second for fine-tuning individual tracks. Our work highlights that the knowledge of all tracks can be transferred to an individual track if the baseline language model has crosslingual features. Our system positioned itself in the top 10 for 4 tracks by scoring 0.7432 macro F1 score for the Hindi track ( 7th rank ) and 0.7322 macro F1 score for the Bangla track ( 9th rank ).

heading: 1 Introduction
text: Multilingual Complex Named Entity Recognition) (Fetahu et al., 2023b) targets the recognition of complex named entities in multilingual and multidomain texts, encouraging researchers to develop new approaches to extract diversified entity types. The semantic structure of data (Fetahu et al., 2023a), which is also provided by organizers for training and validation and testing purposes, has six types of coarse-grained entities: person, location, group, product, medical and creative work with more than 30 fine-grained entities of these 6 coarse-grained entities. The dataset was provided for 12 mono-lingual tracks, and, in addition, this task also provided multi-language data encouraging the development of more generic and adaptive systems. NER task is used in many other NLP tasks. Our team participated in 11 mono-lingual tracks ( Hindi, Bangla, English, German, Spanish, French, Italian, Portuguese, Swedish, Ukrainian, and Chinese ). Our system scores 7th rank in Hindi and is secure in the top 10 in 4 languages (Fetahu et al., 2023b).
Our system leveraged the advantage of crosslingual learning through multi-stage training similar to (Wang et al., 2022) and (He et al., 2022). For this in the first stage, we have selected Crosslingual pretrained models which cover a group of languages of our task and we fine-tune these pretrained language models for a combined dataset of all tracks in which languages are covered by these language models for 10 epoch. It creates a model checkpoint which has knowledge of multiple languages for our task. In the second stage, we fine-tuned the model checkpoints which are generated in the first stage for each track individually. For the Hindi and Bangla tracks our best model was achieved by Muril large model (Khanuja et al., 2021) which is combinedly fine-tuned for three tracks (English, Bangla, Hindi) in the first stage and thereafter the model checkpoints fine-tuned for Hindi and Bangla. For the best model for all other languages, we fine-tuned xlm-r (Conneau et al., 2020) for all the tracks combinedly in the first stage and in the second stage fine tuned all mono-lingual tracks.
In addition to multilingual, there were two key challenges in this year’s task, fine-grained entity taxonomy with over 30 different classes, and noisy testing data which is generated by adding simulated errors to the test set. Our best prediction model is available here1.

heading: 2 Related Work
text: Transformer-based language models like Bert (Devlin et al., 2019) shows the competitive result for several NLP tasks like NER but the problem of low-context situations, long-tail entities, emerging entities and complex entities outlined by the Meng et al. (2021) hard to handle by these models since these models generate contextual word embeddings and pretrained on existing data only.
1https://huggingface.co/sumitrsch
1183
The problem of emerging entities for Web queries in multi-Lingual Code-Mixed context addressed by (Fetahu et al., 2021). These challenges also occur in (Malmasi et al., 2022b) and (Fetahu et al., 2023b). (Fetahu et al., 2021) and Meng et al. (2021) integrated a gazetteer with language models as a solution for these challenges. There are some challenges with the use of a gazetteer, one of them is gazetteer should cover the entities in a sufficient ratio and also for new domain and entity types a specific gazetteer is required, which is subject to the availability of the gazetteer. Another challenge described in Meng et al. (2021) is weak learning of the model due to the model knowing the entity type in advance for the entities, which are covered by the gazetteer. Our work is the continuation of the multilingual NER task started in 2022 (Malmasi et al., 2022b), which has only 6 entities present for 11 monolingual tracks with the multi-language and codemixed track (Malmasi et al., 2022a). The Challenge of (Malmasi et al., 2022b) was to predict complex entities like Creative works with low context for the multi-lingual track. One of the leading approaches to generate a system for (Malmasi et al., 2022b) the problem is based on a Knowledge-based System (Wang et al., 2022) for lower context examples by augmentation of text from Wikipedia. (Chen et al., 2022) also performed well for all tracks by integrating language models like BERT (Devlin et
al., 2019) with a gazetteer network by minimizing the KL divergence. (He et al., 2022) and (Wang et al., 2022) apply multi-stage training for leveraging the advantage of multi-lingual tracks. (He et al., 2022) also used Ensemble architecture and data augmentation for making a more robust model for noisy environments. (Singh et al., 2022) fine tuned muril-large and achieve 0.69 and 0.59 F1-Scores for Hindi and Bangla tracks.

heading: 3 Data
text: This shared task provide 11 mono-lingual track and 1 multi-lingual track (Fetahu et al., 2023a) to encourage NLP researchers to develop NER systems. All tracks have split training, development and testing data however testing data is provided after the final ranking is done. The size of training, development and testing data of all tracks are tabulated in Table 1.
Along with complex entities like creative works, there are 6 coarse-grained entities and more than 30 fine-grained entities are there for all tracks, which are challenging as they are harder to recognize. Organizers (Fetahu et al., 2023a) provided around 9700 sentences only for training for Hindi, Bangla, Chinese and German tracks, this made these tracks more challenging than other tracks as for other mono-lingual tracks around 16500 training data were available.

heading: 4 Methodology
text: All track data are divided into training and validation data by the task organizer. Similar to (He et al., 2022) and (Wang et al., 2022) our system selected strong cross-lingual transfer models ( XLMR (Conneau et al., 2020), pre-trained on more than 100 languages and Muril (Khanuja et al., 2021), pre-trained on all Indic and English languages ) for two-stage training as the base model. In the first stage, we performed multi-lingual training in which training data of all tracks were utilized in selected language models. Figure 1 showed that in the first stage, our model utilized the annotations of all the tracks. This experiment was done with
different hyperparameters for 10 epochs and the best multi-lingual checkpoint was selected based on average loss on validation data. Thereafter in the second stage, we use the fine-tuned multilingual checkpoint of the first stage as the initial model for fine-tuning each mono-lingual track. In the second stage, training of each mono-lingual track is performed with different hyperparameters and we select the final model based on the loss of validation data of the corresponding track. As shown in Figure 1, first of all, the model tokenizer tokenizes the input sentence further preprocessing is done like (Singh et al., 2022). model_max_length of tokenizers set to 92 for better GPU utilization. Thereafter language model generates word embeddings for each token. A linear layer is applied on each token embedding that projects this embedding into logits, a vector of 67 dimensions since the total unique annotations in the dataset are 67. Further, for classification, we have used the cross-entropy loss function.

heading: 5 Experimental setup
text: Our best score was achieved with Muril setup for the Hindi and Bangla track and the other track’s best score was achieved with the XLM-R setup. Detail about both the setup defined in the next subsection. Implementation of our task with the Cross entropy classification layer is done by xxxtokenclassificaion class defined in (Wolf et al., 2020), where xxx refers to the selected model.

heading: 5.1 Hindi and Bangla track
text: Muril model selected for Hindi and Bangla track referring to (Singh et al., 2022). Muril (Khanuja et al., 2021) pretrained on text in 17 languages which include English, Hindi and Bangla with explicitly augmented monolingual text corpora with translated and transliterated document pairs. Therefore for this setup, in the first stage, we combined the English, Hindi and Bangla track datasets for training and validation. Based on validation error after fine-tuning for 10 epochs, Muril multi-lingual checkpoint was constructed. Further, in the second stage, Muril multi-lingual checkpoint is utilized for fine-tuning three mono-lingual tracks separately. Again best model was selected by validation loss after 20 epochs of training. We trained the model in both stages with 1e-6, 5e-6 and 1e-5 learning rates and 16, 32 and 64 batch sizes. Hyperparameters for the best model showed in Table 2.

heading: 5.2 Other mono-lingual track
text: In this setup XLM-R (Conneau et al., 2020) was selected as the baseline model since this model has strong cross-linguality transferability over more than 100 languages and it covers all mono-lingual tracks. Similar to the previous muril setup, this setup was also done in two stages. In the first stage, all tracks are combinedly and trained for 10 epochs and the xlm-r multilingual checkpoint is constructed. In the second stage xlm-r multilingual checkpoint is utilized for fine-tuning all individual mono-lingual tracks for our task. We trained mode in both stages with 1e-6, 5e-6 and 1e-5 learning rates and 16,32 and 64 batch sizes. Hyperparameters for the best model for this setup are shown in Table 2 .

heading: 6 Results and Analysis
text: Our best macro F1 score with official rank for 11 tracks is tabulated in Table 4. Our system scored 7th rank for the Hindi track with 74.32% F1 score and 9th rank for the Bangla track with 73.22% F1 score. Both the scores were achieved by muril setup. For all the other tracks best score was
achieved by xlm-r setup. Our model vs baseline We have also finetuned XLM-R (Conneau et al., 2020) for all tracks and Muril (Khanuja et al., 2021) for Hi, Bn and En tracks individually with the hyperparameters tabulated in Table 2. We considered these finetuned checkpoints as baseline models. Comparison between our models with baseline models is shown in Table 3. The improvement over the baseline model evidently shows that our two-stage strategy learns knowledge of multi-tracks in the first stage and transfers this learning during the second stage of training of the monolingual track.
Muril setup vs xlm-r setup for Hindi, Bangla and English Results shown in Table 3 indicate that Muril setup is better than xlm-r setup for the Hindi and Bangla track while for the English track xlm-r setup performs better compared to the Muril setup.
Clean vs noisy test data Table 6 showed that our system performs lower on noisy data compared to clean data. This might be possible because training data was structured data and we didn’t augment other data for robust training.
Fine-grained vs coarse-grained Table 4 showed that the macro F1 score for coarse-grained classes is better than for fine-grained classes. This infers that the model mispredicted the finegrained classes while correctly predicting the
coarse-grained classes for some inputs. Entity-wise evaluation Entity-wise F1 score for all tracks tabulated in Table 7. Table 5 showed the average of entity-wise F1-score for 11 monolingual tracks. The average F1 score of entity HumanSettlement is the best (0.8814) among all other entities, this entity is part of the Location entity which showed good results in the previous year’s multiconer task but it is also has observed that the OtherLOC entity has 0.5768 F1 score and it is also part of Location Entity. With empirical results, we can infer that the model is confused during the prediction of OtherLOC with other Location entities.

heading: 7 Conclusion
text: In this paper, we performed multi-lingual track knowledge utilization for mono-lingual track for the MultiCoNER II shared task. Our team achieved the top 10 on 4 tracks. We show that the performance of monolingual models can be enhanced by two-stage fine-tuning by allowing them to learn from the training data of all the languages. Also, we have compared two cross-lingual language models, Muril and XLM-R for multi-lingual knowledge transferability for the Hindi, Bangla and English track of this task. Muril performed better for Hindi and Bangla tracks although XLM-R performed better for the English track. For future work, we plan to explore more language models for utilizing their knowledge for NER and other NLP tasks.

heading: Future Work
text: For future work, we plan to explore more language models for utilizing their knowledge for NER and other NLP tasks