Sakura at SemEval-2023 Task 2: Data Augmentation via Translation
Year: 2023

Abstract:
We demonstrate a simple yet effective approach to augmenting training data for multilingual named entity recognition using machine translation. The named entity spans from the original sentences are transferred to the translations via word alignment and then filtered with the baseline recognizer to retain high quality annotations. The proposed data augmentation approach improves the baseline performance of XLM-Roberta on the multilingual dataset.

heading: 1 Introduction
text: Named Entity Recognition (NER) is an essential task in natural language processing (NLP) that involves identifying and classifying entities such as people, organizations, locations, and more recently, complex entities such as events, creative works, diseases. Multilingual Complex Named Entity Recognition (MultiCoNER) shared task (Malmasi et al., 2022a,b) is organized to promote research in multilingual NER for complex entities. MultiCoNER II (Fetahu et al., 2023b) features one multilingual track and twelve monolingual tracks for the following languages: English, Spanish, Hindi, Bangla, Chinese, Swedish, Farsi, French, Italian, Portuguese, Ukrainian, and German. The datasets contain fine-grained tagsets that mainly cover three domains (Wiki sentences, questions, and search queries) and include six top-level tags: Location (LOC), Creative Work (CW), Group (GRP), Person (PER), Product (PROD), and Medical (MED).
Complex named entities are highly ambiguous and can take the form of any linguistic constituency: “Sir” (Creative-Work/Visual-Work), “Catch Me If You Can” (Creative-Work/Visual-Work), “Jeremy Corbyn” (Person/Politician). Thereby, making a granular distinction between entity classes (e.g., Politician vs. Scientist) and identifying rare entities solely from the context are challenging problems. To alleviate this challenge, we propose to
augment multilingual training data with translations, which we believe can help the NER model to disambiguate more entities by learning from diverse and noisy contexts. Table 1 shows a few examples of such translations with the aligned entity annotations.
The proposed augmentation strategy involves three steps: translating training example, transferring entity annotations from the source sentences to the target, and filtering out low-quality annotations. To produce translations, we use publicly-available mBART-50 (Tang et al., 2020). To transfer entity annotations, we compute the token alignment between the source and its translation using SimAlign (Jalili Sabet et al., 2020) and then build an entity alignments on top of the token alignments. To ensure high-quality annotations, we use XLMRoBERTA (XLM-R) (Conneau et al., 2020) finetuned on the original training data to filter out low-quality annotations from the translated data. We find that XLM-R fine-tuned on the augmented dataset outperforms the same model fine-tuned on the original training dataset.

heading: 2 Related Work
text: NER is one of the most popular NLP task and is widely explored across domains: news (Tjong Kim Sang and De Meulder, 2003), biomedical (Settles, 2004), social media (Ritter et al., 2011; Ashwini and Choi, 2014). With the recent advances in representation learning, models such as XLM-R (Conneau et al., 2020) and BERT (Devlin et al., 2019) achieve the state-of-the-art NER performance with minimal feature-engineering efforts. In this work, we use XLM-R as our base model for fine-tuning on the NER task.
Data augmentation via translation has been widely used. An example of this is backtranslation (Sennrich et al., 2016), a methodology of obtaining parallel data by translating sentences from the target language to the source language.
1718
Although such data augmentation approaches may introduce noise, increasing the lexical diversity has been effective (Vanmassenhove et al., 2019; Poncelas et al., 2019; Burchell et al., 2022). Translation have been successfully applied to cross-lingual NER problem under unsupervised transfer settings with limited resources (Xie et al., 2018; Mayhew et al., 2017). Schäfer et al. (2022) use machine translation and word alignment techniques to transfer inferred named entity annotations from a rich resource language to a low resource language. We draw inspiration from these works and employ the same approach to augmenting the training data for NER, in addition we assume that we have enough data in the target language to train a filtering model.

heading: 3 Data Augmentation via Translation
text: We employ mBART-501 (Tang et al., 2020) to generate translations for every sentence in the original dataset (Fetahu et al., 2023a). mBART-50 is trained on 50 languages including the 12 languages of the shared task. We pivot translations via English, as we generally observe that pivoting produces better translations than the direct approach, e.g., Hindi → English → Chinese vs. Hindi → Chinese. The translations are generated from every source language to every target language except English, as our focus is on languages with less resources.
After generating the translations we align the tokens using SimAlign (Jalili Sabet et al., 2020). Through the token alignment we transfer the entity annotations from the source sentences to their translations. Although the word order varies in each language, name entities comprise mostly indivisible spans in different languages, which makes transfer generally feasible. To align the entity boundaries, we collect all the target tokens aligned to any of the tokens in the source entity and confirm the transfer only if the target tokens form a continuous span in the target sentence (see Figure 1).
Although mBART-50 supports all the task lan-
1https://github.com/facebookresearch/fairseq/ tree/main/examples/multilingual#mbart50-models
guages, the translation quality is not guaranteed especially for the low-resource languages, besides the alignment quality also contributes to the compounding error. Thus, to ensure the quality of the augmented dataset, we first retain the translations that only have matching number of entities with the sources. Then, we apply the baseline NER model fine-tunted on the original dataset to the translated data and retain the sentences that agree with baseline on the number of entities and their boundaries. Table 2 shows the data stats before and after augmentation and filtering. By adding the translations, we can increase the size of the data between 30K and 102K sentences depending on the language. Note that in the augmented dataset, the artificiallygenerated sentences are the majority.

heading: 4 Named Entity Recongition
text: We formulate Named Entity Recognition (NER) as a sequential labelling task, where the goal is to assign each token in a given text a label that corresponds to its entity type. For example, in the sentence "John Smith works at New York", the label "B-PER" would be assigned to "John", the label "I-PER" would be assigned to "Smith", the label "B-LOC" would be assigned to "New", and the label "I-LOC" would be assigned to "York".
The sequential labeling is a well-studied tasks and enables us to easily leverage standard model architectures. We use XLM-Roberta (Conneau et al., 2020) as our baseline model. As the premise of our exploration is the data augmentation strategy only, we make no modifications to the XLM-R architecture and use it as-is for fine-tuning on the data variants.
Table 3 shows the macro measures (precision, recall, and F1) computed on the multilingual test set (Fetahu et al., 2023a) for the model trained with the original data and after our augmentation. The data augmentation via transation improves the baseline by 0.85 F1. Table 4 shows fine-grained results of the F1 of each named entity class for the model trained on the original and augmented data. We see
that the F1 of the augmented model is higher for most of the name entities. Only for four of entities, i.e., HumanSettlement, Station, MusicalWork, Food show marginal deterioration in the performance.
We fine-tune XLM-R on a machine with 8 x NVIDIA A100 80Gb with effective batch size of 1024, learning rate of 2 · 10−5. Every model is trained for 250 epoch using the cross entropy loss. During the inference malformed tag sequences are removed. The output of the final model is submitted for evaluation.
Table 5 show the results of our proposed approach in each track.2 This table includes not only the overall F1 scores but also a fine-grained evaluation on the clean and noisy subsets. In the column Rank Status we also indicate the position our system in each track.
2Details of the final ranking for all the 13 tracks can be found in https://multiconer.github.io/results

heading: 5 Conclusion
text: In this work we have shown that data augmentation via translation is an useful approach to improve the performance of NER models. On the multilingual test set we observe an improvement over the baseline XLM-R by 0.85 F1. Our approach uses open-source components and can potentially benefit NER in various domains.
In the future we plan to explore how different translation configurations can boost NER models. Reducing the noise of augmented sentences should have a positive impact on the performance. This can be achieved by improving the performance of the translation model. Alternatively, the dataset could be augmented with translations from several models. Therefore, using a selection of sentences from different sources (Poncelas and Way, 2019; Soto et al., 2020) to augment the data could also be investigated.

heading: Future Work
text: In the future we plan to explore how different translation configurations can boost NER models. Reducing the noise of augmented sentences should have a positive impact on the performance. This can be achieved by improving the performance of the translation model. Alternatively, the dataset could be augmented with translations from several models. Therefore, using a selection of sentences from different sources (Poncelas and Way, 2019; Soto et al. , 2020) to augment the data could also be investigated