Frustratingly Easy Label Projection for Cross-lingual Transfer
Year: 2023

Abstract:
Translating training data into many languages has emerged as a practical solution for improving cross-lingual transfer. For tasks that involve span-level annotations, such as information extraction or question answering, an additional label projection step is required to map annotated spans onto the translated texts. Recently, a few efforts have utilized a simple mark-then-translate method to jointly perform translation and projection by inserting special markers around the labeled spans in the original sentence (Lewis et al., 2020; Hu et al., 2020). However, as far as we are aware, no empirical analysis has been conducted on how this approach compares to traditional annotation projection based on word alignment. In this paper, we present an extensive empirical study across 57 languages and three tasks (QA, NER, and Event Extraction) to evaluate the effectiveness and limitations of both methods, filling an important gap in the literature. Experimental results show that our optimized version of mark-then-translate, which we call EasyProject, is easily applied to many languages and works surprisingly well, outperforming the more complex word alignment-based methods. We analyze several key factors that affect the end-task performance, and show EasyProject works well because it can accurately preserve label span boundaries after translation. 1

heading: 1 Introduction
text: Zero-shot cross-lingual transfer, where models trained on a source language (e.g., English) are directly applied to other target languages, has the potential to extend NLP systems to many languages (Nooralahzadeh et al., 2020; Keung et al., 2020; Chen and Ritter, 2021; Niu et al., 2022; Huang
1Our code and data is available at: https://github. com/edchengg/easyproject
et al., 2022a). Yet, its performance still lags behind models that are directly fine-tuned on labeled data (if available) from the target language. Recent work has shown that combining training data in a source language together with its automatic translation to the target language leads to consistent performance improvements (Xue et al., 2021; Hu et al., 2020). However, for NLP tasks that involve span-level annotations, an additional label projection step is needed to map the span annotations onto the translated texts (see Figure 1).
Traditionally, this annotation projection step is performed based on word alignment after machine translation (Akbik et al., 2015a; Aminian et al., 2019). To avoid the use of complex word alignment models, several recent efforts (Lewis et al., 2020; Hu et al., 2020) directly translated sentences with span annotations wrapped between special markers (e.g., <a> and </a>). However, due to limited analysis presented in prior work, it is unclear (1) how well this approach works across different language families, (2) how robust MT systems are in handling special markers, as inserting markers inevitably degrades the translation quality, and (3) how well marker-based projection works in comparison to traditional alignment-based methods.
In this paper, we present the first systematic study of the mark-then-translate annotation projection technique, which includes careful evaluation of the choice of markers, projection accuracy, impact on translation quality, robustness to different MT systems, as well as a comparison to traditional alignment-based method across 57 languages (including 18 from Africa) on 5 datasets and 3 NLP tasks. We also propose an improved variant of marker-based projection, EASYPROJECT, that consistently outperforms the alignmentbased approach, while being incredibly easy to use to project a variety of annotations (QA, entities,
5775
relations, events) across many languages. The key is to use language-agnostic square bracket markers, combined with an efficient fine-tuning strategy to encourage the multilingual MT system to better preserve the special markers during translation.
Our main findings include (1) the marker-based method is surprisingly robust across different translation systems and languages, but the choice of markers matters (§3.2); (2) EasyProject can project annotated spans more accurately and is better at preserving span boundaries than the alignmentbased methods, which is key to its success (§5.1); (3) fine-tuning an MT system for only 200 steps is sufficient to improve its robustness in handling special markers during translation (§4); (4) the margin of improved cross-lingual transfer is related to the language/script family and amount of pre-training data included in the multilingual model (§5.2). We hope our work will inspire more research on robust models that better handle text markup for the purpose of generating span annotations.

heading: 2 Background and Related Work
text: Alignment-based Projection. Projecting annotations via word alignment typically consists of the following steps: machine translate the available training data into the target language; run word alignment tools on the original and translated sentences; and finally, apply heuristics to map the span-level annotations from the original to translated texts. Statistical word alignment tools such as GIZA++ (Och and Ney, 2003) and fastalign (Dyer et al., 2013) have been widely adopted for projecting part-of-speech tags (Yarowsky et al., 2001; Eskander et al., 2020), semantic roles (Akbik
et al., 2015b; Aminian et al., 2017; Daza and Frank, 2020; Fei et al., 2020), slot filling (Xu et al., 2020), semantic parser (Moradshahi et al., 2020; Nicosia et al., 2021), and NER labels (Ni et al., 2017; Stengel-Eskin et al., 2019). Recent progress on supervised neural aligners (Jalili Sabet et al., 2020; Nagata et al., 2020; Dou and Neubig, 2021; Lan et al., 2021) and multilingual contextualized embeddings (Devlin, 2018; Conneau et al., 2020) has further improved alignment accuracy. However, this pipeline-based method suffers from error propagation, translation shift (Akbik et al., 2015b), and non-contiguous alignments (Zenkel et al., 2020). Our analysis in §5.1 shows that the alignmentbased methods are more error-prone when projecting span-level annotations, compared to the marker-based approaches.
Marker-based Projection. A few efforts have used mark-then-translate label projection method to translate question answering datasets into other languages (Lee et al., 2018; Lewis et al., 2020; Hu et al., 2020; Bornea et al., 2021). However, the focus of these papers was not the label projection task itself and there was no in-depth analysis on the effectiveness of the approach. For instance, Lewis et al. (2020) used quotation marks to translate the SQuAD training set into other languages but did not present empirical comparison to any other label projection methods. Similarly, Hu et al. (2020) used XML tags for the same purpose when creating the XTREME dataset, but this was only briefly mentioned in a few sentences in appendix. Besides QA, MulDA (Liu et al., 2021; Zhou et al., 2022) is a labeled sequence translation method that replaces entities with variable names for cross-lingual NER.
2
However, no comparison with existing projection methods was presented, as the main focus is generating synthetic labeled data using language models.

heading: 3 Analysis of Marker-Based Projection
text: The idea of marker-based label projection is straightforward – wrap labeled spans with special marker tokens, then translate the modified sentence (see an example in Figure 1b). The projected spans can be directly decoded from the translation if the markers are retrained. However, inserting markers inevitably degrades the translation quality. In this section, we analyze several questions left open by prior work (Lewis et al., 2020; Hu et al., 2020), including (1) how well are the special markers being preserved in translation, (2) the impact of different marker choices on translation quality and the performance of cross-lingual transfer.

heading: 3.1 Experimental Setup
text: We conduct experiments on three NLP tasks and 57 languages with five multilingual datasets to comprehensively evaluate the marker-based method. Most multilingual datasets are created by either (1) directly collecting and annotating data in the target language, or (2) translating English data with human or machine translation and then projecting labels manually or automatically. Four of our selected datasets were created with the first method, as evaluation on the translated datasets may overestimate performance on a target language, when in fact a model might only perform well on translationese (Riley et al., 2020).
Datasets. Our experiments include NER via the WikiANN (Pan et al., 2017; Rahimi et al., 2019) and MasahkaNER 2.0 (Adelani et al., 2022) datasets (§5.1), in addition to CoNLL-2002/2003 multilingual NER datasets (Tjong Kim Sang, 2002; Tjong Kim Sang and De Meulder, 2003) for comparison with Liu et al. (2021) (§F.1). For event extraction, we use the ACE05 corpus (Walker et al., 2006), which consists of six sub-tasks: entity and relation extraction, event trigger/argument identification and classification. For QA, we use TyDiQAGoldP (Clark et al., 2020), which contains challenging questions written in eight languages. Data statistics are shown in Table 1 and 10 in Appendix.
IE and QA Models. We use XLM-RoBERTalarge (Conneau et al., 2020) as the backbone model, except where noted.2 For NER and QA, we fine-tune XLM-R with standard tagging and SQuAD-style span prediction layers. For event extraction, we use the OneIE framework (Lin et al., 2020), a joint neural model for information extraction with global features. We report average F1 scores over three runs with different random seeds. More implementation details can be found in Appendix D.
MT Systems. We experiment with two MT systems: (1) the Google Translation (GMT) API,3 and (2) an open-sourced multilingual translation model NLLB (No Language Left Behind) (Costa-jussà et al., 2022) with 3.3 billion parameters, supporting the translation between any pair of 200 languages.4

heading: 3.2 Choice of Markers
text: Ideally, a good span marker should minimize the impact on translation quality while having a high chance of being preserved during translation. However, prior works used quotation marks (“ ”) (Lee et al., 2018; Lewis et al., 2020) and XML/HTML tags (e.g., <a> or <PER>) (Hu et al., 2020; Ahmad et al., 2021) without much justification, which we address below.
Preserved in Translation. In a pilot study, we experimented with several markers, including XML tags, [], “ ”, (), <>, and {}, etc. We found that both MT systems work reasonably well to retain square brackets ([]) and XML markers during the translation across many languages, while other markers that have language-specific formats
2We also experiment with mT5large, mT5XL, and mT5XXL (Xue et al., 2021) on WikiANN-NER (Table 7).
3https://cloud.google.com/translate 4https://github.com/facebookresearch/fairseq/
tree/nllb
3
are easily lost in translation. For example, quotation marks (“ ”) are often translated in a languagespecific way, e.g., «» in Russian, and are sometimes lost entirely in Arabic and Finnish, leading to low projection rates: 53% for Russian, 76% for Arabic, and 79% for Finnish based on TyDiQA dataset. The projection rate is measured by the percentage of data in which the numbers and type of special markers in the translations match with the source sentences. To improve the robustness of MT system in handling markers, we found further finetuning the MT system on synthetic data, where the special markers are inserted around name entities in parallel sentences, for only 200 steps is sufficient to boost the projection rate while maintaining translation quality (more details in §4). Impact on Translation Quality. After narrowing down the choices to XML tags and square brackets, we further measure the impact of adding markers on the translation quality by adopting the evaluation setup used by Fan et al. (2021). We compare translation quality, with and without markers inserted, from English to various target languages using BLEU score. Table 2 presents the experimental results with Google Translation. Examples of errors are shown in Table 3. We find that inserting special markers indeed degrades translation quality, but overall, square brackets have less negative impact compared to XML tags. We hypothesize this is because using [] introduces less number of extra subword tokens in the encoding and decoding of the text during translation, compared to XML tags. More results on 55 languages using the
with GMT. In #1, a necessary Chinese verb “报道 (report)” is lost in the XML-marked translation, while tags (<e>, /e>) are also mismatched due to the word reordering of “记者 (reporter)”. In #2, []-marked translation fails to preserve the square brackets ([]) around the Arabic translation of “WTO” (marked by underline).
NLLB translation system and more details about the evaluation setup can be found in Appendix G.1.
Impact on Transfer Learning. We next evaluate the impact of different marker choices on the performance of cross-lingual transfer. The results on the TyDiQA dataset are presented in Table 4. On average, square brackets ([]) have the best transfer learning performance. We also directly compare with the projection data released by Hu et al., which utilizes XML tags and a Google internal translation system in the year 2020 to translate QA datasets. More results on NER and event extraction tasks, and comparison with the alignment-based projection methods are presented in Table 8.
4
methods on the WikiANN dataset using NLLB translation system. EasyProject (§4) outperforms alignmentbased approach on F1 scores for most languages, though inserting markers degrades translation quality. The experimental setting is detailed in §5.1.

heading: 4 EASYPROJECT
text: Based on our analysis, we develop an optimized version of the mark-then-translate method, which we call EASYPROJECT.5 Our improvements target the two weaknesses of the marker-based approach: (1) special markers may get lost during translation; and (2) although square brackets ([]) show strong performance, they don’t carry the correspondence between original spans and the ones in the translation (e.g., [ Churchill ] was born in [ England ] in [ 1874 ].), as the XML tags (e.g., <a> Churchill </a> was born in <b> England </b> in <c> 1874 </c>.). If multiple annotated spans with different labels exist in one sentence, it is challenging to assign labels to the projected entities in the translation, as word order can change between languages.

heading: 4.1 Fine-tuning NLLB
text: To improve the robustness of the MT system in handling special markers, we further fine-tuned the NLLB model on synthetic data. We utilize parameter-efficient fine-tuning by only updating the last layer of the encoder and decoder, which take 4.2% of all parameters. We found fine-tuning 200 steps is sufficient to improve the projection rate on TyDiQA dataset from 70% to 96.4% while maintaining the translation quality.
Creating Synthetic Data. We first construct a parallel corpus where the special markers are inserted around the corresponding name entities in source and target sentences, with following steps.
5This name was inspired by Daumé III (2007).
1. Detect named entities on the English side of the parallel corpus, using the SpaCy NER system,6 which covers 18 types of NER labels.
2. Translate the English entity names into the target language, and use string matching to find the corresponding entities from the target sentence. Given a pair of entities in the source and target sentences, we add square brackets ([]) around both of them.
3. Select all sentence pairs that contain more than one []-marked entity. We also sort the rest of the data based on length, and include the top-k sentence pairs. In total, we use 5,000 sentence pairs for each language pair.
We utilize the training data of NLLB model7 as the source of the parallel sentences, and use the sentence pairs from high-resource language pairs (en to {de,es,nl,zh,ar}), which are selected based on the CoNLL-2002/2003 ({de,es,nl}) and ACE
6https://spacy.io/models/en#en_core_web_sm 7https://huggingface.co/datasets/allenai/nllb
5
datasets ({zh,ar}).
Parameter-efficient Fine-tuning. To save compute and preserve the translation ability of the model, we only update the weights in the last layer of the encoder and decoder for 200 steps, with a learning rate of 5e-5 and a batch size of 8, which takes around 2 minutes on an A40 GPU. The changes in projection rate and the translation quality during the fine-tuning process are shown in Figure 3. The fine-tuned NLLB model is able to improve the projection rate on TyDiQA from 70% to 96.4%. TyDiQA is particularly challenging due to its relatively long sentences, and the translation model sometimes may ignore the inserted markers. By fine-tuning on high-resource languages, we found the model is able to generalize well to the other language pairs. In pilot study, we notice that fine-tuning on low-resource languages, such as African languages in MasakhaNER corpus, doesn’t generalize well and leads to lower translation quality. We will release all the fine-tuned models.
Fuzzy String Matching. To identify the corresponding labels when more than one projected entity exists in the translation, we design a fuzzy string-matching method. We first translate each annotated span in the original sentence independently, resulting in a set of labeled mentions. To identify labels for the unlabeled spans in the []- marked translation, we compare each unlabeled span with the labeled mentions using the ratio() function in the difflib library. 8 Two strings are considered matched if they have >50% matched subsequences, and the associated label is assigned to the bracketed span. We also experiment with matching span labels left-to-right based on their relative position in the text. The results are shown in Table 5. Using fuzzy string-matching leads to overall better performance since it can assign the span labels more accurately.
Putting all the improvements together, we call this improved variant of marker-based method EASYPROJECT for easily projecting labels.

heading: 5 Experiments
text: In this section, we comprehensively evaluate the effectiveness of EASYPROJECT and analyze the
8https://docs.python.org/3/library/difflib. html#difflib.SequenceMatcher.ratio
key factors that impact the performance of crosslingual transfer learning.

heading: 5.1 Comparison to Alignment-based Method
text: We first compare EasyProject with the traditional pipeline approach based on state-of-the-art bilingual word alignment models. For both methods, we apply a simple filtering rule that removes sentences with different numbers of annotations before and after projection.
Bilingual Word Alignment. We experiment with two state-of-the-art neural word aligners: (1) the unsupervised version of Awesome-align (Dou and Neubig, 2021) and its supervised version, which we extended from 5 to 39 languages for this paper, and (2) QA-align (Nagata et al., 2020) which formulates the word alignment problem as SQuAD-style QA task. More details on the word alignment models can be found in Appendix C.
Transfer Learning Results. As summarized in Figure 2, EasyProject outperforms alignmentbased projection for most languages, even though
6
∆: -10 -5 -1 0 0 +1 +5 +10 .
7
projection methods are marked. For the alignmentbased projection, “克斯” is incorrectly missed. The translations are based on Google Translation.
span markers degrade translation quality. In Table 6 and 8, we show that EasyProject almost always outperforms alignment-based projection on NER, QA, and the more challenging event extraction tasks, when training on a combination of English data and the translated projected data in target languages. In addition, we find that EasyProject generally performs better than using XML tags, as the former has less impact on the translation quality. We also notice the relatively low zero-shot performance in ja, zh, and th on WikiAnn dataset, which is consistent with scores reported in prior literature (He et al., 2021b). We suspect this is due to their distinct script systems, and adding EasyProject data brings significant improvements to all of them – Japanese (+25.5 F1), Chinese (+18.8 F1), and Thai (+14.4 F1). EasyProject (GMT) also improves the performance of mT5large and mT5XL by 7.3 and 5.7 F1 on average across all target languages, and mT5XXL by 2.2 F1 on a subset of 8 languages, as shown in Table 7. Full results of the mT5 model are provided in Table 22 in Appendix.
Accuracy of Projected Annotations. To answer why EasyProject can outperform alignment-based method even though it degrades translation quality, we manually inspect 400 sentences sampled from the WikiANN training set. EasyProject correctly projects 100% and 97.5% of the label spans, when using Google Translation and NLLB, respectively. Whereas the traditional method based on Awesomealign only achieves 97.5% and 93.4% accuracy. We found EasyProject can more accurately preserve the boundaries of the label span. For the alignmentbased method, most errors are caused by partial or missed alignments, as demonstrated in Table 9. More analyses are provided in the Appendix G.2.

heading: 5.2 Size of Pre-training Data vs. Improvement in Performance
text: Figure 4 shows improvements in NER F1 using EasyProject vs. size of data for each language in XLM-RoBERTa’s pre-training corpus. EasyProject provides larger improvements on low-resource languages and languages without whitespaces. For high-resource languages in the Indo-European (e.g., Germanic and Romance) or Uralic families, using projected data struggles to significantly improve over a strong fine-tuning baseline.

heading: 5.3 Transfer from non-English Languages
text: Recent work has suggested that English may not always be the best language to transfer from (Turc et al., 2021). We demonstrate that the markerbased method is not limited to English-centric transfer learning; rather, it can be used for transfer learning from any language to any language provided with the availability of multilingual MT systems. In Figure 5, we show the relative F1 improvements of using EasyProject over fine-tuning on source language only for 9 different languages (81 directions in total), leveraging the multilingual capabilities of NLLB. Fine-tuning models only on source-language data does not work well when transferring to or from Chinese, consistent with observations from Hu et al. (2020). The markerbased method addresses this problem by providing substantial improvements in F1 on the WikiANN dataset for Chinese. Transferring to Arabic and from Russian are also challenging, but again, the marker-based method greatly boosts performance.
8

heading: 5.4 More Experiments and Analyses in the Appendix
text: We also compare EasyProject against MulDA (Liu et al., 2021) (§F.1) and bitext projection (F.2), as well as evaluating it on low-resource languages: Māori and Turkmen (§F.3). In addition, we analyze the two branches of label projection methods from other aspects, including projection rate (§G.2) and translation speed (§G.3). Due to space limits, we present all of them in the appendix. On the data side, we fix a sentence splitting issue for 12 extremely long sentences in the ACE05 Arabic testset (§E ). This issue is noticed by other researchers (Huang et al., 2022b) as well. We will release the improved ACE Arabic dataset to the community.

heading: 6 Conclusion
text: In this paper, we presnet a thorough empirical assessment of various approaches for cross-lingual label projection. We also design an improved variant of the mark-and-translate method, which we call EASYPROJECT. Experiments on 57 target languages and three well-studied NLP tasks show that EasyProject consistently outperforms the alignment-based methods and effectively improves the performance of cross-lingual transfer.
Limitations
While our study shows that EasyProject can effectively translate the source sentences with special markers inserted to the target languages, using the
Google Translation and NLLB model, it is unclear whether all translation models can work well when special markers are inserted. To generalize this approach to future MT systems, we design a simple and computationally efficient approach to improve the robustness of MT systems in handling special markers. However, the translation quality for the marker-inserted text still falls behind the original text. We leave the work of further optimizing the translation quality as future work.

heading: Acknowledgements
text: This material is based upon work supported by the NSF (IIS-2052498) and IARPA via the BETTER and HIATUS programs (2019-19051600004, 2022- 22072200004). The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of NSF, ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

heading: A Data statistics for CoNLL-2002/2003
text: The statistics of the CoNLL-2002/2003 multilingual NER dataset are provided in Table 10.

heading: B Full Results on MasakhaNER2.0
text: MasahkaNER2.0 is a NER dataset in the news domain, including the annotations on 20 African languages. Following the setting in the original paper (Adelani et al., 2022), we use CoNLL-03 dataset (Tjong Kim Sang and De Meulder, 2003) as the source corpus, and train the mDeBERTv3 (He et al., 2021a) model on it. Then the trained model is evaluated on the test set of MasahkaNER2.0, with a focus on the PER, ORG, and LOC types.

heading: C Details of Word Alignment Models
text: Awesome-align. This aligner (Dou and Neubig, 2021), when used in the unsupervised setting, primarily relies on the normalized similarity scores of all word pairs between the two sentences, which are calculated based on pre-trained multilingual word embeddings taken from specific Transformer layers. In the supervised setting, with access to parallel text, Awesome-align can be further improved by fine-tuning towards a set of self-training and language model objectives. We include experiments of both the unsupervised (Awesome) and supervised (Awesomeft) versions of Awesome-align based on multilingual BERT, which has shown to achieve better word alignment results than XLMRoBERTabase. For the supervised version, we finetune an individual Awesome-align model for each of the 39 target languages in WikiANN using parallel sentences sampled from the M2M model’s (Fan et al., 2021) training datasets: CCAligned (ElKishky et al., 2020) and CCMatrix (Schwenk et al., 2021b). Specifically, we randomly sample 200k parallel sentences from the CCAligned corpus for language pairs from English to {te, ka, kk, my, th, yo}, and the rest from the CCMatrix.
We use the codebase9 from Dou and Neubig (2021) with the default softmax configuration to extract alignment. We do not apply the consistency optimization objective when fine-tuning the models because it may trade precision for recall, as suggested in the official instruction written by the authors.
QA-align. This is a state-of-the-art supervised approach (Nagata et al., 2020) that formulates the word alignment problem as a SQuAD-style question answering task by fine-tuning multilingual BERT. Specifically, given a word in the source sentence, the model predicts the aligned span in the target sentence and reconciles both source-totarget and target-to-source directions by averaging and thresholding probabilities. We trained the QAalign model for English to Arabic, German, French, Chinese, and Japanese, where gold annotated word alignment data is available.
We use the codebase from Nagata et al. (2020).10 For the training data of word alignment between
9https://github.com/neulab/awesome-align 10https://github.com/nttcslab-nlp/word_align
14
en and {de, zh, ja, fr}, we use the same data as in Nagata et al. (2020). For en-ar, we use the GALE English-Arabic word alignment data from LDC11, and use 80% of the sentence pairs for training. The data statistics can be found in Table 12.
D Implementation Details of IE models
We follow the same learning rates and number of epochs reported in prior work: Hu et al. (2020) for QA, He et al. (2021b) and Pfeiffer et al. (2020) for NER (the latter for mi and tk) and Yarmohammadi et al. (2021) for ACE. For WikiANN NER (Pan et al., 2017), CoNLL-2002/2003 NER (Tjong Kim Sang, 2002; Tjong Kim Sang and De Meulder, 2003), MasakhaNER2.0 (Adelani et al., 2022), and TyDiQA-GoldP (Clark et al., 2020), we use the codebase from the XTREME benchmark (Hu et al., 2020),12 and MasakhaNER2.0 13, which is based on the Huggingface transformers library (Wolf et al., 2019). The hyperparameters of mDeBERTaV3 (276M) for MasakhaNER2.0 and XLMRoBERTalarge(550M) and for other datasets are presented in Table 13 following (Hu et al., 2020; He et al., 2021b; Liu et al., 2021; Adelani et al., 2022). We report the average result of three random seeds and select models based on the English development set.
mT5 NER Model. Training mT5XXL (Xue et al., 2021) models, which have over 11 billion parameters, for the NER task is computationally challenging. We formulate the WikiANN NER task as generating a sequence of tokens with special entity tags (e.g. <per>, </per>) inserted around the entity spans. To fit the model into GPU memory
11LDC2014T05, LDC2014T10, LDC2014T14, LDC2014T19 LDC2013T10, LDC2013T14, LDC2014T03, LDC2014T08
12https://github.com/google-research/xtreme 13https://github.com/masakhane-io/
masakhane-ner
for training, we freeze the embedding layer and the bottom 12 layers of both encoder and decoder during fine-tuning. We also use the DeepSpeed (Rajbhandari et al., 2020) ZeRo3 with 32-bits configurations. We first fine-tune the model on English data for 20 epochs with a learning rate of 1e-4. To speed up the training process, we initialize the model from the English fine-tuned checkpoint and further fine-tune it on the combination of English and EasyProject + GMT data with a learning rate of 5e-5 for 5 epochs. We report results of mT5large by averaging over three random seeds. We use one random seed for the XL and XXL models due to the heavy computing cost. Experiment results of average performance across languages are shown in Table 7, and results of each language are reported in Table 22 in Appendix.
ACE05. For ACE05 event extraction (Walker et al., 2006), we use the OneIE joint model v0.4.8 codebase14 with the same hyperparameters as Yarmohammadi et al. (2021). For evaluation, we use the OneIE scoring tool to report F1 scores for entities, relations, event triggers identification (Trig-I) and classification (Trig-C), argument role identification (Arg-I) and classification (Arg-C). We train models on the combination of English and projected Chinese data from scratch in the Chinese experiment and select the model based on the English development set. In the Arabic experiment, we initialize the model from the English fine-tuned checkpoint. We fine-tune the argument role classifier for event extraction tasks (Entity, Trig-I, Trig-C, Arg-I, Arg-C) and relation classifier in relation task for 5 epochs. We set the learning rate of task-specific classifiers at 1e-6 and the encoder at 5e-4. During the decoding process of relation classification, we only consider the joint model’s relation and entity prediction scores.
14http://blender.cs.illinois.edu/software/ oneie/
15
On the Arabic data annotation side, the ACE Arabic data contains language-specific annotations on pronoun entities due to morphological stemming (Zitouni et al., 2005), where we observe individual Arabic letter (prefix or suffix) is annotated as a pronoun entity. Because such annotations don’t exist in English data, the label projection process may cause inconsistency in translated-Arabic data. Thus, we remove the pronoun entities in both Arabic test data and English training data for the Arabic experiment. The complete statistics of the Arabic test set is in Table 14. We report the average results of three random seeds.

heading: E Fixing Issues in the Arabic ACE Data
text: The ACE data are pre-processed using the code from Lin et al. (2020). We use the same document splits as Lin et al. (2020) for English (ACE05E+) and Chinese (ACE05-CN). For Arabic, we use the document splits from Lan et al. (2020) following Yarmohammadi et al. (2021).
In the processed Arabic test set from Yarmohammadi et al. (2021), we observed 12 extremely long sentences with an average length of 381 tokens, which are significantly longer than the rest of the sentences with an average length of 28. This issue was also independently noticed by Huang et al. (2022b). A closer look reveals that these 12 sentences are 12 full articles in the original LDC release, which appear to be missing punctuation. We hire a native Arabic speaker to manually split them into sentences, resulting in 106 additional sentences. The data statistics are shown in Table 14. Because the ACE data is licensed, we will release the processing script instead.

heading: F More Experiments
text: In this section, we present more experiments to compare EasyProject with other approaches.
F.1 Comparison to MulDA Table 15 shows a direct comparison of EasyProject with MulDA (Liu et al., 2021), another translationbased label projection approach that has been recently proposed for NER. MulDA replaces named entities with placeholders one by one, such as ‘PER0’ and ‘LOC1’, then invokes the MT system separately for each entity to translate and project the data. Thus, MulDA is more time-consuming and costly than the EasyProject, which only requires one invocation of the MT system per sentence. We find that EasyProject outperforms MulDA in German, Spanish, and Dutch at much less time cost. In this experiment, we follow MulDA’s experimental setup, which uses the CoNLL NER dataset and trains only on the projected data.
In terms of translation speed, we calculate the relative time cost of EasyProject compared to translating the original sentences in CoNLL English data using the NLLB model on one A40 GPU. In Table 15, we observe marker-based (XML and []) translation takes 1.2× and 1.3× longer of time to translate, due to the additional markers in both the input and output. More analysis of the translation speed is provided in Appendix G.3.
F.2 Comparison to Bitext Projection Besides translation-projection, another alternative is bitext-projection, in which bilingual parallel corpora are used in place of a machine translation system. For example, we can apply a trained English IE model to the English side of the bilingual parallel corpus, then use word alignment to project the
16
automatically predicted labels to the corresponding sentences in target languages.
In Table 16, we show that bitext-projection improves F1 of WikiANN NER on 6 out of 8 languages used in (Yarmohammadi et al., 2021) over the fine-tuning baseline (Fint-tuneen), but is outperformed by EasyProject. For this experiment, we randomly sample 100,000 parallel sentences for each of the eight languages from WikiMatrix (Schwenk et al., 2021a), an automatically mined bitext corpus from Wikipedia that matches the domain of WikiANN. We use an XLMRoBERTalarge NER model trained on WikiANN English data with 83.9 F1 to generate named entity labels, and then apply Awesome-align to project labels to the target language. Finally, we train the XLM-RoBERTalarge model on English and bitextprojected data together for 2 epochs (Bitext100k).
Bitext100k loses 13.9 F1 score for Vietnamese (vi), most likely due to Awesome-align projection errors being magnified by fine-tuning on 100,000 projected sentences. One surprising finding is that the Bitext100k improves by an absolute 4.4 F1 score on Spanish and 1.1 F1 on Russian. Translationprojection approaches struggle on these two languages as shown in Table 8.
Lang. FTen EasyProject Bitext100k EasyProject +Bitext100k
F.3 Experiments on Low-resource Languages To investigate the effectiveness of label projection on very low-resource languages (Pfeiffer et al., 2020), we conduct experiments on Māori (mi) and Turkmen (tk), which are not covered by the pretrained language models (i.e., XLM-RoBERTa and mBERT) and have a small number of Wikipedia articles (∼1.2k for Māori and ∼0.5k for Turkman). As shown in Table 17, EasyProject improves F1
score by an absolute 22.7 F1 score on Māori and 5.9 F1 on Turkmen compared to fine-tuning on English data only. We also include a lexicon-based baseline, replacing English words with their wordto-word translations based on PanLex (Kamholz et al., 2014), a commonly used multilingual dictionary. Both EasyProject and Awesome-align significantly outperform the word-level translations, likely because word-level translations still follow the English word orders and fail to capture the variation of word orders in Māori and Turkmen. For example, Māori has a verb-subject-object word order, while Turkmen uses a subject-object-verb. The improvement is less significant on Turkmen than Māori, potentially because Turkmen is close to Turkish, which is covered by both mBERT and XLM-RoBERTa. This is also a plausible reason why Awesome-align that uses mBERT did better on Turkmen.

heading: G More Analysis on EasyProject
text: Here, we present more analysis of the EasyProject method in comparison with the traditional pipeline approach based on word alignment.
G.1 Translation Quality
To further measure the impact of adding special markers on the translation quality for the NLLB model, we adopt the evaluation setup used by NLLB (Costa-jussà et al., 2022) which utilizes the professional human-translated FLORES-200 parallel corpora (1000 sentences per language). For the marker-based approaches (“XML” and “[]”), special markers are removed from the outputs before calculating the BLEU scores. Table 18 presents the BLEU scores for the original NLLB model (3.3B) and the NLLB model further fine-tuned with three
17
types of parallel sentences (original, inserted with XML and [] markers).
As there is no gold NER annotation on the parallel corpus, we first train a NER model based on XLM-RoBERTlarge on the WikiAnn dataset, achieving an F1 score of 83.9. We then apply the trained model to the English side of the parallel corpus and apply the EasyProject method to translate sentences into the target language. After removing the special markers from the translation outputs, we use the sacreBLEU15 to calculate the BLEU scores by comparing the translations against gold references. We follow the NLLB evaluation setting and use multilingual tokenizations (flores200).16
15https://github.com/mjpost/sacrebleu 16https://github.com/mjpost/sacrebleu/blob/
master/CHANGELOG.md
G.2 Projection Rate
We then compare the projection rate for all label projection methods, for which we divide the number of annotations after projection by the number of annotations occurring in the original training data. We also include the average number of successfully projected sentences after filtering out the incorrect ones, which have a different number of annotations compared to the source sentence. For example, the source sentence has a LOC and a PER entity, but the projected sentence has two LOC entities. Such sentences will be filtered. For QA-align in WikiANN NER, we show the average statistics for 5 languages {ar, de, fr, ja, zh} that have supervised training data.
As shown in Table 21, Google Translation (GMT) is very robust in handling special markers, and EasyProject has a nearly perfect 100% projec-
18
tion rate, higher than any word alignment-based method. Our manual inspection of 100 sentences, randomly sampled from the WikiANN training set for English to Chinese projection, also reveals that GMT+EasyProject successfully projects all the sentences without mistakes on any target named entities, whereas Awesome-align only projected 94 sentences and caused 4 entity projection errors. According to our manual analysis, EasyProject is less likely to introduce errors than the word alignmentbased method because the use of special markers encourages full-span projection.
We found that most errors are caused by partial or missed alignments, which often occur when a span contains multiple words, a sentence contains many spans, or when both a Chinese transliteration and the original English name occur together in the translated sentence, which is a correct way to translate but poses challenges for label projection. More examples of alignment errors can be found in Table 19 in Appendix.
G.3 Translation Speed Additional special markers added to the source sentence will affect the translation speed. In Table 20, we show the number of tokens in the input and translation output. We use the CoNLL-2002/2003 NER English training set as the source sentences, and translate them into German. All sentences are tokenized by the NLLB tokenizer. We also show the translation time per sentence and per entity on an A40 GPU with a batch size of 32.
We estimate using XML tags takes 1.2× time compared to translating the original sentences, and EasyProject takes 1.3× time as it requires the additional translation of each entity span, for identifying the label correspondence.
19
20
ACL 2023 Responsible NLP Checklist
A For every submission: 3 A1. Did you describe the limitations of your work?
Limitation section
3 A2. Did you discuss any potential risks of your work? Limitation section
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Appendix A-F
3 B1. Did you cite the creators of artifacts you used? Appendix A-F
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Appendix F
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix E
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? we use existing published datasets.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Table 14
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Table 1 & 10
C 3 Did you run computational experiments? Section 3 & 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix E and H.3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix E
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix E
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix C and Appendix H.1
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Appendix F
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix F
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Appendix F

heading: Future Work
text: To generalize this approach to future MT systems, we design a simple and computationally efficient approach to improve the robustness of MT systems in handling special markers. However, the translation quality for the marker-inserted text still falls behind the original text. We leave the work of further optimizing the translation quality as future work