T-Projection: High Quality Annotation Projection for Sequence Labeling Tasks
Authors: Iker García-Ferrero
Rodrigo Agerri
German Rigau
Venue: EMNLP 2023 Findings (2023)
Keywords: annotation projection
low-resource
sequence labeling
text2text language models
machine translation
automatic data generation
TL;DR: T-Projection outperforms previous methods in annotation projection for sequence labeling, offering a solution to generate annotated data in low-resource languages.

Abstract: In the absence of readily available labeled data for a given sequence labeling task and language, annotation projection has been proposed as one of the possible strategies to automatically generate annotated data. Annotation projection has often been formulated as the task of transporting, on parallel corpora, the labels pertaining to a given span in the source language into its corresponding span in the target language. In this paper we present T-Projection, a novel approach for annotation projection that leverages large pretrained text2text language models and state-of-the-art machine translation technology. T-Projection decomposes the label projection task into two subtasks: (i) A candidate generation step, in which a set of projection candidates using a multilingual T5 model is generated and, (ii) a candidate selection step, in which the generated candidates are ranked based on translation probabilities. We conducted experiments on intrinsic and extrinsic tasks in 5 Indo-European and 8 low-resource African languages. We demostrate that T-projection outperforms previous annotation projection methods by a wide margin. We believe that T-Projection can help to automatically alleviate the lack of high-quality training data for sequence labeling tasks. Code and data are publicly available.

T-Projection: High Quality Annotation Projection for Sequence Labeling Tasks
Iker Fereia-Ferrero Rodrigo Agerri German Rigau HiTZ Center Ixa, University of the Basque Country UPV/EHU { iker.garciaf, rodrigo.agerri, german.rigau @@ehu.eus
Abstract
<Locatlon-New Yort--Locctioo>>
Labeled sertinnee source language
absence readily available labeled data for a given sequence labeling task and language, annotation projection has been proposed as of the possible strategies automatically generate annotated data. Annotation projec- tion has often been formulated as the task of transporting, parallel corpora, the lahels per- taining to given span in the source language into corresponding span the target lan- guage. In this paper present T-Projection, novel approach for annotation projection that leverages large pretrained text-to-text language models and state-of-the-art machine translation technology. T-Projection decomposes the la- bel projection task into two subtasks: (i) A candidate generation step, in which set of projection candidates using multilingual model is generated and, (ii) candidate selec- tion step, in which the generated candidates are ranked based on translation probabilities. We conducted experiments intrinsic and ex- trinsic tasks in 5 Indo-European and 8 low- resource African languages, We demostrate that T-projection outperforms previous annota- tion projection methods by a wide margin. We believe that T-Projection can help automati- cally alleviate the lack of high-quality training data sequence laheling tasks. Code data are publicly !vailable.!
Mc encanta Nueva York
Unlabaled sentence target language
Me encanta Nueva York. <Locatin-NONE-LLommm
Candidate generation
Muttllngual T5
search
<LoocationNueva York--Location>
encanta
most probable predictions
New York
Condidate selection
transiation system
Transiation probability
Nueva York 0.98
0.22
Nuwvi York EinoiiionNNee Yorkei oraaiion
Labeled senfence in barget language
Figure 1: T-Projection two-step method to project se- quence labeling annotations across languages.
1 Introduction
annotated data for each upplication domain and lan- guage, an unfeasible task in terms of monetary cost and human effort. As a result, for the majority languages in the world the amount of manually an- notated data for many downstream tasks is simply nonexistent (Joshi et al., 2020).
The performance of supervised machine-learning methods for Natural Language Processing, includ- ing advanced deep-neural models (Devlin al., 2019; Conneau al., 2020; Xue al., 2021; Zhang et al., 2022), heavily depends on the availability manually annotated training data. addition, supervised models show a significant decrease in performance when evaluated in out-of-domain set- tings (Liu et al., 2021). This means that obtaining optimal results would require manually generate
The emergence of multilingual language models (Devlin et al., 2019; Conneau et al., 2020) allows for zero-shot cross-lingual transfer. A model fine- tuned on one language, typically English, can be directly applied to other target languages. However, better results can be obtained by either machine translating the training data from English into the
'https://githiin www.ittuub.comieeee com//kkergaciia199/ T-Projection

---

target languages or, conversely, translating the test data from the target language into English (Hu et al., 2020; Artetxe al., 2023).
generated by various systems with manually pro- jected annotations. On average, T-Projection im- proves the current state-of-the-art annotation projection methods by more than 8 points in F1 score, which constitutes a significant leap quality over previous label projection approaches, Additionally, we performed real-world NER task evaluation involving eight low-resource African languages. In this downstream evaluation, T- Projection outperformed other annotation projec- tion methods by 3.6 points in F1 score.
Sequence labeling tasks, which involve span- level annotations, require an additional step called annotation projection. This step involves identi- fying, in the translated sentence, the sequence of words that corresponds to the labeled spans in the source text (Yarowsky al., 2001; Ehrmann et al., 2011). The majority of previous published work on this line of research explores the application of word-alignments (Ehrmann et al.. 2011). How- ever, projection methods based on word-ali word-alignments have achieved mixed results they often produce partial, incorrect or missing annotation projections Farcea-FFereer al., 2022). This is due to the fact that word alignments are computed on a word- by-- word basis leveraging word co-occurrences or simi- larity between word vector representations. That is, without taking into consideration the labeled spans or categories to be projected. Other techniques have also been proposed, such as fine-tuning lan- guage models in the span projection task (Li et al., 2021), translating the labeled spans independently from the sentence (Zhou al., 2022) or includ- ing markers during the machine translation step (Chen et al., 2023). However, automatic annotation projection remains an open and difficult challenge.
2 Background
While most of the previous approaches for anno- tation projection are based on the application of word alignments, other techniques have also been explored.
2.1 Word alignment methods
Ehrmann al. (2011) use the statistical aligmment of phrases to project the English labels of a multi- parallel corpus into the target languages. Instead of using discrete labels, Wang and Manning (2014) project model expectations with the aim of facil- itating the transfer of model uncertainty across languages. Ni et al. (2017) aim to filter good- quality projection-labeled data from noisy data by proposing a of heuristics. Other works have pro- posed to use word alignments generated by Giza++ (Och and Ney, 2003) to project parallel labeled data from multiple languages into single target language (Agerri et al., 2018). Fei et al. (2020) use the word alignment probabilities calculated with FastAlign (Dyer et al., 2013) and the POS tag distributions of the source and target words project from the source corpus into target lan- guage machine-translater corpus. Finally, Gareía- Ferrero et al. (2022) propose an annotation pro- jection method based on machine translation and AWESOME (Dou and Neubig, 2021), Transformer- based word alignments to automatically project datasets from English to other languages.
In this paper we present T-Projection, a novel approach to automatically project sequence label- ing annotations across languages. Our method is illustrated by Figure 1. We split the annotation projection task into two steps. First, we use mT5 (Xue et al., 2021) text-to-text model to generate a set of projection candidates in the target sentence for each labeled category in the source sentence. This step exploits the labeled categories as well the cross-lingual capabilities of large pretrained multilingual language models. Second, rank the candidates based the probability of being gener- ated translation of the source spans. We use the M2M100 (Fan et al., 2021) and NLLB200 (Costa- jussà et al., 2022) state-of-the-e MT models to compute the translation probabilities (Vamvas and Sennrich, 2022).
2.2 Other projection methods
With respect to projection methods which do not word alignments, Jain et al. (2019) first gener- a list of projection candidates by orthographic and phonetic similarity. They choose the best matching candidate based on distributional statis- tics derived from the dataset. Xie et al. (2018) propose a method to find word translations based
We conduct intrinsic evaluation on three dif- ferent tasks, Opinion Target Extraction (OTE), Named Entity Recognition (NER) and Argument Mining (AM), and five different target languages (French, German, Italian, Russian and Spanish). In this evaluation we compare the label projections

---

on bilingual word embeddings. Li al. (2021) use a XLM-RoBERTa model (Conneau et al., 2020) trained with the source labeled part of parallel corpus to label the target part of the corpus. Then they train a new improved model with both labeled parts. Zhou et al. (2022) first replace the labeled sequences with placeholder token in the source sentence. Second, they separately translate the sen- tence with the placeholders and the labeled spans into the target sentences. Finally, they replace the placeholders in the translated sentence with the translation of the labeled spans. Chen et al. (2023) jointly perform translation and projection by in- serting special markers around the labeled spans in the source sentence. To improve the translation accuracy and reduce translation artifacts, they fine- tune the translation model with synthetic label protection dataset.
love New York cxctioo coctooe
No encarii Naaea Yor cuarce KONE-1
Muilinguu
Mullingga
coadooowaaaeee contee
Figure 2: Illustration of the candidate generation step. For each label, we generate a set of probable candidates.
labeled spans and their categories in the source sentence as well as the zero-shot cross-lingual ca- pabilities of large pretrained multilingual language models, the candidate selection step applies state- of-the-art MT technology to find those projection candidates that constitute the best translation for each source labeled span. As demonstrated by our empirical evaluation in Sections 5 and 6, we be- lieve that these two techniques leverage both the available information and knowledge from the an- notated text and language models thereby allowing us to obtain better annotation projections. These two steps are described in detail in the following two subsections.
To summarize, previous work does not take ad- vantage of all the information which is available while performing annotation projection. For ex- ample, word alignment models do not take into account the labeled spans and their categories dur- ing alignment generation. Instead, they simply rely on information about word co-occurrences or similarity between word representations. Those techniques based on MT to generate the target part of the parallel corpus ignore additional knowledge that the MT model encodes. Furthermore, methods that utilize MT models for both translation and pro- jection often introduce translation artifacts, which can affect the quality and accuracy of the projec- tions. In contrast, our T-Projection method exploits both the labeled spans and their categories together with the translation probabilities to produce high- quality annotation projections.
3.1 Candidate Generation
When trying to project labeled sequences from some source data into its parallel target dataset, we would expect both the source and the target to contain the same number of sequences labeled with the same category. For example, given the En- glish source sentence Oharma went to <Location>>>e York</Location>" and its parallel Spanish unlabeled target sentence "Ohama fue a Nueva York", we would expect to find the same two entities (person and location) in the tar- get sentence. To solve the task of candidate genera- tion, finetune the text-to-text mT5 (Xue et al., 2021) model using a HTML-tag-style prompt tem- plate (Huang al.. 2022). As illustrated by Fig- ure 2, the input consists of concatenating the unla- beled sentence followed by a list of tags ("<Cat- NNoe=</Catooorr/>>> with the category of each labeled span that we expect to find in the sen- tence. If two or more spans share the same category then we append the tag as many times spans are expected with that category.
3 T-Projection
Given source sentence in which we have se- quences of words labeled with a class, and its paral- lel sentence in a target language, we want to project the labels from the source into the target. As illus- trated in Figure 1, T-Projection implements two main steps. First, a set of projection candidates in the target sentence are generated for each labeled sequence in the source sentence. Second, each pro- jection candidate is ranked using machine trans- lation model. More specifically, candidates are scored based on the probability being generated as a translation of the source labeled sequences. While the candidate generation step exploits the
Unlike Huang et al. (2022), we do not encode the tags for each category as special tokens. Instead, we verbalize the categories (i.e PER->Person) and

---

Source Span
Candidates for Location
Translation Probabiltty
ing the pretrained M2M100 (Fan et al., 2021) NLLB200 (Costa-jusså al., 2022) MT models and the NMTScore library² (Vamvas and Sennrich, 2022). Thus, given the source span A and the can- didate B the translation probability is computed as follows (Vamvas and Sennrich, 2022):
Nueva
0:38
X
York
0.82
LOCATION
encariaa
0:22
X
Po.. (A B) = [11 Po, (A' B, A [
The translation probability is normalized:
Ps, (AIB)
Candidate selection: candidates scored based on the probability of being generated as transla- tion of the originally labeled sequences.
Figure 3:
sim(A B)
Ps. (A]4)
As the translation probability varies depending on the direction of the translation, the scores are symmetrized by computing the scores of both trans- lation directions and averaging them:
the token representations already existing in the model. We expect that, thanks the language modeling pretraining. T5 would have good se- mantic representation categories such as Person, Location, Claim, etc.
sim( (A, B) {A) B) + ( A)
Finally, for each labeled span in the source sen- tence, we choose the candidate in the target with the highest translation probability. Once candidate has been selected, that candidate, and any other that overlaps with it, is removed from the of possible candidates. this way we prevent assigning the same candidate in the target to two different spans in the source.
As Figure 2 illustrates, we finetune mT5 with the labeled source dataset. We train the model replace the token None with the sequence of words in the input sentence that corresponds that category. We use Cross-Entropy loss for training. At inference, we label the target sentences which are parallel translations of the source sentences. As the source tells us how many labeled spans should we obtain in the target, we use the labels of the corresponding source sentence to build the prompts. In other words, our objective to label parallel translations of the sentences we used for training. We take advantage of the zero-shot cross-lingual capabilities of mT5 to project the labels from the source to the target sentence. The output tokens are generated in an autoregressive manner. We use beam search decoding with 100 beams to generate 100 candidates for each input tag.
Experimental Setup
In order evaluate our method we perform both intrinsic and extrinsic evaluations.
Intrinsic evaluation: We select a number of datasets that have been manually projected from English into different target languages. The man- ual annotations are used to evaluate and compare T- T-Projection with respect previous sate-of-the- art label projection models. Results are reported by computing the usual F1-score used for sequence labelling (Tjong Kim Sang, 2002) with the aim of evaluating the quality of the automatically gener- ated projections with respect to the manual ones.
3.2 Candidate Selection
As depicted in Figure 3, all the generated candi- dates are grouped by category. In other words, if the previous step has generated multiple spans with the same category (i.e, two locations in a sentence) then all the candidates are included in a single set. Furthermore, all the candidates that are not sub- sequence of the input sentence are filtered out. For each labeled span in the source sentence, we rank all the projection candidates that share the same category as the source span using their translation probabilities (also known as translation equivalence) which have been obtained by apply-
Extrinsic evaluation: In this evaluation sess the capability of T-PProeection to automatically generate training data for sequence labeling tasks, NER in this particular case. The process begins by utilizing the machine translation system NLLB200 (Costa-jussà et al., 2022) to translate data from En- glish into 8 low-resource African languages. We then project the labels from English onto the respec- tive target languages. The automatically generated datasets then employed to train NER models,
hhttps://githhi httpa///..coomttrcicc//////teeee

---

Serves reaily
sush
Obama PERSON
visitod
France LOCATION
Monday
Nausea the only notable symplom, paliette in group suffered severe
TARGET
PREMISE
CLAIM
Opirion Target Estraction
Named Enily Racagnilion
Argument Mining
Figure 4: Sequence labeling tasks in our experiments
which are evaluated using relatively small manu- ally annotated test set. The same procedure is per- formed with other state-of-the-an label projection models. The comparison of the results obtained reported in terms of F1 score.
widely used in the field. We also evaluate two Transformer-based systems. SimAlign (Jalili Sa- bet et al., 2020) and AWESOME (Dou and Neu- big, 2021), which leverage pre-trained multilingual language models to generate alignments. As the authors recommend, we use multilingual BERT (mBERT) (Devlin et al., 2019) as the backbone. We tested different models as backbone with no improvement in performance (see Appendix C). We use these four systems compute word align- ments between the source and the target sentences and generate the label projections applying the al- gorithm published by García-Ferree et al. (2022)².
4.1 Datasets
The datasets used correspond to three sequence labeling tasks which are illustrated by Figure 4. Opinion Target Extraction (OTE) Given re- view, the task is to detect the linguistic expression used to refer to the reviewed entity. We use the English SemEval 2016 Aspect Based Sentiment Analysis (ABSA) datasets (Pontiki et al., 2014). Additionally, for the evaluation we also used the parallel versions for Spanish, French and Russian generated via machine translation and manual pro- jection of the labels (Garefa-Ferrerd et al.. 2022). Named Entity Recognition (NER) The NER task consists of detecting named entities and clas- sifying them according to some pre-defined cate- gories. We use an English. Spanish, German, and Italian parallel NER dataset (Agerri et al., 2018) based on Europarl data (Koehn, 2005). Manual provided fol-
We also experiment with EasyProjedt (Chen et al., 2023), a system that jointly performs trans- lation and projection by inserting special markers around the labeled spans in the source sentence. As this model generates its own translations it is there- fore not suitable for the intrinsic evaluation which is why we only used it for the extrinsic evaluation. We implemented two additional baselines for comparison. In the first baseline, inspired by Li et al. (2021), we use XLM-RoBERTa (Conneau et al., 2020) 3 billion parameter model (same pa- rameter count as the mT5 model that use in T--Prjection) with token classification layer (lin- ear layer) on top of each token representation, We train the model in the source labeled dataset and we predict the entities in the translated target sen- tences. The second baseline adopts a span transla- tion approach inspired by Zhou et al. (2022). We translate the labeled spans in the source sentence using the pretrained M2M100 12 billion parameter model and we match them with the corresponding span in the target sentence. For example, given the labeled source sentence "I love <Location> New York </Location>." and the target sentence "Me en- canta Nueva York", we translate the span New York into the target language, resulting in the translated span Nueva York which is then matched in the tar- get sentence. We employ beam search to generate 100 possible translations, and select the most probable one that matches the target sentence.
annotation for the 4 languages lowing the CoNLL 2003 (Tjong Kim Sang, 2002) guidelines. In the extrinsic evaluation, we use MasakhaNER 2.0 (Adelani et al., 2022), a human- annotated NER dataset for 20 African languages. A Argument Mining (AM) In the AbstRCT English dataset two types of argument components, Claims and Premises, are annotated medical and scien- tific texts collected from the MEDLINE database (Mayer et al., 2020). For evaluation its Spanish parallel counterpart which was gener- ated following an adapted version of the method described above for OTE (Yeginbergenova and Agerri, 2023). In contrast to NER and OTE, the se- quences in the AM task consist of very long spans of words, often encompassing full sentences. We use the Neoplasm split.
used
4.2 Baselines
We experiment with 4 different word alignment systems. Two statistical systems, Giza++ (Och and Ney, 2003) and FastAlign (Dyer et al., 2013),
'tttps: htt.///gttttuute/ wtt.eiooooiieeeeeee aom/ikergarcia1996. Easy-Label-Prone Projjctton

---

4.3 Models Setup
tences from the WMT19 Biomedical Translation Task (Bawden al., 2019). We evaluate the mod- els with respect to the manually projected labels by Yeginbergenova and Agerri (2023).
We use the 3 billion parameters pretrained mT5 (Xue et al., I for the candidate generation step while candidates are selected using the M2M100 12 billion parameter machine translation model (Fan al., 2021). In the case of MasakhaNER, since not all languages are included in M2M100, we resorted to NLLB200 (Costa-jussà et al., 2022) 3 billion parameter model instead, which was also used by the EasyProject method (Chen et al., 2023). Both MT models demonstrate compara- ble performance. For detailed hyperparameter set- tings, performance comparison of T- T-Projection us- ing models of different sizes, and comparison between T-Projection performance using M2M100 and NLLB200, please refer to the Appendix.
5.1 Annotation Projection Quality
Table I reports the results of the automatically projected datasets generated by each projection method with respect to the human-projected ver- sions of those datasets. The systems based on word alignments obtain good results across the board, especially those using language models, namely, SimAlign and AWESOME. In particular, AWE- SOME achieves good results for OTE and NER but very poor in AM. Manual inspection of the projec- tions found out that AWESOME struggles to align articles and prepositions which are included in long sequences.
5 Intrinsic Evaluation
In this section we present set of experiments to evaluate T-Projection with respect to current state- of-the-art approaches for annotation projection. We also analyze separately the performance of the can- didate generation and candidate selection steps.
XLM-RoBERTa-xX shows a strong zero-shot cross-lingual performance. However, the gener- ated datasets are of lower quality than the ones generated by the word-alignment systems. The sults of the Span Translation approach are quite disappointing. especially when dealing with the long sequences of the AM task. Translating the labeled spans independently produce translations that, in many cases, cannot be located in the target sentence.
For the OTE task we train T-Projection and BLERRa with the English ABSA 2016 training set. We also train the four word align- ment systems (excluding SimAlign which is an unsupervised method) using the English training set together with the respective translations par- allel corpora. We augment the parallel data with 50,000 random parallel sentences from ParaCrawl v8 (Esplå et al., 2019). Models are evaluated with respect to the manually label projections generated by Garefa-Ferrero et al. (2022). As the Europarl- based NER dataset (Agerri et al., 2018) provides only test data for each language, T-Projection and XLM-RoBERTa are trained using the full English CoNLL 2003 dataset (Tjong Kim Sang, 2002) to- gether with the labeled English Europarl test data, The word alignment models are in turn trained with the the parallel sentences from the Europarl- -ased NER data plus 50,000 parallel sentences extracted from Europarl v8 (Koehn, 2005). We evaluate the model with respect to the manual annotations pro- vided by Agerri et al. (2018). With respect to Ar- gument Mining, from the AbstRCT dataset to train T-Projection and XLM-RoBERTa, adding its Spanish translation as parallel corpora for the word alignment systems.
Our T-Projection method achieves the best re- sults for every task and language. In OTE, it out- performs every other method by more than points in FI score averaged across the three languages. This suggests that T-Projection robustly projects labeled spans into
data. The NER evaluation is slightly different because the parallel data was translated by human experts. In this setting, T-Projection clearly improves AWE- SOME's results by 4.7 points, which constitutes a significant leap the quality of the generated datascts.
Despite the fact that the word alignment systems have been trained using Europarl domain-specific data, and that most of the training data used for T- Projection is coming from the 20N0 dataset (news domain) plus very few annotated sentences (699) from Europarl, T-Projection still clearly ob- tains the best results in NER label projection. This suggests that our system can also be applied in out-of-domail settings.
the Neoplasm training set
As this is a medical text corpus, the parallel cor- pora is complemented with 50,000 parallel sen-
Finally, T-Projection obtains the overall high- est scores for Argument Mining which means that

---

OTE
NER
AM
Avg
ES
FR
RU
ES
DE
IT
ES
Giza++ (Och and Ney, 2003)
77.0
73.3
72.4
73.3
75.3
68.4
86.6
77.7
FastAlign (Dyer et al,, 2013)
75.0
72.9
76.9
70.2
77.0
67.0
85.7
77.4
SimAlign (Jalili Sabet et al., 2020)
86.7
86.3
87.7
85.4
87.4
81.3
84.1
85.3
AWESOME (Dou und Neubig. 2021)
91.5
91.1
93.7
87.3
90.7
83.1
54.8
78.0
XLM BLM-RoBERTisxl (Conneau al., 2020)
80.2
76.2
74.5
73.9
68.3
73.9
66.5
71.8
Span Translation
66.5
46.3
58.7
63.5
69.2
48.7
T- T-Projection
95.1
92.3
95.0
93.6
94.0
87.2
96.0
93.9
Table I: Fl scores for annotation projection in the OTE, NER and Argument Mining tasks.
our approach is particularly good projecting long sequences. Thus, T-Projection outperforms the second best model by 9.4 points in F1 score. In fact, the 96.0 Fl result obtained indicates that T- Projection is almost correctly projecting all the examples in the dataset.
very similar to each other, and this makes select- ing the right candidate a more challenging task, Thus, this experiment proves that our mT5 candi- date generation approach is crucial to obtain good performance.
We also replace the candidate selection method with the most probable candidate. In other words, only use the most probable beam generated by mTS label the target sentence. When us- ing mT5 by itself, it obtains competitive results, close to those of the word alignment systems Table 1. Still, the average performance drops by 9.2 points. This further confirms that both the can- didate generation and selection steps are crucial for the T-Projection method.
If we look the average over the three tasks and 5 languages, T-Projection improves by 8.6 points in Fl score the results of the second-best system, SimAlign. These results constitute huge improve- ment over all the previous annotation projection approaches. To the best of our knowledge, these are by wide margin the best annotation projection results published for sequence labeling.
5.2
The Role of the Candidates
In final experiment define upperbound for candidate selection consisting of assuming that our model will always select the correct projection contained among the generated candidates. The upper bound achieves an average Fl score of 98. This result confirms with very high probability that the correct candidate is almost always among the 100 candidates generated by mT5.
OTE
NER
AM
Avg
ES
FR
RU
ES
DE
IT
ES
T-Prejecttonn
95.1
92.3
95.0
93.6
94.0
87.2
96.0
93.9
Ngrams
Candidate
73.3
Selection
Most Probable
83.7
87.2
85.3
79.5
82.8
72.3
90.9
84.8
Candidate
m555-
Extrinsic Evaluation
Upper bound
98.6
97.0
97.9
98.0
98.5
94.0
99.3
98.0
In this section we evaluate T-projection in real- world low-resource scenario, namely, Named En- tity Recognition in African Languages. We com- pare the results obtained by training on NER dataset automatically generated by T-Projection with respect to those automatically projected using two state-of-the-art label projection systems: AWE- SOME (The second- NER system in Table 1) and EasyProject. We use the exactly same settings as Chen et al. (2023). For each target language in MasakhaNER2.0, we first translate the English CoNLL dataset using the NLLB-200 3 billion pa- rameter model. Next, we project the English labels into the target language. It should be noted that ErsyProject performs both these processes in a
Table 2: Fl scores for different candidate generation and candidate selection methods.
We perform a set of experiments to measure the relevance and performance of the candidate gen- eration and candidate selection tasks. First, we replace mT5 with an agredd candidate gener- ation approach. We consider as candidate spans ev- cry possible ngram with size cecentenne_lennghe (i.e "Serves", "really", "good", "sushi", "Serves re- ally"...). Table shows that this approach results in lower performance compared with our technique using mT5. Ngrams are much noisier than the candidates generated by mT5, most of them are

---

Language
No. of Speakers
Lang family
Finc-tunc
AWESOME-EN
EnsyProjcctEN
T-Projection
T-Projection-EB EN
Hausa
63M
Afro-Asiatic XChadic
71.7
72.7
72.2
72.7
72.0
Igbo
27M
NC/ Volta Niger
59.3
63.5
65.6
71.4
71.6
Chichewa
14M
Einglish: Creole
English-Creooe
79.5
75.1
75.3
77.2
77.8
chiShona
NC/
74.3
Kiswahili
98M
NC/ Bantu
87.7
82.4
83.6
84.1
isiXhrsa
9M
NC/ Bantu
24.0
61.7
71.1
72.3
71.7
Yoruba
42M
NC/ Volta-Niger
36.0
38.1
36.8
42.7
42.1
isiZalu
27M
NC/ Bantu
43.9
68.9
73.0
66.7
64.9
AVG
54.7
66.5
66.7
70.3
69.8
Table Fl scores on MasakhaNER2.00 for mDebertaV3 trained with projected annotations from different systems. "+EN" denotes concatenation of the automatically generated target language dataset with the source English dataset.
single step. Subsequently, we train an mDebertaV3 (He et al., 2021) model using the automatically gen- erated datasets for each target language. Finally, this model is evaluated in the gold MasakhaNER2.0 test data. We only evaluate the 8 languages in MasakhaNER2.0 supported by mT5. We focus named entities referring to Person, Location and Organization.
ku [European Union] komiti yezvokurapa mhuka WWerner Zwingmann]". When training mDeber- taV3 with T-projection generated data, which pre- serves the agglutinated words, we achieve better results compared to EasyProject that introduce arti- facts by separating agglutinated words during trans- lation and projection. Our system is only inferior in the Zulu language; however, on average, we im- prove the results by 3.6 Fl points. contrast with previous work, our experiments revealed that con- catenating English and translated data did not yield better results, potentially due to the superior quality of the data generated by T-Projection.
Table 3 presents the results of the evaluated mod- els on the gold MasakhaNE.R2.0 test sets. For T- projection, present the results of training with the automatically generated data for the target lan- guage only, and also by adding the original English CoNLL data concatenated with the automatically generated data for each target language. Regarding other systems, we only show the former results, as it was the only metric reported by previous work. In order to train and evaluate the NER models we apply the same hyperparameter settings and code the authors of EasyProject.
To the best of our knowledge, these are the best zero-shot results achieved for MasakhaNER2.00 un- derscoring the significant benefits of T-projection for NLP tasks in low-resource languages.
Concluding Remarks
The results show that T-projection achieves su- perior performance for seven out of the eight lan- guages. Our model demonstrates more pro- nounced performance difference in agglutinative languages such Igho and Shona. As outlined in Section 5, our model produces superior align- ments compared to AWESOME. On the other side, we found that EasyProject, which utilizes mark- ers for simultaneous translation and projection, in- troduces translation artifacts that hinder the per- formance the downstream model. These arti- facts are particularly noticeable in agglutinative languages, as EasyProject tends to separate words. For instance, in the case of Shona, consider the En- glish sentence "[Germany]'s representative to the [European Union]'s veterinary commitlee [Werner Zwingmann]". Our system produces the Shona sen- tence "Mumiriri [weGermany] kukomiti yemhuka [yeEuropean Union] [Werner Zwingmann]", while ErsyProject produces "Mumiriri we [Germany]
In this paper we introduce T-Projection. new anno- tation projection method that leverages large multi- lingual pretrained text-to-text language models and state-of- oftee----therrrt machine translation systems. We conducted experiments on intrinsic and extrinsic tasks in 5 Indo-European languages and 8 African languages. T-projection clearly outperforms previ- ous annotation projection approaches, obtaining a new A comprehensive analysis shows that both the generation candidate and the candidate selection steps crucially contribute to the final performance of T- T-Pooeection. Future work includes adding more tasks and languages, especially those with different segmentation such Japanese Chinese. Un- like word alignment systems, T-Projection does not need to segment the words do the projec- tion which why we think that our model can also be competitive to project annotations for many language pairs.
result for this task.

---

Limitations
References
David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani. Michael Beukman, Chester Palen- Michel, Constantine Lignos, Jesujoba Alabi, Sham- suddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure P. Dossou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe, Derguene Mbaye, Amelia Taylor, Fatoumata Ka- bore, Chris Chinenye Emezue, Anuoluwapo Aremu, Perez Ogayo, Catherine Gitau, Edwin Munkoh- Buabeng. Victoire Memdjokam Koagne, Allah- sera Auguste Tapo, Tebogo Macuewa, Vukosi Mari- vate, Mboning Tchiaze Elvis, Tajuddeen Gwad- abe, Tosin Adewumi, Orevaoghene Ahia, Joyce Nakatumba-Nabende, Neo Lerato Mokono, Ig- natius Ezeani, Chiamaka Chukwuneke, Mofetoluwa Oluwaseun Adeyemi, Gilles Quentin Hacheme, Idris Abdulmumin. Odunayo Ogundepo. Oreen Yousuf, Tatiana Moteu, and Dietrich Klakow. 2022. MasakhaNER 2.0: Africa-centric transfer learning for named entity recognition. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
We evaluate the performance of T-Projection to project labels in sequence labeling tasks from En- glish into a set of 5 Indo-European languages and 8 African languages. It would be interesting to eval- uate the performance for other language families, which we leave for future work. Our model re- quires training a 3B parameter m'S model. While training a 3B model is computationally expensive and requires GPU with at least 48GB of VRAM, automatically generating a dataset is one-off en- deavour which results in a dataset usable for many occasions and applications, and much cheaper than manual annotation. Furthermore, we believe that the huge gains obtained by T-Projection justify the computation requirements. In any case, we expect that, thanks to the rapid development of computer hardware, the cost of TT-Projection will be reduced in the near future. From a software perspective. recent advancements like 4-bit / 8-bit quantization (Dettmers et al., 2022b,a; Frantar et al., 2022) and Low Rank Adaptation (Hu et al., 2022) have the potential to reduce the hardware requirements of T-Projection.
Abu
Rodrigo Agerri, Yiling Chung, Itziar Aldabe, Nora Aranberri, Gorka Labaka, and German Rigau, 2018. Building named entity recognition taggers via par- allel corpora. In Proceedings of the Eleventh In- ternational Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Acknowledgments
Mikel Artetxe, Vedanuj Goswami, Shruti Bhosale, An- gela Fan, and Luke Zettlemoyer. 2023. Revisiting machine translation for cross-lingual classification. CoRR, abs/2305. 1it/230551424.....
This work has been partially supported by the HiTZ center and the Basque Government (Research
group funding IT-1800-22). We also acknowl-
Mikel Artetxe and Holger Schwenk. 2019. Mas- sively multilingual sentence embeddings for zero- shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics. 7:597-610.
edge the funding from the following projects:
(i)
DeepKnowledge (PID2021-12777708
C21)
and ERDF A way of making Europe;
(ii)
Disargue (TEDD201-1308100-C21)
Rachel Bawden, Kevin Bretonnel Cohen, Cristian Grozea, Antonio Jimeno Yepes, Madeleine Kittner, Martin Krallinger, Nancy Mah, Aurelie Neveol, Mar- iana Neves, Felipe Soares, Amy Siu, Karin Verspoor, and Maika Vicente Navarro. 2019. Findings of the WMT 2019 biomedical translation shared task: Eval- uation for MEDLINE abstracts and biomedical termi- nologies. In Proceedings of the Fourth Conference on Machine Translation ( (Volume 3: Shared Task Papers, Day 2), pages 29-53, Florence, Italy, Association for Computational Linguistics.
MCII/AAE/10..3033//500 100110009/5110001110000
and
European Union NextGenerattionU//RRRR 12070-2--
(iii)
Antidote
/TA.A////0..300//110000000000000 and by
European Union NextGennerationEUU/PPRRR:
(iv)
DeepR3 (1ED2021-130295B-C3) by MCIN/AEE//1.1103399 10001 and EU NextGeneration programme EU/PRTR. Rodrigo
Agerii currently holds the RYC-2017-23647
fellowship (MCIN/AEI/10.1 1..30.39/1 10001 1033 and by ESF Investing in your future). Iker García- Ferrero is supported by a doctoral grant from the Basque Government (PRE_ 2REE_2022_30 2_0208).
Yang Chen, Chao Jiang, Alan Ritter, and Wei Xu. 2023. Frustratingly easy label projection for cross- lingull transfer. In Findings of the Association for Compu- tational Linguistics: ACL 2023, Toronto, Canada, July 9-1 2023, pages 5775-5796 Association for Computational Linguisties.

---

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
annotated corpus using annotation projection. In Pro- ceedings of the International Conference Recent Ad- vances in Natural Language Processing 201 1, pages 118-124, Hissar, Bulgaria, Association for Computa- tional Linguistics.
Miquel Esplä, Mikel Forcada, Gema Ramírez-Sáncher and Hieu Hoang. 2019. ParaCrawl: Web-scale paral- lel corpora for the languages of the EU. In Proceed- ings of Machine Translation Summit XVII: Translator. Project and User Tracks, pages 118 119, Dublin, Ire- land. European Association for Machine Translation.
Marta R. Costa-jussä, James Cross, Onur Çelebi, Maha Elbayad. Kenneth Heafield, Kevin Heffer- nan, Flahe Kalhassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun. Skyler Wang, Guillaume Wenzek, Al Youngblood. Bapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett. Kaushik Ram Sadagopan. Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Rop- ers, Saflyyah Saleem, Holger Schwenk. and Jeff Wang. 2022. No language left behind: Scal- ing human-centered machine translation. CoRR, abs/2207.0467.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Ar- mand Joulin. 2021. Beyond english-centric multi- lingual machine translation. J. Mach. Learn. Res., 22: 12::0011107:4:
Hao Fei, Meishan Zhang, and Donghong Ji. 2020. Cross-lingual semantic role labeling with high- quality translated training corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7014-7026, On- line. Association for Computational Linguistics.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022a. Llm.int8(): 8-bit ma- trix multiplication for transformers at scale. CoRR,
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. GPTQ: accurate quantization for generative pre-trained transformers. CoRR, abs/2210. 17323.
abs/2208.07339.
Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2022b. 8-bit optimizers via block-wise quantization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Iker Gareía-Ferrero, Rodrigo Agerri, and German Rigau. 2022. Model and data transfer for cross-lingual se- quence labelling in zero-resource settings. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2022, pages 6403-66166 Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguistics.
Jacob Devlin, Ming-Wei Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Pengeheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: decoding-enhanced bert with disentangled attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net.
Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings on parallel corpora. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 2112-2128, Online. Association for Computational Linguistics.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameteriza- tion of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644-648, Atlanta, Georgia. Association for Computational Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi- task benchmark for evaluating cross- cross-lingus lingual gener- alisation. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 441 4411-4421. PMLR
Maud Ehrmann, Marco Turchi, and Ralf Steinberger. 2011. Building a multlingual named entity-

---

Kuan-Hao Huang. I-Hung Hsu, Prem Natarajan, Kai- Wei Chang, and Nanyun Peng. 2022. Multilin- gual generative language models for zero-shot cross- lingual event argument extraction. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Papers), pages 4633-4646, Dublin, Ireland. Association for Computational Linguistics.
Jian Ni, Georgiana Dinu, and Radu Florian. 2017. Weakly supervised cross-lingue named entity recog- nition via effective annotation and representation pro- jection. In Proceedings of the 55th Anmual Meeting the Association for Computational Linguistics ( (Vol- ume I: Long Papers), pages 1470-1480, Vancouver, Canada. Association for Computational Linguistics.
Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19-55 51.
Alankar Jain, Bhargavi Paramjape, and Zachary C. Lip- ton. 2019. Entity projection via machine transla- tion for cross-lingual NER. In Proceedings of the 2019 Conference on Empirical Methods in Natu- val Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1083-1092, Hong Kong, China. Association for Computational Linguistics.
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Har- ris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (Se- mEval 2014), pages 27-35, Dublin, Ireland. Associa- tion for Computational Linguistics.
Masoud Jalili Sabet, Philipp Dufter, François Yvon, and Hinrich Schütze. 2020. SimAlign: High qual- ity word alignments without parallel training data using static and contextualized embeddings. In Find- ings of the Association for Computational Linguistics: EMNLP 2020, pages 1627-1643, Online. Association for Computational Linguistics.
Noam Shazeer and Mitchell Stem. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüs- san, Stockhoim, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1603-4611. PMLR.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NIP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282-6293, Online. Association for Computational Linguisties.
Brian Thompson and Matt Post. 2020. Automatic ma- chine tramslation evaluation in many languages via zero-shot paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 90-121, Online. Association for Computational Linguistics.
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79-86, Phuket, Thailand.
Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: hanguage-independee named entity recognition. In COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002).
Bing Li, Yujie Hc, and Wenjin Xu. 2021. Cross-lingure named entity recognition using parallel corpus: A new approach using xlm-roberta alignment. ArXiv preprint, abs/2101..1111
Jannis Vamvas and Rico Sennrich. 2022. NMTScore: A multilingual analysis of canslation-based text similar- ity measures. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 198- 213, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pas- cale Fung. 2021. Crossner: Evaluating cross-domain named entity recognition. In Thirty-Fifth AAAI Con- ference on Artificial Intelligence, AAAT 2021, Thirty- Third Conference on Innovative Applications of Arti- ficial Intelligence, IAAI 2021, The Eleventh Sympo- sium on Educutional Advances in Artificial Intelli- gence, EAAI 1021, Virtual Event, February 2-9, 2021, pages 13452-13460. AAAI Press.
Mengqiu Wang and Christopher Manning. 2014. Cross-lingual projected expectation regularization for weakly supervised learning. Transactions of the Association for Computational Linguistics, 2:55-66.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's 's transformers: State-of -fee--ffthhh-arre the-hhh-rr art natural language processing, ArXiv preprint, abs/1910.03771.
Tobias Mayer, Filena Cabrio, and Serena Villata. 2020. Transformer- bassed argument mining for healthcare applications. In ECAI 2020- 24th European Confer- ence on Artificial Intelligence, 29 August-8 Septem- ber 2020, Santiago de Compostela. Spain, August 29- September 2020- Including 10th Conference on Prestigious Applications of Artificial Intelligence (PAIS 2020), volume 325 of Frontiers in Artificial In- telligence and Applications, pages 2108-2115. IOS Press.
Jiateng Xie. Zhilin Yang, Graham Neubig, Noah A. Smith, and Jaime Carbonell. 2018 Neural cross- lingual named entity recognition with minimal re- sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,

---

pages 369-379, Brussels, Belgium. Association for Computational Linguistics.
Linting Xue, Noah Constant, Adam Roherts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics.
David Yarowsky, Grace Ngai, and Richard Wicentowski. 2001. Inducing multilingual text analysis tools via robust projection across aligned corpora. In Proceed- ings of the First International Conference Human Language Technology Research.
Anar Yeginbergenova and Rodrigo Agerri. 2023. Cross- lingual argument mining in the medical domain. CoRR, abs/2301.10527.
Susan Zhang, Stephen Roller, Naman Goyal. Mikel Arteixe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Out, Sam Shleifer, Kurt Shus- ter, Daniel Simig, Punit Singh Koura, Anjali Srid- har, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: open trained transformer language mod- CORR, abs/2205.0106 0.066..
Ran Zhou, Xin Li, Lidong Bing, Erik Cambria, Luo Si, and Chunyan Miao. 2022. Comner: Consistency training for cross-lingual named entity recognition. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 8438-8444 Association Com- putational Linguistics.

---

A How many candidates do need?
M2M100 models of different sizes. The results show that the size of the translation model does not have a significant impact on the performance of T- Projeection.
Generating candidates is expensive. The number of flops and memory usage increases linearly with the number of beams computed. Generating 20 candidates is twice as expensive as generating 10 candidates. We also need to add the extra workload of computing the similarity between more candi- dates. Figure shows the average Fl score for each task when generating a different number candi- dates. For OTE and NER small improvements are obtained when generating more than 25 candidates. However, in Argument Mining using a large num- ber of candidates hurts T-Projection's 's performance, which performs optimally with just 10 candidates, While the results reported have been obtained gen- erating 100 candidates, which computationally very expensive, this analysis shows that we can use a much lower number of candidates and still achieve similar or even better results.
However, the size of the mTS model used does have big impact on the final performance of the system. Although for OTE and NER switching from 3B to 738M parameter mT5 model pro- duces competitive results, this not the case when applied AM. The overall trend is that when de- creasing the number of parameters results keep decreasing. Summarizing, in order to achieve com- petitive performance for every task T-Projection re- quires a mT5 model with 3B parameters, although 738M parameter model is still competitive for OTE and NER.
C Tunning the Word Alignment Systems
To validate results and further demonstrate the performance of T-Projection, we conduct set of experiments that evaluate the performance of word-alignment systems under different settings, We first compare the annotation projection perfor- mance when using and not using 50,000 parallel sentences as data augmentation for training the word aligners. Note that in Section 5 all the re- sults we show correspond to using 50,000 extra parallel sentences for training the word-alignment systems. As Table 5 shows, using the augmented dataset achieves the best performance. SimAlign (Dou and Neubig, 2021) and AWESOME (Dou and Neubig, 2021) recommend using their systems with paatlllinnuua-bbe---asssse (Devlin et al., 2019) backbone. However, we also test ALM-ROBERTTa xl (Conneau et al., 2020) 3 billion parameter model with SimAlign and XL.M-RBBERTa-larg (355M parameters) model with AWESOME (The released AWESOME code at the time of writing this pa- per doesn't support XLM-RoBERTa-xU) Using produce worse results than us- ing mBERT. These experiments show that we are using the word-alignment systems in their best- performing settings.
OOE
NER Argument Mning
94
scoro
02
02
91
92
10.
08
01
D8
02
1
20
25
30
25
40
45
50
55
05
00 00 85 100
Number of condidates
Figure 5: F1 score when generating a different number candidates.
B Model size vs Performance
Model
OTE
Average
418M
MTT
m2m100
m2m100 100
12B
94.1
91.6
96.0
93.9
m'T5-smal
00.0
--Ss-
220M
72.8
86.2
33.6
64.2
mT5 size
miT -lagee
738M
90.9
90.1
65.3
82.1
mT5-xl
3B
94.1
91.6
96.0
93.9
Table 4: F1 scores of T-Projection when using transla- tion and mT5 models of different size
D MT models vs Laser
We conducted experiments using 12M100-12B (Fan al., 2021), NLLB200-3B (Costa-jussà al., 2022) and prism (Thompson and Post, 2020) model for computing translation probabilities. We also experiment with using LASER 2.0 (Artetxe and Schwenk, 2019, 9) sentence representations in-
We analyze the performance of T-Projection when using an mT5 model and translation sys- tem with different number of parameters. Table shows the average Fl performance across all the tasks and languages. First, we experiment with

---

OTE
NER
AM
Average
System
Data Augmenlation
Backbone
ES
FR
RU
ES
DE
IT
ES
Giza++ (Och and Ney. 2003)
mBERT
76.2
73.8
78.2
71.4
66.6
65.7
86.4
76.8
FastAlign (Dyer al., 2013)
mBERT
72.3
70.4
74.6
60.3
64.0
57.5
84.0
72.4
SimAlign (Jalili Sahet al., 2020)
mBERT
86.7
86.3
87.7
85.4
87.4
81.3
84.1
85.3
AWTSOME: (Dou and Neuhig, 2021)
mAIERT
88.9
89.8
91.2
86.1
89.4
83.0
57.1
77.8
Giza++ (Och and Ney, 2003)
50000
mBERT
77.0
73.3
72.4
73.3
75.3
68.4
86.6
77.7
FastAlign (Dyer et al., 2013)
50000
mAIERT
75.0
72.9
76.9
70.2
77.0
67.0
85.7
77.4
AWESOME (Dou and Neubig, 2021)
50000
mBERT
91.5
91.1
93.7
87.3
90.7
83.1
54.8
78.0
SimAlign (Jalili Sahet ul., 2020)
86.2
86.1
89.5
85.8
88.4
81.2
76.9
83.1
AWESOME (Dou and Ncubig, 2021)
50000
86.1
86.1
87.4
87.2
87.5
83.1
54.8
75.8
I-Projection
95.1
92.3
95.0
93.6
94.0
87.2
96.0
93.9
Table 5: Results of the different word alignment systems when we train with and without data augmentation corpus and different backbone models
OTH
NER
AM
Average
Candidate Scorer
ES
FR
RU
ES
DE
IT
ES
Prism- 745M
91.4
86.8
94.3
93.8
93.4
85.4
96.3
92.7
M2M1 10M1000-10 12B
95.1
92.3
95.0
93.6
94.0
87.2
96.0
93.9
NLLB200-3B
96.6
90.5
95.6
91.0
94.3
87.7
93.9
93.0
LASER 2.0
89.0
80.6
91.3
91.2
91.6
86.5
70.4
82.4
Table 6: Results of T- TT-Prjeection when selecting candidates using translation probability scores with different MT systems vs using the cosine similarity of the multilingual vector representations of the candidates computed using LASER 2.0
stead of the translation probabilities of NMTscore. We encode the source span as well as all the pro- jection candidates using LASER encoder. We then rank them using cosine similarity. Table 6 shows the results. LASER2.0 is competitive when deal- ing with the short labeled sequences in the OTE and NER task. But the performance decreases when dealing with large sequences in the AM task. M2M100, NLLBB00,, and Prism exhibit compara- ble performance, with some of them achieving the best results in specific languages, but overall, their average performance is very similar.
implementation of M2M100, and we use m2m1 100- 12B- 22B-last-ckpt checkpoint of M2M100 released by the authors. We use the firnct-translation function of the NMTscore library to compute the transla- tion probabilities. For MasakhaNER2.0 we use the training script and evaluation script developed by the authors and the same hyper-parament setup than Chen et al. (2023).
F Dataset details
Training details
We train the HuggingFace's (Wolf et al., 2019) im- plementation of mT5 (3 billion parameter model) in the candidate generation step using the following pyper-parameters. Batch size 8, 0.0001 learning rate, 256 tokens sequence length, cosine scheduler with 500 warn up steps and no weight decay. We use AdaFactor (Shazeer and Stern, 2018) optimizer. We train the model for 10 epochs in the OTE task, and 4 epochs for the NER and AM tasks. In the can- didate selection step, also use HuggingFace's
We list the size (number of sentences) of the dataset use in Table 7. Note that all the datasets are parallel in all the languages, and the number of sentences is the same for all the languages.
hnttps://hugginggacc co/facebook/ m2m10e-128-last-ch
chttps://gis thub. www.tttw....eaeeeeeeeeeeeeeeeeee com/masskhane-ioo
masakhane- weaeeeenrbbbbbbaeae httr///lll//aaii//aaaaaaaaaaaaa 0/ 0/scripts/
ctt co//goolee/mt-xl
mdeberta.sh

---

Task
Split
Sentence No
ABSA
ABSA (Peatiki 2014)
Train
2000
ABSA (Pontiki al., 2014)
Test
676
NER
Europari (Agerri 2018)
CoNLLO3 (Tjong Kim Sang. 2002)
Train
14987
CoNLLO3 (Tjong Kim Sang. 2002)
Dev
3466
CoNLLO3 (Tjong Kim Sang. 2002)
Test
3684
MasakhaNEk2.00 (Adelani et al.. 2023)
Test (hau)
1632
MasakhaNER2.00 (Adelami et ul., 2022)
Test (ibo)
2180
MasakhaNER22.0 (Adelani et ul., 2022)
Test (sna)
1772
MavakhaNTER2.0 (Adelami et ul., 2022)
Test (swa)
1882
MasakhaNER2.0 (Adelani et al., 2022)
(xho)
MasakhaNEER2.0 (Adelani al., 2022)
Test (yor)
MasakbaNE82.0 (Adelani 2022)
(nya)
MasakhaNER2.01 (Adelani 2022)
Test (zal)
1669
AbsRCT Neoplasm (Mayer et 2020)
AbsRCT Neoplasm (Mayer 2020)
AbsRCT Neoplasm (Mayer al., 2020)
Text
1251
Table 7:
Size (Number of sentences) of the dataset use to train and evaluate our systems.
For OTE, we use the SemEval-2016 Task As- pect Based Sentiment Analysis (ABSA) dataset (Pontiki et al., 2014). We train T-Projection with the concatenation of the English train and test splits. We evaluate all the systems by projecting the train- ing split. For NER use the English, Span- ish, German, Italian europarl parallel dataset from (Agerri al., 2018). We train T-Projection with the concatenation of the English europarl dataset with the train, dev and test splits the English CoNLL 2003 dataset (Tjong Kim Sang, 2002). We eval- uate the systems by projecting the English NER europarl test splits. For Argument Mining, we use the AbstRCT Neoplasm English dataset (Mayer al., 2020) and the Spanish AbsRCT corpus gen- erated by machine translating the English AbstRCT corpus with DeepL and manually projecting the la- bels. We train T-Projection of the concatenation the English Ncoplasm train, dev and test split. We evaluate the systems by projecting the English Neoplasm train split.
G Hardware used
We perform all our experiments using a single NVIDIA A A100 GPU with 80GB memory. The machine used has two AMD EPYC 7513 32-Core Processors and 512GB RAM.


Reviews:
[{"review_id": "9PvqlI8DZq", "reviewer": "Reviewer_LiTa", "summary": "", "questions": "1) As observed in Table 1, both Giza++ and FastAlign, given their antiquity, might seem unexpected choices for inclusion. Nevertheless, it is intriguing to note their commendable F1 score performance on the AM task. Additionally, there's a notable decline in performance for Span Translation on the AM task. It would be beneficial for the authors to delve into these phenomena for clearer understanding.\n\n2) Upon examination, it appears that T-Projection primarily employs pre-existing translation models. Consequently, one could postulate that enhancements in the underlying translation model could potentially boost T-Projection's performance, correct?\n\n3) It would be instructive for the author to provide a detailed analysis concerning name translation. Specifically, delineating which segments of a name undergo translation, transliteration, or remain unaltered, stratified by each language.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and feedback. \n\nFirstly, we made a thorough revision on  previous state-of-the-art on annotation projections, and no other prior work employs text2text models to generate annotation projection candidates and uses translation models to rank those candidates. Furthermore, T-Projection outperforms any other previously proposed method both in intrinsic and extrinsic evaluations by a wide margin. If lack of originality is going to be claimed, it would be appreciated for the reviewer to provide references to previous works to support the claim.\n\nSecondly, we would like to clarify our use of translation models in T-Projection. Translation models are never used to translate spans of text in T-Projection, they are solely used to rank the projection candidates generated by an mT5 model. During inference, T-Projection uses source sentence labels’ and the unlabeled target sentence as input. Any generated candidate by mT5 that is not a subsequence of the target sentence is discarded (Section 3.2). Therefore, name translation/transliteration does not take place at any step in T-Projection; we use words as they appear in the target sentence. As empirically demonstrated in Section 5.2, this novel combination of using text-to-text language models to generate candidates and  using translation models to select the best candidate is the source of T-Projection's performance. How the target sentence is generated and the potential mistranslations that may arise if they are obtained using machine translation systems are beyond the scope of our paper.\n\nEnhancements to the underlying translation model can improve T-Projection performance, since more accurate translation probabilities would result in a better selection of candidates (section 3.2). In Section 5.2, we compute the upper bound scores of always choosing the correct candidate. This enables us to analyze the pre-trained translation model quality that we use in the candidate selection step.\n\nIt's true that Giza++ and FastAlign are both based on the IBM alignment models from the '90s. However, these models have been the state-of-the-art in many word alignment and annotation projection algorithms and can be considered as baseline systems. While transformer-based word-alignment systems have surpassed these statistical models, they are extremely cost-effective to train and use for inference, and their performance remains competitive. Given their exceptional cost/performance ratio and for historical reasons, we find it valuable to include them as baselines. \n\nThe Argument Mining (AM) task differs from other tasks in the length of the annotated spans. In this task, the annotated spans are notably long. The baselines were computed using the algorithm from Garcia-Ferrero et al., 2022, to generate annotation projections from word-alignment graphs. This algorithm rectifies “gaps” in the word-alignment graph (“If a labeled span in the source sentence is split into two or more parts in the target sentence, we fill the gap, considering the unlabeled words between the parts as labeled words of the same class if the gap is one word”). With longer sequences, the algorithm becomes more robust, and the quality of the word alignments becomes less critical as long as the first and last words of the projected spans are accurate. As for AWESOME, it underperforms in the AM task because it consistently misaligns articles at the beginning of sentences; a detailed study on this was conducted by Yeginbergenova et al., 2023. Regarding the Span Translation baseline,  the performance of this baseline degrades with long annotated spans, as the likelihood of producing a translation that is a subsequence of the target sentence decreases with the number of tokens in the labeled span. The Span Translation baseline was improved by Chen et al,. 2023. We also compare T-projection with their work (EasyProject), although as explained in section Section 4.2, it was only possible for us to compare T-Projection with EasyProject in the extrinsic evaluation. \n\n## References\nGarcía-Ferrero, I., Agerri, R., & Rigau, G. (2022, December). Model and Data Transfer for Cross-Lingual Sequence Labelling in Zero-Resource Settings. In Findings of the Association for Computational Linguistics: EMNLP 2022 (pp. 6403-6416).\n\nYeginbergenova, A., & Agerri, R. (2023). Cross-lingual Argument Mining in the Medical Domain. arXiv preprint arXiv:2301.10527.\n\nChen, Y., Jiang, C., Ritter, A., & Xu, W. (2023). Frustratingly easy label projection for cross-lingual transfer. In Findings of the Association for Computational Linguistics: ACL 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This study offers detailed experimental evaluations, covering both intrinsic and extrinsic tasks, across five Indo-European languages and eight African languages. Furthermore, a thorough analysis elucidates the pivotal roles of both the generation candidate and the candidate selection phases in determining the ultimate performance.", "reasons_to_reject": "The proposed method lacks originality, given that T-Projection predominantly relies on machine translation systems, which historically grapple with the intricacies of translating names. Arguably, the paper would be more impactful if it solely centered on Named Entities. This specificity is justified by the inherent complexities of name translations/transliterations across languages, setting them apart from aspects like Opinion Target or Argument Mining.", "questions_to_authors": "1) As observed in Table 1, both Giza++ and FastAlign, given their antiquity, might seem unexpected choices for inclusion. Nevertheless, it is intriguing to note their commendable F1 score performance on the AM task. Additionally, there's a notable decline in performance for Span Translation on the AM task. It would be beneficial for the authors to delve into these phenomena for clearer understanding.\n\n2) Upon examination, it appears that T-Projection primarily employs pre-existing translation models. Consequently, one could postulate that enhancements in the underlying translation model could potentially boost T-Projection's performance, correct?\n\n3) It would be instructive for the author to provide a detailed analysis concerning name translation. Specifically, delineating which segments of a name undergo translation, transliteration, or remain unaltered, stratified by each language.", "paper_topic_and_main_contributions": "This paper presents T-Projection, an innovative annotation projection methodology that capitalizes on the capabilities of extensive multilingual pretrained text2text language models in conjunction with leading-edge machine translation systems. Through comprehensive experimental evaluations encompassing both intrinsic and extrinsic tasks across five Indo-European languages and eight African languages, our findings underscore the superiority of T-Projection over established annotation projection techniques. Consequently, T-Projection establishes a new benchmark in the realm of annotation projection tasks.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "q32apwvke4", "reviewer": "Reviewer_GnKV", "summary": "", "questions": "a. Can you use llm(gpt3.5-burbo/text-davinci-003/gpt4) with simple end2end prompting or self-consistency prompting under few/zero-shot setting to complete the annotation projection task to prove that the proposed techniques is still a must in the era of llm.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and feedback. \n\nAs far as we are concerned, we don’t agree with the statement that our method being simple or intuitive involves a lack of novelty. The originality of our work resides in the way T-projection combines both text2text and translation models. Characterizing an approach as not novel because it uses pretrained language models and MT models would deem almost every single paper that gets published in NLP as \"not novel\".  We comprehensively reviewed  previous state-of-the-art on annotation projections and we present a new method which outperforms all of them. Therefore, we would appreciate references to previous work on annotation projection that justifies characterizing our work as “incremental” or “not novel”.\n\nOpenAI products are impressive. Unfortunately, as private products we don’t know much about the underlying models, how they were trained or which data was used. On top of that, this product is regularly updated and previous versions are not accessible, therefore, any experiment performed with OpenAI models would not be reproducible. Moreover, ChatGPT has been proven to massively underperform in named entity recognition and other information extraction / sequence labeling tasks (Lai et al.,2023, Tan et al., 2023). Besides, it does only support a few high-resource languages, so it is not suitable for projection annotations into low resource languages such as the African ones we use in our experiments.\n\nIn contrast to ChatGPT, our models, data, and algorithms are publicly available, can be run locally, and our research can be reproduced by any other researcher.  Our system significantly outperforms current state-of-the-art systems and it is close to human performance.  The existence of an opaque model behind a paywall, so demanding that it requires the most powerful GPUs, doesn't diminish our contributions. In fact, we claim that achieving such an outstanding performance with a system that can run on consumer-grade hardware constitutes a remarkable strength of T-Projection.\n\nRegarding the ablation study, due to the 8-page limit, we were unable to include all the experiments in the paper. We selected those we deemed most relevant. In the appendix, you can find additional experiments that provide insight into why T-Projection performs so effectively. These include generating varying numbers of candidates, using different models and parameter sizes, adjusting hyper-parameters, and exploring alternative candidate selection algorithms. We would appreciate feedback on any other ablation experiments you believe might be relevant.\n\n## References \n\nLai, V. D., Ngo, N. T., Veyseh, A. P. B., Man, H., Dernoncourt, F., Bui, T., & Nguyen, T. H. (2023). Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning. arXiv preprint arXiv:2304.05613.\n\nTan, Z., Huang, S., Jia, Z., Cai, J., Li, Y., Lu, W., ... & Jiang, Y. (2023). DAMO-NLP at SemEval-2023 Task 2: A Unified Retrieval-augmented System for Multilingual Named Entity Recognition. SemEval-2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* It proposes a. simple method, T-Projection, for cross-lingual annotation projection that achieves new state-of-the-art results. The technique is innovative in leveraging both multilingual pretrained language models and machine translation models.\n* The gains over prior work are substantial. This is a large advance for a task where progress has been incremental.\n* The method is evaluated extensively on 3 different sequence labeling tasks and 5-13 languages. The consistent sizable gains demonstrate the general applicability of the approach.\n* The writing clearly explains the limitations of prior projection methods and how T-Projection addresses them. The overall paper is well-structured and easy to follow.", "reasons_to_reject": "* The core idea lacks sufficient novelty. Using pretrained language models and machine translation models for cross-lingual transfer is naive and intuitive.\n* The ablation studies are limited. More experiments could be done to provide insight into why this method works so well.\n* The gains, while substantial, come from access to very large pretrained models. It is unclear if the techniques proposed are the key factors behind the improvement. If you substitute mt5 with gpt3.5-burbo/text-davinci-003/gpt4, maybe just simple end2end prompting can work better.", "questions_to_authors": "a. Can you use llm(gpt3.5-burbo/text-davinci-003/gpt4) with simple end2end prompting or self-consistency prompting under few/zero-shot setting to complete the annotation projection task to prove that the proposed techniques is still a must in the era of llm.", "paper_topic_and_main_contributions": "This paper presents T-Projection, a new method for projecting sequence labeling annotations like named entities from one language to another for cross-lingual transfer learning. The key ideas are to use a text-to-text transformer to generate label projection candidates and then rank candidates using translation probabilities from a machine translation model.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xUcILWeMw5", "reviewer": "Reviewer_nRSb", "summary": "", "questions": "I think this method heavily relies on the transferability of the mT5 model. Given a mT5 model trained in English, its inference on another language can have errors, for example, false positive and false negative.\n\nI guess the false positive can be fixed by transition probability. Therefore the precision of this method should be high. But how about the recall? If some entities are missed, they can't be recovered, right? Is this a problem in practice? ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your comments and feedback. Regarding to your question about recall, it can be controlled by adjusting the number of generated candidates. Generating more candidates reduces the number of False Negatives. However, as the candidate set expands, it becomes noisier, leading to more False Positives, making the selection of the correct candidate increasingly challenging. In Section 5.2, we experimented with both generating just one candidate and considering every possible subsequence of the target sentence as candidates. We generated 100 candidates per entity for all our experiments. This decision was informed by a preliminary analysis of the performance at the candidate generation stage. In the appendix A, we conducted experiments with varying numbers of candidates, ranging from 5 to 100, and found that 10-25 candidates is the optimal range. We chose not to rerun all the experiments with the optimal number of candidates, as selecting a hyperparameter based on test set results is neither fair nor good research practice.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method is intuitive and makes sense. I like the design of symmetrized translation probability.\n2. The experimental setup is reasonable and well executed. Strong baselines are included, and standard models are used.\n3. I think the experimental results well support their claim.\n4. The analysis is informative, showing why this method works. \n", "reasons_to_reject": "N/A", "questions_to_authors": "I think this method heavily relies on the transferability of the mT5 model. Given a mT5 model trained in English, its inference on another language can have errors, for example, false positive and false negative.\n\nI guess the false positive can be fixed by transition probability. Therefore the precision of this method should be high. But how about the recall? If some entities are missed, they can't be recovered, right? Is this a problem in practice? ", "paper_topic_and_main_contributions": "This paper studies annotation projection --  a data argumentation technique to improve the performance of sequence labeling tasks in low-resource languages.  This task assumes the existence of labeled data in high-resource language and un-labeled in low-resource language. The goal is to annotate the data in low-resource languages automatically. \n\nThe authors propose a method called T-projection, which consists of two steps: (1) candidate generation by bream search using a fine-tuned mT5 model; (2) candidate selecting using translation probability.\n\nThe authors perform intrinsic evaluation by evaluating the accuracy of label projection. The proposed method outperforms the word alignment-based methods.\n\nIn the extrinsic evaluation, by incorporating labeled data generated by T-projection, the authors demonstrate a significant performance boost in three downstream sequence labeling tasks, including opinion target detection, NER, and argument mining, on five Indo-European languages and eight African languages supported by mT5. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The article presents a new method for projecting annotations, which improves on the performance of the state of the art. The reviewers suggested a number of experiments to better understand and analyse the results, which I think are worthwhile, which is why I think this article is only “borderline”.
Decision: Accept-Findings