Constrained Decoding for Cross-lingual Label Projection
Authors: Duong Minh Le
Yang Chen
Alan Ritter
Wei Xu
Venue: ICLR 2024 poster (2024)
Keywords: constrained decoding
label projection
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Zero-shot cross-lingual transfer utilizing multilingual LLMs has become a popular learning paradigm for low-resource languages with no labeled training data. However, for NLP tasks that involve fine-grained predictions on words and phrases, the performance of zero-shot cross-lingual transfer learning lags far behind supervised fine-tuning methods. Therefore, it is common to exploit translation and label projection to further improve the performance by (1) translating training data that is available in a high-resource language (e.g., English) together with the gold labels into low-resource languages, and/or (2) translating test data in low-resource languages to a high-source language to run inference on, then projecting the predicted span-level labels back onto the original test data. However, state-of-the-art marker-based label projection methods suffer from translation quality degradation due to the extra label markers injected in the input to the translation model. In this work, we explore a new direction that leverages constrained decoding for label projection to overcome the aforementioned issues. Our new method not only can preserve the quality of translated texts but also has the versatility of being applicable to both translating training and translating test data strategies. This versatility is crucial as our experiments reveal that translating test data can lead to a considerable boost in performance compared to translating only training data. We evaluate on two cross-lingual transfer tasks, namely Named Entity Recognition and Event Argument Extraction, spanning 20 languages. The results demonstrate that our approach outperforms the state-of-the-art marker-based method by a large margin and also shows better performance than other label projection methods that rely on external word alignment.

Published as a conference paper at ICLR 2024
CONSTRAINED DECODING FOR CROSS-LINGUAL LABEL PROJECTION
Duong Minh Le, Yang Chen, Alan Ritter & Wei Xu Georgia Institute of Technology (dminh6, yange)@gatech..em edu, falan. ritter, wei. sei.xux@cccgateeee edu
ABSTRACT
Zero-shot cross-lingual transfer utilizing multilingual LLMs has become popular learning paradigm for low-resource languages with labeled training data. How- ever, for NLP tasks that involve fine- fine-grained predictions on words and phrases, the performance of zero-shot cross-lingual transfer learning lags far behind su- pervised finettuiig methods. Therefore, it is common to exploit translation and label projection to further improve the performance by (1) translating training data that is available in a high-resource language (e.g.. English) together with the gold labels into low- low-resource languages, and/or (2) translating test data in low-resource languages to a high-source language run inference
then projecting the pre- dicted span-level labels back onto the original test data. However, state-of-the-an marker-based label projection methods suffer from translation quality degradation due to the extra label markers injected in the input to the translation model. In this work, we explore a new direction that leverages constrained decoding for label projection to overcome the aforementioned issues. Our new method not only can preserve the quality of translated texts but also has the versatility of being applica- ble to both translating training and translating test data strategies. This versatility is crucial as our experiments reveal that translating test data can lead to consider- able boost in performance compared to translating only training data. We evaluate on two cross- loossllnggul transfer tasks, namely Named Entity Recognition and Event Argument Extraction, spanning 20 languages. The results demonstrate that our approach outperforms the state-of-the-art marker-based method by a large margin and also shows better performance than other label projection methods that rely on external word alignmen""
INTRODUCTION
Large language models (LLMs) have demonstrated the potential to perform a variety of NLP tasks in zero or few-shot learning settings. This is attractive because labeling data is expensive anno- tating fine-tuning data across many languages for each task is not feasible. However, for traditional NLPP tasks that involve word/phrase-level predictions, such as named entity recognition or event extraction, the performance of zero and few-shot learning lags far behind supervised fine-tuning methods that make use of large amounts of labeled data (Lai et al., 2023). Prior work has therefore trained multilingual models that support cross-lingual transfer from high-resource language (c.g., English), where fine-tuning data is available to many low-resource languages where data may not be available (e.g.. Bambara, which is spoken primarily in Africa). Encoder-based LLMs such as XLM-RoBERTa (Conneau et al.) (020)) or mDeBERTa (He et al. (02T)) work surprisingly well for cross-lingual transfer, yet the performance of models that are fine-tuned target-language data is still significantly better (Xue et al. (20TT). Motivated by this observation, we present a new approach automatically translate NLP training datasets into many languages that uses constrained decoding to more accurately translate and project annotated label spans from high to low-resource languages.
Our approach builds on top of EasyProject (Chen et al. 2023a), simple, yet effective state-of-the- art method for label projection, that inserts special markers (see Figure la) into the source sentences
Our code is available at: https::/gittubb.com/duuee

---

Published as a conference paper at ICLR 2024
English
Only France lioe and Britain boo backed Fischler Jecn 's proposel.
Only France and Britain becked Fischier 's propoeal.
Only [ (Fance Joo and Britain JLoe backed [Fischier Jren proposall
Tanslate
France] [ (Grande-Bratagee ] doran de
Tomplate
Translate
Codec
[Fischler] ] ka walepumandan deme.
Match span Britain wa Grande- Bretagne
Constrained decoding
Iolentical
Bambara
France June ni Grande Bretagne lian daran?
Faransl ni Anglletari daran de Fischlar ka lapini deme...
Faransl June ni Anglleteri Joo daron
yo Fischier Iren walepumandon dome...
Fischler Iren lapini deme...
(a) EasyProject (Chen et al. 2023a
(b) CODEC: Translate w/o markers then insert markers with constrained decoding
Translate w/ markers
Figure 1: The goal of label projection is to automatically construct annotated data in low-resource language (e.g.. Bambara) by translating from annotated data in high-resource language (e.g., En- glish) while preserving the span-level labels. EasyProject (Left): The source sentence is first in- jected with marker pairs around entities and then translated to the target language; a span-matching step is performed to map each span to the corresponding label (i.e., types of entities). This method has issues related to the translation quality (e.g., the word "France" is not properly translated), due to the existence of markers in the input to the translation model. CODEC (Right): The source sen- tence first translated to the target language, given the source sentence without marker injected, CODEC then performs constrained decoding to insert markers to the translated sentence.
to mark annotated spans, then runs the modified sentences through a machine translation (MT) sys- tem, such as NLLB (Costa-jussà et al.] (022) or Google Translate. A key limitation of EasyProject, as noted by Chen et al. (2023a). is that inserting special markers into the source sentence then trans- lating it degrades the translation quality; nevertheless, EasyProject was shown to be more effective than prior work for label projection that largely relied on word alignment (Yarmohammadi et al.) 2021 To address the problem of translation quality degradation, in this paper, present new approach, Constraint Decoding for Cross Cross-lingual Label Projection (CODEC), for translating training datasets using a customized constrained decoding algorithm. The training data in the high-resource language is first translated without markers followed by a second constrained decoding pass to inject the markers (see Figure Lb). Since the source sentence does not include markers during the transla- tion phase, the final translated text quality from CODEC is preserved. The second decoding pass of CODEC relies on translation model that is conditioned on the modified input sentence with markers (thus is noisier) in order to find the appropriate positions for inserting markers. Using specially designed constrained decoding algorithm, however, we can retain the high-quality translation while having the right number of labels projected by enforcing both as constraints during decoding.
In essence, CODEC only explores the search space which contains valid hypotheses, i.e., translated outputs that conform to (i) the high-quality translation from the first decoding pass without markers' interference and (ii) having the correct number of markers inserted. A brute-force enumeration of all possible such hypotheses is intractable, as the number of sequences that would need to be scored using the translation model is O(n2'), where n is the sequence length and m is the number of la- beled spans to be projected, we show in $3 We therefore design constrained decoding algorithm method (Stahlberg & Byrne 2019). in which a depth-first search is
based on the branch-and- conducted to identify a lower-bound on the best complete hypothesis, and branches that do not have any solutions with a better score than the current lower bound are pruned from the search space. However, even when pruning branches using this bound, decoding time is still prohibitively long. To speed up decoding, we introduce a new heuristic lower bound, which removes branches more aggressively. We also introduce technique to prune unlikely positions for the opening markers in advance. Putting everything together, compared to exact branch-and-boume search, our proposed method significantly reduces decoding time with only a slight drop in performance in a few lan- guages. For example, for the Bambara language, CODEC is about 60 times faster than exact search, while only losing 0.6 absolute F1, a 1.1% drop in performance.
We conduct extensive experiments to evaluate CODEC two popular cross-lingual tasks (i.e., Named Entity Recognition and Event Argument Extraction), covering 20 language pairs. In our experiment, CODEC and other label projection baselines used to project the label from English datasets to their translated version to augment the data in the target language, which is referred to as translate-train (Hu et al. 2020). The results demonstrate that on average, the model fine-tuned
2

---

Published as a conference paper at ICLR 2024
on Copec-augmented data outperforms models fine-tuned the data produced by other label projection basclines by a large margin. This reinforces our hypothesis that preserving translation quality is essential and constrained decoding can improve the accuracy of label projection. More- over, since CODEC separates two phases of translation and marker insertion, it can also be used to improve cross-lingual transfer by using machine translation inference time, sometimes referred to as translate-test (Artetxe et al. 2023). This approach translates test data from the low-resource lan- guage to the high-resource language, uses fully-supervised NLP model to automatically annotate the translation on the high-resource side, then projects the annotations back to the original language. Experiments show that, compared to translate- train alone, using CODEC in the translate-tess setting further boosts cross-lingual transfer performance in the Named Entity Recognition task.
2 LABEL. PROJECTION AS CONSTRAINED TRANSLATION
In this section, we describe how we formulate Label Projection as a constrained generation problem before proposing to tackle it by designing a constrained decoding algorithm (detailed in $ Given a source sentence with label spans and its translation in the target language, our goal to map the label spans from the source to the target sentence. Formally, let be the source sentence with m labeled spans (e.g., m named entities) and markk be the same text as x but with m pairs of special markers (e.g. square brackets) surrounding every label span. In previous work, an MT model will find the translation yarrk of the sentence with markers inserted, mark: mark arg max log P+ (y]lxmmnk))
(1)
where T is machine translation (MT) system. However, as previous work has found that inserting markers degrades translation quality (Chen et al. 2023a), we introduce variant of the approach.
Our approach uses the translation of the source sentence without markers, which will act as a tem- plate to be injected with markers later to create the annotated sentence in the target language:
y lmpl arg max log Pr (y(z)
(2)
In general, without the interference of special markers in the input, the translation y fmpi is expected to have higher quality than markk, as shown in (Chen al. 2023a). Let ) be the set of all possible valid hypotheses with m marker pairs injected into y " In our work, we do not consider the case of overlapped or nested label spans, thus the size of y is ("=²?m²) or O(n2m), where n is the length (# of tokens) of ymm!!
Our goal is find the best hypothesis y from V. in which all the markers are inserted at correct positions into rmpp with each span found in y mapped to its corresponding label in x mank We can cast this task as constrained translation problem, which enforces two constraints: (i) y" contains exactly m valid pairs of markers as x: mark (i.e., no mismatched brackets) and (ii) the plain text (with all markers removed) of y the exactly the same as fyfppp Consequently, we can solve this problem by designing specialized constrained decoding algorithm, which explores every hypothesis in the search space y and finds the one with the highest generative probability:
arg max log Pr (yllamorrk ;y (mpl) y
yEY
log P- (y) Amply
log Pr (+ (y x: mark, k mmp)) Σ
(4)
Starting with a translation prefix E (e.g., a language code <bam>), the decoding algorithm will itera- tively expand the hypothesis; when the end-of-sentence token (i.e., 
) is generated, candidate projection of the labeled spans is found.
CONSTRAINED DECODING
In this section, we will propose a constrained decoding algorithm, CODEC, particularly designed for cross-lingual transfer learning. It uses approximation to reduce the computational complexity of the original problem and ranking method to identify hypotheses with accurate span projections. CODEC has three main steps: (1) prune all unlikely gpening-marker positions, (2) search for k hypotheses with the highest probability, and (3) re-rank to find the best hypothesis.
3

---

Published as a conference paper at ICLR 2024
Input: "Onty
1 and [Brtsii|]
aransi Angleeton donn Fischier lapitt dome.
Fischler's proposal
Fisentor 's proposal
Step 1: Prune opening marker positions
Step Search for hypothesss with highest probabilities
log Ply,
1,27
(Condilioeed on source lec()
0.57
0.56
0.54
Farandl
log PLy
1x2 m [Conditionn ad on source best wi markers
2044
.0.0
.0.0
X
Farursi
Log probabiitty 13.53 fafs bebw the Jnver bound (..)
001
2.04
5, 1p
In! Opacing posltions "Frvons"
Compon
Pruse
Curnet bed
Figure 2: The first two steps of CODEC. Step 1 (Left)): CODEC prunes search branches based on unlikely opening marker positions in the target language by comparing probabilities conditioned the source language with and without span markers inserted. Step (Right): A branch-and-bound search algorithm used find & hypotheses with the highest probabilities (k 3). From each node of the search tree (e.g.. "Faransi"), CODEC expands to the next token from the translation template gmmp! (e.g., "nni") or a marker (i.e., "I" or "I"). A search branch is pruned if its score falls below a heuristic lower bound. Two branches of different lengths might have different values of lower bound (e.g., the lower bound for the top and bottom branches are -3.1 and -2.8, respectively).
3.1 PROBLEM APPROXIMATION
As discussed in 2 if we naively project m label spans into the template translation y/mpl the research space y has a size of O(n2m) where n is the number of tokens in y Therefore, instead of solving this m-projection problem, we propose to address 1-projection problems, whose search space is only (2+²) or O(n²) each, and approximate the original solution by combining the solutions from the m problems. In other words, we will project m sentences, each of which only contains one label span surrounded by one marker pair. This approximation also brings another merit, since there is only one span projected at a time, we can identify the label of the span (e.g.. types of entities) on the target language without the need for the label matching step in EasyProject (see Figure Tap.
3.2 PRUNING OPENING-MARKER POSITIONS
To further reduce the search space and speed up the decoding algorithm, propose heuristic method to detect which position in the translation template can be inserted with the opening marker (e.g., left square bracket '['). During the search process, all hypotheses that have the opening marker inserted in other positions will get pruned. The key idea of our method is to track the difference in conditional word (more accurately subword token) probabilities of generating the same translation template, when being conditioned on two different inputs: the source sentence with and without markers. The intuition is that, if we decode the translation template but conditioned on the marker sentence, at the position that needs to be inserted an opening marker, the model would give a high probability to this token, and consequently, would assign low probability to the token from the template. This intuition is illustrated in Figure (Step 1). Formally, we define A, the difference of the two log- prgabbbbaillisess when generating the 7th token from the template:
A log P(y)
impl empl Ves x markk - llo P P(y tmpl impl pyeeo ,x)]
(5)
where tmpll indicates all tokens before the 7hh token and, from hereon, P is used in place of P, for simplicity. We determine the possible positions of the opening marker by choosing all with a high value of A; (i.e., larger than threshold). All hypotheses, which have the opening marker inserted at other positions, are pruned during the search.
3.3 SEARCHING FOR TOP-K HYPOTHESES
According to our preliminary study, the hypothesis with the highest probability in many cases is not necessarily the best output, i.e., the one with markers injected correctly. This is typically called
4

---

Published as a conference paper at ICLR 2024
(translation) model error in machine translation research, in contrast search enror, where the highest scored sequence is a good translation but the decoding algorithm (often approximate search) fails to find it. Previous research on translation decoding (Stahlberg & Bymee 2019 Eikema & Aziz, 2020) also found that model errors occur very often when using the exact search.
To address this problem, we propose first find top-k hypotheses with the highest probabilities, then use a ranking function (details in $3.4) to choose the best one (i.e., re-ranking). The main idea to traverse the entire search space in depth-first order and adopt the branch-and-bound method Stahhbeeg & Byrne] 2019) to reduce the decoding time see Figure 2 for an illustration. Compared to the search algorithm in the previous work (Stahlberg & Byme 2019), CODEC uses different search strategy to enforce additional constraints for label projection. More specifically, in order for the generated text to follow the translation template, at each decoding iteration, the decoding scheme only considers at most two candidates for the next token: the next token from the template and the next span marker. Besides the search strategy, CODEC also introduced a new heuristic lower bound. Before presenting the lower bound of CODEC, we first discuss the one introduced by [Stahlberg & Byrne (019) and the reason why we need to define a new lower bound for this task. The purpose of the lower bound is to prune branches that do not have any solutions with better score. The score of a branch, which has expanded a sequence to the length of j (i.e., F1.5), ), is the generative log- probability of this sequence (i.e., log P(y P(yusslamark) ) = >i I log P(y (illyye,, mmrk))) Previous work the current best hypothesis. Generally,
has defined an exact lower bound as the log-probability let y be the hypothesis whose tog-probability is the hthh highest at specific time during the search (Is = when only need to find the best hypothesis). Let L be a list containing generative log- probabilities of all prefix of yd (i.e., L log P(. (iliy/lmarr (mmarr)). The exact lower bound is defined as:
(6)
One problem with the exact lower bound is that it uses the probability of a complete hypothesis (i.e., the hypothesis ends with 
) define the lower bound. This probability is often small, and thus cannot prune off short partial hypotheses early enough. As illustrated in (nguree[(tte 2), the exact lower bound with k 3 at this search stage is 36..7, much smaller than the proobbbbaisl of other expanded partial hypotheses. Therefore, we propose a heuristic lower bound, whose value changes according to the length of each partial hypothesis (i.e., having a larger value when compared to shorter partial hypotheses), to help expedite the pruning. The heuristic lower bound for partial hypothesis of length j is defined as:
7 = I
(7)
d = min (max (j + 5, 5.4).1v
(8)
where y is the position of the opening marker in y² (g = if the marker is not selected yet) and a hyperparameter of the lower bound. The lower bound with larger value of á is closer to the exact lower bound. With the new lower bound, CODEC can prune off the expanding hypotheses as shown in Figure 2 (Step22 2).
3.4 RE-RANKING
The goal of this step is to pick the best hypotheses, which have markers injected in the correct positions, among the top-k candidates found in the previous step ($3.3). Two scores are used for this purpose: (i) hypothesis-level score: the generative log-probability of a hypothesis (Eq 4 and (ii) span-level score: the log-probability of generating the original span given the label span found in hypothesis. In particular, for label span ere in the source sentence, we have & hypotheses of projecting erre to the target language after the search step. Let e; be the label span found in the jth hypothesis. The span-level score of the hypothesis 1th is defined as:
fgt
Sppan log Preare tgt
The same MT model is used to compute both hypothesis-level and span-level scores. The k hy- potheses are first ranked by the hypothesis- score. The span-level score is then used to re-rank all hypotheses, whose label spans are equal to are subsequences of the label span of the current top-1 hypothesis. The best hypothesis is the new top-1 after the re-ranking. More details about CODEC can be found in the Appendix
5

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
We evaluate CODEC on two benchmark cross-lingual NLP tasks, including Named Entity Recogni- tion and Event Argument Extraction, covering 20 languages in total.
4.1 DATASETS
For Named Entity Recognition (NER), we use English CONLL03 (Toong Kim Sang 2002 as train/dev data and use MasakhaNER2.0 (Adelani et al.) 2022) which consist of for 20 African languages as test data. Following Chen et al. (2023a), we consider three main types of entities (i.e., person, organization, and location) in the dataset and skip Ghomala and Naija (two languages that NL..B does not support). We conduct experiments on MasakhaNE.R2., instead of WikiANN (Pan et al. 2017). 7). the latter was automatically constructed dataset and contains higher level of noises (Lignos et al., 2022). For Event Argument Extraction (EAE), we use the ACE-2005 (Doddington et al. 2004). a multilingual dataset that covers English, Chinese, and Arabic. The task is to identify the event arguments and their roles in the text given the event triggers. We use the English training sets of CoNLLO3 (Tjong Kim Sang, 2002) and ACE-2005 to act as the source language to perform cross-lingual transfer for the NER and EAE tasks, respectively.
data
4.2 EXPERIMENTAL SETTINGS AND RESULTS
Setup We use NLLB (No Language Left Behind) as the translation model (Costa-jussà et al.)
2022) in our experiments. We fine-tune mDeBERTa-v3 (276M) to act as a NER tagger following
(Chen et al.) 2023at; and fine-tune - (Xue et al. 202T) following the X-Gear framework
(Huang et al. 2022) for EAF. For a direct comparison with existing work (Chen et al. 2023b a
Huang et al. 2022), we report the average F1 scores across five random seeds for NER and three
random seeds for EAE. More details are provided in the Appendix BB
We compare CODEC with the following baselines: (1) EasyProject (Chen et al. 2023a): state-of-the-art marker-based label projection approach; (2) Awes-align: an alignment-basee based label projection approach, in which the state-of-the-art word aligner (Dou & Neubig (021 is used to align the text spans between the source and target sentences; (3) FTEE: the multilingual models (the ones described in "Setup" above) stned directly on the English data, which have shown to be a very strong baseline for lingul transfer; (4) GPT-4: in the cross-lingual NER task, we also prompt GPT-4-061 (Achiam et al. 2023) to annotate each word using BIO scheme, following Chen al. (2023b). Due to cost constraints, we only evaluate GPT-4 on 200 examples for each language of MaR2.. EasyProject uses an NLLB model fine-tuned on synthetic dataset as the MT model, in which the marker pairs are inserted around the label spans in both source and target sentences. We also use the fine-tuned version of NLLB-600M, that they provide in CODEC for a more fair comparison.
Baselines
Results Table Table[ shows the performance of different approaches on the test set of MasakhaNER2.0. O. On average, label projection methods outperform both GPT-4 and FT by large margin. Among the label projection methods, CODEC achieves the best results in both translate-train and translate- test settings, average. Compared to EasyProject, similar method to CODEC but using a dif- ferent decoding algorithm, CODEC has achieved better or comparable performance in most of the languages, with improvements of at least 1.5 F1 in nine languages. The improvement is especially evident for the chiShona language, which is +16.5 FI. Having higher translation quality may be the reason why CODEC can outperform EasyProject in many languages. One noticeable issue with the translation of EasyProject is that the label spans are more often left untranslated (similar to the example Figure Ta). Other types of error from EasyProject and Awes-align are demonstrated in Figure 4 in the Appendix. In addition, we observe that cross-lingual NER in African languages fa- vors translate-test approaches more than translate-train overall. Compared to when being used in the translate-train setting, both CODEC and Awes-align have significant increases in the translate-test. In the translate-tess setting, CODEC also shows better results than the latter method in 13 out of 18 lan- guages. Another interesting observation is that FT achieves the best performance in Chichewa and Kiswahili, likely because the entities in these two languages are often kept the same as their English form. Therefore, fine-tuning the NER model on English data only is sufficient for the two languages

---

Published as a conference paper at ICLR 2024
Table 1: Cross-lingual NER results on the test set of MasakhaNER2.0 (the best Fl score of lan- guage is in bold, Arr: calculated against FTE,. GPT-4 is evaluated on subset of 200 examples from the test set of each language due to cost constraints). On average, CODEC outperforms other approaches by large margin especially when used with the translate-test strategy; EasyProject is not applicable to translate-test.
Translate-train
Translate-test
Lang.
GPT-4"
FT
Awes-align
EasyProject
CODEC (Ar)
Awes-align
CODEC (AFt)
Bambara
46.8
37.1
45.0
45.8
45.8 (+8.7)
50.0
55.6 (+18.5)
Ewe
75.5
75.3
78.3
78.5
79.1 (+3.8)
72.5
79.1 (+3.8)
Fon
59.3
61.4
65.5 (+15.9)
61.4 (+11.8)
Hausa
70.7
71.7
72.7
72.2
72.4 (+0.7)
70.0
73.7 (+2.0)
Igho
51.7
59.3
63.5
65.6
(+11.6))
77.2
72.8 (+13.5)
Kinyarwanda
59.1
66.4
63.2
71.0
71.2 (4-.8)
64.9
78.0 (+11.6)
Luganda
73.7
75.3
77.7
76.7
77.2 (+1.9)
82.4
82.3 (+7.0)
Luo
55.2
35.8
46.5
50.2
49.6 (+13.8)
52.6
$2.9 (+17.1)
Mossi
44.2
45.0
52.2
53.1
$5.6 (+10.6)
48.4
50.4 (+5.44
Chichewa
75.8
79.5
75.1
75.3
76.8 (-2.7)
78.0
76.8 (-2.7)
chiShona
55.9
(+37.2)
67.0
78.4 (+43.2)
Kiswahili
82.6
87.7
82.4
83.6
83.1 (-4.6)
80.2
Setswana
62.0
64.8
73.8
74.0
74.7 (49.9)
81.4
80.3 (+15.5)
Akan/Twi
52.9
50.1
62.7
65.3
64.6
72.6
73.5 (423.4)
Wolof
62.6
44.2
54.5
58.9
63.1 (+18.9)
58.1
67.2 (+23.0)
isiXhosa
69.5
24.0
61.7
71.1
70.4 (146.4)
52.7
69.2 (145.2)
Yorubu
58.2
36.0
38.1
36.8
41.4 (45.4)
49.1
58.0 (+22.0)
isiZulu
60.2
43.9
68.9
73.0
74.8 (+30.9)
64.1
76.9 (+33.0)
AVG
60.4
54.5
63.6
64.9
67.1 (+12.7)
65.8
70.4 (+16.0)
in this setting. As for the EAE task, we observe a similar trend in Table 2 that CODEC outperforms the other label projection approaches average. In each language, CODEC also has comparable or better performance than each baseline. We only report the performance in the translate-train set- ting for the EAE task, because the typical experiment setup requires the gold event trigger input, but for translate-test, this information in English can only achieved by another pass of label pro- jection (from a target language to English) during inference time. More discussions about the two cross-lingual tasks are in Appendix C
4.3 ABLATION STUDY
In this section, we study the efficiency and accuracy the two heuristic steps in CODEC.
Setup We modify each module of CODEC and evaluate the performance of each setting in translate-dev on a sample MasakhaNER2.00 dev set for five languages (i.e., Bambara, Fon, Mossi, Yoruba, and isiZulu). We evaluate the perfor- mance of exact (+re-rank): exact search algo- rithm, similar to CODEC except using the exact lower bound (diseussed in (3) and does prune the opening marker positions; exact: similar to exact (+re_rank) but does not conduct the re-ranking and only return hypothesis with the highest probability; CODEC (b=1), CODEC (Ó=3): CODEC with different value of 6, the hyperparameter of the heuristie lower bound in Eq. (8; CODEC (6=1+)), CODEC (6=3+(): CODEC with different val- ues of 8 and does pruning the opening marker positions. Since the exact search takes an extremely long time to complete, we sample only 100 examples with up label spans for each language.
Table 2: Fl scores of different methods on ACE- 2005 dataset in the translate-tran setting.
Lang.
FT
Awes-align EasyProject CODEC
Arabic 44.8
48.3
45.4
48.4
Chinese 54.0
57.3
59.7
59.1
AVG 49.4
52.8
52.6
53.8
Results The performance and decoding time of different search settings are shown in Figure Ba and Figure 3b respectively. Firstly, compared to exact (+re-rank), only returning the hypothesis with the highest probability (exact) resulted in a significant drop in performance in all languages. This observation shows the necessity of the re- re-ranking step to choose the best hypothesis. In terms of the heuristic lower bound, CODEC (6=3) has roughly the same FI as exact (+re-rank), showing that the heuristic lower bound with = is good approximation of the exact lower bound while having much better decoding speed. Finally, pruning the opening marker positions further speeds up the

---

Published as a conference paper at ICLR 2024
70
exait 1+ rer31kk1
Codtec 15 3)
[8-3) 2)
Codac 2-11
Coder (5 3-]1
exact (#re-rank)
Codoc 15 31
--- Codec (6 3-11
Codec (8
Codec 15 1+[
700
200
11
200
100
50-
bam
ton
mos
yor
(a) Fl scores on dev (translate-dev).
(b) Average decoding time per example.
Figure 3: Ablation study on MasakhaNER2.00 dev set of different search settings for five languages, including Bambara (bam), Fon (fon). Mossi (mos). Yoruba (yor), and isiZulu (zul). exact (+re- rank): exact search with re-ranking; exact: exact search and return the top-1 hypothesis. '8' is the hyperparameter of the heuristic lower bound. [' indicates pruning unlikely opening-marker positions beforehand. Compared to constrained decoding with exact search, CODEC with 8 = 3 significantly reduces the decoding time, while retaining the performance measured by Fl scores.
decoding speed by at least about 1.4 times when 8 = 3, while causing only slight drop of F1 (the biggest drop 0.7 absolute F1 in Yoruba). One interesting observation that pruning the possible opening-marker position can boost the performance in many cases (i.e., all languages when ó 1; Fon, Mossi, and isiZulu when = 3). One explanation is that the translation model is imperfect and may give bad hypotheses high probabilities. sometimes even higher than the probability of the correct hypothesis. With the marker-positinne pruning step, those noisy hypotheses are avoided. More analyses about CODEC in Appendix
MANUAL ASSESSMENT OF CODEC AND DISCUSSION
In this section, we analyze the behavior of each sub-component in CODEC. In particular, manu- ally inspect outputs of CODEC when being used in translate-train and translate-test settings for the cross-lingual NER task from English to Chinese and Vietnamese (100 examples inspected for
each language). We categorize them into error types based on the source of errors: (1) Translation:
error from the MT model, (2) NER: error from the NER tagger (translate-test only), (3) CODEC;
error from CODEC. For all studies, we use the translation from Google Translation (GMT) AP (which has better translation quality for these two languages) to focus the analysis on CODEC, other than the MT system. In addition, we also inspect the outputs of CODEC in the scenarios when the outputs from the MT model and NER tagger are perfect (i.e., Oracle). More details are provided in the Appendix +
Results From Table 3 we see that there is a considerable amount errors from CODEC in Chi- nese. Further inspecting the error outputs from CODEC, we observe that more than 60% of the cases, the correct hypothesis is in the top-k found by CODEC, but the re-ranking step fails to re- trieve it. Therefore, studying how to select the best hypothesis one promising direction to improve CODEC further. Also from the table, there are many eтrors from the MT models in the translate- test setting. Vietnamese, most of the translation eтror incorrectly translating the entities, while in Chinese, most of the time, the error is not due to incorrect translation but the translation style. Particularly, different styles of translation to English may change the entity type, e.g., "Chinese journalists" and "journalists from Chinaroc" can be the translation from the same text, but have different entity types. One possible direction to tackle this issue is to use constrained decoding to control the translation style (e.g., use words that mean nation instead of nationality).
6 RELATED WORK
Label Projection There has been research that involves projecting label spans between bilingual parallel texts for cross- lingull transfer. One popular approach is to utilize word alignment models to
https::/c/l oud goo.g com/ rttal/tnnallaaa
8

---

Published as a conference paper at ICLR 2024
Table 3: Manual analysis of CODEC outputs in translate-train and translate-test settings for Viet- namese and Chinese. The number indicates how many examples out of the 100 sampled for each language are correct or have corresponding enrors (one example may have more than one error).
Translate-train
Translate-test
Language
MT
Correct
Translation
CODEC
Correct
NER
Translation
CODEC
GMT
94
51
33
Vietnamese
Oracle
85
GMT
80
6
14
34
36
24
Chinese
Oracle
16
69
0
31
project the label span between the source and target sentence. This approach has been adopted for a
wide range of tasks: Part-of-speech Tagging (Yarowsky & Ngail [200]] Duong et al. 2013 Eskander
jet al.] 2020). Semantic Role Labeling (Akbik et al.] Fei et al.
2020), Named Entity Recogni- 2023 Behzad et al. 2023). and
tion (Ni et al. 2017 Stengel-Eskin et al. 2019 Sarcia-Ferrero et al.
Slot Filling (Xu et 2020). With the advancement of the MT systems, several recent efforts have adopted the marker- based approach, which directly uses the MT system perform the label projec-
tion (Lee et al. 2018 Lewis et al. 2020 Hu et al., 2020 Bomna et al. 2021 Chen et al., 2023a). Between the two directions, the alignment-based approach is more adaptable terms of being ap- plicable to both translate-train and translate-test, and can preserve the translation quality; however, is shown be less accurate than the marker- approach (Chen et al. 2023a). CODEC has the advantages of both approaches. There is also research that takes different directions to transfer label spans, Daza & Frank (2019) presents an encoder-decoder model to translate and generate Semantic Role labels on the target sentence at the same time, while Guo & Roth (222T) proposes to use con- strained decoding to construct the target sentence from pre-translated labeled entities. In parallel to our work, Parekh et al. (2023) also propose to translate the original sentence without markers, be- fore label projection, to address the translation quality degradation. In addition to having access to a translation system, their approach requires available in many languages) to detect the label spans in the target language. Different from their method, we adopt an approach based constrained decoding, which only requires a translation model, such as NLLB, that can support much wider range of low-resource languages. Outside the topic of label projection, Razumovskaia al. (2023) share similar idea of re-purposing multilin- gual models with ours, and introduces a two-stage framework to adapt multilingual encoders to the task of slot labeling, in the transfer-free scenarios, where no annotated English data available.
Instruction-tunee large language model (currently not
Lexically constrained decoding While constrained decoding has previously been explored in ma- chine translation and text generation, it is typically used to constrain the vocabulary or the length of the outputs (Anderson et al.] [017]] Hokamp & Liu, 2017| Post & Vilar 2018 Miao et al.) 2019 Zhang et al.] 2020 Lu et al.) 2021: 2022 Qin et al. 2022). Compared to these past works, CODEC has some important features that make it more suitable for the label projection problem. Firstly, instead of only constraining the occurrence of some words, CODEC constrains the whole de- coding sentence to follow a pre-defined template. This feature important as it allows CODEC to do the projection to a fixed target sentence, which is requirement for being applicable to the translate- test and preserving translation quality during translate-tre train. Secondly, CODEC also constrains the number of occurrences of each token, including the marker, which is essential as it prevents the gen- erative model from dropping markers, addressing a serious issue with the marker-based approach for label projection. The idea of using constrained decoding to guarantee the validity of the output space has also been adopted in the task of semantic parsing (Scholak et al. 2021.
7 CONCLUSION
In this work, we introduced a new approach of using constrained decoding for cross-lingual label projection. Our new method CODEC not only addresses the problem of translation-quality degradation, which is a major issue of the previous also adaptable in terms of being applicable to both translate- -raite--riinnee and translate-test Experiments on two cross- lingual tasks over 20 languages show that, on average, CODEC has shown improvements over strong fine-tuning baseline and other label projection methods.
maakeer-bssed label projection methods, but is
9

---

Published as a conference paper at ICLR 2024
8 ACKNOWLEDGMENTS
We would like thank Vedaant Shah for helping us with the GPT-4 experiments, and we also thank four anonymous reviewers for their helpful feedback on this work. We would like to thank Azure's Accelerate Foundation Models Research Program for graciously providing access to API- based models, such as GPT-4. This research is supported by the NSF (IIS-2052498) and IARPA via the HIATUS program 2022-22072200004). The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of NSF, ODNI, IARPA, or the U.S. Govermment. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXivv2303.08774 2023.
David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba Alabi, Shamsuddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula. Rooweither Mabuya, Bonaventure F. P. Dos- sou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe, Derguene Mbaye, Amelia Taylor, Fatoumata Kabore, Chris Chinenye Emezue, Anuoluwapo Aremu, Perez Ogayo, Catherine Gitau, Edwin Munkoh-Buabeng, Victoire Memdjokam Koagne, Allahsera Auguste Tapo, Tebogo Macucwa, Vukosi Marivate, Mboning Tchiaze Elvis, Tajuddeen Gwadabe, Tosin Adewumi, Orevaoghene Ahia, Joyce Nakatumba-Nabendde Neo Lerato Mokono, Ignatius Ezeani, Chiamaka Chukwuneke, Mofetoluwa Oluwaseun Adeyemi, Gilles Quentin Hacheme, Idris Abdulmumin, Odunayo Ogundepo, Oreen Yousuf, Tatiana Moteu, and Dietrich Klakow. MasakhaNER 2.0: Africa-centric transfer learning for named entity recognition. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 4488 4508, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https: http:///caaatthhooccoo org/ 2022. emnlp main. 298
Alan Akbik, Laura Chiticariu. Marina Danilevsky, Yunyao Li, Shivakumar Vaithyanathan, and Huaiyu Zhu. Generating high quality proposition Banks for multilingual semantic role label- ing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 397-407, Beijing, China, July 2015, Association for Computational Linguistics. doi: 10.3115/v1/P15-1000 1039.
Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. Adapting pre- trained language models to African languages via multilingual adaptive fine-tuiing In Nico- letta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Ni- anwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na (eds.), Proceedings of the 29th International Conference on Computational Linguistics, pp. 4336-4349, Gyeongju, Republic of Korea, October 2022. Inter- national Committee on Computational Linguistics. URL https::/aclanthol..ooo org//
382
Peter Anderson, Basura Fernando. Mark Johnson, and Stephen Gould. Guided open vocabu- lary image captioning with constrained beam search. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pp. 936 945, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.17133/ URL
1098
Mikel Artetxe, Vedanuj Goswami, Shruti Bhosale, Angela Fan. and Luke Zettlemoyer. Revisiting machine translation for cross-lingual classification. arXiv preprint arXiv:2305. /2223005..44.... 2023.
10

---

Published as a conference paper at ICLR 2024
Shabnam Behzad, Seth Ebner, Mare Marone, Benjamin Van Durme, and Mahsa Yarnohammadi. The effect of aligmment correction on cross-lingual annotation projection. In Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII), pp. 244-251, 2023.
Mihaela Bornea, Lin Pan, Sara Rosenthal, Radu Florian, and Avirup Sil. Multilingual transfer learning for qa using translation as data augmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 12583-12591, 2021.
Yang Chen and Alan Ritter. Model selection for cross-lingual transfer. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021.
Yang Chen, Chao Jiang, Alan Ritter, and Wei Xu. Frustratingly easy label projection for lingsllingual transfer. In Findings of the Association for Computational Linguistics: ACL Toronto, Canada, July 2023a. Association for Computational Linguis- tics. doi: 10. 8153866/2222333 fin.222..nnning--a..555557 URLhttps: https////aaaaaahhaaaaeooeo org/ Findings sarrings-acl... .35
2023, pp.
Yang Chen, Vedaant Shah, and Alan Ritter. Better low-resource entity recognition through transla- tion and annotation fusion. arXiv preprini arXiv:2305. 13582, 2023b.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-8451, Online, July 2020. Association for Computational Linguistics. doi: 10. 1w..1866/112222.... aciimmnn.. 747. URLhetea //aclanthology. wttppoyy.rgg//00/o www.aclmmmmnn....mmmmmmmmmmmm..a main. 747
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04 04672, 2022.
Angel Daza and Anette Frank. Translate and label! an encoder-decoder approach for cross-lingual semantic role labeling. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.). Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP--JCNIP), pp- 603-615, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v 100/655////D11--.1
URL
www.olyyorr//99911000 1056
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers). pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Com- putational Linguistics. doi: 10. 00.11863//1/N199 1423. URL https: http://aaaaaatoo.....o.oo org/
N1 N19-1423
George Doddington, Alexis Mitchell, Mark Przybocki. Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. The automatic content extraction (ACE) program tasks, data, and evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC*04), Lisbon, Portugal, May 2004, European Language Resources Association (ELRA). URL //t:://wwww 1rec- conf wwggpp/cceeeiigg//// http://cceei.nnrrrreeeee//////f/ http:///t/00///////////////
Zi-Yi Dou and Graham Neubig. Word alignment by fine-tuning embeddings on parallel corpora. In Proceedings of the 16th Conference of the European Chapter of the Association for Computa- tional Linguistics: Main Volume, pp. 2112-2128, Online, April 2021. Association for Computa- tional Linguistics. doi: 10.18653/v1/209 1aw.l..eacl.mmmmeeemm main. URL https: / htc/2221.eacl-maall
Long Duong, Paul Cook, Steven Bird, and Pavel Pecina. Simpler unsupervised POS tagging with bilingual projections. In Proceedings of the 51st Annual Meeting of the Association for Compu- tational Linguistics (Volume 2: Short Papers), pp. 634 639, Sofia, Bulgaria, August 2013. Asso- ciation for Computational Linguistics. URL WRI.httaa: //acl mothology.org/PII1
11

---

Published as a conference paper at ICLR 2024
Bryan Eikema and Wilker Aziz. Is map decoding all you need? the inadequacy of the mode in neu- ral machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 4506-4520 2020.
Ramy Eskander, Smaranda Muresan, and Michael Collins. Unsupervised cross-lingual part-of- speech tagging for truly low-resource scenarios. In Proceedings of the 2020 Conference Em- pirical Methods in Natural Language Processing (EMNLP), pp. 4820-4831, Online, November 2020. Association for Computational Linguisties. doi: 10. 11t11/////1122000mmmmmmmm.m (aaunllpmmin.39)) URL
em..p- maii..................mmmm..mm 391
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Man- deep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. The Journal of Machine Learning Research, 22(1):4839 4886, 2021.
Hao Fei, Meishan Zhang, and Donghong Ji. Cross-lingual semantic role labeling with high- qualitu translated training corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7014-7026, Online, July 2020. Association for Computational Linguistics. doi: 10. 18653/ 11111166/6202200.tt 1/2020 acl-main.m2777
URL https://aclanthol..o org/
2020.ac1-mainn 627.
Iker Caarcfa-Ferrent Rodrigo Agerri, and German Rigau. T-projection: High quality annotation projection for sequence labeling tasks. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (Findings), 2023.
Ruohao Guo and Dan Roth. Constrained labeled data generation for low-resource named entity recognition, In Findings of the Association for Computational Linguisties: ACI-IICNLP 202 I, pp. 4519-4533, 2021.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanee bert with disentangled attention. In International Conference on Learning Representations, 2021. URL htt.:hs:///peeeeeiieeeee
Chris Hokamp and Qun Liu. Lexically constrained decoding for sequence generation using grid beam search. Proceedings of the 55th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 1535-1546, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18663/w1/P17-1111. URL https://
aclanthology org/P17-114)
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisa- tion. In International Conference on Machine Learning, pp. 4411 14211. PMLR, 2020.
Kuan-Har -ao Huang, I-Hung Hsu, Prem Natarajan, Kai-Wei Chang, and Nanyun Peng. Multilingual
generative language models for zero-shot cross-lingual event argument extraction. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I;
Long Papers), pp. 4633-4646, Dublin, Ireland, May 2022. Association for Computational Lin- UtL/tttss: http://aaaaaatttooo....
guistics. doi: 1/206222/12222222l2llllll acl-long. 317
/ org/2022.
Ayyoob ImaniGooghari, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André Martins, François Yvon, and Hinrich Schütze. Glot500: Scaling multilingual corpora and language models to 500 languages. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1082-1117, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653//12023..ccl aonno. 61. URL http:atss//aaaaatllllooooo
.rg. 2023 wwtp///g....o.....oo -ong..ll-oon.... 61
Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Deronccuur, Trung Bui, and Thien Huu Nguyen. ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning, arXiv preprint N31v:23044006133.. 2023.
12

---

Published as a conference paper at ICLR 2024
Kyungjae Lee. Kyoungho Yoon, Sunghyun Park, and Seung- won Hwang. Semi Semi- supervised training data generation for multilingual question answering. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018.
Gina-Anne Levow. The third international chinese language processing bakeoff: Word segmenta- tion and named entity recognition. In Proceedings of the Fifth SIGHAN workshop on Chinese lunguage processing, pp. 108-117, 2006.
Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. Mlqa: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7315-7330, 2020.
Constantine Lignos, Nolan Holley, Chester Palen-Michel. and Jonne Sälevä. Toward more mean- ingful resources for lower-resoureed languages. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Compulational Linguistics: ACL 2022, pp. 523-532, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 18.13/ v1/2022. windiiisnninnnlc..... URL
wrg.oooo...r///222/2 wtt.:////..222.................. Findings-acll
44
Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. NeuroLogic decoding: (un)supervised neural text generation with predicate logic constraints. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.
Online, June htw.:///./2a.llllll-e maiaclmmiin339 URL nwa.l-aaaaaaaaaam. 339
2021. Association for Computational Linguistics. doi: 10. 20.18653/v1/2022 https: hatp:://aclaattholooo
Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lianhui Qin, Youngjae Yu, Rowan Zellers, Noah A. Smith, and Yejin Choi. NeuroLogic essquue decoding: Constrained text generation with lookahead heuristics. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, pp. 780-799, Seattle, United States, July 2022. As- sociation for Computational Linguistics. doi: 10.18653/v
URL https:
//aclanthology wtq/ naacl main... 57
Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li. Cgmh: Constrained sentence generation by metropolis-hase hastings sampling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 6834 6834-6842, 2019.
Huyen T M Nguyen, Quyen T Ngo, Luong Vu, Vu M Tran, and Hien T T Nguyen. Visp shared task: Named entity recognition. Journal of Computer Science and Cybernetics, 34(4): 283-294, Jan. 2019. doi: 00.15625/8133-666/444/116666
URL https:/ http://ssacc.....
index. phpp jec http:/aatcceeeeeeeeeeaeeeec 1131611ww111/
Jian Ni, Georgiana Dinu, and Radu Florian. Weakly supervised cross-lingual named entity recog- nition via effective annotation and representation projection. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1470-1480, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v//117-1133 URL httph:tt////mm.... URI www.eeaaaaaoooooooooooooo..a
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. Cross- lingual name tagging and linking for 282 languages. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers). pp. 1946-1958, Vancouver, Canada, July 2017. Association for Compu- tational Linguistics. doi: 10. 10.18653/v1/P17-1177 URL https:
org/
P17-1 1178
Tanmay Parekh, Hsu, Kuan-Hao Huang, Kai-Wei Chang, Nanyun Peng, et al. Contextual label projection for cross-lingual structure extraction. arXiv preprint arXiv:2309 08943, 2023.
Matt Post and David Vilar. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation. In Proceedings of the 2018 Conference of the North American Chap- fer of the Association for Computational Linguistics: Human Language Technologies, Volume
13

---

Published as a conference paper at ICLR 2024
(Long Papers), pp. 1314-1324, New Orleans, Louisiana, June 2018. Association for Compu- tational Linguistics. doi: 10. 10.18653/11/N18-111 1119. URL hat.ps://ccaathooooyyy o
118-1119
Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: Energy-based con- strained text generation with langevin dynamics. Advances in Neural Information Processing Systems, 35:9538-9955 9551, 2022.
Evgeniia Razumovskaia, Ivan Vulić, and Anna Korhonen. Transfer-free data-efficient multilingual slot labeling. arXiv preprint arXiv:2305. 135288, 2023.
Torsten Scholak, Nathan Schucher, and Damitry Bahdanau. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. In Marie-Francine Moens, Xu- anjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.). Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pp. 9895-9901, Online and Punta Cana, Dominican Republic, November 2021, Association for Computational Linguis- tics. doi: 10.1 10118663///22221.mmmmmm..... mmw.ppmmiim7m79. URL.ttpss: //ccl//cccccc antho 1 logy. org/2021. emnip-main. main. 779
Felix Stahlberg and Bill Bymne On NMT search errors and model errors: Cat got your tongue? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1366-3362, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 18.186 1/D/3///1/D1111/// 1331. httpttps::
antho Macchologgh ooy. wtw.:/o/1.0//91110/0330
Elias Stengel-Eskin, Tzu-ray Su, Matt Post, and Benjamin Van Durme. A discriminative neural model for cross-lingual word alignment. In Proceedings of the 2019 Conference on Empir- ical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNL) pp. 910-920, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: 10.18653 URL
wrg./10 1084
Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. Multilingual translation with extensible multilingual pretraining and finetuning. 2020.
Erik F. Tjong Kim Sang. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002), 2002. URL.ttta http://aaaaattooo...ooooooooooat org. /0////2222000 2024
Weijia Xu, Batool Haider, and Saab Mansour. End-to-end slot alignment and recognition for cross- lingual NLU. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5052-5063, Online, November 2020. Association for Computational Linguistics. doi: 2020. www.pmmmin.n.eemmmmmmmmmmmmmmmmm 410
URL https::/accaanteett .orggoggyyrooooooooooooooooooogr
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483-498, 2021.
Mahsa Yarnohammadi, Shijie Wu, Mare Marone, Haoran Xu, Seth Ebner, Guanghui Qin, Yunmo Chen, Jialiang Guo, Craig Harman, Kenton Murray, et al. Everything is all it takes: A multi- pronged strategy for zero-shot cross-lingual information extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1950-1967, 2021.
David Yarowsky and Grace Ngai. Inducing multilingual POS taggers and NP bracketers via ro- bust projection across aligned corpora. In Second Meeting of the North American Chapter of the Association for Computational Linguistics, 2001. URL https: http::/:///aaaahhoooooooo .org
NO1 - 0026
14

---

Published as a conference paper at ICLR 2024
Maosen Zhang, Nan Jiang, Lei Li, and Yexiang Xue. Language generation via combinatorial con- straint satisfaction: A tree search enhanced Monte-Carlo approach. Pindings of the Associ- ation for Computational Linguistics: EMNLP 2020, pp. 1286-1298, Online, November 2020. Association for Computational Linguistics. doi: 10.18633/v I 1/t.2/22222nn.o.. www.a/nniiiiiinnnmnnn...... 15. URL httpss://aclanthol.ooy. htw.://c...tttooooooooooo///000 f idi ngg 115
15

---

Published as a conference paper at ICLR 2024
A CODEC IMPLEMENTATION DETAILS
When we approximate the original m-projection problem by solving m 1-projection sub- sub-problems, there are cases where some projections overlap with each other. In translate-train, remove all examples which have this issue.
A.1 PRUNING OPENING-MARKE POSITIONS
Let Mi be a set of all position i, in which the difference between the two log probabilities is greater than threshold 01- Let M2 be neighbor set of M1, which includes all positions that near each position in M, (within a window of size a) and have a difference greater than a second threshold ag (a2 ai). The set of candidates for the opening marker position M the union of the two sets M1 and M2. In our experiments, we choose 0r = 0.5, 0.1, a = 5.
A.2 SEARCHING FOR TOP-K K HYPOTHESES
In our implementation, for efficiency, we send a batch of partial hypotheses in Line 19 We set the batch size equal 16 and 12 for NER and EAE experi- ments, respectively. For all experiments, we search for 5 hypotheses with the highest probabilities (i.e., k 5).
This step is described in Algorithm [
In Eq 8 the max operation is to avoid the case where the current-chosen hypothesis "saves" the markers until the end. In this case, the probability of prefix of the current-chosen hypothesis is high at the beginning, thus can mistakenly prune the correct hypothesis, but degrades significantly after a marker is inserted. We set å equal to for translate-train experiments the MasakhaNER2.0 dataset and set å equal to 5 for all other experiments.
A.3 RE-RANKING
For the cranslae-trrin experiments, we also use an additional lexical-span-leved score to filter noisy augmented data. Particularly, we translate each label span in the source sentence independently. let etrans be the translation of a label span e the lexical-span-lever score of the ith hypothesis is the lexical similarity between errans and Agt where e/st is the span found in the ith hypothesis, following the fuzzy string matching method in Chen et al.) (2023a). An example will be filtered out if both its lexical-span-level score and the span-level score defined in $3.4 are smaller than the two corresponding thresholds. We set the threshold for the lexical- and bobbabiitt-spannleee scores as 0.5 and -5 respectively.
Augmented low-resource languages
English Data
EasyProject
Awesome-align
Codec
Indiacse and Pakistarus have fought region
Indiaaue Pakistankec ye Kashmin
Indieuuc nePakistan
Indiaus nePakistanue
on
yeKashmiin.c.
yeKashmineer.
State media quated Chinaus: 's top negotiator with Taipekuce. Tang Shubelvse, ... from Talwanuus ...
Imithombo wB Chinaaoe ne Taipeluss, uTTang Shubeless, alivela TTaiwan.ce
Imithombo.co wasseChing neTaipal uuTan Shubelea, elivela aTalwan ...
Imithombo waseChinauce
neTaipeluse ulTang Shubeleas elivela eTalwan.ce
Figure Examples of using different approaches to project label spans from English to low-resource languages (i.e., chiShona (middle) and isiZulu (bottom)) in the translate- -rain setting for Cross- lingual NER. In each example, label spans in English data and their corresponding projections in the target language have the same color, the projection errors are underline. In the two examples: (1) EasyProject incorrectly splits some words and only marks a part of them as an entity (e.g., "Pakistan" instead of "nePakistan"), "); (2) Awes-align cannot project all label spans and incorrectly map "China" to "Imithombo" the second example; (3) CODEC has the correct projections both examples.
16

---

Published as a conference paper at ICLR 2024
Algorithm Constrained DFS: Searching for top-k best hypotheses
Input x m Source sentence with marker, y: translation prefix (default: e), y " translation template,
L: [log P(y (r)), log (()):(z).... log P(y]ar]] (default=(0.0)). M: opening marker positions
H: min heap to record the results, number of hypotheses, ó: lower bound hyperparameter
1: flag [check if all markers generated]
2: if Viul </>> and flag TRUE: then
H. push( L.yg,,,,)) y))
H sorts by first element
4:
if (elen(H] (H) > k then
5:
H. pop()
6:
else
7:
T
8:
W1 + {get the next token in fmpl)
9:
T TU [[wi, log (wwi [y, xxankk)))
10:
j < lyl +
b position token to generated next
11:
W2 < [get the next marker)
12:
if 3 and not (W2 = 'I' land j i M) then
13:
T TU ((w2, log P(waly, x (=aak))))))))))))))>)>>)))))))) })}
T { T by the second element in decreasing order)
15:
for (w,p) p) E do
16:
logp <- L|2 +p 7 {compute lower bound following 7
17:
18:
if logg > y then
19:
Constrained DFS( mark y w. pmpi M,
20:
return
B EXPERIMENT DETAILS
For all experiments, use A40 GPUs (48GB each). About fine-tuning backbone models, for the NER model, we fine-tune mDDeBRTTaV3 and train for epochs (except for the baseline, which is trained for 10 epochs) provided by Chen et al. (2023b). Due to the high variance of cross-lingual transfer performance (Chen & Ritter 2021). report average results random seeds. About the Event Argument Extraction model, for fair comparison, we fine-tune mT55- using the X-Gear (Huang et al. 2022) codebase using their provided hyperparameters when fine-tuning and using their scripts for evaluation.
(276M) using the learning rate of 2e-5, batch size 16,
For all experiments. we use the NLLB-3.3B version to generate the translation template and and use NLLB-600M for decoding in CODEC, unless specified otherwise.
C MORE EXPERIMENTS ON MASAKHANER2.0
C.I MORE BASELINES WITH COMPETITIVE PRE-TRAINED MODELS
In $ we fine-tune mDeberta V --bees model on English data (FT as a baseline NER model, and use mBERT-base (Devlin et al. (211)) as the multilingual models in Awesome-align. In this section, we consider using other pre-trained models, which are more competitive for African languages, for the two aforementioned baselines: (1) AfroXLMR-large (Alabi et al.) (022), model which adapted to 17 African languages, and (2) Glot500-base (ImaniGooghari et al. 2023). which is pre-trained on 500+ languages. We consider the following baselines: (1) FT fine-tuning mDe- berraV3//AfroXLMMM/Gooo0o00 on English data and directly evaluate the model, (2) Awes-align: us- ing Awesome-align with CODEC. Since Awes-align and CODEC achieve the best average performance in translate-test (as discussed in $ in this experiment, we evaluate these two approaches in this setting. For all translate-test experiments, mDebertaV3-benn is still used as the English NER model.
the underlined multilingual models, (3)
Table shows the performance of the above methods on the test set of MasakhaNER2.0 dataset. Replacing mDeBERTaV3 (for FTE.) and mBERT (for Awes-align) with models that are pre-trained
17

---

Published as a conference paper at ICLR 2024
Table 4: Cross-lingual NER results of: (1) FTE:: Fine-tuning different pre-trained models (mDeBERTaV3, AfroXLMR- and Glot500-base) on English data only, (2) Awes-align: Awesome-align with different multilingual models (mBERT, AfroXLMR, and Glot500), (3) CODEC: CODEC with two different MT models (M2M-100 and NLLB) for decoding, the test set of MasakhaNER2.0. CODEC and Awesome-align are evaluated the translate-test setting. CODEC with NLLB outperforms all other baselines on average ("-": these low-resource languages are not supported by the M2M 100 MT system).
FT
Awes align
CODEC
Languages
mDeBERTa
AfroXLMR
Glot500
mBERT
AfroXLMR
Glot500
NLLB
M2M-100
Bambara
37.1
42.8
53.0
50.0
53.8
52.0
55.6
73.1
75.4
72.5
76.8
Fon
49.6
54.6
59.4
62.8
66.0
61.2
61.4
Hausa
71.7
74.8
68.0
70.0
73.7
73.1
73.7
73,1
Igbo
59.3
74.4
65.4
77.2
71.2
72.8
72.8
Kinyarwanda
66.4
68.4
67.0
64.9
68.2
68.1
78,0
Luganda
78.9
80.9
82.4
82.5
82.3
Luo
35.8
40.4
42.0
52.6
51.3
51.2
52.9
Mossi
45.0
45.3
55.5
48.4
49.1
48.7
50.4
Chichewa
73.8
chiShona
35.2
38.4
37.6
67.0
78.6
75.6
78.4
Kiswahili
87.7
88.1
84.7
80.2
80.5
79.5
81.5
82.9
Setswana
74.4
81.4
80.5
Akan/Twi
50.1
41.9
57.9
72.6
71.7
73.5
73.5
Wolof
44.2
49.0
64.5
58.1
59.0
57.1
67.2
isiXhosa
24.0
26.8
27.8
52.7
63.9
69.2
Yonba
36.0
57.0
56.1
49,1
52.7
49.2
58.0
53.8
isiZalu
43.9
47.3
46.5
64.1
75.5
74.9
76.9
75.7
AVG
58.8
60.2
65.8
68.7
67.6
70.4
or adapted to African languages further improve the average results of both approaches. However, the two approaches still fall behind CODEC with NLLB average.
C.2 USING THE M2M MT SYSTEM IN CODEC
In this section, experiment with M2M-100 (418M parameters) (Fan al. 202 another MT system that can support many African languages, in CODEC for constrained decoding. Results are shown in Table CODEC with M2M-100 has similar performance CODEC with NLLB on most of the languages that it can support, except for isiXhosa, and also outperforms the baseline of fine- tuning on English data with mDeBERTa on 5 over supported languages.
C.3 IMPACT OF THE SCALE OF THE MT MODEL. TO CODEC
There are two places where MT systems are used in CODEC: one for generating the translation template, and the other for decoding. In this section, we explore the impact of the scale of each MT model on the performance CODEC on the MasakhaNER2.0 dataset, the translate-test setting.
We first analyze the impact of using different sizes of NI.LB (i.e., 600M, 1.3B, and 3.3B) generate the translation template, and use the same MT' system (i.e., NI.LB-600M) for constrained decoding.
Overall, the average Fl scores of methods using NLLB-600M. NLLB- 1.LB-113B and NLLB-3.3B as the template-transe translator are 68.6, 70.6 and 70.4 respectively (details are in Table 9). We observe the performance significantly improve when changing from NLLB-600M, to 1.3B, where NLLB's translation quality also improves the most (Costa-jussà et al. 2022).
Given the same translation templates (i.e., using NLLB-3.3B as the template generator), explored using CODEC with WILB-660M NLLB-1.32 1.3B and NLLB-3.31 B.33 as the constrained decoding module, and observed that the average performance of the three models are 70.4, 70.1, 70.1 respectively.
18

---

Published as a conference paper at ICLR 2024
Table 5: Fl scores different methods of two Cross-lingual tasks: Event Trigger Detection (ETD)
and Event Argument Extraction (EAE) on ACE-2005 dataset in the translate- -aainn setting.
Languages
Tasks
FT
Awes-align
FasyProject
CODEC
ETD
48.0
54.6
47.2
51.3
Arabic
EAE
26.7
33.1
26.9
33.6
AVG
37.4
43,9
37.1
42.5
ETD
52.8
54.4
51.5
53.1
Chinese
EAE
33.5
37.1
37.2
41.2
AVG
43.2
45.8
44.4
47.2
D MORE EXPERIMENTS ON ACE2005 DATASET
D.I AN END-TO-END PIPELINE FOR CROSS-LINGUAL EVENT EXTRACTION
In the main paper, following many prior works, assume access the gold event trigger the inference time and only focus on the Event Argument Extraction (EAE) task. In this section, we ex- plore full pipeline for the EAE task, which includes first using Event Trigger Detection (ETD) module to detect the event trigger, then using an EAF module to extract the event arguments. To train the ETD model Arabic/Chinese, we project the event triggers from English the target language, and fine-tune mDeBERTaV3-bass on the concatenation of the English and projected data. We consider the same baselines and follow the same setup as f4r fine-tuning the EAE model. We train each model three times with three different random seeds and report the average F1 scores. We also consider using label projection methods (i.e., Awes-align, CODEC) in the translate-test setting, in which use English ETD and EAE models to extract the event triggers and event argu- ments, and use label projection to project the information from English to target languages. Table shows the performance of different methods on the test set of ACE 2005. Overall, Awes-Align (an embedding- based word alignment method) is a very strong baseline for high-resource languages, such Arabie and Chinese. CODEC is marker-based method, which is to EasyProject (the state-of-the-ar marker-base method). CODEC'S strengths are more apparent for low-resource languages and event arguments (i.e., varied nouns, pronouns, noun phrases, and entity mentions) where the multilingual LLMeembeddings are less performant. Event triggers are often the main verbs or action nouns (e.g., "married", "attack"), where word aligner seems be doing better, especially in high-resource languages.
directly comparable
D.2 USING THE MBART MT SYSTEM IN CODEC
this section, we explore using different MT system for the cross-lingual event extraction task. In particular, choose mBART50 many-to-many translation system (Tagg al.) 2020). which a competitive open-source MT system for Chinese and Arabic. We use for generating the translation template (for the three label projection methods) and decoding (for CODEC only). Except of changing the underlining MT systems, we follow the same experiment setup as described in $4.2 We also fine-tune mBART50 many-to-many on the synthetic data which Chen et al. (2023a) provided. The fine-tuned checkpoint in used as the MT system of EasyProject and is used as the decoding MT of CODEC.
loblerccompares the performance of different label projection methods while using mBART50 MT system. CODEC achieves competitive performance compared the other label projection baselines. It achieves the best results in Arabic, and is slightly worse than EasyProject in Chinese.
D.3 TRANSLATE-TEST RESULTS
In this section, we explore the usage of label projection methods in the translate-test setting of the Cross-lingus EAE task. In the setting, label projection methods (i.e., Awesome-alige CODEC) first used to project event triggers from target languages to English, then an English EAE model run extract the arguments, and finally the label projection methods are used to project the
19

---

Published as a conference paper at ICLR 2024
Table 6: Fl scores of different label projection methods using mBART50 MT system on ACE2005.
mBART50 is used for both template generation and decoding in CODEC.
mBAART50-many-to-mann
Lang.
FTB
Awes-align
EasyProject
CODEC
Arabic
44.8
49.1
45.4
49.5
Chinese
54.0
56.6
57.6
57.2
AVG
49.4
52.9
51.5
53.4
extracted arguments back target languages. We experiment with two MT systems for translation template NI.LB-3.38 and Google Machine Translation (GMT). We observe that the tokenizer used in NLLB models lacks some Chinese characters and encodes numerous tokens in the Chinese test sentences as "unknown", therefore, in experiments of CODEC with Chinese data, we use mBART50 many-to-many (Tang al. 2020) to decode. While in the experiments of CODEC with Arabic data, we continue using NLLB 600M to decode. In and D.2] we use the fine-tuned version of NLLB- 600M and mBART50 many-to-many as the decoding MT for CODEC. Specifically, the checkpoints are fine-tuned on the synthetic data from Chen et al. (2023a). The data contains parallel sentences, where the entities on both sides are surrounded by markers. Since the text spans within markers only entities, the fine-tuned checkpoints might not be performant when
in the synthetic data projecting other types of span, such as event triggers, which are often main verbs or action nouns. Consequently, to improve the projection accuracy of CODEC in the EAE task, we explore creating new synthetic data, whose parallel sentences having event triggers and arguments surrounded by markers. This data then used to fine-tune the decoding MT of CODEC. For constructing the data, we use English Event Trigger Detection (ETD) and EAE models to annotate English sentences. We follow the setup in $D.1 obtain the two models. The extracted event trigger and arguments are then projected to sentences in target languages by utilizing string matching or alignment- based approaches. For the former method, we use NLLB-3.33 to translate event triggers and arguments, and use string matching to find the corresponding text spans in the target sentences, following Chen jet al.] (2023a). For the latter approach, we utilize Awesome-align to align the event spans between the source and target sentences. We use the training data of NLLB to obtain parallel sentences, following Chen et al. (2023a). In total, our synthetic data has 10,000 sentence pairs for each language Aaglish-Arabic, Chinese- English, Chinese-English,
pairs (i.e., English-Chhnese
(rabic-English).
In Table 7 we report the performance of zero-shot cross-lingual transfer approach (i.e., FTE.), and translate-test results of label projection methods (i.e., Awes-align and CODEC). We study three versions of CODEC, with the difference comes from the synthetic data used to fine-tune the decoding MT: CODEC: the synthetic data is from Chen et al. (2023a), whose parallel sentences having entity spans surrounded by markers, CODFCmatch and CODECalign our new synthetic data, whose parallel sentences having event triggers and arguments surrounded by markers, constructed by following the string matching and alignment-based approach, respectively. Overall, we observe that the translation quality has big impact on the performance of translate-test methods, especially in Chinese. For Chinese, label projection methods can only outperform FTE. when switching to use GMT as the template translator. Compared to Awes-align, CODEC and variants have better performance for most of the cases, especially in Arabic. Finally, except for CODECmaach in Chinese when using NLLB- 3.3B as the template translator, CODECmath and CODECalign significantly outperforms CODEC. The observation demonstrates the benefit of using the synthetic data with event labels surrounded when fine-tuning the decoding MT of CODEC.
E COMPARE TO BEAM SEARCH IN THE CONSTRAINER SEARCH SPACE
In this section, we compare CODEC with a modified version of beam search that has the same search space, which we name as CODEC to Constrained-Spee -Sacee Beam Search (CSBS). EasyProject method that uses un-modified version of Beam Search. Similar to CODEC, at each decoding iteration, instead of looking at the whole vocabulary, CSBS considers only the next token from the
fact
20

---

Published as a conference paper at ICLR 2024
Table 7: Cross-lingual EAE results of label projection methods using different MT systems in the translate-tess setting. "MT" is the MT system used for translation template. In CODEC and vari- ants, mBART50 and NLLB-600M are used for decoding for Chinese and Arabic data, respectively. CODEC CoDECCiin the decoding MT in CODEC is fine-tuned synthetic parallel data, where event arguments and triggers are surrounded by markers (more details are in D
Translate-test
I Languages
MT
FT
Awes-align
CODEC CODECmuch CODECalge
NLLB-3.3B
41.9
42.8
45.2
47.1
Arabic
44.8
GMT
44.5
49.1
53.9
52.2
NL..B-3.3B
45.8
47.8
44.6
Chinese
54.0
GMT
56.3
54.7
56.9
Table Cross-lingual NER results of using Constrained-Space Beam Search (CSBS) with different beam sizes and CODEC in the translate-test setting in MasakhaNER2.00 CODEC outperforms CSBS for every language.
CSBS
Languages
CODEC
beam=2
beam=4
beam=8
beam=16
Bambara
32.6
35.9
38.5
42.4
55.6
Ewe
54.8
61.0
66.8
70.6
79.1
Fon
21.3
24.2
29.0
32.7
61.4
Hausa
54.4
66.4
68.7
70.1
73.7
Igho
46.2
52.3
56.5
60.3
72.8
Kinyarwanda
61.5
70.7
72.7
74.7
78.0
Luganda
62.0
74.2
77.8
80.3
82.3
Luo
27.4
31.8
34.4
41.1
52.9
Mossi
22.2
25.3
29.4
34.5
50.4
Chichewa
51.9
59.0
62.5
67.8
76,8
chiShona
68.0
74.0
75.3
76.2
78.4
Kiswahili
57.8
73.5
75,8
77.3
81.5
Setswana
69.2
74.7
76.7
78.1
80.3
Akan/Twi
57.3
63.1
65.4
68.1
73.5
Wolof
39.1
45.8
50.7
54.7
67.2
isiXhosa
49.9
59,9
64.2
67.0
69.2
Yoruba
38.1
45.1
48.4
51.7
58.0
isiZulu
54.2
69.4
72.2
74.6
76.9
AVG
48.2
55.9
59.2
62.3
70.4
translation template and/or a marker. We compare the two search algorithms in the translate-test setting the MasakhaNER2.0 dataset. Table T illustrates the performance of CSBS with different beam sizes (i.e., 2, 4, 8, 16) and CODEC. As shown in the table, higher beam size will increase the performance of CSBS, however, even with the beam size 16, CSBS's average Fl score still falls behind CODEC with a big gap.
F MANUAL ASSESSMENT SETUP
Setup In the translate-trail setting, we translate 100 English examples from the CoNLL- 2002/2003 multilingual NER datasets (Tjong Kim Sang 2002) Vietnamese and Chinese, and use CODEC to project label spans. For the translate-tess test setting, we inspect 100 Vietnamese examples from the VLSP 2016 NER dataset (Nguyen al. 2019) and 100 Chinese examples from MSRA- NER dataset (Levow 2006). When using the NLLB model to encode texts from the MSRA dataset, we observe that NLLB lacks many Chinese tokens. Therefore, for experiments on this dataset, we fine-tuned version of the M2M-100 model (Fan et al. 2021) instead. M2M-100 (418M) is fine-tuned on the same synthetic dataset and follows the same script which is used to fine- tune the
NLL.B models used in CODEC and EasyProject (Chen et al. 2023a).
21

---

Published as a conference paper at ICLR 2024
Table 9: Cross-lingual NER results of CODEC using different scalses of NLLB (600m, 1.3B, and
3.3B) for template translation and for constrained decoding. Results are reported for the translate- test setting of MasakhaNER2.0 dataset ("Translate": the NLLB model used for template translation; "Decode": the NLLB model used for decoding in CODEC).
TranslaeeNNLLB-3338
Decode=NLLB-600V
Languages
Dsco0eNLLLB-60M
NLLB-1.3B
NLLB-3.3B
Transat==NLLB-GOMM
NLLB-1.3B
Bambara
55.6
56.0
55.8
48.4
54.9
Ewe
79.1
79.3
77.7
78.0
78.2
Fon
61.4
62.4
60.2
59.7
61.2
Hausa
73.7
73.5
72.2
73.0
Igbo
72.8
72.8
70.5
71.3
Kinyarwanda
78.0
71.3
71.6
76.8
78.4
Luganda
82.3
82.4
81.7
80.9
82.5
Luo
52.9
53.1
53.1
50.2
54.7
Mossi
50.4
52.1
50.7
48.4
53.7
Chichewa
76.8
76.7
76.9
71.5
75.8
chiShona
78.4
78.0
78.3
77.7
79.2
Kiswahili
81.5
82
82.5
81.2
81.1
Setswana
80.3
81.0
81.2
78.3
79.3
Akan/Twi
73.5
74.9
73.9
Wolof
67.2
66.2
67.5
65.2
69.8
isiXhosa
69.2
69.6
69.1
69.6
69.4
Yoruba
58.0
56.5
58.7
56.2
58.6
isiZulu
76.9
76.6
76.6
74.2
75.6
AVG
70.4
70.1
70.1
68.6
70.6
22


Reviews:
[{"review_id": "S52f8rlk7q", "reviewer": "Reviewer_TbXV", "summary": "=== AFTER THE RESPONSE ===\nI would like to thank the authors for taking the time to provide a very detailed response which clarified my main concerns and extra questions. I still think that the method could have been evaluated on a larger selection of tasks, but this doesn't invalidate the soundness of the proposed methodology, and I'm happy to increase my score\n===========================\n\nThis paper targets cross-lingual transfer for sequence labeling task, where the main problem in previous work has been projecting labeled spans from the source language to the correct spans in the target language, a problem sometimes referred to as the labeled span mismatch. Previous work typically solved this problem via two different approaches: 1) using external word aligners to do the label projection from source to target, or 2) inserting marker tokens directly into the input of a strong (N)MT system and basically conducting a standard translate-train approach (but with those extra markers). However, both prior approaches have issues as the former critically relies on the quality of the external word aligner, while the latter yields to degraded MT performance (due to the insertion of markers).\n\nThis paper basically proposes an extension to the latter approach, aiming to preserve the original quality of the MT system by bypassing the direct insertion of markers, and proposes a two-step approach where in Step 1) the original text can be translated (via translate-train or translate-test), and in Step 2) projection is added via constrained decoding, keeping the translation from Step 1 as a fixed template. The main technical contribution is then a computationally feasible technique for the constrained decoding, bypassing the need to conduct exhaustive brute force search while maintaining strong performance. The separation of the translation and marker insertion steps also allows the approach to be applied to the translate-test setting, and the results confirm the usefulness of the technique.", "questions": "A similar two-step idea, but which is not MT-based but encoder-based has been investigated here: https://arxiv.org/pdf/2305.13528.pdf (the idea there is slightly different and is based on classification - in the first step, the system just decides whether something should be a labeled span or not; in the second step, the actual label is added to each span detected as 'labeled span'. The paper should also discuss ideas like this one in related work and they seem highly relevant.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper clearly defines the problem, which is very concrete, and sets out to solve the problem following a clear line of thinking: from the conceptual level all the way to low-level technical execution aiming to improve the performance-versus-efficiency trade-off.\n- The idea of constrained decoding which fixes the entire sentence (instead of focusing only on lexical constraints during constraints) is quite new (at least to the best of my knowledge) and could have applications beyond cross-lingual transfer tasks discussed in this work.\n- The paper is well written and it is easy to link the main hypotheses to the concrete experiments and analyses. The core section of the paper on \"constrained decoding\" is also nicely described and easy to follow.\n- The results on the two tasks seem to support the main claims although the paper requires more experiments (see also under weaknesses).", "weaknesses": "- The main issue with the work is 1) the lack of recognition of other (recent and less recent) work on the same problem of cross-lingual label projection, which consequently leads to the 2) lack of more comprehensive comparisons to more baselines. The main baseline is definitely the EasyProject method and I agree with that, but I feel that not enough care has been provided to optimise the word alignment-based baselines which also shows reasonable performance, and is quite competitive in the EAE task.\n-- For instance, there has been some recent work on alignment correction for label projection (https://aclanthology.org/2023.law-1.24.pdf), and there are also other very relevant papers which should be cited and discussed (and ideally even compared against): https://aclanthology.org/2021.findings-acl.396.pdf or https://d-nb.info/1203127499/34, \n-- The number of evaluation tasks is slightly underwhelming and the paper should extend the scope of tasks to other sequence labeling tasks (e.g., slot labeling in dialogue, dependeny parsing or semantic role labeling) - NER with only 3 NE classes is a (relatively) simple task (from the perspective of its experimental setup), and the paper would have more impact with a wider experimental setup.\n-- I would also like to see a wider exploration of different MT systems and chosen encoder-decoder models and their impact on the performance of both alignment-based approaches as well as EasyProject and CODEC. For instance, how would larger variants of NLLB affect the performance? Would the scale of the NMT system recover for its deficiencies?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kl5cXjvxPn", "reviewer": "Reviewer_kHv7", "summary": "This paper focuses on improving label projection for zero-shot cross-lingual transfer learning. They claim that existing label projection techniques cannot generate accurate translation and there for affect the downstream performance. They accordingly propose a constrained decoding to decide which positions to insert markers conditioned on a better translation template. Some heuristic tricks are presented to accelerate the search process. Experiments on NER and EAE show the potential of the proposed method.\n\n\n==== After response ====\nGiven that the authors promise that they will make the description clearer and add the translate-test results for EAE, I consider increasing the score.", "questions": "Please see above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The writing is clear.\n- The proposed method performs well on two tasks.", "weaknesses": "- The author say that inserting markers would degrade the translation quality. However, although they provide a translation template to guide the model during translation, the proposed method still relies on markers, which is not completely solving this issue. \n- When searching, they mention the assumption:\n```\nIf we decode the translation template but conditioned on the marker sentence, at the position that needs to be inserted an opening marker, the model would give a high probability to this token, and thus assign a low probability to the token from the template.\n```\nThis assumption largely relies the translation model’s ability to handle markers. Different translators may have different behaviors to handle the markers. I suggest the authors to report the results of different translators to show the stability of the proposed method.\n- The proposed method is based on some heuristic. It would be great if the authors can provide some theoretical bound to justify the heuristic.\n- It seems like that they follow the experimental setting of previous work (Chen et al. 2023a). However, they consider MasakhaNER 2.0 rather than WikiAnn (reported by Chen et al. 2023a) for NER without any explanation. Is it because that the proposed method works better for low-resource languages? I suggest to report the scores on WikiAnn as well for more comprehensive comparisons.\n- I am a little bit confused by the reason for not considering translation-test for EAE. The authors mention that the English gold trigger is needed but not feasible. However, this can be obtained by applying the proposed constraint label project from the target language to English. It would be interesting to study more about this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J1awxz4JTb", "reviewer": "Reviewer_VywY", "summary": "This work proposes a constrained-translation-based label projection method for the cross-lingual transfer of two mention extraction problems (named entity recognition and event argument extraction). Instead of directly translating marked sentences, the proposed the method adopts a two-stage approach: first translate the original source without markers, then perform constrained decoding with the marked source and the translation in the first pass. The decoding algorithm consists several interesting parts, including marker-position pruning, branch-and-bound searching and re-ranking. With evaluations on multiple target languages, the proposed method is shown to provide benefits over existing baselines.", "questions": "- It would be nice to discuss and measure the efficiency of different methods, especially considering the extra stages of the proposed method. This can be important for “translate-test“, and maybe also for “translate-train” if the cost difference is too much.\n- I’m wondering whether it would still be effective to replace the searching algorithm with some simpler alternatives, such as greedy pruning (like in a QA-MRC model). Since the problem itself is inserting a pair of markers, the output space is much smaller than the translation.\n- For the event task, it seems that the event triggers are assumed already given? How about considering the full event structures? This might be straight-forward since only span-projection would be enough (it would also be very interesting to consider pairs of spans when projecting).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n- The proposed method is intuitive and effective.", "weaknesses": "- The approach relies on an external MT system, whose performance may influence the effectiveness of the label projection. It would be nice if there can be an analysis on the influence of translation quality.\n- In some cases, the proposed method does not perform well, for example, the NER results are worse for Chichewa and Kiswahili, and the EAE results seem close to the baselines. It would be much better if there can be more analysis on why these happen to provide some guidance on how to select the label-projection methods for a new language.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8DdP4K6Weq", "reviewer": "Reviewer_NyaM", "summary": "This work describes CODEC, a method of generating instances of data in new languages with fine-grained labels transferred from high-resource languages (i.e., English) to low-resource languages (e.g., Bambara). This work intelligently identifies that prior methods such as EasyProject have the drawback of non-natural markers (e.g., BIO tokens) degrading translation quality. To counter this, CODEC instead uses an unconstrained translation as a template and proposes a constrained decoding algorithm to reconcile the template with the annotated input. \n\nThis changes the formula of EN to BAM from:\n```\nEN -> add markers -> MT -> BAM + markers\n```\nto:\n```\nEN -> BAM,    EN -> add markers -> [EN+markers, BAM] -> MT w/ constrained decoding -> BAM + markers\n```\nThis removes the issue of MT errors near annotation tokens and provides some reference to check approximate validity during CODEC. This work applies CODEC to both the translate-train and translate-test scenarios of cross-lingual transfer to identify that CODEC has benefits nearly everywhere we can use silver-standard data in cross-lingual transfer. Experiments on NER and event argument extraction identify how CODEC benefits cross-lingual transfer across a wide range of low-resource languages. Ablations and analysis across multiple languages are honest and interpretable in discussing where CODEC is beneficial and does not improve.", "questions": "- Constrained decoding is also a large topic in semantic parsing and the authors could acknowledge work such as https://arxiv.org/abs/2109.05093\n\n- The sentence \"The intuition is that, if we decode the translation template but conditioned on the marker sentence, at the position that needs to be inserted an opening marker, the model would give a high probability to this token, and thus assign a low probability to the token from the template, as illus- trated in Figure 2 (Step 1).\" is very long and hard to parse. Consider revising.\n\n- Math format mistake in F2 caption, k -> $k$\n\n- Consider a bulleted list at the end of the intro to make your contributions clearer.\n\n- I think the introduction of Bambara as a language from Africa undersells the low resource importance. Consider a more quantitative phrasing such as \"Bambara is a Manding language primarily from Mali with approximately 15 million speakers\", using information from WALS and Ethnologue.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- This is a very original contribution with wide ranging impact to low-resource cross-lingual transfer. Provided a sufficiently user-friendly codebase, the contributions of CODEC to the field could be widespread. This work also provides a more holistic and thoughtful contribution to the problem than the concurrent https://arxiv.org/abs/2309.08943 . Overall, I think this paper absolutely should be accepted.\n\n- The improvement in both translate-train and translate-test scenarios identify the method as a strong new idea with wide applicabiility. Provided _some_ MT capability, this work helps mitigate the cross-lingual transfer gap to languages with very little study. The work smartly focuses on this scenario (i.e., through MasakhaNER) to support that CODEC works in (approx) the lowest resource scenarios available in modern NLP. \n\n- Succintely describing a constrained decoding method is not easy and this work smartly describes the method visually and mathematically for excellent clarity of the contribution. The frank discussion of complexity and the heuristic approximations for tractability are also honest with tradeoffs discussed in detail to inform future practice.", "weaknesses": "- [Minor]: the work could be stronger if this could also be extended to larger models (e.g., >1-5B) and the discussion of applicability on more architectures (enc-only, enc-dec and dec-only) could be more details. \n\n- [Minor]: the paper could also be improved with more comparison against zero-shot results from larger multilingual LLMs. This would be hard on the given 1 48GB GPU setup, but could strengthen the vailidity of the improvement using CODEC. In essence, asking if CODEC works on a larger scale would make the findings more universal.\n\n- [Minor]: it would be enlightening to see CODEC across a benchmark such as XTREME-UP but this likely should be future work not included here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)