Improving Cross-lingual Transfer through Subtree-aware Word Reordering
Authors: Ofir Arviv
Dmitry Nikolaev
Taelin Karidi
Omri Abend
Venue: EMNLP 2023 Findings (2023)
Keywords: multilingual
cross lingual
cross lingual transfer
reordering
syntax
TL;DR: A new reordering algorithm that benifits even modern multilingual LLMs.

Abstract: Despite the impressive growth of the abilities of multilingual language models, such as XLM-R and mT5, it has been shown that they still face difficulties when tackling typologically-distant languages, particularly in the low-resource setting. One obstacle for effective cross-lingual transfer is variability in word-order patterns. It can be potentially mitigated  via source- or target-side word reordering, and numerous approaches to reordering have been proposed.  
However, they rely on language-specific rules, work on the level of POS tags, or only target the main clause, leaving subordinate clauses intact.
To address these limitations, we present a new powerful reordering method, defined in terms of Universal Dependencies, that is able to learn fine-grained word-order patterns conditioned on the syntactic context from a small amount of annotated data and can be applied at all levels of the syntactic tree.
We conduct experiments on a diverse set of tasks and show that our method consistently outperforms strong baselines over different language pairs and model architectures. This performance advantage holds true in both zero-shot and few-shot scenarios.

Improving Cross-Lingual Transfer through Subtree-Aware Word Reordering
Ofir Arviv!
Dmitry Nikolaev2 Taelin Karidi
Omri Abend
IMS, University of Stuttgart (ofir.arvivy taelin. taelin.kaiddi omri abend) } @mail www.i...ji...... ac.ji.ac.il dnikolaevl fastmail. com
Hebrew University of Jerusalem
Abstract
feasible task, and result speakers of underrep- resented languages are unable to reap the benefits of modern NLP capabilities (Joshi et al., 2020). Numerous studies have shown that key chal- lenge for cross-lingual transfer is the divergence in word order between different languages, which often causes a significant drop in performance (Ra- sooli and Collins, 2017; Wang and Eisner, 2018;
Despite the impressive growth of the abilities of multilingual language models, such XLM-R and m'5,, it has been shown that they still face difficulties when tackling typologically-diss distant languages, particularly in the low- www-resouree set- ting. One obstacle for effective cross- pooss-lingual transfer is variability in word-order patterns. It can be potentially mitigated via source- or target-side word reordering, and numerous ap- proaches reordering have been proposed. However, they rely on language-specifte rules, work the level of POS tags, only target the main clause, leaving subordinate clauses intact. To address these limitations, we present a new powerful reordering method, defined in terms of Universal Dependencies, that is able to learn fine-grained word-order patterns conditioned the syntactic context from small amount of annotated data and can be applied at all levels of the syntactic tree. conduct experiments on a diverse set of tasks and show that method consistently outperforms strong baselines over different language pairs and model architec- tures. This performance advantage holds true both zero-shot and few-shot scenarios.'
Ahmad et al., 2019; Liu et al.. 2020a; Ji et al., 2021; Nikolaev and Pado, 2022: Samardžić et al., 2022), This is unsurprising, given the complex and inter- dependent nature of word-order (e.g.. verb-fina final languages tend to have postpositions instead of prepositions and place relative clauses before nom- inal phrases that they modify, while SVO and VSO languages prefer prepositions and postposed rel- ative clauses, see Dryer 1992) and the way is coupled with the presentation of novel information in sentences (Hawkins, 1992). This is especially true for the majority of underrepresented languages, which demonstrate distinct word order preferences from English and other well-resourced languages.
Motivated by this, we present reordering method applicable to any language pair, which can be efficiently trained even on small amount of data, is applicable all levels of the syntactic tree, and powerful enough boost the performance of modern multilingual LMs. The method, defined in terms Universal Dependencies (UD), based on pairwise constraints regulating the linear order of subtrees that share a common parent, which we term POCs for "pairwise ordering constraints".
1 Introduction
Recent multilingual pre-trained language models (LMs), such as mBERT (Devlin et al., 2019), XLM- RoBERTa (Conneau et al., 2020), mBART (Liu et al., 2020b), and mT5 (Xue et al., 2021), have shown impressive cross-lingual ability, enabling effective transfer in a wide range of cross-lingual natural language processing tasks. However, even the most advanced LLMss are effective when dealing with less-represented languages, shown by recent studies (Ruder et al., 2023; Asai al., 2023; Ahuja al., 2023). Furthermore, annotating sufficient training data in these languages is not a
We estimate these constraints based on the prob- ability that the two subtree labels will appear one order or the other when their parent has given label. Thus, in terms of UD, we expect, e.g., lan- guages that use pre-nominal adjectival modification
From a slightly different perspective, this topic has also been actively studied in the machine translation literature (cf. Steinherger, 1994; Chang and Toutanova, 2007; Murthy et al., 2019).
'Code available https: thub. com/ ofirArviv/ur udd bas--baaee--ee woreddwwor-eees reoddrendd

---

assign a high probability amods' preceding their headword, while languages with post-nominal adjectival modification are expected to have a high probability for the other direction.
scribes the setup for our zero-shot and few-shot experiments, the results of which are presented in Section Section 6 investigates the compara- tive performance of encoder- based and sequence- to-sequence models, and Section concludes the paper.
The estimated POCs are fed as constraints to an SMT solver' to produce general reordering algorithm that can affect reordering of all types of syntactic structures. In addition to being effec- tive, POCs are interpretable and provide detailed characterization of typical word order patterns in different languages, allowing interpretability as to the effect of word order on cross- -oogull transfer.
2 Related Work
A major challenge for cross-lingual transfer stems from word-order differences between the source and target language. This challenge has been the subject of many previous works (e-g.. Ahmad al., 2019; Nikolaev and Pado, 2022), and numerous approaches overcome have been proposed.
We evaluate our method on three cross- mooss-lingual tasks - depeedency parsing, task-oriented semantic parsing, and relation classification in the zero- shot setting. Such setting is practically useful (see. e.g., Ammar et al. 2016; Schuster et al. 2019; Wang al. 2019; Xu and Koehn 2021 for successful ex- amples of employing ZS learning pros--inggally) and minimizes the risk of introducing confounds into the analysis.
One of the major approaches of this type is re- ordering, i.c. rearranging the word order in the source sentences to make them more similar to the target ones vice versa. Early approaches, mainly in phrase-based statistical machine trans- lation, relied on hand-written rules (Collins et al., 2005), while later attempts were made to extract reordering rules automatically using parallel cor- pora by minimizing the number of crossing word- alignments (Genzel, 2010; Hitschler et al., 2016). More recent works focusing reordering relied on statistics of various linguistic properties such POS VOS-tags (Wang and Fisner, 2016, 2018; Liu al., 2020a) and syntactic relations (Rasooli and Collins, 2019). Such statistics can be taken from typologi- cal datasets such as WALS (Meng et al., 2019) or extracted from large corpora (Aufrant et al., 2016). Other works proposed to make architectural changes in the models. Thus, Zhang et al. (2017a) incorporated distortion models into attention-based NMT systems, while Chen et al. (2019) pro- posed learning reordering embeddings as part of Transformer-based translation systems. More re- cently, Ji et al. (2021) trained a reordering module as component of a parsing model to improve cross-lingual structured prediction. Meng et al. (2019) suggested changes to the inference mech- anism of graph parsers by incorporating target- language-specific constraintsin inference.
We further evaluate our method in the scarce- data scenario on the semantic parsing task. This scenario is more realistic as in many cases it is fea- sible annotate small amounts of data in specific languages (Ruder al., 2023).
Experiments show that our method consistently presents a noticeable performance gain compared to the baselines over different language pairs and model architectures, both the zero-shot and few- shot scenario. This suggests that despite recent advances, stronger multilingual models still faces difficulties in handling rross-lingual word order di- vergences, and that reordering algorithms, such as ours, can provide much needed boost in perfor- mance in low-resource languages.
Additionally, we investigate the relative ef- fectiveness of our reordering algorithm on two types of neural architectures: encoder-decoder (seq2seq) vs. classification head stacked on top of a pretrained encoder. Our findings show that the encoder-decoder architecture underperforms in cross-lingual transfer and benefits more strongly from reordering, suggesting that may struggle with projecting patterns over word-order diver- gences.
Our work is in line with the proposed solutions to source-sentence reordering, namely treebank re- ordering, which aim to rearrange the word order of source sentences by linearly permuting the nodes of their dependency-parse trees. Aufrant al. (2016) and Wang and Eisner (2018) suggested permut- ing existing dependency treebanks to make their surface POS POS-sequence statistics close to those of
The structure of the paper is as follows: Sec- tion surveys related work. The proposed ap- proach is introduced in Scction 3. Section 4 de-
'An extension of the SAT solver that can, among other things, include mathematical predicates such as + and in its constraints and assign integer values to variables.

---

the target language, in order improve the per- formance of delexicalized dependency parsers in the zero-shot scenario. While some improvements were reported, these approaches rely on short POS n-grams and do not capture many important pat- terns." 4 Liu et al. (2020a) proposed similar method but used a POS-based language model, trained on a target-language corpus, to guide their algorithm. This provided them with the ability capture more complex statistics, but utilizzigg black-box learned models renders their method difficult to interpret. Rasooli and Collins (2019) proposed reorder- ing algorithm based on UD, specifically, the domi- nant dependency direction in the target language, leveraging the rich syntactic information the anno- tation provides. Their method however, leverages only a small part UD richness, compared to our method.
defined by the parse-tree structure, intact.
We define set of language-specific constraints- based on the notion of Puirwise Ordering Distri- butions, the tendeney of words with specific UD labels to be linearly ordered before words with other specific labels, conditioned on the type of subtree they appear in. To implement reodering algorithm we use these constraints as input to an SMT solver.
3.1 Pairwise Ordering Distributions
Let ((s) be the Universal Dependencies parse tree of sentence language L, and (T...... Ta) the set of all UD labels. We denote the pairwise or- dering distribution (POD) in language of two UD nodes with dependency labels ni Ty. in subtree with the rool label Ty. with:
Papma P.P [0.
(1)
We note that previous work on treebank re- ordering usually only evaluated their methods on UD parsing, using delexicalized models or simple manually aligned cross- lins-llingual word embeddings, which limited the scope of the analysis. In this pa- per, we experiment with two additional tasks that are not reducible to syntactic parsing: relation clas- sification and semantic parsing. We further extend previous work by using modern multilingual LMs and experimenting with different architectures.
where is the probability of node with label T; to be linearly ordered before node with label æj. in subtree with a root of label πTς- Note that being linearly ordered before a node with index means having an index of j < i, and that the nodes are direct children of the subtree root. We include a copy of the root node in the computation as one of its own children. Thus we can distinguish between node acting representative of its subtree and the same node acting the head of that subtree.
3 Approach
3.2 Pairwise Ordering Constraints and Reordering
Given a sentence s 81, 82, Sn in source lan- guage I.g, we aim to permute the words in it to mimic the word order a target language L, Sim- ilarly to previous works (Wang and Eisner, 2018; Liu et al., 2020a), we make the assumption that a contiguous subsequence that forms constituent in the original sentence should remain a contiguous subsequence after reordering, while the inner order of words in it may change. This prevents subtrees from losing their semantic coherence and is also vital when dealing with tasks such as relation ex- traction (RE), where some of the subequence must stay intact in order for the annotation to remain valid. Concretely, instead of permuting the words of sentence we permute the subtrees of its UD parse tree, thus keeping the subsequences of as
Given the pairwise ordering distribution of target language L,, denoted as dist = P, we define a set of pairwise constraints based on it. Concretely, for dependency labels Th, TH, Tj. we define the fol- lowing constraint:
11 if P P.A.,.,Ty > 0.5 otherwise
T (TT < <T))
(2)
where The : (TT < π;) = indicates that a node with dependency label тi should be linearly or- dered before node n' with dependency label πj if they are direct children of node with label m&-
Using these constraints, we recursively reorder the tokens according to the parse tree T(s) in the following way. For each subtree T; E T(s) with
'Aufrant et al. (2016) further experimented with manually crafting permutations rules using typological data on POS sequences from WALS. This approach is less demanding in terms of data but is more labor intensive and does not lead to hetter performance.
'In principle, UD allows for linearly overlapping subtrees. such cases are rare in the existing treebanks. "n example set of learned POC statistics is given in Ap- pendix C.

---

UD label πy and children ni, 22, Rms with UD labels n1, "". Bmz
Estimating POCs without a Treebank. While the UD treebank vast, there are still hundreds of widely spoken languages missing from it. The coverage of our method can be improved by us- ing annotation projection (Agié et al., 2016) 6) on a massively parallel corpus, such as the Bible Cor- pus (McCarthy et al., 2020). Approximate POCs can then be extracted from the projected UD trees. While we do not experiment with this setting in this work due to resource limitations, we mention it as a promising future work venue, relying on the work done by Rasooli and Collins (2019), which used this approach successfully, to extract UD statistics, and utilize them in their reordering algorithm on top of annotation projection.
1. We extract the pairwise constraints that apply to T; based on the UD labels of its root and children.
2. We feed the pairwise constraints to the SMT solver' and use it to compute legal ordering of the UD labels, i.e. an order that satisfies all the constraints.
3. If there is such an ordering, reorder the nodes in T; accordingly. Otherwise, we revert to the original order.
4. We proceed recursively, top-down, for every subtree T, until all of ((s) is reordered to match dist
Experimental Setup
We evaluate our reordering algorithm using three tasks - UD parsing, task-oriented semantic parsing, and relation extraction - and over 13 different target languages, with English as the source language. For each task and target language, we compare the performance of model trained the vanilla English dataset against that of model trained transformed (reordered) version of the dataset, using the target-language test set in zero-shot fashion.
For example, assuming the constraints nsubj root, obj - root, and obl + root for the main clause and obl -> case, corresponding cal SOV language, and assuming that the target language does not have determiners, 8 the sentence
typi-
Shensubi putroos [the booklabj [oNaase the table obl
will be first reordered as
We explore two settings: STANDARD, where we reorder the English dataset according to the target language POCs and use it for training, and EN- SEMRIE, where train models on both the vanilla and the reordered English datasets, The main motivation for this is that any reordering al- gorithm bound to add noise to the data. First, the underlying multilingual LMs were trained on the standard word order of English, and feeding them English sentences in an unnatural word or- der will likely produce sub-optimal representations. Secondly, reordering algorithms rely surface statistics, which, while rich, are product of statis- tical estimation and thus imperfect. Lastly, the use of hard constrains may not be justified for target languages with highly flexible word-order!! The
Shensubi [the book) lbj [oncase the table lobi put and then as
Shensubj [the Jobj [the table on caseelobi yoot-
3.3 Estimating the Pairwise Ordering Constraints
In this section we describe two possible methods for estimating the POCs of language, one relying on the availability of UD corpus in the target language and one relying on the Bible Corpus.
Using The UD Treebank. The first method we use to estimate POCs is by extracting them from corresponding empirical PODs in a UD treebank. When there multiple treebanks per language, select of them as a representative treebank. We use v2.10 of the Universal Dependencies dataset, which contains treebanks for over 100 languages.
"We Figlishh as source language because it has the biggest variety of training for different tasks. It has been shown that for some tasks using a source language with strict word order, such as Russian or Czech, can lead to improvements (Nikolaev and Pado, 2022), but practice they are rather minor and not outweigh the benefits having access to more corpora.
We use Python bindings the open-source SMT solver Moura and Bjömer, 2008). "Whose relative position for the sake of the example thus can he selected arbitrarily.
IIn the worst-case scenario, given very flexible order of a particular pair of syntactic elements in the target language, with PT..TLT 8s 0.51, our method will select only one or-

---

ENSEMBLE setting mitigates these issues and im- proves the "signal to noise ratio" of the approach.
so. Such cases mostly arise due to parsing errors and are very rare. The myper-parameters for all the models are given in Appendix D.
We use the vanilla multilingual models and the reordering algorithm by Rasooli and Collins (2019) as baselines. To the best of our knowledge, Ra- sooli and Collins proposed the most recent pre- processing reordering algorithm, which also relies on UD annotation. We re- e-immlement the algorithm and use it in the same settings as our approach.
4.2.1 UD Parsing
Dataset. We use v2.10 of the UD dataset. For training, we use the UD English-EWT corpus with the standard splits. For evaluation, we use the PUD corpora of French, German, Korean, Spanish, Thai and Hindi, as well as the Persian-Seraji, Arabic- PADT, and the Irish-Twittrrish treebanks.
Lastly, we evaluate our method in the scarce-data setting. We additionally train the models fine-tuned on the vanilla and reordered English datasets on a small number of examples in the target language and record their performance. Due to the large amount of experiments required we conduct this experiment using only our method in the context of the aamnantic-parsing task, which is the most chal- lenging one in our benchmark (Asai et al., 2023; Ruder et al., 2023), on the mT5 model, and in the ENSEMBLE setting.
We note that our results are not directly compa- rable to the vanilla baseline because model has indirect access to a labeled target dataset, which is used estimate the POCs. This issue is less of a worry in the next tasks, which are not defined based on UD. We further note that we do not use the same dataset for extracting the information about the target language and for testing the method.
Model. We use the AllenNLP (Gardner et al., 2018) implementation the deep biaffine attention graph-based model of Dozat and Manning (2016). We replace the trainable GloVe embeddings and the BiLSTM encoder with XLM- RoBERTa-lage (Conneau et al., 2020). Finally, we do not use gold (or any) POS tags. We report the standard labeled and unlabeled attachment scores (LAS and UAS) for evaluation, averaged over runs.
4.1
Estimating the POCs
We estimate the POCs ($3.2) by extracting the em- pirical distributions from UD treebanks (see $3.3). While this requires the availability of an external data source in the form of a UD treebank in the tar- get language, we argue that for tasks other than UD parsing this is reasonable setting the UD cor- pora are available for wide variety of languages. Furthermore, we experiment with various treebank sizes, including ones with as few as 1000 sentences, Further experimentation with even smaller tree- banks is deferred to future work. Appendix B lists the treebanks used and their sizes.
4,2.2 Task-oriented Semantic Parsing
Datasets. We use the MTOP (Li et al., 2021) and Multilingual TOP (Xia and Monti, 2021) datasets. MTOP covers 6 languages (English, Span- ish, French, German, Hindi and Thai) across 11 domains. In our experiments, we use the decou- pled representation of the dataset, which removes all the text that does not appear in a leaf slot. This representation is less dependent on the word or- der constraints and thus poses higher challenge to reordering algorithms. The Multilingual TOP dataset contains examples in English, Italian, and Japanese and is based on the TOP dataset (Gupta et al., 2018). Similarly the MTOP dataset, this dataset uses the decoupled representation. Both datasets are formulated as a seq2seq task. We use the standard splits for training and evaluation.
4.2 Evaluation Tasks
In this section, we describe the tasks we use for evaluation, the models we use for performing the tasks, and the datasets for training and evaluation. All datasets, other than the manually annotated UD corpora, are tokenized and parsed using Trankit (Nguyen et al., 2021). Some datasets contain sub- sequences that must stay intact in order for the annotation to remain valid (e.g.. proper-name sequence such as The New York Times may have internal structure but cannot reordered). cases where these subsequences are not part of a single subtree, we manually alter the tree to make them
Models. We use two seq2seq models in our eval- uation, a pointer-generator network model (Ron-
dering as the "correct" one. If this ordering contradicts the original English one, it will be both nearly 50% incorrect and highly unnatural for the encoder. Fnsembling thus ensures that the effect of estimation crrors is bounded.
Note that Thai has only one published UD treebank, so for this experiment we split it in two parts, 500 sentences cach, for estimating POCs and testing.

---

gali et al., 2020) and mT5 (Xue al., 2021). The pointer-generaie
5 Results and Discussion
network was used in previous works on these datasets (Xia and Monti, 2021; Li et al., 2021); ;;; it includes SLM-RoBERTa-knn (Conneau et al., 2020) as the encoder and an unini- tialized Transformer as a decoder. In this method, the target sequence is comprised of ontology 1o- kens, such as :NN:END MESSAGE in the MTOP dataset, and pointer tokens representing tokens from the source sequence (e.g. ptro, which repre- sents the first source-side token). When using mT5, we use the actual tokens and not the pointer tokens as mT5 has abling the model to utilize it. In both models, we report the standard exact-match (EM) metric, aver- aged over 10 runs for the pointer-generator model and runs for mT5.
The results on the various tasks, namely UD pars- ing, semantic parsing, and relation classification, are presented in Tables 1, 2 and 3 respectively. The few-shot experiment results are in Table Stan- dard deviations are reported in Appendix E.
In UD parsing, the ENSEMBLE setting presents noticeable improvements for the languages that are more typologically distant from English (2.3-4.1 LAS points and 1.8-3.5 UAS points), with the ex- ception of Arabic, in which the scores slightly drop. No noticeable effect is observed for structurally closer languages.
built into it, thus en-
In the STANDARD setting, a smaller increase in performance is present for most distant languages, with decrease in performance for closes ones, Persian and Arabic. This is in agreement with pre- vious work that showed that reordering algorithms are more beneficial when applied structurally- divergent language pairs (Wang and Fisner, 2018; Rasooli and Collins, 2019). The ENSEMBLE ap- proach, therefore, seems to be essential for a gener- ally applicable reordering algorithm.
4.2.3 Relation Classification
Datasets. We use two sets of relation-extraction
datasets: (i) TACRED (Zhang et al., 2017b) (TAC)
and Translated TACRED (Arviv et al., 2021)
(Trans-TAC), and (ii) IndoRE (Nag et al., 2021).
TAC is relation dataset with over 100K examples in English, covering 41 relation types. The Trans-TAC dataset contains 533 parallel exam- plcs sampled from TAC and translated into Russian and Korean. We use the TAC English dataset for training and Trans-TAC for evaluation. As the TAC train split is too large for efficient training, we only use the first 30k examples.
The algorithm by Rasooli and Collins (2019) (RC19), in both settings, presents smaller increase in performance for some typologically-diitant lan- guages and no noticeable improvements for others, while sometimes harming the results. This suggests that for this task the surface statistics the algorithm uses are not enough and a more fine-grained ap- proach in needed.
IndoRE (Nag et al., 2021) contains 21K sen- tences in Indian languages (Bengali, Hindi, and Telugu) plus English, covering 51 relation types. We use the English portion of the dataset for train- ing. and the Hindi and Telugu languages for evalu- ation.! 12
In the semantic- partiiipppinnnggg task, the reordering algo- rithm presents substantial improvements for all lan- guages but Italian in the ENSEMBLE setting (2-6.1 increase in exact match), for the RoBERTa based model. Noticeably, the gains are achieved not only for typologically- languages but also for lan- guages close to English, such as French. In the MTOP dataset, the ENSEMBLE setting proves big- ger gains over the STANDARD for all languages. In Multilingual-
Model. We use the relation-clan classificiisiiiiatiionn part of the LUKE model (Yamada et al., 2020). 13 The model uses two special tokens to represent the head and the tail entities in the text. The text is fed into an encoder, and the task is solved using a linear classifier trained on the concatenated representation of the head and tail entities. For consistency, use XLM- MoM--RoBERTaaaaa (Conneau et al., 2020) the encoder. We report the micro-F1 and macro- F1 metrics, averaged over 5 runs.
surprisingly observe the op- posite. Given that Japanese terms of word order is comparable Hindi and Italian, to French, we tend to attribute this result to the peculiarities of the dataset. This, however, merits further analysis.
When compared to RC19, the proposed algo- rithm consistently outperforms it, by an average of about 2 points (in the ENSEMBLE setting).
The Bengali UD treebank extremely small (17 sen- tences), which makes impossible to extract high-quality POCs. Phttps:// //g: thub. com/ hommsstudia. wowoooosaaauoe
For mT5 we observe increase in performance, in the ENSEMBLE setting, of 2.5 and 5 points in Thai and Hindi, respectively. For the other languages,

---

Ours
OursE
RC19
RC19E
OursE
Target
LAS gain
UAS
LAS
UAS
LAS
UAS
LAS
UAS
LAS
UAS
LAS
Arabic
67.5
52.3
65.9
50
66.5
51.2
66.9
53.2
67.6
53.7
-1.1
Spanish
84.7
77.2
83.6
75.8
84.8
77.1
84,4
77.3
84.3
77.2
-0.1
German
86.6
80.6
83,5
77.1
86.7
80.7
85.5
78.3
85,3
78.1
0.1
French
85
79.2
82
76.3
84.9
79.4
84.5
79.9
84.4
79,9
0.2
Persian
66.2
52.3
51
40.1
66.4
52.7
66.4
54,1
53.7
43.1
0,4
Korean
64
46.9
66.6
49.2
65.8
49.2
63.1
46,4
63.1
46,2
2.3
Irish
54.4
38,9
54.4
38.4
58.4
41.5
60
42.4
59
41.6
2.6
Thai
73.8
54.5
75.5
57.7
76
58
73.7
52.7
72.7
51.4
3.5
Hindi
60.4
49.8
61.1
50.9
63.9
53.9
60.7
49.9
61.4
50.9
4.1
Table 1: The results (averaged over 5 models) of the application the reordering algorithm to cross- lingull UD parsing. Columns correspond to evaluations settings and score types; rows correspond to evaluation-dataae languages. The best LAS and UAS scores per language are represented in boldface and underlined, respectively. Abbreviations: RC19 the algorithm by Rasooli and Collins: E- the ENSEMBLE setting.
Target
Arch
Base
Ours
OursE
RC19
RC19E
Micro and Macro points). In the STANDARD set- ting, there is a drop in performance for Hindi and Telugu. When compared to RC19, our algorithm outperforms it in the ENSEMBLE setting, by more than 5 points for Korean, but only by 0.5 points for
XLM
36.9
41.5
42.9
39.3
40.4
Hi
mT5
22.7
24.8
27.7
21.8
24
XDM
13.3
73.8
17.9
T5.9
17.3
Th
mTS
31.2
31.8
33.6
31.7
32,8
XLM
52.9
54
31.1
35.9
Fr
mT5 46.7
43.9
46.3
44.4
46.7
Russian. In Hindi and Telugu, the performance of
XLM
55.3
35.5
59.6
34.6
37.6
both algorithms is close, and RC19 does perform
mT5
46.9
43.2
46.2
47
47.9
XLM
53.9
33.7
56.4
31.3"
34.7
better in some cases.
Ge
mT5
39.9
38.4
40.6
37.8
41.4
Ja
XLM
5.5
9.7
7.5
8.5
7,9
Comparison between Encoder-with- Classifier-Head and Seq2Seq Models
It
XLM
54.4
54.7
54
53.2
Table 2: The results (averaged over 10 models for XLM and mT5 respectively) of the application of the reordering algorithm to MTOP and Multilingual- Top (above and below the horizontal line respectively). Values are exact-match scores. The best score per lan- guage is represented in boldface. Abbreviations: XLM XLM-RoBERTa; RC19 the algorithm by Rasooli and Collins; E the ENSEMBLE setting. Language abbrevia- tions: Hi Hindi, Th Thai, Fr French, Sp - Spanish, Ge German, Ja Japanese, It Italian.
Past work has shown that the architecture is an im- portant predictor of the ability of a given model to generalize over cross-lingual word-order diver- gences. For example, Ahmad et al. (2019) showed that models based on self-attention have better overall cross- liosslln transferability to distant lan- guages than those using RNN-based architectures.
One of the dominant trends in recent years for-
NLP has been using the mulation to solve an increasing variety of tasks (Kale and Rastogi, 2020; Lewis et al., 2020). De- spite that, recent studies (Finegan-Dollak et al., 2018; Keysers et al., 2019; Herzig and Berant, 2019) demonstrated that such models fail com- positional generalization, that is, they do not gen- eralize to structures that were not seen training time. Herzig and Berant (2021) showed that other model architectures can prove advantageous over seq2seq architecture in that regard, but their work was limited to English.
we do not observe a strong impact. We note how- ever, that in French and Spanish, there is a slight drop in the score (less then 1 point). When com- pared RC19, our method provide larger gains in Hindi and Thai.
In the few-shot scenario, observe improved performances for all languages and sample sizes. Surprisingly, the improvements hold even when training on large sample size of 500, indicating that the model not able to easily adapt to the target word-order.
Here, take the first steps in examining the cross-lingual transfer capabilities of the seq2seq encoder-decoder architecture (S2s) vs. a classifica- tion head stacked over an encoder (E+C), focusing on their ability to bridge word-order divergences.
Lastly, the ellatin-classificattee task, in the ENSEMBLE setting we observe an increase in per- formance for all languages (2.3-10.4 increase the

---

Base
Ours
OursE
RC19
RCI9E
Target
Mic-Fi Fl
Mac-F]
Mic-Fl
Mac-FI
Mic-FI
Mac-F1
Mic-Fl
Mac-FI
Mic-FI
Mac-FI
Korcan
40.3
33.4
41
35.4
50
43.8
48.2
43.1
44.2
38.8
Russian
62.7
58.1
66
61.7
68.1
64.2
61.1
55.8
67.6
63.8
Hindi 78.5
74.1
77.1
72.2
80.9
76.7
80
75.4
80.7
77
Telugu
66.6
58.9
65.5
58.3
69.1
62
70.3
61.8
69.4
62.3
Table 3: The results (averaged over 5 models) of the application of the reordering algorithm to Translated Tacred and IndoRE (above and below the horizontal line respectively). Columns correspond to evaluations settings and score types; rows correspond evaluation-dataset languages. The best Miero-FI and Macro-FI scores, per language, are represented in boldface and underlined, respectively. Abbreviations: Mic-FI Micro-FI, F1; Mac-F1 Macro-F1; RC19 the algorithm by Rasooli and Collins; the ENSEMBLE setting.
Hindi
Thai
French
Spanish
German
Sample size
Base
OursE
Base
OursE
Base
OursE
Base
OursE
Base
OursE
100
45.2
48.3
47.6
48.4
62.3
63.1
62.9
64.2
57.8
58.7
300
53.3
56
54.3
55.8
67
67.2
67.7
68.1
61.6
62.9
500
56.9
58.4
58.6
59.7
67.7
68.8
70
71
63.2
65.1
Table 4: The results (averaged over 5 models) of the application of the reordering algorithm MTOP in the few-shot scenario, on the mT5 model. Columns correspond to evaluations settings; rows correspond to evaluation-datase languages. Values are exact-match scores. E the ENSEMBLE setting.
6.1 Experimental Setup
Base
Ours
OursE
Target
We compare the performance of an E+C model against a s2s one on the task of UD parsing over various target languages. Similar to $4, we train each model on the vanilla English dataset and com- pare it against model trained on version of the dataset reordered using algorithm. We evaluate the models using the target-language test set in zero-shot setting.
E+C
$28
E+C
$2s
B+C
$2s
French
85,9
46.6
82.7
42.7
86.12
German
87.4
$4.5
83.6
56.9
87.5
67.8
20.6
55.2
58.4
48.7
Korean
50.9
14.2
53.4
13.8
53.4
16.5
Persian
56.7
21.4
54.1
24.5
57.1
27.3
Spanish
83.7
63.4
82.2
59.5
83.6
64.7
Thai
59.9
28
61.1
29.5
63.7
37.9
Table 5: LAS results (averaged over 5 models) of the application of the reordering algorithm cross-lingual UD parsing, normalized by the English test set LAS. Columns correspond to the evaluations settings, and model types; rows correspond evaluation dataset lan- guage. The best scores for the E+C and s2s settings, per language, are represented in boldface and under- lined, respectively. Abbreviations: setting; E+C classification head stacked over an en- coder architecture; s2s - seq2seq encoder-decoder architecture.
Dataset and POC's Estimates. We use the same UD dataset and POCs as in $4. For the s2s task, we linearize the UD parse tree using the method by Li et al. (2018).
Models. For the E+C model we use the deep biaffine attention graph-based model with XLM- RoBERTa-large as the encoder, as in $4.2.1. For s2s model, we use the standard transformer archi- tecture with XLM-RoBERRTa-larg as the encoder and an uninitialized self-attention stack as the de- coder. The perammparaeeters for the models are given in Appendix D.
the ENSEMBLE
then 50%), despite relying on the same underline multilingual LM. Furthermore, the s2s architecture benefits more strongly from reordering for distant languages (more than twice much), compared to the E+C one. This suggests that seq-to-sequence architecture may be less effective handling cross- lingual divergences, specifically word-order diver- gences, and may gain more from methods such as reordering.
6.2 Results and Discussion
The results for LAS (averaged over 5 runs), nor- malized by the base parser performance on the English test-set, are presented in Table (UAS follow the same trends. See full scores in Ap- pendix F). The zero-shot performance of the s2s model is subpar compared to the E+C one (less

---

7 Conclusion
for closely-related languages. Secondly, as this is pre-processing algorithm, it incurs time and com- putation overhead for tokenization, parsing, and reordering of the source dataset. Most importantly, if one wishes to train a model on several word order patterns, it will be necessary to apply the al- gorithm repeatedly to create new dataset for each pattern, increasing the overall size of the dataset. Furthermore, extracting the POCs requires external resources, either UD corpora in the target language or multi-paralle corpus for annotation projec- tion. More technical limitations of the reordering algorithm are described in Appendix A.
We presented a novel pre-process pre-processing reordering ap- proach that is defined in terms of Universal Depen- dencies. Experiments on three tasks and numerous architectures and target languages demonstrate that this method is able to boost the performances of modern multilingual LMs both in the zero-shot and few-shot setting. Our key contributions include a new method for reordering sentences based on fine-grained word-order statistics, the Pairwise Or- dering Distributions, using an SMT solver to con- vert the learned constraints into a linear ordering, and a demonstration of the necessity of combin- ing the reordered dataset with the original one (the ENSEMBLE setting) in order to consistently boost performance.
Acknowledgements
This work was supported in part by the Israel Sci- ence Foundation (grant no. 2424/21).
Our results suggest that despite the recent im- provements in multilingual models, they still face difficulties in handling cross-lingual word order di- vergences, and that reordering algorithms, such as ours, can provide a much needed boost in perfor- mance in low-resource languages. This result holds even in the few-shot scenario, when the model is trained on few hundred examples, underscoring the difficulty of models adapt varying word or- ders, as well as the need for more typologically diverse data, additional inductive bias at the train- ing time, or a pre-processing approach such as ours to be more effective. Furthermore, our experiments suggest that seq2seq encoder-decoder architectures may suffer from these difficulties to a bigger extent than more traditional modular ones.
References
Željko Agić, Anders Johannsen, Barbara Plank, Héc- for Martínez Alonso, Natalic Schluter, and Anders Søgaard. 2016. Multilingual projection for parsing truly low-resource languages. Transactions of the As- sociation for Computational Linguistics, 4:301-312.
Wasi Ahmad, Zhisong Zhang, Xuezhe Ma. Eduard Hovy, Kai-Wei Chang, and Nanyun Peng. 2019. On difficulties of cross- cross-lingual transfer with order dif- ferences: case study on dependency parsing. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2440-2452, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Future work will include, firstly, addressing the limitations of the proposed approach in order to make it less language- pair dependent and reduce the computational and storage overhead, and sec- ondly, leveraging the POCs in order to compute the word-order distance between languages in a rich, rigorous corpus-based way, 14 and to more pre- cisely predict when the reordering algorithm will be beneficial as well to provide fine graiinained analysis of the connection between word order and cross-lingual performance, in line with Nikolaev et al. (2020).
Kabir Ahuja, Rishav Hada, Millicent A. Ochieng. Prachi Jain, Harshita Diddce, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. 2023. Mega: Multilingual evalua- tion of generative ai. ArXiv, abs/2303. 12528.
Waleed Ammar, George Mulcaire, Miguel Ballesteros. Chris Dyer, and Noah A. Smith. 2016. Many lan- guages, one parser. Transactions of the Association for Computational Linguistics, 4:431 144.
Ofir Arviv, Dmitry Nikolaev, Taclin Karidi, and Omri Abend. 2021. On the relation between syntactic di- vergence and zero-shot performance. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 4803-4817, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Limitations
There are several limitation to our work. First, as shown in the experiment results, for some tasks, reordering, even with ensembling, is not beneficial
Akari Asai, Sneha Kudugunta, Xinyan Velocity Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannaneh Hajishirzi.
S, for example, only provides single categorical label for "dominant word order".

---

2023. Buffet: Benchmarking large language mod- els for few-shot cross- minssllingull transfer. ArXiv, abs/2305. 148577
Catherine Finegan- Finegan-Dollak,, Jonathan K. Kummerfeld. Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving text- to-SQL evaluation methodology. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 351-360, Melhourne, Australia. Association for Computational Linguistics.
Lauriane Aufrant, Guillaume Wisniewski, and François Yvon. 2016. Zero-resource dependency parsing: Boosting delexicalized cross-lingual transfer with linguistic knowledge. In Proceedings of COLING 2016, the 26th International Conference on Computa- tional Linguisties: Technical Papers. pages 119-1300 Osaka, Japan. The COLING 2016 Organizing Com- mittee.
Matt Gardner, Joel Grus, Mark Neumann. Oyvind Tafjord, Pradeep Dasigi, Nelson F, Liu, Matthew Pe- ters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language pro- cessing platform. In Proceedings of Workshop for NIP Open Source Software (NLP-OSS) pages 1-6, Melbourne, Australia. Association for Computational Linguisties.
Pi-Chuan Chang and Kristina Toutanova. 2007. A dis- criminative syntactic word order model for machine translation. In Proceedings of the 45th Annual Meet- ing of the Association of Computational Linguistics, pages 9-16, Prague. Czech Republic. Association for Computational Linguistics.
Dmitriy Genzel. 2010. Automatically learning source- side reordering rules for large scale machine transla- tion. In Proceedings of the 23rd International Con- ference on Computational Linguistics (Coling 2010), pages 376-384, Beijing. China. Coling 2010 Orga- nizing Committee.
Kehai Chen, Rui Wang, Masao Utiyama, and Eiichiro Sumita. 2019, Neural machine translation with re- ordering embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1787-1799, Florence, Italy. Asso- ciation for Computational Linguistics.
Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Ku- mar, and Mike Lewis. 2018. Semantic parsing for task oriented dialog using hierarchical representa- tions. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2787-2792, Brussels, Belgium. Association for Computational Linguistics.
Michael Collins, Philipp Koehn, and Ivona Kučerová. 2005. Clause restructuring for statistical machine translation. In Proceedings of the 43rd Annual Meet- ing of the Association for Computational Linguistics (ACL '05), pages 531-540, Ann Arbor, Michigan. As- sociation for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
John A Hawkins. 1992. Syntactic weight versus infor- mation structure in word order variation. Informa- tionsstruktur und grammatik, pages 196-219.
Jonathan Herzig and Jonathan Berant. 2019. Don't para- phrase, detect! rapid and effective data collection for semantic parsing. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP)), pages 3810-3820, Hong Kong, China, Association for Computational Linguistics.
Leonardo de Moura and Nikolaj Björner. 2008 Z3: An efficient smt solver. In Tools and Algorithms for the Construction and Analysis of Systems. pages 337- 340, Berlin, Heidelberg. Springer Berlin Heidelberg.
Jonathan Herzig and Jonathan Berant. 2021. Span- based semantic parsing for compositional general- ization. Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th 1thh International Joint Conference on Natu- ral Language Processing (Volume I: Long Pupers). pages 908-921, Online. Association for Computa- tional Linguisties.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Julian Hitschler, Laura Jehl, Sariya Karimova, Mayumi Ohta, Benjamin Körner, and Stefan Riezler. 2016. Otedama: Fast rule-based pre-ordering for machine translation. The Prague Bulletin of Mathematical Linguistics, 106:159- 168.
Timothy Dozat and Christopher D. Manning. 2016. Deep biaffine attention for neural dependency pars- ing. ArXiv, abs/161 1.01734.
Matthew Dryer. 1992. The Greenbergian word order correlations. Language, 68(1):81-138.
Tao Ji, Yong Jiang, Tao Wang, Zhonggiang Huang. Fei Huang, Yuanbin Wu, and Xiaoling Wang. 2021.

---

Word reordering for zero-shot cross-lingual struc- tured prediction. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 4109-4120, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
pre-training for neural machine translation. Transac- tions of the Association for Computational Linguis- tics, 8:726-742.
Arya D. McCarthy, Rachel Wicks, Dylan Lewis, Aaron
Mueller, Winston Wu, Oliver Adams, Garrett Nicolai, Matt Post, and David Yarowsky. 2020. The Johns Hopkins University Bible corpus: 1600+ tongues for typological exploration. In Proceedings of the Twelfth Language Rexources and Evaluation Confer- ence, pages 2884-2892, Marseille, France. European Language Resources Association.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages
6282-6293, Online. Association for Computational Linguistics.
Tao Meng, Nanyun Peng, and Kai-Wei Chang. 2019. Target language- constrained inference for cross booss-lingual dependency parsing. In Proceedings of the 2019 Conference on Empirical Methods Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLPP) pages 11 1117-112 128, Hong Kong, China. Association for Computational Linguistics.
Mihir Kale and Abhinav Rastogi. 2020. TexI-to-text
pre-training for data-to-text tasks. Proceedings of the 13th International Conference on Natural Lan- guage Generation, pages 97-102, Dublin, Ireland. Association for Computational Linguisties.
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang. Mare van 7ee, and Olivier Bousquet. 2019. Measur- ing compositional generalization: A comprehensive method on realistic data. ArXiv, abs/
Rudra Murthy, Anoop Kunchukuttan, and Pushpak Bhat- tacharyya. 2019. Addressing word-order divergence in multilingual neural machine tramslation for ex- tremely low resource languages. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3868-3873, Minneapolis, Min- nesota. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Chazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. pre-training
BART: Denoising sequence-to-sequeee for natural language generation, translation, and com- prehension. Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Arijit Nag. Bidisha Samanta, Animesh Mukherjee, Niloy Ganguly, and Soumen Chakrabarti. 2021. A data bootstrapping recipe for low-resource multi- lingual relation classification. In Proceedings of the 25th Conference Computational Natural Lan- guage Learning, pages 575-587, Online. Association for Computational Linguistics.
Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad, 2021. MTOP: A comprehensive multilingual task-orienn oriented semantic parsing benchmark. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 2950-2962, Online. Association for Computa- tional Linguisties.
Minh Van Nguyen, Viet Dac Lai, Amir Pouran Ben Vey- sch, and Thien Huu Nguyen. 2021. Trankit: A light- weight transformer-baa ural language processing. In Proceedings of the 16th Conference of the European Chapter of the Associa- tion for Computational Linguisties: System Demon- strations, pages 80-90, Online. Association for Com- putational Linguistics.
toolkit for multilingual nat-
Zuchao Li. Jiaxun Cai, Shexia Hc, and Hai Zhao. 2018. Seq2seq dependency parsing. In Proceedings of the 27th International Conference on Computational Lin- guistics, pages 3203-3214, Santa Fe. New Mexico, USA. Association for Computational Linguistics.
Dmitry Nikolaev, Ofir Arviv, Taelin Karidi, Neta Ken- neth, Veronika Mitnik, Lilja Maria Saeboe, and Omri Abend. 2020. Fine-grained amalysis of cross- linguistic syntaetic divergences. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, pages 1159-1176, Online. As- sociation for Computational Linguistics.
Lu Liu, Yi Zhou, Jianhan Xu, Xiaoqing Zheng, Kai- Wei Chang. and Xuanjing Huang. 2020a. Cross-lingus dependency parsing by POS-guided word reordering. In Findings of the Association for Computational Online.
Dmitry Nikolaev and Sebastian Pado. 2022. Word- order typology in multilingual BERT: A case study in subordinate-claus the 4th Workshop on Research in Computational Lin- guistic Typology and Mulrilingual NLP, pages 11-21, Seattle, Washington. Association for Computational Linguistics.
Linguistics: EMNLP 2020, pages Association for Computational Linguisties.
detection. In Proceedings of
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Bdunov, Marjan Ghazvininejad, Mike Lewis. and Luke Zettlemoyer. 2020b. Multilingual denoising

---

Mohammad Sadegh Rasooli and Michael Collins. 2017. Cross-lingual syntactic transfer with limited re- sources. Transactions of the Association for Compu- tational Linguistics, 5:279-293.
Brussels, Belgium, Association for Computational Linguistics.
Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu. 2019. Cross-lingual BERT transformation for zero-shot dependency parsing. In Proceedings of the 2019 Conference on Empirical Methods Natu- ral Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pages 5721-5727, Hong Kong. China. Association for Computational Linguistics.
Mohammad Sadegh Rasooli and Michael Collins. 2019, Low-resource syntactic transfer with unsupervised source reordering. In Proceedings of the 2019 Con- ference of the North American Chapter of the Asso- ciation for Compitational Linguistics: Human Lan- guage Technologies, Volume I (Long and Short Pa- pers), pages 3845-3856, Minneapolis, Minnesota. Association for Computational Linguistics.
Menglin Xia and Emilio Monti. 2021. Multilingual neu- ral semantic parsing for low-resourced languages. In Proceedings of *SEM 2021: The Tenth Joint Confer- ence on Lexical and Computational Semantics, pages 185-194, Online. Association for Computational Lin- guistics.
Subendhu Rongali, Luca Soldaini, Emilio Monti, and Wael Hamza. 2020. Don't parse, generate! se- quence to sequence architecture for task- oriented se- mantic parsing. Proceedings of The Web Conference 2020.
Haoran Xu and Philipp Koehn. 2021. Zero-shot cross- lingual dependency parsing through contextual em- bedding transformation. In Proceedings of the Sec- ond Workshop on Domain Adaptation for NLP, pages Kyiv, Ukraine. Association for Computa- tional Linguistics.
Sebastian Ruder, Clark, Alexander Gutkin, Mihir Kale, Min Ma, Massimo, Nicosia, Shruti Rijhwani, Parker Riley, Jean Michel A. Sarr, Xinyi Wang, John Wieting. Nitish Gupta, Anna Katanova, Christo Kirov, Dana L. Dickinson, Brian Roark, Bidisha Samanta, Connie Tao, David Ifeoluwa Adelani, Vera Axelrod, Isaac Caswell, Colin Cherry, Dan Gar- rette, R. Reeve Ingle, Melvin Johnson, Dmitry Pan- teleev, and Partha Pratim Talukdar. 2023, Xtreme- up: A user-centric scarce-data benchmark for under- represented languages. ArXiv, abs/2305. 1938.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami A1-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. m5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ler of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics.
Tanja Samardžić, Ximena Gutierrez- Vasques, Rob van der Goot, Max Müller-Eberstein, Olga Pelloni, and Barbara Plank. 2022. On language spaces, scales and cross-lingual transfer of UD parsers. In Proceed- ings of the 26th Conference on Computational Nat- ural Language Learning (CoNLL), pages 266-281, Abu Dhabi, United Arab Emirates (Hybrid), Associa- tion for Computational Linguistics.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep contextualized entity representations with entity- aware self-attention. In Proceedings of the 2020 Conference Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 1442266544 On- line. Association for Computational Linguistics.
Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019, Cross-lingue alignment of con- textual word embeddings, with applications to zero- shot dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computaatonall Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 1599-1613 1613, Minneapolis, Minnesota. Association for Computational Linguistics.
Jinchao Zhang, Mingxuan Wang, Qun Liu, and Jie Zhou. 2017a. Incorporating word reordering knowledge into attention- based neural machine translation. In Proceedings of the 55th Anmual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers), pages 1524-1534, Vancouver, Canada. Association for Computational Linguistics.
Yuhao Zhang, Victor Zhong. Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017b. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 35-45, Copenhagen. Demmark. Association for Com- putational Linguistics.
Ralf Steinberger. 1994, Treating 'free word order' in machine translation. In COLING 1994 Volume 1:
The 15th International Conference Computational Linguistics, Kyoto, Japan.
Dingquan Wang and Jason Eisner. 2016. The galac- tic dependencies treebanks: Getting more data by synthesizing new languages. Transactions of the As- sociation for Computational Linguistics, 4:491-50s
A Shortcomings of the Reordering Algorithm
Dingquan Wang and Jason Eisner. 2018. Synthetic data made to order: The case of parsing. In Proceed- ings of the 2018 Conference on Empiricul Methods in Natural Language Processing, pages 1325-1337,
We note a couple of possible shortcomings to this approach. First, while a pair of constraints can- not be straightforwardly contradictory (both values

---

cannot be set to 1), it can be uninformative (both values set to 0) when not enough label-orderigg data is presented the training treebank, which means that the ordering of the corresponding nodes is not subject to any constraint.
Language
UD Treebank
Size
French
French-GSD
14450
Spanish
Spanish-GSi GSDD
14187
German
German GSD
13814
Italian
Italian-ISDD
13121
Moreover, is possible to encounter loops or transitivity conflicts when joining different con- straints, which makes it priori impossible for the solver to satisfy them. To alleviate this problem, for each subtree we aim reorder, we only consider the constraints that are relevant it. For example. if the subtree does not contain any token with label nmod, we discard all the constraints which include this label, such as amod: (nmod < amod). This, together with the tendency of languages to have preferred ordering to their constituent elements, makes it that only small percentage of subtrees cannot be ordered.
Russian
Russian-GSD
3850
Irish
Irish-IDT
4005
Arabic
Arabic-PUl
1000
Korean
Korean Korean-GSD
4400
Japanese
Japanese-GSD
7050
Thai
Thai-PUD
1000
Persian
Persian-PerDD
26196
Hindi
Hindi-HDTB 13306
Telugu
Telugu-MTG 1051
Table 6: The UD treebanks used to estimate the POCs.
Each row corresponds to a Language-treebank pair. Tree- bank size is measured in sentence counts.
Last, sparsity issues may prevent some con- straints from being statistically justified, and round- ing the constraints to a hard 0 or 1 may re- sult in information loss and thus be detrimen- tal. For example, if our pairwise distributions are P. ramolaaa = 0.51 and prodd 0.49, deriving a constraint of nmod (nmod amod) may not be warranted. This may also hap- pens in the case of a highly flexible order of particular pair of syntactic elements in the target language. If this ordering contradicts the origi- nal English one, it will be both nearly 50% incor- rect and highly unnatural for the encoder. This is, however, partially mitigated by the ENSEMBLES method. For the purposes of this work we do not distinguish between POCs according their tistical validity and defer this question to future work.
Base
Ours
OursE
Target
UAS
LAS
UAS
LAS
UAS
LAS
French
44.3
39.1
41.1
35.8
45.4
40.1
German
51.2
45.7
53.7
47.7
62,4
56.8
Spanish
59.7
53.1
56.5
49.9
60.6
54.2
Korean
26.4
11.9
26.6
11.6
28.3
13.8
Persian
27.3
17.9
29
20.5
32.4
22.9
Hindi
26.9
17.3
41.1
31.9
50.4
40.8
Thai
35.6
23.5
37.7
24.7
45.5
31.8
Table The results (averaged 5 models) of the application of the reordering algorithm to cross-lingual s2s UD parsing. Columns correspond to evaluations settings and score types; correspond evaluation- dataset languages. Abbreviations: RC19 - the algorithm by Rasooli and Collins; E the ENSEMBLE setting.
acl vs. advmod
advmod->ucl:
acl-ooddood:
B Estimating the POCs
acl vs. amod
We estimate the POCs ($3.2) by extracting the em- pirical distributions from UD treebanks. The UD treebanks used are reported in Table 6.
amod->ael:
acl->amod: 1
acl vs. appos
C Example of Learned Distributions
appos-sacl: 3
acl->appos: 4
Here are the statistics of the pairwise ordering of main elements of the matrix c5 learned on the Irish-IDT treebank:
acl vs. nsubj
nsubj->acl:
acl->nsubj: 8
acl vs. advel
acl vs. obj
acl->advel: 10
obj->acl: 3
advel->acl: 4
acl vs. obl
15 Elements that are directly under the root node.
obl->acl: 14

---

Base
OursE
obj >advmod:
Tagett
advmod-2obj: 77
UAS
LAS
UAS
LAS
UAS
LAS
French
1.1
0.5
0.6
1.6
1.7
advmod vs. ohl
German
1.2
1.2
0.4
0.3
0.4
0.3
obl->advmod: 231
Spanish
0.7
0.6
0.4
0.4
0.2
0.3
Korean
0.5
0.5
0.7
0.6
0.4
0.5
advmod-2obl: 326
Persian
0.5
0.4
0.7
0.6
0.4
0.4
advmod vs. root
Hindi
0.7
0.5
0.6
0.5
0.7
0.7
Thai
0.9
0.7
2.8
2.6
1.2
2.2
advmod-2root: 111
root- root->advmod: 486
Table 8: The standard deviation of the results (aver- aged over 5 models) of the application of the reordering algorithm to cross-lingual s2s UD parsing. Columns correspond to evaluations settings and score types; correspond to ovaluation-datasel languages. Abbrevia- tions: RC19 algorithm by Rasooli and Collins; E. the ENSEMBLE setting.
amod vs. appos
amod->appos: 2
amod vs. nsubj
nsuhj->amod: 13
amod->nsubj: 41
amod vs. ohj
obj->amod: 3
acl->obl:
amod vs. obl
acl vs. root
obl->aamo:: 7
root->acl: 144
ammod-sobl:
advel vs. advel
amod root
advel-sadvel:
rroot>>mmo:
advel vs. advmod
amod->root: 6
adveod-zadvel: 72
appos nsubj
advel->advmod: 22
nsubj->appos:
advel vs. amod
appos->nsubj: 4
amod->advel:
appos obl
advcl-samod: 5
obl->appos:
advel vs. appos
appos root
advel-2appos: 2 advel vs. nsubj
root->appos: 32
nsubj obj
advel->nsubj: 132
insuj--2avvcl: 320
nsubj->obj: 469
obj->nsubj: 2
advel vs. obj
nsubj vs. obl
-bbj->ddvcl: 128
advel-zohj: 48
nsubj->ohl: 1699
oobl>>nubbj: 314
advel vs. obl
nsubj vs. root
obbl>>adell: 339
advel-Sobl: 189
root->nsubj: 2423
nsubj->root:
vs. root
-root-savel:: 513
ohj vs. obl
advel-2root: 243
-obj->obl: 755
advmod vs. amod
obb->>ojj: 219
advmod-samod: 3
obj vs. root
amod-zadvmod:
root->obj:
adymod vs. nsubj
obj->root: 13
nsubj->advmod: 337
obl
advmod->nsubj: 83
root->obl: 2566
adymod vs. obj
obl->root: 425

---

As expected, Irish behaves as strict head-initial language: root overwhelmingly precedes all other constituents, including subordinate clauses, and modifier subordinate clauses (ac1, advel) follow nominal clause participants (nsubj, obj, obl). Adjectival modifiers, however, mostly pre- cede nominal elements; this may be due to the fact that some frequent pronominal adjectives, such as uile 'all' and gach 'every' do not follow the general rule and precede the nouns they modify.
The position of adverbial modifiers is not re- stricted by the grammar, and it may noted that generally follows nominal subjects but as often as not precedes direct objects, and in 3/5 of cases pre- cedes obliques, which suggests the general order root nsubj >> advmod/ obj >> obl.
D Models Hyperparameters
The hyperparameters of the UD parser are given in Table 9. For the seq2xeq pointer-generator network model in Table 10, for mT5 in Table 11, and for LUKE relation classification model in Table 12. the same hy-
For the UD Seq2Seq parser, perparameters as for the seq2seq pointer-generator network model, with the following exceptions: train for only 50 epochs and the learning rate to lc-5 for both the encoder and decoder.
E Standard Deviations
The standard deviations of the results of the periments in UD parsing, semantic parsing, and relation classification are presented in Tables 14, 13, and 15 respectively.
UD Seq2Seq Model Performances
The full results (averaged over 5 models) of the s2s model in UD parsing, are presented in Table 7. The standard deviations in Table 8.

---

Hyperparameter
Value
Dataset Reader
input tokens
100
Encoder
type
xim-roberta-large
train parmmpareetters
true
Model
(General)
lype:
biaffine__aane parser
wwwpeeeentttinnnnnnoo __dim
500
ing_representationndim
100
dropout
0.1
input_dropout
0.3
Training
num num__pncchs patience
100
grad_norm
Optimizer
type
huggingface adamw
le-5
weight_decay
0.01
Table 9: Hyperparameterss for the biaffine UD parser.
Hyperparameter
Value
Dataset
imput max tokens
100
Reader
Encoder
type
Alm-roberta-large
train parameters
true
Decoder
type
stacked_sell attention
stacked self
num layers
num num attention heads
decoding dim
1024
target_eemhedee wwweedding......e dim feedforward hidden dim pointers_con copy__opppp..c. copy_mcchanisem mechanisem (if using pointers)]
1024
512
BilinearAttentin
Model (General)
label www.ssmmoooraattttt ratio
beaa_eeaaii___ee__aaee_aee size
4
beam_ m_a__ma__ss___ steps
100
Training
epochs
100
patience
None
grad_nov
5.0
Optimizer
type
hugginglace_adee _adamm
encoder: 1e-3
Ir
decoder: lc-5
veight_decay learning_ rate scheduler
slanted triangular
gradual_unfrezzing
true
Table 10: Hyperparameters for the pointer network generator seq2seq model.
Hyperparameter
Value
Dataset Reader
input max tokens
100
Model
(General)
type:
gongle/mt5-hass
lahel smoothimottgggggoo ratio
0.1
beam_ sea_seeachhhhhee search_beam_sinee beam_size
4
heam_ seam_ssarchh.. max_mmaaattt steps
100
Training
num epochs
50
patience
None
grad norm lype Ir
5.0
Optimizer
huggingface-aalane
le-5
weight_dccay
0.01
Table 1I: Hyperparameters for m m55

---

Hyperparameter
Value
Encoder
type
xim-roberta-large
train parameters
true
Seq2vee Encoder
type: dropout
bert pooler
0.1
Training
num_epochs
100
patience
grad
5.0
Optimizer
type
huggingface_adaen
le-5
weight_decay
0.01
Table 12: Hyperparameters for the LUKE relation classification model.
Target
Arch
Ours
OursE
RC19
RC19E
XLM
1.5
1.1
1.2
1.1
0.8
Hi
mT5
0.8
1.3
0.9
XLM
1.8
1.9
2.7
1.6
2.1
Th
mT5
0.5
0.8
0.8
0.2
XLM
1.6
1.3
1.1
0.8
Fr
mT5
0.9
0.5
0.5
1.8
XI.M
1.3
0.9
Sp
mT5
0.3
0.9
0.3
XI.M
1.1
Ge
0.6
0.3
Ja
XI.M
1.7
1.4
1
1.1
XLM
0.4
0.7
1.1
0.5
0.8
Table 13: Standard deviation (averaged over an models XLM and mT5 respectively) the results of the application of the reordering algorithm MTOP and Multilingual-Tog (above and below the horizontal line respectively). For the mT5 model, we report results only for the MTOP dataset, due to time constraints. Columns correspond evaluations settings; rows correspond valuation-datass languages and model types. Values are exact-match scores. Abbreviations: XLM XLMMRR,ERTa; RC19 the algorithm by Rasooli and Collins; E the ENSEMBLE setting, Language abbreviations: Hindi, Th- Thai, French, Sp - Spanish, German, Ja Japanese, It- Italian.
Base
Ours
Ours-E
Ras
Ras-E E
Target
UAS
LAS
UAS
LAS
UAS
LAS
UAS
LAS
UAS
LAS
French
0.1
0.1
0.9
1
0.2
0.2
0.2
0.3
0.2
0.2
German
0.3
0.4
0.7
0.6
0.1
0.3
0.3
0.4
0.3
0.2
Hindi
0.9
1.1
0.9
0.7
0.6
0.5
1.1
1.1
1.2
Korean
0.5
1
0.5
0.8
0.4
0.4
1.3
0.5
Persian
0.6
0.7
0.8
0.5
0.8
0.5
0.7
0.5
1.1
Spanish
0.1
0.1
0.6
0.5
0.1
0.2
0.2
0.2
0.4
0.4
Thai
1.8
1.4
0.9
0.5
0.3
0.3
1.3
1.6
1.4
3.1
Irish
2.1
1.4
1.2
0.9
1.8
1.4
0.7
0.8
0.2
0.8
Arahic
0.7
0.5
Table 14: Standard deviations (over 5 models) of the results the application the reordering algorithm to cross-lingual UD parsing. Columns correspond evaluations settings and score types; correspond evaluation-dataset languages. Abbreviations: Ras the algorithm by Rasooli and Collins; - the ENSEMBLE setting.

---

Base
Ours
OursE
RC19
RC19E
Target
Mic-FI
Mac-Fi
Mic-F1
Mac-F1
Mic-FI
Mac-F1
Mic-F1
Mac-F1
Mic-F1
Mac-F1
Korcan
5.3
6.4
3.4
3.6
5.3
5.9
5.4
5.8
8.4
9.1
Russian
4
2.5
3.1
3.1
3.5
4.8
6.5
2.2
2.8
Hindi
2.5
2.7
Telugu
4.1
3.1
1.8
2.7
Table 15: The standard deviations of the results (averaged over 5 models) of the application of the reordering algorithm Translated Tacred IndoRE (above and below the horizontal line respectively). Columns correspond evaluations settings and score types; correspond to ialuuation-datase languages. Abbreviations: Mic-F1 Micro-Fl; Mac-Fl Macro-F1; RC19 the algorithm by Rasooli and Collins; E - the ENSEMBLE setting.

---

Target
Sample Size
Base
OursE
100
Hi
100
Th
0.9
Fr
0.7
300
0.5
0.5
300
0.5
0.8
500
0.5
0.6
100
0.8
1.1
Ge
300
0.5
0.8
500
0.8
0.9
Table 16: The standard deviations of the results (aver- aged over 5 models) of the application of the reordering algorithm MTOP in the few-shot scenario. Columns correspond to evaluations settings; rows correspond to ovaluation-datte dataset languages. Values are exact-match scores. Abbreviations: E- the ENSEMBLE setting. Lan- guage abbreviations: Hi Hindi, Th - Thai, Fr - French, Sp Spanish, Ge German.


Reviews:
[{"review_id": "VpCXP2olFf", "reviewer": "Reviewer_W6Rf", "summary": "", "questions": "[Question A] Lines 75-80, I don't understand the point made here. Isn't \"use pre-nominal adjectival modification\" the same as \"high probability for adjectival modifiers to precede the headword\"? What difference did the authors want to make here?\n\n[Question B] Line 280 states \"assuming that the target language does not have determiners\", but it is unclear how this information has been used in the given example (has it?). How is the absence of determiners accounted for in the method? (Side remark related to [Re-A] above: Aufrant et al. also considered the case of absent determiners)\n\n[Question C] Line 300 mentions an approach to estimate POCs without a treebank, but it is not clear where in the paper this is experimented with. For which languages has this method been used, and in which Table are the corresponding results?\n\n[Question D] In the §4.2.1 experiment, what has been done exactly for the Thai data? Because line 390 states \"never use the same dataset\", but both line 381 and Table 6 mention Thai-PUD. Which one is correct?\n\n[Question E] Line 447 mentions the use of TAC English for train and Trans-TAC for test, but Trans-TAC is already a translation of TAC English, so there may be leaking. Is it the case that the Trans-TAC data used is solely translated from the **test** split of TAC English? Can you clarify and justify why this is sound evaluation?\n\n[Question F] For the few-shot scenario (line 518 and Table 4), is the experiment conducted with the models based on XLM or mT5?\n\n[Question G] Line 1025, how does it help mitigating conflicts to discard the irrelevant constraints? If those constraints are not applicable to that subtree (because root labels do not match), then they do not take part in the computation of the solver, right? So why would they be the ones preventing the solver from finding an ordering?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review, for the reference, and for the suggestions for improvement. We will incorporate them.\n* **[Re-A]** We thank the reviewer for the reference, which we have indeed missed, and agree that the hard constraints imposed by the original formulation of our method can be detrimental for some target languages. We will emphasize this. However, as the reviewer mentions, the requisite smoothing is essentially recovered by combining the pre-processed and the original training data in the Ensemble setting (more precisely, it mitigates the negative effect of imposing wrong constraints relative to only using vanilla English data). \n\n    We disagree that this “significantly changes the interpretation of the results” because the variability in the target word order, of which we are aware and which we will stress more in the revised version, is not the only source of noise that the ensemble setting mitigates: there is also the issue of the conflict between the word order seen by the encoder in pre-training vs. fine-tuning. For example, the models fine tuned on the reordered English almost always obtain lower results on the reordered English validation sets than the vanilla models on the vanilla validation set. Another source is the errors in frequency estimation for rarer dependency-label pairs (notice that ensembling improves the results for almost any type of target language). \n\n\n    We also do not think that knowledge of this work would have materially influenced our methodology as the approach by Aufrant et al. (2016) , while being quite comprehensive, retains some of the common issues we wanted to address: it is local (reorderings are only allowed inside 3-word windows), it is not tree-aware (transformations are defined based on POS sequences, not parent-conditioned dependency labels), and in addition to the statistics of target-language POS tags, it relies on manually-defined rules derived from WALS. Furthermore, as mentioned in the Shortcomings section, we fully agree that this is an important future work avenue, but one that requires substantial additional effort.\n\n\n    _“Actually, the mention of \"statistical validity\" (line 1041) looks as if the authors consider that either \"nmod &lt; amod\" or \"amod &lt; nmod\" is the appropriate constraint and when measuring P=0.51 it only prevents to identify which one is, NOT that the appropriate one would indeed be \"half of each\"._” —  We meant to acknowledge the issue, without offering concrete solutions.  Furthermore,  “Half of each” is not a constraint, but rather a probabilistic matching of the target word order, and thus the way to model this is not straight-forward (in our method). Furthermore, it ignores lexical effects, information structure, and other sources of WO variability. Generally, our approach in the ensemble setting is based on mixing the prevalent target WO (which we can estimate with some confidence) with the source word order (which we know). We also experimented with setting higher thresholds for hard constraints, i.e. setting them only when P >= 0.75 or 0.8, but this did not lead to improved performance. \n\n* **[Re-A1]** We agree that we need to stress the impact of ensemble setting with targets with soft constraints, but we consider the proposed reasoning to only provide a partial explanation. If the variability in the target order were the dominant factor, ensembling would be markedly damaging for languages with strict deviant word orders, such as Korean or Irish, because it would only add noise by preserving misleading English structures. This, however, is not the case as ensembling almost always yield the best results, with only Japanese Multilingual Top demonstrating a real drop in performance (for UD Korean there is a slight drop, but for RE Korean there is a substantial increase). We agree that we need to investigate this case further and will acknowledge this in the final version.\n* **[Re-A2]** As mentioned above, we think that while discretizing the constraints might be a reason for it, it is not the only one. If it was, then there would be language-pairs for which the ensemble method would be damaging. While Aufrant et al. did point out an important issue (which we acknowledged in the Shortcomings section), they made different design choices (hand-written rules are allowed, they are using WALS, etc.), which are not easily applicable to our work. As mentioned in the Shortcomings section, we agree that this is an important future work avenue.\n* **[Re-A3]** As mentioned above, ensembling to a large extent infuses the training data with the source, i.e. English, word order bias. Therefore if that was the only reason, ensembling should somewhat lower performance gains on languages with _hard WO preferences different from English_, which it often does not. Japanese and Italian have relatively free WOs, so the reversal we are witnessing is indeed surprising. Furthermore, while we agree with the comment and will address it, we personally suspect it to be more a case of peculiarities in the multilingual-TOP dataset.\n* **[Re-B]** We acknowledge this and agree with your statement. We do feel that we were careful in mentioning the limitations of our approach, but we will go over the paper again and make sure to clarify it. \n* **[Re-B1]** We do not require a treebank of the size comparable to that of French, Spanish, or German with tens of thousands of annotated sentences. For example, the PUD treebanks we used contain only 1000 sentences. The  “impossible to extract” case that we mention contains a mere 17 examples. We will add a clearer mention of the PUD size so it is clearer, and we acknowledge that experimenting with even lower sizes of treebank is an important avenue for future work.\n*  **[Re-B2]** We mentioned the approach as a possible extension, relying on the fact that RC19 used this approach successfully with their reordering algorithm on top of annotation projection (on UD). While their setting is different, as they did not use modern-multilingual models (and thus we cannot make direct conclusions from their results), we expect to see similar gains. This is because the texts to which we can more or less reliably project UD annotations, are restricted to the New Testament, which is not lexically representative. Additionally, annotation projection introduces its own kind of noise , which is different from the one reordering algorithms introduce. We will address this in the final version. \n* **[Re-C1]** We feel these results are largely in line with our other observations and did not find anything insightful to add.\n* **[Re-C2]** Despite some differences in grammar, English and French possess many typological commonalities, which makes them both parts of the Standard Average European linguistic area (see the discussion and references in https://en.wikipedia.org/wiki/Standard_Average_European). Past work often did not see improvements for European languages such as French and often saw a decrease in performance. In our work too, French did not see gains in UD parsing, and the same can be observed for the application of RC19 to the MTOP mT5 model. So we were indeed surprised by the mentioned gains.\n* **[Re-C3]** Experimenting with more source languages is indeed impractical. The choice of English as a source language was motivated by the fact that most datasets are in English, and most multilingual models had more English training data. As some related work (e.g., Nikolaev & Pado, Word-order Typology in Multilingual BERT, SigTyp 2022) shows, replacing English with another high resource source language that has a more flexible word order and richer morphology (such as Czech or Russian) can lead to some improvements in cross-lingual transfer, but in practice those are rather minor. We touched upon it in the introduction, and did not mention it further as using English as the source language is the standard setting in all past works. We will add a comment on this and clarify the motivation further.\n* **[Re-C4]** In this study, we did not attempt to explain the unsatisfactory performance of base models but to initiate discussion on the issue. Our hypothesis that s2s models fail to project to diverging word orders seems to be well supported by Table 5, and while the exact reason for this is a fascinating research problem, we leave it for future work.\n* **[Question A]** Here we restate the constraint usually formulated in terms of POS (Adj–Noun vs. Noun–Adj) in terms of parent-aware dependency relations: amod–nsubj/obj/obl, etc. vs. the other way around. We will clarify this point.\n* **[Question B]** The absence of determiners in the target language allowed us to leave the position of determiners on the source side unrestricted. In the example, we kept them in place for presentational clarity, but their position is largely irrelevant. \n \\“Side remark related to **[Re-A]** above/: Aufrant et al. also considered the case of absent determiners” — yes, they removed them, and elements of other types, when appropriate, based on information from WALS. We do not use WALS and restrict ourselves to using subtree preserving reorderings. (We note that the definition of a determiner in UD and WALS can differ, even across corpora. For example, deictic pronouns can be treated as determiners or amod’s in languages without articles, such as Japanese, which makes blanket removal of word categories dangerous.)\n* **[Question C]** We only mentioned the approach as a possible extension, relying on the fact that RC19 used this approach successfully with their reordering algorithm. We will make this clearer. As a side note, we did experiment with this approach, and saw some success, but a comprehensive study of this setting involves retraining all of the reported models for all languages and experimental settings, which is too computationally demanding.\n* **[Question D]** You are correct, this is a mistake on our end. Thanks for catching it. Thai has a single UD treebank and this is the source of the mistake. We will clarify that or split the treebank and re-run the experiments if time permits (this will also allow us to report results on an even smaller treebank than 1000 sentences).\n* **[Question E]** We only used a subset of the TACRED training set (as mentioned in the paper), which does not include the Trans-TACRED examples. We will emphasize that. In any case, the vanilla and the reordered models are trained exactly the same way and with the same data, so the comparison is valid.\n* **[Question F]** mT5. Thank you for noticing this omission\n* **[Question G]** We had a choice between solving for a universal ordering of each subtree type (by label) once, assigning positions to all possible children, and repeating the analysis on a case-to-case basis. The second option boils down to not including irrelevant constraints on a sub-tree basis, which is essentially a mitigation because, e.g., a CFG, even a probabilistic one, would assign a recommended total ordering to all subtrees, thus increasing the number of subtree types that have ordering conflicts. For example, in a specific language, we might have a constraint of amod->nmod, for a subtree of type “subj”, which contradicts another constraint under the same subtree. If in the specific subtree of label “subj” we are reordering, amod does not appear, we will remove this constraint. This way we are removing irrelevant constraints that might contradict one another.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "[Ac-A] The proposed method is thought-enriching and opens new research avenues.\n\n[Ac-B] The authors experiment with a large variety of languages (typologically diverse), tasks, settings (zero-shot, few-shot, different architectures, etc.), which yields more comprehensive and generalizable results.\n\n[Ac-C] Much appreciated the initiative to include a longer discussion on the method's shortcomings in the annex (beyond the \"Limitations\" paragraph).\n\n[Ac-D] Annexes provide comprehensive experimental details and results, including standard deviations, which is very good methodology.", "reasons_to_reject": "**[Re-A]** The idea underlying this contribution is quite interesting (using a SMT solver to improve upon prior similar work), but it also seems that the authors missed an important related work (Aufrant et al. 2016, see Missing references), and that if accounting for it they would possibly have made some choices differently, and interpreted some of their results differently.\n\nIn particular, Equation (2) shows that they discretize the statistics measured empirically (turning a slight preference into a deterministic hard constraint). Why? Aufrant et al. (2016) have precisely discussed (and implemented accordingly) how it is beneficial to have \"*smooth transformations (with mean preference rate objectives and error margins)*\", in other words to avoid deterministic word order, because it is a linguistic fact that for some languages the word order preference is not necessarily deterministic for a given label pair. And they also show how deterministic reordering can be detrimental in practice, by losing useful (balanced) information in the source treebank.\n\nImpact:\n- [Re-A1] The argument made on line 331 (that ENSEMBLE works better because statistical estimation creates imperfection) appears dubious in light of that prior work. On the contrary: it is because the proposed method overlooks the statistical nature of word order preference, that ENSEMBLE works better. Indeed, when source has mostly prenominal adjectives and the target has a very slight preference for postnominal, then STANDARD contains 100% postnominal, whereas ENSEMBLE contains half prenominal / half postnominal (= much closer to the target)… which is exactly what Aufrant et al. advocated for. This sheds a completely new light on ENSEMBLE, and significantly changes the interpretation of the results.\n- [Re-A2] Same for the case of performance decrease in case of close languages: a simple interpretation is that close languages have already similar word order ratios for a given label pair, and the discretized constraints move them further. This is exactly the \"French to Italian\" case analyzed by Aufrant et al. So these observations may just be an artefact of the choice to discretize the constraints, not an evidence of the applicability of reordering in general depending on the language pair.\n- [Re-A3] And for the remark line 506 on MTOP results opposite to Multilingual-TOP ones: since language sets are different, that again may just be an artefact of the fact that target languages on one side or the other have more deterministic or more balanced word order preferences.\n\nThe authors acknowledge on line 1039 that there may be an issue with this discretization, but the analysis does not go as far as observing how that impacts (and possibly invalidates) some of their own conclusions and analyses in the paper. Actually, the mention of \"statistical validity\" (line 1041) looks as if the authors consider that either \"nmod < amod\" or \"amod < nmod\" is the appropriate constraint and when measuring P=0.51 it only prevents to identify which one is, NOT that the appropriate one would indeed be \"half of each\".\n\n**[Re-B]** There is also an issue with the realism of the targeted scenarios. Often when working on low-resourced scenarios, there are irremediable issues that prevent from being fully realistic (e.g. evaluation can only be done when sufficient data exists, so not for actually low-resourced languages). So it is fully OK to assume that such work is done in best-effort mode. But this does not mean discarding the corresponding issues as being irrelevant, but rather to acknowledge them and acknowledge they are not solvable. Plus, care must be taken that the proposed methods would actually be meaningful in a real low-resourced scenario.\n\nMore precisely:\n- [Re-B1] Regarding the pre-requisite for an UD treebank to estimate the POCs, line 359 accurately observes that UD is available for many languages anyway. However, treebank availability does not mean availability of a treebank with size comparable (hence the same reliability of POC estimation) to the large treebanks used in Table 5 for French, Spanish, or German. So this presumably overestimates a lot the quality of the POCs estimated in actually low-resourced settings. In particular, the treatment made of footnote 6 is questionable: precisely this is a real-world scenario of low-resourced language, so the conclusion \"impossible to extract\" raises major questions on the applicability of the method to actual low-resourced cases.\n- [Re-B2] The scenario for estimating POCs without a treebank also does not seem very convincing. If using annotation projection to produce a treebank in the target language, why only using it for estimating POCs (and then training on a treebank from a different language), rather than directly using the projected trees as training treebank? And same for the other tasks, if there is parallel data to do annotation projection, then isn't it appropriate to project the available TOP & RC annotations through that parallel corpus, instead of resorting to a convoluted approach through another corpus and through data transforms? Or has this issue already been thought of, and were there specific reasons to prefer that approach?\n\n**[Re-C]** Finally there is a number of places where the analysis is too shallow, or sometimes completely missing:\n- [Re-C1] For the relation-classification task, lines 524-534 only describe results, without any comment on what can be concluded from those results.\n- [Re-C2] The remark line 502 on English vs French overlooks that English and French are indeed related, but have nevertheless a number of important typological differences (pre/postnominal adjectives for instance), so it is unsurprising to observe gains. What is the concept of \"typologically distant\" used here?\n- [Re-C3] Only English is used as source language, which is clearly not neutral when considering typology (English being rather atypical among the 7000 languages, regarding typology). Surely these experiments are computationally expensive, so experimenting with many sources was probably not doable. But the motivations and impact of that choice would have deserved at least a comment.\n- [Re-C4] In Table 5, a number of s2s results are so low that it is hard to consider that the model does anything (or at least, it does not outperform a crude heuristic such as \"dependency to the left neighbour, with the majority-class as relation label\"). This raises many questions on how to interpret the score increase, so this would have at least deserved a comment (more than just calling it \"subpar\").", "questions_to_authors": "[Question A] Lines 75-80, I don't understand the point made here. Isn't \"use pre-nominal adjectival modification\" the same as \"high probability for adjectival modifiers to precede the headword\"? What difference did the authors want to make here?\n\n[Question B] Line 280 states \"assuming that the target language does not have determiners\", but it is unclear how this information has been used in the given example (has it?). How is the absence of determiners accounted for in the method? (Side remark related to [Re-A] above: Aufrant et al. also considered the case of absent determiners)\n\n[Question C] Line 300 mentions an approach to estimate POCs without a treebank, but it is not clear where in the paper this is experimented with. For which languages has this method been used, and in which Table are the corresponding results?\n\n[Question D] In the §4.2.1 experiment, what has been done exactly for the Thai data? Because line 390 states \"never use the same dataset\", but both line 381 and Table 6 mention Thai-PUD. Which one is correct?\n\n[Question E] Line 447 mentions the use of TAC English for train and Trans-TAC for test, but Trans-TAC is already a translation of TAC English, so there may be leaking. Is it the case that the Trans-TAC data used is solely translated from the **test** split of TAC English? Can you clarify and justify why this is sound evaluation?\n\n[Question F] For the few-shot scenario (line 518 and Table 4), is the experiment conducted with the models based on XLM or mT5?\n\n[Question G] Line 1025, how does it help mitigating conflicts to discard the irrelevant constraints? If those constraints are not applicable to that subtree (because root labels do not match), then they do not take part in the computation of the solver, right? So why would they be the ones preventing the solver from finding an ordering?", "paper_topic_and_main_contributions": "This paper presents a new method to improve cross-lingual transfer based on source-side word reordering (operated at the level of dependency subtrees). The authors do so by estimating word order preferences from data, then casting them as constraints to an SMT solver to identify the optimal reordering. The method is experimented on 3 tasks and on typologically diverse languages. Results are ambivalent (usually beneficial but detrimental for some language pairs) but the authors propose a mitigation measure (concatenating original and reordered data) that is successful. Experiments are extended to the few-shot scenario and to comparison among architectures, which yields new insights on the generalizability of the method as well as on properties of existing algorithms.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Missing a key reference: Aufrant et al. (2016), Zero-resource Dependency Parsing: Boosting Delexicalized Cross-lingual Transfer with Linguistic Knowledge\n\nFor section §2 it is appropriate to mention, because a) it has anteriority over Liu et al. (2020a) on the POS-based language model idea, and b) it also includes an instance of the \"using WALS\" approach but with source-side (training-time) reordering, as this work does (whereas Meng et al. 2019 is at inference time).\n\nBut perhaps more importantly, it already provides answers to some of the issues encountered in this work: how reordering can be detrimental, why avoiding deterministic reorderings, etc. See [Re-A] above.", "Typos_Grammar_Style_and_Presentation_Improvements": "- Line 56, it is not discussed what \"old\" and \"new\" mean here, so it is hard to understand what phenomenon is referred to. This seems very marginal compared to the topic of the paper, so maybe not worth a long discussion, but is it possible to rephrase in a more explicit manner?\n- Typo line 107: suggest --> suggests\n- Typo lines 167, 221, 251, 259, 267: constrains --> constraints\n- Line 261 (and possibly elsewhere), sub-tree --> subtree (to match the spelling in the rest of the paper)\n- Typo line 265: on of --> on\n- Typo line 256: the the --> the\n- Typo line 490: the the --> the\n- Typo line 531: then --> than\n- Table 2 formatting: it would be useful to display differently the results that are an increase and a decrease (to see more easily which languages are concerned). Use for instance bold, underline, decorate with an asterisk after… at least for Table 2 since this is explicitly discussed at line 474, but if possible for other Tables as well.\n- Tables 4 & 5: use the same precision for all numbers (e.g. 67.0 instead of 67 for Fr-300-Base)\n- Typo line 587: tends --> trends\n- Typo line 594: suggest --> suggests\n- Typo line 604: able boost --> able to boost\n- Typo line 620: results --> result\n- Typo line 628: suggests --> suggest\n- Typo line 633: remove \"more\"\n- Line 655, \"multiply-parallel\" is awkward, usually this is called \"multi-parallel\""}}, {"review_id": "iNf3nsHt5m", "reviewer": "Reviewer_YA2U", "summary": "", "questions": "[1] l. 477 What about Irish? There seems to be an advantage of your method there.\n\n[2] ll. 478-479 \"No noticeable effect is observed for structurally closer languages.\" However there is once again a decrease in performance in Arabic, and a terrible drop in Persian!\n\n[3] ll. 512-513 What do you mean by significant? Did you test it with some statistical test? One option could be the McNemar test on paired nominal data. It is true that the difference is numerically small, but statistical significance (p-values) depend on the variance of the samples. Very small differences can be statistically significant with large sample sizes. \n\n[4] ll. 528-529 How do you explain the drop in performance in Hindi and Telugu? It's in contrast with your previous observation, where more typologically distant languages benefited the most from your approach.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review, for the reference and for the suggestions for improvement and we will incorporate them.\n\n\n\n* **[R1]** We regard this as an important avenue for future work. Furthermore, while not mentioned in the paper, the estimates can be used in order to compute the word-order distance between languages in a rigorous corpus-based way, while WALS only provides a single categorical label for “dominant word order”.\n* **[R2]** Again, we view this as an important avenue for future work. Time permitting, we will provide an appendix with an overview of learned constraints for some target language. \n* **[Q1]** If you are referring to the fact that we wrote “No noticeable effect is observed for structurally closer languages” while Irish did gain from our ENSEMBLE method - this is because we consider Irish a distant language. Typologically, Irish is not much closer to English than Arabic (in addition to being a VSO language and lacking most of Standard Average European features, it has neither infinitives nor the verb “to have”, its verbal system is heavily reliant on the use of nominalised verb forms, it has a complex system of morphophonological alternations known as “mutations”, it employs conjugated prepositions to index prenominal objects, etc.).\n* **[Q2]** These lines refer to the ENSEMBLE setting (see line 472), in which Persian has a very slight increase.  In line 476 we state “with the exception of Arabic”. In line 478-481 we refer to the STANDARD and state the decrease in performances, but we should indeed be clearer there and we will fix that.\n* **[Q3]** “Significant” here refers to the strength of the effect, not its statistical significance; we will rephrase the passage. Generally, we tried to make the results more robust by running the experiments and examining and reporting standard deviations, but with only 5 data points proper statistical significance testing is not really possible (and running more than 5 was prohibitively costly).\n* **[Q4]** They still benefit from the ENSEMBLE approach. As to why in the STANDARD approach there was a drop in performance, this can be because of the idiosyncrasies of the RE dataset. It could be that this dataset is not as difficult, or that the annotations do not require the model to leverage syntactic information to the same extent, such that the added noise of the method was larger than the potential benefit. For example, in Table 1, it can be seen that Hindi did benefit from the approach in the STANDARD setting. Validating these hypotheses was beyond scope of this work.\n* **[Presentation Improvements 2]** This is due to the technical need to position the tables to fit 8 pages. With the extra page we will change the table numbering.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "[1] The proposed method is simple, effective, and particularly suited to post-hoc interpretability.\n\n[2] The paper considers both few-shot and zero-shot experiments.\n\n[3] The \"Related work\" section is very informative and clear. Personally, I do not work on machine translation, and I really enjoyed quality of the overview. \n\n[4] Interestingly, the proposed method is associated with a higher performance gain in the seq2seq setting, highlighting a difficulty of encoder-decoder architectures in dealing with variable word order patterns.", "reasons_to_reject": "[1] I think that the study could greatly benefit from an objective quantification of the typological / word-order distance between the languages considered. One of the point I found most interesting about this work is that there seems to be an increase in performance for languages that are distant from English (e.g., Japanese, Hindi), and a decrease in performance (sometimes) for languages that are close to English (e.g., Spanish, German, especially with mT5). It would be great to assess this trend more formally, with a metric of typological similarity (e.g., using WALS). \n\n[2] The authors state that their approach is suited for interpretability; however, the way POCs can be interpreted is never truly addressed in the body of the paper. It would have been a very nice follow-up analysis/appendix.", "questions_to_authors": "[1] l. 477 What about Irish? There seems to be an advantage of your method there.\n\n[2] ll. 478-479 \"No noticeable effect is observed for structurally closer languages.\" However there is once again a decrease in performance in Arabic, and a terrible drop in Persian!\n\n[3] ll. 512-513 What do you mean by significant? Did you test it with some statistical test? One option could be the McNemar test on paired nominal data. It is true that the difference is numerically small, but statistical significance (p-values) depend on the variance of the samples. Very small differences can be statistically significant with large sample sizes. \n\n[4] ll. 528-529 How do you explain the drop in performance in Hindi and Telugu? It's in contrast with your previous observation, where more typologically distant languages benefited the most from your approach.", "paper_topic_and_main_contributions": "The paper presents an interesting approach to reordering for multilingual NLP, which instead of reordering the words of sentence, it reorders the sub-trees of its universal dependencies parse tree. The proposed method is shown to outperform previous approaches both in zero- and few-shots settings.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Joshi et al., 2020. The State and Fate of Linguistic Diversity and Inclusion in the NLP World\n- Could be useful for the case you're making in the introduction", "Typos_Grammar_Style_and_Presentation_Improvements": "[1] Highlighting the best results per condition in bold in the tables would increase readability a lot!\n\n[2] Why is table 2 discussed before table 1?"}, "extra_text": {}}, {"review_id": "bPH9oGRRXy", "reviewer": "Reviewer_gucq", "summary": "", "questions": "n/a", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review and for the suggestions for improvement and we will incorporate them.\n\n\n\n* **[R1]** What we meant is that while Rasooli and Collins (2019) leveraged the rich syntactic information of Universal Dependencies, they only used a very small part of the information UD provides (although they did put it to great use), in contrast to our work. The “richness” remark refers to UD. We will clarify this.\n\n    About the need to compare with more methods - this was not done for a couple of reasons: first, other methods are not easily reproducible. Specifically, comparing with Liu et al. (2020a) would have required training POS-based language models, and then reproducing their experiments to make sure we managed to train comparable models. This is in itself a big task. We did not find any released code or models available online. \n\n\n    We also note that we needed to create our own implementation of Rasooli and Collins (2019) method to test it.\n\n\n    We plan to release our method (and our implementation of Rasooli and Collins (2019)), to facilitate future replications.\n\n\n    Secondly, the amount of resources required for this is too high - a proper comparison would have required training of hundreds of additional models, and our study being very computationally intensive as it is, we needed to choose our experiments very carefully.\n\n\n    Lastly, to our knowledge Rasooli and Collins (2019) is the most comparable method to ours (being pre-processing-based, interpretable, and based on UD), and it produces strong results. \n\n* **[R2]** For Japanese and Italian in the Multilingual-TOP dataset, we suspect it is due to the peculiarities of the dataset, as this is the only dataset where we see this effect. Regarding Irish and Arabic in Table 2, this happens only for Rasooli and Collins’s (2019) method. In general, we agree that this (and understanding exactly why the ENSEMBLE is better) is an interesting avenue to check, but due to the amount of work already done, it was outside of scope for this work.\n* [**Presentation Improvements 1[** We will add average gains per language class to Table 2, as suggested.\n* [**Presentation Improvements 2]** “Additionally, it may be beneficial to employ the average reordering time per sentence as a quantitative measure of structural distance” — it is hard to measure this properly as for each subtree it happens that either (i) we get an output from the SMT solver (using which we can apply some kind of index-permutation distance to quantify the local WO distance) or (ii) SMT fails to solve the subtree, which makes the estimation of the structural distance impossible. As we do not know the distribution of subtrees that fail to solve, using only solved subtrees to measure the distance can be arbitrarily biased. It is possible, however, to measure WO differences between languages using all learned pairwise constraints. We regard this as an avenue for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tExtensive experiments on different tasks (UD paring, relation classification, and semantic parsing) and settings (zero-shot and few-shot), which show the necessity of reordering. \n2.\tDetailed analyses on different architectures. \n", "reasons_to_reject": "1.\tInsufficient comparison with related works.\nFirst, the differences with the most related work (Rasooli and Collins, 2019) should be clearer. Their approach is reported to leverage rich syntactic information, as stated on line 190, which is controversial with the expression “superficial statistics” in the line 192. Second, it is insufficient to solely compare the proposed methodology with that of Rasooli and Collins (2019). It would be prudent to also consider similar work, such as that of Liu et al. (2020a).\n\n2.\tThe effectiveness of these two settings (STANDARD and ENSEMBLE) varies depending on the tasks and languages. As shown in table 5, ENSEMBLE outperforms STANDARD for the majority of languages, with the notable exceptions of Japanese and Italian. A similar phenomenon is observed for Thai and Irish, as indicated in Table 2, warranting further investigation.\n", "questions_to_authors": "n/a", "paper_topic_and_main_contributions": "This paper presents a source-sentence reordering method to narrow the gap between typologically-distant languages in cross-lingual transfer. They learn reordering rules based on Universal Dependencies and apply them at all levels of the syntactic tree. Extensive experiments show the effectiveness of the proposed approach in enhancing cross-lingual transfer, particularly in the context of low-resource languages. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "n/a"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1.\tLine 265: “based on of the UD” -> “based on the UD”\n2.\tThe average scores for both typologically-distant and structurally-similar languages should be included in Table 2 for comparison. Additionally, it may be beneficial to employ the average reordering time per sentence as a quantitative measure of structural distance.\n"}}]

Author rebuttal:

Meta-review: The authors propose an interesting and promising approach, but the reviewers have identified several points that need to be improved before this work can be published: quantification of similarity between languages, analyses that are not sufficiently detailed, etc.
Decision: Accept-Findings