Model Selection for Cross-Lingual Transfer using a Learned Scoring Function
Authors: Yang Chen
Alan Ritter
Venue:  (2021)

Abstract: Transformers that are pre-trained on multilingual text corpora, such as, mBERT and XLM-RoBERTa, have achieved impressive cross-lingual transfer learning results.  In the zero-shot cross-lingual transfer setting, only English training data is assumed, and the fine-tuned model is evaluated on another target language.  No target-language validation data is assumed in this setting, however substantial variance has been observed in target language performance between different fine-tuning runs.  Prior work has relied on English validation/development data to select among models that are fine-tuned with different learning rates, number of steps and other hyperparameters, often resulting in suboptimal choices.  In this paper, we show that it is possible to select consistently better models when small amounts of annotated data are available in an auxiliary pivot language.
We propose a machine learning approach to model selection that uses the fine-tuned model's own internal representations to predict its cross-lingual capabilities.  In extensive experiments we find that our approach consistently selects better models than English validation data across five languages and five well-studied NLP tasks, achieving results that are comparable to small amounts of target language development data.

Under review as a conference paper at ICLR 2021
MODEL SELECTION FOR CROSS- TRANSFER USING A LEARNED SCORING FUNCTION
Anonymous authors Paper under double-blind review
ABSTRACT
Transformers that are pre-trained on multilingual text corpora, such as, mBERT and XLMMRoBERTa, have achieved impressive cross-lingual transfer learning re- sults. In the zero-shot cross-lingual transfer setting, only English training data is assumed, and the fine-tuned model is evaluated on another target language. No target-language validation data is assumed in this setting, however substan- tial variance has been observed target language performance between different fine-tuning runs. Prior work has relied on English validation/ddveloppmen data to select among models that are fine-tuned with different learning rates, num- ber of steps and other hyperparameters, often resulting in suboptimal choices. In this paper, we show that it is possible to select consistently better models when small amounts of annotated data available in auxiliary pivot language. We propose a machine learning approach to model selection that uses the fine-tuned model's own internal representations predict its cross-lingual capabilities. In extensive experiments we find that our approach consistently selects better mod- els than English validation data across five languages and five well-studied NLP tasks, achieving results that are comparable to small amounts of target language development data
INTRODUCTION
Pre-trained Transformers (Vaswani et al. 2017: Devlin et al. 2019) have lead to state-of-the-art results on a wide range of NLP tasks, for example, named entity recognition, relation extraction and question answering, often approaching human inter-rater agreement (Joshi et al., 2020a). These models have also been demonstrated learn effective cross-lingual representations, even without access to parallel text or bilingual lexicons (Wu & Dredze, 2019 Pires et al. 2019). Multilingual pre-trained Transformers, such as mBERT and XoBRRRoERTT (Conneau et al. 2019), support surprisingly effective zero-shot cross- cross-lingus lingual transfer, where training and development data are only assumed a high resource source language (e.g. English), and performance is evaluated on another target language.
Because no target language annotations are assumed in this setting, source language data is typically used select among models that fine-tuned with different hyperparameters and random seeds. However, recent work has shown that English dev accuracy does not always correlate well with target language performance (Keung et al., 2020). In this paper, we propose alternative strategy for model selection zero-shot setting. Our approach, dubbed Learned Model Selection (LMS), learns function that scores the compatibility between fine-tuned multilingual transformer, and target language. The compatibility score is calculated based on features of the multilingual model's learned representations and the target language. model's features are based on its own internal representations: this is done by aggregating representations over an unlabeled target language text corpus. These model-specific features capture information about how the cross-lingual representa- tions transfer to the target language after fine-tuning on source language data. In addition to model- specific representations. we also make use Iearned language embeddings from the lang2vec package (Malaviya al. 2017 which have been shown to encode typological information, for example, whether a language has prepositions or postpositions. To measure compatibility between
We will make our code and data available on publication. httpps://// thub. com/ antonn sa/1 1ang2vec

---

Under review as a conference paper at ICLR 2021
MBEERT
En-Dev F1
MBERT
[CLS]
Scores
0.5
Bootng
English
0.5
Unlabeled
Target
Development
Language
Data
Data
0.1
0.6
(a) English Development Selection
(b) Learned Model Selection (LMS)
Figure 1: An illustration of our approach to select the best model for zero-shot cross-lingual transfer. (a) Prior works select the best model using source language development data. (b) LMS: A learned function scores fine-tuned models based on their hidden layer representations when encoding unla- beled target language data.
a multilingual model's fine-tuned representations and target language, the model- and language- specific representations are combined in bilinear layer. Parameters the scoring function are op- timized to minimize a pairwise ranking loss on a set of held-out models, where the gold ranking is calculated using standard performance metrics, such as accuracy or Fj, on set of pivot languages (not including the target language). Our method assumes training data in English, and small amounts of annotated data in one or more pivot languages (not the target language). This corresponds to a scenario where new multilingual NLP task needs to be quickly applied to new language. LMS does not rely on any annotated data in the target language, yet it is effective in learning to predict whether fine-tuned multilingual representations
good match.
In experiments on five well-studied NLP tasks (part of speech tagging. named entity recognition, question answering, relation extraction and event argument role labeling). find LMS consistently selects models with better target-language performance than those chosen using English dev data. Appendix A.5 demonstrates that framework supports multi-task learning, which can be helpful in settings where some langet-l annotations are available. but not for the desired task. Finally, show that LMS generalizes to both mBERT and BBBRRa in Appendix A.4
2 BACKGROUND: ZERO-SHOT CROSS LINGUAL TRANSFER
The zero-shot setting considered in this paper works follows. A transformer model is first pre- trained using a standard masked language model objective. The only difference from the monolin- gual approach contextual word representations (Peters al., 2018 Devlin al. 2019 is the pre- training corpus, which contains text written in multiple languages. For example, mBERT is trained on text written in 104 languages from Wikipedia. After pre-training on multilingual corpus, the resulting transformer encodes language-independerr representations that support surprisingly effec- tive cross-ling cross-lingual transfer. simply by fine-tuning the pre-trained parameters using English training data. For example, after fine-tuning mBERT using the English portion of the CoNLL Named Entity Recognition dataset, the resulting model can used to perform inference directly on Spanish text, achieving an F, score around 75, and outperforming prior work using cross-lingual word embed- dings (Xie et al. 2018 Mikolov et al. 2013). A challenge with this approach, however, that there is relatively high variance in this performance across training runs. Although the mean F, score on Spanish 75, the performance of 60 models fine-tuned with different learning rates and random seeds ranges from around 70 F1 to 78 (Figure 3). In zero-shot learning, validation/ddeellommen data assumed in the target language, motivating the need for a machine learning approach to model selection.
2

---

Under review as a conference paper at ICLR 2021
3 RANKING MODEL COMPATIBILITY WITH A TARGET LANGUAGE
Given set of multilingual English training set using different hyperparameters and random seeds, our goal is to select the model that performs the best target language, Inarget. Our approach (LMS) learns rank set of models based on two sources of information: (I) the models' own internal representations, and (2) lang2vec representations of the target language (Malaviya et al.) 2017). By integrating model's target language representations together with anng2ec embeddings, LMS predicts which model will perform best.
models, M m. m....... Myy that fine-tuned
We adopt pairwise approach to learning to rank (Burges et al. 2005; Köppel et al. 2019). The learned ranking computed using a scoring function, 8(m, () [yMBERT [m)) fnaggvvee (7)), where IMBERT (m) is feature vector for model w., which is computed from the model's ' own hidden state representations, and M1ang2ved (I) is the lang2vec representation language The model and language features are each passed through feed-forward neural network and then combined using
a bilinear layer to calculate a final score as follows:
s(m, (goRFRr(m)) grangtrac()))
= (INN(gmBEr((m)
Using the above score, represent the probability that model m, performs better than my on language
P(m; Py my) (s(m,,))) 1) ())
where ol)) is the sigmoid function. To tune the parameters of the scoring function, which include the feed-forward and bilinear layers, we minimize cross-entropy loss:
C= Σ Σ
Σ
JELi, M.EM MyEM
where
Cmi.mj.l
[1n De my] log P(m, Dy m2) + [m mi] log P(my m²)
Here [m, Py mi] is an indicator function that has the value if my outperforms m. as evaluated using labeled development data in language
The first sum in Equation Tranges over all languages where development data is available, excluding the target language. After tuning parameters minimize cross-entropy loss on these languages, the models are ranked based on their scores for the target language, and the highest scoring model, m arg max,, smm, lurge)), is selected.
4 TASKS AND DATASETS
Section 3 presented an approach selecting a model for zero-shot cross-lingual transfer. evaluate its effectiveness, a set of fine-tuned models is split into meta-trraindeev/tett sets. We then use models the meta-train set to tune parameters of the scoring function, s(-), develop input features with models in the meta-dev set (see 77 for details), and report our main results using models in the meta-test set Table
We experiment with five well-studied NLP tasks in the zero-shot transfer setting: named entity recognition (NER), part-of-speech (POS) tagging, question answering (QA), relation extraction (RE), and event-argument role labeling (ARL). Labeled training data for each task is assumed in English and trained models are evaluated on the target language. The task/language combinations covered by experiments summarized in Table
The following subsections describe the tasks and datasets used in our experiments; more details are presented in Appendix A.1

---

Under review as a conference paper at ICLR 2021
ar
bg
da
de
fa
hu nl pt
sk
sl
zh
NER
POS
V
QA
RE
ARL
Table 1: Seventeen target languages and five tasks used in our experiments. English is used as the source language. Arabic, bg: Bulgarian, da: Danish, de: German, es: Spanish, fa: Persian, hi: Hindi, hu: Hungarian, it: Italian, nl: Dutch, pt: Portuguese, ro: Romanian, sk: Slovak, sl: Slovene, Swedish, vi: Vietnamese, zh: Chinese.
4.1 NAMED ENTITY RECOGNITION
We use data from the CoNLL 2002 and 2003 (Sang 2002 Sang & Meulder 2003) shared tasks (four target languages: en, de, es, and nl), in addition Chinese NER dataset (Levow 2006), following Wu & Dredze (2019). The data is tagged using the BIO tagging scheme with four types of named entities. We formulate this as tagging task and add linear classification layer with a softmax function to obtain word-level predictions. Because the labels are at the word level, but mBERT operates on the subword level, representation of the first subword is used make word- level predictions, following (Devlin et al. 2019). The CoNLL evaluation script is used to compute Fl scores.
4.2 PART-OF-SPEECH TAGGING
subset of Universal Dependency (UD) Treebanks (v1.4) (Nivre et al. 2016) (fifteen target lan- guages: ar, bg, da, de, es, få, hu, it, nl, pt, ro, sk, sl, sv, zh) is used, following the setup Kim et al.) (017)). There are eighteen POS tags in total across all languages. We formulate this as a tagging task, following the same approach as described for named entity recognition, and report the accuracy of the predicted POS tags (Acc).
4.3 QUESTION ANSWERING
The Multilingual Question Answering (MLQA) dataset (Lewis al. 2020) is used to evaluate model selection for cross-lingual transfer in the context of question answering. We use the MLQA dataset, consisting of six target languages: ar, de, es, hi, vi, zh. Following the setup of [ewiis et al. (2020), models trained using English data from SQuAD v1.1 (Rajpurkar al. 2016). and the MLQA English development set is used for the En-Dev baseline. We use the standard BERT approach for question answering (Devlin et al., 2019), where (question, passage) pair is encoded using special tokens as follows: [BOS] Question [SEP] Passage [SEP]. ]. The classification layer predicts both the start and end index of the answer span from the text passage. The formal evaluation script of MLQA is adopted to compute F1 scores.
4.4 RELATION EXTRACTION
Given a pair of entity mentions, RE aims to identify a relation between them and classify its type. We use the ACE 2005 corpus (Walker et al.) (006)). which contains two target languages, Arabic and Chinese, using the same preprocessed dataset as (Subburathinam et al. (201)). ACE contains eighteen relation types and an additional label indicating no relation (19-way classification). We adopt the best performing model from [Soares et (2019), [ENTITY MARKERS ENTITY START), which adds special tokens [E1start], [E1and] and [E2gtart]: [E2ana] surrounding entity mention pairs in sentence. The modified sentence is then fed into mBERT and representations of the starting markers [Elstart], [E2gtart] are concatenated as inputs to a linear classification layer. We follow evaluation protocols established in prior work, using gold entities as inputs to control for differences in performance due to named entity recognition errors (Subburathinam et al.) 2019). A predicted relation mention considered correct if its type and the heads of both entities match the gold data.
4

---

Under review as a conference paper at ICLR 2021
4.4.1 EVENT-ARGUMENT ROLE LABELING
Event-Argument Role Labeling (ARL) aims to identify event triggers and their arguments in texts. For example, in the sentence Facebook aquired Instagram, an event trigger is aquired, whose ar- guments are Facebook and Instagram, with roles BUYER and SELLER. Again we draw on the ACE 2005 corpus (Walker et al. 2006) (two target languages: ar. zh), using the same preprocessed dataset as (Subburathinam et al. (2019). There are thirty-five role labels in ACE. Following a sim- ilar approach as was described for relation extraction, we add special markers [Tatart], [Tand] and [Estart], [Eena] around the trigger and candidate entity mentions. The modified sentence is fed into mBERT, and the marker representations are concatenated as input to classification layer that pre- dicts the entity's 's role with respect to the trigger word.
5 EXPERIMENTAL DESIGN
For a downstream task with n languages L :4,,...,4, our goal is to select the model that performs best on target language, lmrget L. We assume the available resources are English training and development data, in addition to a small development set in one or more pivot languages (that do include the target language). First, a set of mBERT models. M, are fine-tuned on an English training set using different hyperparameters and random seeds and shuffled into meta- meta-train/dew train/dev/testtt sets. We then evaluate each model, m,, on the pivot languages' dev sets calculate gold ranking, Pt. that is used in the cross-entropy loss (Equation [ Model-specific features are then extracted from the fine-tuned mBERTs, by feeding unlabeled pivot language text (dev set examples) as input.
Development and Evaluation mBERT models in the meta-dev set are used to experiment with different model and language features. Target language dev data was used to experiment with a handful of model and language features, as described in 7 Evaluation is performed using the meta- test set. For each target language, we rank models using the learned scoring function, select the highest scoring model, and report results in Table 2
5.1 BASELINES
En-Dev is our main baseline following standard practice for model selection in the zero-shot set- ting (Wu Dredze 2019 Pires et al. 2019). Model selection with the pivot language dev set included as another baseline, where the pivot language is selected based on the highest similar- ity with the target language, as measured using cosine similarity between Lang2ved embeddings among candidates in {ar, de, es, nl, 2h). Also, we compare our results with models selected using 100 sentences annotated in the target language to understand how our approach compares to the more costly alternative of annotating small amount of target language development data. Finally, an All-Target oracle that picks the best model based on target language development data is adopted. These baselines and oracles are summarized below:
En-Dev (baseline): choose an mBERT with the English dev set.
Pevot-Dev (haseline): choose mBERT with the pivot language dev set.
100- Target (oracle): choose mBERT with 100 target language dev set instances.
All-Target (oracle): choose an mBERT with the full target language dev set.
5.2 HYPERPARAMETERS AND OTHER SETTINGS
To train the scoring function, s--), we use Adam Kingma & Ba 2015 and select the batch size
among [16, 32, 64, 128), learning rate among {1x x 10 5 10 11 10 5 1x1xx 10 "),
and train for }3} epochs. The scoring function, s(-), contains a 2-layer FENN with 1024 hidden units and ReL.U activation (Glorot et al. 201 T). The base cased mBERT has 179M parameters and a vocabulary of around 120k wordpieces. Both the pre-trained transformer layers and task-specific layers are fine-tuned using Adam, with 31 = 0.9, B2 0.999, and an L2 weight decay 0.01.
wawee-on--annngge--aee language-out cross-validation over {ar, de, es, zh), To evaluate performance on target languages in {bg, da, fa, hi, hu, it. pt, ro, sk, sl, su, vi} for POS and QA, single LM was trained using pivot languages in far, ar, de, es, nl, zh).
'Evaluation was performed using leave-one one
5

---

Under review as a conference paper at ICLR 2021
Model candidates fine-tuned with varying learning rates and number of epochs with the following settings: learning rate E x3x x 10 7 10 10-" number epochs E (3, 4, 5, 6); batch size {32}; random seeds {0, 239}. 240 mBERT models with different random seeds are fine- tuned with 12 different hyperparameter settings (20 random seeds for each set of hyperparameters), and then split into meta-train/dev/test sets (120/60/60). All models are trained on an RTX 2080 Ti.
EVALUATION
Below we report model selection results on mBERTss in the meta-test set for each of the five tasks.
Task
Lang
Ref
En-Dev
Pivot-Dev
LMS
100-Target
All-Target
# All-Target
de
69.56
69.89
70.72 (nl)
69.89
66.74
72.07
2867
74.96
74.61
73.10 (nl)
75.74
75.74
75.73
1915
NER (F,)
nl 77.57
78.74
79.27 (de)
78.85
78.67
80.26
2895
zh 51.90
54.90
52.99 (de)
55.05
55.36
56.88
4499
ar
49.66
50.28 (de)
51.61
50.56
52.70
786
de
89.8
89.27
88.75 (nl)
89.77
89.45
89.98
799
es 85.2
84.84
85.26 (nl)
85.56
84.82
85.11
1552
nl 75.9
75.68
75.93 (de)
75.87
75.54
76.04
zh
66.94
66.90 (de)
68.00
67.25
68.84
500
bg 87.4
87.13
87.05 (es)
87.92
87.95
87.95
1115
da 88.3
88.64
88.87 (nl)
88.94
88.64
89.20
POS (Acc)
fa 72.8
71.64
71.63 (CS)
73.63
73.63
73.75
599
hu 83.2
82.55
82.05 (de)
83.26
83.26
83.11
179
it 84.7
84.47
84.89
85.23
85.37
85.84
489
pt 82.1
81.80
81.88 (es)
82.16
81.82
82.18
271
ro 84.7
83.83
84.19
84.71
84.43
85.37
1191
sk 83.6
83.73
83.65 (es)
84.23
83.65
84.80
1060
sl 84.2
84.49
83.48 (cS)
85.16
83.82
85.53
735
Sv 91.3
91.39
91.83 (nl)
91.66
91.33
91.76
504
ar 45.7
47.72
49.36 (de)
49.26
49.36
49.36
517
de
57.9
55.27
55.83 (ar)
55.86
57.12
55.83
512
QA (F1)
es
64.3
64.92
64.74 (ar)
64.95
64.51
65,08
500
zh 57.5
58.03
58.09 (de)
58.11
58.14
58.38
hi 43.8
39.09
42.05 (cs)
42.43
38.85
42,89
507
vi 57.1
57.36
56.88 (ar)
58.21
59.12
58.12
511
ar 39.43
36.10
35.35 (zh)
39.54
34.68
41.92
4482
RE (F1)
zh 32.74
67.68
67.43 (ar)
70.75
68.20
69.13
7096
16.48
44.11
48.08 (zh)
47.08
44.11
47.15
1221
ARL (F.)
zh 23.49
60.96
61.26 (ar)
62.05
62.52
63.81
2226
Table 2: Model scores selected based on LMS for NER, POS, QA, RE, and ARL. En-Dev / Pivot-Dev / 100-Target / All-Target: model selection based the highest Fl of English dev set / Pivot language dev set (pivot language in bracket) / 100 target language dev set examples tar- get language dev set. LMS: model selection based the highest scores for the target language: arg max,, (m, lurge); "# All-Target" is the number of labeled target-language sentences used for model selection in the All-Target oracle.
NER As illustrated Table our method selects models with higher Fl score than En-Dev except in the case of German (de). Besides, it outperforms model selection using small amounts of target-language annotations (100-Target) on Dutch (nl) and selects a model that performs as well on Spanish (es). On average, LMS achieves 0.86 point increase in Fl score relative to Pivot-Dev. We use (Wu& & Dredze 2019) as references for zero-shot cross-lingual transfer with MBERT.
POS Table 2 presents POS accuracies the test set, using various approaches to model selection for the fifiteen target languages. LMS outperforms En-Dev and Pivot-Dev except in the case of Swedish (sv) and Dutch (nl). Interestingly, model selection for Italian with Spanish dev set does not outperform LMS. We use (Wu & Dredze 2019) as references for zero-shot cross- lingual transfer with MBERT.

---

Under review as a conference paper at ICLR 2021
QA Our method selects model with higher F1 across all languages compared with En Dev, al- though we find that Pivot-Dev performs slightly better on Arabic (ar). We use (Lewis et al. 2020 references for zero-shot cross- Pross-lingual transfer with mBERT.
ARL and RE In Table 2 our method selects models with higher Fl scores compared to En-Dev. also outperforms 100-Target across both languages. We hypothesize this is because 100 target- language examples is not sufficient for effective model selection, as the dataset contains a large proportion of negative examples (no relation). Also, RE and ARL have large label sizes (18 and 35) so a random sample of 100 instances might not cover every label. In contrast, the full dev set contains thousands of examples. We use GCNeeimp as a reference in the imbalanced dataset (see Appendix A.I for details). Reference models were selected using the English dev set.
Model Score Distributions Figure 2 visualizes the En-Dev and LMS results on the test set in the context of the score distributions of the 60 models in the meta-test set, using kernel density estima- tion. English development data tends to select models that perform only slightly better than average, whereas LMS does significantly better. Similar visualizations for NER, POS, and QA are presented Appendix A.2 In addition to the model score, we run a statistical analysis of the results for POS and QA in Appendix A.3 LMS is statistically significantly higher than En-Dev while Pivot-Dev fails in three target languages.
RE Fl Distribution
ARL F1 Distribution
En-Dey
1401
44%
260 725
Figure 2: Model Fl score distributions for RE and ARL. Red line: LMS and blue line: En-Dev. X-axis is Fl score. Selecting models with LMS achieve better results compared to En-Dev.
ANALYSIS
In Section we empirically demonstrated that our learned scoring function, s(.), consistently selects better models than the standard approach (En-Dev), and is comparable to small amounts of labeled target language data. Section presents additional analysis of our approach, exploring the impact of various modeling choices with (ar, de, es, nl, zh). In addition, analysis of generalization beyond mBERT, across tasks capability, and size of training are present in Appendices A.4 A.5 and A.6
7.1 MODEL AND LANGUAGE FEATURES
This section explores the impact of different choices for model and language representations for LMS. Four types of model features and two language embeddings are explored. We start by delin- eating possible choices for representations, then describe the details of our experiments, results, and the final choices used in [6
Four model-specific features are described below. Note [CLS] vectors are extracted from mBERT by feeding unlabeled text as input.
[Eng]: Averaged [CLS] vectors computed over unlabeled English text corpus used for both training and testing 4
[Pivot] ]: During training, [CLS] vectors are averaged over an unlabeled text corpus in
the pivot language. At test time, [CLS] embeddings are averaged over unlabeled corpus in the target language. We use the target-language development set (ignoring labels) for this purpose in experiments.
[Target]: [CLS] vectors are averaged over text corpus in the target language (for both training and testing).
'In experiments, sentences in the English dev set are used for this purpose (ignoring the labels).

---

Under review as a conference paper at ICLR 2021
Fusion: A linear combination of the above features. learned during training.
Weights each representation
Two types of language embeddings are examined.
1 ang2vec: 512-dimensional vectors learned by neural network trained for typological prediction (Malaviya et al. 20177)
syntax: 103-dimensional binary vectors, which capture syntax features from the URIEL knowledge base (Littell et al. 2017).
First, determine the choice of model-specific features by averaging performance across both language embeddings. Table 3 reports averaged evaluation metrics for each model-specific repre- sentation across all target languages with En-Dev as a baseline.
Averaged evaluation metrics across all target languages for each language embedding are reported in Table In addition to evaluating the effectiveness of each language embedding, also exper- imented with a variant of our scoring function that does not include any language embeddings as input. Results are reported on mBERT models in the meta-dev set and the target languages' dev sets for all experiments in this section.
Took
Ei-Dev
[Eng]
[Pivet]
[Target]]
Fusion
NER
70.45
70.64
71.18
71.87
70.66
POS
74.69
75.58
75.54
75.48
75.04
QA
56.31
56.49
56.79
56.68
56.63
RE 51.81
$4.92
55.57
$5.56
54.57
50.98
51.99
53.74
$2.31
Avg
60.85
61.92
62.60
62.38
62.32
Table 3: Model-specific feature analysis. We use mBERT models in the meta-dev set for analysis. Each number represents average of scores across all the target languages in particular task.
Task
En Dev
lang2voc
syntax
Nane
NER
70.45
71.37
70.98
70.08
QA
56.31
56.81
56.77
56.49
55,92
55.22
52.53
ARL
50.98
53.60
53.88
53,14
Avg
60.85
62.68
62.44
61.49
Table Language embedding analysis across lang2vec, synt and no language embedding. We use mBERT models in the meta-dev set for analysis. Each number represents average of scores across all the target languages in a particular task.
Table [2IVOT] features achieve top-2 performance in all five tasks. [Eng] and [ [Target] achieve mixed results, and the fusion of three features does not effectively incorporate the advantages of each representation, except in the case ARL. Table 4 shows that lang2vec outperforms syntax for all tasks but ARL and also outperforms not included. Thus, ang2vec and [P TVOT] are used for all experiments in Section
approach when language embeddings
8 RELATED WORK
Recent work has explored hyper-parameter optimization Klein et al. (2019), and model selection for a new task. task2vec (Achille et al.) 2019) presents a meta-learning approach to selecting pre-trained feature extractor from a library for resents tasks a vector space and is capable of predicting task similarities and taxonomic relations. It encodes task2ved we select a trained model for a specific task, and we represent a trained model with model-specific features target language.
visual task. More concretely, task2vec rep-
task and selects the best feature extractor trained the most similar task. Unlike
8

---

Under review as a conference paper at ICLR 2021
MAML (Finn et al. 2017: Rajeswaran et al. 2019) is another approach to meta-learning, pre-training a single model with a meta- moss to initialize set of parameters that can be quickly fine-tuned for related tasks. Nooralahzadeh et al. (2020) explore the of MAML in the cross-lingual transfer setting. MAMI, is designed to support few-shot learning through better initialization of model pa- rameters and does not address the problem of model selection. In contrast, approach improves model selection in the zero-shot cross-lingual transfer setting.
Most relevant to our work. Xiaa on an NLP task. They formulate this as a regression problem based on features of the task (dataset size, average sentence length, etc.), incorporating a discrete feature to represent the choice of model. In contrast, LMS inspects a model's 's internal representations, thus it is suitable for predicting which out of a set of fine-tuned models will best transfer to a target language. Also relevant prior work on learning to select the best language to transfer from Lin et al. 2019).
(2020) use regression methods predict a model's performance
Finally, we note that there is need for more NLP research on low-resource languages (Joshi al, 2020b). Lauscher et al. (2020) present number of challenges in transferring languages with few resources using pre-trained transformers. The languages used in our experiments could consid- ered high-resource, however, our experiments do cover fairly diverse set of languages, including Arabic and Chinese. We believe that there is still a need for more research on multilingual NLP for high-resource languages well, this is not solved problem. Finally, note that there several other prominent benchmarks for evaluating cross- pross-lingual transfer including XTERME (Hu et al. 2020) and XGLUE (Liang al. 2020), both of which include some datasets used in this work.
CONCLUSION
In this paper, we presented machine learning approach to model selection for zero-shot cross- lingual transfer, which is appropriate when small amounts of development data are available one or more pivot languages, but not in the target language. We showed that our approach improves over the standard practice of model selection using source language development data. Experiments on five well-studied NLP tasks show that by inspecting internal representations, our method consis- tently selects better models. LMS also achieves comparable results to the more expensive alternative of annotating small amounts of target-languue development data.
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless Fowlkes, Stefano Soatto, and Pictro Perona. Task2vec: Task embedding for meta-learnig. In Proceedings of the International Conference on Computer Vision, 2019.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguisties, 2017.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hul- lender. Learning rank using gradient descent. In Proceedings of International Conference on Machine Learning. 2005.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the Association for Computational Linguistics, 2019.
Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta seatallerning for fast adaptation of deep networks. In Proceedings of International Conference Machine Learning, 2017.

---

Under review as a conference paper at ICLR 2021
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of Artificial Intelligence and Statistics, 2011.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. Xtreme: massively multilingual multi-task benchmark for evaluating cross-lingual generaliza- tion. arXiv preprint arXiv:2003. 1080, 2020.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel Weld, Luke Zettlemoyer, and Omer Levy. Span- BERT: Improving vee-traiinng by representing and predicting spans. Transactions of the Associ- ationfor Computational Linguistics, 2020a.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the nlp world. arXiv preprint arXiv:2004.09095, 2020b.
M.G. Kendal. A new measure of rank correlation. Biometrika, (30):81-93, 1938.
Phillip Keung, Yichao Lu, Julian Salazar, and Vikas Bhardwaj. On the evaluation of contextual embeddings for zero-shot cross-lingual transfer learning. arXiv preprint 1rXiv:2004.1500 2020.
URL httpp::/aaaxx..ooopp//////00000/ wtt. 15001. pddf
Koo-Kyung Kim, Young-Bure Kim, Ruhi Sarikaya, and Eric Fosler-Lussier. Cross-lingual transfer learning for POS tagging without cross-lingual resources. In Proceedings of Empirical Methods in Natural Language Processing. 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of International Conference on Learning Representation, 2015.
Aaron Klein, Zhenwen Dai, Frank Hutter, Neil Lawrence, and Javier Gonzalez. Meta-surrogate benchmarking for hyperparameter optimization. Proceedings of Advances in Neurul Informa- tion Processing Systems, 2019.
Marius Köppel, Alexander Segner, Martin Wagener, Lukas Pensel, Andreas Karwath, and Ste- fan Kramer. Pairwise learning to rank by neural networks revisited: Reconstruction, theo- retical analysis and practical performance. arXiv preprint arXiv:1909.02768, 2019. URL
org/pd//////////// 1909. 02768. pdf
Anne Lauscher, Vinit Ravishankar, Ivan Vulié, and Goran Glavaš. From zero to hero: On the
limitations of zero-shot language transfer with multilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, November 2020. Association for Computational Linguistics. htth:t// /wwwww..... aclweb. oxq/anthology//22
emnlp-main. 363
Gina-Anne Levow. The third international chinese language processing bakeoff: Word segmenta- tion and named entity recognition. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, 2006.
Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. Mlqa: Evaluating cross-lingual extractive question answering. acl, 2020.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, et al. Xglue: A new benchmark dataset for cross-lingual pre-training, understanding and generation. arXiv preprint wxX1::2004.01401, 2020.
Yu-Hsiang Lin, Chian-Yo Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, et al. Choosing transfer languages for lingull learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, 2019.
Patrick Littell, David R Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. Uriel and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceedings of the European Chapter the Association for Computational Linguistics, 2017.
10

---

Under review as a conference paper at ICLR 2021
Chaitanya Malaviya, Graham Neubig, and Patrick Littell. Learning language representations for typology prediction. In Proceedings of Empirical Methods in Natural Language Processing, 2017.
Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. Exploiting similarities among languages for ma- chine translation. arXiv preprint arXiv: 1369,4168, 2013. URL httpttsss///xxi...t.c/ org/pdf/
1309. pdd
Joakim Nivre, Zeljko Agié, Lars Ahrenberg, Maria Jesus Aranzabe, Masayuki Asahara, Aitz- iber Atutxa, Miguel Ballesteros, John Bauer, Kepa Bengoetxea, Yevgeni Berzak, Riyaz Ahmad Bhat, Eckhard Bick, Carl Börstell, Cristina Bosco, Gosse Bouma, Sam Bowman, Gülşen Ce- biroglu Eryigit, Giuseppe G. A. Celano, Fabricio Chalub, Çağrı Çöltekin, Miriam Connor, Eliz- abeth Davidson, Marie-Catherine de Marneffe, Arantza Diaz de Harraza, Kaja Dobrovolje, Tim- othy Dozat, Kira Droganova, Puneet Dwivedi, Marhaba Eli, Tomaž Erjavec, Richárd Farkas, Jennifer Foster, Claudia Freitas, Katarína Gajdošová, Daniel Galbraith, Marcos Garcia, Moa Gärdenfors, Sebastian Garza, Filip Ginter, lakes Goenaga, Koldo Gojenola, Memduh Gökırmak, Yoav Goldberg, Xavier Gómez Guinovart, Berta Gonzáles Saavedra, Matias Grioni, Normunds Grüzïtis, Bruno Guillaume, Jan Hajič, Linh Hà Mý, Dag Haug, Barbora Hladká, Radu Ion, Blena Irimia, Anders Johannsen, Fredrik Jørgensen, Hüner Kaşıkara, Hiroshi Kanayama, Jenna Kanerva, Boris Katz, Jessica Kenney, Natalia Kotsyba, Simon Krek, Veronika Laippala, Lu- cia Lam, Phuong Lê H' ông. Alessandro Lenci, Nikola Ljubešić, Olga Lyashevskaya, Teresa Lynn, Aibek Makazhanov, Christopher Manning, Cätälina Märănduc, David Mareček, Héctor Martínez Alonso, André Martins, Jan Mašek, Yuji Matsumoto, Ryan McDonald, Anna Missilä, Verginica Mititelu, Yusuke Miyao, Simonetta Montemagni, Keiko Sophie Mori, Shunsuke Mori, Bohdan Moskalevskyi, Kadri Muischnek, Nina Mustafina, Kaili Müürisep, Lu' ng Nguy ên Thi, Huy ên Nguy ēn Thi Minh, Vitaly Nikolaev, Hanna Nurmi, Petya Osenova, Robert Östling, Lilja Øvrelid, Valeria Paiva, Elena Pascual, Marco Passarotti, Cenel-Augusto Perez, Slav Petrov, Jussi Pütulainen, Barbara Plank, Martin Popel. Lauma Pretkalniga, Prokopis Prokopidis, Tina Puo- lakainen, Sampo Pyysalo, Alexandre Rademaker, Loganathan Ramasamy. Livy Real, Laura Rit- uma, Rudolf Rosa, Shadi Saleh, Baiba Saulīte, Sebastian Schuster, Wolfgang Seeker, Mojgan Ser- aji, Lena Shakurova, Mo Shen, Natalia Silveira, Maria Simi, Radu Simionescu, Katalin Simkó, Mária Šimková, Kiril Simov, Aaron Smith, Carolyn Spadine, Alane Suhr, Umut Sulubacak, Zsolt Szántó, Takaaki Tanaka, Reut Tsarfaty, Francis Tyers, Sumire Uematsu, Larraitz Uria, Gertjan van Noord, Viktor Varga, Veronika Vincze, Lars Wallin, Jing Xian Wang, Jonathan North Wash- ington, Mats Wiren, Zdenčk Zabokrtsky, Amir Zeldes, Daniel Zeman, and Hanzhi Zhu. Universal Dependencies 1.4, 2016.
Farhad Nooralahzadeh, Giannis Bekoulis, Johannes Bjerva, and Isabelle Augenstein. Zero-shot cross-lingual transfer with meta learning. arXiv preprint arXiv:2003.0277 2020.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018
Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual BERT? In Pro- ceedings of the Association for Computational Linguistics, 2019.
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with im- plicit gradients. In Proceedings of Advances in Neural Information Processing Systems, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of Empirical Methods Natural Language Processing, 2016.
Erik F. Tjong Kim Sang. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In Proceedings of International Conference on Computational Lin- guistics, 2002.
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-inddeendent named entity recognition. In Proceedings of the Conference of the North
11

---

Under review as a conference paper at ICLR 2021
American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, 2003.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. Matching the blanks: Distributional similarity for relation learning. In Proceedings of the Association for Com- putational Linguistics, 2019.
Ananya Subburathinam, Di Lu, Heng Ji, Jonathan May, Shih-Fu Chang, Avirup Sil, and Clare Voss. Cross-lingual structure transfer for relation and event extraction. In Proceedings of Empirical Methods in Natural Language Processing and the International Joint Conference Natural Lan- guage Processing, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of Advances in Neural Information Processing Systems, 2017.
Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. ACE 2005 multilingual training corpus. Linguistic Data Consortium, 2006. LDC2006T06.
Shijie Wu and Mark Dredze. Beto, Bentz, Becas:
The surprising cross-lingual effectiveness of BERT. In Proceedings of Empirical Methods in Natural Language Processing and the Interna- tional Joint Conference on Natural Language Processing, 2019.
Mengzhou Xia, Antonios Anastasopoulos, Ruochen Xu, Yiming Yang, and Graham Neubig. Pre- dicting performance for natural language processing tasks. In Proceedings of the Association for Computational Linguistics, 2020.
Jiateng Xie. Zhilin Yang, Graham Neubig, Noah A Smith, and Jaime G Carbonell. Neural cross- lingual named entity recognition with minimal resources. In Proceedings of Empirical Methods in Natural Language Processing, 2018
Wei Ye, Bo Li, Rui Xie, Zhonghao Sheng, Long Chen, and Shikun Zhang. Exploiting entity BIO tag embeddings and multi-task learning for relation extraction with imbalanced data. In Proceedings of the Association for Computational Linguistics, 2019.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 IMBALANCED DATA FOR RE AND ARL
In this section, we describe details of the imbalanced dataset for RE and ARL. Table reports the statistics of the dataset and
references and baseline results.
The number non-relation entity pairs or trigger-entity pairs (negative instances) far exceeds pos- itive instances, which negatively affects model's performance. However, the preprocessed dataset from (Subburathinam et al.) 2019) is artificially balanced, which does not reflect practical set- ting. Therefore, create additional imbalanced dataset using the ACE2005 corpus Walker al. (2066, which more closely replicates the setting a model will be faced with in a real-world infor- mation extraction scenario. First, we shuffle documents into 80%/10%//05 splits for train/dev/test, then extract candidate entity-pairs from each document. For RE, the first approach in (Ye et al,, 2019) is adopted extract negative instances. Negative instances whose entity-type combination has never appeared as positive example in the training data are filtered out. This results in positive/negative ratio of 1:8.9 within 80,256 mention pairs for English. We repeat the same pro- cess for Chinese/ rabic and end up with positive/negative ratio of 1:9.4/1:8. and 73,08 73,082/42,106 instances. For ARI,, we create negative instances by pairing each trigger with every entity sen- tence. This leads to positive/negative ratio of 2.6/2.7/2. 7/2.9 and 27,823/19, 3333//14,336 instances in English/Chinese/AAabbl Details on the two datasets are summarized in Table
As baseline for the imbalanced dataset, reimplement the Graph Convolutional Network (GCN) model of (Subburathinam et al.) 2019) using multilingual embeddings learned by fastText (Bo- janowski et al. (017) Wikipedia (GCNRelmp)- Tables 6 display Fl for zero-shot cross-lingual
transter in both balanced and imbalanced datasets.
Task
Lang
Train
Dev
Test
Pos/Neg
en 63177
10218
6861
1:8.9
RE
zh
57824
7096
8162
1:9.4
32984
4482
4638
1:8.9
21875
3345
2603
1:2.6
ARL
15095
2226
2017
1:2.7
ar
11587
1221
1568
1:2.9
Table 5: Statistics of the imbalanced dataset. Number of instances and the total positive/negative ratio.
RE (F,) zh
ARL (F1) zh
Balanced Datasel
GCN (Subburathinam et al.
2019
58.70
42.50
61.80
59.00
GCNgelup
56.10
41.70
62.08
55.10
BBERT
79.56
59.27
68.00
Imbalanced Dataset
GCNRaup
39.43
32.74
16.48
23.49
Model Selection (Imbalanced Dataset)
En-Dev
36. 10
67.66
44.11
60.96
LMS
39.54
70.75
47.08
62.05
100- Target
34.68
68.20
44.11
62.52
All-Target
41.92
69.13
47.15
63.81
Table 6: F1 scores for relation extraction and argument role labeling on the test set. En-Dev/100- Target/All-Targee model selection based on the highest Fl of English dev set/100 target language dev set examples/target language dev set. Ours: model selection based on the highest scores for the target language: arg max,,, sm,, lunget).
13

---

Under review as a conference paper at ICLR 2021
NER F1 Distribution
de
es
nl
zh
En-Dev
LMS
61
65
70
70.0
72.5
75.0
77.5
76
78
50.0
52.5
55.0
57.5
POS Accuracy Distributiin
de
2h
En Dev
LMS
50
40
75
76
97
QA F1 Distribution
ar
de
es
zh
En-Dev LMS
46
48
50
52
54
56
58
63
64
65
66
57
58
59
Figure 3: Model score distribution of models in the meta-test for NER, POS, and QA. LMS (Red) and En-Dev (Blue). X-axis is F1 score for NER and QA and accuracy for POS.
A.2 MODEL SCORE DISTRIBUTION
Details of model distribution explained and analyzed in this section. Distribution figures of NER, POS, and QA are shown in Figure
To better interpret model selection results, we visualize the distribution model scores in the meta- test set with the kernel density estimate (KDE) plot. KDE represents the data using continuous probability density curve in one more dimensions. LMS (Red) and En-Dev (Blue) selection results are marked in the model score distribution. The majority model scores are concentrated at the center of the distribution but the variations are large. For example, Arabic POS accuracy in Figure 3 ranging from 48 to 53, where the majority of models achieve score around 50. En-Dev fails to select a model from the middle of the distribution, which leads to a score below average models. However, LMS is able to not only select model better than the En-Dev but also select model that is better than the majority of candidates. strategy when it comes to linguistically different languages such as Arabic and Chinese. For Arabic POS, Arabic QA, and Chinese POS, En-Dev selects a model that performs worse than an average model from the candidates (left side of the distribution). This might suggest the model is over- wettffttiig to English data but fails to transfer to a distanced language. Our observations motivate us to develop better model selection method LMS.
fact, En-Dev is not a good model selection
A.3 STATISTICAL TEST OF LMS PERFORMANCE
In this section, we present statistical analysis of model selection results for POS and QA. Table shows LMS on average improve point of 0.74 relative to and a point 0.44 relative to Pivot-Dev.
We train single LMS with pivot languages in far, de, es, nl, zh} for POS and {ar, de, es, es, 2h) for QA, following by testing it on the target languages. All the results are reported with mean and standard deviation with five runs (different random seeds). A z-test is performed to the differences between LMS/Pivot-Des and En-Dev.
LMS is statistically significantly (p-value < 0.05) higher than En-De baseline across all languages and two tasks while Pivot-Dev fails in three languages. LMS also obtains lower standard deviation for the model scores except for Swedish (sv) and Vietnamese (vi).
14

---

Under review as a conference paper at ICLR 2021
Methoch
POS
Ta Rin
185tion
21.Suam
M--iia
13
87/Wino
9:33ic
3000
53.10
71.5
1725
125t
343axar
428
MMMo
83idian
84.
Al-liaps
1236in
of Al-llage
122
200
211
199
1990
235
304
30
301
Table 7: Model scores (mean + sd) selected based on LMS for POS and QA over 5 runs. Bold indicates the best score and underline indicates the second best. indicates the LMS/Pivot-Dev is statistically significantly (p-value < 0.05) higher than En-Dev.
A.4 DOES THIS APPROACH GENERALIZE TO 3LM-ROBERTA?
In Section we showed that our approach consistently selects better fine-tuned models than those chosen using English development data. To test the robustness of our approach with different multilingual pre-trained transformer, we re-train and evaluate using XLM-RoBERTa- base (Conneau
et al. 2019), with the same settings used for mBERT in Section 6 when testing RE and ARL with imbalance dataset.
RE In the left section of Table our approach selects a model with higher F1 score compared to En-Dev in Chinese and on par with En-Dev Arabic.
ARL In the right section of Table our approach selects model with higher F1 score compared
to En-Dev in Arabic but performs worse on Chinese. Overall, our approach appears to be effective when used with XLM-RoBERTa.
RE (F,)
ARL (F;)
ar
ar
GCNgampp
39.43
32.74
16.48
23.49
36.10
Model Selection
En-Dev LMS
40.79
64.48
50.65
62.73
40.79
52.96
100-Target
42.33
All Target
44.66
65.75
53.09
62.27
Table 8: XLM-RoBERTa experiment: F1 of relation extraction and argument role labeling the imbalanced dataset. Model selection results are based on XLM-RoBERTa base models in the meta- test set.
A.5 CAN MULTI-TASK LEARNING HELP?
Our setting does not assume access to the labeled data in the target language for particular task. However, labeled data in the target language may be available for a relevant auxiliary task, which could help the scoring function learn better estimate whether model good match for the target language.
To test whether an auxiliary task in the target language might help select better model for the target task, we fine-tune sets of mBERT models for ARI. and RE. Gold rankings on the models are then computed for each language using the pivot languages' dev sets. Also, another "silver" ranking is computed for each language using the auxiliary task. The scoring function is then trained rank mBERT models for both tasks. To differentiate the two tasks, a variant of the scoring function, s(m, which concatenates randomly initialized task embedding with the language embedding adopted. Following Section 7 we use model selection results from mBERT models in the meta- dev set and report average target language dev-set results across Chinese and Arabic.
In Table our approach can select a model with higher Fl score for RE. However, multi-tase task does not benefit ARI. but still outperforms En- As for future direction, believe LMS that is
15

---

Under review as a conference paper at ICLR 2021
Hew isaiiing wie effecis LMS7
LM5
Kenial Pank Corpobaioon Cooffidantt
Figure 4: The similarity between En-Dev/LMS ranking and the golden ranking for models in the QA meta-test set.
trained auxiliary dataset can be transferred to the target dataset, hence release the requirement of small amount of pivot language development data in the target dataset.
Task
Fen
([Pivot), lang2vec)
Multi-task
51.81
55.92
$7.31
ART.
50.98
$3.60
51.99
Table Multi-task analysis using additional training data in the target language from another task. We use mBERT models in the meta-dev set for analysis. Model selection is based the highest scores for the target language and target task: arg max s(m, Aaargtt (arggt))
A.6 How MANY MODELS DO WE NEED TO TRAIN LMS?
We analyze how practical LMS is in this section. In particular, we aim to explore the question of can the LMS be trained without large amount models. LMS can be trained with only six models.
LMS outperforms the En-Dev in three out of four target languages by training on only six models (Table [O). Varying the number of models in the meta-training set from 6 to 120 for QA, LMS performs consistently across a different number of training sizes. It outperforms the En-Dev when trained with 72 models.
To better understand how LMS performs compared to En-Dev, we use Kendall Rank Correlation Coefficient (KRCC) (Kendal 1938) to measure the similarity of the orderings. KRCC (-1 1) be- tween two model rankings will be high when having a similar rank. In Figure KRCC calculated LMS
between En- Dev/LMSS and golden model rankings for 60 models in the QA meta-test outperforms En-Dev with only six models and the KRCC improves as more models are used to train LMS. It is interesting see LMS has the highest KRCC Arabic. We hypothesis this is because of the high variation of Fl score in Arabic the standard deviation of the Arabic Fl score is 3.3 as opposed 2.4 and 1.3 in German and Spanish.
ar
de
En-Dev
47.72
55.27
64.92
58.03
of models to learn LMS
48.95
55.21
65.28
58.27
48.95
55.21
65.28
58.27
72
48.95
56.03
64.95
58.27
120
49.26
55.86
64.95
58.11
Table 10: Model selection results for QA (F1). A varying number of models to train LMS from 6 to 120.
16


Reviews:
[{"review_id": "l6I9tAxnoKi", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an instance of a meta-learning approach, dedicated to performing model selection in a zero-shot setting. This Learned Model Selection (LMS) approach consists of a learned function scoring the compatibility between a fine-tuned multilingual transformer and a target language, without access to any data in target language (hence zero-shot).\n\nThe empirical results show that the LMS technique is effective in predicting when a multilingual model’s representations are a good match for the target language, as opposed to a baseline in which the model selection is done using the performance on En dev data. Moreover, it performs similarly with the performance of having access to some small amount of target data (the 100-Target experimental condition), and not very far from the full target data condition (the Full-Target experimental condition).\n\nInterestingly, the authors show that the 512-dimensional vectors learned by a neural network trained for typological prediction (lang2vec) are the most effective features to be used in their approach. This is important because it validates that the degree to which this meta-learning approach works depends on the family of languages for which the task is attempted (the fit between the training-data language family and the target language family).\n\nThe paper is well-written, with the experimental section well executed.\n\nSuggestions:\nSome of the results are a bit counterintuitive, and I would like to understand a bit better if there are some explanations or intuitions on why that would be; for instance, the LMS approach has better performance on some tasks (RE, ARL) than the 100-Target approach; given that access to the target language should provide inherently some advantages, what is the explanation for that? Is that the 100-Target baseline somehow misuses this advantage?\n\nSomewhat related to the above: I could not see how much data the Full-Target condition is using; seeing that the Full-Target actually performs clearly better on these (RE, ARL) than both LMS and 100-Target, it would be illuminating to understand the threshold N (in terms of data amounts) at which LMS and N-Target would be on-par, especially if that illustrates a correlation between the closeness in the language family for target  vs the value of N.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ennhxFJGFwm", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nResearch Problem: Zero-shot cross-lingual transfer, with model selection using source language dev set, has high variance with different hyperparameters or even different random seeds.\n\nThis paper proposes learning a model selection function using pivot languages dev set. The model selection function is a scoring function, taking the contextual representation and external language embeddings as input. The scoring function is trained with a set of fine-tuned models and its performance ranking in pivot languages dev set. This paper conducts experiments on 5 NLP tasks and cross-lingual transfer from English to 5 target languages, and shows outperforming model selection using the English dev set. It also presents ablation study on the feature of the scoring function and the number of models for training the scoring function.\n\n## Pros\n1. This paper presents an interesting approach to learn model selection using model representation and external language embeddings.\n2. In the conducted experiments, the proposed method is shown to outperform selecting models using the English dev set.\n3. This paper also conducts an ablation study on modeling decisions of the scoring function.\n \n## Cons\n1. The modeling assumption seems relatively limited. It assumes only one target language at a time per scoring function (“our goal is to select the model that performs the best on a target language, $l_\\text{target}$”). In this experiment where there are multiple target languages of interest, I am assuming it adapts a leave-one-out approach. However, it is not clearly stated. If my understanding is correct, the number of scoring functions increase linearly with the number of target languages. While training the scoring function might be fast (not discussed in the paper), this paper should discuss this assumption in more detail.\n2. The experiment's setup is not clearly presented. Aside from the previous example, the experiment's setup is sometimes hard to follow, which makes it relatively hard to assess the claim in the experiment. For example,\n- This paper states 240 mBERT models are fine-tuned. But it’s unclear how exactly this count is computed. mBERTs are fine-tuned with 3(learning rate)x4(epochs)x1(batch size)=12 hyperparameter combinations. While it mentioned different random seeds are considered, it is unclear how many seeds are used.\n- It is unclear what is the source of unlabeled text.\n- It is unclear how many checkpoints each set of hyperparameters produce.\n- Most importantly, the meta-test set should come from a set of models with at least **different random seeds**, it’s unclear whether the current split can test the generalization ability of the scoring function.\n3. Get back to the original research problem, recall that cross-lingual transfer has high variance. From this paper's main result, it is unclear whether the proposed method has lower variance compared to model selection with source language dev set, as only single performance is reported.\n\n## Questions during rebuttal period\n1. In appendix A.3., is a new scoring function trained for XLM-R or is it the same scoring function of mBERT? If the latter is true, I would not refer to it as “generalization”, and the results are quite interesting. If the former is true, more results on other tasks are beneficial as the results so far on XLM-R are mixed.\n\n## Reasons for score\nOverall, I am leaning toward rejecting. While I find the idea of learning model selection quite interesting, the experiments presented in this current version cast doubt on the empirical result of this paper. Hopefully the authors can clarify and address my concern in the rebuttal period. \n  \n## Minor comment\n1. Referring languages other than target language as pivot language seems misleading, as this paper does not assume any similarity between target language and “pivot language”.\n2. Referring 100-Target or Full-Target as oracle seems misleading, as the proposed method sometimes even outperforms the “oracle”. The oracle should be the best possible performance in the meta-test set.\n3. In table 1, es should have checked instead of nl.\n\n## After rebuttal\nThank you for answering my questions! It address my concern and I revise my recommendation.\n\n- Please clarify in the next version how the checkpoint is selected (e.g. which epoch is selected?) given a product of seed and hyperparameter.", "extra_scores": {}, "extra_text": {}}, {"review_id": "TeInpc2CW6o", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n- The most common approach to model selection in zero-shot learning is to use English development data.\n- Using English as selection proxy is biased and likely yields suboptimal models.\n- The paper proposes a meta-learning based approach to select models by using its own internal representations to predict cross-lingual performance.\n- Learned Model Selection (LMS) learns a scoring function that scores how compatible a fine-tuned multilingual transformer is to the target language.\n- The target languages are represented by using lang2vec as a reference point.\n\nStrengths:\n- Very simple and intuitive model.\n- Strong results in comparison to the English dev data baseline.\n- Thorough discussion and overall enjoyable writeup.\n\nWeaknesses:\n- A disappointing lack of cross-lingual breadth: only six languages in the experiment.\n- Especially displeasing that the Universal Dependencies POS test includes only 5 rather resource-rich languages.\n- While CoNLL NER data is well-established, there is data for 250+ languages by Pan et al. for NER as well.\n\nOverall:\n- I am strongly in favor of the hypothesis and model that the paper proposes and thus I vote accept.\n- Would have voted strong accept if not for the underwhelming subset of languages that feature in the experiment.\n- I kindly suggest that the final-version revision includes more languages and typological analysis in the experiments.\n\nQuestions:\n1. How biased is the usage of English development data with respect to language (dis)similarity to target language in these tasks?\n2. What would happen if we replaced lang2vec by another representation, e.g., from WALS features or subsets of those?", "extra_scores": {}, "extra_text": {}}, {"review_id": "3-QHRTV2DId", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=== Update after the revision and the author response ===\n\nI would like to thank the authors for the additional work and effort invested into improving the paper presentation. This has made me increase my score; I still have some doubts regarding the experimental setup (using target dev sets and taking this information for granted), but maybe a high-level question I posed in my review really does go beyond this work. Given that the main premise is 'quick adaptation' to new (and unseen) languages, the inclusion of at least one-two truly low-resource languages would have still been nice to link the motivation with the experimental setup.\n\n===\n\nThis paper tackles one interesting problem pertaining to zero-shot cross-lingual transfer, which was observed in previous work. Namely, doing model selection based on English (as the typical source language) dev data often displays suboptimal transfer performance in a wide range of tasks and for a wide range of target languages. The authors set out to tackle this problem by proposing a new approach based on the pairwise learning-to-rank (LTR) framework which combines the multilingual pretrained model's internal representations with some typological knowledge coming from available language vectors (lang2vec vectors are used). The results across five tasks show that doing better model selection (effectively discarding model selection on English dev set) suggest that the LTR-based method does yield improvements in transfer performance. Overall, I see this work as a potentially nice small contribution to the growing area of cross-lingual transfer learning while isolating and motivating the concrete problem in a solid way, but I see a number of ways on how to improve the paper and make this a stronger submission, and I cannot accept it in its current format.\n\n1. While disguising the method as a less resource demanding than the standard approach based on EN dev set, the method is in fact much more resource-demanding. It assumes existence of development data in a sufficient number of target languages (excluding test languages, ofc) which are used to run the LTR training over a number of pretrained mBERT models. Is this a reasonable assumption? How does the amount of languages for which we have development data (and its size) affect the LTR method?\n\n2. Following my previous comment, one pretty obvious baseline/method is missing from the comparisons, if we assume existence of dev data in many languages. For instance, instead of running LTR, why not using development data of a language which is quite similar to the test language, and optimise performance on that 'neighbouring/pivot' language? For instance, if we want to optimise our model to transfer well for Italian NER, and we assume that we possess NER dev data for Spanish, why don't we simply do model selection relying on Spanish NER data (or using Spanish + Catalan or Spanish + Catalan + French NER data if we have dev data for all these languages)? Given that the gains with the LTR approach are still quite small compared to selection based on EN dev, I wonder how this baseline approach would work.\n\n3. Unfortunately, all the experiments are run only on languages which could be considered high-resource languages (e.g., see the work of Joshi et al., ACL 2020 or Lauscher et al.. EMNLP 2020). However, this work in particular should pay particular attention to truly low-resource scenarios, and not limit the evaluations only to a small number of high-resource languages. Low-resource languages are exactly the ones that benefit most from (zero-shot or few-shot) transfer, and the ones where a more careful model selection should really make the difference. However, the paper does not offer any experiments in those setups. Perhaps running some additional experiments with evaluation data assembled in the recent XTREME and XGLUE benchmarks could be insightful?\n\n4. Model-specific feature representation - given that there are so many different ways (e.g., mean/max pooling, using [CLS], using only first-occurring during subwords, layerwise averaging) to extract model-specific feature representations, the authors should provide more justification on choosing these particular feature representations? Have they explored a wider set of options here?\n\n5. Another question relates to a higher-level relationship between model selection versus in-task fine-tuning - related to my comments 1) and 2) above, if we assume existence of some pivot language dev data, the question is whether it is better to use the data for model selection or simply do task-specific fine-tuning on the dev data from related languages to increase the model capability to deal with the final test language? Do the authors have any insights here?\n\n6. I find the use of term 'meta-learning' a bit misleading in this context, as it does not fully relate to the standard research domain of 'meta-learning', and it might even mislead the reader. The work is not strictly 'meta-learning'.\n\n\nMinor:\n- Is there a way to automatically define the number of models to train LMS and optimise their differences so that we do not use redundant models?\n- Given that there are multiple possible approaches to choosing a suitable model for transfer, can the authors provide additional discussion and justification for their chosen approach? The problem is indeed most intuitively tackled as a learning-to-rank problem, but there are also many different flavours of LTR. Why this particular implementation?\n- I would suggest to cite this highly relevant paper, which actually resembles the main ideas from this paper quite a bit, but instead of choosing between many models from the same language, it chooses models from different languages (1 model in each language = many models, but 1 per language) to optimise for transfer performance: https://arxiv.org/pdf/1905.12688.pdf ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the paper has merits, the experiments are lacking in important respects: I agree with Reviewer 1 that it is a serious problem that the approach is not evaluated on truly low-resource languages - since a significant pivot-to-target language bias is to be expected (as also suggested by Reviewer 2). I also agree with the sentiment that the work is not properly baselined, without considering alternative ways of using the pivot language development data. I also agree with Reviewer 3 that the 1:1 assumption is limiting, given that multi-source transfer has been de facto standard since 2011 (see, e.g., work by McDonald, Søgaard, Cohen, etc.). I’m also a little worried about using dev data for unlabelled data, since this is data from the exact same sample as the test data. In practice, dev data will be biased, and artificially removing this bias will lead to overly optimistic results. 
Decision: Reject