Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: === Update after the revision and the author response ===

I would like to thank the authors for the additional work and effort invested into improving the paper presentation. This has made me increase my score; I still have some doubts regarding the experimental setup (using target dev sets and taking this information for granted), but maybe a high-level question I posed in my review really does go beyond this work. Given that the main premise is 'quick adaptation' to new (and unseen) languages, the inclusion of at least one-two truly low-resource languages would have still been nice to link the motivation with the experimental setup.

===

This paper tackles one interesting problem pertaining to zero-shot cross-lingual transfer, which was observed in previous work. Namely, doing model selection based on English (as the typical source language) dev data often displays suboptimal transfer performance in a wide range of tasks and for a wide range of target languages. The authors set out to tackle this problem by proposing a new approach based on the pairwise learning-to-rank (LTR) framework which combines the multilingual pretrained model's internal representations with some typological knowledge coming from available language vectors (lang2vec vectors are used). The results across five tasks show that doing better model selection (effectively discarding model selection on English dev set) suggest that the LTR-based method does yield improvements in transfer performance. Overall, I see this work as a potentially nice small contribution to the growing area of cross-lingual transfer learning while isolating and motivating the concrete problem in a solid way, but I see a number of ways on how to improve the paper and make this a stronger submission, and I cannot accept it in its current format.

1. While disguising the method as a less resource demanding than the standard approach based on EN dev set, the method is in fact much more resource-demanding. It assumes existence of development data in a sufficient number of target languages (excluding test languages, ofc) which are used to run the LTR training over a number of pretrained mBERT models. Is this a reasonable assumption? How does the amount of languages for which we have development data (and its size) affect the LTR method?

2. Following my previous comment, one pretty obvious baseline/method is missing from the comparisons, if we assume existence of dev data in many languages. For instance, instead of running LTR, why not using development data of a language which is quite similar to the test language, and optimise performance on that 'neighbouring/pivot' language? For instance, if we want to optimise our model to transfer well for Italian NER, and we assume that we possess NER dev data for Spanish, why don't we simply do model selection relying on Spanish NER data (or using Spanish + Catalan or Spanish + Catalan + French NER data if we have dev data for all these languages)? Given that the gains with the LTR approach are still quite small compared to selection based on EN dev, I wonder how this baseline approach would work.

3. Unfortunately, all the experiments are run only on languages which could be considered high-resource languages (e.g., see the work of Joshi et al., ACL 2020 or Lauscher et al.. EMNLP 2020). However, this work in particular should pay particular attention to truly low-resource scenarios, and not limit the evaluations only to a small number of high-resource languages. Low-resource languages are exactly the ones that benefit most from (zero-shot or few-shot) transfer, and the ones where a more careful model selection should really make the difference. However, the paper does not offer any experiments in those setups. Perhaps running some additional experiments with evaluation data assembled in the recent XTREME and XGLUE benchmarks could be insightful?

4. Model-specific feature representation - given that there are so many different ways (e.g., mean/max pooling, using [CLS], using only first-occurring during subwords, layerwise averaging) to extract model-specific feature representations, the authors should provide more justification on choosing these particular feature representations? Have they explored a wider set of options here?

5. Another question relates to a higher-level relationship between model selection versus in-task fine-tuning - related to my comments 1) and 2) above, if we assume existence of some pivot language dev data, the question is whether it is better to use the data for model selection or simply do task-specific fine-tuning on the dev data from related languages to increase the model capability to deal with the final test language? Do the authors have any insights here?

6. I find the use of term 'meta-learning' a bit misleading in this context, as it does not fully relate to the standard research domain of 'meta-learning', and it might even mislead the reader. The work is not strictly 'meta-learning'.


Minor:
- Is there a way to automatically define the number of models to train LMS and optimise their differences so that we do not use redundant models?
- Given that there are multiple possible approaches to choosing a suitable model for transfer, can the authors provide additional discussion and justification for their chosen approach? The problem is indeed most intuitively tackled as a learning-to-rank problem, but there are also many different flavours of LTR. Why this particular implementation?
- I would suggest to cite this highly relevant paper, which actually resembles the main ideas from this paper quite a bit, but instead of choosing between many models from the same language, it chooses models from different languages (1 model in each language = many models, but 1 per language) to optimise for transfer performance: https://arxiv.org/pdf/1905.12688.pdf 
review_title: An interesting problem, a good motivation for the work, but needs more work...

final_score: rating: 7: Good paper, accept
review_content: The paper presents an instance of a meta-learning approach, dedicated to performing model selection in a zero-shot setting. This Learned Model Selection (LMS) approach consists of a learned function scoring the compatibility between a fine-tuned multilingual transformer and a target language, without access to any data in target language (hence zero-shot).

The empirical results show that the LMS technique is effective in predicting when a multilingual model’s representations are a good match for the target language, as opposed to a baseline in which the model selection is done using the performance on En dev data. Moreover, it performs similarly with the performance of having access to some small amount of target data (the 100-Target experimental condition), and not very far from the full target data condition (the Full-Target experimental condition).

Interestingly, the authors show that the 512-dimensional vectors learned by a neural network trained for typological prediction (lang2vec) are the most effective features to be used in their approach. This is important because it validates that the degree to which this meta-learning approach works depends on the family of languages for which the task is attempted (the fit between the training-data language family and the target language family).

The paper is well-written, with the experimental section well executed.

Suggestions:
Some of the results are a bit counterintuitive, and I would like to understand a bit better if there are some explanations or intuitions on why that would be; for instance, the LMS approach has better performance on some tasks (RE, ARL) than the 100-Target approach; given that access to the target language should provide inherently some advantages, what is the explanation for that? Is that the 100-Target baseline somehow misuses this advantage?

Somewhat related to the above: I could not see how much data the Full-Target condition is using; seeing that the Full-Target actually performs clearly better on these (RE, ARL) than both LMS and 100-Target, it would be illuminating to understand the threshold N (in terms of data amounts) at which LMS and N-Target would be on-par, especially if that illustrates a correlation between the closeness in the language family for target  vs the value of N.

review_title: MODEL SELECTION FOR CROSS-LINGUAL TRANSFER USING A LEARNED SCORING FUNCTION

final_score: rating: 7: Good paper, accept
review_content: Summary:
- The most common approach to model selection in zero-shot learning is to use English development data.
- Using English as selection proxy is biased and likely yields suboptimal models.
- The paper proposes a meta-learning based approach to select models by using its own internal representations to predict cross-lingual performance.
- Learned Model Selection (LMS) learns a scoring function that scores how compatible a fine-tuned multilingual transformer is to the target language.
- The target languages are represented by using lang2vec as a reference point.

Strengths:
- Very simple and intuitive model.
- Strong results in comparison to the English dev data baseline.
- Thorough discussion and overall enjoyable writeup.

Weaknesses:
- A disappointing lack of cross-lingual breadth: only six languages in the experiment.
- Especially displeasing that the Universal Dependencies POS test includes only 5 rather resource-rich languages.
- While CoNLL NER data is well-established, there is data for 250+ languages by Pan et al. for NER as well.

Overall:
- I am strongly in favor of the hypothesis and model that the paper proposes and thus I vote accept.
- Would have voted strong accept if not for the underwhelming subset of languages that feature in the experiment.
- I kindly suggest that the final-version revision includes more languages and typological analysis in the experiments.

Questions:
1. How biased is the usage of English development data with respect to language (dis)similarity to target language in these tasks?
2. What would happen if we replaced lang2vec by another representation, e.g., from WALS features or subsets of those?
review_title: Relevant contribution, thorough empirical exposition, terribly limited language breadth

final_score: rating: 7: Good paper, accept
review_content: ## Summary
Research Problem: Zero-shot cross-lingual transfer, with model selection using source language dev set, has high variance with different hyperparameters or even different random seeds.

This paper proposes learning a model selection function using pivot languages dev set. The model selection function is a scoring function, taking the contextual representation and external language embeddings as input. The scoring function is trained with a set of fine-tuned models and its performance ranking in pivot languages dev set. This paper conducts experiments on 5 NLP tasks and cross-lingual transfer from English to 5 target languages, and shows outperforming model selection using the English dev set. It also presents ablation study on the feature of the scoring function and the number of models for training the scoring function.

## Pros
1. This paper presents an interesting approach to learn model selection using model representation and external language embeddings.
2. In the conducted experiments, the proposed method is shown to outperform selecting models using the English dev set.
3. This paper also conducts an ablation study on modeling decisions of the scoring function.
 
## Cons
1. The modeling assumption seems relatively limited. It assumes only one target language at a time per scoring function (“our goal is to select the model that performs the best on a target language, $l_\text{target}$”). In this experiment where there are multiple target languages of interest, I am assuming it adapts a leave-one-out approach. However, it is not clearly stated. If my understanding is correct, the number of scoring functions increase linearly with the number of target languages. While training the scoring function might be fast (not discussed in the paper), this paper should discuss this assumption in more detail.
2. The experiment's setup is not clearly presented. Aside from the previous example, the experiment's setup is sometimes hard to follow, which makes it relatively hard to assess the claim in the experiment. For example,
- This paper states 240 mBERT models are fine-tuned. But it’s unclear how exactly this count is computed. mBERTs are fine-tuned with 3(learning rate)x4(epochs)x1(batch size)=12 hyperparameter combinations. While it mentioned different random seeds are considered, it is unclear how many seeds are used.
- It is unclear what is the source of unlabeled text.
- It is unclear how many checkpoints each set of hyperparameters produce.
- Most importantly, the meta-test set should come from a set of models with at least **different random seeds**, it’s unclear whether the current split can test the generalization ability of the scoring function.
3. Get back to the original research problem, recall that cross-lingual transfer has high variance. From this paper's main result, it is unclear whether the proposed method has lower variance compared to model selection with source language dev set, as only single performance is reported.

## Questions during rebuttal period
1. In appendix A.3., is a new scoring function trained for XLM-R or is it the same scoring function of mBERT? If the latter is true, I would not refer to it as “generalization”, and the results are quite interesting. If the former is true, more results on other tasks are beneficial as the results so far on XLM-R are mixed.

## Reasons for score
Overall, I am leaning toward rejecting. While I find the idea of learning model selection quite interesting, the experiments presented in this current version cast doubt on the empirical result of this paper. Hopefully the authors can clarify and address my concern in the rebuttal period. 
  
## Minor comment
1. Referring languages other than target language as pivot language seems misleading, as this paper does not assume any similarity between target language and “pivot language”.
2. Referring 100-Target or Full-Target as oracle seems misleading, as the proposed method sometimes even outperforms the “oracle”. The oracle should be the best possible performance in the meta-test set.
3. In table 1, es should have checked instead of nl.

## After rebuttal
Thank you for answering my questions! It address my concern and I revise my recommendation.

- Please clarify in the next version how the checkpoint is selected (e.g. which epoch is selected?) given a product of seed and hyperparameter.
review_title: Interesting Idea but Concern on Experiment

Meta-review: While the paper has merits, the experiments are lacking in important respects: I agree with Reviewer 1 that it is a serious problem that the approach is not evaluated on truly low-resource languages - since a significant pivot-to-target language bias is to be expected (as also suggested by Reviewer 2). I also agree with the sentiment that the work is not properly baselined, without considering alternative ways of using the pivot language development data. I also agree with Reviewer 3 that the 1:1 assumption is limiting, given that multi-source transfer has been de facto standard since 2011 (see, e.g., work by McDonald, Søgaard, Cohen, etc.). I’m also a little worried about using dev data for unlabelled data, since this is data from the exact same sample as the test data. In practice, dev data will be biased, and artificially removing this bias will lead to overly optimistic results. 
Decision: Reject