Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper looks at the problem of unsupervised cross-lingual transfer (termed UCL) in the paper through the optics of domain adaptation. After empirically analysing and validating that distributional shifts in class priors might cause a huge problem for UCL (which wasn't tackled in previous research), the authors proceed with an introduction of a new method that aims to mitigate that problem. The idea is to get rid of that shift through a approach called importance-weighted domain adaptation (IWDA), which is largely the adaptation of the work from Tachet des Combes et al. (NeurIPS 2020) to the UCL problem.

The results on two tasks in the UCL setup (NER and MARC classification) show slight gains over the standard zero-shot transfer when IWDA is applied, with more prominent gains reported when a stronger domain shift is observed - however, such a setup has been created mostly artificially, to further demonstrate the benefits of modelling the shift in the model.
main_review: ===== UPDATE AFTER THE RESPONSE =====
I would like to thank the authors for the new set of experiments and their very thorough author response. The new experiments indeed strengthen the paper and better outline the key contributions of the work. Some of my main concerns still do remain though: I am not fully certain that, given its mixed performance in the transfer tasks, the proposed framework will be very useful with more powerful transfer methods. This preliminary evidence has not fully convinced me on its future impact, and I am still not fully convinced by the choice of baselines, and whether to understand this preliminary empirical evidence as fertile ground for future enhancements in UCL. Some other clarifications were provided in the response, so I have increased my score, but as said - I still have reservations when it comes to the paper's empirical contributions and consequent impact.
=====


Overall, the paper provides some nice insights, especially treating the problem of cross-lingual transfer as largely a domain transfer problem, which allows it to use some domain adaptation (DA) machinery from prior work and apply it to the UCL problem. The empirical gains, although not huge, do demonstrate the alignment between the research hypothesis and empirical scores. The main strength of the paper imo is Section 2 which delves deeper into analysing factors that affect cross-lingual transfer. 

The ideas of having more invariant representations to improve cross-lingual transfer are not new, and they date back to the work on cross-lingual word embeddings (e.g., see the work of Dubossarsky et al. EMNLP 2020; or Patra et al., ICLR 2019, or Zheng et al., ACL 2019). Also the idea of treating cross-lingual transfer as a DA problem is also not novel - e.g., PhD thesis of Ruder discusses the similarity of the two problems in a nice and detailed way.

One of the major concerns I have with the current paper and its current presentation is that it is very difficult to discern between novelty of this work and what was done in prior research and simply reapplied to UCL. For instance, it seems a bit that this work is a (largely incremental) application of the prior (more fundamental) work of Tachet des Combes et al. (NeurIPS 2020) to a new (but highly similar) problem. For instance, the authors should clearly note in Section 3 if they bring any methodological contribution here or they just describe the previous method of Tachet des Combes et al.

In a similar vein, the paper can also be seen as an extension of the work from Keung et al., adding this mitigation of prior class shift into the mix, which yields slight improvements. 

I am also not convinced by the results and the entire evaluation protocol, detecting several potential problems and weaknesses here:
- Evaluation is conducted only on high-resource languages (from the NLP perspective), while the main promise of UCL is to improve on NLP tasks for lower-resource languages as done in plenty of contemporary NLP research. I wonder how IWDA would behave for such languages and whether it would bring any benefits. Doing zero-shot transfer from EN to DE really does not make much sense imo...
- Evaluation is conducted on two reasonably simple tasks: NER and MARC sentiment analysis (which is a classification task with a small number of classes) - a more detailed empirical analysis on other higher-level semantic tasks (e.g., XNLI, PAWS-X) is warranted for a clearer picture of the benefits of the proposed approach.
- The gains are typically not very high, and are close to none (when compared to the simple alignment method of Wu et al.) in the NER task. Moreover, performance on MARC sentiment is measured only against the vanilla zero-shot transfer and an older self-training (ST) baseline. What precludes the authors to again compare against Wu et al. or some other more recent techniques that go beyond the simplest zero-shot transfer protocol?
- The paper does not really put results into the perspective of plenty of related work that has been conducted in this area recently: e.g., check the leaderboards of XTREME and XGLUE for more sophisticated baselines: a true gain would be showing that applying IWDA along with these stronger models can yield further benefits. With the current set of experiments, I believe that the paper will have very limited impact.
- Along the same line, why is only mBERT evaluated? All concurrent work also provides evaluations with a stronger XLM-R Base model, and I also wonder whether these gains would remain with a larger and an even stronger XLM-R Large model. 

The paper also misses some very relevant related work, e.g., it seems very close to this work in its optics and design: https://arxiv.org/pdf/2011.11499.pdf 
- I would like to see a discussion regarding their (dis)similarity.
Some other papers that should have been briefly discussed and cited in this paper:
- https://arxiv.org/pdf/2104.07908.pdf
- https://arxiv.org/pdf/2005.00396.pdf (analysis of mBERT's multilinguality)
summary_of_the_review: The paper presents a DA-inspired view on (unsupervised) cross-lingual transfer, offering some insightful analyses, but it seems as an eclectic (mostly incremental) work, with insufficient and lacking empirical validations, incomplete baselines, and inadequate positioning against previous work in this area, which would negatively affect its potential impact and its overall contributions - more work and a stronger empirical foundation are needed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors provide substantial analyses on the cross-lingual transfer performance in the multilingual neural language models and reported that the performance is strongly correlated with representation invariance and negatively affected by distributional shift in class priors between data in the src/tgt languages. Based on these findings, the authors propose an unsupervised cross-lingual learning method, called importance-weighted domain adaptation (IWDA), where it performs feature alignment, prior shift estimation, and correction. The authors experimented on two different NLP tasks such as multilingual NER and multilingual sentiment analysis tasks, and experimentally showed the effectiveness. Besides that, they demonstrated that the proposed approach improves performance further, when combined with existing semi-supervised learning approaches.
main_review: The paper provides extensive analyses on cross-lingual transfer in the commonly used approaches that follow the pretrain-fine-tune strategy. In this strategy, the multilingual model finetuned on English task data is known to have zero-shot capability in the other languages, however, it is not well studied that, in unsupervised cross-lingual learning such as multilingual language model, what the role of shared representations is. Considering this, the paper gives a good start with substantial analyses that are helpful to understand what are the key factors to successful cross-lingual transfer learning.
Since XTREME benchmark provides a variety of multilingual NLP tasks, the experiment section could be extended with more results. The targeted languages in the current experiment are considered as a high-resource language. It would be better if the authors could move the remaining results into the main 0 pages and give more discussion on them. 
summary_of_the_review: The paper is mostly well organized, and provides extensive analyses and experimental results. They will be helpful to the studies in crosslingual transfer learning. However, some important descriptions on the model settings or experimental results are shown in Appendix, which makes the paper difficult to read. I would suggest to revisit and reorganize the sections for better readability. Regarding the experiments, since mBERT that the authors used as a pretrained model serves diverse language representations, they could provide deeper discussion, by moving Table 4 to the main 9 pages.
####
I read the responses from authors, and appreciate their response and showing more results. I am okay with accepting the paper, but keep the score 6. Because one concern might be that those results/analyses are mostly described in Appendix. The authors would need to reconstruct the manuscript by moving them into the main pages.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper first demonstrates the importance of feature invariance (language invariant representations) and class-prior invariance across languages on zero-shot cross-lingual performance. By analyzing the zero-shot performance on different languages on the MARC reviews and WikiANN NER tasks, and comparing against the class conditional distance between the source language (English) and target language feature representations, the authors illustrate the how high feature invariance results in better zero-shot performance. By also synthetically modifying the class prior for the target language, the authors demonstrate how increasing differences in class priors result in decreasing zero-shot cross-lingual transfer performance.

Building on their observations, the authors propose an approach that:
(i) Introduces an adversarial loss term to penalize distortion in average class conditional feature representations between the source and target languages.
(ii) Adds an importance weighting term to ensure the approach doesn't fail under class prior shifts.

Empirical studies demonstrate that the proposed approach improves significantly over the vanilla zero-shot model on both MARC sentiment analysis and NER tasks, also improving over self-training on sentiment analysis. Comparisons on synthetic datasets (sub-sampled from NER and MARC datasets) that enhance the class prior shift highlight the robustness of the approach under large class prior shifts where previous approaches fail.
main_review: Overall, the paper first presents insightful analysis that highlights the role of feature invariance and class prior shifts on the extent of zero-shot cross-lingual transfer. The insights from the analysis are then adapted to develop Importance-weighted Domain Adaptation for zero-shot crosslingual learning, resulting in improved performance on MARC sentiment analysis and WikiANN NER; with significantly improved robustness under class-prior shifts. 

Strengths:
1. Well written and easy to understand.
2. Insightful analysis that grounds the presented approach. 
3. Results from analysis with synthetically modified class prior distributions support the hypothesis that IWDA is improving robustness to class-prior shifts.

Weaknesses:
1. The paper evaluates on a limited set of tasks; having additional results on a wider range of tasks (for eg. additional tasks from the Xtreme benchmark) could significantly strengthen the results.

Suggestions / comments / questions:
1. Incorporating feature-invariance for domain adaptation has been studied for several NLP applications, including multilingual pre-training and zero-shot Neural Machine Translation. Discussion on several references is missing in the paper. [1, 2, 3]
2. Last paragraph on page 3: pratrained -> pre-trained
3. While the analysis in Figure 1 suggests that F1-score is directly correlated with conditional feature shift, is it possible there are other confounds in this analysis; for eg. the amount of pre-training data used for each of these languages?
4. Could the relatively weaker results on NER also be caused by challenges with aligning representations at a sub-word level across languages (with different levels of tokenization granularity across different languages)?

References:
[1] Explicit Alignment Objectives for Multilingual Bidirectional Encoders, Hu et al.
[2] The Missing Ingredient in Zero-Shot Neural Machine Translation, Arivazhagan et al.
[3] Improving Zero-shot Translation with Language-Independent Constraints, Pham et al.
summary_of_the_review: Overall, the paper first presents insightful analysis that highlights the role of feature invariance and class prior shifts on the extent of zero-shot cross-lingual transfer. The insights from the analysis are then adapted to develop Importance-weighted Domain Adaptation for zero-shot crosslingual learning, resulting in improved performance on MARC sentiment analysis and WikiANN NER; with significantly improved robustness under class-prior shifts. 

The paper produces some valuable insights and develops a well-grounded approach that improves the robustness of zero-shot crosslingual learning. However, the empirical results are limited to just two (relatively) small scale tasks. Having additional results on a wider range of tasks (for eg. additional tasks from the Xtreme benchmark) could significantly strengthen the results.

Given the thorough analysis, but limited range of tasks, I am leaning towards acceptance. If authors include additional empirical results I would be willing to update my recommendation to strong accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates a method for improving the cross-lingual transfer of pretrained multilingual models. The paper first empirically analyzed the influence of representation invariance and distributional class shift. Then, the paper proposed a method to improve the representation invariance and correcting the class shift. Experiments showed its superiority under large prior shifts.
main_review: Strengths:

The proposed method is well motivated based the empirical analysis. 

Experiments show effectiveness of the proposed method.


Weaknesses:

The experimental results are not convincing enough. 1) Only two of the tasks are considered. 2) only mBERT is tested. Would the same observations and conclusions hold when other widely-used pretrained models such as XLM are used. 3) According to Table 1, the proposed method is comparable with the existing work (Wu et al., 2020) on average. 4) There’s no baselines in Table 2. 5) Seemingly the proposed method is much more time consuming during training than prior works. 

summary_of_the_review: The proposed method is well motivated given that the empirical analysis reveal the influence of representation invariance and class shift. Experiments show the effectiveness of the proposed method under large class shift. However, the experimental results are not convincing enough and the paper can be improved by conducting more experiments and analysis.
review_title: null

Meta-review: The paper presents a domain adaptation approach based on the importance weighting for unsupervised cross-lingual learning. The paper first analyzes factors that affect cross-lingual transfer and finds that the cross-lingual transfer performance is strongly correlated with feature representation alignments as well as the distributional shift in class priors between the source and the target. Then the paper designs an approach based on the observations.

Pros: 
+ The paper is well written and the proposed approach is well motivated. 
+ The analysis about which factors affect cross-lingual transfer is interesting and provides some great insight. 

Cons: 
- As the reviewer pointed out, the experiments for verifying the proposed approach are relatively weak.

Overall, the paper presents nice insights to connect cross-lingual transfer with domain adaptation. All reviewers lean to accept the paper and I also found the paper is in general interesting.
Decision: Accept (Poster)