Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes cross-lingual mixup, a technique which performs manifold mixup of source and target sequences. This technique also includes mixup ratio which factors in MT quality and scheduled sampling which deals with exposure bias. Experiments on 3 task types from the XTREME benchmark show that x-mixup leads to significant gains over translate-train baselines and other strong submissions to the leaderboard. The authors also present some analyses with a few ablations.
main_review: 
Strengths:
- x-mixup is a clear idea and seems straightforward to include on top of existing models like mBERT or XLM-R.
- The improvements over strong baselines on multiple XTREME tasks are convincing.

Weaknesses:
- One big downside I see is that the authors use mixup at different layers for different tasks. What happens if a single setting is picked and used for all tasks? How does the comparison look?
- The regularization term to account for MT quality should be further tested by controlling the quality of MT and seeing if it's robust to it. Otherwise it's not clear how valuable it is.
- Another downside is that this method relies on translation data (both translate-train and translate-test) during training. I would like to see an ablation where only translate-train data is used or only translate-test data is used. This would be fair comparison to the XTREME submissions which only use translate-train and not translate-test.


Detailed comments:
- The related work section is a bit weak.
- Some missing citations:
     - there's a whole body of work on using translation data for cross-lingual learning: please cite
                 VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
                 Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation
                 Explicit Alignment Objectives for Multilingual Bidirectional Encoders
                 nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?
- Footnote #9 is a very important detail. It should be moved to the main paper.
- Some minor grammar mistakes but overall clear paper.
- Please have a discussion on how this be applied to the retrieval tasks in XTREME. Can this approach be moved to the pre-training set up by leveraging some parallel data?
- Please also discuss the impact on training time by adding x-mixup compared to translate-train. 


summary_of_the_review: The authors propose x-mixup a technique which leverages parallel data and forces cross-lingual representations to be aligned. The proposed method leverages translation data (translate-train and translate-test) during training. This leads to improvements in several XTREME downstream tasks over strong baselines. The authors perform some ablations which showcase the importance of the different components in the proposed method.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a technique which the authors call *cross-lingual manifold mixup* or *X-mixup*. The approach has been inspired by *mixup* (Zhang et al., 2018) and *Filter* (Fang et al., 2020). The general idea is to combine the hidden representations corresponding to the source-language input with the hidden representations corresponding to the target-language input (in a smarter way than, e.g., simply concatenating them). The authors perform multiple experiments using two pretrained models (mBERT and XLM-R) and show that their proposed approach improves over multiple strong baselines.
main_review: This paper proposes a technique called *cross-lingual manifold mixup* or *X-mixup*, which has been inspired by *mixup* (Zhang et al., 2018) and *Filter* (Fang et al., 2020). The general idea is to combine the hidden representations corresponding to the source-language input with the hidden representations corresponding to the target-language input (in a smarter way than, e.g., simply concatenating them). The authors combine this with 1) a specific mixup ratio based on translation quality and 2) scheduled sampling. They perform multiple experiments using two pretrained models (mBERT and XLM-R) and show that their proposed approach improves over multiple strong baselines.

Strengths: 
- The approach works: the authors show improved results over multiple strong baselines.
- The paper is well written and the authors perform a nice analysis/investigation of smaller research questions in Section 6.

Weaknesses:
- I am unsure if framing *X-mixup* as a cross-lingual transfer version of *mixup* is the best framing. I only read the *mixup* paper superficially, but it seems to me that that paper's goal is to generate additional training examples by averaging (independent) pairs for both *x* and *y* to make the model more robust. In contrast, what the authors do here is more closely related to simply concatenating source and target sentence in the input. 
- The experiment section isn't very clean. For example, it is unclear why different methods are used on top of mBERT and XLM-R, respectively (see Table 3).
summary_of_the_review: The paper presents *X-mixup*, a method for cross-lingual transfer based on translate-train, which improves performance over multiple strong baselines. However, I have doubts regarding the framing of the paper as well as regarding the conducted experiments. Thus, I think the paper should be revised before being published.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes X-Mixup, a model that considers the source languages and target languages together for cross-lingual transfer. The designed model takes a pair of sentences (or the translated sentences) in a source language and a target language as the input and computes the cross-attention between them. The author also adds the consistency loss to encourage similar predictions for the source and the target. They also propose scheduled sampling to handle the different distributions during training and testing. The experimental results and the ablation studies show promising performances for cross-lingual transfer.
main_review: Overall, I like this idea that considers the source languages and the target languages at the same time to encourage the model to learn more similar representations for different languages. Here are some suggestions and questions:
- Figure 4 only contains a few languages and they are mostly high-resource languages. It would be more convincing if the authors can show results of more languages, especially low-resource languages.
- What if there are multiple source languages? Do we randomly sample one of them during training and testing?
- It seems like we need to know what target languages we will care about in advance. What if there are some new target languages? Some analysis on this can be quite interesting. For example, only consider part of target languages during training and see if those languages help the model learn good representations for unseen languages as well.
- Why the consistency loss consists of two terms: MSE and KL. Any ablations on this?
- Does more CKA score improvement lead to more task performance improvement? Any relations between them?

summary_of_the_review: The proposed idea is interesting. They show good and promising experimental results and ablation studies. Some questions need to be clarified and some more analysis is suggested.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The work applies and adjusts manifold mixup in the subject area of zero-shot cross-lingual transfer. The work first identifies the challenges with the current landscape of cross-lingual transfer methods, in which the performance of the target languages still lags for far behind the source language. The paper make the gap associated with cross-lingual representation discrepancy. To take care this, the authors propose a X-Mixup method that interpolates the representations between the source and target languages, and give this compromised representation for target language prediction. Compared to strong baselines, the work obtains significant gains on the XTREME benchmark.
main_review: Strengths

1. The paper is well written and the motivation is clear. The paper first conducts some priori experiments to strength the motivation, and transforms the performance gap to the representation discrepancy through the visiualization analysis.
2. The manifold mixup seems to be valuable for bridging the cross-lingual representation discrepancy across languages.
3. The improvements on XTREME over XLM-R is pretty strong. 

Cons:

1. What does $\tilde{\mathcal{D}}_{S}^{Train}$ means? I didn't find its definition in the "Notations" part.
2. I would like to see ablations for the mixup ratio $\lambda$ where you use randomly sampled value for $\lambda$ or compute a gate using $h_{T|S}$ and $h_{T}$, rather depend on $A$. 
3. Why you need the consistency loss in the training objective? What does $P_S$ and $P_T$ exactly mean? How to deal with the KL collapse issue?
4. What does $\tilde{D}_S$ (appeared in the Scheduled Sampling and Inference parts) means? 
5. The scheduled sampling strategy has a great impact on training efficiency. Can you provide the training time of the X-Mixup and translate-train respectively?
summary_of_the_review: I score this paper a 6. I think the manifold mixup is very interesting for this use-case and made my reading and evaluation of this paper more interesting. I look forward to the responses from the authors.
review_title: null

Meta-review: This paper proposes X-Mixup, a model that considers the source languages and target languages together for cross-lingual transfer. The designed model takes a pair of sentences (or the translated sentences) in a source language and a target language as the input and computes the cross-attention between them.  

The empirical results are convincing. Reviewers think this paper is well-written and the idea is interesting.
Decision: Accept (Poster)