Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposed a new data augmentation framework for low-resourse (and zero resource) cross-lingual task adaptation by combining several methods (entropy regluarized training, self-training). The authors conducted extensive experiments on three cross-lingual tasks, demonstrating the effectiveness of XLA. In addition, the authors compared different choices in the XLA distilation stage and claimed that the gain from XLA is beyond the model ensemble effect.

According to the experiment results, the XLA framework has a remarkable gain over previous methods. However, it's not clear which component actually contributed to this gain. An thorough ablation study on different model component will help clarify this concern.

The writing style is a bit redundant and confusing. For instance, the author use "model distillation" to define the label selection procedure, yet, this term is another technique in machine learning literature.

Lastly from a methodology perspective, I think the algorithm seems like a combination of existing methods and its novelty is incremental to me. I hope the authors can edit the paper more carefully with a clarification on their novelty.
review_title: Good experiment results yet limited novelty

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors present an unsupervised data augmentation framework for cross-lingual NLP. Their method, called XLA, combines self-learning with co-learning and filtering. They generate additional synthetic examples by replacing words with predictions from pretrained multilingual masked LM (taken from XLM Conneau 2020). The key contribution in this paper is how they get reliable labels by simultaneously co-training three student models and using them to filter examples for training each other to avoid confirmation bias.

Strengths: 
The authors present three representative cross-lingual transfer tasks: XNER, XNLI, and PAWS-X with strong results for all three. For XNER XLA gets SoTA in all languages, and they show higher scores for structurally dissimilar and low-resource languages.

Weaknesses:
The method is quite elaborate requiring three epochs, with different training sets at each step, three different models for labelling, and then a combination of co-teaching, co-distillation and "co-guessing". I think that the justification for all these design decisions is not entirely convincing and it feels slightly over-engineered. How robust is this framework? How generalisable? 

I also think that the explanation of the architecture is not as clear as it could be. The text in the intro to Section 3 is augmented with an Algorithm and a diagram in the appendix. Each of these explanations are presented separately, and neither the algorithm not the figure are fully explained. I think the algorithm should go in the appendix and be fully explained. I think the figure should go in the main article and be referred to in the text when describing the model at a high level.

Even with these limitations, I think this paper makes enough of a contribution to warrant being accepted. Hopefully some of my concerns can be addressed before the camera ready paper. 

Table 1: Ensemble of what three models? 

Why not report the supervised results ie. Like the Conneau et al 2020 paper? I know the main point is the zero-shot performance, but it is an interesting data point anyway especially as you have already included the baseline results for this.

Figure 3: Needs to be described - how did you plot this graph? And how is the reader supposed to see that the right hand plot is more robust to OOD data??

The vicinal risk minimization principle takes up a lot of space and adds quite a bit of complexity to the description of the model. I think you could leave it out and improve the paper - this kind of data augmentation is being done already: 
	- Sosuke Kobayashi. 2018. Contextual augmentation:Data augmentation by words with paradigmatic re-lations. InProceedings of the 2018 Conference ofthe North American Chapter of the Association forComputational Linguistics: Human Language Tech-nologies, Volume 2 (Short Papers), pages 452–457,New Orleans, Louisiana. Association for Computa-tional Linguistics

The distillation by clustering method is not fully leveraged as extra information is not present in the experiments. It would be a nice extra contribution to the paper if features such as sentence length were indeed included. As it is you might as well leave clustering out of the paper and use the space for something else. 

Your appendix is extremely large. Please consider what is truly important and remove the rest. 

------

Response to author's replies:
I am impressed by the detailed response and the changes they have made to their paper and I am happy for this paper to be accepted. I still feel like the XLA framework is quite involved and it would have been good to understand which components of this framework are crucial to its success which is why I do not increase my score. 

review_title: Important problem, strong results

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper improves 1-to-1 cross-lingual NLP by proposing a new data augmentation method. Overall, the paper is clearly written. The proposed method is intuitive to understand and is novel in crafting augmentation texts using masked language model of BERT and relabeling these vicinal examples.  

Experiments on cross-lingual NER, NLI and paraphrase detection tasks demonstrates the effectiveness of XLA, outperforming  XLM-R methods by a large margin and setting up new baselines for future cross-lingual models.

However, as the paper focuses on data augmentation in NLP, it should discuss about data augmentation methods in mono-lingual (e.g. English) and cross-lingual NLP (if there exists), pointing out their advantages/disadvantages under the paper's setting.

As the paper crafts examples with and without vicinity separately for the target language, which part contributes more to the improvement of XLA over XLM-R? 

Pros:
- The augmentation algorithm is clear and results are promising.

Cons:
- Lacking comparisons and discussions with other data augmentation methods in NLP.
- Hard to infer the contribution of $D_t^\prime$, $\tilde{D_t}$ and $\tilde{D_s}$ separately from Fig.1.
review_title: A good data augmentation (DA) method for cross-lingual NLP, however, lacks enough discussions and comparisons with other existing DA methods in NLP.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: The paper  presents a data augmentation framework for zero-shot cross-lingual transfer learning. The framework uses different types of data (labeled source data, unlabeled source data, automatically generated augmented data) for training a model for the target language. Experiments are conducted on three different tasks: Named Entity Recognition (NER), Natural Language Inference (NLI) and paraphrase identification (PAWS). The approach combines multiple components together namely self-training, augmented sentence generation and confidence penalty.

Data Augmentation:
The paper states that data augmentation has been successfully used in images but not so much in  text (excluding back-translation). In fact, replacing creating augmenting text by masking and replacing word has been used in NLP both before and after pre-trained LMs such as BERT, etc.

- Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations. Kobayshi. NAACL 2018
- Conditional BERT Contextual Augmentation. Wu et al. 2018.
- AUG-BERT: An Efficient Data Augmentation Algorithm for Text Classification. Shi et al. CSPS 2019
- A lexical and frame-semantic embedding based data augmentation approach to automatic categorization .... Wang and Yang. EMNLP 2015.

The paper argues that generating new samples for data augmentation using the vicinity distribution of the source and target samples is better than back-translation since you cannot transfer the labels in sequence tagging tasks with back translation. However, similar problems would occur with vicinity distributions based augmentation since even changing a single word may result in changing the meaning of the sentence and hence the label (as the paper argues in Section 3.3.) . Given that, it would be useful to see more discussion/experiments comparing the two types of augmentation strategies especially that the self-training step can leverage augmented data without labels

Experiments:
The paper provide a lot of interesting ablation and analysis. However one of the key questions that I couldn't get an answer to is what is the value of each source of data and could they be used in any different way. For example, what happens if we only do data augmentation for the source only, or the target only or by using translation or back-translation, etc. 

On a related note, it looks like the specific order in which the datasets are used is important as shown in the experiments but it is not very clear what is the intuition behind that choice and whether other choices were considered or tried.

Other questions:

What is the benefit of Successive cross (vs. successive max)?

Wikiann is much bigger and has covers 40 languages. any reason why only 3 languages were considered from Wikiann and other languages from CoNLL?

 -----
Edited after authors responses:
I would like to thank the authors for the detailed response and the changes they have made to the paper. 
 
- Regarding contextual data augmentation (Kobayshi et al., Wu et al., etc.): Thanks for pointing out that these method use the labeled data to finetune the LM to make sure that words are replaced with other "label-compatible" words. Note that the comparison is not intended to necessarily show that the proposed method outperforms these baselines. Rather it is intended to guide the reader into making a decision about which method is more appropriate for which problem. If the findings are that the performance is comparable but one method will eliminate the additional finetuning step, that would be a useful finding to share. Also , it is not clear that  these methods would require labeled data in the target language for the finetuning or not.  For example, can the source labeled data be used for the finetuning step?

- Regarding translation: Cross-Lingual transfer via Machine Translation does suffer from the label transfer problem for sequence tagging tasks (transferring labels for sentence-level tasks is straightforward). However, there has been several methods to address this in the literature by using unsupervised word alignment  (e.g., Yarowsky et al., 2001; Ni et al., 2017) or attention weights from NMT models (Schuster et al., 2019), heuristic approaches (e.g., Ehrmann et al., 2011) or co-learning alignment and tagging (Xu et al., 2020).

A detailed comparison and discussion of the trade-off between the performance of each method and the resources they require would make the paper much stronger 
review_title: Official Blind Review #2 

Meta-review: The paper clearly has merits, presenting a reasonable approach to zero-shot cross-lingual learning with good results, but with limited novelty, perhaps. I am sympathetic to the departure from XTREME on NER, agreeing with the authors that using CoNLL data is more interesting than WikiANN. 

The post-rebuttal discussion centered on novelty and baselining - and specifically, whether other approaches to unsupervised data augmentation exist that should be used to baseline the proposed work. The authors argued that most of the approaches mentioned by the reviewers were in some way supervised. I personally think the confusion is a result of the paper being somewhat poorly framed:

Reviewer 2, for example, suggests a bunch of baselines. Some of these require gold labels for supervised fine-tuning to condition the MLM, but this seems like a trivial difference, which is orthogonal to using the augmentation strategies as baselines? Also, other papers have been presented that do not require gold labels, e.g. https://www.aclweb.org/anthology/D18-1100.pdf

Also, on the discussion of Täckström et al. (2012): Older approaches relying on distributional clusters *are* in fact data augmentation methods. Training on augmented data with words replaced is, in the limit, equivalent to training with clusters, when replacement words are sampled from clusters. Others have in the past proposed to use FSAs or clusters induced from static embeddings.

What the authors suggest is a form of co-training procedure, so similarly, semi-supervised algorithms - e.g., tri-training - could have been used as baselines.  

In sum, I think the sentiment shared across the reviewers is that the results are largely unsurprising, and could likely be obtained in different ways, including jointly training with a target language modeling objective, tri-training, etc. Finally, I agree with Reviewer 2 that a “detailed comparison and discussion of the trade-off” between the different approaches to data augmentation, even beyond what’s apples-to-apples, would benefit the paper. Maybe there's other advantages to the proposed approach over other baselines (effectiveness, robustness)? 

Decision: Reject