Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:
The paper presents HICTL which enables models to learn sentence level representations and uses contrastive learning to force better language agnostic representations for large multilingual encoders. They obtain significant gains on the XTREME benchmark and also on standard MT benchmarks.

Reasons for score:
I score this paper a 6. The constrastive losses introduced in the paper are interesting. The sentence-level and word-level CTL definitely seem to have an impact on the downstream performance. The improvements on XTREME over XLM-R is pretty strong. However, the improvements in MT are not that convincing. I would have also liked to see more ablations. Finally, the authors initialize from XLM-R and fine-tune on 15 languages with both monolingual and parallel data. I would have liked to see this number (i.e. number of languages) be much larger.

Cons:
- Why did you only choose to fine-tune on the 15 languages from XNLI? 
- Can you present the breakdown of results in different tasks by language as an appendix? Specifically, I would like to see if the improvements are only coming in the 15 languages you fine-tune on or on others as well. Do you have an answer for this?
- I would like to see ablations for XTREME where you only did sentence-CTL or only did word-CTL. Other interesting ablations without parallel data would have also added more value to the paper. Ablations of the amount of data used would also be interesting.
- Writing could have been clearer. I found a lot of grammar mistakes in the paper which need to be corrected.
- Why did you not make an official submission to the XTREME leaderboard? That would have been a more fair evaluation of your system.


Minor comments:
- Word-level CTL: change "are in two folds" to "are two fold".
- Table 2: It would be good to have an average column which will make it easier for the reader.
review_title: HICTL presents some interesting contrastive losses for representation learning

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a pre-trained language model variant which extends XLM-R (multilingual masked model) with two new objectives. The main difference to most other models is that the new losses are contrastive losses (however, as pointed out by the authors, other contrastive losses had been used before in e.g. ELECTRA). The first additional loss is a sentence-level one - where a [CLS] token is trained to be close to the positive sample, the paired sentence, with other sentences as negative samples. The same is done at word level, where the bag of words constructed from two sentences becomes the set of positive samples and other vocabulary words are negative samples. 
Contrastive losses are promising and the paper shows positive results when adding them to the previously proposed XLM-R model. The review of previous work is thorough and very helpful to place the work proposed in the existing literature. However I had difficulties understanding the impact of the changes proposed and disentangling the different factors that may have led to the results. At the end of reading this paper, I am not sure if implementing what the authors proposed, versus other variations of existing models, would have given the same improvements: While these improvements can be seen across many data sets, they are often modest. The proposal does not offer any other advantages, such as computational efficiency. 
For the NMT experiments, additional experiments on En-Es and En-Ro (to follow experiments in Zhu et al 2020), and/or back-translation experiments would have made the impact of the method clearer. Given that the main contribution of the paper is empirical (none of the ideas are new), better and more comprehensive experimental results would have strengthened this work.
The following are clarification questions/comments: 
- The query and key terminology used in section 2 is confusing: why not use negative/positive sample notation from the Saunshi et al, 2019 and  Oord et al, 2018 papers? Section 3.1 introduces r_x, which is yet another notation for the query q. 
- Figure 1: please clarify the notation used in the caption (e.g. the set B is defined only later, similarly n, m, V). 
- The losses in equations (2) and (3) are symmetric: if the data pairs are symmetric, which seems to be the case, why distinguish between queries and keys at all and define two identical, but symmetric losses? 
- First paragraph in Word-Level CTL in Section 3.1: This should be rephrased in order to clarify the motivation for the word level loss. 
- I couldn't find details regarding the negative samples for the sentence loss: no of negative samples, how are they obtained, etc.

review_title: extension of cross lingual pre-trained models with sentence- and word-level contrastive losses

final_score: rating: 7: Good paper, accept
review_content: Summary

The work applies and adjusts contrastive learning in the subject area of pre-training language models. The work first identifies the challenges with the current landscape of Masked Language Models with limits to learning sentence-level representations and semantic alignments in sentences of different languages. To take care of these gaps, the authors propose using HCTL as an approach that can learn more universal representations for sentences across different languages. The work builds on top of the BERT models, with the adjusted contrastive learning objective goal.

Strengths

The paper is well written. The authors work to clearly identify shortcomings in the current literature. The identification of contrastive learning as a possible approach to learn better representations. 

Questions

- Using the [CLS} token has been one of the ways to capture the sentence embedding for BERT, correct? I do understand that there are other ways to try to extract sentence embedding from pre-trained BERT, but your approach is not new in this sense?

- I am not sure I missed it, but I do not have an objective view on the computational overhead (especially on having to sample the right-hand side of the non-similar keys). Simply, how much more time do I spend using this type of training as compared to the prior approaches? 

- With the prior question, the results, show consistent improvements (your biggest strength along with the representation itself being comparable) but are a slight improvement in individual metrics (a point or two). Given the goal of developing universal cross-lingual representations, why is it that if we do learn these better representations are we not doing Very Very well?

- Might it be that the prior models, already have these representations already embedded in their models and your work just extracted them? Could there be another approach to XML-R for example, that extracts these representations?

- Thank you for including different sizes of corpora and also some low-resource languages. Kiswahili is the smallest parallel data that you have, and I wondered what would happen with a smaller language? For low-resource languages, it would be insightful to also have examples of where the failures happen and why.

- Please expand the future 3 captions. it is hard to understand what it represents from the caption. Just make sure the caption is completely descriptive. The figure is also very small and hard to read. 

I think the contrastive learning approach is very interesting for this use-case and made my reading and evaluation of this paper more interesting. I look forward to the responses from the authors. 
review_title: Universal representations, at what cost?

Meta-review: This paper presents two new representation learning tasks (losses) based on contrastive learning that---when combined with a language modeling loss---result in a better multilingual model. Experiments on machine translation and XTREME demonstrate the benefit of the proposed method compared to strong baselines.

I think this is an interesting paper that advances multilingual representation learning. The authors have incorporated many suggestions from the reviewers to improve the paper during the rebuttal period. I recommend to accept the paper, but also strongly suggest the authors to make an official submission to XTREME to validate their results.
Decision: Accept (Poster)