Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents Conditional Masked Language Modeling (CMLM), which integrates sentence representation learning into MLM training by conditioning on the encoded vectors of adjacent sentences. It is shown that the English CMLM model achieves strong performance on SentEval, and outperforms models learned using (semi-)supervised signals. It is also found that a multilingual CMLM model co-trained with bitext retrieval (BR) and natural language inference (NLI) tasks outperforms the previous state-of-the-art multilingual models by a large margin. The paper further proposes a principle component based approach to remove the language identifying information from the representation while still retaining sentence semantics.

-Strengths

Learning sentence representations on large scale unlabeled corpora is an important research problem. This paper presents a heavily empirical study, with a series of experiments to evaluate the proposed sentence representation learning method. Multilingual experiments are conducted, with interesting results on language agnostic.

The proposed method, as shown in Figure 1, is somewhat new.

-Weaknesses

The study is mainly empirical.
The authors should provide more details about the three-layer neural network as the projection P (·).
Another concern is that the contribution of this paper to research community may be weak, if the code is not released and the results are not easily reproduced.

--------update after reading the response-----------

Thanks for the authors' response. Mainly empirical and limited in methodology novelty. So I tend to keep the score.
review_title: The results are good but mainly empirical

final_score: rating: 7: Good paper, accept
review_content: The authors present conditional masked language modeling (CMLM), a new method for unsupervised pretraining, in which the skip-thought notion of conditioning on neighboring sentences is adopted for masked language modeling. The upshot of the proposed approach is that it generates single sentence embeddings that perform competitively on SentEval. In the multilingual setting, the authors combine their CMLM method with a bitext retrieval objective (selecting a sentence’s translation from the other sentences of the language in the batch) that increases performance on a version of the SentEval tasks translated into 14 other languages. In their analysis, the authors make further claims about multilingual embeddings capturing language ID information in their first principle components, a conclusion somewhat substantiated by their results. The authors provide a small amount of ablation experiments for experimental/model design choices.

The underlying idea is worth pursuing, the execution and description could be improved, people will be interested in the results that are present (but then have questions).


Overall

Why only a subset of SentEval for the English experiments (3.2) but then the full SentEval in the multilingual XEval experiment (4.5.1)?  Especially if you are trying to make a single-sentence encoder but then evaluating on SICK-E instead of SICK-R, which is arguably a more applicable eval set. 

Why no XLM-R in the amazon reviews analysis (4.5.2)?

Sec 1.
Fig 1: box of chocolate*s*


Sec 2.
--“conditional”, →  “conditional,”

--no quantitative comparison of using max vs mean pool vs CLS embedding

--first sentence is feed → first sentence is fed

--three-layer neural network → three-layer MLP

--refer to using the same set of encoder weights for different inputs as siamese networks, as done in the sentence-bert paper https://en.wikipedia.org/wiki/Siamese_neural_network

-- v_d is used but not defined


Sec 3.
--Skip-thought originally used a sentence to predict the generation of both the preceding and succeeding sentences. This is functionally equivalent to your flip-flopping the order of the consecutive sequences. I would make the point that these steps are equivalent.  Note that this is also not necessarily making the task “more challenging” (and moreover I am not sure why “more challenging” equates with “better pretraining method for language understanding” -- and an ablation of this step is not included to show that it is in fact necessary and useful).

-- similarly, no analysis of masking rate, nor explanation for why ‘more challenging’ is better.

-- “We explore two transformer configurations, base and large, same as in the original BERT paper.” – fragmented

-- The number of projections N = 15. – fragmented

-- SentBERT → SentenceBERT or SBERT

-- On the specific subset of SentEval tasks you’ve selected, the majority of the performance discrepancy is in the SICK-E  task--otherwise, the overall #’s are rather interchangeable. How does this change if you add in the rest of the SentEval tasks, and why were they omitted? Analysis/exploration for why you get such a performance boost only on SICK-E would also be useful.

-- “the length … set to be 256 tokens”: please clarify whether the “length” refers to the maximum length, or each sentence is a fixed-length chunk consisting of 256 tokens

--typo:  “we also exploring”


Sec 4.
--If your introduced bitext retrieval objective uses batch size, experiments comparing the effect of batch size is necessary. 

-- Please specify the value of the margin m being used in the experiments

--Choice of number of projections is also not motivated (and in fact contradicted by the ablation  experiment finding that 15 is better)

-- the motivation and contribution for XEVAL are great-- the explanation of the dataset is lacking. What translation API was used? How was the XEVAL score computed for each language? Is it the full set of SentEval downstream tasks?

-- cite precedent for using the concatenation of u,v, u-v and u*v. (or show its effect via ablation)

-- BR  → CMLM+BR configuration not evaluated

-- choice of different training step #’s in each configuration is not particularly motivated.

-- “after exploring options including [CLS] representations and max pooling.” what was the performance drop?

-- typo: “has a significant upon mBERT”


Sec 5.
--It is not clear that you can make the claim that the first PC *only* encodes language-identification information?

--I assume Figure 3 is 2-dimensional TSNE (needs a cite), which comes with its own set of caveats as a visualization tool. Quantitative clustering analysis such as silhouette coefficient might be more appropriate than a plot.  If Figure 3 is not t-SNE, please specify the meaning of X and Y axes.

-- did not try higher than n=15 projections but claimed it was optimal

-- The description of the “skip” ablation is unclear: please clarify what is meant by “concatenated with the sequence outputs of s2”.

-- typos: “removing the first principal component … effectively eliminate”, “for both two models”, “representations … generally exhibits”
review_title: SkipThought + MLM: worth exploring, more details would improve the article

final_score: rating: 4: Ok but not good enough - rejection
review_content: I appreciate the response from the authors to my review, as well as to others. 

My concerns on the intuition are most not solved. Although in this DNN dominating era, we cannot expect the explainability as we had before, I still believe that a solid work should be grounded on a reasonable basis, which could be in a high level, such as BERT and SBERT. Let's refer to the example given in the model architecture. The projection of the sentence vector of "Life is a box of chocolates" is left-concatenated with the masked embeddings of the second sentence. This operation is very much lacking in intuition, how come the projection of a sentence representation can be concatenated with the embeddings? In addition, "The second encoder shares the same weights with the encoder used to embed s1", considering their inputs are very different, weight sharing for the two encoders are also problematic.

Another point I just noticed, although the authors claimed that their model is better than SBERT, and did a comparison with SBERT-large, they did not compare with SBERT-base, which makes the conclusion unreliable.

---------------------------------------------------------------------------------------------------------------------------
This paper proposes a method called "Conditional Masked Language Model" for unsupervised sentence representation learning. The method involves two-sentence encoder, where one sentence depends on the encoded sentence level representation of the adjacent sentence. The experimental results are good overall, as the proposed method tends to give the best results across monolingual and multilingual benchmark datasets.

There are still some concerns about the novelty of this paper.  First, I think the explanations for the intuition of the proposed model can be clarified, especially in the introduction section. Second, the baselines used for comparison are not complete, which makes me concern about the effectiveness of the model. The proposed model is the combination of Skip-Thought (Kiros et al., 2015) and BERT masked LM (Devlin et al., 2019).  Their experimental results show a detailed comparison of BERT but ignore much about the Skip-Thought. Although the authors mentioned the results of the Skip-Thought model on the SentEval benchmark, the encoder used in the Skip-Thought (Kiros et al., 2015) is RNN while the author used the Transformer Encoder.  I would appreciate a better and fair comparison of the Skip-Thought model by using same transformer encoder and same training corpus.

I am not sure why you used the concatenation when you do the masked LM.  Are there any other ways to do that?  It can be more convincing to see some analysis or results here. Additionally, there is another work titled “DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations”, which also focuses on unsupervised sentence representation learning.  Although it is from arxiv, it would be nice that the authors can mention this work.

It seems good that the authors performed many experiments over many different datasets across monolingual and multilingual.  The exploration of the same language bias of the learned representations is also very interesting.

To summarize, the paper is a bit limited in terms of technical contribution, particularly considering that there is not a good intuition explanation, but some analysis in this paper looks good.

review_title: A reasonable work, but a bit limited in terms of technical contribution, particularly considering that there is not a good intuition explanation.

final_score: rating: 5: Marginally below acceptance threshold
review_content: ----------

I appreciate the response from authors and the additional experiments. I do think the semantic search task adds value to the paper. However, the paper continues to be centered around the SentEval benchmark results. While SentEval is a useful benchmark to evaluate sentence representations, it doesn't reflect well how these representations will be used in practice. A fine-tuned BERT model will likely perform strongly on these tasks. The paper would be far more compelling if the authors can provide strong evidence that the sentence embeddings do well on tasks where using a BERT model is either less effective due to performance or computational reasons. 

I prefer to keep my score. 

------------

This work proposes a self-supervised training objective called CMLM (conditional masked language model) for learning sentence representations. An encoder produces multiple fixed length representations of a given sentence and a decoder reconstructs the adjacent sentence given it’s masked version and the encoded representations. CMLM performs well on the SentEval benchmark. CMLM is further extended to the multilingual setting via bi-text retrieval contrastive training and training on NLI data. The multilingual version is shown to work well on multiple translated versions of the SentEval benchmark (SentEval data translated into other languages using an off the shelf translation system) and Amazon reviews (sentiment classification).   


Pros
* This work addresses the important problem of (unsupervised) sentence representation learning. Extracting fixed-length sentence representations from popular language model based encoders is a non-trivial problem and this work attempts to provide a solution.
* Experiments go beyond the standard English setting and evaluate sentence representations in the multilingual setting as well. 
* Interesting modeling approaches.

Cons
* Experiments are weak. It is unclear to what extent the tasks + evaluation protocol considered here are reflective of language understanding. I don’t think strong baselines were considered. Some of the evaluation benchmarks considered seem arbitrary.
* Model is largely based on prior work. The main contribution is not clear. There are many settings considered in the paper and it is unclear if the proposed contributions are truly significant due to weak baselines are differences in data used for training different methods.

There are several issues with the experimental setup.
* Evaluation protocol: It is unclear if the evaluation protocol considered is measuring language understanding capability well. Representations from the encoders are held fixed and linear classifiers are trained on top of these fixed representations on downstream tasks using labelled data. To me, this is not a setting that demands sentence vectors. It only shows that the sentence vectors capture useful features. I would suggest focusing on a setting where the advantage of the sentence vectors can be demonstrated such as a retrieval problem.
* Baselines: It is unfair to compare the proposed method against baselines like BERT which are not designed to produce fixed length encodings. 
* Data used for pre-training: It is difficult to gauge how good the method is in comparison to other models due to differences in the data used for pre-training. Ideally, there should be a table comparing models that use the exact same resources. In Table 1, although BERT-base/large is trained on the same data, it is not a strong baseline since mean pooled representations from the encoder are treated as a sentence representation. Ideally, the model should be compared against a skip-thought baseline or an unsupervised sentence representation learning method that uses the same resources for training. 

I would have expected the authors to evaluate multilingual representations on existing benchmarks as well. I don’t find the proposed benchmark XEVAL very convincing. Claims would have been stronger if authors had also included results on existing benchmarks. 

The authors need to make it clear exactly what resources are used for training each method. 

Presentation can be improved, especially the organization of the paper. It is difficult to follow the paper and identify the main contributions in the current presentation. 

The paper touches upon several things - conditional masked language modeling, bitext retrieval, NLI training, language agnosticism, etc, and I find the paper quite incoherent. I suggest the authors make a focused contribution and provide strong experimental evidence to support that contribution. Right now there’s too many things which makes it hard to make sense of the paper as whole.

While the approaches considered in the paper have some merit, the significance of this work is unclear due to issues in the evaluation. 

review_title: Low significance, issues in experiments and setup

Meta-review: This paper proposes a Conditional Masked Language Modeling (CMLM) method to enhance the MLM by conditioning on the contextual information. 

All of the reviewers think the results are good. However, the reviewers also think the intuition and experiments are not so convincing. The responses and revisions still not satisfy all the reviewers' major concern.
Decision: Reject