Sentence embedding with contrastive multi-views learning

Abstract: In this work, we propose a self-supervised method to learn sentence representations with an injection of linguistic knowledge. Multiple linguistic frameworks propose diverse sentence structures from which semantic meaning might be expressed out of compositional words operations. We aim to take advantage of this linguist diversity and learn to represent sentences by contrasting these diverse views. Formally, multiple views of the same sentence are mapped to close representations. On the contrary, views from other sentences are mapped further. By contrasting different linguistic views, we aim at building embeddings which better capture semantic and which are less sensitive to the sentence outward form.
      

Reviews and author responses:

The paper proposes a new sentence embedding method. The novelty is to use dependency trees as examples in the self-supervised method based on contrastive learning. The idea to use linguistic knowledge in the design of sentence embeddings is attractive. The sentence representation is computed by a bi-LSTM and dependency tree representations are computed by Tree LSTM. The softmax classifier is trained using the negative log-likelihood loss.

In my opinion, the paper could not be accepted. As said before, the idea is attractive but the paper lacks motivations for the choice of dependency trees as additional linguistic knowledge. Indeed, the goal of the proposed algorithm should be made more precise. It is, in my opinion, very difficult to do better than existing sentence embedding methods and the proposed method should be used for specific downstream tasks where the structure of sentences is meaningful. Moreover, the proposed method do not scale well and empirical results on classical downstream tasks are not convincing. Last, in my opinion, the redaction of the paper should be improved and the bibliography should be updated. For instance, the best up-to-date sentence embedding methods are not cited (ELMO and BERT).

Detailed comments.

* Abstract and introduction. The description of the contribution is not precise enough. Please make precise what are "multiple views", "different linguistic views". Please explain why you choose dependency trees and explain why their use can improve sentence embeddings. 
* Related work. Please consider only word embeddings and sentence embeddings because the literature is sufficiently large in the last few years. Please update your related work with methods such as ELMO and BERT and subsequent work. Also recent papers study how BERT embeddings embed structural information and these should be discussed as you consider dependency trees in the construction of sentence embeddings.
* The method does not scale well. The paper does not propose ideas to solve this problem. Why don't you consider the approach used in Logeswaran et al. 
* The qualitative analysis shows that similar sentences have a similar structure. This is not surprising because dependency trees are used for learning. But this should give ideas of downstream tasks for which the approach could be fruitful.
* Many typos.

Overview: This work proposes to learn sentence embeddings using both contrastive learning and multiple "views" of sentences.  This work largely builds off of [1], including using the same objective, but uses a multi-view approach to modeling.
    - They apply the concept of multi-view models, specifically combining tree and linear LSTMs to learning sentence representations.
    - They prepare a new, large-scale book dataset, which is useful because the previously commonly used book dataset was taken down for legal reason.
    - They provide a fairly broad set of analyses on their model, both quantitative and qualitative, performance-driven and analysis-driven.
- Review: The ideas and models presented in this paper are not new, while the supporting experiments are not very well done or convincing. Overall, I recommend rejecting this work.
- The models are contrastively learned in that they are trained to embed "similar" sentences nearby in the embedding space, and "dissimilar" sentence far away, where "similar" sentences are defined as consecutive sentences. This method of learning textual representations is well-established in the NLP literature, mostly prominently in recent years with word embedding models like Skip-Gram and in sentence embedding models like in [1], [2], [3] (the next sentence prediction task), and several more.
- In practice, the multiple views of each sentence that this paper considers boils down to encoding the sentence with a bidirectional LSTM and a TreeLSTM and concatenating the representations from each encoder. This idea again has been established in the literature ([4], [5], [6]).
- The experiments don't seem setup to demonstrate that the multiple views are beneficial over a single view. In Table 1, there are rows for just an LSTM or just a TreeLSTM, but they seem to be trained with labeled data whereas the proposed method is trained self-supervised. A more informative comparison to demonstrate the value of using multiple views would be to train the LSTM and TreeLSTM with the same objective (and ideally model size). Overall, I don't think the claims in the paper are well-supported by the model proposed or the experiments.
- I have a number of concerns about the experiments.
    - "Models are trained on a single epoch on the entire corpus without any train-test split": so there is no early stopping? Why stop training after one epoch? Was there any indication you were overfitting the data?
    - "The training phase was stopped after 33 hours of training": Why stop there? Computational constraints? Later comments suggest this is quite premature ("training phase was completed on only 4.6M sentences among the 78M available").
    - The results seem to indicate that this method underperforming recent work significantly.

Areas of improvement
- Some of the language in the introduction and conclusion are a bit of a stretch. Using a linear and tree LSTM (based on dependency parses) doesn't really represent a "diversity of linguistic structures". 
- Related work: There's no mention of pretained language models, which could be seen as a form of representation learning for language, and have been hugely impactful in NLP.
- Method
    - Missing negative in the log likelihood
    - Why do you use inner product if other works "report excellent results" with other scoring functions?
    - "assumes the underlying structure of the sentence to be a sequence, while allowing for long term dependencies": If anything, the treeLSTM more easily allows for long-term dependencies than the linear LSTM.
    - "Negative examples are obtained using the dependency Tree LSTM": I'm not totally sure how the negatives are obtained here.
    - "The target sequence is encoded using the sequential Tree LSTM, while the positive and negative samples are encoded using the ChildSum Tree LSTM": why are the sentences not all encoded with the same encoder?
    - It looks really odd that most of Table 1 is empty. Given your model, I imagine it can't have been that difficult to evaluate more baselines (BiLSTM and TreeLSTM) on the rest of the tasks.
    - It'd be nice if you could clearly indicate in Table 1 which method is yours.
- Results and Analysis
    - The standard evaluation setting for sentence embeddings would be GLUE or SuperGLUE.
    - A glaringly missing baseline is BERT (or any of its relatives), which is also self-supervised.
    - The results are underwhelming, and as the author admits, somewhat premature as training didn't seem to finish.
    - 5.2: what are the contrastive LSTM and Tree LSTM? Are those the learned encoders from the "Contrastive Tree" in Table 1, or are they trained from scratch?
    - I don't think the analyses in Sections 5.2 and 5.3 or Figure 2 are particularly useful.
- There are a noticeable number of typos. For example, in the abstract: "this linguist[ic] diversity" and "better capture semantic[s]". It'd be worthwhile to look over the paper closely for typos.


[1] AN EFFICIENT FRAMEWORK FOR LEARNING SENTENCE REPRESENTATIONS. Lajanugen Logeswaran and Honglak Lee
[2] Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning. Yacine Jernite, Samuel R. Bowman, David Sontag
[3] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova
[4] Enhancing and Combining Sequential and Tree LSTM for Natural Language Inference. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang.
[5] Enhanced LSTM for Natural Language Inference. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, Diana Inkpen.
[6] Improving Sentence Representations with Consensus Maximisation. Shuai Tang, Virginia R. de Sa.

This paper describes a self-supervised sentence embedding approach that incorporates a different view from plain text where some extent of linguistic knowledge is incorporated through the application of tree LSTM. The training procedure is standard contrastive framework where the model is encouraged to distinguish between context sentence (sentences appearing close to the target sentence) and negative samples. Evaluations are conducted on 1) downstream tasks, but with a simple logistic regression model on top of sentence embeddings; 2) probing tasks that more focus on surface information prediction, syntactic and semantic tasks; and 3) qualitative analysis with nearest 5 sentences.

Although the experiments are thorough, I am in favor of rejecting this paper with the following reasons:

First, the proposed model is trained with 4.6M sentences among 78M available for 33 hours. It is unclear why authors stop the training at this early stage but the results on all three evaluations seem to be inferior to the state-of-the-art by a big margin. I am happy to raise my score if authors can show the results of a well trained proposed model.

Second, the paper has some room for improvement in terms of clarity, to name a few:
1) Authors can strengthen the motivation for multi-views learning in related work; 
2) Formula 1 for softmax is wrong;
3) Contrastive LSTM and contrastive tree LSTM are not clearly defined in the paper, although the former should refer to quick-thoughts and the latter means the proposed method;
4) In qualitative analysis, for the last example, there is exactly the same candidate with similarity score 0.012. According to cosine similarity, wouldn’t this be 0 and also show up in the baseline model regardless of the embeddings?

Meta-review: This paper proposes a method to learn sentence representations that incorporates linguistic knowledge in the form of dependency trees using contrastive learning. Experiments on SentEval and probing tasks show that the proposed method underperform baseline methods.

All reviewers agree that the results are not strong enough to support the claim of the paper and have some concerns about the scalability of the implementation. They also agree that the writing of the paper can be improved (details included in their reviews below). 

The authors acknowledged these concerns and mentioned that they will use them to improve the paper for future work, so I recommend rejecting this paper for ICLR.

Decision: reject