Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The paper investigates a new training objective, contrastive tension (CT), for obtaining unsupervised sentence embeddings. The objective operates by initializing two models with identical weights and then training the models to produce similar sentence embeddings to each other for identical sentences and dissimilar representations for different sentences. This objective encourages the paired models to agree on positive examples, but at the same time encourages divergence in their models weights by providing different sentences to each encoders for the negative pairs containing different sentences. The new objective is applied as an unsupervised finetuning tasks for BERT, Sentence-BERT, Distill BERT, multilingual BERT, XLNet and XLMR.

The paper demonstrates consistently strong results empirical results on the unsupervised semantic textual similarity (STS) task. The results on supervised STS are mixed with more modest gains and losses over the baseline for some configurations. The paper provides reasonably good analysis on demonstrating the impact of the proposed technique within different layers of a pre-trained model as well as on the effect on the model scores vs. human labels. I found the analysis particularly interesting that showed CT helped the model to better discriminate/score pairs with lower similarity scores (Figure 3). I also like the breadth of the experiments that included a number of different models, finetuning corpora and STS datasets includes multilingual STS.

In terms of potential improvements, I found the results in Figure 1 somewhat surprising in that they show that without further fine-tuning on either unsupervised or supervised data the default representations for many pre-training models are poorly suited for STS. It is well established that the final layers are not useful, but I haven't yet seen an analysis looking at all the layers of so many models. The presentation could possible be improved by including for contrast one existing transformer model that performs well on the STS task (e.g., S-BERT or maybe USE). 

While presenting the worst-performing results across runs is a refreshing change from other papers that might be cherry picking their results, I found the inclusion of just the worst-performing results makes it a little hard to full understand the performance of the model. If possible update all of the results with the worst-performance across runs with average and worst, maybe using something like AvgScore[WorseScore]. If this makes the tables too crowded, consider including more comprehensive results in the appendix of the paper.


review_title: Contrastive Tension (CT) credibly improves performance on unsupervised STS 

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper describes a method for improving pre-trained language representations for sentence similarity tasks.
During the training, two independent models, with identically initialized weights, are trained to maximise the dot product between their sentence representations for identical sentences.
This is shown to improve performance on different sentence similarity benchmarks.

The method could potentially be useful and shows positive empirical results. But there are several questionable aspects that are not addressed in the paper.

The evaluation is currently not convincing. SOTA results are claimed on STS 2012-2016 datasets, but very few contemporary models have been reported on those datasets. Results on the much more widely used STS-B are considerably lower than existing SOTA. Many models, including XLNet, RoBERTa and ALBERT report over 92%, whereas the proposed model achieves 86% on STS-B.

The core of the idea is that the output of two identical models is optimized to be similar to each other. As far as I can tell based on the paper, the only difference then is that negative samples are always passed through one particular model. Without this, the models would remain identical throughout training and there would be no point in optimizing the representations to be similar. It should be more clearly motivated and investigated why this difference would provide any benefit to the model representations.

The method should be compared to other methods that use unsupervised sentence-level objectives for learning sentence representations. QuickThoughts and DeCLUTR are mentioned in the related work section but not compared.

Citations for BERT, Electra, XLNet and GPT-2 are missing in Section 3.

In section 3 it is demonstrated that lower or intermediate layers are generally much more competitive on sentence similarity tasks, compared to the top layer of transformer models. However, it seems that the proposed CT method is only applied to and compared to representations from the top layer of the models. Why not start with a representation that already performs better? At least for the baselines the performance of the best layer should be reported, as simply taking a different layer from the pre-trained model would be a much simpler approach.

The paper says that the worst-performing model is reported. Why? That seems very unusual. It would be more informative to report min and max results or the average.
But also worst among what? Only one model configuration is described, so what are the other (better?) models?

In the supervised setting it is unclear how the experiments are set up. Does CT training happen before, after or parallel with the supervised training?
review_title: Interesting idea but some issues with the explanation and evaluation

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes Contrastive Tension, a self-supervised method to improve sentence representations from pre-trained language models for Semantic Textual Similarity tasks. This work is motivated by the observation by previous work that the final layers of pre-trained model are often biased towards token-level pre-training objectives, and perform poorly for sentence similarity tasks. The proposed method counters this bias by introducing a sentence-level self-supervised task where two different models are encouraged to generate similar representation for the same input sentence, and different representations for different inputs. Experiments show the proposed method significantly improves over previous SotA methods on STS benchmarks.

Pros:
1) The proposed method only relies on unlabeled data.
2) The experimental results on STS benchmarks are strong. The authors also emphasizes reproducibility by reporting lowest performing numbers and promise to release code/model, which is a plus.
Cons:
1) The training objective involving two separate models is not clearly motivated. Since the two models are initialized with identical weights from pre-trained models, the loss of contrastive tension should be very small at the start of fine-tuning. More analysis on the training process, such as training loss curve, could be helpful in this regard.
2) The paper propose to improve the extraction of representations from pre-trained models, but the experiment is only done on STS datasets. I would like to see results on other tasks such MNLI where the sentence level representations are used.

Overall, I find the paper interesting, but the applicability of the method is not clearly demonstrated since the experiment is only done on STS datasets. Thus, I give this paper a weak accept rating. 
review_title: Weak accept

final_score: rating: 7: Good paper, accept
review_content: The paper studies the problem of finding effective representation for semantic text similarity (STS). The paper first investigates the effectiveness of pre-trained masked language models (e.g. BERT) in the STS task. They found different layers of BERT have different performance when employed in STS task -- in particular, the popular method of using the last layer does not usually lead to  good performance. In fact, the last layers are worse than those preceding layers in STS. This is universal in several models including BERT, Electra, XLNet, and GPT-2, but not RoBerta. The paper then propose to use dual contrastive training method (basically use two BERT branches) to further fine-tune BERT, where the additional objective is defined by bringing two models' output closer for the same input sentence and further for randomly sampled different sentences. 
The authors evaluated their method on standard benchmark of STS task and obtain consistent improvement over BERT (unsupervised setting) and S-BERT (supervised setting)

Strong points of the paper:
S1. The observation of earlier layers of pre-trained masked language models work better than later layers on STS is interesting and surprising. It is consistent in all models except Roberta-base. 
S2. The proposed method is relatively simple to implement. 
S3. The experimental results on STS task verifies that the proposed method achieves consistent and significant improvement over the plain BERT and supervised BERT.

Weak points:
W1. Although the observation is interesting, the connection between the observation on layer-wise performance of BERT on STS and the proposed contrastive training method is not clear. 
W2. The paper lacks details in the training of BERT with CT and BERT-Distil. It is also unclear whether the BERT-base-CT is using BERT-base as the initialization. 
W3. Though the method works for semantic text similarity, it does not work for other NLP tasks in particular on GLUE. It is unclear why. 

Additional comments and questions:
Since it only works on STS, the title of the paper may appear too broad. The authors may consider revising the title. 
Some statements in the introduction are not backed by evidence. For example, the layers close to the final objective function will be more task-specific.  
In the abstract, it states "impose a significant bias on the the final layers". What bias?
The authors may want to avoid using too long compound sentences, which hurts readability.
Figure 1. what is the range of y-axis?

review_title: Interesting observation with simple method on text representation learning for STS

Meta-review: 
This paper described a model that improves the performance of LM-based pre-trained sentence representation on semantic text similarity tasks (STS). The proposed approach is motivated by the observation that top-layers in transformer-based LMs are quite poor at this task per se. This paper proposes Contrastive Tension, a self-supervised objective that drags representations of same sentence together, and pulls away representations of different sentences. The proposed method only relies on unlabelled data, and is relatively simple to implement. The paper demonstrates consistently strong results empirical results on the unsupervised semantic textual similarity (STS) task. Moreover, the paper provides reasonably good analysis.

On a negative side, the reviewers noted that the paper lacks a bit of analysis about the objective. The connection between the observation on layer-wise performance of BERT on STS and the proposed contrastive training method is not clear. Second, while the result is interesting, its applicability is limited to STS.

Taking into account all the above, the reviews constitue a case for a solid weak-accept. Therefore I recommend acceptance as a Poster contribution.
Decision: Accept (Poster)