Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary
* For the fine-tuning of pre-trained language models, the authors proposed a supervised learning method that combines cross-entropy loss and contrastive loss.
Experimental results show that the proposed method improves over cross-entropy loss on several classification tasks of the GLUE benchmark set.
The proposed method outperforms cross-entropy loss in few-shot learning tasks and noisy datasets generated by English-German and German-English translation.

Strong points

* The proposed loss function is reasonable and the effect of supervised contrastive learning was not reported for NLP applications before, the experimental results are valuable.
* The paper is well-organized and well-written.
* Without using extra datasets for fine-tuning, the proposed method consistently improves the baseline method.
* The generation of noisy examples using the back-translation model in Section 5.3 is an interesting approach to analyze model robustness.

Weak points
* Although the supervised contrastive learning term as previously proposed in (Khosla et al. 2020), it is not cited in the section.  
* The benchmark results in Table 2 are not comparable with conventional methods since the experimental setting does not follow the finetuning procedures from prior work (Devlin et al., 2019) which reports the test set performance obtained from GLUE submissions.

Decision reason
* The technical contribution of this paper is limited since the proposed method is a rather strait-forward expansion of Khosla et al. 2020.  In addition, although it is novel to apply supervised contrastive learning for NLP applications, the impact of these results is also limited because the experimental results are not directly comparable with previous work.

Questions
* Why did you "sample ... taking the label distribution of the original validation set into account" in Section 4.1?   I am worried that this sampling procedure may make the few-shot task easier.

Additional Feedback
* Since the subtraction between two values in percentage is not a ratio,    the percentage is not an appropriate unit for it.  For example "1.2% improvement on SST-2" in Section 5.1 should be "1.2 point improvement on SST-2".
* Since Khosla et al. 2020 proposed a two-stage training procedure, supervised contrastive learning at the first stage and the learning of the output layer at the second stage, I would like to see the qualitative comparison with the proposed joint training procedure.
* Instead of using different original sentences for each T values, it is clear and compact to use the same original sentence for each T values.
review_title: Good paper but small contribution

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes using a combination of two losses in the fine-tuning stage when using a pre-trained model: the standard CE one, plus a supervised contrastive loss SCL (combined linearly via a \lambda hyperparameter). The supervised contrastive loss uses a normalization summation over the batch examples, with a temperature hyperparameter \theta.

The empirical results cover nicely three scenarios: (a) the impact of adding the SCL loss, in the presence of all the fine-tuning data; (b) the impact of adding the SCL loss in few-shot learning scenarios; and (c) the impact of SCL in the presence of training noise (induced via back-translation through German, using a standard WMT-trained MT model).
The results as presented are encouraging, and support the main hypothesis of the paper, namely that training with the added SCL loss improves performance over all three scenarios mentioned above.

I have a few suggestions that could be seen as minor, and a few observations that are major.
Minor ones:
(i) adding the SCL loss clearly impacts the training speed, yet there is no mention of that (especially as a function of the batch size); in particular, Table 3 would offer the perfect place for mentioning how the training time (for a fixed number of training steps) is affected by the increase batch size, so that the reader can understand both the “upsize” (the improved performance)  as well as the “downsize” (training cost).
(ii) I could see no mention of the settings for the hyperparameters used (\lambda and \theta), nor any ablation experiments that would indicate how their values have been chosen; in the interest of both reproducibility and increased understanding of the value of SCL, please add a sub-section that discusses this issue.

Major ones:
(i) It appears that the authors have used the Roberta_large model and run their own experiments with fine-tuning, w/o and w/ the SCL loss, with little regard for reporting against the published numbers for the GLUE tasks; for instance, in Table 2, they report with CE-only to have RTE performance at 85.0, while the Roberta paper shows 86.6 for it; in this case, the CE+SCL at 85.6 no longer looks like a convincing win; a bit different but nevertheless troublesome is the result reported for CoLA, at 86.4 (CE-only); the numbers for CoLA are normally much lower than that, eg the Roberta paper shows 68.0 (CE-only); this disparity throws a lot of doubt over the accuracy of the results reported in Table 2.
(ii) I commend the authors for showing the variance across their results in both Table 3, 4, and 5; however, it is unclear to me that the claims that the CE+SCL approach is better are being supported by the results. It is not like the CE+SCL gives lower variance, as it is clearly the case that sometimes it is higher than the CE alone; and the fact that, under this high variance, the CE method sometimes performs better CE+SCL puts the whole conclusion under doubt from an empirical standpoint.

One one result that seems to hold strong is the one for few-shot learning, which appears to support the main hypothesis of the paper. However, the main issues mentioned above would need to be addressed in order to have the paper reach the level of clearing the bar for ICLR publication.


Re: References A lot of the references use the Arxiv version for papers that have been peer-reviewed and published. Please fix.

review_title: SUPERVISED CONTRASTIVE LEARNING FOR PRE-TRAINED LANGUAGE MODEL FINE-TUNING

final_score: rating: 7: Good paper, accept
review_content: This work adds a SCL (supervised contrastive learning) loss term during the fine-tuning stage of RoBERTa. The results show that the model with the SCL term and cross-entropy (CE) achieve better GLUE scores than the classic baseline that only uses CE loss, especially when the numbers of supervised training data are small and the data is noisy. 

Pros:
The method is simple and the improvement looks significant under various settings

Cons:
It is not very clear whether and why SCL loss improves the results (see the detailed comments below).


Clarity: 
The text is fluent and the paper cites lots of related work, but the paper does not well explain why the method performs well.

Originality: 
The SCL is not novel because it comes from computer vision but this is the first paper I have seen that successfully applies SCL in NLP tasks.

Significance of this work:
If the authors can really show that the improvement comes from SCL, it may become a popular tool in the fine-tuning stage.


It is possible that the source of improvement comes from the temperature tau and l2 normalization instead of SCL loss itself. Both of the tricks could be also applied to CE loss. Thus, could you do control experiments that replace the SCL loss with the CE loss but keeping l2 normalization and retune the tau and lambda. You can report it as CE+CE.

As shown in Figure 1, the authors suggest that the main reason that SCL loss is better because SCL loss tends to encourage the samples belonging to the same class. However, I believe that CE loss could achieve the same goal and maybe more aggressively than SCL loss. My understanding is that while minimizing CE loss, we encourage each class embedding close to all its samples, so the class embedding tends to become the cluster center of all the points in the class. In the meanwhile, we encourage the samples in each class close to its class embedding, so the samples within the same class will also become closer together, right (e.g., in Figure 3, CE loss could also separate the two classes with a large margin)? The difference is that CE loss encourages the samples close to the average of the samples in the same class, but SCL loss encourages the samples close to each sample in the same class. From this perspective, could you tell us why SCL loss is better (or tell me why this perspective is wrong)? In the introduction, you cite several studies that mention the limitations of cross-entropy loss. I think the motivation of the paper will be much stronger if the authors could briefly and intuitively introduce why CE loss is worse (e.g., what does it mean poor margins and why CE leads to them) and why SCL loss could fix that issue. It would be even better if you can design experiments to demonstrate that (e.g., measure and compare the margins of CE+CE with CE+SCL).

The effect of hyperparameters on performances is not clear. You mention that lambda is tuned in each downstream task. It is possible that in many applications, lambda is 0. Could you show the lambda value for each downstream application? In addition, could you show the final tau value(s) as well? In the experiment setup, you mention that you report the best model out of 10 seeds. Could you also report the average performance? This will tell us more about how stable this SCL method is. In Table 3, could you also report CE only and CE+CE performance?

I believe the issues could be resolved by conducting more controlled experiments and analyses, so I currently vote weak accept. If the concerns are not addressed well during the rebuttal period, I am very likely to change my vote to rejection.

Minor:
1. In equation (2), I think putting CE loss for multi-class classification would be more general.
2. Before the "Relationship to Self-Supervised Contrastive Learning", you mention lower temperature creates harder negatives. The meaning of negative here is unclear. I think you mean harder examples here.
review_title: The results look good but we still need more experiments to identify the source of improvement

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a new training objective for fine-tuning pre-trained models: a weighted sum of the classical cross-entropy (CE) and a new supervised contrastive learning term (SCP). The latter uses the (negated) softmax over the embedding distances (i.e. dot products) between a training instance and all other instances in the batch with the same label. In contrast to the more traditional self-supervised contrastive learning (where positive pairs are obtained by applying transformations to the original data instance), there is no data augmentation; two examples with the same label constitute a positive pair.

Experiments on the GLUE benchmark compare the baseline (RoBERTa-Large with CE loss) against the proposed objective (RoBERTa-Large with CE+SCP loss). There are 4 sets of experiments:
1) When training on the full datasets, results are quite modest (+0.4 increase in accuracy on average over 6 GLUE tasks).
2) In the few-shot setting, CE+SCP does meaningfully better than the baseline (for instance, when fine-tuning on only 20 data points, CE+SCP improves accuracy by more than 10%); these gains decrease as the dataset size increases.
3) When the datasets are noisy (effect obtained via back-translation), CE+SCP shines again (for instance, when the degree of corruption is very high, MNLI accuracy goes from ~47% up to ~53%).
4) Finally, the authors look at domain shift; they fine-tune a model on SST-2, then apply few-shot learning on other sentiment classification datasets. This set of experiments has quite high error margins, so I didn't find it as convincing as 2) and 3).

Here are some questions/suggestions for the authors regarding their experiments:

a) "In all our experiments, [...] we sample half of the original validation set of GLUE benchmark and use it as our test set, and sample ~500 examples for our validation set from the original validation set [...]" -- Evaluating the models on a *subset* of the validation set makes it harder to compare it against other papers that fine-tune RoBERTa-Large. I think that, at least for Table 2, it would be useful for posterity if you could either i) get the true test scores from the GLUE server, or ii) use part of the training set for validation, and then test on the full dev set, which is more standard practice.

b) "We run each experiment with 10 different seeds, and pick the top model out of 10 seeds based on
validation accuracy and report its corresponding test accuracy" -- I am assuming this statement describes how evaluation numbers are reported for a fixed set of hyperparameters. Why do you choose to pick the *top* model as opposed to reporting the *average* accuracy across the 10 runs?

c) "we observe that our proposed method does not lead to improvement on MNLI [...]. We believe
this is due to the fact that number of positive example pairs are quite sparse [...] with batch size 16 [...]. We show evidence for this hypothesis in our ablation studies that we show in Table 3" -- Then why doesn't Table 3 include MNLI? Am I missing something?

d) This method excels in the few-shot setting, at least compared to the CE baseline. So I think it would be a lot more impactful to focus on this particular use case and convince the reader that CE+SCP is better than some other standard few-shot learning baselines (e.g. meta-learning objectives). I do appreciate that the current message of the paper is crystal-clear (adding a SCP term to the loss leads to better fine-tuning), but I also think that the results in Table 2 are too weak for this somewhat general statement. There is quite a bit of real-estate in the paper that could be re-allocated to something more substantive (e.g. Table 1).

Strengths:
- The presentation of the paper is extremely clean, and the goal is clear.
- In the few-shot learning scenario, CE+SCP performs meaningfully better than the CE baseline.

Weaknesses:
- The main weakness is related to my suggestion d) above. I believe marketing CE+SCP as a general fine-tuning solution with somewhat underwhelming results in Table 2 is a missed opportunity to lead with potentially strong results on few-shot learning. I'm calling the results "underwhelming" because there is evidence that a thorough hyperparameter sweep can boost fine-tuning accuracy on GLUE by quite a bit. For instance, Dodge et al. [1] show that fine-tuning BERT carefully can increase SST-2 accuracy by ~2% without any changes in the pre-trained model or fine-tuning objective.

[1] Dodge et al., Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
review_title: Clear goal, clear presentation, somewhat limited novelty / impact

Meta-review: This paper introduces supervised contrastive learning loss on top of typical cross-entropy loss for fine-tuning language model for downstream tasks. While the idea is simple and has been used in vision literature (as pointed out by R1 & R4), its application LM is first introduced in this paper. The experimental gain is small in the regular setting but clearer gains in a few-shot learning setting and noisy training dataset (through back translation) setting. Overall the paper is clearly written and experiments are carefully studied. During the discussion phase, the authors provided results on the full GLUE dataset as well as other ablation studies (e.g., CE+CE recommended by R2), improving the paper.  
Decision: Accept (Poster)