Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning
Authors: Beliz Gunel
Jingfei Du
Alexis Conneau
Veselin Stoyanov
Venue: ICLR 2021 Poster (2021)
Keywords: pre-trained language model fine-tuning
supervised contrastive learning
natural language understanding
few-shot learning
robustness
generalization

Abstract: State-of-the-art natural language understanding classification models follow two-stages: pre-training a large language model on an auxiliary task, and then fine-tuning the model on a task-specific labeled dataset using cross-entropy loss. However, the cross-entropy loss has several shortcomings that can lead to sub-optimal generalization and instability. Driven by the intuition that good generalization requires capturing the similarity between examples in one class and contrasting them with examples in other classes, we propose a supervised contrastive learning (SCL) objective for the fine-tuning stage. Combined with cross-entropy, our proposed SCL loss obtains significant improvements over a strong RoBERTa-Large baseline on multiple datasets of the GLUE benchmark in few-shot learning settings, without requiring specialized architecture, data augmentations, memory banks, or additional unsupervised data. Our proposed fine-tuning objective leads to models that are more robust to different levels of noise in the fine-tuning training data, and can generalize better to related tasks with limited labeled data.

Published as a conference paper at ICLR 2021
SUPERVISED CONTRASTIVE LEARNING FOR PRE-TRAINES LANGUAGE MODEL GUNINTUNING
Beliz Gunel", Jingfei Dul,Alest Conneau', Ves Stoyanov?
Stannoor University, FFacebook Al
ABSTRACT
State-of-the-art natural language understanding classification models follow two- stages: pre-training large language model on an auxiliary task, and then fine- tuning the model on task-specific labeled dataset using cross-entropy loss. How- ever, the cross-entropy loss has several shortcomings that can lead to sub-optimal generalization and instability. Driven by the intuition that good generalization requires capturing the similarity between examples in one class and contrasting them with examples in other classes, we propose a supervised contrastive learning (SCL) objective for the fine-tuning stage. Combined with cross-entropy, our pro- posed SCL loss obtains significant improvements over a strong RoBERTa-Land Large baseline multiple datasets of the GLUE benchmark in few-shot learning settings, without requiring specialized architecture, data augmentations, memory banks, or additional unsupervised data. Our proposed fine-tuning objective leads models that are more robust to different levels noise in the fine-tuning training data, and can generalize better related tasks with limited labeled data.
INTRODUCTION
Statc-of-the-ar for most existing natural language processing (NLP) classification tasks is achieved by models that are first pre-trained auxiliary language modeling tasks and then fine-tuned on the task of interest with cross-entropy loss (Radford et al., 2019 Howard & Ruder 2018 Liu et al. Deilin et al. 2019). Although ubiquitous, the cross-entropy loss the KL. KL-divergence between one-hot vectors of labels and the distribution of model's output logits has several shortcomings. Cross entropy loss leads poor generalization performance (Liu et al. 2016: Cao et al. 2209)), and it lacks robustness to noisy labels (Zhang & Sabuncu 2018 Sukhbaatar et al. 2015) or adversarial examples (Elsayed et 2018 Nar al.) 2019). Effective alternatives have been proposed modify the reference label distributions through label smoothing (Szegedy et al.) 2016 Müller et al. 2019).
Mixup (Zhang et al. 2018), CutMix (Yun et al. (2019), knowledge distillation (Hinton et al. 2015) or
self-training Yalniz et al. 2019 Xie et al. 2020).
Fine-tuning using cross entropy loss in NLP also tends to be unstable across different runs Zhang et al. 2020; Dodge et al. 2020), especially when supervised data is limited, scenario in which pre-training is particularly helpful. To tackle the issue unstable fine-tuning and poor generalization, recent works propose local smoothness-inducing regularizers (Jiang et al. 2020) and regularization methods inspired by the trust region theory (Aghajanyan et al. 2020) to prevent representation collapse. Empirical evidence suggests that fine-tuning for more iterations. reinitializing top few layers (Zhang et al. 2020), and using debiased Adam optimizer during fine-tuning (Mosbach et al.) 2020) can make the fine-tuning stage
stable.
Inspired by the learning strategy that humans utilize when given a few examples, we seek to find the commonalities between the examples of each class and contrast them with examples from other classes. We hypothesize that a similarity-based loss will be able to hone in on the important dimensions of the multidimensional hidden representations hence lead to better few-shot learning results and be more stable while fine-tuning pre-trained language models. We propose novel objective for fine-tuning that includes a supervised contrastive learning (SCL) term that pushes the examples from the same class close and the examples from different classes further apart. The SCL
Work done during Facebnok AI research internship, correspondence bluuunl @atanforddeddd

---

Published as a conference paper at ICLR 2021
term is similar to the contrastive objectives used in self superviid representation learning across
image, speech, and video domains. (Sohn 2016; Oord et al. 2018 Wu et al. 2018 Bachman et al.
|0I9|Henaft Henaff al. 2019 Baevski et al. 2020/Cooneee
et al. 2020 Lian et al. 2020; Hjelm et al.
2019 Han et al. 2019 He et al. 2020; Misra & Maaten 2020 Chen et a (2020a b). Unlike these
methods, however, we use contrastive objective for supervised learning of the final task, instead of contrasting different augmented views of examples.
In few-shot learning settings (20, 100, 1000 labeled examples), the addition of the SCL term to the fine- tuning objective significantly improves the performance on several natural language understanding classification tasks from the popular GLUE benchmark (Wang et al. 2019) over the very strong baseline of fine-tuning RoBERTa-Large with cross-entrope loss only. Furthermore, pre-trained language models fine-tuned with our proposed objective are not only robust to noise in the fine-tuning training data, but can also exhibit improved generalization to related tasks with limited labeled task data. Our approach does not require any specialized network architectures (Bachman et al. 2019)
Hénaff et al. 2019), memory banks (Wu et al. 2018 Tian al. 2020 Misra & Maaten 2020 data augmentation of any kind, or additional unsupervised data. To the best our knowledge, our work is the first to successfully integrate supervised contrastive learning objective for fine-tuning pre-trained language models. We empirically demonstrate that the new objective has desirable properties across several different settings. Our contributions in this work are listed in the following:
We propose novel objective for fine-tuning pre-trained language models that includes supervised contrastive learning term, as described in Section 2.
We obtain strong improvements in the few-shot learning settings (20, 100, 1000 labeled examples) as shown in Table 2 leading up 10.7 points improvement subset of GLUE benchmark tasks (SST-2, QNLI, MNLI) for the 20 labeled example few-shot setting, over a very strong baseline RoBERTa- Large fine-tuned with cross-entropy loss.
We demonstrate that our proposed fine-tuning objective is more robust, in comparison to RoBERTa-Large fine-tuned with cross-entropy loss, across augmented noisy training datasets (used to fine-tune the models for the task of interest) with varying noise levels as shown in Table leading up to points improvement on a subset of GLUE benchmark tasks (SST-2, QNLI, MNLI) across augmented noisy training datasets. We use back- translation model to construct the augmented noisy training datasets of varying noise levels (controlled by the temperature parameter), described in detail in Section 4.2.
We show that the models fine-tuned with proposed objective have improved gener- alizability to related tasks despite having limited availability of labeled task data (Table (7). This led to 2.9 point improvement Amazon-2 the task model fine-tuned with cross-entropy loss only. Moreover, it considerably reduced the variance across few-shot training samples, when transferred from the source SST-2 sentiment analysis task model.
2
APPROACH
We propose novel objective that includes a supervised contrastive learning term for fine-tuning pre-trained language models. The loss is meant to capture the similarities between examples of the same class and contrast them with the examples from other classes.
For multi-clan classification problem with C classes, we work with a batch of training examples of size N, {#,, y4 J= (-) E Rd denotes an encoder that outputs the l2 normalized final encoder hidden layer before the softmax projection; Ny, is the total number of examples in the batch that have the same label as yı: t > 0 is an adjustable scalar temperature parameter that controls the separation of classes; Vi.c denotes the label and fr.c denotes the model output for the probability of the ith example belonging to the class c; À is a scalar weighting hyperparameter that we tune for each downstream task and setting. The overall loss is then given in the following:
2

---

Published as a conference paper at ICLR 2021
= (1 XcE + ALScL
(1)
N
Ler = N
Vic loggiic
(2)
exp (D(x,) ((z))/T) kmm Ligh exp (#(x;) 2 (xx))/T)
LSCL. Σ
Σ
1 log
(3)
The overall loss is weighted average of CE and the proposed SCL loss, as given in the equation (1). The canonical definition of the multi-class CE loss that we use is given in equation (2). The novel SCL loss is given in the equation (3).
This loss can be applied using a variety of encoders ((.) E Rd for example a ResNet for a computer vision application or a pre-trained language model such BERT for NLP applica- tion. In this work, we focus on fine-tuning pre-trained language models for single sentence and sentence-per pair classification settings. For single sentence classification, each example x; consists of sequence of tokens prepended with the special [CLS] token The length of sequence is constrained such that < Lmax- Similarly, for sentence-pair clas- sification tasks, each example zÿ is concatenation of two sequences of tokens [e, 22 (x] and [81, 18, S2,..., SM] corresponding to the sentences with special tokens delimiting them: a, = [[CLS), t2, strained such that + < Lmax- both cases, ((x;) E R uses the embedding of [CLS] token as the representation for example z;- These choices follow standard practices for fine-tuning pre-trained
[[CLS], f1, 12,..., [EOS]]
[SEP), $1, 82, ................................ SM, [EOS]]. The length of concatenated sequences is con-
language models for classification (Devlin et al. 2019 Liu et al. 2019).
"Works hard to establish rounded charaateess, but then has nothing fresh or particularly intoresting to say about them."
"It's worth sceing fust on basis of the wisdom, and times, the startiing optinism, of the childreen."
CE SC.L
"Iff you vafuee your time and monay, find an escape clausse and avoid secing this trite, predictaba rehash."
"If you enjoy thoughtful comedies with interesting confficted characters; this one is for you."
embaddding from class A embedddig from class B
Figure 1: Our proposed objective includes a cross-entropy term (CE) and a supervised contrastive learning (SCL) term, and is formulated to push examples from the same class close and examples from different classes further apart. We show examples from the SST-2 sentiment analysis dataset from the GLUE benchmark, where class A (shown in red) is negative movie reviews and class (shown blue) is positive movie reviews. Although we show a binary classification case for simplicity, the loss is generally applicable any multi-class classification setting.
Empirical observations show that both l normalization of the encoded embedding representations and an adjustable scalar temperature parameter improve performance, Lower temperature increases the influence of examples that are harder separate, effectively creating harder negatives. Using hard negatives has been previously shown to improve performance in the context of margin-based loss formulations such as triplet loss (Schroff al., 2015). The empirical behavior of the adjustable temperature parameter is consistent with the observations of previous work related to supervised contrastive learning. (Chen et al. 2020a| Khosla et al. 2020).
Relationship to Self-Supervised Contrastive Learning Self-supervised contrastive learning has shown success in learning powerful representations, particularly in the computer vision domain. (Chen al. 2020a| He al. 2020 Tian et al. 2020 Mnih & Kavukcuoglu, 2013) Gutmann & Hyvärinen 2012 Kolesnikov et al. 2019) Self-supervised learning methods do not require any labeled data;
instead they sample mini batch from unsupervised data and create positive and negative examples
3

---

Published as a conference paper at ICLR 2021
from these samples using strong data augmentation techniques such as AutoAugment (Cubuk et al. 2019) or RandAugment (Cubuk et al. 2020) for computer vision. Positive examples are constructed by applying data augmentation to the same example (cropping, flipping, etc. for an image), and negative examples are simply all the other examples in the sampled mini batch. Intuitively, self- supervised contrastive objectives positive pairs; while maximizing the distance between negative pairs. The distance metric used is often the inner product or the Euclidean distance between vector representations of the examples.
learning representations that invariant different views of
For a batch of size N. self-supervised contrastive loss is defined as:
2N
exp ( (0(2)111) (x22)/T)
Saels
Σ
log
2N lize exp (f(x)) (17))/7)
where ((-) R2 denotes an encoder that outputs the l2 normalized final encoder hidden layer before the softmax projection; r > 0 is a scalar temperature parameter. A is defined as data augmentation block that generates two randomly generated augmented examples, 2/2 and -22 from the original example æy: A((2;, yi)i-I,... {ar )=1. As example, A can be RandAugment for computer vision application; or could be a back-translate model for an NLPP application.
3 RELATED WORK
Traditional Machine Learning and Theoretical Understanding Several works have analyzed the shortcomings of the widely adopted cross-entropy loss, demonstrating that it leads to poor generalization performance due to poor margins (Liu et al., 2016 Cao al., 2019), and lack of robustness to noisy labels (Zhang & Sabuncu 2018 Sukhbaatar et al. 2015) or adversarial examples (Elsayed et al. 2018 Nar et al. 2019). On the other hand, there has been a body work that has explored the performance difference for classifiers trained with discriminative (i.e., optimizing for p(y(z), where y is the label and x is the input) losses such as cross-entropy loss and generative losses (i.e. optimizing for p(zly)). ). Ng & Jordan (2001) show that classifiers trained with generative losses can outperform their counterparts trained with discriminative losses in the context of Logistic Regression and Naive Bayes. Raina al. (2003) show that a hybrid discriminative and generative objective outperforms both solely discriminative and generative approaches. In the context of contrastive learning. (Saunshi al. (2019) propose a theoretical framework for analyzing contrastive learning algorithms through hypothesizing that semantically similar points are sampled from the same latent class, which allows showing formal guarantees on the quality of learned representations.
Contrastive Learning There has been several recent investigations for the use of contrastive ob- jectives for sulf-supervised, semi- supervsueee,sed, and supervised learning methods, primarily in the computer vision domain. Chen et al. (2020a) propose a framework for contrastive learning of vi- sual representations without specialized architectures or a memory bank, and show state-of-the-an results on ImageNet ILSVRC-201 (Russakovsky et al. 2015) outperforming previous methods for self-supervised, semi-supervised and transfer learning. Similarly, Khosla et al. (000)) propose supervised contrastive loss that outperforms cross entropy loss and gets state-of-the art results on ImageNet on both ResNet-50 and ResNet-200 (He et al.) 2016) with AutoAugment (Cubuk et al. 2019) data augmentation. They also show increased robustness the ImageNet--C dataset (Hendrycks & Dietterich] 2019). and demonstrate that supervised contrastive loss is less sensitive to different hyperparameter settings for optimizers or data augmentations compared to the cross-entropy ietiimiinntivegeneratt
training of energy-based models where they approximate the generative term with contrastive loss using large batch sizes and show improved classification accuracy of WideResNeet28- 10 (Zagoruyko & Komodakis 2016) on CIFAR-10 and CIFAR-100 (Krizhevsky) (2009) datasets, outperforming state-of-the-art discriminative and generative classifiers. They also demonstrate improved performance for WideResNNet228. 10 on robustness, out-of-distribution detection, and calibration, compared to other state-of-the -he--art gener- ative and hybrid models. Finally, Fang & Xie (2020) propose pre-training language models using a self-supervised contrastive learning objective at the sentence level using vack-translation as the augmentation method, followed by fine-tuning by predicting whether two augmented sentences originate from the same sentence demonstrating improvements over fine-tuning BERT on a subset of GL GLUE benchmark tasks.
loss. Liu & Abbeel (2020) propose a hybrid
4

---

Published as a conference paper at ICLR 2021
Stability and Robustness of Fine-tuning Pre-trained Language Models There has been recent works on analyzing the stability and robustness of fine-tuming pre-trained language models, since they have been shown to overfit the labeled task data while fine-tuning and hence fail to generalize to unseen data when there is limited labeled data for the task (Aghajanyan et al. (020). To improve the generalization performance, Daaaaanneennnnngeee.a et al. (2020) propose a local smoothness- smmothhess--mddccin regularizer to manage the complexity of the model and Bregman proximal point optimization method, an instance of trust-region methods, to prevent aggressive updating of the model during fine-tuning. They show state-of-thee of-the-rrt performance on GLUE, SNLI (Bowman et al. 2015), SciTail (Khot) al. 2018), and ANLI (Nie et al., (020)) natural language understanding benchmarks. Similarly, Aghajanyan et al. (2020) propose regularized fine-tuning procedure inspired by trust-region theory that replaces adversarial objectives with parametric noise sampled from normal or uniform distribution in order to prevent representation collapse during fine-tuning for better generalization performance, without hurting the performance. They show improved performance on a range of natural language understanding and generation tasks including DailyMai/CCNN (Hermann et al. 2015). Gigaword (Napoles et al., 2012), Reddit TIFU (Kim et al.] 2019), and the GLUE benchmark. There has also been some empirical analysis that suggests fine-tuning for more epochs, reinitializing top few layers (Zhang et al. 2020) instead of only the classification head, and using debiased Adam optimizer
instead of BERTAdam (Devlin et al. 2019) during fine-tuning (Mosbach al. 2020) can make the fine-tuning procedure more stable across different runs.
4 EXPERIMENTAL SETUP
4.1 DATASETS AND TRAINING DETAILS
We use datasets from the GLUE natural language understanding benchmark (Wang al. 2019) for evaluation. We include both single sentence classification tasks and sentence-pair classification tasks to test whether our hypothesis is generally applicable across tasks. We summarize each dataset based on their main task, domain, number of training examples, and number of classes in Table
In our few-shot learning experiments, we sample half of the original validation set of the GLUE benchmark and use it as our test set, and sample ~500 examples for our validation set from the original GIUEE validation set, both taking the label distribution of the original validation set into account. For each task, we want the validation set to be small enough to avoid easy overfitting on the validation set, and big enough to avoid high-variance when early-stopping at various epochs for the few-shot learning experiments. For full dataset experiments, such as the ones shown in Table 5 Table 6 Table 8 and T we sample a validation set from the original training set of the GLUE benchmark based on the size of the original validation set of GLUE, and report test results on the original validation set of GLUE
We run each experiment with 10 different seeds, and report the average test accuracy, standard deviation, along with p-values with respect to the baseline. We pick the best hyperparameter combination based on the average validation accuracy across 10 seeds. For few-shot learning experiments, such as the ones shown in Table 2 Table 3 and Table I0 we sample 10 different training set samples based on the total number of examples N specified from the original training set of the GLUE benchmark, taking the label distribution of the original training set into account. We report the average and the standard deviation of the test accuracies of the top 3 models based on their validation accuracies out of 10 random training set samples. Best hyperparameter combination is picked based on the average validation accuracy of the top 3 models. The reason why we focus on the top3 3 models for this setting is that we would like to reduce the variance across training set samples.
We use fairseq Ott et al. 2019) library and the open-source RoBERTa-Large model for all of our experiments. During all the fine-tuning runs, we use Adam optimizer with learning rate of le-5, batch size of 16 (unless specified otherwise), and dropout rate 0.1. For each experiment that includes the SCL term, we conduct grid- hyperparameter sweep for À {0.1, 0.3, 0.5, 0.7, 0.9, 1.0} and 7 E {0.1, 0.3, 0.5, 0.7]. We observe that models with best test accuracies across all experimental settings overwhelmingly use the hyperparameter combination 0.3 and À 0.9.
5

---

Published as a conference paper at ICLR 2021
Dataset
Task
Domain
#Train
# #Classes
SST-2
sentiment analysis
movie reviews linguistic publications news
67k
2
CoLA
grammatical correctness
8.5k
2
MRPC
paraphrase
3.7k
2
RTE
textual entailment
news/Wikipedia
2.5k
2
QNLI
question ans answering/textual entailment
Wikipedia
105k
2
MNLI
textual entailment
multi multi-domain
393k
3
Table I: GLUE Benchmark datasets used for evaluation.
4.2 CONSTRUCTING AUGMENTED NOISY TRAINING DATASETS
Machine learning researchers or practitioners often do not know how noisy their datasets are, as input examples might be corrupted or ground truth labeling might not be perfect. Therefore, it is preferable to use robust training objectives that can get more information out of datasets of different noise levels, where there is limited amount of labeled data. We construct augmented noisy training datasets (used to fine-tune the pre-trained language models for the task of interest) of different noise levels using a back-translation model (Edunov et 2018), where to create more noisy examples. trecksttansltiion refers to the procedure of translating an example in language A into language B and then translating back to language A, and it is a commonly used data augmentation procedure for NLP applications, as the new examples obtained through mack-translation provide targeted inductive bias to the model while preserving the meaning of the original example. Specifically, we use WMT' 18 English-German and BermannEEnglih translation models, use random sampling to get more diverse examples, and employ and augmentation ratio of 1:3 for supervised axamplesaaummented examples. We observe that employing random sampling with a tunable temperature parameter is critical to get diverse paraphrases for the supervised examples, consistent with the previous work (Edunov al. 2018 Xie et al. (2019), since commonly used beam search results in very regular sentences that do not provide diversity to the existing data distribution.
increase the temperature parameter
We keep the validation and test sets same with the experiments shown in Table 2
ANALYSIS AND RESULTS
5.1 GLUE BENCHMARK FEW-SHOT LEARNING RESULTS
We proposed adding the SCL term inspired by the learning strategy of humans when they are given few examples. In Table [ we report our few-shot learning results on SST-2, QNLI, and MNLI from the GLUE benchmark with 20, 100, 1000 labeled training examples. Details of the experimental setup are explained Section 4. We use very strong baseline of fine-tuning RoBERTa-Large with cross-entropy loss. We observe that the SCL term improves performance over the baseline significantly across all datasets and data regimes, leading to 10.7 points improvement on QNLI, 3.4 points improvement MNLI, and 2.2 points improvement SST-2, where we have 20 labeled examples for fine-tuning. This shows that our proposed objective is effective both for binary single sentence classification such sentiment analysis; and sentence pair classification tasks such as textual entailment and paraphrasing when we are given only few labeled examples for the task. We that as we increase the number of labeled examples, performance improvement over the baseline decreases, leading to 1.9 points improvement on MNLI for 100 examples and 0.6 points improvement on QNLI for 1000 examples. We also would like acknowledge that improvements over the baseline when N=1000 on both SST-2 and MNLI are not statistically significant. In addition, we conduct an ablation study where we investigate the importance of l2 normalization and temperature scaling where we replace SCL loss with CE loss but keep the la normalization and temperature scaling, as shown in Table in the Appendix under the method name CE+CE.
In Figure 2 we show (SNE plots of the learned representations of the CLS embeddings on SST-2 test set when RoBERTa-Large is fine-tuned with 20 labeled examples, comparing CE with and without the SCL term. We can clearly that the SCL term enforces with the same label; while the distribution of the embeddings learned with CE is close to random. We include more detailed comparison for CE and CE+SCL showing learned representations of
compact clustering of examples

---

Published as a conference paper at ICLR 2021
examples as tSNE plots, where have 20, 100 labeled examples and full dataset respectively for fine-tuning in Figure in the Appendix.
Model
Loss
N
SST-2
QNLI
MNLI
RoBERTaLare
CE
20
85.94-2.1
65.0--2.0
39.3+-2.5
RoBERTa Laggg
Large
CE + SCL
20
88.11:3.3
75.7
42.71:4.6
p-value
5c-10
1c-46
1c-8
RoBERTay aagERTa Large
CE
100
91.1111.
81.9:10.4
59.242.1
RoBERTa Large
+ SCL
100
92.8+11.3
82.5++0.4
61.1+3.00
p-value
3e-17
le-20
2e-4
RoBERTaLare
CE
1000
94.010.6
89.2:10.6
81.410.2
RoBERTa
CE + SCL
1000
94.1110.5
89.8:0.4
81.51002
p-value
0.6
le-12
0.5
Table 2: Few-shot learning test results on the GLUE benchmark where we have N=20,100,100
labeled examples for training. Reported results are the mean and the standard deviation of the test accuracies of the top 3 models based on validation accuracy out of 10 random training set samples, along with p-values for each experiment.
Figure 2: ISNE plots of the learned CLS embeddings the SST-2 test set in the few-shot learning setting of having 20 labeled examples to fine-tune on comparing RoBERTa-Larg fine-tuned with CE only (left) and with proposed objective CE+SCL (right) for the SST-2 sentiment analysis task. Blue: positive examples; red: negative examples.
5.2 ROBUSTNESS ACROSS AUGMENTED NOISY TRAINING DATASETS
In Table we report our results on augmented noisy training sets with varying levels of noise. We have 100 labeled examples for fine-tuning for each task, and augment their training sets with noisy examples using a back-translation model, as described in detail in Section 4.2. Note that we use the back-translation model simulate training datasets of varying noise levels and not method to boost model performance. Experimental setup follows what is described in Section 4 for few-shot learning experiments. T is the temperature for the tack-translation model used augment the training sets, and higher temperature corresponds to more noise in the augmented training set.
We observe consistent improvements over the RoBERTa-Large baseline with our proposed objective across all datasets across all noise levels, with 0.4 points improvement on SST-2, 2.5 points improve- ment on QNLI, and 7 points improvement on MNLI on average across augmented training sets. The improvement is particularly significant for inference tasks (QNLI, MNLI) when the noise levels are higher (higher temperature), leading to 7.7 points improvement on MNLI when T=0.7, and 4.2 points improvement QNLI when T=0.9. We show some samples of the augmented examples used in this robustness experiment in Table 4 For T=0.3, examples mostly stay the same with minor changes in their phrasing, while for T=0.9, some grammatical mistakes and factual errors introduced.
7

---

Published as a conference paper at ICLR 2021
Dataset
Loss
Original
T=0.3
T=0.5
T=0.7
T=0.9
Average
SST-2
CE CE + SCL
91.1
9,30++1.3
91,4+11.
91.7++1.3
90.0
91.3+11.2
S
9..811.3
92.611.9
91.5+11.0
91.2400.6
91.511.0
91.7110
QNLI
CR 81.9+0.4
81.1+2.3
80.0+2.9
78.9+3.7
75.9+4..0
79.0+3.5
QNLI
CE + SCL
82.510.4
82.7119
81.912.5
81.30..6
80.1
81.51200
MNLI
CE 59.2.12.1
54.0.11.1
$5.312.4
54.641..
47.0.1..8
52.713.9 59.7114.3
MNLI
CE + SCL
61.1113.0
61.2.1.33
6..1
62.3
53.0.12.1
Table 3: Results on the GLUE benchmark for robustness across noisy augmented training sets. Average shows the average performance across augmented training sets.
Dataset
Type
Sentence
SST-2
Original
possihly the best actor working movies today, As perhaps the best actor who now stars in films.
SST-2
Augmented (T=0.3)
SST-2
Original
The young stars are too cute; the story and ensuing complications are too manipulative. The babies are too cute, the image and complications that follow too manipulative.
SST-2
Augmented (T=0.9)
QNLI
Original
Brain tissuc is naturally soft, but can be stiffened with what liquid? Brain tissue is omitted naturally, but with what fluid can be stiffened?
QNLI
Augmented (T=0.3)
QNLI
Original
In March 1968, and Sony formed CBS/Sony Records, Japanese business joint venture.
QNLI
Augmented (T=0.9)
CBS was founded by CBS and Sony Records in March 1962, Japanese company.
MNLI
Original
However, the link did not transfer the user to comment box particular to the rule at issue. However, the link did not send the to comment field specificully for the rule.
MNLI
Augmented (T=0.3)
MNLI
Original
Tenants could not enter the apartment complex due to dangerous chemical spill.
MNLI
Augmented (T=0.9)
Tenants were banned from entering the medical property because of a blood positive substance.
Table 4: Sample of augmented examples with different noise levels for the robustness experiment
shown in Table 3 Higher temperature (T) corresponds to more noise in the augmented training set.
5.3 GLUE BENCHMARK FULL DATASET RESULTS
In 5 we report results using our proposed objective on six downstream tasks from the GLUE benchmark. We use a very strong baseline of fine-tuning RoBERTa-Large with cross-entropy loss, which is currently the standard practice for the state-of-the-art NLP classification models. Details of the experimental setup are explained in Section 4.
We observe that adding the SCL term to the objective improves the performance over the RoBERTa- Large baseline that lead to 3.1 points improvement on MRPC, 3.5 points improvement on QNLI, and an average improvement of 1.2 points across all 6 datasets. We conduct these experiments to investigate the effect the SCL term in high-data regimes, as we observe that it's effective in few-shot learning settings. We acknowledge that only MRPC and QNLI results are statistically significant, and we report the results on the other datasets as a finding for the sake of completeness.
We hypothesize larger batch sizes lead to better performance, but we leave that for future work that requires additional engineering effort. We show evidence for this hypothesis in our ablation studies that we show in Table 6 where conduct the full dataset experiments for CE+SCL with the same experimental setup described here for Table SST-2, CoLA, QNLI, and MNLI for batch sizes 16, 64, and 256 using RoBERTa-Baan improves significantly across all datasets. Specifically, we observe 0.3 points improvement on SST-2, 0.8 points improvement on CoLA. 0.4 points improvement on QNLI, and 1.3 points improvement on MNLI, when we increase the batch size from 16 to 256 for CE+SCL. We also investigate the effect of SCL term in the overall training speed, and we measure that with average updates per second metric, shown in 6 For batch size 16, the batch size we use throughout the paper across all experimental settings, effect of SCL is negligible decreasing average updates per second from 15.9 to 15.08. As we increase the batch size, effect of SCL to training speed becomes more significant decreasing average updates per second from 2,46 to 1.54 for batch size 256. In addition, we conduct an ablation study where we investigate the importance of l2 normalization and temperature scaling where we replace SCL loss with CE loss but keep the normalization and scaling (denoted as CE+CE) both for full dataset results in Table & and for batch size ablation in Table 9 in the Appendix.
We observe that we increase the batch size, performance
8

---

Published as a conference paper at ICLR 2021
Model
Loss
SST-2
CoLA
MRPC
RTE
QNLI
MNLI
Avg
RoBERT
CE
96.01::.4
86.0+0.5
86.41:2.4
85,5+18
90.4
88.41
88.8
RoBERTaLarge
CE + SCL
96.3+-0.4
86.1--0.8
89.5++0.9
85.7+0..
93.94.077
88.6+
90
p-value
0.07
0.63
0.01
0.06
0.01
0.16
Table 5:
Test results on the validation set of GLUE benchmark. We compare fine-tuning RoBERTa- Large with CE with and without SCL. Best hyperparameter configuration picked based on average validation accuracy, We report average accuracy across 10 seeds for the model with best hyperparam- eter configuration, its standard deviation, and p- values.
Model
Loss
Bsz
SST-2
CoLAA
QNLI
MNLI
Avg ups/sec
RoBERTaBase
CE
16
94.1
83.310.7
88,210.8
8410.6
15.9
RoBERTaBase
CE + SCL
16
94.9110.6
83.710.9
92.510.4
85.310.5
15.08
RoBERTaBsee
CE
64
94.2.10.4
83.310.5
89.2.10.5
8410.4
8.43
RoBERTaBes
CE + SCL
64
94.7=0.2
83.8+0.6
92.6+0.5
85.7+0.7
7.44
RoBERTaßss
CE
256
94.1=004
8440.5
90=00.7
84.4
2.46
RoBERTaBase
CE + SCL
256
95.2:10.3
84.5+0.5
92.910.3
86.6210.6
1.54
Table 6: Ablation study on performance and training speed shown as average updates per second (Avg ups/sec) for fine-tuning RoBERTa-Base with respect the batch size (Bsz).
5.4 GENERALIZATION ABILITY OF TASK MODELS
In this experiment, we first fine-tune RoBERTa-Large on SST-2 using its full training set and get task model with and without SCL term. Then, we transfer this task model to two related single sentence sentiment analysis binary classification tasks for the movie reviews domain - Amazon-2 and Yelp-2 (Zhang al. 2015). For both, sample 20 labeled examples for each class, and follow the few-shot learning experimental setup described in Section 4. In Table 7 we demonstrate that using the SCL term for both source (SST-2) and target domains (Amazon-2, Yelp-2) lead to better generalization ability, with 2.9 points improvement on Amazon-2 and 0.4 points improvement on Yelp-2 along with significant reduction variance across training set samples.
Model
Loss
N
Amazon-2
Yelp-2
RoBERTaLarg
CE
40
87.4+6..4
90.8++2.2
RoBERTaLaage
CE + SCL
40
90.3
91.2-10.4
Table 7: Generalization of the SST-2 task model (fine-tuned using the full training set) to related tasks (Amazon-2, Yelp-2) where there are 20 labeled examples for each class.
6 CONCLUSION
We propose a supervised contrastive learning objective for fine-tuning pre-trained language models and demonstrate significant improvements over a strong RoBERTa-Large baseline on multiple datasets of the GLUE benchmark in the few-shot learning settings. We also show that our proposed objective leads to models that are more robust to different levels of noise in the training data and can generalize better to related tasks with limited labeled task data. Currently, data augmentation methods in NLP and their effects on the downstream tasks are neither as effective nor as well understood their counterparts in the computer vision domain. In future work, we plan to study principled and automated data augmentation techniques for NLP that would allow extending our supervised contrastive learning objective both semi-supervised and self-supervised learning settings.
REFERENCES
Armen Aghajanyan. Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better fine-tuning by reducing representational collapse. ArXiv, abss/2088.03156 2020.

---

Published as a conference paper at ICLR 2021
Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NeurlPS, 2019.
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for supe-vse learning of speech representations. In NeurIPS, 2020.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In EMNLP, 2015.
Kaidi Cao, Colin Wei, Adrien Gaidon, N. Aréchiga, and Tengyu Ma. Learning imbalanced datasets with labelldiistrbuttion-aaar margin loss. In NeurlPS, 2019.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey E. Hinton, A simple framework for contrastive learning of visual representations. In /CML, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. Big self-supervised models are strong semi-supervised learners. In NeurIPS, 2020b.
Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. Unsupervised cross-lingual representation learning for speech recognition. arXiv preprint arXiv:2006. / 13979, 2020.
E. Cubuk, Barret Zoph. Dandelion Mané, V. Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation strategies from data. 2019 IEEF/CVF Conference Computer Vision and Patlern Recognition (CVPR), pp. 123, 2019.
E. D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated data augmentation with reduced search space. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 3008-301 2020.
Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah A. Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. ArXiv, abs/2002.06305, 2020.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back-translation scale. In EMNLP, 2018. 2018.
Gamaleldin F. Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. Large margin deep networks for classification. In NeurlPS, 2018.
Hongchao Fang and Pengtao Xie. Cert: Contrastive self-supervised learning for language understand- ing. ArXiv, abs/2005.1 12766, 2020.
M. Gutmann and A. Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. J. Mach. Learn. Res., 13:307-361, 2012.
Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. 2019 JEFF/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 1483-1492, 2019.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference Computer Vision and Patlern Recognition (CVPR), pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. 2020 IEEE/CVF Conference on Computer Vision and 2020.
Pattern Recognition (CVPR), pp.
Olivier J. Hénaff, A. Srinivas, J. Fauw, Ali Razavi, C. Doersch, S. Eslami, and A. Oord. Data-efficient image recognition with contrastive predictive coding. ArXiv, 0its/1905.099 2019.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
10

---

Published as a conference paper at ICLR 2021
K. Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, W. Kay, Mustafa Suleyman, and P. Blunsom. Teaching machines to read and comprehend. In NeurIPS, 2015.
Geoffrey E. Hinton, Oriol Vinyals, and J. Dean. Distilling the knowledge in a neural network. In NeurlPS Deep Learning and Representation Learning Workshop, 2015.
R. Devon Hjelm, A. Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), volume 1, pp. 328-339, 2018.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regu- larized optimization. In ACL, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Samna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurlPS, 2020.
Tushar Khot, A. Sabharwal, and Peter Clark. Scitail: A textual entailment dataset from science question answering. In AAAI, 2018.
Byeongchang Kim. Hyunwoo Kim, and Gunhee Kim. Abstractive summarization of reddit posts with multi-level memory networks. NAACL-HLT, 2019.
A. Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual representation learning. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 1920-1929,40 2019.
A. Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Hao Liu and P. Abbeel. Hybrid discrininative-generaavve training via contrastive learning, ArXiv, 0as/2007.09070, 2020.
Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolu- tional neural networks. In ICML, 2016.
Y. Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zeitlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.1 1692, 2019.
I. Misra and L. V. D. Maaten. Self-supervised learning of pretext-invariant representations. 2020 IEFF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 6706-6716 16. 2020.
A. Mnih and K. Kavukcuoglu. Learning word embeddings efficiently with noise-contrastive estima- tion. In NeurIPS, 2013.
Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines. ArXiv, fads/200.04488 2020.
R. Müller, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In NeurIPS, 2019.
Courtney Napoles, Matthew R. Gormley, and Benjamin Van Durme. Annotated gigaword. In 2012.
K. Nar, O. Ocal, S. Sastry, and K. Ramchandran. Cross-entropy loss and low-rank features have responsibility for adversarial examples. ArXiv, abss/19110034. 2019.
Andrew Y. Ng and Michael Jordan. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. In NeurlPS, 2001.

---

Published as a conference paper at ICLR 2021
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, J. Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. 2020.
A. Oord, Y. Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, ArXiv, abs/18072 807.03748 2018.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-IILT 2019; Demonstrations, 2019.
A. Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
Rajat Raina, Yirong Shen, Andrew Y. Ng, and Andrew McCallum. Classification with hybrid generativeddisciiinative models. In NeurlPS, 2003.
Olga Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Zhiheng Huang, A. Karpathy, A. Khosla, M. Bernstein, A. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 11 11:22112222. 2015.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. volume 97 of Proceedings of Machine Learning Research. pp. 5628-5637, Long Beach, California, USA, 09-15 Jun 2019.
PMLR. URL http::/procceedingg
mir press v27 swwnshilllaaaa 1 html
Florian Schroff, D. Kalenichenko, and J. Philbin. Facenet: A unified embedding for face recognition and clustering. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 815-823, 2015.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In NeurlPS, 2016.
Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir D. Bourdev, and Rob Fergus. Training convolutional networks with noisy labels. ICLR, 2015.
Christian Szegedy, V. Vanhoucke, S. loffe, Jon Shlens, and Z. Wojna. Rethinking the inception archi- tecture for computer vision. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 2818 2818-280 2826. 2016.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, 2020.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019. URL
net. forum?ide idd r...2=5554mmmm
Zhirong Wu, Yuanjun Xiong, S. Yu, and D. Lin. Unsupervised feature learning via non- non-parametric instance discrimination. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3733-3742, 2018.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoe Le. Unsupervised data augmentation for consistency training. arXiv: Learning, 2019.
Qizhe Xie, Minh- Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pp. 10687-10698. 2020.
I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi- supervised learning for image classification. arXiv preprint 0rXiv:1955.00544 2019.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. 2019 IEEE/CVF International Conference Computer Vision (ICCV), pp. 6022-6031, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. ArXiv,
2016.
12

---

Published as a conference paper at ICLR 2021
Hongyi Zhang, M. Cissé, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Weinberger, and Yoav Artzi. Revisiting few-sample bert fine-tuning. ArXiv, abs/2006.05987,, 2020.
X. Zhang, J. Zhao, and Y. LeCun. Character-level convolutional networks for text classification. In NeurIPS, 2015.
Zhilu Zhang and Mert R. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. NeurlPS, 2018.
13

---

Published as a conference paper at ICLR 2021
A
APPENDIX
labeled examples
full
100
20
CE
CE+SCL
Figure 3: tSNE plots of learned CLS embedding on SST-2 test set where we have 20, 100 labeled examples, and full dataset respectively, comparing CF with and without SCL. term. Blue: positive examples; red: negative examples.
Model
Loss
SST-2
CoLA
MRPC
RTE
QNLI
MNLI
Avg
RoBERTaLage
CE
96.0++0.4
86.0+-0.5
86.4+22.4
85.5+
90.4++0.8
88.4+1
88.8
RoBERTa
CE + SCL
96.3 0.4
86.1
89,5
85.7
93.9
88.6
90
p-value
0.07
0.63
0.01
0.06
0.16
RoBERTa
CE + CE
96:
86.3++0.4
89:t1
84,9
93.9++0.8
89
89.9
p-value
0.39
0.13
0.01
0.12
RoBERTa
Khosla al, 2020
96++0.3
86.7+1
89.3+11.2
85.2+
92.4+0.7
88.8
88.8+00.9
89.7
p-value
0.4
0.42
0.22
Table Test results on the validation set of GLUE benchmark. We compare tuning RoBERTa- Large with CE with and without SCL, CE+CE and the two-stage method of Khosla et al. (2020). Best hyperparameter configuration is picked based on the average validation accuracy. We report average accuracy across 10 seeds for the model with the best hyperparameter configuration, its standard deviation, and p-values. CE+CE refers to the case where we replace SCL loss with the CE loss but keep normalization and temperature scaling.
14

---

Published as a conference paper at ICLR 2021
Model
Loss
Bsz
SST-2
CoLA
QNLI
MNLI
Avg ups/sec
RoBERTage Basse
CE
16
94.110.5
83.3:0.7
88.210.8
8440.6
15.9
RoBERTA Base
CE + SCL
16
94.910.6
83.7=0.9
92.5=0.4
85.310.5
15.08
RoBERTapae Base
CE + CE
16
94.8+0.77
83.6=0.4
91.6=
85+0.3
15.25
RoBERT
CE
94.2++.44
83.3=0.5
89.2
8440.4
8.43
RoBERTag.c.
CE + SCL
64
94.7+0.2
83,8=0.66
92.610.5
85.710.7
7.44
RoBERTape Rase
CE + CE
64
94.6+t0.7
83.5=0.6
92.1=0.8
85+0.88
7.64
RoBERTape Rase
CE
256
94.1=00.4
84
90
84.4
2.46
RoBERTap TaBERTaaasee
CE + SCL
256
9..2++0.3
84.5=0.5
92.9-0.3
86.6+00.6
1.54
RoBERTagese
CE CE
256
94.3+1..5
83.5=0.3
91.9=0.4
84.610.8
1.77
Table Ablation on performance and fine-tuning speed shown as average updates per second (Avg ups/sec) for fine-tuning RoBERTa-Base with respect the batch size (Bsz). CE+CE refers to the case where we replace SCL. loss with the CE loss but keep 12 normalization and temperature scaling.
Model
Loss
N
SST-2
QNLI
MNLI
RoBERTaLang
CE
20
85.94-2.1
65.0+2.0
39.3+
RoBERTaLang
+ SCL.
20
88.1+-3.3
75.7+
42.744.6
p-value
5e-10
1e-46
1e-8
RoBERTaLasg
CE + CE
20
86.5+2.2
75.1
40.8
p-value
0.03
4e-68
3e-4
RoBERTa Taj.argr
CE
100
91.1+:1.3
81.9400.4
59.2+2.1
Large
RoBERTa, Large
CE + SCL
100
92.8++1.3
82.5+
61.1+33.0
p-value
3e-17
1e-20
2e-4
RoBERTaLars
CE + CE
100
91.7100.5
81.7+0..5
5614.0
p-value
le-4
3e-4
2e-8
RoBERTa Lagge
CE
1000
94.0+10.6
89.2+-0.6
81.4+00.2
RoBERTa Lrgg
CE + SCL
1000
94.1+10.5
89.8
81.5++0.2
p-value
0.6
le-12
0.5
RoBERTaLarg
CE + CE
1000
94:0..7
89.3+1
81
81.2-10.2
p-value
0.78
0.06
0.12
Table 10: Few-shot learning test results on the GLUE benchmark where we have N=20,100,1000
labeled examples for fine-tuning, Reported results are the mean and the standard deviation of the test accuracies of the top models based on the validation accuracy out of random training set samples, along with p-values for each experiment. CE+CE refers to the case where we replace SCL. loss with the CE loss but keep 12 normalization and temperature scaling.
15


Reviews:
[{"review_id": "w1QOCSeoQ1t", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new training objective for fine-tuning pre-trained models: a weighted sum of the classical cross-entropy (CE) and a new supervised contrastive learning term (SCP). The latter uses the (negated) softmax over the embedding distances (i.e. dot products) between a training instance and all other instances in the batch with the same label. In contrast to the more traditional self-supervised contrastive learning (where positive pairs are obtained by applying transformations to the original data instance), there is no data augmentation; two examples with the same label constitute a positive pair.\n\nExperiments on the GLUE benchmark compare the baseline (RoBERTa-Large with CE loss) against the proposed objective (RoBERTa-Large with CE+SCP loss). There are 4 sets of experiments:\n1) When training on the full datasets, results are quite modest (+0.4 increase in accuracy on average over 6 GLUE tasks).\n2) In the few-shot setting, CE+SCP does meaningfully better than the baseline (for instance, when fine-tuning on only 20 data points, CE+SCP improves accuracy by more than 10%); these gains decrease as the dataset size increases.\n3) When the datasets are noisy (effect obtained via back-translation), CE+SCP shines again (for instance, when the degree of corruption is very high, MNLI accuracy goes from ~47% up to ~53%).\n4) Finally, the authors look at domain shift; they fine-tune a model on SST-2, then apply few-shot learning on other sentiment classification datasets. This set of experiments has quite high error margins, so I didn't find it as convincing as 2) and 3).\n\nHere are some questions/suggestions for the authors regarding their experiments:\n\na) \"In all our experiments, [...] we sample half of the original validation set of GLUE benchmark and use it as our test set, and sample ~500 examples for our validation set from the original validation set [...]\" -- Evaluating the models on a *subset* of the validation set makes it harder to compare it against other papers that fine-tune RoBERTa-Large. I think that, at least for Table 2, it would be useful for posterity if you could either i) get the true test scores from the GLUE server, or ii) use part of the training set for validation, and then test on the full dev set, which is more standard practice.\n\nb) \"We run each experiment with 10 different seeds, and pick the top model out of 10 seeds based on\nvalidation accuracy and report its corresponding test accuracy\" -- I am assuming this statement describes how evaluation numbers are reported for a fixed set of hyperparameters. Why do you choose to pick the *top* model as opposed to reporting the *average* accuracy across the 10 runs?\n\nc) \"we observe that our proposed method does not lead to improvement on MNLI [...]. We believe\nthis is due to the fact that number of positive example pairs are quite sparse [...] with batch size 16 [...]. We show evidence for this hypothesis in our ablation studies that we show in Table 3\" -- Then why doesn't Table 3 include MNLI? Am I missing something?\n\nd) This method excels in the few-shot setting, at least compared to the CE baseline. So I think it would be a lot more impactful to focus on this particular use case and convince the reader that CE+SCP is better than some other standard few-shot learning baselines (e.g. meta-learning objectives). I do appreciate that the current message of the paper is crystal-clear (adding a SCP term to the loss leads to better fine-tuning), but I also think that the results in Table 2 are too weak for this somewhat general statement. There is quite a bit of real-estate in the paper that could be re-allocated to something more substantive (e.g. Table 1).\n\nStrengths:\n- The presentation of the paper is extremely clean, and the goal is clear.\n- In the few-shot learning scenario, CE+SCP performs meaningfully better than the CE baseline.\n\nWeaknesses:\n- The main weakness is related to my suggestion d) above. I believe marketing CE+SCP as a general fine-tuning solution with somewhat underwhelming results in Table 2 is a missed opportunity to lead with potentially strong results on few-shot learning. I'm calling the results \"underwhelming\" because there is evidence that a thorough hyperparameter sweep can boost fine-tuning accuracy on GLUE by quite a bit. For instance, Dodge et al. [1] show that fine-tuning BERT carefully can increase SST-2 accuracy by ~2% without any changes in the pre-trained model or fine-tuning objective.\n\n[1] Dodge et al., Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping", "extra_scores": {}, "extra_text": {}}, {"review_id": "hb3eRpkLnrO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work adds a SCL (supervised contrastive learning) loss term during the fine-tuning stage of RoBERTa. The results show that the model with the SCL term and cross-entropy (CE) achieve better GLUE scores than the classic baseline that only uses CE loss, especially when the numbers of supervised training data are small and the data is noisy. \n\nPros:\nThe method is simple and the improvement looks significant under various settings\n\nCons:\nIt is not very clear whether and why SCL loss improves the results (see the detailed comments below).\n\n\nClarity: \nThe text is fluent and the paper cites lots of related work, but the paper does not well explain why the method performs well.\n\nOriginality: \nThe SCL is not novel because it comes from computer vision but this is the first paper I have seen that successfully applies SCL in NLP tasks.\n\nSignificance of this work:\nIf the authors can really show that the improvement comes from SCL, it may become a popular tool in the fine-tuning stage.\n\n\nIt is possible that the source of improvement comes from the temperature tau and l2 normalization instead of SCL loss itself. Both of the tricks could be also applied to CE loss. Thus, could you do control experiments that replace the SCL loss with the CE loss but keeping l2 normalization and retune the tau and lambda. You can report it as CE+CE.\n\nAs shown in Figure 1, the authors suggest that the main reason that SCL loss is better because SCL loss tends to encourage the samples belonging to the same class. However, I believe that CE loss could achieve the same goal and maybe more aggressively than SCL loss. My understanding is that while minimizing CE loss, we encourage each class embedding close to all its samples, so the class embedding tends to become the cluster center of all the points in the class. In the meanwhile, we encourage the samples in each class close to its class embedding, so the samples within the same class will also become closer together, right (e.g., in Figure 3, CE loss could also separate the two classes with a large margin)? The difference is that CE loss encourages the samples close to the average of the samples in the same class, but SCL loss encourages the samples close to each sample in the same class. From this perspective, could you tell us why SCL loss is better (or tell me why this perspective is wrong)? In the introduction, you cite several studies that mention the limitations of cross-entropy loss. I think the motivation of the paper will be much stronger if the authors could briefly and intuitively introduce why CE loss is worse (e.g., what does it mean poor margins and why CE leads to them) and why SCL loss could fix that issue. It would be even better if you can design experiments to demonstrate that (e.g., measure and compare the margins of CE+CE with CE+SCL).\n\nThe effect of hyperparameters on performances is not clear. You mention that lambda is tuned in each downstream task. It is possible that in many applications, lambda is 0. Could you show the lambda value for each downstream application? In addition, could you show the final tau value(s) as well? In the experiment setup, you mention that you report the best model out of 10 seeds. Could you also report the average performance? This will tell us more about how stable this SCL method is. In Table 3, could you also report CE only and CE+CE performance?\n\nI believe the issues could be resolved by conducting more controlled experiments and analyses, so I currently vote weak accept. If the concerns are not addressed well during the rebuttal period, I am very likely to change my vote to rejection.\n\nMinor:\n1. In equation (2), I think putting CE loss for multi-class classification would be more general.\n2. Before the \"Relationship to Self-Supervised Contrastive Learning\", you mention lower temperature creates harder negatives. The meaning of negative here is unclear. I think you mean harder examples here.", "extra_scores": {}, "extra_text": {}}, {"review_id": "WEB_kY5THRD", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes using a combination of two losses in the fine-tuning stage when using a pre-trained model: the standard CE one, plus a supervised contrastive loss SCL (combined linearly via a \\lambda hyperparameter). The supervised contrastive loss uses a normalization summation over the batch examples, with a temperature hyperparameter \\theta.\n\nThe empirical results cover nicely three scenarios: (a) the impact of adding the SCL loss, in the presence of all the fine-tuning data; (b) the impact of adding the SCL loss in few-shot learning scenarios; and (c) the impact of SCL in the presence of training noise (induced via back-translation through German, using a standard WMT-trained MT model).\nThe results as presented are encouraging, and support the main hypothesis of the paper, namely that training with the added SCL loss improves performance over all three scenarios mentioned above.\n\nI have a few suggestions that could be seen as minor, and a few observations that are major.\nMinor ones:\n(i) adding the SCL loss clearly impacts the training speed, yet there is no mention of that (especially as a function of the batch size); in particular, Table 3 would offer the perfect place for mentioning how the training time (for a fixed number of training steps) is affected by the increase batch size, so that the reader can understand both the “upsize” (the improved performance)  as well as the “downsize” (training cost).\n(ii) I could see no mention of the settings for the hyperparameters used (\\lambda and \\theta), nor any ablation experiments that would indicate how their values have been chosen; in the interest of both reproducibility and increased understanding of the value of SCL, please add a sub-section that discusses this issue.\n\nMajor ones:\n(i) It appears that the authors have used the Roberta_large model and run their own experiments with fine-tuning, w/o and w/ the SCL loss, with little regard for reporting against the published numbers for the GLUE tasks; for instance, in Table 2, they report with CE-only to have RTE performance at 85.0, while the Roberta paper shows 86.6 for it; in this case, the CE+SCL at 85.6 no longer looks like a convincing win; a bit different but nevertheless troublesome is the result reported for CoLA, at 86.4 (CE-only); the numbers for CoLA are normally much lower than that, eg the Roberta paper shows 68.0 (CE-only); this disparity throws a lot of doubt over the accuracy of the results reported in Table 2.\n(ii) I commend the authors for showing the variance across their results in both Table 3, 4, and 5; however, it is unclear to me that the claims that the CE+SCL approach is better are being supported by the results. It is not like the CE+SCL gives lower variance, as it is clearly the case that sometimes it is higher than the CE alone; and the fact that, under this high variance, the CE method sometimes performs better CE+SCL puts the whole conclusion under doubt from an empirical standpoint.\n\nOne one result that seems to hold strong is the one for few-shot learning, which appears to support the main hypothesis of the paper. However, the main issues mentioned above would need to be addressed in order to have the paper reach the level of clearing the bar for ICLR publication.\n\n\nRe: References A lot of the references use the Arxiv version for papers that have been peer-reviewed and published. Please fix.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "S5vKvANpZ5h", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n* For the fine-tuning of pre-trained language models, the authors proposed a supervised learning method that combines cross-entropy loss and contrastive loss.\nExperimental results show that the proposed method improves over cross-entropy loss on several classification tasks of the GLUE benchmark set.\nThe proposed method outperforms cross-entropy loss in few-shot learning tasks and noisy datasets generated by English-German and German-English translation.\n\nStrong points\n\n* The proposed loss function is reasonable and the effect of supervised contrastive learning was not reported for NLP applications before, the experimental results are valuable.\n* The paper is well-organized and well-written.\n* Without using extra datasets for fine-tuning, the proposed method consistently improves the baseline method.\n* The generation of noisy examples using the back-translation model in Section 5.3 is an interesting approach to analyze model robustness.\n\nWeak points\n* Although the supervised contrastive learning term as previously proposed in (Khosla et al. 2020), it is not cited in the section.  \n* The benchmark results in Table 2 are not comparable with conventional methods since the experimental setting does not follow the finetuning procedures from prior work (Devlin et al., 2019) which reports the test set performance obtained from GLUE submissions.\n\nDecision reason\n* The technical contribution of this paper is limited since the proposed method is a rather strait-forward expansion of Khosla et al. 2020.  In addition, although it is novel to apply supervised contrastive learning for NLP applications, the impact of these results is also limited because the experimental results are not directly comparable with previous work.\n\nQuestions\n* Why did you \"sample ... taking the label distribution of the original validation set into account\" in Section 4.1?   I am worried that this sampling procedure may make the few-shot task easier.\n\nAdditional Feedback\n* Since the subtraction between two values in percentage is not a ratio,    the percentage is not an appropriate unit for it.  For example \"1.2% improvement on SST-2\" in Section 5.1 should be \"1.2 point improvement on SST-2\".\n* Since Khosla et al. 2020 proposed a two-stage training procedure, supervised contrastive learning at the first stage and the learning of the output layer at the second stage, I would like to see the qualitative comparison with the proposed joint training procedure.\n* Instead of using different original sentences for each T values, it is clear and compact to use the same original sentence for each T values.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces supervised contrastive learning loss on top of typical cross-entropy loss for fine-tuning language model for downstream tasks. While the idea is simple and has been used in vision literature (as pointed out by R1 & R4), its application LM is first introduced in this paper. The experimental gain is small in the regular setting but clearer gains in a few-shot learning setting and noisy training dataset (through back translation) setting. Overall the paper is clearly written and experiments are carefully studied. During the discussion phase, the authors provided results on the full GLUE dataset as well as other ablation studies (e.g., CE+CE recommended by R2), improving the paper.  
Decision: Accept (Poster)