Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning

1 Introduction . Pre-training a deep neural network on a large database and then fine-tuning it on downstream tasks has been a popular training scheme . Recently , contrastive self-supervised learning ( CSL ) has attracted increasing attention on model pre-training , since it does not rely on any hand-crafted annotations but even achieves more promising performance than supervised pre-training on downstream tasks [ 6 , 7 , 20 , 22 , 47 ] . Specifically , CSL leverages unlabeled data to train visual models via contrastive learning , which maximizes the feature similarity for two augmentations of the same instance and minimizes the feature similarity of two instances [ 58 ] . The learned models provide instance-discriminative visual representations that are uniformly scattered in the feature space [ 53 ] . Although there have been substantial CSL studies on model pre-training [ 23 , 47 ] , few have explored the fine-tuning process . The common practice is to directly fine-tune CSL models with the crossentropy loss [ 6 , 13 , 20 ] . However , we empirically ( cf . Table 1 ) find that different fine-tuning methods significantly influence the model performance on downstream tasks , and fine-tuning with only crossentropy is not the optimal strategy . Intuitively , although cross-entropy tends to learn separable features among classes , the resulting model is still limited in its capability for reducing intra-class feature scattering [ 37 , 55 ] that exists in CSL models . Meanwhile , most existing fine-tuning methods [ 33 , 35 ] are devised for supervised pre-trained models and tend to enforce regularizers to prevent the finetuned models changing too much from the pre-trained ones . However , they suffer from the issue of negative transfer [ 9 ] , since downstream tasks are often different from the pre-training contrastive task . In this sense , how to fine-tune CSL models remains an important yet under-explored question . ∗Corresponding to : Yifan Zhang < yifan.zhang @ u.nus.edu > 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Considering that optimizing the unsupervised contrastive loss during pre-training yields models with instance-level discriminative power , we investigate whether applying contrastive learning to fine-tuning would bring further benefits . To answer this , we analyze the contrastive loss during fine-tuning ( cf . Section 3 ) and find that it offers two benefits . First , integrating the contrastive loss into cross-entropy can provide an additional regularization effect , as compared to cross-entropy based fine-tuning , for discriminative representation learning . Such an effect encourages the model to learn a low-entropy feature cluster for each class ( i.e. , high intra-class compactness ) and a high-entropy feature space ( i.e. , large inter-class separation degree ) . Second , optimizing the contrastive loss will minimize the infimum of the cross-entropy loss over training data , which can provide an additional optimization effect for model fine-tuning . Based on the optimization effectiveness as well as the regularization effectiveness on representations , we argue that optimizing the contrastive loss during fine-tuning can further improve the performance of CSL models on downstream tasks . Considering the above benefits , a natural idea is to directly add the contrastive loss to the objective for fine-tuning , e.g. , one recent study [ 18 ] simply uses contrastive learning to fine-tune language models . However , such a method can not take full advantage of contrastive learning , since it ignores an important challenge in contrastive fine-tuning . That is , contrastive learning highly relies on positive/negative sample pairs , but the majority of sample features are easy-to-contrast ( cf . Figure 1 ( a ) ) [ 19 , 56 ] and may produce negligible contrastive loss gradients . Ignoring this makes the method [ 18 ] fail to learn more discriminative features via contrastive learning and thus can not fine-tune CSL models well . In this paper , to better fine-tune CSL models and enhance their performance on downstream tasks , we propose a contrast-regularized tuning approach ( termed Core-tuning ) , based on a novel hard pair mining strategy . Specifically , Core-tuning generates both hard positive and hard negative pairs for each anchor data via a new hardness-directed mixup strategy ( cf . Figure 1 ( b-c ) ) . Here , hard positives indicate the positive pairs far away from the anchor , while hard negatives are the negative pairs close to the anchor . Meanwhile , since hard pairs are more informative for contrastive learning [ 19 ] , Core-tuning further assigns higher importance weights to hard positive pairs based on a new focal contrastive loss . In this way , the resulting model is able to learn a more discriminative feature space by contrastive fine-tuning . Following that , we also explore how to better exploit the learned discriminative feature space in Core-tuning . Previous work has found that the decision boundary simply trained with cross-entropy is often sharp and close to training data [ 52 ] , which may make the classifier fail to exploit the high inter-class separation degree in the discriminative feature space ( cf . Figure 1 ( d ) ) , and also suffer from limited generalization performance . To address this , Coretuning further uses the mixed features to train the classifier , so that the learned decision boundaries can be more smooth and far away from the original training data ( cf . Figure 1 ( e ) ) . The key contributions are threefold . 1 ) To our knowledge , we are among the first to look into the fine-tuning stage of CSL models , which is an important yet under-explored question . To address this , we propose a novel Core-tuning method . 2 ) We theoretically analyze the benefits of the supervised contrastive loss on representation learning and model optimization , revealing that it is beneficial to model fine-tuning . 3 ) Promising results on image classification and semantic segmentation verify the effectiveness of Core-tuning for improving the fine-tuning performance of CSL models . We also empirically find that Core-tuning benefits CSL models in terms of domain generalization and adversarial robustness on downstream tasks . Considering the theoretical guarantee and empirical effectiveness of Core-tuning , we recommend using it as a standard baseline to fine-tune CSL models . 2 Related Work . Contrastive self-supervised learning ( CSL ) . Self-supervised learning is a kind of unsupervised learning method based on self-supervised proxy tasks , e.g. , rotation prediction [ 16 ] , colorization prediction [ 30 ] and clustering [ 60 ] . Recently , CSL has become the most popular self-supervised paradigm , which treats each instance as a category to learn instance-discriminative representations . State-of-the-art CSL methods include InsDis [ 58 ] , MoCo [ 20 ] , SimCLR [ 6 , 7 ] and InfoMin [ 47 ] . Most CSL studies are devoted to network pre-training , but few have explored the fine-tuning process . As an effective data augmentation method , mixup [ 66 ] has recently been applied to instance augmentation for CSL [ 23 , 26 , 31 , 45 ] . Among these methods , the work [ 23 ] uses mixup to generate hard negative pairs for better instance discrimination . However , all these methods focus on unsupervised pre-training and can not accurately generate hard pairs regarding classes . Comparatively , Core-tuning focuses on the fine-tuning of CSL models and can generate accurate hard positive/negative pairs for each class . Note that the hardness-directed mixup strategy in Core-tuning is different from manifold mixup [ 52 ] that can not be directly used to generate hard sample pairs . Pre-training and Fine-tuning . In deep learning , it is a popular scheme to first pre-train a deep neural network on a large database ( e.g. , ImageNet ) and then fine-tune it on downstream tasks [ 35 , 34 ] . Supervised learning is the mainstream method for pre-training [ 27 ] , whereas self-supervised learning is attracting increasing attention since it does not rely on rich annotations [ 6 , 7 ] . Most existing methods for fine-tuning , like L2-SP [ 35 ] and DELTA [ 33 ] , are devised for supervised pre-trained models and tend to enforce some regularizer to prevent the fine-tuned models changing too much from the pre-trained ones . However , they may be unsuitable for contrastive self-supervised models , since downstream tasks are often different from the contrastive pre-training task , leading to negative transfer [ 9 ] . Very recently , one work [ 18 ] explored contrastive learning to fine-tune language models . However , it simply add the contrastive loss to the objective of fine-tuning and can not theoretically explain why it boosts fine-tuning . More critically , it ignores the challenge of hard pair mining in contrastive fine-tuning and thus can not fine-tune CSL models well . 3 Effects of Contrastive Loss for Model Fine-tuning . We start by analyzing the benefits of the contrastive loss during fine-tuning , which will motivate our new method . Before that , we first define the problem and notations . Problem Definition and Notation . This paper studies the fine-tuning of contrastive self-supervised visual models that are pre-trained on a large-scale unlabeled database . During fine-tuning , let { ( xi , yi ) } ni=1 denote the target task dataset with n samples , where xi is an instance with one-hot label yi∈RK and K denotes the number of classes . The neural network model is denoted by G , which consists of a pre-trained feature encoder Ge and a new predictor Gy specific to the target task . Based on the network , we extract visual representations by zi=Ge ( xi ) and make a prediction by ŷi=Gy ( zi ) . Such a contrastive self-supervised model is generally fine-tuned with the cross-entropy loss [ 13 , 20 ] . Following [ 1 ] , we define the random variables of samples and labels as X and Y , and those of embeddings and predictions as Z|X∼Ge ( X ) and Ŷ |Z∼Gy ( Z ) , respectively . Moreover , let pY be the distribution of Y , p ( Y , Z ) be the joint distribution of Y and Z , and pY |Z be the conditional distribution of Y given Z . We define the entropy of Y as H ( Y ) : =EpY [ − log pY ( Y ) ] and the conditional entropy of Y given Z as H ( Y |Z ) : =Ep ( Y , Z ) [ − log pY |Z ( Y |Z ) ] . Besides , we define the cross-entropy ( CE ) between Y and Ŷ byH ( Y ; Ŷ ) : =EpY [ − log pŶ ( Y ) ] and the conditional CE given Z byH ( Y ; Ŷ |Z ) : =Ep ( Y , Z ) [ − log pŶ |Z ( Y |Z ) ] . Before our analysis , we first revisit contrastive loss . Contrastive loss . We use the supervised contrastive loss [ 25 ] for fine-tuning , which is a variant of InfoNCE [ 43 ] . Specifically , given a sample feature zi as anchor , the contrastive loss takes the features from the same class to the anchor as positive pairs and those from different classes as negative pairs . Assuming features are ` 2-normalized , the contrastive loss is computed by : Lcon= − 1 n n∑ i=1 1 |Pi| ∑ zj∈Pi log e ( z > i zj/τ ) ∑ zk∈Ai e ( z > i zk/τ ) , ( 1 ) where τ is a temperature factor , while Pi and Ai denote the positive pair set and the full pair set of the anchor zi , respectively . We next analyze the contrastive loss and find it has two beneficial effects .

Summary: This paper proposes a fine-tuning method called core-tuning for Contrastive Self-supervised Learning (CSL). Core-tuning employs contrastive loss as a regularizer for preserving intra-class features and further improves the finetuned performance. Besides, hard sample mining, focal loss, and mixup are integrated to further boost the performance. The contributions are 1. explore the fine-tuning problem for CSL. 2. provide the theoretical analysis 3. good experiment result.