When does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?

1 Introduction . Image classification has been revolutionized by convolutional neural networks ( CNNs ) . In spite of CNNs ’ generalization power , the lack of adversarial robustness has shown to be a main weakness that gives rise to security concerns in high-stakes applications when CNNs are applied , e.g. , face recognition , medical image classification , surveillance , and autonomous driving [ 1–5 ] . The brittleness of CNNs can be easily manifested by generating tiny input perturbations to completely alter the models ’ decision . Such input perturbations and corresponding perturbed inputs are referred to as adversarial perturbations and adversarial examples ( or attacks ) , respectively [ 6–10 ] . One of the most powerful defensive schemes against adversarial attacks is adversarial training ( AT ) [ 11 ] , built upon a two-player game in which an ‘ attacker ’ crafts input perturbations to maximize the training objective for worst-case robustness , and a ‘ defender ’ minimizes the maximum loss for an improved robust model against these attacks . However , AT and its many variants using min-max optimization [ 12–21 ] were restricted to supervised learning as true labels of training data are required 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . for both supervised classifier and attack generator ( that ensures misclassification ) . The recent work [ 22–24 ] demonstrated that with a properly-designed attacker ’ s objective , AT-type defenses can be generalized to the semi-supervised setting , and showed that the incorporation of additional unlabeled data could further improve adversarial robustness in image classification . Such an extension from supervised to semi-supervised defenses further inspires us to ask whether there exist unsupervised defenses that can eliminate the prerequisite of labeled data but improve model robustness . Some very recent literature [ 25–29 ] started tackling the problem of adversarial defense through the lens of self-supervised learning . Examples include augmenting a supervised task with an unsupervised ‘ pretext ’ task for which ground-truth label is available for ‘ free ’ [ 25 , 26 ] , or robustifying unsupervised representation learning based only on a pretext task and then finetuning the learned representations over downstream supervised tasks [ 27–29 ] . The latter scenario is of primary interest to us as a defense can then be performed at the pretraining stage without needing any label information . Meanwhile , selfsupervised contrastive learning ( CL ) has been outstandingly successful in the field of representation learning : It can surpass a supervised learning counterpart on downstream image classification tasks in standard accuracy [ 30–34 ] . Different from conventional self-supervised learning methods [ 35 ] , CL , e.g. , SimCLR [ 30 ] , enforces instance discrimination by exploring multiple views of the same data and treating every instance under a specific view as a class of its own [ 36 ] . The most relevant work to ours is [ 27 , 28 ] , which integrated adversarial training with CL . However , the achieved adversarial robustness at downstream tasks largely relies on the use of advanced finetuning techniques , either adversarial full finetuning [ 27 ] or adversarial linear finetuning [ 28 ] . Different from [ 27 , 28 ] , we ask : ( Q ) How to accomplish robustness enhance- ment using CL without losing its finetuning ef- ficiency , e.g. , via a standard linear finetuner ? Our work attempts to make a rigorous and comprehensive study on addressing the above question . We find that self-supervised learning ( including the state-of-the-art CL ) suffers a new robustness challenge that we call ‘ crosstask robustness transferability ’ , which was largely overlooked in the previous work . That is , there exists a task mismatch from pretraining to finetuning ( e.g. , from CL to supervised classification ) so that adversarial robustness is not able to transfer across tasks even if pretraining datasets and finetuning datasets are drawn from the same distribution . Different from supervised/semi-supervised learning , this is a characteristic behavior of selfsupervision when being adapted to robust learning . As shown in Figure 1 , our work advances CL in the adversarial context and the proposed method outperforms all state-of-the-art baseline methods , leading to a substantial improvement in both robust accuracy and standard accuracy using either the lightweight standard linear finetuning or end-to-end adversarial full finetuning . Contributions Our main contributions are summarized below . ∂ We propose ADVCL , a unified adversarial CL framework , and propose to use original adversarial examples and high-frequency data components to create robustness-aware and generalization-aware views of unlabeled data . ∑ We propose to generate proper pseudo-supervision stimulus for ADVCL to improve cross-task robustness transferability . Different from existing self-supervised defenses aided with labeled data [ 27 ] , we generate pseudo-labels of unlabeled data based on their clustering information . ∏ We conduct a thorough experimental study and show that ADVCL achieves state-of-the-art robust accuracies under both PGD attacks [ 11 ] and Auto-Attacks [ 37 ] using only standard linear finetuning . For example , in the case of Auto-Attack ( the most powerful threat model ) with 8/255 ` 1-norm perturbation strength under ResNet-18 , we achieve 3.44 % and 3.45 % robustness improvement on CIFAR-10 and CIFAR-100 over existing self-supervised methods . We also justify the effectiveness of ADVCL in different attack setups , dataset transferring , model explanation , and loss landscape smoothness . 2 Background & Related Work . Self-Supervised Learning Early approaches for unsupervised representation learning leverages handcrafted tasks , like prediction rotation [ 38 ] and solving the Jigsaw puzzle [ 39 , 40 ] , geometry prediction [ 41 ] and Selfie [ 42 ] . Recently contrastive learning ( CL ) [ 30 , 33 , 34 , 43–45 ] and its variants [ 31 , 32 , 36 , 46 ] have demonstrated superior abilities in learning generalizable features in an unsupervised manner . The main idea behind CL is to self-create positive samples of the same image from aggressive viewpoints , and then acquire data representations by maximizing agreement between positives while contrasts with negatives . In what follows , we elaborate on the formulation of SimCLR [ 30 ] , one of the most commonly-used CL frameworks , which this paper will focus on . To be concrete , let X = { x1 , x2 , ... , xn } denote an unlabeled source dataset , SimCLR offers a learned feature encoder f✓ to generate expressive deep representations of the data . To train f✓ , each input x 2 X will be transformed into two views ( ⌧1 ( x ) , ⌧2 ( x ) ) and labels them as a positive pair . Here transformation operations ⌧1 and ⌧2 are randomly sampled from a pre-defined transformation set T , which includes , e.g. , random cropping and resizing , color jittering , rotation , and cutout . The positive pair is then fed in the feature encoder f✓ with a projection head g to acquire projected features , i.e. , zi = g f✓ ( ⌧i ( x ) ) for j 2 { 1 , 2 } . NT-Xent loss ( i.e. , the normalized temperature-scaled cross-entropy loss ) is then applied to optimizing f✓ , where the distance of projected positive features ( z1 , z2 ) is minimized for each input x. SimCLR follows the ‘ self-supervised pretraining + supervised finetuning ’ paradigm . That is , once f✓ is trained , a downstream supervised classification task can be handled by just finetuning a linear classifier over the fixed encoder f✓ , leading to the eventual classification network f✓ . Adversarial Training ( AT ) Deep neural networks are vulnerable to adversarial attacks . Various approaches have been proposed to enhance the model robustness . Given a classification model ✓ , AT [ 11 ] is one of the most powerful robust training methods against adversarial attacks . Different from standard training over normal data ( x , y ) 2 D ( with feature x and label y in dataset D ) , AT adopts a min-max training recipe , where the worst-case training loss is minimized over the adversarially perturbed data ( x + , y ) . Here denotes the input perturbation variable to be maximized for the worst-case training objective . The supervised AT is then formally given by min ✓ E ( x , y ) 2D max k k1✏ ` ( x+ , y ; ✓ ) , ( 1 ) where ` denotes the supervised training objective , e.g. , cross-entropy ( CE ) loss . There have been many variants of AT [ 19–21 , 47–50 , 22–25 ] established for supervised/semi-supervised learning . Self-supervision enabled AT Several recent works [ 26–29 ] started to study how to improve model robustness using self-supervised AT . Their idea is to apply AT ( 1 ) to a self-supervised pretraining task , e.g. , SimCLR in [ 27 , 28 ] , such that the learned feature encoder f✓ renders robust data representations . However , different from our work , the existing ones lack a systematic study on when and how self-supervised robust pretraining can preserve robustness to downstream tasks without sacrificing the efficiency of lightweight finetuning . For example , the prior work [ 26 , 27 ] suggested adversarial full finetuning , where pretrained model is used as a weight initialization in finetuning downstream tasks . Yet , it requests the finetuner to update all of the weights of the pretrained model , and thus makes the advantage of self-supervised robust pretraining less significant . A more practical scenario is linear finetuning : One freezes the pretrained feature encoder for the downstream task and only partially finetunes a linear prediction head . The work [ 28 ] evaluated the performance of linear fintuning but observed a relatively large performance gap between the standard linear finetuning and adversarial linear finetuning ; see more comparisons in Figure 1 . Therefore , the problem–how to enhance robustness transferability from pretraining to linear finetuning–remains unexplored . 3 Problem Statement . In this section , we present the problem of our interest , together with its setup . Robust pretraining + linear finetuning . We aim to develop robustness enhancement solutions by fully exploiting and exploring the power of CL at the pretraining phase , so that the resulting robust feature representations can seamlessly be used to generate robust predictions of downstream tasks using just a lightweight finetuning scheme . With the aid of AT ( 1 ) , we formulate the ‘ robust pretraining + linear finetuning ’ problem below : Pretraining : min ✓ Ex2X max k k1✏ ` pre ( x+ , x ; ✓ ) ( 2 ) Finetuning : min ✓c E ( x , y ) 2D ` CE ( ✓c f✓ ( x ) , y ) , ( 3 ) where ` pre denotes a properly-designed robustness- and generalization-aware CL loss ( see Sec . 4 ) given as a function of the adversarial example ( x + ) , original example x and feature encoder parameters ✓ . In ( 2 ) , ✓c f✓ denotes the classifier by equipping the linear prediction head ✓c ( with parameters ✓c to be designed ) on top of the fixed feature encoder f✓ , and ` CE denotes the supervised CE loss over the target dataset D. Note that besides the standard linear finetuning ( 3 ) , one can also modify ( 3 ) using the worst-case CE loss for adversarial linear/full finetuning [ 27 , 28 ] . We do not consider standard full finetuning in the paper since tuning the full network weights with standard cross-entropy loss is not possible for the model to preserve robustness [ 26 ] . Cross-task robustness transferability . Different from supervised/semi-supervised learning , selfsupervision enables robust pretraining over unlabeled source data . In the meantime , it also imposes a new challenge that we call ‘ cross-task robustness transferability ’ : At the pretraining stage , a feature encoder is learned over a ‘ pretext ’ task for which ground-truth is available for free , while finetuning is typically carried out on a new downstream task . Spurred by the above , we ask the following questions : • Will CL improve adversarial robustness using just standard linear finetuning ? • What are the principles that CL should follow to preserve robustness across tasks ? • What are the insights can we acquire from self-supervised robust representation learning ?

Summary: The submission investigates the adversarial robustness of self-supervised pretraining using contrastive learning with the NT-Xent loss [29]. The goal is to improve robustness of the pretraining step such that a linear classifier suffices to be fine-tuned for state-of-the-art performance, and to increase robustness across tasks. The proposed design, AdvCL, extends the loss formulation to contain four designed views (Eq. 9), as well as a supervision stimulus based on K-means clustering [50]. Experimental results show that the resulting system significantly increases adversarial robustness in the self-supervised setting. The paper also contributes a visualization of qualitative results as well as an ablation study of some of the key parameters.