When does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?

1 Introduction . Image classification has been revolutionized by convolutional neural networks ( CNNs ) . In spite of CNNs ’ generalization power , the lack of adversarial robustness has shown to be a main weakness that gives rise to security concerns in high-stakes applications when CNNs are applied , e.g. , face recognition , medical image classification , surveillance , and autonomous driving [ 1–5 ] . The brittleness of CNNs can be easily manifested by generating tiny input perturbations to completely alter the models ’ decision . Such input perturbations and corresponding perturbed inputs are referred to as adversarial perturbations and adversarial examples ( or attacks ) , respectively [ 6–10 ] . One of the most powerful defensive schemes against adversarial attacks is adversarial training ( AT ) [ 11 ] , built upon a two-player game in which an ‘ attacker ’ crafts input perturbations to maximize the training objective for worst-case robustness , and a ‘ defender ’ minimizes the maximum loss for an improved robust model against these attacks . However , AT and its many variants using min-max optimization [ 12–21 ] were restricted to supervised learning as true labels of training data are required 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . for both supervised classifier and attack generator ( that ensures misclassification ) . The recent work [ 22–24 ] demonstrated that with a properly-designed attacker ’ s objective , AT-type defenses can be generalized to the semi-supervised setting , and showed that the incorporation of additional unlabeled data could further improve adversarial robustness in image classification . Such an extension from supervised to semi-supervised defenses further inspires us to ask whether there exist unsupervised defenses that can eliminate the prerequisite of labeled data but improve model robustness . Some very recent literature [ 25–29 ] started tackling the problem of adversarial defense through the lens of self-supervised learning . Examples include augmenting a supervised task with an unsupervised ‘ pretext ’ task for which ground-truth label is available for ‘ free ’ [ 25 , 26 ] , or robustifying unsupervised representation learning based only on a pretext task and then finetuning the learned representations over downstream supervised tasks [ 27–29 ] . The latter scenario is of primary interest to us as a defense can then be performed at the pretraining stage without needing any label information . Meanwhile , selfsupervised contrastive learning ( CL ) has been outstandingly successful in the field of representation learning : It can surpass a supervised learning counterpart on downstream image classification tasks in standard accuracy [ 30–34 ] . Different from conventional self-supervised learning methods [ 35 ] , CL , e.g. , SimCLR [ 30 ] , enforces instance discrimination by exploring multiple views of the same data and treating every instance under a specific view as a class of its own [ 36 ] . The most relevant work to ours is [ 27 , 28 ] , which integrated adversarial training with CL . However , the achieved adversarial robustness at downstream tasks largely relies on the use of advanced finetuning techniques , either adversarial full finetuning [ 27 ] or adversarial linear finetuning [ 28 ] . Different from [ 27 , 28 ] , we ask : ( Q ) How to accomplish robustness enhance- ment using CL without losing its finetuning ef- ficiency , e.g. , via a standard linear finetuner ? Our work attempts to make a rigorous and comprehensive study on addressing the above question . We find that self-supervised learning ( including the state-of-the-art CL ) suffers a new robustness challenge that we call ‘ crosstask robustness transferability ’ , which was largely overlooked in the previous work . That is , there exists a task mismatch from pretraining to finetuning ( e.g. , from CL to supervised classification ) so that adversarial robustness is not able to transfer across tasks even if pretraining datasets and finetuning datasets are drawn from the same distribution . Different from supervised/semi-supervised learning , this is a characteristic behavior of selfsupervision when being adapted to robust learning . As shown in Figure 1 , our work advances CL in the adversarial context and the proposed method outperforms all state-of-the-art baseline methods , leading to a substantial improvement in both robust accuracy and standard accuracy using either the lightweight standard linear finetuning or end-to-end adversarial full finetuning . Contributions Our main contributions are summarized below . ∂ We propose ADVCL , a unified adversarial CL framework , and propose to use original adversarial examples and high-frequency data components to create robustness-aware and generalization-aware views of unlabeled data . ∑ We propose to generate proper pseudo-supervision stimulus for ADVCL to improve cross-task robustness transferability . Different from existing self-supervised defenses aided with labeled data [ 27 ] , we generate pseudo-labels of unlabeled data based on their clustering information . ∏ We conduct a thorough experimental study and show that ADVCL achieves state-of-the-art robust accuracies under both PGD attacks [ 11 ] and Auto-Attacks [ 37 ] using only standard linear finetuning . For example , in the case of Auto-Attack ( the most powerful threat model ) with 8/255 ` 1-norm perturbation strength under ResNet-18 , we achieve 3.44 % and 3.45 % robustness improvement on CIFAR-10 and CIFAR-100 over existing self-supervised methods . We also justify the effectiveness of ADVCL in different attack setups , dataset transferring , model explanation , and loss landscape smoothness . 2 Background & Related Work . Self-Supervised Learning Early approaches for unsupervised representation learning leverages handcrafted tasks , like prediction rotation [ 38 ] and solving the Jigsaw puzzle [ 39 , 40 ] , geometry prediction [ 41 ] and Selfie [ 42 ] . Recently contrastive learning ( CL ) [ 30 , 33 , 34 , 43–45 ] and its variants [ 31 , 32 , 36 , 46 ] have demonstrated superior abilities in learning generalizable features in an unsupervised manner . The main idea behind CL is to self-create positive samples of the same image from aggressive viewpoints , and then acquire data representations by maximizing agreement between positives while contrasts with negatives . In what follows , we elaborate on the formulation of SimCLR [ 30 ] , one of the most commonly-used CL frameworks , which this paper will focus on . To be concrete , let X = { x1 , x2 , ... , xn } denote an unlabeled source dataset , SimCLR offers a learned feature encoder f✓ to generate expressive deep representations of the data . To train f✓ , each input x 2 X will be transformed into two views ( ⌧1 ( x ) , ⌧2 ( x ) ) and labels them as a positive pair . Here transformation operations ⌧1 and ⌧2 are randomly sampled from a pre-defined transformation set T , which includes , e.g. , random cropping and resizing , color jittering , rotation , and cutout . The positive pair is then fed in the feature encoder f✓ with a projection head g to acquire projected features , i.e. , zi = g f✓ ( ⌧i ( x ) ) for j 2 { 1 , 2 } . NT-Xent loss ( i.e. , the normalized temperature-scaled cross-entropy loss ) is then applied to optimizing f✓ , where the distance of projected positive features ( z1 , z2 ) is minimized for each input x. SimCLR follows the ‘ self-supervised pretraining + supervised finetuning ’ paradigm . That is , once f✓ is trained , a downstream supervised classification task can be handled by just finetuning a linear classifier over the fixed encoder f✓ , leading to the eventual classification network f✓ . Adversarial Training ( AT ) Deep neural networks are vulnerable to adversarial attacks . Various approaches have been proposed to enhance the model robustness . Given a classification model ✓ , AT [ 11 ] is one of the most powerful robust training methods against adversarial attacks . Different from standard training over normal data ( x , y ) 2 D ( with feature x and label y in dataset D ) , AT adopts a min-max training recipe , where the worst-case training loss is minimized over the adversarially perturbed data ( x + , y ) . Here denotes the input perturbation variable to be maximized for the worst-case training objective . The supervised AT is then formally given by min ✓ E ( x , y ) 2D max k k1✏ ` ( x+ , y ; ✓ ) , ( 1 ) where ` denotes the supervised training objective , e.g. , cross-entropy ( CE ) loss . There have been many variants of AT [ 19–21 , 47–50 , 22–25 ] established for supervised/semi-supervised learning . Self-supervision enabled AT Several recent works [ 26–29 ] started to study how to improve model robustness using self-supervised AT . Their idea is to apply AT ( 1 ) to a self-supervised pretraining task , e.g. , SimCLR in [ 27 , 28 ] , such that the learned feature encoder f✓ renders robust data representations . However , different from our work , the existing ones lack a systematic study on when and how self-supervised robust pretraining can preserve robustness to downstream tasks without sacrificing the efficiency of lightweight finetuning . For example , the prior work [ 26 , 27 ] suggested adversarial full finetuning , where pretrained model is used as a weight initialization in finetuning downstream tasks . Yet , it requests the finetuner to update all of the weights of the pretrained model , and thus makes the advantage of self-supervised robust pretraining less significant . A more practical scenario is linear finetuning : One freezes the pretrained feature encoder for the downstream task and only partially finetunes a linear prediction head . The work [ 28 ] evaluated the performance of linear fintuning but observed a relatively large performance gap between the standard linear finetuning and adversarial linear finetuning ; see more comparisons in Figure 1 . Therefore , the problem–how to enhance robustness transferability from pretraining to linear finetuning–remains unexplored . 3 Problem Statement . In this section , we present the problem of our interest , together with its setup . Robust pretraining + linear finetuning . We aim to develop robustness enhancement solutions by fully exploiting and exploring the power of CL at the pretraining phase , so that the resulting robust feature representations can seamlessly be used to generate robust predictions of downstream tasks using just a lightweight finetuning scheme . With the aid of AT ( 1 ) , we formulate the ‘ robust pretraining + linear finetuning ’ problem below : Pretraining : min ✓ Ex2X max k k1✏ ` pre ( x+ , x ; ✓ ) ( 2 ) Finetuning : min ✓c E ( x , y ) 2D ` CE ( ✓c f✓ ( x ) , y ) , ( 3 ) where ` pre denotes a properly-designed robustness- and generalization-aware CL loss ( see Sec . 4 ) given as a function of the adversarial example ( x + ) , original example x and feature encoder parameters ✓ . In ( 2 ) , ✓c f✓ denotes the classifier by equipping the linear prediction head ✓c ( with parameters ✓c to be designed ) on top of the fixed feature encoder f✓ , and ` CE denotes the supervised CE loss over the target dataset D. Note that besides the standard linear finetuning ( 3 ) , one can also modify ( 3 ) using the worst-case CE loss for adversarial linear/full finetuning [ 27 , 28 ] . We do not consider standard full finetuning in the paper since tuning the full network weights with standard cross-entropy loss is not possible for the model to preserve robustness [ 26 ] . Cross-task robustness transferability . Different from supervised/semi-supervised learning , selfsupervision enables robust pretraining over unlabeled source data . In the meantime , it also imposes a new challenge that we call ‘ cross-task robustness transferability ’ : At the pretraining stage , a feature encoder is learned over a ‘ pretext ’ task for which ground-truth is available for free , while finetuning is typically carried out on a new downstream task . Spurred by the above , we ask the following questions : • Will CL improve adversarial robustness using just standard linear finetuning ? • What are the principles that CL should follow to preserve robustness across tasks ? • What are the insights can we acquire from self-supervised robust representation learning ?

Summary: The focus of the paper is to make contrastive pre-training (e.g. SimCLR), which has seen great success in recent years, more robust to adversarial attacks. To this end, the authors propose AdvCL, which makes the following modifications to prior contrastive methods: 1) high-frequency components of images are shown to be important for robustness and are leveraged when building the views. 2) augmenting contrastive learning with "pseudo-supervision" obtained by clustering the features further improves performance. The authors evaluate test accuracy on CIFAR10, CIFAR100, STL-10 under adversarial perturbations (AutoAttack and standard PGD) using both linear head fine-tuning and full fine-tuning and compare against recent SOTA methods. AdvCL seems to outperform the rest mostly across the board. Ablative studies are conducted.