Self-supervised Adversarial Robustness for the Low-label, High-data Regime

1 INTRODUCTION . As neural networks tackle challenges ranging from ranking content on the web ( Covington et al. , 2016 ) to autonomous driving ( Bojarski et al. , 2016 ) via medical diagnostics ( De Fauw et al. , 2018 ) , it has becomes increasingly important to ensure that deployed models are robust and generalize to various input perturbations . Unfortunately , despite their success , neural networks are not intrinsically robust . In particular , the addition of small but carefully chosen deviations to the input , called adversarial perturbations , can cause the neural network to make incorrect predictions with high confidence ( Carlini & Wagner , 2017a ; Goodfellow et al. , 2014 ; Kurakin et al. , 2016 ; Szegedy et al. , 2013 ) . Starting with Szegedy et al . ( 2013 ) , there has been a lot of work on understanding and generating adversarial perturbations ( Carlini & Wagner , 2017b ; Athalye & Sutskever , 2017 ) , and on building models that are robust to such perturbations ( Papernot et al. , 2015 ; Madry et al. , 2017 ; Kannan et al. , 2018 ) . Robust optimization techniques , like the one developed by Madry et al . ( 2017 ) , learn robust models by trying to find the worst-case adversarial examples ( by using gradient ascent on the training loss ) at each training step and adding them to the training data . Since Madry et al . ( 2017 ) , various modifications to their original implementation have been proposed ( Zhang et al. , 2019 ; Pang et al. , 2020 ; Huang et al. , 2020 ; Qin et al. , 2019 ) . We highlight the simultaneous work from Carmon et al . ( 2019 ) ; Uesato et al . ( 2019 ) ; Zhai et al . ( 2019a ) ; Najafi et al . ( 2019 ) that pioneered the use of additional unlabeled data using pseudo-labeling . While , theoretically , robustness can be achieved with only limited amount of labeled data , in practice , it remains difficult to train models that are both robust and accurate in the low-label regime . 1 Finally , we note that there has been little work towards learning adversarially robust representations that allow for efficient training on multiple downstream tasks ( with the exception of Cemgil et al. , 2019 ; Kim et al. , 2020 ) . Learning good image representations is a key challenge in computer vision ( Wiskott & Sejnowski , 2002 ; Hinton et al. , 2006 ) , and many different approaches have been proposed . Among them state-of-the-art methods include contrastive methods ( Chen et al. , 2020b ; Oord et al. , 2018 ; He et al. , 2020 ) and latent bootstrapping ( Grill et al. , 2020 ) . However , none of these recent works consider the impact of adversarial manipulations , which can render the widespread use of general representations difficult . As an example , Fig . 1 demonstrates the effect that a non-robust representation has on a content retrieval task , where two seemingly identical query images are matched to widely different images ( i.e. , their nearest neighbors in representation space ) . In this paper , we tackle the issue of learning robust representations that are adversarially robust on multiple downstream tasks in the low-label regime . Our contributions are as follows : • We formulate Bootstrap Your Own Robust Latents ( BYORL ) , a modification of Bootstrap Your Own Latents ( BYOL ) ( Grill et al. , 2020 ) that enables the training of robust representations without the need for any label information . These representations allow for efficient training on multiple downstream tasks with a fraction of the original labels . • Most notably , even with only 1 % of the labels , BYORL comes close to or even exceeds previous state-of-the-art which uses all labels . For example , for ` 2 norm-bounded perturbations of size = 128/255 on CIFAR-10 , BYORL achieves 75.50 % robust accuracy compared to 72.91 % for the previous state-of-the-art using all labels . BYORL reaches 77.61 % robust accuracy when using all available labels ( and additional unlabeled data extracted from 80M-TINYIMAGES ; Torralba et al. , 2008 ) . • Finally , we show that the representations learned through BYORL transfer much better to downstream tasks ( i.e. , downscaled STL-10 ( Coates et al. , 2011 ) and CIFAR-100 ( Krizhevsky et al. , 2014 ) ) than those obtained through pseudo-labeling and standard adversarial training . Importantly , we also highlight that classifiers trained on top of these robust representations do not need to be trained adversarially to be robust . 2 RELATED WORK . Adversarial robustness . Biggio et al . ( 2013 ) and Szegedy et al . ( 2013 ) observed that neural networks , while they achieve high accuracy on test data , are vulnerable to carefully crafted inputs perturbations , called adversarial examples . Since then , there has been several work on building stronger adversarial examples as well as defense against such adversarial examples ( Carlini & Wagner , 2017b ; Athalye & Sutskever , 2017 ; Goodfellow et al. , 2014 ; Papernot et al. , 2015 ; Madry 1In Uesato et al . ( 2018 ) and Carmon et al . ( 2019 ) , robust accuracy drops by 10 % when limiting the number of labels to about 10 % . et al. , 2017 ; Kannan et al. , 2018 ) . Arguably , the most successful approach for learning adversarially robust models is adversarial training as proposed by Madry et al . ( Athalye et al. , 2018 ; Uesato et al. , 2018 ) . This classic version of adversarial training has been augmented in different ways – with changes in the attack procedure ( e.g. , by incorporating momentum ; Dong et al. , 2017 ) , loss function ( e.g. , logit pairing ; Mosbach et al. , 2018 ) or model architecture ( e.g. , using attention ; Zoran et al. , 2020 ) . We also highlight Zhang et al . ( 2019 ) , who proposed TRADES which balances the trade-off between standard and robust accuracy . By construction , to the contrary of our proposed method , all aforementioned approaches use label information and are not capable of learning generic representations that might be useful to multiple downstream tasks . Semi- and self-supervised learning . Since human annotations can be expensive , semi- and selfsupervised learning approaches that leverage both labeled and unlabeled data have been proposed to improve model performance ( Chapelle et al. , 2009 ; Bachman et al. , 2014 ; Berthelot et al. , 2019 ; Laine & Aila , 2017 ; Miyato et al. , 2018 ; Sajjadi et al. , 2016 ; Xie et al. , 2019 ) . A common approach is to train networks to solve a manually-predefined pretext task ( e.g. , predicting the relative location of image patches ) for representation learning , and later use the learned representation for a specific supervised learning task ( Dosovitskiy et al. , 2014 ; Doersch et al. , 2015 ; Noroozi & Favaro , 2016 ) . Recently , contrastive learning that uses different views of multiple augmented images has been an effective tool to learn rich representation from unsupervised data ( Oord et al. , 2018 ; Chen et al. , 2020b ; He et al. , 2020 ; Tian et al. , 2020 ) , as these methods achieve comparable performance to fully-supervised models . While these works focus on improving standard generalization , we leverage representation learning , as proposed by Grill et al . ( 2020 ) , to improve adversarial generalization . Semi- and self-supervised learning for adversarial robustness . Schmidt et al . ( 2018 ) showed that learning adversarially robust models requires more data . As such , adversarial robustness with unlabeled data has recently drawn a lot of attention . We highlight the works by Uesato et al . ( 2019 ) ; Carmon et al . ( 2019 ) ; Zhai et al . ( 2019a ) which leveraged labeled data to train a standard classifier that is in turn used to pseudo-label the remaining unlabeled data . However , as shown by Uesato et al . ( 2019 ) ; Carmon et al . ( 2019 ) ; Zhai et al . ( 2019a ) , when only 10 % of the CIFAR-10 labels are available the robust accuracy drops significantly . In this paper , we focus on improving adversarial robustness in the low-label regime by leveraging unlabeled data ( e.g. , when 1 % –10 % of labels are available ) to build robust representations . The result is a technique that significantly outperforms state-of-the-art pseudo-labeling techniques in the low-label regime and remains competitive with adversarial training when all labels are available . Chen et al . ( 2020a ) also study adversarial pretraining on self-supervised tasks and demonstrate that they can train robust representations . However , to the contrary of our approach , their method does not preserve the robustness of their resulting representations on downstream tasks ( unless robust fine-tuning is used ) . Hendrycks et al . ( 2019 ) combine supervised adversarial training with an additional self-supervised head . They demonstrate that they can improve on standard adversarial training , but do not learn general representations . We also highlight the recent work by Kim et al . ( 2020 ) which combines adversarial training with contrastive learning . Our method reaches comparable robust accuracy , but is more scalable ( as it is not based on contrastive learning which requires large batch sizes ) . To the contrary of Kim et al. , we also study the transferability of robust representations and focus on the low-label regime . 3 METHOD . In this section , we explain BYORL which elegantly combines adversarial training with BYOL . Hence , we start by giving a brief description of adversarial training and BYOL . 3.1 ADVERSARIAL TRAINING . Madry et al . ( 2017 ) formulate a saddle point problem whose goal is to find model parameters θ that minimize the adversarial risk : E ( x , y ) ∼D [ max δ∈S l ( f ( x+ δ ; θ ) , y ) ] ( 1 ) where D is a data distribution over pairs of examples x and corresponding labels y , f ( · ; θ ) is a model parametrized by θ , l is a suitable loss function ( such as the 0− 1 loss in the context of classification tasks ) , and S defines the set of allowed perturbations ( i.e. , the adversarial input set or threat model ) . Several methods ( also known as “ attacks ” ) have been proposed to find adversarial examples ( and effectively solve the inner maximization problem in Eq . 1 ) . Classical adversarial training as proposed by Madry et al . ( 2017 ) uses Projected Gradient Descent ( PGD ) ,2 which replaces the impractical 0− 1 loss l with the cross-entropy loss l̂ and computes an adversarial perturbation δ̂ = δ ( K ) in K gradient ascent steps of size α as δ ( k+1 ) ← projS ( δ ( k ) + α∇δ ( t ) l̂ ( f ( x+ δ ( k ) ; θ ) , y ) ) ( 2 ) where δ ( 0 ) is chosen at random within S , and where projA ( a ) projects a point a back onto a set A . Finally , for each example x with label y , adversarial training minimizes the loss given by LATθ = l̂ ( f ( x+ δ̂ ; θ ) , y ) ≈ max δ∈S l̂ ( f ( x+ δ ; θ ) , y ) ( 3 ) where δ̂ is given by Eq . 2 and l̂ is the softmax cross-entropy loss .

Summary: The paper proposes a new self-supervised technique, Bootstrap Your Own Robust Latents (BYORL), based on an existing technique, BYOL. BYORL proposes to provide adversarially robust representations for low-label regimes. The paper claims that BYORL achieves state-of-the-art performance on CIFAR-10 even with data that is labeled as low as 1%. In fact, the authors highlight that the representations resulted from BYORL avoid the explicit training for adversarial robustness, because they are already robust.