Adversarial Robustness as a Prior for Learned Representations

1 INTRODUCTION . Beyond achieving remarkably high accuracy on a variety of tasks ( Krizhevsky et al. , 2012 ; He et al. , 2015 ; Collobert & Weston , 2008 ) , a major appeal of deep learning is the ability to learn effective representations of data . Specifically , deep neural networks can be thought of as linear classifiers acting on learned feature representations ( also known as feature embeddings ) . A major goal in representation learning is for these embeddings to encode high-level , interpretable features of any given input ( Goodfellow et al. , 2016 ; Bengio et al. , 2013 ; Bengio , 2019 ) . Indeed , learned representations turn out to be quite versatile—in computer vision , for example , they are the driving force behind transfer learning ( Girshick et al. , 2014 ; Donahue et al. , 2014 ) , and image similarity metrics such as VGG distance ( Dosovitskiy & Brox , 2016a ; Zhang et al. , 2018 ) . Still , deep networks ’ feature embeddings exhibit some shortcomings that are at odds with our idealized model of a linear classifier on top of interpretable high-level features . For example , the existence of adversarial examples ( Biggio et al. , 2013 ; Szegedy et al. , 2014 ) —and the fact that they may correspond to flipping predictive features ( Ilyas et al. , 2019 ) —suggests that deep neural networks make predictions based on features that are vastly different from what humans use , or even recognize . ( This message has been also corroborated by several recent works ( Brendel & Bethge , 2019 ; Geirhos et al. , 2019 ; Jetley et al. , 2018 ; Zhang & Zhu , 2019 ) . ) An more direct example of such a shortcoming is pinpointed by Jacobsen et al . ( 2019 ) , who show that one can find image pairs that appear completely different to a human but are nearly identical in terms of their feature embeddings . 1https : //github.com/cantankerousdolphin/robust-learned-representations . Our contributions . Motivated by the limitations of standard representations , we propose using robust optimization ( in particular , adversarial training ( Goodfellow et al. , 2015 ; Madry et al. , 2018 ) ) to enforce a prior on features that models learn ( and thus on their learned feature representations ) . It turns out that the resulting “ robust representations ” ( the embeddings learned by adversarially trained neural networks ) are significantly better-behaved than their standard counterparts . We demonstrate this fact by looking at two well-studied tasks that are typically used to study representations : • Representation inversion ( Section 5 ) : In contrast to standard representations—for which representation inversion is a significant challenge , e.g. , ( Mahendran & Vedaldi , 2015 ) — robust representations are naturally approximately invertible . In particular , adversarially robust networks provide an embedding of the input such that images with similar representations are semantically similar , and the salient features of an image are easily recoverable from its robust feature representation . This property also naturally enables feature interpolation between arbitrary inputs . • Feature visualization ( Section 6 ) : Direct maximization of the coordinates of robust representations suffices to visualize easily recognizable features of the model . This is again a significant departure from standard representations where ( a ) without explicit regularization at visualization-time , feature visualization often produces unintelligible results ; and ( b ) even with regularization , visualized features in the representation layer are scarcely humanrecognizeable ( Olah et al. , 2017 ) . As a result of this property , robust representations enable the addition of specific features to images through direct first-order optimization . The tasks above and their respective applications are illustrated in Figure 2 . Broadly , our results suggest robust optimization as a promising avenue for learning better-behaved image representations . 2 RELATED WORK . Our work studies the feature representations of adversarially robust networks . As discussed in Section 3 , these are networks trained with the robust optimization framework ( Wald , 1945 ; Goodfellow et al. , 2015 ; Madry et al. , 2018 ) and were originally proposed in the context of defending against adversarial perturbations ( Biggio et al. , 2013 ; Szegedy et al. , 2014 ) . Adversarial robustness has been studied extensively in the context of machine learning security ( see e.g. , Carlini & Wagner ( 2017 ) ; Athalye et al . ( 2018b ; a ) ; Papernot et al . ( 2017 ) ) , and as an independent phenomenon ( see e.g. , Gilmer et al . ( 2018 ) ; Schmidt et al . ( 2018 ) ; Jacobsen et al . ( 2019 ) ; Ilyas et al . ( 2019 ) ; Tsipras et al . ( 2019 ) ; Su et al . ( 2018 ) . Several recent works have studied the qualitative properties of robust models . Zhang & Zhu ( 2019 ) find that adversarially robust models behave more predictably in the face of various outof-distribution data , appearing to use more global features . Tsipras et al . ( 2019 ) observe that large adversarial perturbation constructed for robust networks actually resemble instances of the target class . Santurkar et al . ( 2019 ) leverage this fact and use robust classifiers for a wide array of image synthesis tasks . Our work is complementary to this line of research , and focuses on understanding properties of robust representations through the lens of “ benchmark ” representation learning tasks ( namely , inversion and component visualization ) . There is also a large body of work dedicated to studying each of the representation learning tasks we study below ( inversion and feature visualization ) —we have embedded discussions of the related work for each task within the relevant sections . 3 BACKGROUND AND MOTIVATION . 3.1 ADVERSARIAL EXAMPLES AND ROBUST TRAINING . In standard settings , supervised machine learning models are trained by minimizing the expected loss with respect to a set of parameters θ , i.e. , by solving an optimization problem of the form : θ∗ = min θ E ( x , y ) ∼D [ Lθ ( x , y ) ] . ( 1 ) We refer to ( 1 ) as the standard training objective—finding the optimum of this objective should guarantee high performance on unseen data from the distribution . It turns out that deep neural networks trained with the standard objective are vulnerable to adversarial examples ( Biggio et al. , 2013 ; Szegedy et al. , 2014 ) —by changing a natural input imperceptibly , one can easily manipulate the predictions of a deep network to be arbitrarily incorrect . A natural approach ( and one of the most successful ) for defending against these adversarial examples is to use the robust optimization framework : a classical framework for optimization in the presence of uncertainty ( Wald , 1945 ; Danskin , 1967 ) . In particular , instead of just finding parameters which minimize the expected loss ( as in the standard objective ) , a robust optimization objective also requires that the model induced by the parameters θ be robust to worst-case perturbation of the input : θ∗ = arg min θ E ( x , y ) ∼D [ max δ∈∆ Lθ ( x+ δ , y ) ] . ( 2 ) This robust objective is in fact common in the context of machine learning security , where ∆ is usually chosen to be a simple convex set , e.g. , an ` p-ball . Canonical instantiations of robust optimization such as adversarial training ( Goodfellow et al. , 2015 ; Madry et al. , 2018 ) ) have arisen as practical ways of obtaining networks that are invariant to small ` p-bounded changes in the input while maintaining high accuracy ( though a small tradeoff between robustness and accuracy has been noted by prior work ( Tsipras et al. , 2019 ; Su et al. , 2018 ) ( also cf . Appendix Tables 4 and 5 for a comparison of accuracies of standard and robust classifiers ) ) . 3.2 ROBUST TRAINING AS A FEATURE PRIOR . Traditionally , adversarial robustness in the deep learning setting has been explored as a goal predominantly in the context of ML security and reliability ( Biggio & Roli , 2018 ) . In this work , we consider an alternative perspective on adversarial robustness—we cast it as a prior on the features that can be learned by a model . Specifically , models trained with objective ( 2 ) must be invariant to a set of perturbations ∆ . Thus , selecting ∆ to be a set of perturbations that humans are robust to ( e.g. , small ` p-norm perturbations ) results in models that share more invariances with ( and thus are encouraged to use similar features to ) human perception . Note that incorporating human-selected priors and invariances in this fashion has a long history in the design of ML models—convolutional layers , for instance , were introduced as a means of introducing an invariance to translations of the input ( Fukushima , 1980 ) . In what follows , we will explore the effect of the prior induced by adversarial robustness on models ’ learned representations , and demonstrate that representations learned by adversarially robust models are significantly better-behaved , and enable many previously-infeasible modes of direct interaction with feature embeddings . It is worth noting that despite the value of ε used for training being quite small , we find that robust optimization globally affects the behavior of learned representations . As we demonstrate in this section , the benefits of robust representations extend to out-of-distribution inputs and far beyond ε-balls around the training distribution . 3.3 STANDARD AND ROBUST REPRESENTATIONS . Our work is primarily focused on studying the representations of trained neural networks . Throughout this work , we define the representation function R ( · ) as a function induced by a neural network which maps inputs x ∈ Rn to vectors R ( x ) ∈ Rk in the representation layer of that network ( the penultimate layer ) . In what follows , we refer to “ standard representations ” as the representation functions induced by standard ( non-robust ) networks , trained with the objective ( 1 ) —analogously , “ robust representations ” refer to the representation functions induced by ` 2-adversarially robust networks , i.e . networks trained with the objective ( 2 ) with ∆ being the ` 2 ball : θ∗robust = arg min θ E ( x , y ) ∼D [ max ‖δ‖2≤ε Lθ ( x+ δ , y ) ] . 4 EXPERIMENTAL SETUP . We train robust and standard ResNet-50 ( He et al. , 2016 ) networks on the Restricted ImageNet ( Tsipras et al. , 2019 ) and ImageNet ( Russakovsky et al. , 2015 ) datasets . Datasets specifics are in in Appendix A.1 , training details are in in Appendices A.2 and A.3 , and the performance of each model is reported in Appendix A.4 . In the main text , we present results for Restricted ImageNet , and link to ( nearly identical ) results for ImageNet in Appendices ( B.1.4 , B.3.2 ) . Unless explicitly noted otherwise , our optimization method of choice for any objective function will be ( projected ) gradient descent ( PGD ) , a first-order method which is known to be highly effective for minimizing neural network-based loss functions for both standard and adversarially robust neural networks ( Athalye et al. , 2018a ; Madry et al. , 2018 ) .

Summary: The paper presents a study about the representations learned by neural networks trained using robust optimization — a type of optimization that requires the model to be robust to small perturbations in the data. Specifically, the paper presents results of ResNet-50 trained on ImageNet with standard optimization and robust optimization. The paper draws three main insights from studying the learned representations of the standard and robust networks. First, the representation of the robust network is approximately invertible. In other words, when recovering an image by matching the representation of a random image to the representation of a target image by adding noise, the recovered images are semantically similar to the target image; the recovered images look similar to a human. Moreover, this is also demonstrated with images from outside of the distribution of the training data. Second, the representation of the robust network, unlike the representation of standard network, shows semantically meaningful high level features without any preprocessing or regularization. This leads to the final insight, feature manipulation is easier in robust networks. This is demonstrated by adding noise to an initial image in order to maximize the activation of a specific higher level feature and stopping early to preserve most of the other features of the original image.