Adversarial Robustness as a Prior for Learned Representations
Authors: Logan Engstrom
Andrew Ilyas
Shibani Santurkar
Dimitris Tsipras
Brandon Tran
Aleksander Madry
Venue: ICLR 2022 Submitted (2022)
Keywords: adversarial robustness
representation learning

Abstract: An common goal in deep learning is to learn versatile, high-level feature representations of input data. However, standard networks' representations seem to possess shortcomings that, as we illustrate, prevent them from fully realizing this goal. In this work, we show that robust optimization can be re-cast as a tool for enforcing priors on the features learned by deep neural networks. It turns out that representations learned by robust models address the aforementioned shortcomings and make significant progress towards learning a high-level encoding of inputs. In particular, these representations are approximately invertible, while allowing for direct visualization and manipulation of salient input features. More broadly, our results indicate adversarial robustness as a promising avenue for improving learned representations. 

Under review as a conference paper at ICLR 2022
ADVERSARIAL ROBUSTNESS AS A PRIOR FOR LEARNED REPRESENTATIONS
Anonymous authors Paper under double-bling review
ABSTRACT
An important goal in deep learning is learn versatile, high-level feature rep- resentations of input data. However, standard networks' representations seem to possess shortcomings that, goal. In this work, we show that robust optimization can be re-cast as a tool for enforcing prior on the features learned by deep neural networks. It turns out that representations learned by robust models address the aforementioned short- comings and make significant progress towards learning a high-level encoding of inputs. In particular, these representations are approximately invertible, while al- lowing for direct visualization and manipulation of salient input features. More broadly, our results indicate adversarial robustness promising avenue for im- proving learned representations.
illustrate, prevent them from fully realizing this
INTRODUCTION
Beyond achieving remarkably high accuracy variety of tasks (Krizhevsky al. 2012 He et al. 2015 Collobert & Weston 2008), a major appeal of deep learning is the ability to learn effective representations of data. Specifically, deep neural networks classifiers acting on learned feature representations (also known as feature embeddings). A major goal in representation learning is for these embeddings to encode high-level, interpretable features of any given input (Goodfellow et al.) 2016 Bengio et al. 2013 Bengio 2019). Indeed, learned representations turn out to be quite versatile
be thought of linear
computer vision, for example, they are the driving force behind transfer learning (Girshick et al., 2014 Donahue et al.] (2014), and image similarity metries such as VGG distance (Dosovitskiy Brox 2016a) Zhang 2018).
Still, deep networks' feature embeddings exhibit some shortcomings that are at odds with our ide- alized model of linear classifier on top of interpretable high-level features. For example, the existence of adversarial examples (Biggio et al. 2013 SZegedy et al. 2014))and the fact that they may correspond to flipping predictive features Ilyas et al. 2019)) suggests that deep neural net- works make predictions based features that vastly different from what humans use, recognize. (This message has been also corroborated by several recent works (Brendel & Bethge, 2019 Geirhos al. 2019 Jetley a shortcoming is pinpointed by Jacobsen et al. (2019), who show that one can find image pairs that appear completely different human but are nearly identical in terms of their feature embeddings.
2018 Zhang & Zhu 2019).) An
direct example of such
R(x²) R(x)) +
Figure 1: A limitation of standard feature embeddings: it is straightforward to construct pairs of images (I1, Ø2) that appear completely different yet have near-identical representations.

---

Under review as a conference paper at ICLR 2022
Direct feature visualizations
Representation Inversion
Feature manipulation
Add "stripes"
rep(x) from noise
Add "water"
"shells"
"eyespots"
Match rep(x) from noise
Add "red limbs"
"branches"
"leathers"
Interpolations between arbitrary inputs
"Fur"
"stripes"
Figure 2: Sample images highlighting the properties and applications of "robust representations" studied in this work. All of these manipulations use only gradient descent on simple, unregularized. direct functions of the representations of adversarially robust neural networks (Goodfellow et al., 2015 Madry et al. 2018).
Our contributions. Motivated by the limitations of standard representations, we propose using robust optimization (in particular, adversarial training (Goodfellow et al. 2 to enforce prior on features that models learn (and thus on their learned feature representations). turns that the resulting "robust representations" (the embeddings learned by adversarially trained neural networks) are significantly better-behaved than their standard counterparts. We demonstrate this fact by looking at two well-studied tasks that are typically used to study representations:
2018)
Representation inversion (Section 5): In contrast standard representations for which representation inversion is a significant challenge, e.g., (Mahendran & Vedaldi 2015) robust representations are naturally approximately invertible. In particular, adversarially robust networks provide an embedding of the input such that images with similar represen- tations are semantically similar, and the salient features of an image are easily recoverable from its robust feature representation. This property also naturally enables feature interpo- lation between arbitrary inputs.
Feature visualization (Section 6): Direct maximization of the coordinates of robust rep- resentations suffices visualize casily recognizable features of the model. This is again a significant departure from standard representations where (a) without explicit regulariza- tion at visualization-timee feature visualization often produces unintelligible results; and (b) even with regularization, visualized features in the representation layer are scarcely human- recognizeable (Olah et al., (2017). As result of this property, robust representations enable the addition of specific features to images through direct first-order optimization.
The tasks above and their respective applications are illustrated in Figure Broadly, our results sug- gest robust optimization promising avenue for learning better- beheeebe image representations.
RELATED WORK
Our work studies the feature representations of adversarially robust networks. As discussed Sec- tion 3 these are networks trained with the robust optimization framework (Wald) 1945) Goodfellow ctt al. 2015 Madry et al., 2018) and were originally proposed in the context of defending against adversarial perturbations (Biggio et al. 2013 Szegedy et al., 2014). Adversarial robustness has been studied extensively in the context of machine learning security (see e.g.. Carlini & Wagner (2017): Athalyy al. (2018b)al a); Papernot al. (2017)), and independent phenomenon (see e.g., Gilmer et al. 2018); Schmidt et al. 2018);
Jacossen et al. (2019); Ilyas et al. (2019); Tsipras
al.. 2019) Su et al. (2018).
2

---

Under review as a conference paper at ICLR 2022
Several recent works have studied the qualitative properties of robust models. Zhang & Zhu (2019) find that adversarially robust models behave more predictably in the face of various out- of-distribution data, appearing to use more global features. Tsipras et al. (2019) observe that large adversarial perturbation constructed for robust networks actually resemble instances of the target class. [Santurkar al. (2019) leverage this fact and use robust classifiers for wide array of image synthesis tasks. Our work is complementary to this line of research, and focuses on understanding properties of robust representations through the lens of "benchmark" representation learning tasks (namely, inversion and component visualization).
There is also a large body of work dedicated to studying each of the representation learning tasks we study below (inversion and feature visualization) we have embedded discussions of the related work for each task within the relevant sections.
BACKGROUND AND MOTIVATION
3.1 ADVERSARIAL EXAMPLES AND ROBUST TRAINING
In standard settings, supervised machine learning models are trained by minimizing the expected loss with respect to set of parameters 0, i.e., by solving optimization problem of the form:
0" min E(E,))> [Co v)]-
(1)
We refer to [ as the standard training abjective-findeee the optimum of this objective should guarantee high performance on unseen data from the distribution.
turns out that deep neural networks trained with the standard objective are vulnerable to adversar- ial examples (Biggio et al. 2013 Szzeedy al. b014)-by changing natural input imperceptibly, one can easily manipulate the predictions of a deep network to be arbitrarily incorrect.
A natural approach (and of the most successful) for defending against these adversarial amples is to use the robust optimization framework: a classical framework for optimization in the presence of uncertainty (Wald] 1945) Danskin 1967). In particular, instead of just finding parame- ters which minimize the expected loss (as in the standard objective), robust optimization objective also requires that the model induced by the parameters be robust to worst-case perturbation of the input:
arg min B(
max Cx BE
(x + +8,))
(2)
This robust objective is in fact common in the context of machine learning security, where A is usually chosen to be simple convex set, e.g., an fp-ball. Canonical instantiations of robust op- timization such as adversarial training (Goodfellow et al., 2013 Madry et al., (018)) have arisen as practical ways of obtaining networks that are invariant small bo-bounded changes in the input while maintaining high accuracy (though a small tradeoff between robustness and accuracy has been noted by prior work (Tsipras et al. 2019 Su et al., 2018) (also ef. Appendix Tables 4 and for a comparison of accuracies of standard and robust classifiers)..
3.2 ROBUST TRAINING AS A FEATURE PRIOR
Traditionally, adversarial robustness in the deep learning setting has been explored as a goal pre- dominantly in the context of ML security and reliability (Biggio & Roli 2018).
In this work, we consider an alternative perspective on adversarial robustness we cast it as a prior on the features that can be learned by model. Specifically, models trained with objective (2) must be invariant to a set of perturbations A. Thus, selecting to be set of perturbations that hu- mans are robust to (c.g., small no-norm perturbations) results in models that share more invariances with (and thus are encouraged to use similar features to) human perception. Note that incorpo- rating human-selecter priors and invariances in this fashion has long history in the design of ML condels-convolutions layers, for instance, were introduced as a means of introducing an invariance to translations of the input (Fukushimal 1980).
In what follows, we will explore the effect of the prior induced by adversarial robustness on models' learned representations, and demonstrate that representations learned by adversarially robust models
3

---

Under review as a conference paper at ICLR 2022
significantly better-behaved, and enable many nneviously-infessbee modes of direct interaction with feature embeddings. is worth noting that despite the value of E used for training being quite small, we find that robust optimization globally affects the behavior of learned representations. As we demonstrate in this section, the benefits of robust representations extend to out-of-distribution inputs and far beyond e-balls around the training distribution.
3.3 STANDARD AND ROBUST REPRESENTATIONS
Our work is primarily focused on studying the representations of trained neural networks. Through- out this work, we define the representation function R(-) as function induced by neural network which maps inputs x E IR" to vectors R(x) E IR* in the representation layer of that network (the penultimate layer). In what follows, refer to "standard representations" functions induced by standard (mon-robust) networks, trained with the objective (T) analogously, "robust representations" refer the representation functions induced by (2-adversarially robust net- works, i.e. networks trained with the objective (2) with A being the lg ball:
the representation
Tobbust arg min E (xxg))
max Lo(a + 6, 6.9) [18l|aae
4 EXPERIMENTAL SETUP
We train robust and standard ResNet-50 (He ct al. 2016) networks on the Restricted Ima- geNet (Tsipras et al. 2019) and ImageNet (Russakovsky al. 2015) datasets. Datasets specifics are in in Appendix I. training details are in in Appendices A.2 and A.3 and the performance of each model reported in Appendix AA.4] In the main text, we present results for Restricted ImageNet, and link to (nearly identical) results for ImageNet in Appendices B.3.22
Unless explicitly noted otherwise, our optimization method of choice for any objective function will be (projected) gradient descent (PGD), a first-order method which is known to be highly effective for minimizing neural network-bast -baeed loss functions for both standard and adversarially robust neural networks (Athalye et al. 2018a; Madry et al. 2018)
INVERTING ROBUST REPRESENTATIONS
A common tool for understanding the semantic content of a feature representation is inversion i.e.. reconstructing image using only data from its feature embedding (Mahendran & Vedaldi 2015
Dosovitskiy & Brox 2016b; Ulyanov et al. 2017). As discussed in the introduction. however, for standard deep networks, given any input x, straightforward to find another input that looks entirely different but has nearly the same representation (c.f. Figure IT. This makes representation inversion a difficult problem, since images are not identifiable from their feature representations alone. Indeed, the "naive" approach to representation inversion, wherein one inverts æ by solving
Tine To arg arg min
[[(xo + 8) R(x)]l2.
5
tends to find solutions Toe that bear very little resemblance to x (Mahendran & Vedaldi 2015). and instead look visually similar to the starting point of the optimization zo (which can be any arbitrary image or even random noise).
Related work and motivation. As result of the phenomenon described above, methods for inverting learned representations typically either impose a "natural image" prior on the inverted image (Mahendran & Vedaldi 2015; Yosinski et al. [0155] Ulyanov et al. 2017), or train a separate model altogether to perform the inversion (Kingma & Welling 2015 Dosovitskiy & Brox 2016b In either case, however, these methods introduce additional information into the inversion process beyond that which is contained in feature embedding, making the inversions not fully faithful to the model. In particular, becomes difficult to disentangle the information that is actually contained in models" feature embeddings from that which is imposed externally by the inversion method.
4

---

Under review as a conference paper at ICLR 2022
More broadly, lack of invertibility runs somewhat counter to the idea that learned representations capture a relevant set input features. After all, if the representation function was truly extracting "high-level" features of the imput conceptualize them, semantically dissimilar images should (by definition) have different representations. We now show that the state affairs is greatly im- proved for robust representations.
Source (a)
Target (x)
Robust Xine
Scandard Nene
Figure 3: Representation inversions for robust and standard models trained on the Restricted Ima- geNet dataset. We minimize (3) to find an Fine which matches (in f2-distance) the representation of the target image (x) starting from each corresponding source image zo (top row) for an adversar- ially trained (second row) and standard (third row) model respectively. When inverting the robust representation, regardless of the optimization starting point Zo. the resulting inversions percep- tually similar to the target image; contrast, the results inverting the standard model appear more similar the (arbitrary) source image used the seed for the optimization. Additional results in Appendix B.I and similar results for ImageNet are Appendix B.1.4
Test set
Out-of-distribution
Result Target
Figure 4: Target: random images from the test set (col. 1-5) and from outside of the training distribution (6-10); Result: images obtained from optimizing inputs (using Gaussian noise as the source image) to minmmize fy-distance to the representations of the corresponding image in the top row. More examples appear in Appendix B.I
Robust representations are (approximately) invertible out-of-the-box. It turns out that in sharp contrast to what we observe for standard models, the images resulting from minimizing (3) for robust models are actually semantically similar to the original (target) images whose representation being matched, and this behavior is consistent across multiple samplings of the starting point (source image) Iı (cf. Figure 3
This inversion property holds even for out-of-distribution inputs, demonstrating that robust repre- sentations can represent features beyond those that are relevant for the specific classification task. In particular, we repeat the inversion experiment (simple minimization distance in representation space) using images from classes not present in the original dataset used during training (Figure right) and structured random patterns (Figure 13 in Appendix B.I.: the reconstructed images con- sistently resemble the targets.
5

---

Under review as a conference paper at ICLR 2022
In fact, the contrast between the invertibilityy of standard and robust representations is even stronger. To illustrate this, we will attempt to match the representation of a target image while staying close to the starting image of the optimization in pixel-wise 2--norm (this is equivalent to putting norm bound on 6 in objective ( With standard models, we can consistently get close to the target image in representation space, without moving far from the source image x1. On the other hand, for robust models, we cannot get close to the target representation while staying close to the source image-this is illustrated quantitatively in Figure This indicates that for robust models, semantic similarity may in fact be necessary for representation similarity (and is not, for instance, merely an artifact of the local robustness induced by robust optimization).
Min distance in rep. space
1.2
Robust network
Standard network
0.8
for robust opt. (Eq. 2
0.6
0.4
0.2
0
21
23 Allowed l distance from source image
2
27
29
Figure 5: Optimizing objective 3 with an E2-norm constraint around the source image. On the a-axxs is the radius of the constraint set, and on the y-axis is the distance in representation space between the minimizer of objective (3) within the constraint set and the target image, normalized by the norm of the representation of the target image: i.e., a point (I;, yı) on the graph corresponds to Notably, we are unable to closely match
[R(x + 5) R( (lee)))ll//// [[[rtrr))ll ( Y/ = min the representation of the target image for the robust network until the norm constraint grows very large, and in particular much larger than the norm of the perturbation that the model is trained to be robust against (s in objective (2) Shown are 95% confidence intervals over random choice of source and target images.
Additional related work. We have already discussed related work in inverting standard represen- tations. In an orthogonal direction, is possible to construct models that are analytically invertible by construction (Dinh et al.] 2014 2017 Jacobsen et al. 2018 Behrmann et al.) 2018). However, the representations learned by these models do not seem to perceptually meaningful (for instance, interpolating between points in the representation space does not lead to perceptual input space inter- polations (Jacobsen et al. 2018), in contrast to robust representations, cf. Appendix (A.5). Another important distinction between the inversions shown here and invertible networks is that the latter are an exactly invertible map from Rd + Rd, while the former shows that we can approximately recover the original input in Rdd from "compressed" representation in Rd for < d.
DIRECT FEATURE VISUALIZATION
In the previous section, we showed how adversarial training enables direct (approximate) input inversion without including extra information at inversion-time. We now explore another common technique for studying representations (namely, feature visualization (Olah al., (017)) and show that here too, robust representations behave significantly better than their standard counterparts.
In optimization based basedfeatunr feature visualization (Olah et al., (017). we maximize a specific feature (com- ponent) in the representation with respect to the input, in order to obtain insight into the role of the feature classification. Concretely, given some E [k) denoting component of the representation vector, we use gradient descent to find an input x' that maximally activates it, i.c. we solve:
To + arg max R(zu + 5);
(4)
for various starting points 2o which might be random images from D random noise.

---

Under review as a conference paper at ICLR 2022
Visualization "fails" for standard networks. For standard networks, optimizing the objective often yields unsatisfying results. While we can easily find images for which the 1th component of R(-) is large (and thus the optimization problem is tractable). these images tends to look meaningless to humans, often resembling the starting point of the optimization. Even when these images are non-trivial, they tend contain abstract, hard-to-discern patterns (c.f. Figure [ (bottom)). As result, just as was the case for representation inversion, state-of-the-art feature visualization methods typically regularize objective (4) with terms that encourage more compelling visualizations. These methods include applying random transformations during the optimization process (Mordvintsev
2015 Olah al., 2017). leveraging deep generative models (Nguyen et al., 2015: 2016
2017
or post-processing the input or gradients (Oygard 2015 Tyka 2016).
Maximizing different coordinates (i)
Sceds (xp)
Robust
Standard
Figure 6: Correspondence between image-level patterns and activations learned by standard and robust models on the Restricted ImageNet dataset. Starting from randomly chosen seed inputs (noise/images), we use PGD to find inputs that (locally) maximally activate a given component of the representation vector (cf. Appendix [A.6.I] for details). In the left column we have the seed inputs To (selected randomly), and in subsequent columns we visualize the result of the optimiza- tion i.e,,'', x', for different activations, with each row starting from the same (far left) input 2o for (top): robust (adversarially trained) and (bottom): a standard model. Additional visualizations in Appendix B.3 and similar results for ImageNet in B.3.2
As was the case for inversion, however, such regularization comes with a few well-known (and widely-studied within the feature visualization literature) disadvantages. First, when one introduces prior information about what makes images visually appealing into the optimization process, be- comes difficult disentangle the effects of the actual model from the effect of the prior information introduced through regularizatior Furthermore, while adding regularization does improve the vi- sual quality of the visualizations, the components of the representation still cannot be shown to correspond to any recognizable high-level feature. Indeed, Olah et al.) (2017) note that in the rep- resentation layer of a standard GoogLeNet, "Neurons not seem to correspond to particularly meaningful semantic ideas" the corresponding feature visualizations are reproduced in Figure 7
Robust representations allow for direct visualization of human-recognizath features. For robust representations, however, we find that easily recognizable high-level features emerge from optimizing objective 4 directly, without any regularization results of this maximization in Figure 6 (top): coordinates consistently represent the same concepts across different choice of starting input Zo (both in and out of distribution). Furthermore, these concepts are not merely an artifact of our visualization process, as they consistently appear in the test-set inputs that most strongly activate their corresponding coordinates (Figure
post-processing. We present the
II fact, model explanations that enforce priors for purposes of visual appeal have heen often found to have little do with the data the model itself (Adebayo et al. 2018).
7

---

Under review as a conference paper at ICLR 2022
Figure 7: Figure reproduced from (Olah al. 2017] a (strongly regularized) visualization few components of the representation layer of GoogLeNet. While regularization (as well as Fourier parameterization and colorspace decor- relation) yields visually appealing results, the visualization does not reveal consistent seman- tic concepts. The situation is significantly worse without regularization (cf. Figure bottom).
Figure 8: A visualization of the first four com- ponents of the representation layer of VGGI6 when regularization via random jittering and ro- tation is applied. Figure produced using the Lu- cid" visualization library.
In fact, we observe that the unregularized feature visualizations from robust representations are sig- nificantly more discemable than even regularized visualizations of standard ones. Specifically, we provide examples of representation-layee visualizations for VGGI6 (which we found qualitatively best among modern architectures) regularized with jittering and random rotations in Figure While these visualizations certainly look better qualitatively than their unregularized counterparts in Fig- ure (6 (bottom), there remains a large gap quality and discernability between these regularized visualizations and the unregularized robust visualizations in Figure (top).
Activation 444 ("long fish")
Activation 939
("insect legs")
Most activated
Most activated
Maximized from noise
Least activated
Maximized from noise
Least activated
Figure 9: Maximizing inputs x' (found by solving with Tn being a gray image) and most least activating images (from the test set) for two random activations of a robust model trained on the Restricted ImageNet dataset. For each activation, we plot the three images from the validation set that had the highest or lowest activation value sorted by the magnitude of the selected activation.
Natural consequence: feature manipulation. The ability to directly visualize high-level, recog- nizable features reveals another application of robust representations, which refer as feature manipulation. Consider the visualization objective (4) shown in the previous section. Starting from some original image, optimizing this objective results in the corresponding feature being introduced in continuous manner. It is hence possible to stop this process relatively early to ensure that the content of the original image is preserved. As heuristic, we stop the optimization process as soon as the desired feature attains a larger value than all the other coordinates of the representation. We visualize the result of this process for a variety of input images Figure T0 where "stripes" or "red limbs" are introduced seamlessly into images without any processing regularization
Additional related work. The latent space of generative adversarial networks (GANs) Goodfel- low et al. 2014) tends to allow for "semantic feature arithmetic" (Radford et al. 2016 Larsen al. 2016) (similar to that word2vec embeddings (Mikolov et al.] 2013)) where one can manipulate salient input features using latent space manipulations. In similar vein, one utilize an image- to-image translation framework to perform such manipulation (e.g. transforming horses to zebras), although this requires task-specific dataset and model (Zhu al. 2017). It possible utilize the
We repeat this process with many additional random images and random features in Appendix B.4
8

---

Under review as a conference paper at ICLR 2022
"Stripes" (selected)
"Red limbs" " (random)
Figure 10: Visualization of the results from maximizing chosen (left) and random (right) rep- resentation coordinate starting from random images for the Restricted ImageNet dataset. In each figure, the top has the initial images, and the bottom row has a feature added. Additional exam- ples in Appendix B.4
deep representations of standard models to perform semantic feature manipulations; however such methods tend to either only perform well on datasets where the inputs are center-aligned (Upchurch al. 2017), or are restricted small set of manipulations (Gatys et al., 2016).
CONCLUSION
We show that the learned representations of robustly trained models align much more closely with our idealized view of neural network embeddings extractors of high-level features. After high- lighting certain shortcomings of standard deep networks and their representations, we demonstrate that robust optimization can actually be viewed inducing prior over the features that models are able to learn. In this way, one can view the robust representations that result from this prior as feature extractors that are more aligned with human perception.
In support of this view, we demonstrate that robust representations overcome the challenges identi- fied for standard representations: they approximately invertible, and moving towards an image in representation space seems to entail recovering salient features of that image in pixel space. Further- more, show that robust representations can be directly visualized with first-order methods with- out the need for post-processing or regularization, and also yield much more human-understandable features than standard models (even when they are visualized with regularization). These two prop- erties (inversion and direct feature visualization). in addition to serving as illustrations of the benefits of robust representations, also enable direct modes of input manipulation such as feature addition.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, lan Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Neural Information Processing Systems (NeurIPS), 2018.
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give false sense of security: Circumventing defenses adversarial examples. In International Conference on Machine Learning (ICML), 2018a.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In International Conference on Machine Learning (ICML), 201
David Bau, Jun- Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, and Antonio Torralba. Inverting layers of a large generator. In ICLR Debugging Machine Learning Models Workshop, 2019.
Jens Behrmann, Will Grathwohi, Ricky Q. Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In arXiv preprint arXiv:1811.00995 2018.
Y. Bengio, A. Courville, and P. Vincent. Representation learning: review and new perspectives. 2013.
Yoshua Bengio. Talk abstract: Learning high-level representations for agents, 2019. URL
htw.://ewwaaaaaa.eeeeeeeeee////e bengio learning high Level wwwpesentations_see....... agentss Abstract for talk given at MIT.
Battista Biggio and Fabio Roli. learning. 2018.
Wild patterns: Ten years after the rise of adversarial machine
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Smdic Pavel Laskov, Gior- gio Giacinto, and Fabio Roli. Evasion attacks against machine learning test time. Joint European conference machine learning and knowledge discovery in databases (ECML-KDD), 2013.
Wieland Brendel and Matthias Bethge. Approximating CNNs with bag-of-local-featuee models works surprisingly well on imagenet. In International Conference on Learning Representations (ICLR), 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations (ICLR), 2019.
Nicholas Carlini and David Wagner. Adversarial examples not easily detected: Bypassing ten detection methods. In Workshop on Artificial Intelligence and Security (AlSec), 2017.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pp. 160-167, 2008
John M. Danskin. The Theory of Max-Min and its Application to Weapons Allocation Problems. 1967.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. In arXiv preprint 1rXiv:/410.888 /6, 2014.
Laurent Dinh, Jascha Sohl-Dicksteen, and Samy Bengio. Density estimation using real nvp. In International Conference on Learning Representations (ICLR), 2017.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Inter- national conference on machine learning (ICML). 2014.
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based deep networks. In neural information processing systems (NeurIPS), 2016a.
10

---

Under review as a conference paper at ICLR 2022
Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks. In Computer Vision and Pattern Recognition (CVPR), 2016b.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, and Dimitris Tsipras. Robustness (python li- brary), 2019. URL httpp:://itthuii htt.://oo////aa./ee http://ddylaabbobuuttoet
Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological cybernetics, 1980.
Lcon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In computer vision and pattern recognition (CVPR), 2016.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias im- proves accuracy and robustness. In International Conference on Learning Representations, 2019.
Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S Schoenholz, Maithra Raghu, Martin Watten- berg, and lan Goodfellow. Adversarial spheres. In Workshop of International Conference on Learning Representations (ICLR), 2018.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for curate object detection and semantic segmentation. In computer vision and pattern recognition (CVPR), pp. 580-587, 2014.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In neural information process- ing systems (NeurIPS), 2014.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR), 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In international conference computer vision (ICCV), 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Andrew Ilyas, Shibani Santurkar, Logan Engstrom, Brandon Tran, and Aleksander Madry. Ad- versarial examples are not bugs, they are features. In Neural Information Processing Systems (NeurIPS), 2019.
Jörn-Henrik Jacobsen, Arnold W.M. Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. In International Conference Learning Representations (ICLR), 2018.
Jorn-Henrik Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge. Excessive invariance
causes adversarial vulnerability. In International Contemporary on Learning Representations, 2019.
Saumya Jetley, Nicholas Lord, and Philip Torr. With friends like these, who needs adversaries? In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2015.
Alex Krizhevsky. Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances Neural Information Processing Systems (NeurIPS), 2012.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using a learned similarity metric. In International Conference on Machine Learning (ICMI), 2016.
11

---

Under review as a conference paper at ICLR 2022
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In computer vision and pattern recognition (CVPR), 2015.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances neural information pro- cessing systems, pp- 31 11-3119, 2013.
Alexander Mordvintsev, Christopher Olah, and Mike Tyka.
Inceptionism: Going
deeper into neural networks, 2015. URL httpps://aa //ai, /twp://g///ooog//////ooooooooooo beooleeblcse c0tp inceptionn ini am-goinge new---oo-n--deeee-----e---- inteerr-ittte-euureee html
Alexander Mordvintsev, Nicola Pezzotti, Ludwig Schubert, and Chris Olah. Differentiable image parameterizations. In Distill, 2018.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High con- fidence predictions for unrecognizable images. In Conference on computer vision and pattern recognition (CVPR), 2015.
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Neural Informa- tion Processing Systems (NeurIPS), 2016.
Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. Plug & play generative networks: Conditional iterative generation of images in latent space. Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. In Distill, 2017.
Audun Oygard. Visualizing googlenet classes, 2015, URL https:: http:://www.audunn/ com/
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Asia Conference on Computer and Communications Security, 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference on Learning Repre- sentations (ICLR), 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Fei- Fei. ImageNet Large Scale Visual Recognition Challenge. In International Journal of Computer Vision (IJCV), 2015.
Shibani Santurkar, Dimitris Tsipras, Brandon Tran, Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Image synthesis with single (robust) classifier. Neural Information Processing Sys- tems (NeurIPS), 2019.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adver- sarially robust generalization requires more data. In Advances in Neural Information Processing Systems (NeurIPS), 2018
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015.
Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi, Pin- Yu Chen, and Yupeng Gao. Is robustness the cost of accuracy? a comprehensive study on the robustness of 18 deep image classification models. In European Conference on Computer Vision (ECCV), 2018.
12

---

Under review as a conference paper at ICLR 2022
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representa- tions (ICLR), 2019.
Mike Tyka. Class visualization with bilateral filters, 2016. URL bttps://ntyka.gg.tt (tteepprraamm2011102/66666616612
vis html
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In ArXiv preprint arXiv: 2017.
Paul Upchurch, Jacob Gardner, Geoff Pleiss, Robert Pless, Noah Snavely, Kavita Bala, and Kilian Weinberger. Deep feature interpolation for image content changes. conference on computer vision and pattern recognition (CVPR), 2017.
Abraham Wald. Statistical decision functions which minimize the maximum risk. In Annals of Mathematics, 1945.
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. In arXiv preprint arXiv:/506.065/1 2015.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Computer Vision and Pattern Recognition (CVPR), 2018
Tianyuan Zhang and Zhanxing Zhu. Interpreting adversarially trained convolutional neural net- works. In International Conference on Machine Learning (ICML), 2019.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image trans- lation using cycle-consistent adversarial networks. international conference on computer vi- sion(ICCV), 2017.
13

---

Under review as a conference paper at ICLR 2022
A EXPERIMENTAL SETUP
In this section we describe the elements of experimental setup. Code for reproducing our results using the robustness library (Engstrom et al. 2019) can be found at athttps:///itte /itphbb/ggttub.t. com/
snappymanateer/robuse learned representaiions
A.1 DATASETS
In the main text, we perform all our experimental analysis on the Restricted ImageNet dataset (Tippas et al. 2019) which is obtained by grouping together semantically similar classes from ImageNet into super-classes shown in Table Attaining robust models for the complete Im- ageNet dataset known to be challenging, both due the hardness of the learning problem itself, as well as the computational complexity.
For the sake completeness, we also replicate our experiments feature visualization and represen- tation inversion on the complete ImageNet dataset (Russakovsky et al. 2013) in Appendices B.3.2 and B.1.4 in particular, cf. Figures and 15
Table I: Classes used in the Restricted ImageNet model. The class ranges are inclusive.
Class
Corresponding ImageNet Classes
"Dog"
151 to 268
"Cat"
281 285
"Frog"
30 to 32
"Turtle"
33 37
"Bird"
80 to 100
"Primate"
365 to 382
"Fish"
389 397
"Crab"
118 to 121
"Insect"
300 319
A.2 MODELS
We use the ResNet-50 architecture (He et al. 2016) for adversarially trained classifiers all datasets. Unless otherwise specified, we use standard ResNet-50 classifiers trained using empir- ical risk minimization work that among standard classifiers, VGG networks (Simonyan & Zisserman (015) tend to have better-behaved representations and feature visualizations (Mordvintsev al. (2018). Thus, we also compare against standard VGG16 networks in the subsequent appendices. All models are trained with data augmentation, momentum 0.9 and weight decay 5e Other hyperparameters are pro- vided in Tables[]and]]
baseline in experiments. Additionally, it has been noted in prior
The exact procedure used train robust models along with the corresponding hyperparameters are described in Section For standard (not ad adversarially trained) classifiers on the complete Ik-class ImageNet dataset, we use pre-trained models provided the PyTorch repository
Table 2: Standard hyperparameters for the models trained in the main paper.
Dataset
Model
Arch.
Epochs
LR
Batch Size
LR Schedule
Restricted ImageNet Restricted ImageNet ImageNet
standard
ResNet-50
110
0.1
256
Drop by 10 at epochs E [30, 60
robust
Re
110
0.1
256
Drop by 10 at epochs e [30. 60
robust
ResNet-50
110
0.1
256
Drop by 10 at epochs [100]
14

---

Under review as a conference paper at ICLR 2022
Test performance of all the classifiers can be found in Section Specific parameters used study the properties of learned representations are described in Section A.6
A.3 ADVERSARIAL TRAINING
To obtain robust classifiers, we employ the adversarial training methodology proposed in (Madry et al., 2018). Specifically, we train against projected gradient descent (PGD) adversary with normalized step size, starting from random initial perturbation the training data. We consider adversarial perturbations in f-norm. Unless otherwise specified, we use the values of provided in Table 3 to train/evaluate our models (the images themselves lie the range [0, I)..
Table 3: Hyperparameters used for adversarial training.
Dataset
steps
Step size
Restricted ImageNet
3.0
0.5
ImageNet
3.0
7
0.5
A.4 MODEL. PERFORMANCE
Standard test performance the models used in the paper are presented Table for the Restricted ImageNet dataset and in for the complete ImageNet dataset.
Additionally, we report adversarial accuracy of both standard and robust models. Here, adversarial accuracies are computed against PGD adversary with 20 steps and step size of 0.375. (We also evaluated against a stronger adversary using more steps (100) of PGD, however this had marginal effect on the adversarial accuracy of the models.)
Table 4: Test accuracy for standard and robust models the Restricted ImageNet dataset.
Model
Standard
Adversarial (eps=3.0)
Standard VGG16
98.22%
2.17%
Standard ResNet-50
98.01 %
4. 4.74%
Robust ResNer-50
92.39%
81.91%
Table 5: 1op-1 accuracy for standard and robust models on the ImageNet dataset.
Model
Standard
Adversarial (eps=3.0)
Standard VGG16
73.36%
0.35%
Standard ResNet-50
76.13%
0.13%
Robust ResNet-50
57.90%
35. 15.16%
A.5 IMAGE INTERPOLATIONS
A natural consequence of the "natural invertibility" property of robust representations is the ability to synthesize natural interpolations between any two inputs #1, 22 E R In particular, given two images 2 and Ø2, define the X-interpolate between them
TA min (A R(x,) + (1- R(x2)) R(x)112
where, for given X, find x by solving 55) with projected gradient descent. Intuitively, this corresponds to linearly interpolating between the points in representation space and then finding a point in image space that has similar representation. To construct a length-(T 1) interpolation,
15

---

Under review as a conference paper at ICLR 2022
choose {0, 7.....11 .). The resulting interpolations, shown in Figure demonstrate that the A-interpolates of robust representations correspond to a meaningful feature interpolation between images. (For standard models constructing meaningful interpolations is impossible due to the brittleness identified Section ?? see Appendix B. 1.3 for details.)
Top: Image-space interpolation
dog + dog
Bottom: Representation-spac interpolation
random endpoints
Figure 11: Image interpolation using robust representations compared to their image-space coun- terparts. The former appear perceptually plausible while the latter exhibit ghosting artifacts. For pairs of images from the Restricted ImageNet test set, we solve (5) for À varying between zero and one, i.e., match linear interpolates in representation space. Additional interpolations appear in Appendix [ Figure 16 We demonstrate the ineffectiveness of interpolation with standard
representations in Appendix B.2.2
Figure 17
Relation to other interpolation methods. We emphasize that linearly interpolating in robust rep- resentation space works for any two images. This generality is in contrast to interpolations induced by GANs (e.g. (Radford et al. 2016; Brock al. 2019)), which can only interpolate between im- ages generated by the generator. (Reconstructions of out-of-range images tend to be decipherable but rather different from the originals (Bau al. 2019).) it worth noting that even for models with analytically invertible representations, interpolating in representation space does not yield semantic interpolations (Jacobsen al. 2018).
A.6 PARAMETERS USED IN STUDIES OF ROBUST/STANDARD REPRESENTATIONS
A.6.1
FINDING REPRESENTATION CORRESPONDENCE
Dataset
# steps
Step size
Restricted ImageNet/lmageNet
1000
200
A.6.2
INVERTING REPRESENTATIONS AND INTERPOLATIONS
Dataset
# steps
Step size
Restricted ImageNet/ImageNet
1000
10000
16

---

Under review as a conference paper at ICLR 2022
B
OMITTED FIGURES
B.1
INVERTING REPRESENTATIONS
B.1.1
RECOVERING TEST SET IMAGES USING ROBUST REPRESENTATIONS
Original
Inverse
Original
(a)
Original
Driginal
(b)
Figure 12: Robust representations yield semantically meaningful inverses: Original: randomly cho-
test set images from the Restricted ImageNet dataset; Inverse: images obtained by inverting the representation of the corresponding image in the top row by solving the optimization problem (??) starting from: (a) different test images and (b) Gaussian noise.
17

---

Under review as a conference paper at ICLR 2022
B.1.2
RECOVERING OUT-OF-DISTRIBUTION INPUTS USING ROBUST REPRESENTATIONS
Orginal
(a) Random kaleidoscope patterns.
Original
(b) Samples from other ImageNet classes outside what the model is trained on.
Figure 13: Robust representations yield semantically meaningful inverses: (Original): randomly chosen out-of-distributio inputs; (Inverse): images obtained by inverting the representation of the corresponding image in the top row by solving the optimization problem (??) starting from Gaussian noise.
B.1.3 INVERTING STANDARD REPRESENTATIONS
Original
Figure 14: Standard representations do not yield semantically meaningful inverses: (Original): ran- domly chosen test images from the Restricted ImageNet dataset; (Inverse): images obtained by inverting the representation of the corresponding image in the top row by solving the optimization problem (??) starting from Gaussian noise.
18

---

Under review as a conference paper at ICLR 2022
B.1.4
REPRESENTATION INVERSION ON THE IMAGENET DATASET
Source (x)
Target (x2)
Robust (xx)
Standard (x()
Figure 15: Visualization of imputs that mapped similar representations by models trained on the ImageNet dataset. Target (I2) & Source (#1): random examples image from the test set; Robust and Standard (a') result of minimizing the objective (3) to match (in (a-distance) the representation of the target image starting from the corresponding source image for (top): a robust (adversarially trained) and (bottom): standard model respectively. For the robust model, observe that the resulting images are perceptually similar to the target image in terms of high-level features, while for the standard model they often look more similar the source image which is the seed for the optimization process.
19

---

Under review as a conference paper at ICLR 2022
B.2 IMAGE INTERPOLATIONS
B.2.1
ADDITIONAL INTERPOLATIONS FOR ROBUST MODELS
Figure 16: Additional image interpolation using robust representations. To find the interpolation in input space, construct images that map to linear interpolations of the endpoints in robust representation space. Concretely, for randomly selected pairs from the Restricted ImageNet test set, use (??) find images that match to the linear interpolates in representation space (5)
20

---

Under review as a conference paper at ICLR 2022
B.2.2
INTERPOLATIONS FOR STANDARD MODELS
Ellxo
18th 120
17h E37.0-0m%
Figure 17: Image interpolation using standard representations. To find the interpolation in input space, we construct images that map to linear interpolations of the endpoints in standard represen- tation space. Concretely, for randomly selected pairs from the Restricted ImageNet test set, use (??) to find images that match to the linear interpolates in representation space (5). Image space interpolations from the standard model appear be significantly less meaningful than their robust counterparts. They are visibly similar to linear interpolation directly in the input space, which is in fact used to seed the optimization process.
21

---

Under review as a conference paper at ICLR 2022
B.3
DIRECT FEATURE VISUALIZATIONS FOR STANDARD AND ROBUST MODELS
B.3.1 ADDITIONAL FEATURE VISUALIZATIONS FOR THE RESTRICTED IMAGENET DATASET
Figure 18: Correspondence between image-level features and representations learned by bust model on the Restricted ImageNet dataset. Starting from randomly chosen seed inputs (noise/images),
constrained optimization process identify input features that maximally activate a given component of the representation vector (cf. Appendix A for details). Specif- ically, (left column): inputs to the optimization process, and (subsequent columns): features that activate randomly chosen representation components, along with the predicted class of the feature.
Figure/ Correspondence between image-level features and representations learned by ro- bust model on the Restricted ImageNet dataset. Starting from randomly chosen seed inputs (noise/images), we use constrained optimization process to identify input features that maximally activate a given component of the representation vector (cf. Appendix A. for details). Specif- ically, (left column): inputs to the optimization process, and (subsequent columns): features that activate select representation components, along with the predicted class of the feature.
22

---

Under review as a conference paper at ICLR 2022
Maximizing different coordinates (i)
Robust (ResNe:50)
Seeds (Xx)
Standard (ResNec55)
Standard (VGG 16)
Figure 20: Correspondence between image-level patterns and activations learned by standard and robust models on the Restricted ImageNet dataset. Starting from randomly chosen seed inputs (noise/images), we use PGD to find inputs that (locally) maximally activate given component of the representation vector (cf. Appendix for details). In the left column we have the original in- puts (selected randomly). and in subsequent columns we visualize the result of the optimization (4) for different activations, with each row starting from the same (far left) input for (top): robust (adversarially trained) ResNet-50 model, (middle); standard ResNet-50 model and (bottom): standard VGG16 model.
23

---

Under review as a conference paper at ICLR 2022
B.3.2
FEATURE VISUALIZATIONS FOR THE IMAGENET DATASET
Maximizing different coordinates (i)
Robust (ResNec50)
Seeds (Xx)
Standard (ResNer550)
Standard (VGG16)
Figure 21: Correspondence between image-level patterns and activations learned by standard and robust models on the complete ImageNet dataset. Starting from randomly chosen seed inputs (noise/images), we use PGD to find inputs that (locally) maximally activate given component of the representation vector (cf. Appendix(A.6 puts (selected randomly), and in subsequent columns visualize the result of the optimization (4) for different activations, with each row starting from the same (far left) input for (top): robust (adversarially trained) ResNet-50 model, (middle): standard ResNet-50 model and (bottom): standard VGG16 model.
details). In the left column we have the original in-
24

---

Under review as a conference paper at ICLR 2022
B.4
ADDITIONAL EXAMPLES OF FEATURE MANIPULATION
orgna
Orgha
Orignnl
Original
Figure 22:
Visualization of the results adding various neurons, labelled on the left, randomly chosen test images. The rows alternate between the original test images, and those same images with additional feature arising from maximizing the corresponding neuron.
25


Reviews:
[{"review_id": "pqKMCOzCDkI", "reviewer": "Reviewer_DfB9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this paper is very well written and the problem is well motivated. I think strongest point of the paper is how it is able to invert representations (very well) without regularization and starting from diverse starting points.\n\nHere are some of my thoughts/criticisms:\n1. I agree with the authors that feature activations seem to sharper via their method (like in Fig 6), but just because it appears sharper to us humans might not necessarily mean they are more interpretable. To their credit, the authors partially answer this in Fig 9 where certain activations seem to correspond to similar looking images. I think that to make their point clear the authors should have mentioned features that have the highest value in a given image (say top 5) and then show the maximally activated images for those features. This way we know that these features truly are important (say top-5) for their entire class  vs important to just a few images in that class. At the moment, Fig 9 seems to me like a post hoc way of analyzing features. \n2. In standard models, usually, the motivation to visualize representations seems to be to understand discriminative features in the input. In robust models (which have lower accuracy), the models seem to be retaining a lot of information not relevant to classification. My question is -  is visualizing a representation useful if that representation /classifier is not very good? I’m not sure about the answer myself but I’m curious what the authors think.\n3. Santurkar et al. have studied robust models in a very similar setting to the results in Fig 10 (image to image translation, feature painting)\n4. This is just a curiosity - I’ve noticed work in this direction (Tsipras et al., Santurkar et al.) choose l2 vs l-inf, is there a reason for this?", "summary_of_the_paper": "This paper posits that adversarial training imposes a prior on intermediate representations of a classifier. The authors then go on to show that this prior makes tasks like representation inversion and visualization easier. ", "summary_of_the_review": "I really like the simple and elegant nature of the solution proposed in this paper but, overall, I’m not convinced about the novelty wrt prior work, so I’m going recommend a borderline reject (however, I’m willing to be persuaded)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dUYgmF4Z5fn", "reviewer": "Reviewer_h5Za", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strength\n\n- This paper conducts instrumental feature visualization experiments, including representation inversion and feature visualization. These experiments demonstrate that adversarial trained neural networks learn a high-level encoding of inputs. \n- This paper's experiments are comprehensive. *Figure 5* is especially interesting since it demonstrates how the feature space of a robust network differs from that of a standard network.\n\n## Weakness and questions\n\n- Tsipras et al. (2019) also finds that robust model's feature are more discernable than those of standard model. Moreoever, They also conducts similar experiments to visualize robust model's features. It would be good if the authors of this paper could explicitly discuss what the different between their work and Tsipras et al. (2019).\n- In section 3.2, the authors write *selecting $\\Delta$ to be a set of perturbations that humans are robust to*. Can the authors explain why $\\ell_2$-norm is chosen? How about $\\ell_{\\infty}$-norm?\n- In the *Robust representations are (approximately) invertible our-of-the-box* subsection, the authors find that *this iversion property holds even for out-of-distribution inputs*. It is unclear why robust networks learn discernable features for *OOD* data even if the network has never observed those *OOD* data during training?\n- In *Figure 5*, 95% confidence intervals are shown. It would be good if the authors could explain what the 95% confidence interval means and how it is computed.", "summary_of_the_paper": "This paper empirically demonstrates that robust optimization encourages deep neural networks to learn a high-level encoding of inputs. Specifically, this paper first utilize $\\ell_2$- norm adversarial training to train robust neural networks. Then, this paper leverage two visualization techniques, i.e., *representation inversion* and *feature visualization*, to demonstrate that features identified by the robust neural network are more discernable. \n\nTo summary, the main contributions of this paper are as follows:\n\n- A comprehensive literature review about findings related to features of standard and robust models;\n- Train a $\\ell_2$-norm robust neural network and visualize its features;\n- Leverage *representation inversion* and *feature visualization* to demonstrate the features in robust neural networks are more discernable.", "summary_of_the_review": "This paper conducts interesting experiments to demonstrate that robust networks' features are more discernable via representation inversion and feature visualization. However, it is unclear to me how this work is different from previous paper (Tsipras et al. (2019)).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LiY420eEDtp", "reviewer": "Reviewer_o96F", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The properties investigated in the paper are interesting but not surprising, especially in the context of existing literature[1,2,3,4]. \n\nThe paper only provides experimental demonstration of this phenomenon without going into a more detailed explanation of the phenomenon. In my opinion this is not enough when the observations, in question, are not very novel and have already been explored in various forms in past published literature.\n\nFor example [1] argues that local lipschitzness provides adversarial robustness (this is not the only paper to argue this) and local lipschitzness is a necessary (and bi-lipschitzness is a suffiicient) criterion for **invertibility**. [2] shows that adversarially robust models captures salient features of the data and combining that with local linearisation (from [1]) allows us  to **visualise  features**. Thus, in the context of [2], [1], and also [3] i find the feature visualisation experiments rather deductive.\n\nAdded to this, [4] provides a whole host of experimental evidence that adversarially robust models can be used to generate salient features of the input data either from scratch, inpaint into an occluded image, or  from hand drawn sketches. Thus it is not a surprise that input space features can be visualised by altering the representation space.\n\nThus given that the observations themselves are quite deductive from existing work, I would have liked to see a deeper dive into why this happens, if this is a property of adversarial training algorithms or perhaps robust models (using other techniques) also exhibit this. It is surprising that there is much mention of adversarial training but no experimental work with (or even mention of) TRADES [5], which is an algorithm of similar flavor but has shown (perhaps) better empirical behavior.\n\nIn the discussion of learned representation, I also did not find mention of recent works [6,7]  that talks about the importance of proper prior for representation learning  in the context of adversarial robustness.\n\nFurther, all the main claims of the paper have been exposed through visual examples. While this fits quite well as the points to be made are very visual points, it raises questions about whether the study has been done systematically. There is a huge body of work in the invertibility of NN literature. Perhaps, the authors could see if that could be used to derive more quantitative claims about adversarially robust models.\n\n\n\n[1] Yang, Yao-Yuan, et al. \"Adversarial robustness through local lipschitzness.\" (2020).  \n[2] Salman, Hadi, et al. \"Do adversarially robust imagenet models transfer better?.\" (2020).  \n[3] Ilyas, Andrew, et al. \"Adversarial examples are not bugs, they are features.\" (2019).   \n[4] Santurkar, Shibani, et al. \"Image synthesis with a single (robust) classifier.\" (2019).   \n[5] Zhang, Hongyang, et al. \"Theoretically principled trade-off between robustness and accuracy.\"  (2019).   \n[6] Montasser, Omar, Steve Hanneke, and Nathan Srebro. \"Vc classes are adversarially robustly learnable, but only improperly.\" (2019).   \n[7] Sanyal, Amartya, et. al. \"How Benign is Benign Overfitting?\" (2020). \n================== Update ====================\n\nThe authors have not provided any response to our reviews. So, I will stick to my initial rating and I am confident of it.", "summary_of_the_paper": "The paper looks at  favorable properties of feature representations of an adversarially robust model. In particular, the authors look at a model trained with PGD training with an $\\ell_p$ adversary. In terms of favourable properties, the authors look at representation inversion and feature manipulation and with experimental evidences claim that adversarially robust models are naturally better at it,", "summary_of_the_review": "Overall, due to the points mentioned above I found the work to be very deductive and not systematic. Thus I vote for rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "38u9tLBbQTv", "reviewer": "Reviewer_Ukbn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n1. The paper is very well-written and well-organized. The observations are interesting and inspiring.\n\n## Weaknesses\n1. My main concern is the novelty of the paper since the current submission does not introduce a new approach or algorithm or theoretical results. The paper also lacks comparison/discussion of recent works.", "summary_of_the_paper": "This paper presents an interesting observation that the learned representations of robustly trained models align much more closely with our idealized view of neural network embeddings as extractors of high-level features that are more aligned with human perception. The paper provides experiments to support this view: (1) robust representations are (approximately) invertible out-of-the-box; (2) robust representations allow for direct visualization of human-recognizable features.", "summary_of_the_review": "Reviewing this paper gives me the impression that the paper is a little outdated (for example the most recent paper cited was in 2019). I found a very similar submission for ICLR 2020 and I agree with the reviewers/CPC's discussions. Although the paper is very well-presented, the previous concerns have not been fully addressed in the current manuscript. Thus I lean toward rejecting the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper looks at the favorable properties of feature representations of an adversarially robust model, which are interesting but not surprising, especially in the context of much existing literature has talked about this. All reviewers gave negative scores. The main issues are: 1) The paper only provides experimental demonstration of this phenomenon without going into a more detailed explanation of the phenomenon. This is not enough when the observations, in question, are not very novel and have already been explored in various forms in past published literature. 2) limited novelty since the current submission does not introduce a new approach or algorithm or theoretical results. The paper also lacks comparison/discussion of recent works. Thus, I cannot recommend accepting the paper to ICLR.
Decision: Reject