Adversarial Robustness as a Prior for Learned Representations
Authors: Logan Engstrom
Andrew Ilyas
Shibani Santurkar
Dimitris Tsipras
Brandon Tran
Aleksander Madry
Venue:  (2020)
Keywords: adversarial robustness
adversarial examples
robust optimization
representation learning
feature visualization

Abstract: An important goal in deep learning is to learn versatile, high-level feature representations of input data. However, standard networks' representations seem to possess shortcomings that, as we illustrate, prevent them from fully realizing this goal. In this work, we show that robust optimization can be re-cast as a tool for enforcing priors on the features learned by deep neural networks. It turns out that representations learned by robust models address the aforementioned shortcomings and make significant progress towards learning a high-level encoding of inputs. In particular, these representations are approximately invertible, while allowing for direct visualization and manipulation of salient input features. More broadly, our results indicate adversarial robustness as a promising avenue for improving learned representations.

Under review as a conference paper at ICLR 2020
ADVERSARIAL ROBUSTNESS AS A PRIOR FOR LEARNED REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
An important goal in deep learning is learn versatile, high-level feature rep- resentations of input data. However, standard networks' representations seem to possess shortcomings that, we illustrate, prevent them from fully realizing this goal. In this work, we show that robust optimization can he re-cast as a tool for enforcing priors on the features learned by deep neural networks. It turns out that representations learned by robust models address the aforementioned short- comings and make significant progress towards learning a high-level encoding of inputs. In particular, these representations are approximately invertible, while al- lowing for direct visualization and manipulation of salient input features. More broadly, our results indicate adversarial robustness promising avenue for im- proving learned representations.
INTRODUCTION
Beyond achieving remarkably high accuracy variety of tasks (Krizhevsky al. 2012 He et al. 2015 Collobert & Weston 2008). a major appeal of deep learning is the ability to learn effective feature representations of data. Specifically, deep neural networks can be thought of as linear classifiers acting on learned feature representations (also known as feature embeddings). A major goal in representation learning for these embeddings encode high-level, interpretable features of any given input (Goodfellow et al.) 2016:; Bengio et al.) 2013 Bengio (019)). Indeed, learned representations turn out to be quite versatile- in computer vision, for example, they are the driving force behind transfer learning Girshick et al. (2014); Donahue al. (2014), and image similarity metrics such as VGG distance Dosovitskiy & Brox (20n6));oohnonn Johnson et al. (2016) Zhang ct al. 2018).
These successes and others clearly illustrate the utility of learned feature representations. Still, deep networks and their embeddings exhibit some shortcomings that are at odds with our idealized model of a linear classifier on top of interpretable high-level features. For example, the existence of adver- sarial examples (Biggio al. 20033] Szegedy et al. (011) and the fact that they may correspond to flipping predictive features llyas et al. (2019) suggests that deep neural networks make predictions based on features that are vastly different from what humans use, or even recognize. (This message has been also corroborated by several recent works (Brendel & Bethge 2019 Geirhos et al. [211]] Jetley et al.) 2018 Zhang Zhu/ 2019).) In fact, we show a more direct example of such a short- coming fc.f. Section 2}. wherein one can construct pairs of images that appear completely different to human but are nearly identical in terms of their learned feature representations.
Our contributions. Motivated by the limitations of standard representations, we propose using the robust optimization framework as tool to enforce (user-specified) priors features that models should learn (and thus on their learned feature representations). We demonstrate that the result- ing learned "robust representations" (the embeddings learned by adversarially robust neural net- works Goodrellow et al. (2013); |); Madry et al.] (2018) address many of the shortcomings affecting standard learned representations and thereby enable new modes of interaction with inputs via manip- ulation of salient features. These findings are summarized below (c.f. Figure | for an illustration):
'Our code and models for reproducing these results is available at a//pp///gghhaaccmmmsammmmmmmccrr learned-reprreennttiions

---

Under review as a conference paper at ICLR 2020
Direct feature visualizations
Representation Inversion
Feature manipulation
Add "stripes"
rep(x) from noise
Add "water"
"shells"
"eyespots"
Match rep(x) from noise
Add "red limbs"
"branches"
"leathers"
Interpolations between arbitrary inputs
"Fur"
"stripes"
Figure I: Sample images highlighting the properties and applications of "robust representations" studied in this work. All of these manipulations use only gradient descent on simple, unregularized. direct functions of the representations of adversarially robust neural networks Goodfellow et al. (011): Madry et al. (2018).
Representation inversion (Section 4.1 In stark contrast standard representations, bust representations are approximately invertible- is, they provide high-level em- bedding of the input such that images with similar robust representations similar, and the salient features of an image are easily recoverable from its robust feature representation. This property also naturally enables feature interpolation between arbitrary inputs.
semantically
Direct maximization of the coordinates of robust representations suffices to visualize easily recognizable features of the model. This is again a significant departure from standard models where (a) without explicit regularization at visualization time, feature visualization often produces unintelligible results; and (b) even with regularization, visualized features in the representation layer are scarcely human- recognizeable Olah et al. (2017).
Simple feature visualization (Section 4.2):
Feature manipulation (Section 4.2.1 Through the aforementioned direct feature visual- ization property, robust representations enable the addition of specific features to images through direct first-order optimization.
Broadly, our results indicate that robust optimization a promising avenue for learning represen- tations that more "aligned" with notion of perception. Furthermore, our findings highlight the the desirability of adversarial robustness as goal beyond the standard security and reliability context.
2 LIMITATIONS OF STANDARD REPRESENTATIONS
Following standard convention, for given deep network we define the representation R(x) R of a given input x Rd as the activations of the penultimate layer the network (where usually k & d). The prediction of the network can thus be viewed as the output of linear classifier on the representation R(x). We refer to the distance in representation space between two inputs (æ1, x2) as the lg distance between their representations (R(z1), R(x2)), i.c., [[()) R(22)
A common aspiration in representation learning is to have that for any pixel-space input x, R(x vector encoding (206)): 2016i:Beeggoo et al.] (2013). These high-level features would be linearly separable with respect to the classification task, allowing the classifier to attain high accuracy.
of "human-meaninge" meaningful" features of x: Bengio (2019) Goodfellow et al.
Running somewhat counter to this intuition, however, we find that it straightforward to construct pairs of images with nearly identical representations yet drastically different content, shown in
2

---

Under review as a conference paper at ICLR 2020
Figure Finding such pairs turns out to be simple as sampling two images #1,22 D, then optimizing one of them to minimize distance in representation space to the other:
2 = ž + arg min R( + 6) - R(22)1l2.
(1)
Indeed, solving objective T yields images that have similar representations, but share no qualitative resemblance (in fact, a/ tends to look nearly identical to æ1). An example of such pair is given in
Figure 2
Note that if representations truly provided an encoding of any image into high-level features, finding images with similar representations should necessitate finding images with similar high-level fea- tures. Thus, the existence of these image pairs (and similar phenomena observed by prior work JJ- cobsen et (2019)) lays bare misalignment between the notion of distance induced via the features learned by current deep networks, and the notion distance as perceived by humans.
R(x1) ~ R(x2)
Figure 2: A limitation of standard neural network representations: it straightforward construct pairs of images (rf, that appear completely different yet map to similar representations.
3 ADVERSARIAL ROBUSTNESS AS A PRIOR
Our analysis in Section 2 and prior work [Jacobsen et al. 2019) prompt the question:
How learn better-behaved representations?
In this work, we demonstrate that the representations learned by adversarially robust neural net- works to address many identified limitations of standard representations, and make significant progress towards the broader goal of learning high-level, human- undrrtaddddrt encodings.
Adversarially robust deep networks and robust optimization. In standard settings, supervised machine learning models are trained by minimizing the expected loss with respect to set of param- eters 0, i.e., by solving an optimization problem of the form:
min (( [o((I, v)]
We refer to 2 as the standard training fijdctive-findng the optimum of this objective should guarantee high performance unseen data from the distribution. It turns out, however, that deep neural networks trained with this standard objective are extraordinarily vulnerable to adversarial examples (Biggio et al. 2013) Szegedy et al. 2014) by changing natural input imperceptibly, one can easily manipulate the predictions of deep network be arbitrarily incorrect.
A natural approach (and one of the most successful) for defending against these adversarial ex- amples to use the robust optimization framework: classical framework for optimization in the presence of uncertainty (Wald) 1945) Danskin 1967). In particular, instead of just finding parame- ters which minimize the expected loss (as in the standard objective), a robust optimization objective also requires that the model induced by the parameters 0 be robust to worst-case perturbation of the input:
0" = =arg aig B(z,,)
max La + 6, 6, BEA
(3)

---

Under review as a conference paper at ICLR 2020
This robust objective is in fact common in the context of machine learning security, where A is usually chosen to be a simple convex set, e.g., an 2p-bal. Canonical instantiations of robust op- timization such as adversarial training (Goodfellow et al.) 2013 Madry et all, 2018) have arisen as practical ways of obtaining networks that are invariant to small f f,-bounded changes in the input while maintaining high accuracy (though small tradeoff between robustness and accuracy has been noted by prior work Tsipras et al. (2019); (Su et al. (2018)also cf. Appendix Tables 4 and 5 for comparison of accuracies of standard and robust classifiers)).
Robust optimization as feature prior. Traditionally, adversarial robustness in the deep learning setting has been explored as a goal predominantly in the context of MI. security and reliability (Big- gio & Roli 2018).
In this work, we consider an alternative perspective on adversarial robustness-we cast it as prior on the features that can be learned by model. Specifically, models trained with objective (3) must be invariant to a set of perturbations A. Thus, selecting A to be a set perturbations that hu- mans are robust to (e.g.. small nn-norm perturbations) results in models that share more invariances with (and thus are encouraged to use similar features to) human perception. Note that incorpo- rating human-selected priors and invariances in this fashion has long history in the design of MI. contol--onnooutionna layers, for instance, were introduced as a means of introducing an invariance translations of the input (Fukushima) 1980).
In what follows, we will explore the effect of the prior induced by adversarial robustness on models' learned representations. and demonstrate that representations learned by adversarially robust models are better behaved, and do in fact seem to use features that are more human -nddestadndr.......ll....
4 PROPERTIES AND APPLICATIONS OF ROBUST REPRESENTATIONS
In the previous section, we proposed using robus! optimization as a way of enforcing user-specified priors during model training. Our goal was mitigate the issues with standard representations identified in Section 2 We now demonstrate that the learned representations resulting from training with this prior indeed exhibit several advantages over standard representations.
Recall that we define representation R(-) function induced by neural network which maps inputs E IR"" vectors R(x) E I in the representation layer of that network (the penulti- mate layer). In what follows, we refer to "standard representations" as the representation functions induced by standard (non-robust) networks, trained with the objective a "robust representations" refer to the representation functions induced by fg-adversarially robust networks, ic. networks trained with the objective 3 with A being the la ball:
8 robustt arg min B( max Lo(a + 6.y)
[118|laSe
It is worth noting that despite the value of used for training being quite small, we find that robust optimization globally affects the behavior of learned representations. As we demonstrate in this section, the benefits of robust representations extend out-of-distribution inputs and far beyond e-balls around the training distribution.
Experimental setup. We train robust and standard ResNet-50 (He et al., 2016) networks on
the Restricted ImageNet (Tsipras et al.] (019)) and ImageNet (Russakovsky et al. (013) datasets. Datasets specifics are in in Appendix [A. training details are in in Appendices A.2 and A.3 and the performance of each model is reported in Appendix A.4 In the main text, we present results for Restricted ImageNet, and link to (nearly identical) results for ImageNet present in the appendices
B.1.4 B.3.2
Unless explicitly noted otherwise, our optimization method of choice for any objective function will be (projected) gradient descent (PGD), first-order method which is known to be highly effective for minimizing neural network- beteork-based loss functions for both standard and adversarially robust neural networks (Athalye et al. 2018a Madry et al.) 2018).
Code for reproducing our results is available at hittps://githee com/
htwp://bbb.ooo/spppmmmaeteeee snappymanatee/
robust- www.llaareeeeeee -tw.aaeeee-rrreeeettttiiieeee
4

---

Under review as a conference paper at ICLR 2020
4.1 INVERTING ROBUST REPRESENTATIONS
As discussed in Section for standard deep networks, given any input it is straightforward to find another input that looks entirely different but has nearly the same representation (c.f. Figure 2). We noted that this finding runs somewhat counter the idea that these learned representations ef- fectively capture relevant input features. After all, if the representation function was truly extracting "high-level" features of the input conceptualize them, semantically dissimilar images should (by definition) have different representations. We now show that the state of affairs is greatly im- proved for robust representations.
Robust representations are (approximately) invertible out of the box. We begin by recalling the optimization objective (I) used in Soc find pairs of images with similar representations, simple minimization of & distance in representation space from a source image a1 to target image Z22
o 21 arg min [R + 6) R(22)ll2-
This process can be seen recovering an image that maps the desired target representation, and hence is commonly referred to as representation inversion (Dosovitskiy & Brox 2016b; Mahendran & Vedaldi 2015 Ulyanov et al. 2017). It turns out that in sharp contrast what observe for standard models. the images resulting from minimizing (4) for robust models are actually semanti- cally similar the original (target) images whose representation being matched, and this behavior is consistent across multiple samplings of the starting point (source image) a1 (cf. Figure
Standard (x)) Robust (x1) Source (X,)
Target (x2)
Figure 3: Visualization of inputs that are mapped to similar representations by models trained on the Restricted ImageNet dataset. Target (22) & Source (x1): random examples image from the test set; Robust and Standard (ar(): result of minimizing the objective (4) match (in Ez-distance) the representation of the target image starting from the corresponding source image for (top): robust (adversarially trained) and (bottom): standard model respectively. For the robust model, we observe that the resulting images are perceptually similar to the target image in terms of high-level features (even though they do not match exactly), while for the standard model they often look more similar to the source image which is the seed for the optimization process. Additional results in Appendix B and similar results for ImageNet are in Appendix B.1.4
Representation proximity seems to entail semantic similarity. In fact, the contrast between the invertibility of standard and robust representations is even stronger. To illustrate this, we will attempt to match the representation of target image while staying close to the starting image of the optimization in pixel-wise f--norm (this is equivalent to putting norm bound on 8 in objective (4). With standard models, we can consistently get close to the target image representation space, without moving far from the source image Fi. On the other hand, for robust models, we cannot get close to the target representation while staying close to the source image- this is illustrated quantitatively in Figure 4 This indicates that for robust models, semantic similarity may in fact be necessary for representation similarity (and not, for instance, merely an artifact of the local robustness induced by robust optimization).
5

---

Under review as a conference paper at ICLR 2020
Min distance in rep. space
1.2
Robust network
Standard network
0.8
for robust opt. (Eq. (3)
0.6
0.4
0.2
2
2
25
27
2
Allowed lz distance from source image
Figure 4: Optimizing objective 4 with PGD and an fo-norm constraint around the source image. On the x-axis is the radius of the constraint set, and on the y-axis is the distance in representation space between the minimizer of objective (4) within the constraint set and the target image, normalized by the norm of the representation of the target image: i.e., a point (Iç, y;) on the graph corresponds to Notably, we are unable closely match the representation of the target image for the robust network until the norm constraint grows very large, and in particular much larger than the norm of the perturbation that the model is trained to be robust against (e in objective
Y/ minj
R[ 5) R((tary))l //lp//////// R(Jaary))ll (atarr/)ll/(((((arrg)))l
Shown are 95% confidence intervals over random choice of
source and target images.
Constraint Constraint Constraint 32 Constraint 128
Target image
Figure 5: visualization of the final solutions to optimizing objective with PGD when constraining the solution to lie in an & ball around the source image for an adversarially robust neural network. We note that even the radius the constraint set small and we cannot match the representation very well, salient features of the target image still arise.
We also find that when 8 is highly constrained (i.e. when the source image and thus cannot match the representation of the target well), the solution to the inversion problem still displays some salient features of the target image (c.f. Figure 5). Both of these observations suggest that the representations of robust networks function much more like we would expect high-level feature representations behave.
forced stay very close to
Inversion of out-of-distribution inputs. We find that the inversion properties uncovered above hold even for dittr--disttibutioon inputs, demonstrating that robust representations capture general features as opposed to features only relevant for the specific classification task. In particular, we repeat the inversion experiment (simple minimization of distance in representation space) using images from classes not present in the original dataset used during training (Figure 6 right) and structured random patterns (Figure 14 in Appendix B. the reconstructed images consistently resemble the targets.
Interpolation between arbitrary inputs. Note that this ability to consistently invert representa- tions into corresponding inputs also translates into the ability to semantically interpolate between any two inputs. For any two inputs I1 and 22, R(z2) in representation space, then use the inversion procedure to get images corresponding to the interpolate representations. The resulting inputs interpolate between the endpoints in percep-
(linearly) interpolate between R(z1) and

---

Under review as a conference paper at ICLR 2020
Test
Out-of-distribution
Target
Result
Figure 6: Robust representations yield semantically meaningful embeddings. Target: random im- ages from the test set (col. 1-5) and from outside of the training distribution (6-10); Result: images obtained from optimizing inputs (using Gaussian noise the source image) to minimize ly-distance to the representations of the corresponding image in the top row. (More examples appear in Ap- pendix B.T
tually plausible manner without any of the "ghosting" artifacts present in input-space interpolation. We show examples of this inversion as well as experimental details in Appendix A.5
4.2 DIRECT FEATURE VISUALIZATION
A common technique for visualizing and understanding the representation function R(-) of a given network is optimization-based feature visualization (Olah et al. 2017). process in which we maxi- mize a specific feature (component) in the representation with respect to the input, in order to obtain insight into the role of the feature in classification. Concretely, given some i E [K] denoting component of the representation vector, we use gradient descent to find an input x' that maximally activates i.e., we solve:
zo + arg max R(zo + 6),
(5)
for various starting points zy which might be random images from D or even random noise.
Visualization "fails"" for standard networks. For standard networks, optimizing the objective ( often yields unsatisfying results. While we can easily find images for which the 7t component of R(-) is large (and thus the optimization problem is tractable). these images tends to look meaningless to humans, often resembling the starting point of the optimization. Even when these images are non-trivial, they tend to contain abstract, hard-to-discern patterns (c.f. Figure | (hottom)). As we discuss later this section, regularization/post propeessi- of visualizations does improve this state of affairs, though not very significantly and potentially at the cost of suppressing useful features present in the representation post-hoc.
Robust representations allow for direct visualization of human-recognizall features. For robust representations. however, we find that easily recognizable high-level features emerge from optimizing objective (5) directly, without any regularization or post-processing. We present the results of this maximization Figure 7 (top): coordinates consistently represent the same concepts across different choice of starting input z5 (both in and out of distribution). Furthermore, these concepts are not merely an artifact of our visualization process, as they consistently appear in the test-set inputs that most strongly activate their corresponding coordinates (Figure
The limitations of regularization for visualization in standard networks. Given that directly optimizing objective (5) does not produce numan-meaninge images, prior work on visualization usually tries regularize objective (5) through a variety of methods. These methods include ap- plying random transformations during the optimization process (Mordvintsev et al.) 2015] Olah ct al. 2017). restricting the space of possible solutions (Nguyen et al. | 2016 2017). or post- processing the input gradients (Oygard 2015 Tyka, 2016). While regularization does in gen- eral produce better results qualitatively, comes with few notable disadvantages that are well- recognized in the domain of feature visualization. First, when introduces prior information about what makes images visually appealing into the optimization process, becomes difficult to disentangle the effects of the actual model from the effect of the prior information introduced

---

Under review as a conference paper at ICLR 2020
Maximizing different coordinates (i)
Seeds (xp)
Robust
Standard
Figure 7: Correspondence between image-level patterns and activations learned by standard and robust models on the Restricted ImageNet dataset. Starting from randomly chosen seed inputs (noise/images), we use PGD to find inputs that (locally) maximally activate a given component of the representation vector (ef. Appendix I for details). In the left column we have the seed inputs To (selected randomly), and in subsequent columns visualize the result of the optimiza- tion (5) i.e., x', for different activations, with each row starting from same (far left) input zi for (top): robust (adversarially trained) and (boltom): a standard model. Additional visualizations in Appendix B.3 and similar results for ImageNet in B.3.2
Activation 444
Activation 939 ("insect legs")
("long fish")
Most activated
Most activated
Maximized from noise
Least activated
Maximized from noise
Least activated
Figure 8: Maximizing inputs x' (found by solving (5) with zn being a gray image) and most or least activating images (from the test set) for two random activations of robust model trained on the Restricted ImageNet dataset. For each activation, we plot the three images from the validation set that had the highest or lowest activation value sorted by the magnitude of the selected activation.
through regularization Furthermore, while adding regularization does improve the visual quality of the visualizations, the components of the representation still cannot be shown to correspond to any recognizable high-level feature. Indeed, Olah et al. (2017) note that in the representation layer of standard GoogLeNet, "Neurons do not seem to correspond to particularly meaningful semantic ideas"-the the corresponding feature visualizations are reproduced in Figure 9 amples of representation-layee visualizations for VGG16 (which we found qualitatively best among modern architectures) regularized with jittering and random rotations Figure TO While these visualizations certainly look better qualitatively than their unregularized counterparts in Figure (bottom), there remains a significantly large gap in quality and discernability between these regular- ized visualizations and those of the robust network in Figure (top).
We also provide ex-
In fact, model explanations that enforce priors for purposes of visual appeal have heen often found to have little do with the data the model itself (Adebayo al. 2018).
8

---

Under review as a conference paper at ICLR 2020
Figure 10: A visualization of the first four com- ponents of the representation layer of VGGI6 when regularization via random jittering and ro- tation is applied. Figure produced using the Lu- cid visualization library.
Figure 9: Figure reproduced from Olah et al., 2017) a visualization of few components the representation layer of GoogLeNet. While regularization (as well as Fourier parameteriza- tion and colorspace decorrelation) yields visu- ally appealing results, the visualization does not reveal consistent semantic concepts.
'htpps: http::://gghthee wtwphu/...oo/ttssrt....oo
lucid/
"Stripes" (selected)
"Red limbs" (random)
30 a
Figure 1I: Visualization of the results from maximizing a chosen (left) and a random (right) rep- resentation coordinate starting from random images for the Restricted ImageNet dataset. In each figure, the top row has the initial images, and the bottom row has a feature added. Additional exam- ples in Appendix B.4
4.2.1 NATURAL CONSEQUENCE: FEATURE MANIPULATION
The ability to directly visualize high-level, recognizable features reveals another application of ro- bust representations, which refer to feature manipulation. Consider the visualization objec- tive (5) shown in the previous section. Starting from some original image, optimizing this objective results in the corresponding feature being introduced in continuous manner. It is hence possible to stop this process relatively early to ensure that the content of the original image is preserved. As heuristic, we stop the optimization process as soon the desired feature attains larger value than all the other coordinates of the representation. We visualize the result of this process for a variety of input images in Figure [T] where "stripes" or "red limbs" " are introduced seamlessly into images without any processing regularization
RELATED WORK
Adversarial Robustness Our work studies the feature representations of adversarially robust net- works. As discussed in Section 3 these are networks trained with the robust optimization frame- work (Wald) 1945; Goodfellow al. 2015 Madry et al. 2018) and were originally proposed the context of defending against adversarial perturbations (Biggio et al. 2013 Szegedy et al. 2014). Adversarial robustness has been studied extensively in the context of machine learning security (see e.g., Carlini & Wagner (2017); Athalyy et al. (2018bla) a); Papernot et al. (2077), and as indepen- dent phenomenon (see e.g.. Gilmer et al. (2018); Schmiid et (2018); Jacobsen et al. (2019): [Iyas]] 2019 Su al. (2018). Recent work also robust models for input 2019 observe that large adversarial perturbation constructed for robust networks actually resemble instances of the target class, and Anon. (2019 demonstrates that robust classifiers can be used for wide array of image synthesis tasks. While our work also manipulates inputs with robust classifiers, we focus on understanding properties robust representations (via inversion and component visualization), rather than perform any downstream tasks.
al. [2019): sipras al.
manipulation: Tsipras et al.
We repeat this process with many additional random images and random features in Appendix B.4 Anonymized for the rebuttal stage.

---

Under review as a conference paper at ICLR 2020
Inverting representations. Previous methods for inverting learned representations typically ei- ther solve an optimization problem similar to (I) while imposing a "natural image" prior on the input Mahendran & Vedaldi (2013); Yosinski et all]|1200)) (2013); Ulyanov et al.] (017)) or train separate model to perform the inversion Kingma & Welling (2015) Dosovitskiy & Brox (016bb a) Note that since these methods introduce priors additional components into the inversion process, their results are not fully faithful to the model. In an orthogonal direction, it is possible to construct mod- els that are analytically invertible by construction Dinh et al. (2014) 2017): Jacobsen et al. (2018); Behrmann et al. (2018). However, the representations learned by these models do not seem to be perceptually meaningful (for instance, interpolating between points in the representation space does not lead to perceptual input space interpolations Jacobsen et al. (2018). Another notable distinction between the inversions shown here and invertible networks is that the latter are an exactly invertible map from ı + IR, while the former shows that we can approximately recover the original input in IR from a representation in IR for l < d.
Feature visualization. Typical methods for visualizing features or classes learned by deep net- works follow an optimization- approach, optimizing objectives similar to objective (5) Since this optimization does not lead meaningful visualizations directly, these methods incorporate domain-specific input priors (either hand-crafted Nguyen et al. (2015) or learned Nguyen et al. [016) 2017)) and regularizers Simonyan et al.] (2013); Mordvintsev al. (2013): Oygard (2013); Yosinski et al. 2 Tyka (2016); Olah et al. (2017) to produce -umaa-disccrnible visualizations.
The key difference of our work is that we avoid the use of such priors or regularizers altogether,
hence producing visualizations that are fully based on the model itself without introducing any ad- ditional bias.
Semantic feature manipulation. The latent space of generative adversarial networks (GANs) Goodfellow et al. (2014) tends to allow for "semantic feature arithmetic" Radford et al. 2016 Larsen al. (2016) (similar that in word2vec embeddings Mikolov et al. (2013)) where one can manipulate salient input features using latent space manipulations. In a similar vein, one can utilize an image-to-ine tonge-tt-image translation framework to perform such manipulation (e.g. transform- ing horses to zebras), although this requires a task-specific dataset and model iZhu et al. (2017). Somewhat orthogonally,
possible to utilize the deep representations of standard models to per- form semantic feature manipulations; however such methods tend to either only perform well on datasets where the inputs are center-aligned (Upchurch et al. (2017), or are restricted to a small set of manipulations Gatys al. (2016).
6 CONCLUSION
We show that the learned representations of robustly trained models align much more closely with our idealized view of neural network embeddings as extractors human-meaningfu.. high-level features. After highlighting certain shorteomings of standard deep networks and their representa- tions, we demonstrate that robust optimization can actually be viewed as inducing a human prior over the features that models able to learn. In this way, one can view the robust representations that result from this prior as feature extractors that are more aligned with human perception.
In support of this view, we demonstrate that robust representations overcome the challenges identi- fied for standard representations: they are approximately invertible, and moving towards an image in representation space seems to entail recovering salient features of that image in pixel space. Further- more, we show that robust representations can be directly visualized with first-order methods with- out the need for post-processing or regularization, and also yield much more human unddrastannet features than standard models (even when they are visualized with regularization). These two prop- erties (inversion and direct feature visualization), in addition to serving as illustrations of the benefits of robust representations, also enable direct modes of input manipulation (interpolation and feature manipulation. respectively).
Overall, our findings highlight robust optimization framework to enforce feature priors on learned models. We believe that further exploring this paradigm will lead to models that are sig- nificantly more human-aligned while enabling wide range of new modes of interactions.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, lan Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Neural Information Processing Systems (NeurIPS), 2018.
Anon. Image synthesis with a single (robust) classifier. In Neural Information Processing Systems (NeurIPS), 2019.
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In International Conference on Machine Learning (ICMI.), 2018a.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In International Conference on Machine Learning (ICML), 2018b.
David Bau, Jun- Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, and Antonio Torralba. Inverting layers of a large generator. In ICLR Debugging Machine Learning Models Workshop, 2019.
Jens Behrmann, Will Grathwohl, Ricky T. Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In arXiv preprint (tf:::8811009999 2018.
Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. 2013.
Yoshua Bengio. URL
Talk abstract: Learning high-level representations for agents, 2019. bengio_learnntntnooooeeenen
h httpaaaetsttt__ee__eeetssttetet www.agtes..oooooo2o2.o 112NNFF Abstract for
level wwwpeeeentttionnn........ talk given at MIT.
Battista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarial machine learning. 2018.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šindić, Pavel Laskov, Gior- gio Giacinto, and Fabio Roli. Evasion attacks against machine learning test time. In Joint European conference on machine learning and knowledge discovery in databases (ECML-KDD), 2013.
Wieland Brendel and Matthias Bethge. Approximating CNNs with hag-of-locallfeattree models works surprisingly well on imagenet. In International Conference on Learning Representations (ICLR), 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations (ICLR), 2019.
Nicholas Carlini and David Wagner. Adversarial examples not easily detected: Bypassing ten detection methods. In Workshop on Artificial Intelligence and Security (AlSec), 2017.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pp. 160-167, 2008
John M. Danskin. The Theory of Max-Min and its Application to Weapons Allocation Problems. 1967.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. In arXiv preprint 4rX.v:1410.8516, 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference Learning Representations (ICIR), 2017.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Inter- national conference machine learning (ICMI), 2014.

---

Under review as a conference paper at ICLR 2020
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. In neural information processing systems (NeurIPS), 2016a.
Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks. In Computer Vision and Pattern Recognition (CVPR), 2016b.
Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological cybernetics, 1980.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In computer vision and pattern recognition (CVPR), 2016.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-traine trained CNNs biased towards texture; increasing shape bias im- proves accuracy and robustness. In International Conference on Learning Representations, 2019.
Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S Schoenholz, Maithra Raghu, Martin Watten- berg, and lan Goodfellow. Adversarial spheres. In Workshop of International Conference on Learning Representations (ICLR), 2018.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for ac- curate object detection and semantic segmentation. In computer vision and pattern recognition (CVPR), pp. 580-587, 2014.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. neural information process- ing systems (NeurIPS), 2014.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR), 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In international conference computer vision (ICCV), 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Andrew Ilyas, Shibani Santurkar, Logan Engstrom, Brandon Tran, and Aleksander Madry. Ad- versarial examples are not bugs, they are features. In Neural Information Processing Systems (NeurIPS), 2019.
Jorn-Henrik Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge. Excessive invariance causes adversarial vulnerability. In International Contemporary on Learning Representations, 2019.
Jörn-Henrik Jacobsen, Arnold W.M. Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. In International Conference Learning Representations (ICLR), 2018.
Saumya Jetley, Nicholas Lord, and Philip Torr. With friends like these, who needs adversaries? In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision (ECCV), 2016.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep volutional neural networks. Advances Neural Information Processing Systems (NeurIPS), 2012.
12

---

Under review as a conference paper at ICLR 2020
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using a learned similarity metric. In International Conference on Machine Learning (ICML), 2016.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In computer vision and pattern recognition (CVPR), 2015.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in neural information pro- cessing systems, pp- 3111-3119, 2013.
Alexander Mordvintsev, Christopher Olah, and Mike Tyka.
Inceptionism:
Going
deeper into neural networks, 2015. URL https: //t::://iiii. -ntererralnt---ural......
Alexander Mordvintsev, Nicola Pezzotti, Ludwig Schubert, and Chris Olah. Differentiable image parameterizations. In Distill, 2018.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High con- fidence predictions for unrecognizable images. In Conference on computer vision and pattern recognition (CVPR), 2015.
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Neural Informa- fion Processing Systems (NeurIPS), 2016.
Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. Plug & play generative networks: Conditional iterative generation of images in latent space. In Conference Computer Vision and Pattern Recognition (CVPR), 2017.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. In Distill, 2017.
Audun Oygard. Visualizing googlenet classes, 2015. URL
com/
2015/00/22 11155/07/2//isuss
c classes/
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Asia Conference on Computer and Communications Security, 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference on Learning Repre- sentations (ICLR), 2016.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. ImageNet Large Scale Visual Recognition Challenge. In International Journal of Computer Vision (IJCV), 2015.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry, Adver- sarially robust generalization requires more data. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint /3Xi:v 2013.
Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi, Pin- Yu Chen, and Yupeng Gao. Is robustness the cost of accuracy? a comprehensive study on the robustness of 18 deep image classification models. In European Conference on Computer Vision (ECCV), 2018.
13

---

Under review as a conference paper at ICLR 2020
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representa- tions (ICLR), 2019.
Mike Tyka. Class visualization with bilateral filters, 2016. URL bttps://ntyka.gg.tt (tteepprraam/2011102/66666611612
vis html
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In ArXiv preprint arXiv: 2017.
Paul Upchurch, Jacob Gardner, Geoff Pleiss, Robert Pless, Noah Snavely, Kavita Bala, and Kilian Weinberger. Deep feature interpolation for image content changes. conference on computer vision and pattern recognition (CVPR), 2017.
Abraham Wald. Statistical decision functions which minimize the maximum risk. In Annals of Mathematics, 1945.
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. In arXiv preprint arXiv:/506.065/1 2015.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Computer Vision and Pattern Recognition (CVPR), 2018
Tianyuan Zhang and Zhanxing Zhu. Interpreting adversarially trained convolutional neural net- works. In International Conference on Machine Learning (ICML), 2019.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image trans- lation using cycle-consistent adversarial networks. international conference on computer vi- sion(ICCV), 2017.
14

---

Under review as a conference paper at ICLR 2020
A EXPERIMENTAL SETUP
A.1 DATASETS
In the main text, we perform all our experimental analysis on the Restricted ImageNet dataset Tsipras et al. (2019) which is obtained by grouping together semantically similar classes from ImageNet into 9 super-classes shown in Table [ Attaining robust models for the complete ImageNet dataset known be challenging, both due to the hardness of the learning problem itself, as well as the computational complexity.
For the sake of completeness, also replicate our experiments feature visualization and represen- tation inversion on the complete ImageNet dataset [Russakovsky et al. (2015) in Appendices B.3.2 and B. 1.1.4 in particular, cf. Figures 22 and T6
Table 1: Classes used in the Restricted ImageNet model. The class ranges are inclusive.
Class
Corresponding ImageNet Classes
"Dog"
151 to 268
"Cat"
281 to 285
"Frog"
30 32
"Turle"
33 to 37
"Bird"
80 100
"Primate"
365 to 382
"Fish"
389 397
"Crab"
118 to 121
"Insect"
300 319
A.2 MODELS
We use the ResNet-50 architecture He et al. (2016) for our adversarially trained classifiers on all datasets. Unless otherwise specified, ical risk minimization as a baseline in our experiments. Additionally, has been noted in prior work that among standard classifiers, VGG networks (Sioonyan & Zisserman (2015) tend to have better-behaved representations and feature visualizations Mordvintsev et al. (2018). Thus, we also compare against standard VGGI6 networks in the subsequent appendices. All models trained with data augmentation, momentum 0.9 and weight decay 5e Other hyperparameters are pro- vided Sablesgandi
standard ResNet-50 classifiers trained using empir-
The exact procedure used to train robust models along with the corresponding hyperparameters are described in Section[A.3 For standard (not adversarially trained) classifiers on the complete Ik-class ImageNet dataset, we use pre-trained models provided in the PyTocc repository
Table 2: Standard hyperparameters for the models trained in the main paper.
Dataset
Model
Arch.
Epochs
LR
Batch Size
LR Schedule
Restricted ImageNet
standard
5e
110
0.1
256
Drop by 10 at epochs e [30, 60
Restricted ImageNet
robust
5e
110
0.1
256
Drop by 10 at epochs e [30, 60
ImageNet
robust
ResNet-50
110
0.1
256
Drop by 10 at epochs [100]
Test performance of all the classifiers can be found in Section Specific parameters used study the properties of learned representations are described in Section A.6
15

---

Under review as a conference paper at ICLR 2020
A.3 ADVERSARIAL TRAINING
To obtain robust classifiers, we employ the adversarial training methodology proposed in Madry al. (2018). Specifically, train against projected gradient descent (PGD) adversary with normalized step size, starting from random initial perturbation of the training data. We consider adversarial perturbations in fn-nor. Unless otherwise specified, we use the values provided in Table to train/evaluate our models (the images themselves lie in the range [0, 1D.
Table 3: Hyperparameters used for adversarial training.
Dataset
steps
Step size
Restricted ImageNet
3.0
7
0.5
ImageNet
3.0
7
0.5
A.4 MODEL PERFORMANCE
Standard test performance for the models used in the paper presented in Table the Restricted ImageNet dataset and in Table for the complete ImageNet dataset.
Additionally, report adversarial accuracy of both standard and robust models. Here, adversarial accuracies are computed against a PGD adversary with 20 steps and step size of 0.375. (We also evaluated against a stronger adversary using more steps (100) of PGD, however this had marginal effect on the adversarial accuracy of the models.)
Table 4: Test accuracy for standard and robust models on the Restricted ImageNet dataset.
Model
Standard
Adversarial (eps=3.0)
Standard VGG16
98.22%
2.17%
Standard ResNet-50
98.01 %
4.74%
Robust ResNet-50
92.39%
81.91%
Table 5: Top-1 accuracy for standard and robust models on the ImageNet dataset.
Model
Standard
Adversarial (eps=3.0)
Standard VGG16
73.36%
0.35%
Standard ResNet-50
76.13%
0.13%
Robust ResNet-50
57.90%
35.16%
A.5 IMAGE INTERPOLATIONS
A natural consequence of the "natural invertibility" property of robust representations is the ability synthesize natural interpolations between any two inputs 2 22 E 12 In particular, given two images # and I2, we define the X-interpolate between them as
IA min (A R( + (1- A) R(x2)) R(x)lla-
(6)
where, for a given X, we find Zy by solving (6) with projected gradient descent. Intuitively, this corresponds to linearly interpolating between the points in representation space and then finding a point in image space that has similar representation. To construct a length-(7 + interpolation,
we choose = {0, 3, 2 1). The resulting interpolations, shown in Figure 12 demonstrate that the X-interpolates of robust representations correspond to meaningful feature interpolation between images. (For standard models constructing meaningful interpolations impossible due to the brittleness identified in Section see Appendix B. for details.)
16

---

Under review as a conference paper at ICLR 2020
Top: Image-space interpolation
dog dog
Bottom: Representation-space interpolation
random endpoints
Figure 12: Image interpolation using robust representations compared their image-space terparts. The former appear perceptually plausible while the latter exhibit ghosting artifacts. For pairs of images from the Restricted ImageNet test set. we solve (6) for À varying between zero and one, i.c., we match linear interpolates in representation space. Additional interpolations appear in Appendix [.2.1 Figure 17 We demonstrate the ineffectiveness of interpolation with standard representations in Appendix B.2.2 Figure 18
Relation to other interpolation methods. We emphasize that linearly interpolating in robust rep- resentation space works for any images. This generality is in contrast interpolations induced by GANs (e.g. (Radford et al. 2016) Brock et al. 2019)), which can only interpolate between im- ages generated by the generator. (Reconstructions of out-of-range images tend be decipherable but rather different from the originals Bau et al. (2019).) It is worth noting that even for models with analytically invertible representations, interpolating in representation space does not yield semantic interpolations Jacobsen et al. (018).
A.6 PARAMETERS USED IN STUDIES OF ROBUST/STANDARD REPRESENTATIONS
A.6.1
FINDING RERRESENTATION-FEATURR CORRESPONDENCE
Dataset
# steps
Step size
Restricted ImageNet/lmageNet
1000
200
1
A.6.2
INVERTING REPRESENTATIONS AND INTERPOLATIONS
Dataset
e
steps
Step size
Restricted ImageNet/ImageNet
1000
10000
17

---

Under review as a conference paper at ICLR 2020
B
OMITTED FIGURES
B.1
INVERTING REPRESENTATIONS
B.1.1
RECOVERING TEST SET IMAGES USING ROBUST REPRESENTATIONS
Original
Inverse
Original
(a)
Original
Driginal
(b)
Figure 13: Robust representations yield semantically meaningful inverses: Original: randomly cho-
test set images from the Restricted ImageNet dataset; Inverse: images obtained by inverting the representation of the corresponding image in the top row by solving the optimization problem (T) starting from: (a) different test images and (b) Gaussian noise.
18

---

Under review as a conference paper at ICLR 2020
B.1.2
RECOVERING OUT-OF-DISTRIBUTION INPUTS USING ROBUST REPRESENTATIONS
Orginal
(a) Random kaleidoscope patterns.
Original
(b) Samples from other ImageNet classes outside what the model is trained on.
Figure 14: Robust representations yield semantically meaningful inverses: (Original): randomly chosen
inputs; (Inverse): images obtained by inverting the representation of the corresponding image the top row by solving the optimization problem starting from Gaussian noise.
B.1.3 INVERTING STANDARD REPRESENTATIONS
Original
Figure 15: Standard representations do not yield semantically meaningful inverses: (Original): ran- domly chosen test images from the Restricted ImageNet dataset; (Inverse): images obtained by inverting the representation of the corresponding image in the top row by solving the optimization problem (T) starting from Gaussian noise.
19

---

Under review as a conference paper at ICLR 2020
B.1.4
REPRESENTATION INVERSION ON THE IMAGENET DATASET
Source (x)
Target (x2)
Robust (xx)
Standard (x()
Figure 16: Visualization of imputs that mapped similar representations by models trained on the ImageNet dataset. Target (I2) & Source (#1): random examples image from the test set; Robust and Standard (a') result of minimizing the objective (4) to match (in (a-distance) the representation of the target image starting from the corresponding source image for (top): a robust (adversarially trained) and (bottom): standard model respectively. For the robust model, observe that the resulting images are perceptually similar to the target image in terms of high-level features, while for the standard model they often look more similar the source image which is the seed for the optimization process.
20

---

Under review as a conference paper at ICLR 2020
B.2 IMAGE INTERPOLATIONS
B.2.1
ADDITIONAL INTERPOLATIONS FOR ROBUST MODELS
Figure 17: Additional image interpolation using robust representations. To find the interpolation in input space, construct images that map to linear interpolations of the endpoints in robust representation space. Concretely, for randomly selected pairs from the Restricted ImageNet test set, use (T) to find images that match to the linear interpolates in representation space (6)
21

---

Under review as a conference paper at ICLR 2020
B.2.2
INTERPOLATIONS FOR STANDARD MODELS
Ellxo
17h
Figure 18: Image interpolation using standard representations. To find the interpolation in input space, we construct images that map to linear interpolations of the endpoints in standard represen- tation space. Concretely, for randomly selected pairs from the Restricted ImageNet test set, use (I) to find images that match to the linear interpolates in representation space (6). Image space interpolations from the standard model appear be significantly less meaningful than their robust fact used to seed the optimization process.
counterparts. They are visibly similar to linear interpolation directly in the input space, which is in
22

---

Under review as a conference paper at ICLR 2020
B.3
DIRECT FEATURE VISUALIZATIONS FOR STANDARD AND ROBUST MODELS
B.3.1
ADDITIONAL FEATURE VISUALIZATIONS FOR THE RESTRICTED IMAGENET DATASET
Figure 19: Correspondence between image-level features and representations learned by bust model on the Restricted ImageNet dataset. Starting from randomly chosen seed inputs (noise/images),
constrained optimization process identify input features that maximally activate a given component of the representation vector (cf. Appendix A for details). Specif- ically, (left column): inputs to the optimization process, and (subsequent columns): features that activate randomly chosen representation components, along with the predicted class of the feature.
Figure 20: Correspondence between image-level features and representations learned by ro- bust model on the Restricted ImageNet dataset. Starting from randomly chosen seed inputs (noise/images), we use constrained optimization process to identify input features that maximally activate a given component of the representation vector (cf. Appendix A. for details). Specif- ically, (left column): inputs to the optimization process, and (subsequent columns): features that activate select representation components, along with the predicted class of the feature.
23

---

Under review as a conference paper at ICLR 2020
Maximizing different coordinates (i)
Robust (ResNe:50)
Seeds (Xx)
Standard (ResNec55)
Standard (VGG 16)
Figure 21: Correspondence between image-level patterns and activations learned by standard and robust models on the Restricted ImageNet dataset. Starting from randomly chosen seed inputs (noise/images), we use PGD to find inputs that (locally) maximally activate given component of the representation vector (cf. Appendix for details). In the left column we have the original in- puts (selected randomly). and in subsequent columns we visualize the result of the optimization (5) for different activations, with each row starting from the same (far left) input for (top): robust (adversarially trained) ResNet-50 model, (middle); standard ResNet-50 model and (bottom): standard VGG16 model.
24

---

Under review as a conference paper at ICLR 2020
B.3.2
FEATURE VISUALIZATIONS FOR THE IMAGENET DATASET
Maximizing different coordinates (i)
Robust (ResNec50)
Seeds (Xx)
Standard (ResNer550
Standard (VGG16)
Figure 22: Correspondence between image-level patterns and activations learned by standard and robust models on the complete ImageNet dataset. Starting from randomly chosen seed inputs (noise/images), we use PGD to find inputs that (locally) maximally activate given component of the representation vector (cf. Appendix(A.6 puts (selected randomly), and in subsequent columns visualize the result of the optimization (5) for different activations, with each row starting from the same (far left) input for (top): robust (adversarially trained) ResNet-50 model, (middle): standard ResNet-50 model and (bottom): standard VGG16 model.
details). In the left column we have the original in-
25

---

Under review as a conference paper at ICLR 2020
B.4
ADDITIONAL EXAMPLES OF FEATURE MANIPULATION
orgna
Orggaa
Orignnl
Figure 23:
Visualization of the results adding various neurons, labelled on the left, randomly chosen test images. The rows alternate between the original test images, and those same images with additional feature arising from maximizing the corresponding neuron.
26


Reviews:
[{"review_id": "rylBUSKFYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper shows that the learnt representations of robustly trained models align more closely with features that the human perceive as meaningful. They propose that robust optimization can be viewed as inducing a human prior over learnt features. Extensive experiments demonstrate that robust representations are approximately invertible, can be visualized yielding more human-interpretable features, and enable direct modes of input manipulations.\n\nThe paper indicate adversarial robustness as a promising avenue for improving learned representations in from several aspects. It is well written and contains extension experimental results. I'd suggest accepting the paper.\n\nQuestions and Comments:\n- Is there a particular reason that $L_{2}$ norm is used throughout the paper? How is the performance if using other ones?\n- Compared to the other methods introducing priors or additional components into the inversion process, how is the quantitative inversion quality and computational complexity of the proposed method?\n- The paper claims that the representations are more perceptually meaningful than the others, which may need to be evaluated with broader human subjective.\n- I think it should be argmin in Equation (1).\n- Also Equation (4) is exactly the same as (1).\n- Some symbols seem to be used somewhat interchangeably. E.g., x' in Equation (1) represents x+\\delta, while x' in Equation (5) is \\delta itself.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "rkljGH8aKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "###  Summary\n1. The paper proposes robustness to small adversarial perturbations as a prior when learning representations. \n2. It demonstrates that representations that satisfy such a prior have non-trivial properties -- they are easier to visualize, are invertible (i.e. optimizing an input that produces the desired activation leads to reasonable images), and allows for direct manipulation of input features (by changing a feature in the representation space and then optimizing the image to satisfy this new representation.)\n\n### Non-blind review \nTHIS IS NOT A BLIND REVIEW\nReviewing this paper reminded me of a recent NeurIPS paper I read. \n\nI went back to that NeurIPS paper (to better compare the similarities and differences) only to found out:\n\n1- The NeurIPS19 paper cites an earlier arxiv version of this paper as an inspiration for its approach. \n2- It is from the exact same authors. \n\nThis, unfortunately, means I know who the authors are (however, there is no conflict of interest). \n\nMore importantly, this paper is too similar to the NeurIPS paper and It's hard to review without taking into account the NeurIPS paper. In this review, I will treat the said NeurIPS19 paper as published work, and evaluate if this work adds more to the discourse. (I've refrained from naming the neurips paper so the anonymity is maintained for other reviewers; the authors, I presume, would immediately know which paper I'm referring to). \n\n### Decisions with reasons\n\nEven though I think the idea introduced in this paper is interesting, I would argue for rejecting this paper for the simple reason: It doesn't add much to the existing discourse. \n\nUsing the proposed framework (i.e. learning robust representations), it demonstrates two phenomena.\n\nFirst, it shows that robust models allow feature inversion. Second, it shows that it's easily possible to directly visualize and manipulate features for such a model. (Both of these are achieved using the same idea: treating input to the model as parameterized, and optimizing for a target activation)\n\nThese are interesting observations and show that robust models learn features that rely on salient parts of the input image. However, the NeurIPS19 paper shows this even more cleary. \n\nAs a result, I'm not convinced that demonstrating the same phenomena with different examples is sufficient for this to be a standalone paper. (Perhaps the two papers could have been one single paper). \n\n### Questions \n\nWhat is the rationale behind dividing examples showing robust models rely on salient parts of input into two papers? Is there a semantic meaning to the grouping i.e. showing feature inversion, feature manipulation, and visualization in one paper and Generation, inpainting, translation, etc in another?\n\nIf I understand correctly, all of these examples exist because the robustly learned representation relies on the salient parts of the input and not on the non-robust features. If that is the case, it makes more sense to show all of these examples in a single paper. \n\n### Update after Author's response\nSince the authors have added and discussed the pertinent NeurIPS paper in this submission, I'm updating my score. \nI still think that the two papers are more similar than they might seem (See Re: Response 3 for more details). \n\n\n### Update 2 \n\nI pointed out the similarities between the three contributions in this paper and the NeurIPS paper in \"Re: Response #3\" below. The authors replied to my concerns. I'm summarizing the author's position to my concerns followed by my response. \n\n#### Author's Position\nThe authors agreed that features manipulation and feature visualization is similar, but pointed out that the chain of dependency is this paper -> NeurIPS paper and not the other way around. They mentioned that the NeurIPS paper cites this paper and acknowledges this. Moreover, they argued that even if we consider NeurIPS paper to be prior work, feature visualization is explored in much more detail in this paper. \n\n#### Response \nI think the direction of the chain of dependency is not that important since neither paper clearly builds on top of the other. The NeurIPS paper is published work now, and it makes sense to consider it prior work (Especially since it is from the same authors). \n\nMoreover, during the NeurIPS review period, the authors did not cite this paper; they only added the citation in the camera-ready version. This means that during the NeurIPS review period, they did, in fact,  take credit for the ideas used in feature painting. (The authors mention that they somehow did not, and just stated the method and showed the pictorial result in the NeurIPS paper. However, I don't see how it is possible to present a method and a pictorial result without citing other work and not take credit for the method and result.) \n\nI would agree with the authors that this paper does go into more detail for feature visualization. More specifically, this paper also looks at visualizing individual features in the representation (The NeurIPS feature painting restricts the visualization using a mask) and demonstrates that the same feature can be used to visualize similar semantic concepts (such as red limbs) on multiple images. This is definitely interesting, but still very related to the feature-painting result. It would have made more sense to include these feature visualization results in the NeurIPS paper instead of adding them in a separate paper.\n\n#### Author's Position 2\nThey disagreed that feature inversion (this paper) is similar to image generation (NeurIPS paper). I did acknowledge in my initial response that feature inversion is slightly more general than image generation, however, the authors suggest that they are completely different.\"\n\n#### Response \nI think representation inversion is more similar to generation than it might seem. Representation for an in-distribution image would correspond to a class with high probability. Maximizing a class probability would indirectly optimize for a representation (Say R_0) that maximizes that class probability. Image generation, as presented in NeurIPS paper, can be seen as inverting R_0.\n \nMoreover, the qualitative results for feature inversion, as presented in this paper, are not extra-ordinary. In the majority of the inverted images, I can not classify the inverted image correctly. That shows the model is still not paying attention to the correct aspects of the input to do classification. As a result, this paper certainly does not solve the feature inversion problem (Ideally, inverted features would highlight parts of the input necessary for making predictions and ignore other parts. Robust models, on the other hand, seem to be uniformly retaining all information of the image including the background and not highlighting the parts important for making predictions. As a result, many inverted images can not be classified by humans). \n\n#### My current position \nAt the end of the day, both this and the NeurIPS paper are demonstration papers (They are empirically demonstrating an unintuitive phenomenon). Both papers are demonstrating that robust models learn features that correspond to salient parts of the input. Even though both papers are nice, either one is sufficient to demonstrate the phenomenon. For this to be a stand-alone paper, the authors would have to do more in my opinion. One option would be to explore and compare different forms of adversarial robustness as priors (The paper is called \"Adversarial Robustness as a Prior for Learned Representations\" and not \"L2 Adversarial Robustness as a Prior for Learned Representations,\" after all). Another option would be to see if such representations are 'quantitatively' better in some settings (Such as for transfer learning). \n\nIn its current form, I feel that the two papers are too similar to recommend acceptance. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Bklg_o8TFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "===== Summary =====\nThe paper presents a study about the representations learned by neural networks trained using robust optimization — a type of optimization that requires the model to be robust to small perturbations in the data. Specifically, the paper presents results of ResNet-50 trained on ImageNet with standard optimization and robust optimization. The paper draws three main insights from studying the learned representations of the standard and robust networks. First, the representation of the robust network is approximately invertible. In other words, when recovering an image by matching the representation of a random image to the representation of a target image by adding noise, the recovered images are semantically similar to the target image; the recovered images look similar to a human. Moreover, this is also demonstrated with images from outside of the distribution of the training data. Second, the representation of the robust network, unlike the representation of standard network, shows semantically meaningful high level features without any preprocessing or regularization. This leads to the final insight, feature manipulation is easier in robust networks. This is demonstrated by adding noise to an initial image in order to maximize the activation of a specific higher level feature and stopping early to preserve most of the other features of the original image. \n\nContributions:\n1. The paper demonstrates that robust optimization enforces a prior on the representation learned by neural networks that results in high correspondence between the high-level features of an image and its representation in the network, i.e., similar images share similar representations. \n2. The paper shows that the features learned by networks trained using robust optimization are semantically meaningful to humans without having to use any form of preprocessing. \n3. The paper demonstrates that robust networks facilitate feature manipulation by injecting noise that maximally activates one of the features in the representation. \n\n===== Decision =====\nI consider that this paper should be accepted. The paper does not introduce any new algorithm or shows any theoretical results, but it is a great source of insight and intuition about robust optimization and deep learning. Moreover, the paper excels at the presentation and careful study of each of the main findings and it is well framed within the robust optimization literature. \n\n===== Comments and Questions =====\n\nThere is still a major question that the paper does not directly address, but that is very relevant to robust optimization. Given that robust optimization seems to result in better-behaved and semantically meaningful representations, as evidenced by the findings in the paper, why is it that the performance of the resulting networks, in terms of classification accuracy, is lower than the performance of standard networks (trained with standard optimization)? It seems counter-intuitive that the robust network have worse accuracy than the standard network  given that it is more robust to small perturbations. I am curious if we could obtain any insights about this issue based on what has already being done in the paper. For example, are there any salient features in the images that the standard network classifies correctly but the robust network does not? \n\n=== Minor Comments ===\n1. I think the operator in Equations (1) and (4) should argmin since the noise is being added to x_1 in order to obtain x’_1.\n\n2. What is the meaning of the error bars in Figure 4? I think this should be mentioned in the caption. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes recasting robust optimization as regularizer for learning representations by neural networks, resulting e.g. in more semantically meaningful representations. 

The reviewers found that the claimed contributions were well supported by the experimental evidence. The reviewers noted a few minor points regarding clarity that seem to have been addressed. The problems addressed are very relevant to the ICLR community (representation learning and adversarial robustness).

However, the reviewers were not convinced by the novelty of the paper. A big part of the discussion focused on prior work by the authors that is to be published at NeurIPS. This paper was not referenced in the manuscript but does reduce the novelty of the present submission. In contrast to the current submission, that paper focuses on manipulating the learned manipulations to solve image generation tasks, whereas the current paper focuses on the underlying properties of the representation. Since the underlying phenomenon had been described in the earlier paper and the current submission does not introduce a new approach / algorithm, the paper was deemed to lack the novelty for acceptance to ICLR. 


Decision: Reject