Year: 2021

Abstract:
An important goal in deep learning is to learn versatile, high-level feature representations of input data. However, standard networks’ representations seem to possess shortcomings that, as we illustrate, prevent them from fully realizing this goal. In this work, we show that robust optimization can be re-cast as a tool for enforcing a prior on the features learned by deep neural networks. It turns out that representations learned by robust models address the aforementioned shortcomings and make significant progress towards learning a high-level encoding of inputs. In particular, these representations are approximately invertible, while allowing for direct visualization and manipulation of salient input features. More broadly, our results indicate adversarial robustness as a promising avenue for improving learned representations. 1

heading: 1 INTRODUCTION
text: Beyond achieving remarkably high accuracy on a variety of tasks (Krizhevsky et al., 2012; He et al., 2015; Collobert & Weston, 2008), a major appeal of deep learning is the ability to learn effective representations of data. Specifically, deep neural networks can be thought of as linear classifiers acting on learned feature representations (also known as feature embeddings). A major goal in representation learning is for these embeddings to encode high-level, interpretable features of any given input (Goodfellow et al., 2016; Bengio et al., 2013; Bengio, 2019). Indeed, learned representations turn out to be quite versatile—in computer vision, for example, they are the driving force behind transfer learning (Girshick et al., 2014; Donahue et al., 2014), and image similarity metrics such as VGG distance (Dosovitskiy & Brox, 2016a; Zhang et al., 2018).
Still, deep networks’ feature embeddings exhibit some shortcomings that are at odds with our idealized model of a linear classifier on top of interpretable high-level features. For example, the existence of adversarial examples (Biggio et al., 2013; Szegedy et al., 2014)—and the fact that they may correspond to flipping predictive features (Ilyas et al., 2019)—suggests that deep neural networks make predictions based on features that are vastly different from what humans use, or even recognize. (This message has been also corroborated by several recent works (Brendel & Bethge, 2019; Geirhos et al., 2019; Jetley et al., 2018; Zhang & Zhu, 2019).) An more direct example of such a shortcoming is pinpointed by Jacobsen et al. (2019), who show that one can find image pairs that appear completely different to a human but are nearly identical in terms of their feature embeddings.
1https://github.com/cantankerousdolphin/robust-learned-representations.
Our contributions. Motivated by the limitations of standard representations, we propose using robust optimization (in particular, adversarial training (Goodfellow et al., 2015; Madry et al., 2018)) to enforce a prior on features that models learn (and thus on their learned feature representations). It turns out that the resulting “robust representations” (the embeddings learned by adversarially trained neural networks) are significantly better-behaved than their standard counterparts. We demonstrate this fact by looking at two well-studied tasks that are typically used to study representations:
• Representation inversion (Section 5): In contrast to standard representations—for which representation inversion is a significant challenge, e.g., (Mahendran & Vedaldi, 2015)— robust representations are naturally approximately invertible. In particular, adversarially robust networks provide an embedding of the input such that images with similar representations are semantically similar, and the salient features of an image are easily recoverable from its robust feature representation. This property also naturally enables feature interpolation between arbitrary inputs.
• Feature visualization (Section 6): Direct maximization of the coordinates of robust representations suffices to visualize easily recognizable features of the model. This is again a significant departure from standard representations where (a) without explicit regularization at visualization-time, feature visualization often produces unintelligible results; and (b) even with regularization, visualized features in the representation layer are scarcely humanrecognizeable (Olah et al., 2017). As a result of this property, robust representations enable the addition of specific features to images through direct first-order optimization.
The tasks above and their respective applications are illustrated in Figure 2. Broadly, our results suggest robust optimization as a promising avenue for learning better-behaved image representations.

heading: 2 RELATED WORK
text: Our work studies the feature representations of adversarially robust networks. As discussed in Section 3, these are networks trained with the robust optimization framework (Wald, 1945; Goodfellow et al., 2015; Madry et al., 2018) and were originally proposed in the context of defending against adversarial perturbations (Biggio et al., 2013; Szegedy et al., 2014). Adversarial robustness has been studied extensively in the context of machine learning security (see e.g., Carlini & Wagner (2017); Athalye et al. (2018b;a); Papernot et al. (2017)), and as an independent phenomenon (see e.g., Gilmer et al. (2018); Schmidt et al. (2018); Jacobsen et al. (2019); Ilyas et al. (2019); Tsipras et al. (2019); Su et al. (2018).
Several recent works have studied the qualitative properties of robust models. Zhang & Zhu (2019) find that adversarially robust models behave more predictably in the face of various outof-distribution data, appearing to use more global features. Tsipras et al. (2019) observe that large adversarial perturbation constructed for robust networks actually resemble instances of the target class. Santurkar et al. (2019) leverage this fact and use robust classifiers for a wide array of image synthesis tasks. Our work is complementary to this line of research, and focuses on understanding properties of robust representations through the lens of “benchmark” representation learning tasks (namely, inversion and component visualization).
There is also a large body of work dedicated to studying each of the representation learning tasks we study below (inversion and feature visualization)—we have embedded discussions of the related work for each task within the relevant sections.

heading: 3 BACKGROUND AND MOTIVATION

heading: 3.1 ADVERSARIAL EXAMPLES AND ROBUST TRAINING
text: In standard settings, supervised machine learning models are trained by minimizing the expected loss with respect to a set of parameters θ, i.e., by solving an optimization problem of the form:
θ∗ = min θ E(x,y)∼D [Lθ(x, y)] . (1)
We refer to (1) as the standard training objective—finding the optimum of this objective should guarantee high performance on unseen data from the distribution.
It turns out that deep neural networks trained with the standard objective are vulnerable to adversarial examples (Biggio et al., 2013; Szegedy et al., 2014)—by changing a natural input imperceptibly, one can easily manipulate the predictions of a deep network to be arbitrarily incorrect.
A natural approach (and one of the most successful) for defending against these adversarial examples is to use the robust optimization framework: a classical framework for optimization in the presence of uncertainty (Wald, 1945; Danskin, 1967). In particular, instead of just finding parameters which minimize the expected loss (as in the standard objective), a robust optimization objective also requires that the model induced by the parameters θ be robust to worst-case perturbation of the input:
θ∗ = arg min θ E(x,y)∼D [ max δ∈∆ Lθ(x+ δ, y) ] . (2)
This robust objective is in fact common in the context of machine learning security, where ∆ is usually chosen to be a simple convex set, e.g., an `p-ball. Canonical instantiations of robust optimization such as adversarial training (Goodfellow et al., 2015; Madry et al., 2018)) have arisen as practical ways of obtaining networks that are invariant to small `p-bounded changes in the input while maintaining high accuracy (though a small tradeoff between robustness and accuracy has been noted by prior work (Tsipras et al., 2019; Su et al., 2018) (also cf. Appendix Tables 4 and 5 for a comparison of accuracies of standard and robust classifiers)).

heading: 3.2 ROBUST TRAINING AS A FEATURE PRIOR
text: Traditionally, adversarial robustness in the deep learning setting has been explored as a goal predominantly in the context of ML security and reliability (Biggio & Roli, 2018).
In this work, we consider an alternative perspective on adversarial robustness—we cast it as a prior on the features that can be learned by a model. Specifically, models trained with objective (2) must be invariant to a set of perturbations ∆. Thus, selecting ∆ to be a set of perturbations that humans are robust to (e.g., small `p-norm perturbations) results in models that share more invariances with (and thus are encouraged to use similar features to) human perception. Note that incorporating human-selected priors and invariances in this fashion has a long history in the design of ML models—convolutional layers, for instance, were introduced as a means of introducing an invariance to translations of the input (Fukushima, 1980).
In what follows, we will explore the effect of the prior induced by adversarial robustness on models’ learned representations, and demonstrate that representations learned by adversarially robust models
are significantly better-behaved, and enable many previously-infeasible modes of direct interaction with feature embeddings. It is worth noting that despite the value of ε used for training being quite small, we find that robust optimization globally affects the behavior of learned representations. As we demonstrate in this section, the benefits of robust representations extend to out-of-distribution inputs and far beyond ε-balls around the training distribution.

heading: 3.3 STANDARD AND ROBUST REPRESENTATIONS
text: Our work is primarily focused on studying the representations of trained neural networks. Throughout this work, we define the representation function R(·) as a function induced by a neural network which maps inputs x ∈ Rn to vectors R(x) ∈ Rk in the representation layer of that network (the penultimate layer). In what follows, we refer to “standard representations” as the representation functions induced by standard (non-robust) networks, trained with the objective (1)—analogously, “robust representations” refer to the representation functions induced by `2-adversarially robust networks, i.e. networks trained with the objective (2) with ∆ being the `2 ball:
θ∗robust = arg min θ
E(x,y)∼D [
max ‖δ‖2≤ε
Lθ(x+ δ, y) ] .

heading: 4 EXPERIMENTAL SETUP
text: We train robust and standard ResNet-50 (He et al., 2016) networks on the Restricted ImageNet (Tsipras et al., 2019) and ImageNet (Russakovsky et al., 2015) datasets. Datasets specifics are in in Appendix A.1, training details are in in Appendices A.2 and A.3, and the performance of each model is reported in Appendix A.4. In the main text, we present results for Restricted ImageNet, and link to (nearly identical) results for ImageNet in Appendices (B.1.4,B.3.2).
Unless explicitly noted otherwise, our optimization method of choice for any objective function will be (projected) gradient descent (PGD), a first-order method which is known to be highly effective for minimizing neural network-based loss functions for both standard and adversarially robust neural networks (Athalye et al., 2018a; Madry et al., 2018).

heading: 5 INVERTING ROBUST REPRESENTATIONS
text: A common tool for understanding the semantic content of a feature representation is inversion—i.e., reconstructing an image using only data from its feature embedding (Mahendran & Vedaldi, 2015; Dosovitskiy & Brox, 2016b; Ulyanov et al., 2017). As discussed in the introduction, however, for standard deep networks, given any input x, it is straightforward to find another input that looks entirely different but has nearly the same representation (c.f. Figure 1). This makes representation inversion a difficult problem, since images are not identifiable from their feature representations alone. Indeed, the “naive” approach to representation inversion, wherein one inverts x by solving
xinv = x0 + arg min δ ‖R(x0 + δ)−R(x)‖2, (3)
tends to find solutions xinv that bear very little resemblance to x (Mahendran & Vedaldi, 2015), and instead look visually similar to the starting point of the optimization x0 (which can be any arbitrary image or even random noise).
Related work and motivation. As a result of the phenomenon described above, methods for inverting learned representations typically either impose a “natural image” prior on the inverted image (Mahendran & Vedaldi, 2015; Yosinski et al., 2015; Ulyanov et al., 2017), or train a separate model altogether to perform the inversion (Kingma & Welling, 2015; Dosovitskiy & Brox, 2016b;a). In either case, however, these methods introduce additional information into the inversion process beyond that which is contained in feature embedding, making the inversions not fully faithful to the model. In particular, it becomes difficult to disentangle the information that is actually contained in models’ feature embeddings from that which is imposed externally by the inversion method.
More broadly, lack of invertibility runs somewhat counter to the idea that learned representations capture a relevant set input features. After all, if the representation function was truly extracting “high-level” features of the input as we conceptualize them, semantically dissimilar images should (by definition) have different representations. We now show that the state of affairs is greatly improved for robust representations.
Robust representations are (approximately) invertible out-of-the-box. It turns out that in sharp contrast to what we observe for standard models, the images resulting from minimizing (3) for robust models are actually semantically similar to the original (target) images whose representation is being matched, and this behavior is consistent across multiple samplings of the starting point (source image) x1 (cf. Figure 3).
This inversion property holds even for out-of-distribution inputs, demonstrating that robust representations can represent features beyond those that are relevant for the specific classification task. In particular, we repeat the inversion experiment (simple minimization of distance in representation space) using images from classes not present in the original dataset used during training (Figure 4 right) and structured random patterns (Figure 13 in Appendix B.1): the reconstructed images consistently resemble the targets.
In fact, the contrast between the invertibility of standard and robust representations is even stronger. To illustrate this, we will attempt to match the representation of a target image while staying close to the starting image of the optimization in pixel-wise `2-norm (this is equivalent to putting a norm bound on δ in objective (3)). With standard models, we can consistently get close to the target image in representation space, without moving far from the source image x1. On the other hand, for robust models, we cannot get close to the target representation while staying close to the source image—this is illustrated quantitatively in Figure 5. This indicates that for robust models, semantic similarity may in fact be necessary for representation similarity (and is not, for instance, merely an artifact of the local robustness induced by robust optimization).
Additional related work. We have already discussed related work in inverting standard representations. In an orthogonal direction, it is possible to construct models that are analytically invertible by construction (Dinh et al., 2014; 2017; Jacobsen et al., 2018; Behrmann et al., 2018). However, the representations learned by these models do not seem to be perceptually meaningful (for instance, interpolating between points in the representation space does not lead to perceptual input space interpolations (Jacobsen et al., 2018), in contrast to robust representations, cf. Appendix A.5.). Another important distinction between the inversions shown here and invertible networks is that the latter are an exactly invertible map from Rd → Rd, while the former shows that we can approximately recover the original input in Rd from a “compressed” representation in Rk for k d.

heading: 6 DIRECT FEATURE VISUALIZATION
text: In the previous section, we showed how adversarial training enables direct (approximate) input inversion without including extra information at inversion-time. We now explore another common technique for studying representations (namely, feature visualization (Olah et al., 2017)) and show that here too, robust representations behave significantly better than their standard counterparts.
In optimization-based feature visualization (Olah et al., 2017), we maximize a specific feature (component) in the representation with respect to the input, in order to obtain insight into the role of the feature in classification. Concretely, given some i ∈ [k] denoting a component of the representation vector, we use gradient descent to find an input x′ that maximally activates it, i.e., we solve:
xvis = x0 + arg max δ R(x0 + δ)i (4)
for various starting points x0 which might be random images from D or even random noise.
Visualization “fails” for standard networks. For standard networks, optimizing the objective (4) often yields unsatisfying results. While we can easily find images for which the ith component of R(·) is large (and thus the optimization problem is tractable), these images tends to look meaningless to humans, often resembling the starting point of the optimization. Even when these images are non-trivial, they tend to contain abstract, hard-to-discern patterns (c.f. Figure 6 (bottom)). As a result, just as was the case for representation inversion, state-of-the-art feature visualization methods typically regularize objective (4) with terms that encourage more compelling visualizations. These methods include applying random transformations during the optimization process (Mordvintsev et al., 2015; Olah et al., 2017), leveraging deep generative models (Nguyen et al., 2015; 2016; 2017), or post-processing the input or gradients (Oygard, 2015; Tyka, 2016).
As was the case for inversion, however, such regularization comes with a few well-known (and widely-studied within the feature visualization literature) disadvantages. First, when one introduces prior information about what makes images visually appealing into the optimization process, it becomes difficult to disentangle the effects of the actual model from the effect of the prior information introduced through regularization2. Furthermore, while adding regularization does improve the visual quality of the visualizations, the components of the representation still cannot be shown to correspond to any recognizable high-level feature. Indeed, Olah et al. (2017) note that in the representation layer of a standard GoogLeNet, “Neurons do not seem to correspond to particularly meaningful semantic ideas”—the corresponding feature visualizations are reproduced in Figure 7.
Robust representations allow for direct visualization of human-recognizable features. For robust representations, however, we find that easily recognizable high-level features emerge from optimizing objective (4) directly, without any regularization or post-processing. We present the results of this maximization in Figure 6 (top): coordinates consistently represent the same concepts across different choice of starting input x0 (both in and out of distribution). Furthermore, these concepts are not merely an artifact of our visualization process, as they consistently appear in the test-set inputs that most strongly activate their corresponding coordinates (Figure 9).
2In fact, model explanations that enforce priors for purposes of visual appeal have been often found to have little to do with the data or the model itself (Adebayo et al., 2018).
ahttps://github.com/tensorflow/lucid/
In fact, we observe that the unregularized feature visualizations from robust representations are significantly more discernable than even regularized visualizations of standard ones. Specifically, we provide examples of representation-layer visualizations for VGG16 (which we found qualitatively best among modern architectures) regularized with jittering and random rotations in Figure 8. While these visualizations certainly look better qualitatively than their unregularized counterparts in Figure 6 (bottom), there remains a large gap in quality and discernability between these regularized visualizations and the unregularized robust visualizations in Figure 6 (top).
Activation 444 (“long fish”)
Activation 939 (“insect legs”)
Maximized from noise
Most activated
Least activated Maximized from noise
Most activated
Least activated
Figure 9: Maximizing inputs x′ (found by solving (4) with x0 being a gray image) and most or least activating images (from the test set) for two random activations of a robust model trained on the Restricted ImageNet dataset. For each activation, we plot the three images from the validation set that had the highest or lowest activation value sorted by the magnitude of the selected activation.
Natural consequence: feature manipulation. The ability to directly visualize high-level, recognizable features reveals another application of robust representations, which we refer to as feature manipulation. Consider the visualization objective (4) shown in the previous section. Starting from some original image, optimizing this objective results in the corresponding feature being introduced in a continuous manner. It is hence possible to stop this process relatively early to ensure that the content of the original image is preserved. As a heuristic, we stop the optimization process as soon as the desired feature attains a larger value than all the other coordinates of the representation. We visualize the result of this process for a variety of input images in Figure 10, where “stripes” or “red limbs” are introduced seamlessly into images without any processing or regularization 3.
Additional related work. The latent space of generative adversarial networks (GANs) (Goodfellow et al., 2014) tends to allow for “semantic feature arithmetic” (Radford et al., 2016; Larsen et al., 2016) (similar to that in word2vec embeddings (Mikolov et al., 2013)) where one can manipulate salient input features using latent space manipulations. In a similar vein, one can utilize an imageto-image translation framework to perform such manipulation (e.g. transforming horses to zebras), although this requires a task-specific dataset and model (Zhu et al., 2017). I t is possible to utilize the
3We repeat this process with many additional random images and random features in Appendix B.4.
deep representations of standard models to perform semantic feature manipulations; however such methods tend to either only perform well on datasets where the inputs are center-aligned (Upchurch et al., 2017), or are restricted to a small set of manipulations (Gatys et al., 2016).

heading: 7 CONCLUSION
text: We show that the learned representations of robustly trained models align much more closely with our idealized view of neural network embeddings as extractors of high-level features. After highlighting certain shortcomings of standard deep networks and their representations, we demonstrate that robust optimization can actually be viewed as inducing a prior over the features that models are able to learn. In this way, one can view the robust representations that result from this prior as feature extractors that are more aligned with human perception.
In support of this view, we demonstrate that robust representations overcome the challenges identified for standard representations: they are approximately invertible, and moving towards an image in representation space seems to entail recovering salient features of that image in pixel space. Furthermore, we show that robust representations can be directly visualized with first-order methods without the need for post-processing or regularization, and also yield much more human-understandable features than standard models (even when they are visualized with regularization). These two properties (inversion and direct feature visualization), in addition to serving as illustrations of the benefits of robust representations, also enable direct modes of input manipulation such as feature addition.

heading: A EXPERIMENTAL SETUP
text: In this section we describe the elements of our experimental setup. Code for reproducing our results using the robustness library (Engstrom et al., 2019) can be found at https://github.com/ snappymanatee/robust-learned-representations.

heading: A.1 DATASETS
text: In the main text, we perform all our experimental analysis on the Restricted ImageNet dataset (Tsipras et al., 2019) which is obtained by grouping together semantically similar classes from ImageNet into 9 super-classes shown in Table 1. Attaining robust models for the complete ImageNet dataset is known to be challenging, both due to the hardness of the learning problem itself, as well as the computational complexity.
For the sake of completeness, we also replicate our experiments feature visualization and representation inversion on the complete ImageNet dataset (Russakovsky et al., 2015) in Appendices B.3.2 and B.1.4—in particular, cf. Figures 21 and 15.

heading: A.2 MODELS
text: We use the ResNet-50 architecture (He et al., 2016) for our adversarially trained classifiers on all datasets. Unless otherwise specified, we use standard ResNet-50 classifiers trained using empirical risk minimization as a baseline in our experiments. Additionally, it has been noted in prior work that among standard classifiers, VGG networks (Simonyan & Zisserman, 2015) tend to have better-behaved representations and feature visualizations (Mordvintsev et al., 2018). Thus, we also compare against standard VGG16 networks in the subsequent appendices. All models are trained with data augmentation, momentum 0.9 and weight decay 5e−4. Other hyperparameters are provided in Tables 2 and 3.
The exact procedure used to train robust models along with the corresponding hyperparameters are described in Section A.3. For standard (not adversarially trained) classifiers on the complete 1k-class ImageNet dataset, we use pre-trained models provided in the PyTorch repository4.
Test performance of all the classifiers can be found in Section A.4. Specific parameters used to study the properties of learned representations are described in Section A.6.

heading: A.3 ADVERSARIAL TRAINING
text: To obtain robust classifiers, we employ the adversarial training methodology proposed in (Madry et al., 2018). Specifically, we train against a projected gradient descent (PGD) adversary with a normalized step size, starting from a random initial perturbation of the training data. We consider adversarial perturbations in `2-norm. Unless otherwise specified, we use the values of provided in Table 3 to train/evaluate our models (the images themselves lie in the range [0, 1]).

heading: A.4 MODEL PERFORMANCE
text: Standard test performance for the models used in the paper are presented in Table 4 for the Restricted ImageNet dataset and in Table 5 for the complete ImageNet dataset.
Additionally, we report adversarial accuracy of both standard and robust models. Here, adversarial accuracies are computed against a PGD adversary with 20 steps and step size of 0.375. (We also evaluated against a stronger adversary using more steps (100) of PGD, however this had a marginal effect on the adversarial accuracy of the models.)
A.5 IMAGE INTERPOLATIONS
A natural consequence of the “natural invertibility” property of robust representations is the ability to synthesize natural interpolations between any two inputs x1, x2 ∈ Rn. In particular, given two images x1 and x2, we define the λ-interpolate between them as
xλ = min x ‖ (λ ·R(x1) + (1− λ) ·R(x2))−R(x)‖2. (5)
where, for a given λ, we find xλ by solving (5) with projected gradient descent. Intuitively, this corresponds to linearly interpolating between the points in representation space and then finding a point in image space that has a similar representation. To construct a length-(T + 1) interpolation,
we choose λ = {0, 1T , 2 T , . . . 1}. The resulting interpolations, shown in Figure 11, demonstrate that the λ-interpolates of robust representations correspond to a meaningful feature interpolation between images. (For standard models constructing meaningful interpolations is impossible due to the brittleness identified in Section ??—see Appendix B.1.3 for details.)
Relation to other interpolation methods. We emphasize that linearly interpolating in robust representation space works for any two images. This generality is in contrast to interpolations induced by GANs (e.g. (Radford et al., 2016; Brock et al., 2019)), which can only interpolate between images generated by the generator. (Reconstructions of out-of-range images tend to be decipherable but rather different from the originals (Bau et al., 2019).) It is worth noting that even for models with analytically invertible representations, interpolating in representation space does not yield semantic interpolations (Jacobsen et al., 2018).

heading: A.6 PARAMETERS USED IN STUDIES OF ROBUST/STANDARD REPRESENTATIONS

heading: A.6.1 FINDING REPRESENTATION-FEATURE CORRESPONDENCE
text: Dataset # steps Step size
Restricted ImageNet/ImageNet 1000 200 1
A.6.2 INVERTING REPRESENTATIONS AND INTERPOLATIONS
Dataset # steps Step size
Restricted ImageNet/ImageNet 1000 10000 1

heading: B OMITTED FIGURES
text: B.1 INVERTING REPRESENTATIONS
B.1.1 RECOVERING TEST SET IMAGES USING ROBUST REPRESENTATIONS

heading: B.1.2 RECOVERING OUT-OF-DISTRIBUTION INPUTS USING ROBUST REPRESENTATIONS
text: B.1.3 INVERTING STANDARD REPRESENTATIONS

heading: B.1.4 REPRESENTATION INVERSION ON THE IMAGENET DATASET
text: B.2 IMAGE INTERPOLATIONS

heading: B.2.1 ADDITIONAL INTERPOLATIONS FOR ROBUST MODELS
text: B.2.2 INTERPOLATIONS FOR STANDARD MODELS

heading: B.3.1 ADDITIONAL FEATURE VISUALIZATIONS FOR THE RESTRICTED IMAGENET DATASET

heading: B.3 DIRECT FEATURE VISUALIZATIONS FOR STANDARD AND ROBUST MODELS

heading: B.3.2 FEATURE VISUALIZATIONS FOR THE IMAGENET DATASET

heading: B.4 ADDITIONAL EXAMPLES OF FEATURE MANIPULATION

Review summaries:
The paper looks at  favorable properties of feature representations of an adversarially robust model. In particular, the authors look at a model trained with PGD training with an $\ell_p$ adversary. In terms of favourable properties, the authors look at representation inversion and feature manipulation and with experimental evidences claim that adversarially robust models are naturally better at it,

This paper empirically demonstrates that robust optimization encourages deep neural networks to learn a high-level encoding of inputs. Specifically, this paper first utilize $\ell_2$- norm adversarial training to train robust neural networks. Then, this paper leverage two visualization techniques, i.e., *representation inversion* and *feature visualization*, to demonstrate that features identified by the robust neural network are more discernable.   To summary, the main contributions of this paper are as follows:  - A comprehensive literature review about findings related to features of standard and robust models; - Train a $\ell_2$-norm robust neural network and visualize its features; - Leverage *representation inversion* and *feature visualization* to demonstrate the features in robust neural networks are more discernable.

The paper presents a study about the representations learned by neural networks trained using robust optimization — a type of optimization that requires the model to be robust to small perturbations in the data. Specifically, the paper presents results of ResNet-50 trained on ImageNet with standard optimization and robust optimization. The paper draws three main insights from studying the learned representations of the standard and robust networks. First, the representation of the robust network is approximately invertible. In other words, when recovering an image by matching the representation of a random image to the representation of a target image by adding noise, the recovered images are semantically similar to the target image; the recovered images look similar to a human. Moreover, this is also demonstrated with images from outside of the distribution of the training data. Second, the representation of the robust network, unlike the representation of standard network, shows semantically meaningful high level features without any preprocessing or regularization. This leads to the final insight, feature manipulation is easier in robust networks. This is demonstrated by adding noise to an initial image in order to maximize the activation of a specific higher level feature and stopping early to preserve most of the other features of the original image. 

The paper shows that the learnt representations of robustly trained models align more closely with features that the human perceive as meaningful. They propose that robust optimization can be viewed as inducing a human prior over learnt features. Extensive experiments demonstrate that robust representations are approximately invertible, can be visualized yielding more human-interpretable features, and enable direct modes of input manipulations.