Adversarial Examples Are Not Real Features
Authors: Ang Li
Yifei Wang
Yiwen Guo
Yisen Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Adversarial Examples
Adversarial Training
Generalization
Generative Models
TL;DR: Non-robust features are not cross-paradigmly useful features.

Abstract: The existence of adversarial examples has been a mystery for years and attracted much interest. A well-known theory by \citet{ilyas2019adversarial} explains adversarial vulnerability from a data perspective by showing that one can extract non-robust features from adversarial examples and these features alone are useful for classification. However, the explanation remains quite counter-intuitive since non-robust features are mostly noise features to humans. In this paper, we re-examine the theory from a larger context by incorporating multiple learning paradigms. Notably, we find that contrary to their good usefulness under supervised learning, non-robust features attain poor usefulness when transferred to other self-supervised learning paradigms, such as contrastive learning, masked image modeling, and diffusion models. It reveals that non-robust features are not really as useful as robust or natural features that enjoy good transferability between these paradigms. Meanwhile, for robustness, we also show that naturally trained encoders from robust features are largely non-robust under AutoAttack. Our cross-paradigm examination suggests that the non-robust features are not really useful but more like paradigm-wise shortcuts, and robust features alone might be insufficient to attain reliable model robustness. Code is available at \url{https://github.com/PKU-ML/AdvNotRealFeatures}.

Adversarial Examples Are Not Real Features
Ang Li Yifei Wang2* Yiwen Guo Yisen Wang
School of Electronics Engineering and Computer Science, Peking University
2 School of Mathematical Sciences, Peking University
Independent Researcher
National Key Lab of General Artificial Intelligence,
School of Intelligence Science and Technology, Peking University Institute for Artificial Intelligence, Peking University
Abstract
The existence of adversarial examples has been a mystery for years and attracted much interest. A well-known theory by Ilyas al. [19] explains adversarial vulner- ability from data perspective by showing that one can extract non-robust features from adversarial examples and these features alone are useful for classification. However, the explanation remains quite counter-intuitive since non-robust features are mostly noise features to humans. In this paper, we re-examine the theory from a larger context by incorporating multiple learning paradigms. Notably, find that contrary to their good usefulness under supervised learning, non-robust features attain poor usefulness when transferred to other self-supervised learning paradigms, such as contrastive learning, masked image modeling, and diffusion models. It reveals that non-robust features not really useful as robust natural features that enjoy good transferability between these paradigms. Meanwhile, for robust- ness, we also show that naturally trained encoders from robust features are largely non-robust under AttoAttack. Our cross-paradigm examination suggests that the non-robust features and robust features alone might be insufficient to attain reliable model robustness. Code available at https: /ttps:///tttte gigghu..bb.... cww. Pwo//KU--- Www.AA/AAAaaaaaeeaaeeaaaaaooaeae
really useful but more like paradigm-wise shortcuts,
Introduction
Alongside the human-level even superior performance of Deep Neural Networks (DNNs) in various
tasks [22, 14, 10), concerns on the existence of adversarial examples constantly rise, regarding them main threats that fool DNN classifiers with invisible perturbations [34. 12, 24, 26). Among many explanations of adversarial examples [11, 36, 31, 30, 44, 45, 48, 28, 2, 3], the robust and non-robust feature perspective developed by Ilyas al. [19] has received wide attention. Compared to previous explanations that regard adversarial examples as "bugs" of neural networks (i.e., the model), they claim that adversarial examples stem from non-robust "features" in the inputs (i.e., the data). Specifically, they argue that each example is composed of human-perceptibble robust features (invariant to attack) and human-imperceptible non-robust features (sensitive attack) that are both useful for classification. Under adversarial attacks, non-robust features from other classes are injected into the adversarial examples and lead to misclassification. To verify this point, they show that datasets containing only non-robust features can attain good classification performance; and similarly, datasets containing robust features can attain good robustness even with natural training. Supported by these observations, the perspective has been widely accepted ever since and many subsequential works are built upon their framework [5, 20, 37].
Equal Contribution.
Corresponding Author: Yisen Wang (yisen. wang @ (ww..cukuu.u.c)
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Natural
Rabust
Non
Figure I: Tiny-ImageNe instances containing natural, robust, and non-robust features, respectively. The robust and non-robust instances are generated following the iterative optimization procedure in Ilyas et al. [19] from random noise. The robust features are semantically aligned to natural images, while the non-robust features are always noise-like.
Although both robust and non-robust features are shown to be useful for classification, the two still have large discrepancies, particularly in perceptibility. As shown by Ilyas et al. [19], robust features usually have rich semantic information such as distinguishable edges and meaningful combinations of colors; on the contrary, non-robust features are always noise-like and meaningless to humans, as shown in Figure 1. Therefore, it is natural for robust features to be useful, while the usefulness of noise-like non-robust features still seems suspicious and counter-intuitive. One may raise a natural question: are non-robust faatusstfeaurre features real (useful) features?
In this paper, we aim re-examine robust and non-robust features in a wider context in order to understand the true distinction between them. Particularly, we notice that a major limitation of Ilyas al. [19] is the scoop of learning paradigms considered, as the usefulness of non-robust features is only evaluated on the supervised image classification. However, from the backdoor literature, we know that even a meaningless backdoor pattern can lead to the desired classification. A feature being useful for classification does not necessarily imply that it is truly useful. Therefore, it is reasonable to assume that truly useful features, e.g., those utilized by humans, should work well for wide range of learning paradigms instead of a single paradigm like the backdoor pattern.
Driven by the analysis, take the first step by defining the pross-paradigm usefulness of robust and non-robust features, as a viable measure of true usefulness that generalizes previous definitions of Ilyas et al. [19]. feature is defined to be truly useful only if it yields good representations across different learning paradigms. Aside from supervised learning, we further consider three modern self-supervised learning paradigms as the representatives: 1) contrastive learning that aligns augmented samples in the latent space, e.g., SimCLR [27, 6, 15]; 2) masked image modeling that predicts the masked patches from the unmasked context, e.g., MAE [9, 4, 16]; and 3) diffusion models that learn to restore images corrupted by Gaussian noise, e.g. DDPM [17, 33]. Among the four paradigms, supervised learning and contrastive learning are discriminative tasks, while masked image modeling and diffusion models are generative tasks. Previous studies show that all these different paradigms yield representations with good downstream performance when pretraining on natural images [6. 16, 50]. Therefore, we would intuitively expect that if non-robust features are as useful as natural image features, we can also learn good representations from non-robust features alone. Similarly, we can define cross-paradign robustness that evaluates whether a feature can yield robust representations (with robust prediction on downstream tasks) across different learning paradigms.
To validate their true usefulness, we follow the same procedure of Ilyas et al. [19] and construct robust (non-robust) datasets that only have robust (non-robust) features extracted from supervised models. To evaluate their cross posssdaar usefulness and robustness, we learn features with the four different learning paradigms on the constructed datasets and evaluate the learned features with a linear probing head for classification (a common evaluation protocol in self-supervised learning). Surprisingly, we find that there exist notable discrepancies in the cross-paradigm usefulness: on the three (transferred) self-supervised paradigms, robust features yield excellent performance that nearly matches natural features, while non-robust features yield much worse prediction that hardly usable, in sharp contrast to its own good performance supervised learning. It clearly conveys that non-robust features

---

are not as (cross-panee prrosdpa useful as robust/natural features. Therefore, unlike robust or natural images, non-robust features do not contain much meaningful information that is truly useful across different paradigms. Intuitively speaking, non-robust features are more like certain backdoor pattern of natural images subject to the chosen learning paradigm (e.g., supervised classification). As paradigm- teiiggm--iseeeee shortcuts, these features are essentially uninformative when examined with other learning paradigms. We further verify this point by showing that adversarial examples crafted under different paradigms can bardly transfer among each other. To this end, we conclude that robust features are truly useful features while non-robust features are not (in the cross-paradige sense). Unlike the counter- cennter-intuitive explanation of Ilyas et al. [19], our result justifies the human intuition that the noise-like non-robust features do not really capture the essence of the images.
Furthermore, we also re-evaluate the robustness of robust and non-robust features in this way. Interestingly, contrary to the findings in Ilyas et al. [19] that natural training on the robust dataset produces a robust classifier, we find these so-called robust features hardly show robustness when learned with other learning paradigms. We also observe that the supervised learned classifier is non-robust under more reliable attacks like AutoAttack [7]. Thus, we arrive at the conclusion that on real-world datasets, both robust and non-robust features extracted by Ilyas et al. [19] are actually non-robust, in both in in-paradigm and pross-paradigm senses. Although robust features are shown to exist in toy models when explicitly designed [39], there is by far no evidence that robust features exist in woal-world datasets, at least not extractable.
To summarize, the main contributions of this work are three folds:
In order to evaluate robust and non-robust features in a broader context, we generalize the notions on the usefulness and robustness of robust and non-robust features to cross- paradigm sense. This perspective enables us to get rid of potential paradigm-wise shortcut effects and evaluate true feature informativeness.
For usefulness, we find that robust features are both in-paradigm and cross poos--paddiggm useful like natural features. Instead, non-robust features are only useful in in-paradigm, and their usefulness dramatically degrades when transferred to other paradigms, suggesting that they are more like paradigm-wis wiis shorteuts instead of real features. We further verify this point by showing that adversarial examples also have poor transferability across different paradigms.
For robustness, we find that robustness obtained from the constructed robust dataset is actually false sense of robustness when evaluated eross-paradigmly on more reliable attacks. The loss of this key evidence would raise the question of whether robust features really exist in real-world datasets.
Last but not least, although the main messages of this paper seem rather negative, this view also points out potential avenues to better adversarial robustness. In particular, the paradigm-wise behaviors of non-robust features suggest that it may be inadequate to perform adversarial training on a single learning paradigm, and mixture of adversarial training on multiple paradigms may come to the aid. In the meantime, the non-robustness robust dataset indicates a more comprehensive view of adversarial vulnerability: input data (or features) are not the only source of adversarial vulnerability, and only combating the vulnerabilities in both data and models can lead to true robustness.
2
A Cross-Paradigm View of Robust and Non-robust Features
2.1 Background
Robust and Non-Robust Features. As in llyas al. [19], features are defined as function mapping from input space to real numbers, namely f X' - R. The robust and non-robust features are then described with the following definitions for a binary classification task:
uuseful features: For a dataset D, a feature f is p-useful if the feature is correlated with the label:
B( f(x)] 2 p.
(1)
7-robustly useful features: For a p-useful feature, it is regarded as 7-robustly useful it remains useful under certain range of perturbation:
B(z,))~D| inf y f(x + 8)) 2 n.
BEA
3

---

Useful, non-robust features: These feature are the ones that are p-useful features, but are not y-robust features for any ? 0.
Construction of Robust and Non-Robust Datasets. Given a classifier C' and a dataset D, llyas et al. [19] propose to construct a dataset D which satisfies:
T B(ww))l((().) vl if f Fc E (( y] =
otherwise,
where Fc represents the set of features utilized by C. Denoting g as the mapping from input the representation layer in C, the new instance z; is obtained from x through following optimization:
min (g(x)) ((xr)ll2
If the classifier C' (e.g., ResNet-50) is adversarially trained, the constructed dataset DD is regarded as robust dataset. As for standardly trained classifier, the dataset DD is regarded as non-robust datatset.
Basic Conclusions of Ilas et al. [19]. Utilizing the constructed datasets, Ilyas et al. [19] empirically show that models trained on the non-robust dataset are useful, i.e., they attain good classification performance on natural test data, comparable to models trained on the natural dataset. This supports their claim that non-robust features are sufficiently useful to obtain good generalization. This perspective well explains the existence of adversarial examples as well as their transferability, since adversarial examples contain non-robust features from the misclassifed classes that are useful for different models. Therefore, as they put it, adversarial examples are features, not bugs. Also, they also show that natural training on the robust dataset (containing robust features alone) can yield robust models. In this way, they view adversarial vulnerability purely from a data (or feature) perspective, regarding them as a result of different priors on extracted features.
Self-Supervised Learning. Aside from the supervised learning paradigm studied in Ilyas et al. [19), self-supervised learning (SSL) also received wide attention in recent years. Without manual labels, SSL methods utilize self-supervision to learn meaningful features from unlabeled data and have achieved impressive progress in recent years [6. 16, 17]. Generally speaking, an SSL algorithm pre- trains a feature encoder f X + Z, mapping from the input space to the latent space Z. Afterward, the learned features are typically evaluated on the so-called linear probing task. Specifically, given a labeled dataset (x, y) ~ D (usually a subset of the pretraining data), we train linear classifier p Z + y on top of learned features and use its classification accuracy (with the composed classifier p f) on the test data as measure of the usefulness of learned features (representations).
2.2 Cross Pross-Paradign Notions of Robust and Non-Robust Features
In this part, we generalize the definitions of robust and non-robust features to wider context beyond the supervised classification. For a rigorous discussion, we introduce some general definitions of the paradigm-wil wise feature usefulness and robustness and then define true usefulness and robustness in the cross-paradigg sense.
Paradigm- Wise Definitions. To facilitate features to generalize across different paradigms, we differ from Ilas et al. [19] and adopt a more classic definition of features, filters in the input space, i.e., g : X X. Note that common image features like textures, edges, and colors naturally fall into this category. We define a learning paradigm T as a specific learning algorithm that learns a feature encoder with loss function LT over given dataset D:
fT = arg min VED LT (f()),)))
(5)
Incorporating feature g, the minimization problem can be further specified as
p()) = arg min EE. (
(6)
With each learning paradigm T. learned representations with a linear probing head p on top of f( on D. The usefulness of a feature g relevant to the paradigm T is defined follows:
train an encoder (ممر for feature g. Then we evaluate the
U(g. D,T) = max VED
(p( (1)) (x)) v].
PEP

---

That is, a feature is useful paradigm T if the linear probing head shows good classification performance with representations learned from the feature g under the learning objective defined by the paradigm T.
Accordingly, the paradigm- gms feature robustness is defined the remaining feature usefulness under local adversarial perturbations,
Rgg, D, T)
max R RY,EED max 1p (x 82)) v)-
PEP
[
In other words, feature is robust paradigm T if standardly trained linear probing head shows good robustness with representations learned from the feature under T.
From this perspective, the conventional definitions of feature usefulness and robustness of Hyas et al. [19] correspond to special case of our definitions when choosing the supervised learning task as the learning paradigm. In this way, the two processes, feature learning and feature evaluation, actually share the same learning objective, which may introduce spurious paradigm- wrisd shortcut effects. To get rid of this potential risk, we propose to re-define these concepts in cross-paradigm sense.
Cross-Paradign Definitions. Given diverse of learning paradigms T {TT T2. Ts}, define feature to truly useful if it can generalize across multiple learning paradigms. Specifically, we define cross parsdigm usefulness the worst paradigm wise usefulness among these paradigms,
(Cross-Paradigm Usefulness) CUT(9, D) min (g, D)
(9)
TET
Similarly, we define cross-paradigm robustness the worst robustness
(Cross-Paradigm Usefulness) CRT D)
min Rig, D)
(10)
TET
Nevertheless, since different paradigms usually attain performance on different levels, a direct comparison their absolute performance may be unfair. In the worst case, if certain paradigm has poor performance even when pretrained on the raw images, it would dominate other paradigms when computing cross-paradigm metrics, which, however, does not reflect the true feature usefulness. To mitigate this potential issue, we propose the paradigm- wise relative metrics the ratio between the performance of the chosen fcature and the performance of using all features (raw inputs),
(Relative Usefulness) (g, D,T) Ug,
(11)
U(D,T)
R(g, D,T)
(Relative Robustness) RR(g D)T)
(12)
R(D,T)
where given an encoder fr learned from the raw images under the paradigm T, we define
UDD,T) max Ea weeelleeeel v), RDD,T) max Ex,yED max [p(fr(( 6x)) 9).
pEp
pEP
[
Accordingly, can define the cross-paradigm relative usefulness and robustness follows:
(Cross-Paradigm Relative Usefulness) CRU = min (U, D,T),
(13)
TET
(Cross-Paradigm Relative Robustness) CRRT min RR(g, D,T).
(14)
3 Cross-Paradigm Usefulness of Robust and Non-robust Features
Built upon the evaluation framework established in Section 2.2, we first investigate the cross- paradigm usefulness of robust and non-robust features real-world datasets in this section. With this generalized notion of feature usefulness, we are trying to answer the key question: are robust and non-to features truly useful?
3.1 Setup
Data Construction. We include two commonly adopted datasets in our study, CIFAR10 [21] and 200gee [49]. Aside from the raw images, following the same construction process in
5

---

Table Evaluation of relative usefulness of robust features and non-robust features on four learning
paradigms: MIM, CL. DM, and SL. The cross pross-paradigm relative usefulness is computed as the worst
relative usefulness over the four paradigms. We also include the usefulness scores (evaluated only on the supervised task) reported in Ilyas et al. [19] for comparison.
Data
Feature
MIM
CL
DM
Cross. Rel. Useful.
Ilyas et al. [19]
Robust
0.896
0.791
0.831
0.880
0.791
0.854
CIFAR-10
Non-Robus
0.307
0.433
0.512
0.914
0.307
0.823
Robust 0.691
0.775
0.861
0.880
0.672
0.407
Tiny-ImageNer
Non-Robust
0.134
0.074
0.141
0.645
0.134
0.396
Ilyas et al. [19] (details in Appendix A), we further construct a robust version and non-robust version for each dataset, which only contain robust and non-robust features, respectively.
Learning Paradigms. Besides supervised learning with the cross-entropy loss, we also include three self-supervised learning paradigms for a pross-paradigm evaluation: SimCLR [6] for Contrastive Learning (CL), MAE [16] for Masked Image Modeling (MIM), and DDPM [17] for Diffusion Models (DM). We then train linear probing heads for the encoders on the same dataset that it was trained on. The probing heads are directly applied to the output of the encoder for SimCLR L and MAE models, while for DDPM model, since the U-Net encoder has complex high-dimensional hidden features, we use the global average pooled feature in the fourth upsampling layer in U-Net following Xiang al. [50], which is shown deliver excellent linear classification performance on par with other self-supervised learning paradigms. Also, we train a ResNet-50 on the datasets with Supervised Learning (SL) as our baseline. To summarize, the four learning paradigms considered in this work are T = (MIM, CL. DM, SL). After pretraining, we evaluate the classifiers' performance on the test sets of the two datasets. More details on training and evaluation are Appendix A.
3.2 Non-robust Features are Not Cross-paradigmly Useful
We first examine the paradigm-wise usefulness of robust and non-robust features extracted from the supervised models. As shown in Table 1, models pretrained from robust datasets (containing only robust features) show comparable performance those pretrained from the natural datasets (containing all features). In comparison, models pretrained from the non-robust datasets (containing only non-robust features) differ a lot across different paradigms: they work well for learning from the supervised task (ResNet-50), but much worse on all the other SSL paradigms (SimCLR, MAE, DDPM). It shows that in contrary to the perceptually aligned robust features that useful for different paradigms, the noise-like non-robust features do not really contain meaningful information that is also useful for other SSL paradigms. Instead, these non-robust features extracted from supervised models only show usefulness on the supervised task. This sharp distinction suggests that non-robust features are more like paradigm- wise shorteut features rather than being truly useful.
For a comprehensive evaluation, we further compute the cross- poossppardiigm relative usefulness following the definitions in Section 2.2. As shown in Table 1, the relative usefulness of non-robust features is significantly lower than that of robust features under various self-supervised learning paradigms, The overall cross-paradigm ends up being 0.307, which is much smaller than robust features (0.791) In comparison, these two kinds features show similar performance (0.854 and 0.823) in Ilyas al. [19] when only considering the supervised task. shows that from paradigm- wrisi view, the usefulness of features dramatically differs from the case where only supervised learning is considered. Again, verifies that non-robust features are not as useful as natural and robust features when transferred to many other learning paradigms. For completeness, in Appendix C.I, we further examine the cross- parsdiga transferability of the non-robust features extracted from the three SSL paradigms. Similarly, we find that these SSL non-robust features are non-transferable as well.
4 Cross-Paradigm Robustness of Robust Features
The results above challenge the claim from Ilyas et al. [19] that non-robust features are useful features by showing that non-robust features are not really useful in the cross-paradigm sense. Aside from the usefulness of non-robust features, another notable finding in Ilyas et al. [19] is that robust features

---

alone are enough for attaining model robustness. Specifically, they extract robust version of a given dataset (e.g., CIFAR-10) by distilling from robust classifier such that the transformed samples only contain robust samples, and they show that a naturally trained classifier on the constructed "robust dataset" has good robustness. In this section, we further examine the robustness of the extracted robust features on different learning paradigms.
4.1 Setup
We construct the robust versions of CIFARIO with a supervised -obbuttst ResNet-18 encoder [46] and self-supervvsed fx-robuss ResNet-18 18 encoder [23], largely following the convention of Ilyas et al. [19]. We then train encoders on the robust datasets with the learning paradigms set T = (MIM, CI., DM, ST.) and use the learned representations to naturally train a linear head. Finally, we evaluate the robustness of the composed classifiers (encoders + linear heads) with a modern reliable attack method, AutoAttack [7], under the perturbation budget of 4/255, instead of the PGD attack [25] that may lead to over-estimated robustness [7]. More experimental details can be found in Appendix A.
4.2 Robust Features are Not Robust both In-Paradigm and Cross Cross-Paradigm
We summarize the results in Table Viewing from the in in-paradigm results in Table 2a, we can observe that in supervised learning, PGD attacks can over-estimate the robustness of models trained from robust datasets, e.g., 15.71 (PGD) v.s. 3.65 (AutoAttack). When evaluated under AutoAttack, robust datasets generated from either E2-robust or -robuutstt models show poor robustness, being hard to match the robustness of adversarially trained source models. In other words, in practice, we believe it difficult to attain good robustness with only robust features (extracted following Ilyas et al. [19]). Their original conclusion on the good robustness of robust datasets could be due to the unreliability of PGD-based robust evaluation (also shown in the AutoAttack paper [7]). The severe over-estimation on robust datasets is potentially due to gradient obfuscation or masking [1] since these robust datasets are essentially generated with the PGD attack.
Extending the finding, we also evaluate the robustness of robust features on other self-supervised learning paradigms in Table 2b. Likewise, robust features also perform poorly and do not attain non-trivial robustness. Interestingly, when training on generative models like MIM and DM, robust features obtain slightly better robustness compared to the discriminative ones (CL and SL), even on natural data. Overall, the cross-paradigm robustness (defined in Section 2.2) still bottlenecked by the SL itself. Based these results, we conclude that robust features still cannot achieve model robustness alone (in either in- paradigm or cross- passsdgg sense), without model-level interventions like ad adversarial training. It suggests that adversarial vulnerability may not come from the data alone, and a joint training strategy against both data-level and model-level vulnerabilities may be needed to attain real robustness. It worths noting that Tsilivis et al. [38] also found the non-robustness of the robust dataset of Ilyas et al. [19] in the supervised domain, while our results here give a full examination of robust features from both in-paradigm and cross-paradigm aspects.
5 Cross-Paradigmn Transferability of Adversarial Attacks
The transferability across different models, even with different architectures [47, 40), is a fascinating fact of adversarial examples. The perspective of Ilyas et al. [19] provides a natural explanation for the phenomenon since non-robust features in adversarial examples are inherently useful such that they can affect the prediction of different models. Nevertheless, our studies Section reveal that non-robust features are only paradigm-wise but not cross-paradigm useful. It implies that adversarial transfcrability is also paradigm-wist wise. In other words, adversarial examples can hardly transfer between different paradigms. To verify the hypothesis above, further investigate the cross-paradiges transfcrability of adversarial attacks in this section.
5.1 Cross-Paradign Transferability of Attack Objectives
Since different paradigms mainly differ by their learning objectives, adversarial attacks w.r.f. different learning objectives (e.g., g. SI., CL., MIM, DM) can generate adversarial examples of different paradigms. To study the transfcrability between different attack objectives, consider two ResNet- 18 backbone
7

---

networks (learnd with SL and CL, respectively), attack them with an objective, and observe the change of both learning objectives.
Setup. We consider two objectives: InfoNCE loss [27] in CL and Cross Entropy (CE) loss in SL.. Adversarial examples generated with lint [utoAAttack[7] bounded by 4/255, using the default attack settings on CIFAR-10 1- (details in Appendix A). To be consistent, we compute the InfoNCE loss for supervised encoder by appending re-trained projection head on top of the fixed encoder. When computing the gradient during attacks, we also retain the standard data augmentations in SimCLR [6] and differentiate through these operators with the Kornia package [32]. For further ablation study on the influence of these components on transferability, see Appendix B.
Mar
Rouck Inle Atuuc CE
Atiack Allack Ct
Allak PO
Allack
Allack
(a) CE loss of fer.
(b) InfoNCE loss of Scr. (c) CE loss of Sst.
(d) InfoNCE loss of SSL
Figure The change loss value v.s. the attack iteration steps when using different attack objectives. CE loss (blue lines) or InfoNCE loss (orange lines), and backbones, trained by SL. (Figure 2a & Figure 2b) or CL (Figure 2c Figure 2d) from different paradigms.
Results. We plot the change of loss values during the attacking process in Figure 2, from which we can identify general trend: there is a significant difference between the changes of different objectives, and maximizing one attack objective has limited effect on the other objective. This distinction clearly implies that adversarial examples generated with attack objectives from different paradigms have poor transferability among each other.
5.2 Cross-Paradigm Transferability of Backbone Encoders
Aside from the attack objective discussed in Section 5.1, different paradigms also learn different paradigm- wiie feature encoders that also have a large influence on the generation of adversarial examples. To further analyze the paradigm- wiise influence of feature encoder on adversarial trans- ferability, we generate adversarial examples with backbone encoders obtained by the four different paradigms, while keeping the attack objective to be the same CE loss (using the linear head learned for each paradigm). Besides the default (eSSL-))0()L-RNN model, we further include three different
Table 2: Absolute robustness of robust features on four paradigms: MIM, CL, DM, and SL, on CIFAR- 10. The robust features are extracted by pretrained models from two different source paradigms: supervised learning and contrastive learning. Different from Ilyas et al. [19] that use 1000-step PGD)
[25] for robust evaluation, we adopt a more reliable attack algorithm, AutoAttack [7].
(a) Robustness evaluation (with PGD and Auto for classifiers naturally trained with robust datasets (SL).
Eg, E 0.5 AutoAttack
loc. 4 4/255 AutoAttack
Source Model
PGD-1000
PGD-1000
robutt classifier
15.71 + 0.21
3.65 + 0.58
6.36 + 1.66
0.54 0.16
robustt classifier
13.96 + 0.12
5.03 0.16
17.63 = 0.16
3.52 0.31
(b) Cross-paradigm robustness (AutoAttack by default) of encoders naturally trained from natural or robust datascts. The robust datasets are generated from &oo -robust encoders trained with the SL CL paradigm.
Training Dataset
CL
MIM
DM
SL
Cross. Rob.
Natural
0.22 + 0.03
2.33 + 0.12
3.41 + 0.32
0.00 + 0.01
0.00
Robust w/ SL model
0.23 + 0.01
2.73 + 0.33
0.20 2.54 0.200
3.52 - 0.31
0.23
Robust w/ CL model
1.28 + 0.01
1.16 + 0.23
1.19
2.21 + 0.29
1.16

---

DM
St-R
St.nCC
MIM
SLIN SLiNC
Figure 3: The cross-paradigm robustness of adversarial examples generated with encoders from different learning paradigms. The (A, B) cell represents the accuracy of adversarial examples generated with an A-paradigm model (encoder with linear head) when evaluated B-paradigm model (encoder with linear head). Darker colors (i.e.. higher accuracy) indicate worse transferability of adversarial examples.
supervised models, DenseNet-121 121 (SL-DN) [18], and Inception- V3 (SL-InC) [35] as baselines. We plot the transferred adversarial robustness in the confusion matrix in Figure 3.
The results in Figure 3 demonstrate two important messages: 1) adversarial examples transfer well across different architectures (SL-RN, SL-DN, SL-InC) under the same paradigm (SL). and 2) adversarial examples transfer poorly between different paradigms, even under the same architecture (for example, both CL. and SI. adopt ResNet-50 backbones). It suggests that for adversarial transferability, paradigms have much more influence than model architectures. This phenomenon can be well explained by the paradigm-we wise shortcut-like behaviors of non-robust features (Section 3), and it indicates that adversarial transferability is also largely paradigm-wist phenomenon.
5.3 Relationship to Natural Transferability between Paradigms
In the discussions above, demonstrate that adversarial examples do not have (good) transferabiliiyy between different learning paradigms. We note that this discovery in adversarial transferability not in contradiction the good natural transferability between different paradigms. Indeed, many existing works [6, 15, 41, 16, 50, 52] show that representations learned from SSL have good downstream performance, particularly when evaluated with linear probing. Moreover, theoretical guarantees on the downstream classification performance have also been established for contrastive learning [13, 42, 43, 8), non-contrastive learning [53], and masked image modeling [51]. Since adversarial examples are essentially out-of- -iittribbitiitoo examples (not drawn from the natural data distribution), the generalization guarantees on natural data do not apply. The fact that these paradigms have good natural transferability and poor adversarial transferability serves as another piece of evidence for our understanding that the misclassification of adversarial examples caused by paradigm-wise shorteuts instead of real useful features.
Conclusion
In this paper, we have investigated the real usefulness and robustness of robust and non-robust from a wider context. By studying their behaviors across four different learning paradigms, we have found that robust features are as useful as natural features, while non-robust features generated by attacking supervised models become largely useless when transferred to other self-supervised learning paradigms, indicating that non-robust features are not real features but more like paradigm- wise shortcuts. Meanwhile, we have also shown that robust datasets containing only robust features are insufficient to attain model robustness under AutoAttack,, indicating that feature non-robustness not the only source of adversarial vulnerability. Posed and non-robust features, this work advocates the idea that their real usefulness and robustness should be examined under cross- pross-paradigm perspective well as more reliable attacks.
challenge to common beliefs of robust

---

Acknowledgements
Yisen Wang was supported by National Key R&D Program of China (2022200160304 National Natural Science Foundation of China (62006153, 62376010, 92370129), Open Research Projects of Zhejiang Lab (No. 2022RC0AB05), Beijing Nova Program (20230484344), and -CF-Baichuan-EB Fund.
References
[1] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In ICML, 2018.
[2] Yang Bai, Xin Yan, Yong Jiang, Shu-Tao Xia, and Yisen Wang. Clustering effect of (linearized) adversarial robust models. In NeurIPS, 2021.
[3] Yang Bai, Yuyuan Zeng, Yong Jiang, Shu-Tao Xia, Xingjun Ma, and Yisen Wang. Improving adversarial robustness via channel-wise activation suppressing. In ICLR, 2021.
[4] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In ICLR, 2022.
[5] Philipp Benz, Chaoning Zhang, and In So Kweon. Batch normalization increases adversarial vulnerability and decreases adversarial transferability: A non-robust feature perspective. In CVPR, pages 7818 88227, 2021.
[6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[7] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with ensemble of diverse parameter-free attacks. In ICML, pages 2206-2216. PMLR, 2020.
[8] Jingyi Cui, Weiran Huang, Yifei Wang, and Yisen Wang. Rethinking weak supervision in helping contrastive learning. ICML, 2023.
[9] Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[11] Alhussein Fawzi, Seyed-Mohsen Moosavi-Dezfoolli and Pascal Frossard. Robustness of classifiers: from adversarial to random noise. NeurIPS, 29, 2016.
[12] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. In ICLR, 2015.
[13] Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. In NeurIPS, 2021.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[15] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick, Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[16] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 16000-16009, 2022.
[17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NewrIPS, 2020.
10

---

[18] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
[19] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In NewrlPS, 2019.
[20] Byunggill Joe, Sung Ju Hwang, and Insik Shin. Learning to disentangle robust and vulnerable features for adversarial detection. arXiv preprint arXiv: 2019.
[21] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[22] Alex Krizhevsky, Ily Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, 2012.
[23] Rundong Luo, Yifei Wang, and Yisen Wang. Rethinking the effect of data augmentation in adversarial contrastive learning. In ICLR, 2023.
[24] Xingjun Ma, Yuhao Niu, Lin Gu, Yisen Wang, Yitian Zhao, James Bailey, and Feng Lu. Understanding adversarial attacks on deep learning based medical image analysis systems. Pattern Recognition, 2020.
[25] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
[26] Dantong Niu, Ruohao Guo, and Yisen Wang. Morié attack (ma): new potential risk of screen photos. In NeurlPS, 2021.
[27] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 2018
[28] Jie Ren, Die Zhang, Yisen Wang, Lu Chen, Zhanpeng Zhou, Yiting Chen, Xu Cheng, Xin Wang, Meng Zhou, Jie Shi, al. A unified game-theoretic interpretation of adversarial robustness, In NeurlPS, 2021.
[29] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. MICCAL Springer, 2015.
[30] Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Ad- versarially robust generalization requires more data. Advances in neural information processing systems, 31, 2018.
[31] Ali Shafahi, W Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein. Are adver- sarial examples inevitable? arXiv preprint arXiv:1809.02100 2018.
[32] Jian Shi, Edgar Riba, Dmytro Mishkin, Francesc Moreno, and Anguelos Nicolaou. Differen- tiable data augmentation with kornia, 2020.
[33] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2020.
[34] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Good- fellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
[35] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
[36] Thomas Tanay and Lewis Griffin. A boundary tilting persepective on the phenomenon of adversarial examples. arXiv preprint arXiv:1608s 07690, 2016.
[37] Lue Tao, Lei Feng. Hongxin Wei, Jinfeng Yi, Sheng-Jun Huang, and Songean Chen. Can adversarial training be manipulated by non-robust features? NeurIPS, 35:26504-26511 2022.
11

---

[38] Nikolaos Tsilivis, Jingtong Su, and Julia Kempe. Can we achieve robustness from data alone? arXiv preprint arXiv:2207.11727, 2022.
[39] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry, Robustness may be at odds with accuracy. In ICLR, 2019.
[40] Xin Wang. Jie Ren, Shuyun Lin, Xiangming Zhu, Yisen Wang, and Quanshi Zhang. A unified approach to interpreting and boosting adversarial transferabilityy In ICLR, 2021.
[41] Yifei Wang, Zhengyang Geng, Feng Jiang, Chuming Li, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Residual relaxation for multi-view representation learning. In NeurIPS, 2021.
[42] Yifei Wang, Qi Zhang. Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is ladder: new theoretical understanding of contrastive learning via augmentation overlap. In ICLR, 2022.
[43] Yifei Wang, Qi Zhang, Tianqi Du, Jiansheng Yang, Zhouchen Lin, and Yisen Wang. A message passing perspective on learning dynamics of contrastive learning. In ICLR, 2023.
[44] Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In ICML, 2019.
[45] Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR, 2020.
[46] Zeming Wei, Yifei Wang, Yiwen Guo, and Yisen Wang. Cfa: Class-wise calibrated fair adversarial training. CVPR, pages 8193-8201 2023.
[47] Dongxian Wu, Yisen Wang, Shu-Tao Xia, James Bailey, and Xingjun Ma. Skip connections matter: On the transferability of adversarial examples generated with resnets. In ICLR, 2020.
[48] Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. In NeurlPS, 2020.
[49] Jiyy Wu, Qixiang Zhang, and Guoxi Xu. Tiny ImageNet challenge, 2017.
[50] Weilai Xiang, Hongyu Yang. Di Huang, and Yunhong Wang. Denoising diffusion autoencoders are unified self-supervised learners. arXiv preprint arXiv:2303.09769 2023.
[51] Qi Zhang, Yifei Wang, and Yisen Wang. How mask matters: Towards theoretical understandings of masked autoencoders. NeurIPS, 35:27127-27139, 2022.
[52] Qi Zhang, Yifei Wang, and Yisen Wang. On the generalization of multi-modal contrastive learning. In JCML, 2023.
[53] Zhijian Zhuo Yifei Wang, Jinwen Ma, and Yisen Wang. Towards unified theoretical under- standing of non-contrastive learning via rank differential mechanism. In ICLR, 2023.
12

---

A
Experimental Details
Here we elaborate the experimental and technical details. First, we specify how the robust/non-robust datasets are constructed in our experiments. Then we detail the pre-training and linear probing hyper-parameters. Finally, we illustrate the settings of our robustness evaluation framework.
A.1 Dataset Construction
We largely follow Ilyas et al. [19] in the process of dataset construction. Unless specified, we use the robust and non-robust datasets generated from images for experiments (Section 3, Section 5). In Section 4, we generate the robust CIFARIO with both supervisedly trained and contrastively trained ResNet- 18. All constructed datasets are obtained via lg PGD optimization. The detailed settings are listed below
Table 3: Experimental configurations in dataset construction. We generate robust and non-robust CIFAR10 10 starting from images (initialized with randomly drawn different images in the dataset) or from noise (random Gaussian noise).
Dataset
Iteration
Step Size
E-Cons.
Robust CIFAR10 10 (From Image)
1000
1.0
0
Robust CIFARIO (From Noise)
10000
1.0
QC
Non-Robust CIFARIO (From Image)
1000
0.1
0,5
Non-Robust CIFARIO (From Noise)
10000
0.1
0.5
Robust Tiny-mmageNet-2000 (From Image)
2000
1.0
00
Non-Robust 200 (From Image)
2000
0.1
0.5
A.2 Pre-Training
For evaluating the usefulness and robustness of features (Section 3, Section 4), we use ResNet- 18 [14] for SL, MAE [16] backboned by ViT-t [10] for MIM, ResNet-18 for CL. and DDPM [17] backhone by UNet [29] for DM all of the three versions of CIFARIO [21] and Imay-ImageNet-200 [49].
We additionally use supervisedly trained ResNet-50, DenseNet-50 [18], and Inception V3 [35] for discussion the paradigm-wa transferability adversarial examples (Section 5). We list the hyper- parameter configurations below. We pick the default settings commonly adopted by the community when training the models and feel it unnecessary to intentionally optimize hypee-parameters since main research goal is focused on rethinking the essence of non-robust features.
Table 4: The pyper-paramettr settings of pre-training on the three versions of CIFARIO.
Paradigm
Model
Epoch
B-Size
LR
LR Sche.
W-Decay
Data Aug.
ResNet-18
50
128
le-1
CosineAmnealingL..
5e-4
ResNet-50
50
128
le-1
CosineAnnellingL..
5e-4
SL
5e
50
128
le-1
CosineAnnealingLL.
5e-4
Inception V3
128
lc-1
CosincAnnealingL.R
5c-4
CL
ResNet-18
2000
512
le-4
CosineAnnellingL.R
le-5
MIM
MAE(ViT-))
2000
512
2e-4
LambdaLR
5e-2
DM
DDPM(UNet)
1000
128
2e-4
Lambdal.R
0
A.3 Linear Probing
We use linear probing to evaluate the representation learned by our encoders and we attach the trained linear heads to the encoders to transform them into classifiers. (Section 3, Section 4, Section 5).
13

---

Table 5: The settings of pre- pre-training on the three versions of JinyIImageNet-200
Paradigm
Model
Epoch
B-Size
LR
LR Sche.
W-Decay
Data Aug.
SL
ResNet-18
100
128
le-2
CosineAmnealingL.R
le-5
CL
ResNet-18
1000
512
2e-3
CosineAmnealing..R
le-5
MIM
MAE(VIT-b)
1000
512
2c-4
LambdaLR
5e-2
V
DM
DDPM(UNet)
1500
128
2e-5
Lambdal.R
Table 6: Hyper- configuration of linear probing.
Dataset
Epoch
B-Size
LR
LR Sche
W-Decay
Data Aug.
CIFARIO I0 Tiny-ImageNet-200
30
256
0.1
CosineAnnealingL..
5e-4
50
256
0.01
CosineAnnealingLRR
le-5
A.4 Robustness Evaluation
To replicate the results reported in Ilyas et al. [19], we use foe PGD- [25] bounded by E 4/255 and f2 PGD- 1000 bounded by = 0.5 as baseline evaluation. Besides, we use fox AutoAttack bounded by ε = 4/255 for more reliable evaluation. Due to the randomness induced in the encoding process of MAE and DDPM, we use the randomized version of AutoAttack when evaluating their robustness, otherwise, we use the standard version AutoAttack.
B Ablation Study on Cross-Paradigm Transferability
In Section 5.1, we have shown that adversarial examples generated by attacking contrastive learning (CL) have poor transferability on attacking supervised learning (SL). To further investigate how this happens, we conduct an experiment that gradually ablates each part of the InfoNCE loss.
Specifically, we notice three major differences when computing the CL and SL objectives: 1) CL usually adopts heavy data augmentations while SI. does not; 2) CL. adopts a projector head after the embedding while SL. does not; and 3) CL and SL. adopt different objectives: InfoNCE loss and CE loss. As shown in Table ablating the augmentation and/or projector indeed brings a better attack success rate (lower robustness), since it bridges the differences between two paradigms. Following the same vein, we further consider ablating the InfoNCE loss with a simple alignment loss between the natural example and adversarial example x', i.e.,
Tadv
arg max fx f
(15)
x!
We can see from Table 7 that this alignment-only attacking objective obtains significantly lower robustness, though the difference to attacking CE is still large. This ablation experiment shows that the distinction between paradigms is not absolute, and gradually eliminating their gap. We leave a more comprehensive study on this aspect to future work.
improve their transferability by
Table 7: Ablation study of model components when transferring adversarial examples generated w.r.t the CL objective to the classification task on CIFAR-10.
Paradigm
Attack Configuration
Robustness (%)
No Attack
78.51
InfoNCE loss Projector + Augmentation (default)
70.31
InfoNCE loss Projector
70.21
CL
InfoNCE loss Augmentation
71.09
InfoNCE loss
59.76
Alignment-only loss
33.20
SL
CE
0.78
14

---

C Cross-Paradigm Transferability of Non-robust features from SSL Paradigms
In the main paper, we have mainly examined non-robust features extracted from SL (supervised learning) model and show their non-transferability
cross-paradigm sense. For completeness. we further examine non-robust features generated from the other three SSL. paradigms considered in our work: CL, MIM, and DM. We follow the same hyperparameter setting as Ilyas et al. [19] for constructing non-robust dataset (Section 2.1) and for each paradigm, we generate adversarial examples by optimizing the perturbation to maximize the SSL pretraining objective per se (mode details below in Appendix C.1).
Table Cross-paradign transferabilityy of non-robust features generated from SSL paradigms. Here the non-robust datasets are generated with noise initialization following the setting of Ilyas et al. [19].
Accuracy
SL.
CL.
MIM
DM
CL-non-robust
18.71
60.02
10.09
13.66
MIM-non-robust
12.02
21.65
22.19
15.47
DM-non-robust
14.07
14.11
18.07
31.08
Table 9: Here the non-robust datasets are generated with noise initialization following the setting of Ilyas et [19]. Here the non-robust datasets are generated with random image initialization following the setting of Ilyas et al. [19].
Accuracy
SL
CL
MIM
DM
CL-non-robust
22.18
41.80
14.33
11.91
MIM-non-robust
14.11
9.29
32.80
21.78
DM-non-robust
12.47
9.86
18.25
39.41
Results are shown Table 8 (with random noise initialization) and Table (with random image initialization). We still observe the same trend observed in the SL-induced non-robust dataset (Section 3), where non-robust features show much better usefulness in-paradigm compared to that of other paradigms. Thus, non-robust features pretrained from SSL paradigms.
conclude that the cross-paradigm tratransferrbbiity still holds for
C.1 Implementation Details
Constrastive Learning: We leverage the non-robust features hidden in CL models by aligning the adversarial examples with the clean images. Specifically, the adversarial examples are obtained via the following optimization
CL. 57 arg min log exp(/((6)//7esp((((t))))
(16)
6;
Σ exp(f(6))/r) exp f(x/)/7
where 6/- denote the n adversarial examples, - denote the n clean image, i.e., the attack target within a batch and f the encoder being attacked. The non-robust features generated this way look like random noise but are semantically close to the clean images for the model, which qualifies them for the definition of adversarial examples.
Masked Image Modeling & Diffusion Model: The non-robust features for MIM and DM are obtained by maximizing the possibility of recovering the target (the clean image) from the adversarial examples. Denoting the procedure random masking in MIM and adding Gaussian noise in DM as Т, we can express the adversarial optimization of the paradigms follows
MIM: 8
arg min [TT()) 2=13
(17)
DM: 8 arg min [TT())) (x (21
(18)
15

---

with f denoting the model and z denoting the clean image. Conceptually, we assume the non-robust features of MIM to be invisible perturbations that contain information for the model to reconstruct the clean image from it. With the diffusion models predicting noise added to the input, the non-robust features of DM are similarly defined as small perturbations fooling the model to predict the difference between the clean images and the adversarial examples. Note that we fix the timestep used in generating the adversarial examples to the same as the one used in turning the DM into classifiers.
16


Reviews:
[{"review_id": "t5BMgfD1E2", "reviewer": "Reviewer_WcXm", "summary": "In this study, the authors dispute the argument from \"Adversarial examples are not bugs, they are features,\" where it was suggested that adversarial examples exist due to non-robust yet useful image features. They used self-supervised learning algorithms on both robust and non-robust datasets to test this theory. The findings contradicted the hypothesis, revealing that the self-supervised models trained on non-robust datasets didn't generalize well, indicating that non-robust features aren't universally useful for different training setups.\n\nAdditionally, it was demonstrated that non-robust features, while beneficial for classification, aren't helpful for reconstruction, highlighting their task-specific utility. \n\nMoreover, models trained solely on robust features lacked robustness, showing high vulnerability to AutoAttack. This contradicts the original study's findings, as these models failed to exhibit robustness even when trained exclusively on robust data. \n\nLastly, an examination of cosine similarity between the adversarial attack directions of various self-supervised models showed significant differences, suggesting adversarial attacks aren't easily transferable between different training setups. This indicates that non-robust features are generally not very useful and are model-specific rather than dataset-specific.", "questions": "Could the author shows whether the current model in Table 3 is vulnerable to both PGD and CW attack? Also, could the authors ablate the experimental differences between the original paper and the current on?", "limitations": "The authors have addressed the limitations adequately", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper principally examined the arguments made by [1], by training models in a self supervised manner on non-robust and robust datasets in [1].\n- I think the finding the models trained on robust features alone are not robust is already a very important finding. As [1] is a paper that the community has been very interested in, it is important that other papers try to reproduce and validate the results.\n\n\n[1] Ilyas, Andrew, et al. \"Adversarial examples are not bugs, they are features.\" Advances in neural information processing systems 32 (2019).\n", "weaknesses": "- I think the authors should really expand on 6.2 and table 3. The results is truly surprising and strongly contradict the original finding in [1]. For example, the authors suggest that the use of AutoAttack is the main cause for the discrepancies with the finding of the original paper. The author should verify whether the current model is vulnerable under PGD and CW attack. If the current model is also vulnerable under PGD and CW attack with more steps, which I think the current model is going to be. Then what are the conditions for reproducing the original finding, and how does the author's setting differ from it? \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### Title: Response to Reviewer WcXm\n\nDear Reviewer WcXm,\n\nWe appreciate for your careful reading of our work as well as recognizing our finding that standard training on robust dataset does not yield true robustness. Below, we address your main concerns.\n\n---\n**Q1.** The author should verify whether the current model is vulnerable under PGD and CW attacks. If the current model is also vulnerable under PGD and CW attack with more steps, which I think the current model is going to be. Then what are the conditions for reproducing the original finding, and how does the author's setting differ from it?\n\n**A1.** In this experiment, we follow the original setting of [1], and the only modification is to change the attacker from PGD/CW to AutoAttack. As it's shown in the table below, under the same setting, we can reproduce [1]'s results and find that the model is indeed **robust under PGD/CW attack**, even after 1000 iterations. However, under AutoAttack, that model has only 0.21% robust accuracy. Therefore, [1]'s evaluation actually gives a false sense of robustness, and their arguments on robust datasets are questionable since the resulting model is essentially non-robust. \n\nWe will add this comparison to Sec 6.2 and Table 3 in revision.\n\n*Robustness of the PreActResNet-18 model obtained from standard training on robust dataset [1] under different attacks on CIFAR-10.*\n| Attack Method | PGD-500 | CW-500 | PGD-1000 | CW-1000 | AutoAttack |\n| ----   | :----: | :----: | :----: | :----: | :----: |\n| **Robust Accuracy** | 32.86% | 32.44% | 32.59% | 32.44% | **0.21%** | \n\nReferences:\n\n[1] Adversarial Examples Are Not Bugs, They Are Features; Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, Aleksander Madry;  Part of Advances in Neural Information Processing Systems 32 (NeurIPS 2019)\n\n---\n\nPlease let us know if there is more to clarify. We are happy to take your further question in the discussion stage."}, {"review_id": "7cxKXdSQA9", "reviewer": "Reviewer_GnEu", "summary": "This work challenges the theory on \"robust\" and \"non-robust\" features from Ilyas et al. With an extended and more generalized formulation, the authors show that \"non-robust\" features are indeed very task-specific, and that even supposedly \"robust\" features are mostly task-specific and hardly provide robustness. The authors experiment with various self-supervised models on CIFAR10 (and its robust/non-robust variants) and answer multiple interesting research questions.", "questions": "In Eq(11) the same $\\epsilon_0$ seems to be added to added for all tasks? This does not seem optimal at all, since different tasks may require different levels of noise. For instance, a binary classifier would require much more noise (happy/sad classifier) as opposed to a more fine-grained task (person identification).", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The original work from Ilyas et al. did mention that the \"robust\" features are task specific, but this work explicitly tests and shows that \"robust\" features are not necessarily truly robust, and and only really \"task-robust\". In some sense, there are 2 levels of generalization- this work shows that robustness at the second stage (same X, Y distribution) does not imply robustness in the first stage (same X, different Y distribution).\n\n- I like Eq (9) - providing a formulation that is relative to the choice of $g$.\n\n- The conclusion in L199-200 is fitting, and ideally should have been a stress point in the original paper on robust features. The authors here have done a good job of making this point explicit to clear up the misconception around \"robust\" features. The fact that the original paper did claim robust features to be a property of the dataset is in direct contradiction to results from this work.", "weaknesses": "- My biggest concern is the interpretation of \"robust\" and \"non-robust\" features. The authors challenge the claim made by Ilyas et al. about the actual robustness of features. However, the cited paper does not claim that the \"robust\" features they identify are universally robust, only that they are robust for the classification task at hand. This is not surprising- 'eye color' would be a robust feature for person identification, but not for smile detection. These features are statistical patterns that ere useful for the **given** task but not of any use otherwise, so not surprising that they do not generalize to multiple tasks. Even on L112, the authors claim \"we believe that the existence of non-robust features is task-reliant\", which is what Ilyas et al. also say.\n\n- L144: \"...meaning that the adversarial perturbations are almost meaningless to DDPM\". This can also mean that the attack is not potent enough. Also, there is no reason to believe that adversarial perturbations should transfer across tasks. The objective when adding perturbations for one task is unrelated to the wanted objective for another task- perturbations meant to fool smiling/non-smiling have no reason to influence classification scores for straight/curly hair, for instance. The former would most likely perturb areas around the face, while the latter would likely look at features close to the head.\n\n- All experiments are focused on CIFAR-10 and its robust/non-robust variants. I would like to see at least one more dataset to be more confident in the generalization of claims made in the paper.\n\n## Minor comments\n\n- This paper assumes that the readers are familiar with the work on \"robust features\" referenced throughout the paper. Please give a brief summary of the referenced paper (from MadryLab) - their crux and notion of \"robust\"/\"non-robust\" features in the Introduction itself.\n\n- L19: \"...and it becomes natural for papers to use terms like...\" - please give some examples of papers that indeed do this.\n\n- L35: \"experimente\" -> \"experiment\"\n\n- L41: Please provide a list of contributions. Although posing these research questions is a good way to pique interest, they should not not be left as unanswered questions until the end of the paper.\n\n- Eq (12) seems to be missing the term $T$\n\n- Figure 2 missing axis labels\n\n- L164: \"...exceeds 80%\" - what is the ASR of a robust model for the same attack?\n\n- L169-172: \"First, the attack method.......from gradient obfuscation\" - this might mean that the \"robust\" features are not entirely robust and have some noisy non-robust features in them too.\n\n- Figure 4: please provide a heatmap legend. This is an interesting figure- why is cosine similarity of perturbations a good metric? Why not look at transferability rates instead. The latter would be a better and more direct indicator.\n\n- References formatting is mixed: conference names are mixed in lower-case, capitalized, etc. For instance, \"Are adversarial examples inevitable\" appears in ICLR but the arxiv version is cited. Some conference names are in full while for others abbreviations are used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### Title: Response to Reviewer GnEu\n\nDear Reviewer GnEu,\n\nAppreciate for your careful reading and acknowledging our contributions on the definitions and evaluation of robust features. Below, we address your main concerns, especially those concerning the task-reliance of robust features.\n\n--- \n**Q1.** My biggest concern is the interpretation of \"robust\" and \"non-robust\" features [...]\n\n**A1.** Thanks for your insightful comments, and we get your point that a feature could be useful for one task while useless for the other. However, this does NOT contradict with our theory. In fact, we did not expect a feature that is generalized across every possible task (e.g., eye color and smile); instead, we define robust features **for a given set of tasks $\\mathcal{T}$** (see L73). The word \"universally robust\" could be a little stretching, but essentially, we define it as \"robust for every task in $\\mathcal{T}$\" (see L82).\n\nFor a valid discussion, in this work we consider the case when $\\mathcal{T}$ contains **a set of relevant tasks where naturally trained features transfer well across different tasks**. Specifically, we consider the transfer from SSL pretraining tasks (CL, MAE, diffusion) to supervised tasks (classification), and it is a known fact that features learned by SSL are very useful for classification tasks (with high linear probing accuracy). Due to this high relevance, one would expect that like natural features, the transferability should happen for non-robust features if they are truly useful for classification (as Ilyas et al suggested). Instead, our experiments give quite surprising results that their transferability is much worse than that of natural / robust features. This supports our claim that non-robust features may not be truly useful features but mainly task-specific spurious features.\n\nWe will state this relationship and make the dependence on the task set $\\mathcal{T}$ clearer to avoid potential confusions. Please let us know if there is more to clarify.\n\n---\n**Q2.** L144: \"...meaning that the adversarial perturbations are almost meaningless to DDPM\". \n\n**A2.** We address your concerns on this statement point by point. \n\n> This can also mean that the attack is not potent enough. \n\nWe note that these non-robust features generated by PGD-1000 are very useful in the classification task, as they can 1) mislead prediction, 2) provide a good classifier when trained on non-robust dataset (Ilyas et al). The fact that it no longer works on DDPM reveals that there exits a clear discrepancy on feature usefulness between the two tasks, and thus justfies our argument here.\n\n>  Also, there is no reason to believe that adversarial perturbations should transfer across tasks. \n\nFollowing the discussion in A1, diffusion models can also be seen as an SSL pretrain method, and we obtain >80% transferred linear classification accuracy with diffusion-learned features (Fig 1). In view of the good transferability of natural features, the intransferability of non-robust features between two tasks reveals that non-robust features are very different from natural features.\n\n---\n**Q3.** All experiments are focused on CIFAR-10 and its robust/non-robust variants. I would like to see at least one more dataset to be more confident in the generalization of claims made in the paper.\n\n**A3.** Within the limit of time, we reproduce some main results on a larger dataset, TinyImageNet [1], which contains 100,000 64x64 images diveded into 200 classes. \n\nAs shown in Table A in our rebuttal PDF, non-robust features still obtains much lower accuracy than natural and robust features, which aligns well with our experiments on CIFAR-10.\n\nWe also evaluate the robustness of ResNet-50, DenseNet-121 with standard training on the constructed robust dataset on TinyImageNet. Simialr to CIFAR-10 results, in Table B (Rebuttal PDF), the trained models are also non-robust under AutoAttack.\n \n[1] Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge.  Stanford University. 2015\n\n---\n\n***Remark.* Due to the limit of characters, we address some of your key concerns in minor points below and will fix the writing problems in revision following your suggestions.\n\n**Q4.** ASR of a robust model under AutoAttack.\n\n**A4.** According to RobustBench [1], the ASR of the SOTA robust model of CIFAR10 dataset is **29.31%**, and for a medium-level adversarially trained model ASR is about **50%**. \n\n[1] Croce et al. RobustBench: a standardized adversarial robustness benchmark. NeurIPS'21.\n\n---\n**Q5.** Figure 4: please provide a heatmap legend. This is an interesting figure-Why is cosine similarity used rather than transferability of adversarial perturbations?\n\n**A5.** We will add the legend. Consine similarity measures the relationship between adversarial perturbations in the input space. As you suggested, transferability is also a good metric for similarity and the results can be seen in Figure A in the Rebuttal PDF, which mostly consistent with previous results: the transferability between SL models is good, but poor between SSL and SL models.\n\n---\n**Q6.** In Eq(11) the same $\\epsilon_0$ added for all tasks? This might not be optimal, since different tasks may require different levels of noise.\n\n**A6.** No, $\\varepsilon_0$ here denotes a sample-wise and task-wise perturabation independently generated for each sample pair $(x,y)$ at each task $T$ *per se*. Rigorously, it should be $\\varepsilon_{x,\\mathcal{T}}$. We will clarify the notation in revision.\n\n---\n\nPlease let us know if you have further concerns/questions."}, {"review_id": "i30zV2B2sM", "reviewer": "Reviewer_nwrB", "summary": "This work analyzes the robustness of deep networks under various tasks, analyzes how adversarial attacks between models trained on different tasks transfers between them, and does so through a newly proposed framework.", "questions": "See Weaknesses for items to address.", "limitations": "See above. No ethical statement, a little could be said about dataset robustness and safety.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "This paper delves into the robustness of models trained under different tasks (SSL training) and runs thorough analysis on CIFAR10 datasets to compare these models. \n\nAs SSL trained models continue to be used, this paper's research direction is interesting and useful.\n\nThe newly proposed framework seems reasonable to compare robustness.\n\n", "weaknesses": "While I like the general idea of this paper, the main thrust of it was quite difficult to follow. While I am familiar with the work \"Adversarial Examples Are Not Bugs, They Are Features\" (or [12]), it wasn't until nearly finishing my first read-through that the submitted manuscript is heavily based on it's work. It also wasn't clear that \"tasks\" in this paper do not represent different datasets (with different objectives) similar to multi-task learning, but \"tasks\" refers to self-supervised learning. These previous points could be made much clearer. Furthermore, it appears that the paper may have been written under time constraints, as there are multiple instances of typographical errors throughout the text (\"datatset\" and \"hekp\" for example). Taking the time to carefully proofread the content would greatly enhance its clarity and overall presentation (e.g., Fig 3 referenced on page 4, but shown on page 6). Lastly, many technical details are left out of the paper which would seriously help understanding: a brief overview of each of the SSL methods, how are the datasets constructed, and where linear probing is performed.\n\nThe proposed \"theory\" framework appears to be heavily based off [12], and it is more of a list of definitions rather than offering any theoretical insights. While these definitions are useful, theoretical insight could be given by analyzing how different (SSL / non SSL) tasks could effect the robustness more comprehensively.\n\nSome of the large claims in this paper, that the robust datasets are in fact not robust, are not quite backed up by the experiments of the paper. The robust dataset was generated by using a classifier, to then train a classifier. I'm not sure why it's surprising that using that robust classifier specific dataset doesn't give robustness to SSL trained models. Here's a natural experiment:\n1. Train an SSL model on the clean dataset\n2. Fit a linear probe for classification\n3. Use that linear probe to generate the robust dataset following [12]\n4. Train a new SSL model on that new robust dataset\n5. Attack that model\nI would be very curious to see what Table 3 would look like after following the above steps.\n\n\nFinally, results on using CIFAR-10 have questionable generalization. It is a little surprising that the \"Restricted ImageNet\" from [12] was not used for experiments.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### Title: Response to reviewer nwrB\n\nWe apperciate your careful reading and constructive comments on the presentation details. We will revise the paper carefully and take into consideration your suggestions in the revision.  \nBelow, we address your concerns on the paper content.\n\n---\n\n**Q1.** While these definitions are useful, theoretical insight could be given by analyzing how different (SSL / non SSL) tasks could effect the robustness more comprehensively.\n\n**A1.** Thanks for suggestions. We can give an intuitive theoretical example as in Ilyas et al to illustrate the differences between SSL features and SL features. \n\nConsider a simple case where $X = USV\\in\\mathbb{R}^{N\\times M}$ is the SVD of input features of $N$ samples with $M$-dimesional features. If there is a spurious correlation between the eigenvectors with smallest singular values and the class labels, they are useful features for classification. However, these features will be discarded in PCA that only extract top eigenvectors. This shows that the features learned by an SSL task and an SL task can be quite different. And some features may only be useful for one task only.\n\nThings are much more complex on real-world data and NN-based SSL methods. As shown in our experiments, although natural features transfer well between SSL and classification, non-robust features are mostly ineffective under various SSL training, showing strong task dependence.\n\n\n---\n**Q2.** Some of the large claims in this paper, that the robust datasets are in fact not robust, are not quite backed up by the experiments of the paper. The robust dataset was generated by using a classifier, to then train a classifier.\n\n**A2.** We highlight that we have evaluated the robust-dataset classifier on both classification (SL->SL) and SSL (SL->SSL) tasks. The former experiment follows **the same evaluation protocol as Ilyas et al [1]'s experiments to validate the robust dataset (Sec 3.1 in their paper)**, that is to perform standard training on the robust dataset and evaluate model robustness. The *only* change we did is to replace their PGD/CW attack with AutoAttack. From Table 3 quoted below, we can find the model robustness vanishes under AutoAttack. It clearly shows that robust datasets are not really robust.\n\n\n| Attack Method | PGD-500 | CW-500 | PGD-1000 | CW-1000 | AutoAttack |\n| ----   | :----: | :----: | :----: | :----: | :----: |\n| **Robust Accuracy** | 32.86% | 32.44% | 32.59% | 32.44% | **0.21%** | \n\n> Why it's surprising that using that robust classifier specific dataset doesn't give robustness to SSL trained models? Here's a natural experiment [...]\n\nThis experiment is to further study the \"universal robustness\" (defined in Sec 3.1) of the robust dataset, i.e., whether its robustness can generalize across different tasks. We find that similarly, SSL models using robust datasets also exhibit no robustness. In all, robust datasets show no robustness on both classification and SSL tasks.\n\nWe also collect new results following the experiment setup that you suggest, using robust dataset generated by SSL (SSL -> SSL). The results are also consistent with the SL->SL and SL->SSL results, and still no nontrivial robustness is observed.\n\n| Model  |  Clean Accuracy | Robust Accuracy |\n| ----   | :----: | :----: |\n| SimCLR | 60.52% | 0.02% |\n| MAE | 28.67% | 0.24% |\n| ResNet-50 | 71.04% | 0.06%  |\n| DenseNet-121 | 72.34% | 0.10% |\n\n[1] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, Aleksander Madry; Adversarial Examples Are Not Bugs, They Are Features; Part of Advances in Neural Information Processing Systems 32 (NeurIPS 2019)\n\n---\n**Q3.** Finally, results on using CIFAR-10 have questionable generalization. It is a little surprising that the \"Restricted ImageNet\" from [12] was not used for experiments.\n\n**A3.** Restricted ImageNet is about 1/10 size of ImageNet and extracting robust/non-robust features requires massive computation. Due to limit time of rebuttal, we instead add experiment results on TinyImageNet-200, which contains 100,000 64x64 images diveded into 200 classes.\n\nAs shown in Table A in our rebuttal PDF, non-robust features still obtains much lower accuracy than natural and robust features, which aligns well with our experiments on CIFAR-10.\n\nWe also evaluate the robustness of ResNet-50, DenseNet-121 with standard training on the constructed robust dataset on TinyImageNet. Simialr to CIFAR-10 results, in Table B (Rebuttal PDF), the trained models are also non-robust under AutoAttack.\n \n[1] Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. Stanford University. 2015\n\n[2] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, Aleksander Madry; Adversarial Examples Are Not Bugs, They Are Features; Part of Advances in Neural Information Processing Systems 32 (NeurIPS 2019)\n\n--- \n\nPlease let us know if you have further concerns/questions."}, {"review_id": "oqnB2AGCZJ", "reviewer": "Reviewer_NVbH", "summary": "This paper builds upon the work of [12] \"Adversarial examples are not bugs, they are features\" written by Ilyas et al. The authors of [12] introduced the concept of robust and non-robust features. According to [12], non-robust features alone can be useful for the classification task. The authors of the current paper show that such features however are not extendable to different tasks. They show this through linear probing and auto encoders. Furthermore, according to the authors of [12], robust features alone are able to create a robust dataset. However, the findings of this paper contradicts this notion.\n\n", "questions": "1. In Figure 2, the clean dataset always performs better than the rest. Why is this the case always? \n2. What are the authors intuition behind non-negligible features having non-negligible performance in Figure 2? Does this mean that some of this features can be transferred across some specific tasks?", "limitations": "1. As mentioned above the presentation of this paper can be improved. In my opinion doing so would elevate the paper further.\n2. The authors ask four questions in the manuscript and answer them through experiments. It would be useful to mention these questions in the intro and comment on the usefulness of each of them.  ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper extends the definition of what means for a feature to be robust vs non-robust. Instead of relying on a single classification task, they extend the definition to a task wise perspective. This allows them to define \"Absolute Useful\" and \"Relative Useful\" features. Intuitively relative usefulness represents the importance of a feature more than absolute usefulness. Using these definitions, the authors then define $\\rho$-useful and $\\gamma$-useful features.\n2. The authors then ask a series of compelling questions. They try to find the answer to these questions empirically. In the first experiment they find that using the task definition, non-robust features perform poorly (even though not non-negligible). This indicates that non-robust features are actually not useful. Next they find that the essence of non-robust features are task-specific and they should not be considered as real features. The next question is the most interesting one. The authors find that robust features alone are not enough to for robust training. This directly contradicts with the finding in [12]. Finally, they show that different tasks capture different non-robust features. These findings together shed more light on the nature of adversarial examples and the features related to them.", "weaknesses": "1. Even though the paper is pretty well written, there are some weakness in the presentation. These include several typos e.g., in (13) and (14) I believe the function $U(g, D, T)$ is mentioned incorrectly instead of $R(g, D, T)$. Similarly, in line 104 Figure 3 is mentioned, but I think that the authors meant Figure 2. Adding to these, I could not find a mention of Figure 1 in the text. This means that the evaluation framework is not properly explained. Some similar other details are also missing reducing the readability of the manuscript. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### Title: Response to reviewer NVbH\n\nDear Reviewer NVbH,\n\nThanks for your careful reading and we are glad to hear that you appreciate the novelty and insights of this work. Below, we address your main concerns.\n\n--- \n\n**Q1.** Some suggestions regarding the presentation.\n\n**A1.** Thanks for your careful reading. We will fix these issues in the revision.\n\n---\n**Q2.** I could not find a mention of Figure 1 in the text. This means that the evaluation framework is not properly explained.\n\n**A2.** Sorry for the confusion. We will add more descriptions of Figure 1 to explain the evaluation famework. Below is an outline:\n- First, we train a feature extractors $f_{T}$ with different kinds of tasks $T$ (SSL and SL ones) on a specific dataset;\n- Second, we perform linear probing by training a linear classification head on top of frozen features using labeled data;\n- Third, we evaluate the accuracy and robustness of the composed classifier (feature extractor + linear head) on test data.\n\n---\n**Q3.** In Figure 2, the clean dataset always performs better than the rest. Why is this the case always?\n\n**A2.** We note that this phenomeon is similar in Ilyas et al's experiments (except that non-robust features perform much worse in ours). There could be two main reasons. **First**, the clean dataset contains both robust and non-robust features, which is supposed to provide more information for classification. **Second**, different from clean features using the original images, the (non-)robust dataset is constrcuted through iterative optimization from random inputs using an imperfect NN classifer. This process will inevitably lose information and introduce some noises to the constructed inputs, leading to performance degradation to some extent.\n\n--- \n**Q4.** What are the authors intuition behind non-negligible features having non-negligible performance in Figure 2? Does this mean that some of this features can be transferred across some specific tasks?\n\n**A3.** Indeed, it suggests that the human perceptible (non-negligible) features have good transferability across tasks, while human inperceptible features do not. Previously, Ilyas et al argue that despite the fact robust features are perceptible and non-robust features are not, non-robust features are still useful for classification. On the contrary, we show that their counter-intuitive understanding is questionable to some extent, as these non-robust features are largely useless on SSL tasks where natural and robust features work well. It indicates a fundamental difference between robust and non-robust features and only those human perceptible (robust) features are truly useful features, which justifies the rationality of human perception instead. \n\n---\nPlease let us know if you have further concerns/questions."}]

Author rebuttal:
The supplementary material for rebuttal.

Meta-review: This paper studies the question of whether or not adversarial examples really are features or bugs.  The reviewers universally liked the paper and after an engaging discussion with the authors believed it was worth accepting. I would encourage the authors to address the potential improvements raised during the discussion period to further improve the paper.
Decision: Accept (poster)