SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ang Li* Yifei Wang²* Yiwen Guo³ Yisen Wang⁴,⁵†
<sup>1</sup> School of Electronics Engineering and Computer Science, Peking University
$^{2}$  School of Mathematical Sciences, Peking University
$^{3}$  Independent Researcher
$^{4}$  National Key Lab of General Artificial Intelligence,
School of Intelligence Science and Technology, Peking University
$^{5}$  Institute for Artificial Intelligence, Peking University
The existence of adversarial examples has been a mystery for years and attracted much interest. A well-known theory by Ilyas et al. [19] explains adversarial vulnerability from a data perspective by showing that one can extract non-robust features from adversarial examples and these features alone are useful for classification. However, the explanation remains quite counter-intuitive since non-robust features are mostly noise features to humans. In this paper, we re-examine the theory from a larger context by incorporating multiple learning paradigms. Notably, we find that contrary to their good usefulness under supervised learning, non-robust features attain poor usefulness when transferred to other self-supervised learning paradigms, such as contrastive learning, masked image modeling, and diffusion models. It reveals that non-robust features are not really as useful as robust or natural features that enjoy good transferability between these paradigms. Meanwhile, for robustness, we also show that naturally trained encoders from robust features are largely non-robust under AutoAttack. Our cross-paradigm examination suggests that the non-robust features are not really useful but more like paradigm-wise shortcuts, and robust features alone might be insufficient to attain reliable model robustness. Code is available at https://github.com/PKU-ML/AdvNotRealFeatures.
Alongside the human-level or even superior performance of Deep Neural Networks (DNNs) in various tasks [22, 14, 10], concerns on the existence of adversarial examples constantly rise, regarding them as main threats that fool DNN classifiers with invisible perturbations [34, 12, 24, 26]. Among many explanations of adversarial examples [11, 36, 31, 30, 44, 45, 48, 28, 2, 3], the robust and non-robust feature perspective developed by Ilyas et al. [19] has received wide attention. Compared to previous explanations that regard adversarial examples as “bugs” of neural networks (i.e., the model), they claim that adversarial examples stem from non-robust “features” in the inputs (i.e., the data). Specifically, they argue that each example is composed of human-perceptible robust features (invariant to attack) and human-imperceptible non-robust features (sensitive to attack) that are both useful for classification. Under adversarial attacks, non-robust features from other classes are injected into the adversarial examples and lead to misclassification. To verify this point, they show that datasets containing only non-robust features can attain good classification performance; and similarly, datasets containing robust features can attain good robustness even with natural training. Supported by these observations, the perspective has been widely accepted ever since and many subsequent works are built upon their framework [5, 20, 37].
*Equal Contribution.
†Corresponding Author: Yisen Wang (yisen.wang@puk.edu.cn).
Figure 1: Tiny-[ImageNet instances containing natural, robust, and non-robust features, respectively. The robust and non-robust instances are generated following the iterative optimization procedure in Ilyas et al. [19] from random noise. The robust features are semantically aligned to natural images, while the non-robust features are always noise-like.
Although both robust and non-robust features are shown to be useful for classification, the two still have large discrepancies, particularly in perceptibility. As shown by Ilyas et al. [19], robust features usually have rich semantic information such as distinguishable edges and meaningful combinations of colors; on the contrary, non-robust features are always noise-like and meaningless to humans, as shown in Figure 1. Therefore, it is natural for robust features to be useful, while the usefulness of noise-like non-robust features still seems suspicious and counter-intuitive. One may raise a natural question: are non-robust features real (useful) features?
In this paper, we aim to re-examine robust and non-robust features in a wider context in order to understand the true distinction between them. Particularly, we notice that a major limitation of Ilyas et al. [19] is the scoop of learning paradigms considered, as the usefulness of non-robust features is only evaluated on the supervised image classification. However, from the backdoor literature, we know that even a meaningless backdoor pattern can lead to the desired classification. A feature being useful for classification does not necessarily imply that it is truly useful. Therefore, it is reasonable to assume that truly useful features, e.g., those utilized by humans, should work well for a wide range of learning paradigms instead of a single paradigm like the backdoor pattern.
Driven by the analysis, we take the first step by defining the cross-paradigm usefulness of robust and non-robust features, as a viable measure of true usefulness that generalizes previous definitions of Ilyas et al. [19]. A feature is defined to be truly useful only if it yields good representations across different learning paradigms. Aside from supervised learning, we further consider three modern self-supervised learning paradigms as the representatives: 1) contrastive learning that aligns augmented samples in the latent space, e.g., SimCLR [27, 6, 15]; 2) masked image modeling that predicts the masked patches from the unmasked context, e.g., MAE [9, 4, 16]; and 3) diffusion models that learn to restore images corrupted by Gaussian noise, e.g., DDPM [17, 33]. Among the four paradigms, supervised learning and contrastive learning are discriminative tasks, while masked image modeling and diffusion models are generative tasks. Previous studies show that all these different paradigms yield representations with good downstream performance when pretraining on natural images [6, 16, 50]. Therefore, we would intuitively expect that if non-robust features are as useful as natural image features, we can also learn good representations from non-robust features alone. Similarly, we can define cross-paradigm robustness that evaluates whether a feature can yield robust representations (with robust prediction on downstream tasks) across different learning paradigms.
To validate their true usefulness, we follow the same procedure of Ilyas et al. [19] and construct robust (non-robust) datasets that only have robust (non-robust) features extracted from supervised models. To evaluate their cross-paradigm usefulness and robustness, we learn features with the four different learning paradigms on the constructed datasets and evaluate the learned features with a linear probing head for classification (a common evaluation protocol in self-supervised learning). Surprisingly, we find that there exist notable discrepancies in the cross-paradigm usefulness: on the three (transferred) self-supervised paradigms, robust features yield excellent performance that nearly matches natural features, while non-robust features yield much worse prediction that is hardly usable, in sharp contrast to its own good performance on supervised learning. It clearly conveys that non-robust features
are not as (cross-paradigm) useful as robust/natural features. Therefore, unlike robust or natural images, non-robust features do not contain much meaningful information that is truly useful across different paradigms. Intuitively speaking, non-robust features are more like a certain backdoor pattern of natural images subject to the chosen learning paradigm (e.g., supervised classification). As paradigm-wise shortcuts, these features are essentially uninformative when examined with other learning paradigms. We further verify this point by showing that adversarial examples crafted under different paradigms can hardly transfer among each other. To this end, we conclude that robust features are truly useful features while non-robust features are not (in the cross-paradigm sense). Unlike the counter-intuitive explanation of Ilyas et al. [19], our result justifies the human intuition that the noise-like non-robust features do not really capture the essence of the images.
Furthermore, we also re-evaluate the robustness of robust and non-robust features in this way. Interestingly, contrary to the findings in Ilyas et al. [19] that natural training on the robust dataset produces a robust classifier, we find these so-called robust features hardly show robustness when learned with other learning paradigms. We also observe that the supervised learned classifier is non-robust under more reliable attacks like AutoAttack [7]. Thus, we arrive at the conclusion that on real-world datasets, both robust and non-robust features extracted by Ilyas et al. [19] are actually non-robust, in both in-paradigm and cross-paradigm senses. Although robust features are shown to exist in toy models when explicitly designed [39], there is by far no evidence that robust features exist in real-world datasets, at least not extractable.
To summarize, the main contributions of this work are three folds:
- In order to evaluate robust and non-robust features in a broader context, we generalize the notions on the usefulness and robustness of robust and non-robust features to a cross-paradigm sense. This perspective enables us to get rid of potential paradigm-wise shortcut effects and evaluate true feature informativeness.
- For usefulness, we find that robust features are both in-paradigm and cross-paradigm useful like natural features. Instead, non-robust features are only useful in-paradigm, and their usefulness dramatically degrades when transferred to other paradigms, suggesting that they are more like paradigm-wise shortcuts instead of real features. We further verify this point by showing that adversarial examples also have poor transferability across different paradigms.
- For robustness, we find that robustness obtained from the constructed robust dataset is actually a false sense of robustness when evaluated cross-paradigmly on more reliable attacks. The loss of this key evidence would raise the question of whether robust features really exist in real-world datasets.
Last but not least, although the main messages of this paper seem rather negative, this view also points out potential avenues to better adversarial robustness. In particular, the paradigm-wise behaviors of non-robust features suggest that it may be inadequate to perform adversarial training on a single learning paradigm, and a mixture of adversarial training on multiple paradigms may come to the aid. In the meantime, the non-robustness of robust dataset indicates a more comprehensive view of adversarial vulnerability: input data (or features) are not the only source of adversarial vulnerability, and only combating the vulnerabilities in both data and models can lead to true robustness.
2 A Cross-Paradigm View of Robust and Non-robust Features
2.1 Background
Robust and Non-Robust Features. As in Ilyas et al. [19], features are defined as a function mapping from input space to real numbers, namely  $f: \mathcal{X} \to \mathcal{R}$ . The robust and non-robust features are then described with the following definitions for a binary classification task:
-  $\rho$ -useful features: For a dataset  $\mathcal{D}$ , a feature  $f$  is  $\rho$ -useful if the feature is correlated with the label:
$$
\mathbb {E} _ {(x, y) \sim \mathcal {D}} [ y \cdot f (x) ] \geq \rho . \tag {1}
$$
-  $\gamma$ -robustly useful features: For a  $\rho$ -useful feature, it is regarded as  $\gamma$ -robustly useful if it remains useful under certain range of perturbation:
$$
\mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \inf  _ {\delta \in \Delta} y \cdot f (x + \delta) \right] \geq \gamma . \tag {2}
$$
- Useful, non-robust features: These features are the ones that are  $\rho$ -useful features, but are not  $\gamma$ -robust features for any  $\gamma \geq 0$ .
Construction of Robust and Non-Robust Datasets. Given a classifier  $C$  and a dataset  $\mathcal{D}$ , Ilyas et al. [19] propose to construct a dataset  $\hat{\mathcal{D}}$  which satisfies:
$$
\mathbb {E} _ {(x, y) \sim \hat {\mathcal {D}}} [ f (x) \cdot y ] = \left\{ \begin{array}{l l} \mathbb {E} _ {(x, y) \sim \mathcal {D}} [ f (x) \cdot y ] & \text {i f} f \in F _ {C} \\ 0 & \text {o t h e r w i s e ,} \end{array} \right. \tag {3}
$$
where  $F_{C}$  represents the set of features utilized by  $C$ . Denoting  $g$  as the mapping from input  $x$  to the representation layer in  $C$ , the new instance  $x_{r}$  is obtained from  $x$  through following optimization:
$$
\min  _ {x _ {r}} \left\| g (x) - g \left(x _ {r}\right) \right\| _ {2}. \tag {4}
$$
If the classifier  $C$  (e.g., ResNet-50) is adversarially trained, the constructed dataset  $\hat{\mathcal{D}}$  is regarded as a robust dataset. As for a standardly trained classifier, the dataset  $\hat{\mathcal{D}}$  is regarded as a non-robust dataset.
Basic Conclusions of Ilyas et al. [19]. Utilizing the constructed datasets, Ilyas et al. [19] empirically show that models trained on the non-robust dataset are useful, i.e., they can attain good classification performance on natural test data, comparable to models trained on the natural dataset. This supports their claim that non-robust features are sufficiently useful to obtain good generalization. This perspective well explains the existence of adversarial examples as well as their transferability, since adversarial examples contain non-robust features from the misclassified classes that are useful for different models. Therefore, as they put it, adversarial examples are features, not bugs. Also, they also show that natural training on the robust dataset (containing robust features alone) can yield robust models. In this way, they view adversarial vulnerability purely from a data (or feature) perspective, regarding them as a result of different priors on extracted features.
Self-Supervised Learning. Aside from the supervised learning paradigm studied in Ilyas et al. [19], self-supervised learning (SSL) also received wide attention in recent years. Without manual labels, SSL methods utilize self-supervision to learn meaningful features from unlabeled data and have achieved impressive progress in recent years [6, 16, 17]. Generally speaking, an SSL algorithm pretrains a feature encoder  $f: \mathcal{X} \to \mathcal{Z}$ , mapping from the input space to the latent space  $\mathcal{Z}$ . Afterward, the learned features are typically evaluated on the so-called linear probing task. Specifically, given a labeled dataset  $(x, y) \sim \mathcal{D}$  (usually a subset of the pretraining data), we train a linear classifier  $p: \mathcal{Z} \to \mathcal{Y}$  on top of learned features and use its classification accuracy (with the composed classifier  $p \cdot f$ ) on the test data as a measure of the usefulness of learned features (representations).
2.2 Cross-Paradigm Notions of Robust and Non-Robust Features
In this part, we generalize the definitions of robust and non-robust features to a wider context beyond the supervised classification. For a rigorous discussion, we introduce some general definitions of the paradigm-wise feature usefulness and robustness and then define true usefulness and robustness in the cross-paradigm sense.
**Paradigm-Wise Definitions.** To facilitate features to generalize across different paradigms, we differ from Ilyas et al. [19] and adopt a more classic definition of features, filters in the input space, i.e.,  $g: \mathcal{X} \to \mathcal{X}$ . Note that common image features like textures, edges, and colors naturally fall into this category. We define a learning paradigm  $T$  as a specific learning algorithm that learns a feature encoder  $f$  with a loss function  $L_{T}$  over a given dataset  $\mathcal{D}$ :
$$
f _ {T} = \underset {f} {\arg \min } \mathbb {E} _ {x, y \in \mathcal {D}} [ L _ {T} (f (x), y) ]. \tag {5}
$$
Incorporating a feature  $g$ , the minimization problem can be further specified as
$$
f _ {T} ^ {(g)} = \underset {f} {\arg \min } \mathbb {E} _ {x, y \in \mathcal {D}} [ L _ {T} (f (g (x)), y) ]. \tag {6}
$$
With each learning paradigm  $T$ , we can train an encoder  $f_{T}^{(g)}$  for feature  $g$ . Then we evaluate the learned representations with a linear probing head  $p$  on top of  $f_{T}^{(g)}$  on  $\mathcal{D}$ . The usefulness of a feature  $g$  relevant to the paradigm  $T$  is defined as follows:
$$
U (g, \mathcal {D}, T) = \max  _ {p \in \mathcal {P}} \mathbb {E} _ {x, y \in \mathcal {D}} \mathbf {1} [ p \left(f _ {T} ^ {(g)} (x)\right) = y ]. \tag {7}
$$
That is, a feature  $g$  is useful on paradigm  $T$  if the linear probing head shows good classification performance with representations learned from the feature  $g$  under the learning objective defined by the paradigm  $T$ .
Accordingly, the paradigm-wise feature robustness is defined as the remaining feature usefulness under local adversarial perturbations,
$$
R (g, \mathcal {D}, T) = \max  _ {p \in \mathcal {P}} \mathbb {E} _ {x, y \in \mathcal {D}} \max  _ {\| \delta \| \leq \varepsilon} \mathbf {1} \left[ p \left(f _ {T} ^ {(g)} \left(x + \delta_ {x}\right)\right) = y \right]. \tag {8}
$$
In other words, a feature is robust on paradigm  $T$  if a standardly trained linear probing head shows good robustness with representations learned from the feature  $g$  under  $T$ .
From this perspective, the conventional definitions of feature usefulness and robustness of Ilyas et al. [19] correspond to a special case of our definitions - when choosing the supervised learning task as the learning paradigm. In this way, the two processes, feature learning and feature evaluation, actually share the same learning objective, which may introduce spurious paradigm-wise shortcut effects. To get rid of this potential risk, we propose to re-define these concepts in a cross-paradigm sense.
Cross-Paradigm Definitions. Given a diverse set of learning paradigms  $\mathcal{T} = \{T_1, T_2, \dots, T_3\}$ , we define a feature to be truly useful if it can generalize across multiple learning paradigms. Specifically, we define cross-paradigm usefulness as the worst paradigm-wise usefulness among these paradigms,
$$
\left(\text {C r o s s - P a r a d i g m U s e f u l n e s s}\right) C U _ {\mathcal {T}} (g, \mathcal {D}) = \min  _ {T \in \mathcal {T}} U (g, \mathcal {D}, T). \tag {9}
$$
Similarly, we can define cross-paradigm robustness as the worst robustness
$$
\left(\text {C r o s s - P a r a d i g m U s e f u l n e s s}\right) C R _ {\mathcal {T}} (g, \mathcal {D}) = \min  _ {T \in \mathcal {T}} R (g, \mathcal {D}, T). \tag {10}
$$
Nevertheless, since different paradigms usually attain performance on different levels, a direct comparison of their absolute performance may be unfair. In the worst case, if a certain paradigm has poor performance even when pretrained on the raw images, it would dominate other paradigms when computing cross-paradigm metrics, which, however, does not reflect the true feature usefulness. To mitigate this potential issue, we propose the paradigm-wise relative metrics as the ratio between the performance of the chosen feature and the performance of using all features (raw inputs),
$$
\text {(R e l a t i v e U s e f u l n e s s)} R U (g, \mathcal {D}, T) = \frac {U (g , \mathcal {D} , T)}{U (\mathcal {D} , T)}, \tag {11}
$$
$$
\text {(R e l a t i v e R o b u s t n e s s)} R R (g, \mathcal {D}, T) = \frac {R (g , \mathcal {D} , T)}{R (\mathcal {D} , T)}, \tag {12}
$$
where given an encoder  $f_{T}$  learned from the raw images under the paradigm  $T$ , we define
$$
U(\mathcal{D},T):= \max_{p\in \mathcal{P}}\mathbb{E}_{x,y\in \mathcal{D}}\mathbf{1}[p(f_{T}(x)) = y],R(\mathcal{D},T) = \max_{p\in \mathcal{P}}\mathbb{E}_{x,y\in \mathcal{D}}\max_{\| \delta \| \leq \varepsilon}\mathbf{1}[p(f_{T}(x + \delta_{x})) = y].
$$
Accordingly, we can define the cross-paradigm relative usefulness and robustness as follows:
$$
\left(\text {C r o s s - P a r a d i g m R e l a t i v e U s e f u l n e s s}\right) C R U _ {\mathcal {T}} (g, \mathcal {D}) = \min  _ {T \in \mathcal {T}} R U (g, \mathcal {D}, T), \tag {13}
$$
$$
\text {(C r o s s - P a r a d i g m R e l a t i v e R o b u s t n e s s)} C R R _ {\mathcal {T}} (g, \mathcal {D}) = \min  _ {T \in \mathcal {T}} R R (g, \mathcal {D}, T). \tag {14}
$$
3 Cross-Paradigm Usefulness of Robust and Non-robust Features
Built upon the evaluation framework established in Section 2.2, we first investigate the cross-paradigm usefulness of robust and non-robust features on real-world datasets in this section. With this generalized notion of feature usefulness, we are trying to answer the key question: are robust and non-robust features truly useful?
3.1 Setup
Data Construction. We include two commonly adopted datasets in our study, CIFAR10 [21] and Tiny-ImageNet-200 [49]. Aside from the raw images, following the same construction process in
Table 1: Evaluation of relative usefulness of robust features and non-robust features on four learning paradigms: MIM, CL, DM, and SL. The cross-paradigm relative usefulness is computed as the worst relative usefulness over the four paradigms. We also include the usefulness scores (evaluated only on the supervised task) reported in Ilyas et al. [19] for a comparison.
Data	Feature	MIM	CL	DM	SL	Cross. Rel. Useful.	Ilyas et al. [19]
CIFAR-10	Robust	0.896	0.791	0.831	0.880	0.791	0.854
Non-Robust	0.307	0.433	0.512	0.914	0.307	0.823
Tiny/ImageNet	Robust	0.691	0.775	0.861	0.880	0.672	0.407
Non-Robust	0.134	0.074	0.141	0.645	0.134	0.396
Ilyas et al. [19] (details in Appendix A), we further construct a robust version and a non-robust version for each dataset, which only contain robust and non-robust features, respectively.
Learning Paradigms. Besides supervised learning with the cross-entropy loss, we also include three self-supervised learning paradigms for a cross-paradigm evaluation: SimCLR [6] for Contrastive Learning (CL), MAE [16] for Masked Image Modeling (MIM), and DDPM [17] for Diffusion Models (DM). We then train linear probing heads for the encoders on the same dataset that it was trained on. The probing heads are directly applied to the output of the encoder for SimCLR and MAE models, while for DDPM model, since the U-Net encoder has complex high-dimensional hidden features, we use the global average pooled feature in the fourth upsampling layer in U-Net following Xiang et al. [50], which is shown to deliver excellent linear classification performance on par with other self-supervised learning paradigms. Also, we train a ResNet-50 on the datasets with Supervised Learning (SL) as our baseline. To summarize, the four learning paradigms considered in this work are  $\mathcal{T} = \{\mathrm{MIM},\mathrm{CL},\mathrm{DM},\mathrm{SL}\}$ . After pretraining, we evaluate the classifiers' performance on the test sets of the two datasets. More details on training and evaluation are in Appendix A.
3.2 Non-robust Features are Not Cross-paradigmly Useful