SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We first examine the paradigm-wise usefulness of robust and non-robust features extracted from the supervised models. As shown in Table 1, models pretrained from robust datasets (containing only robust features) show comparable performance to those pretrained from the natural datasets (containing all features). In comparison, models pretrained from the non-robust datasets (containing only non-robust features) differ a lot across different paradigms: they work well for learning from the supervised task (ResNet-50), but much worse on all the other SSL paradigms (SimCLR, MAE, DDPM). It shows that in contrary to the perceptually aligned robust features that are useful for different paradigms, the noise-like non-robust features do not really contain meaningful information that is also useful for other SSL paradigms. Instead, these non-robust features extracted from supervised models only show usefulness on the supervised task. This sharp distinction suggests that non-robust features are more like paradigm-wise shortcut features rather than being truly useful.
For a comprehensive evaluation, we further compute the cross-paradigm relative usefulness following the definitions in Section 2.2. As shown in Table 1, the relative usefulness of non-robust features is significantly lower than that of robust features under various self-supervised learning paradigms. The overall cross-paradigm ends up being 0.307, which is much smaller than robust features (0.791). In comparison, these two kinds of features show similar performance (0.854 and 0.823) in Ilyas et al. [19] when only considering the supervised task. It shows that from a paradigm-wise view, the usefulness of features dramatically differs from the case where only supervised learning is considered. Again, it verifies that non-robust features are not as useful as natural and robust features when transferred to many other learning paradigms. For completeness, in Appendix C.1, we further examine the cross-paradigm transferability of the non-robust features extracted from the three SSL paradigms. Similarly, we find that these SSL non-robust features are non-transferable as well.
4 Cross-Paradigm Robustness of Robust Features
The results above challenge the claim from Ilyas et al. [19] that non-robust features are useful features by showing that non-robust features are not really useful in the cross-paradigm sense. Aside from the usefulness of non-robust features, another notable finding in Ilyas et al. [19] is that robust features
alone are enough for attaining model robustness. Specifically, they extract a robust version of a given dataset (e.g., CIFAR-10) by distilling from a robust classifier such that the transformed samples only contain robust samples, and they show that a naturally trained classifier on the constructed "robust dataset" has good robustness. In this section, we further examine the robustness of the extracted robust features on different learning paradigms.
We construct the robust versions of CIFAR10 with a supervised  $\ell_{\infty}$ -robust ResNet-18 encoder [46] and a self-supervised  $\ell_{\infty}$ -robust ResNet-18 encoder [23], largely following the convention of Ilyas et al. [19]. We then train encoders on the robust datasets with the learning paradigms set  $\mathcal{T} = \{\mathrm{MIM}, \mathrm{CL}, \mathrm{DM}, \mathrm{SL}\}$  and use the learned representations to naturally train a linear head. Finally, we evaluate the robustness of the composed classifiers (encoders + linear heads) with a modern reliable attack method, AutoAttack [7], under the perturbation budget of 4/255, instead of the PGD attack [25] that may lead to over-estimated robustness [7]. More experimental details can be found in Appendix A.
4.2 Robust Features are Not Robust both In-Paradigm and Cross-Paradigm
We summarize the results in Table 2. Viewing from the in-paradigm results in Table 2a, we can observe that in supervised learning, PGD attacks can over-estimate the robustness of models trained from robust datasets, e.g., 15.71 (PGD) v.s. 3.65 (AutoAttack). When evaluated under AutoAttack, robust datasets generated from either  $\ell_2$ -robust or  $\ell_{\infty}$ -robust models show poor robustness, being hard to match the robustness of adversarially trained source models. In other words, in practice, we believe it difficult to attain good robustness with only robust features (extracted following Ilyas et al. [19]). Their original conclusion on the good robustness of robust datasets could be due to the unreliability of PGD-based robust evaluation (also shown in the AutoAttack paper [7]). The severe over-estimation on robust datasets is potentially due to gradient obfuscation or masking [1] since these robust datasets are essentially generated with the PGD attack.
Extending the finding, we also evaluate the robustness of robust features on other self-supervised learning paradigms in Table 2b. Likewise, robust features also perform poorly and do not attain non-trivial robustness. Interestingly, when training on generative models like MIM and DM, robust features obtain slightly better robustness compared to the discriminative ones (CL and SL), even on natural data. Overall, the cross-paradigm robustness (defined in Section 2.2) is still bottlenecked by the SL itself. Based on these results, we conclude that robust features still cannot achieve model robustness alone (in either in-paradigm or cross-paradigm sense), without model-level interventions like adversarial training. It suggests that adversarial vulnerability may not come from the data alone, and a joint training strategy against both data-level and model-level vulnerabilities may be needed to attain real robustness. It worth noting that Tsilivis et al. [38] also found the non-robustness of the robust dataset of Ilyas et al. [19] in the supervised domain, while our results here give a full examination of robust features from both in-paradigm and cross-paradigm aspects.
5 Cross-Paradigm Transferability of Adversarial Attacks
The transferability across different models, even with different architectures [47, 40], is a fascinating fact of adversarial examples. The perspective of Ilyas et al. [19] provides a natural explanation for the phenomenon since non-robust features in adversarial examples are inherently useful such that they can affect the prediction of different models. Nevertheless, our studies in Section 3 reveal that non-robust features are only paradigm-wise but not cross-paradigm useful. It implies that adversarial transferability is also paradigm-wise. In other words, adversarial examples can hardly transfer between different paradigms. To verify the hypothesis above, we further investigate the cross-paradigm transferability of adversarial attacks in this section.
5.1 Cross-Paradigm Transferability of Attack Objectives
Since different paradigms mainly differ by their learning objectives, adversarial attacks w.r.t. different learning objectives (e.g., SL, CL, MIM, DM) can generate adversarial examples of different paradigms. To study the transferability between different attack objectives, we consider two ResNet-18 backbone
networks (learned with SL and CL, respectively), attack them with an objective, and observe the change of both learning objectives.
Setup. We consider two objectives: InfoNCE loss [27] in CL and Cross Entropy (CE) loss in SL. Adversarial examples are generated with  $l_{\mathrm{inf}}$  AutoAttack[7] bounded by  $\varepsilon = 4 / 255$ , using the default attack settings on CIFAR-10 (details in Appendix A). To be consistent, we compute the InfoNCE loss for a supervised encoder by appending a re-trained projection head on top of the fixed encoder. When computing the gradient during attacks, we also retain the standard data augmentations in SimCLR [6] and differentiate through these operators with the Kornia package [32]. For further ablation study on the influence of these components on transferability, see Appendix B.
(a) CE loss of  $f_{CL}$
Figure 2: The change of loss value  $\nu.s.$  the attack iteration steps when using different attack objectives, CE loss (blue lines) or InfoNCE loss (orange lines), and backbones, trained by SL (Figure 2a & Figure 2b) or CL (Figure 2c & Figure 2d) from different paradigms.
(b) InfoNCE loss of  $f_{CL}$
(c) CE loss of  $f_{SL}$
(d) InfoNCE loss of  $f_{SL}$
Results. We plot the change of loss values during the attacking process in Figure 2, from which we can identify a general trend: there is a significant difference between the changes of different objectives, and maximizing one attack objective has a limited effect on the other objective. This distinction clearly implies that adversarial examples generated with attack objectives from different paradigms have poor transferability among each other.
5.2 Cross-Paradigm Transferability of Backbone Encoders
Aside from the attack objective discussed in Section 5.1, different paradigms also learn different paradigm-wise feature encoders that also have a large influence on the generation of adversarial examples. To further analyze the paradigm-wise influence of feature encoder on adversarial transferability, we generate adversarial examples with backbone encoders obtained by the four different paradigms, while keeping the attack objective to be the same CE loss (using the linear head learned for each paradigm). Besides the default ResNet-50(SL-RN) model, we further include three different
Table 2: Absolute robustness of robust features on four paradigms: MIM, CL, DM, and SL, on CIFAR-10. The robust features are extracted by pretrained models from two different source paradigms: supervised learning and contrastive learning. Different from Ilyas et al. [19] that use 1000-step PGD [25] for robust evaluation, we adopt a more reliable attack algorithm, AutoAttack [7].
(a) Robustness evaluation (with PGD and AutoAttack) for classifiers naturally trained with robust datasets (SL).
Source Model	l2, ε = 0.5	l∞, ε = 4/255
PGD-1000	AutoAttack	PGD-1000	AutoAttack
l2-robust classifier	15.71 ± 0.21	3.65 ± 0.58	6.36 ± 1.66	0.54 ± 0.16
l∞-robust classifier	13.96 ± 0.12	5.03 ± 0.16	17.63 ± 0.16	3.52 ± 0.31
(b) Cross-paradigm robustness (AutoAttack by default) of encoders naturally trained from natural or robust datasets. The robust datasets are generated from  $\ell_{\infty}$ -robust encoders trained with the SL or CL paradigm.
Training Dataset	CL	MIM	DM	SL	Cross. Rob.
Natural	0.22 ± 0.03	2.33 ± 0.12	3.41 ± 0.32	0.00 ± 0.01	0.00
Robust w/ SL model	0.23 ± 0.01	2.73 ± 0.33	2.54 ± 0.20	3.52 ± 0.31	0.23
Robust w/ CL model	1.28 ± 0.01	1.16 ± 0.23	1.19 ± 0.14	2.21 ± 0.29	1.16
Figure 3: The cross-paradigm robustness of adversarial examples generated with encoders from different learning paradigms. The  $(A, B)$ -th cell represents the accuracy of adversarial examples generated with an  $A$ -paradigm model (encoder with linear head) when evaluated on a  $B$ -paradigm model (encoder with linear head). Darker colors (i.e., higher accuracy) indicate worse transferability of adversarial examples.
supervised models, DenseNet-121 (SL-DN) [18], and Inception-V3 (SL-InC) [35] as baselines. We plot the transferred adversarial robustness in the confusion matrix in Figure 3.
The results in Figure 3 demonstrate two important messages: 1) adversarial examples transfer well across different architectures (SL-RN, SL-DN, SL-InC) under the same paradigm (SL), and 2) adversarial examples transfer poorly between different paradigms, even under the same architecture (for example, both CL and SL adopt ResNet-50 as backbones). It suggests that for adversarial transferability, paradigms have much more influence than model architectures. This phenomenon can be well explained by the paradigm-wise shortcut-like behaviors of non-robust features (Section 3), and it indicates that adversarial transferability is also largely a paradigm-wise phenomenon.
5.3 Relationship to Natural Transferability between Paradigms
In the discussions above, we demonstrate that adversarial examples do not have (good) transferability between different learning paradigms. We note that this discovery in adversarial transferability is not in contradiction to the good natural transferability between different paradigms. Indeed, many existing works [6, 15, 41, 16, 50, 52] show that representations learned from SSL have good downstream performance, particularly when evaluated with linear probing. Moreover, theoretical guarantees on the downstream classification performance have also been established for contrastive learning [13, 42, 43, 8], non-contrastive learning [53], and masked image modeling [51]. Since adversarial examples are essentially out-of-distribution examples (not drawn from the natural data distribution), the generalization guarantees on natural data do not apply. The fact that these paradigms have good natural transferability and poor adversarial transferability serves as another piece of evidence for our understanding that the misclassification of adversarial examples is caused by paradigm-wise shortcuts instead of real useful features.
In this paper, we have investigated the real usefulness and robustness of robust and non-robust from a wider context. By studying their behaviors across four different learning paradigms, we have found that robust features are as useful as natural features, while non-robust features generated by attacking supervised models become largely useless when transferred to other self-supervised learning paradigms, indicating that non-robust features are not real features but more like paradigm-wise shortcuts. Meanwhile, we have also shown that robust datasets containing only robust features are insufficient to attain model robustness under AutoAttack, indicating that feature non-robustness is not the only source of adversarial vulnerability. Posed as a challenge to common beliefs of robust and non-robust features, this work advocates the idea that their real usefulness and robustness should be examined under a cross-paradigm perspective as well as more reliable attacks.
Yisen Wang was supported by National Key R&D Program of China (2022ZD0160304), National Natural Science Foundation of China (62006153, 62376010, 92370129), Open Research Projects of Zhejiang Lab (No. 2022RC0AB05), Beijing Nova Program (20230484344), and CCF-Baichuan-EB Fund.
[1] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In ICML, 2018.
[2] Yang Bai, Xin Yan, Yong Jiang, Shu-Tao Xia, and Yisen Wang. Clustering effect of (linearized) adversarial robust models. In NeurIPS, 2021.
[3] Yang Bai, Yuyuan Zeng, Yong Jiang, Shu-Tao Xia, Xingjun Ma, and Yisen Wang. Improving adversarial robustness via channel-wise activation suppressing. In ICLR, 2021.
[4] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In ICLR, 2022.
[5] Philipp Benz, Chaoning Zhang, and In So Kweon. Batch normalization increases adversarial vulnerability and decreases adversarial transferability: A non-robust feature perspective. In CVPR, pages 7818-7827, 2021.
[6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[7] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, pages 2206-2216. PMLR, 2020.
[8] Jingyi Cui, Weiran Huang, Yifei Wang, and Yisen Wang. Rethinking weak supervision in helping contrastive learning. In ICML, 2023.
[9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[11] Alhussein Fawzi, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. Robustness of classifiers: from adversarial to random noise. NeurIPS, 29, 2016.
[12] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
[13] Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. In NeurIPS, 2021.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[15] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[16] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dólar, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 16000-16009, 2022.
[17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
[38] Nikolaos Tsilivis, Jingtong Su, and Julia Kempe. Can we achieve robustness from data alone? arXiv preprint arXiv:2207.11727, 2022.
[39] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In ICLR, 2019.
[40] Xin Wang, Jie Ren, Shuyun Lin, Xiangming Zhu, Yisen Wang, and Quanshi Zhang. A unified approach to interpreting and boosting adversarial transferability. In ICLR, 2021.
[41] Yifei Wang, Zhengyang Geng, Feng Jiang, Chuming Li, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Residual relaxation for multi-view representation learning. In NeurIPS, 2021.
[42] Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In ICLR, 2022.
[43] Yifei Wang, Qi Zhang, Tianqi Du, Jiansheng Yang, Zhouchen Lin, and Yisen Wang. A message passing perspective on learning dynamics of contrastive learning. In ICLR, 2023.
[44] Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In ICML, 2019.
[45] Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR, 2020.
[46] Zeming Wei, Yifei Wang, Yiwen Guo, and Yisen Wang. Cfa: Class-wise calibrated fair adversarial training. In CVPR, pages 8193-8201, 2023.
[47] Dongxian Wu, Yisen Wang, Shu-Tao Xia, James Bailey, and Xingjun Ma. Skip connections matter: On the transferability of adversarial examples generated with resnets. In ICLR, 2020.
[48] Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. In NeurIPS, 2020.
[49] Jiayu Wu, Qixiang Zhang, and Guoxi Xu. Tiny ImageNet challenge, 2017.
[50] Weilai Xiang, Hongyu Yang, Di Huang, and Yunhong Wang. Denoising diffusion autoencoders are unified self-supervised learners. arXiv preprint arXiv:2303.09769, 2023.
[51] Qi Zhang, Yifei Wang, and Yisen Wang. How mask matters: Towards theoretical understandings of masked autoencoders. NeurIPS, 35:27127-27139, 2022.
[52] Qi Zhang, Yifei Wang, and Yisen Wang. On the generalization of multi-modal contrastive learning. In ICML, 2023.
[53] Zhijian Zhuo, Yifei Wang, Jinwen Ma, and Yisen Wang. Towards a unified theoretical understanding of non-contrastive learning via rank differential mechanism. In ICLR, 2023.
Here we elaborate the experimental and technical details. First, we specify how the robust/non-robust datasets are constructed in our experiments. Then we detail the pre-training and linear probing hyper-parameters. Finally, we illustrate the settings of our robustness evaluation framework.
A.1 Dataset Construction
We largely follow Ilyas et al. [19] in the process of dataset construction. Unless specified, we use the robust and non-robust datasets generated from images for experiments (Section 3, Section 5). In Section 4, we generate the robust CIFAR10 with both supervisedly trained and contrastively trained ResNet-18. All constructed datasets are obtained via  $\ell_2$  PGD optimization. The detailed settings are listed below
Table 3: Experimental configurations in dataset construction. We generate robust and non-robust CIFAR10 starting from images (initialized with randomly drawn different images in the dataset) or from noise (random Gaussian noise).
Dataset	Iteration	Step Size	ε-Cons.
Robust CIFAR10 (From Image)	1000	1.0	∞
Robust CIFAR10 (From Noise)	10000	1.0	∞
Non-Robust CIFAR10 (From Image)	1000	0.1	0.5
Non-Robust CIFAR10 (From Noise)	10000	0.1	0.5
Robust Tiny-[ImageNet-200 (From Image)	2000	1.0	∞
Non-Robust Tiny-[ImageNet-200 (From Image)	2000	0.1	0.5
A.2 Pre-Training
For evaluating the usefulness and robustness of features (Section 3, Section 4), we use ResNet-18 [14] for SL, MAE [16] backboned by ViT-t [10] for MIM, ResNet-18 for CL, and DDPM [17] backbone by UNet [29] for DM in all of the three versions of CIFAR10 [21] and Tiny-ImageNet-200 [49]. We additionally use supervisedly trained ResNet-50, DenseNet-50 [18], and InceptionV3 [35] for discussion in the paradigm-wise transferability of adversarial examples (Section 5). We list the hyperparameter configurations below. We pick the default settings commonly adopted by the community when training the models and feel it unnecessary to intentionally optimize the hyper-parameters since our main research goal is focused on rethinking the essence of non-robust features.
Table 4: The hyper-parameter settings of pre-training on the three versions of CIFAR10.
Paradigm	Model	Epoch	B-Size	LR	LR Sche.	W-Decay	Data Aug.
SL	ResNet-18	50	128	1e-1	CosineAnnealingLR	5e-4	✓
ResNet-50	50	128	1e-1	CosineAnnealingLR	5e-4	✓
DenseNet-50	50	128	1e-1	CosineAnnealingLR	5e-4	✓
InceptionV3	50	128	1e-1	CosineAnnealingLR	5e-4	✓
CL	ResNet-18	2000	512	1e-4	CosineAnnealingLR	1e-5	✓
MIM	MAE(ViT-t)	2000	512	2e-4	LambdaLR	5e-2	✓
DM	DDPM(UNet)	1000	128	2e-4	LambdaLR	0	✓
A.3 Linear Probing
We use linear probing to evaluate the representation learned by our encoders and we attach the trained linear heads to the encoders to transform them into classifiers. (Section 3, Section 4, Section 5).
Table 5: The hyper-parameter settings of pre-training on the three versions of Tiny-ImageNet-200.
Paradigm	Model	Epoch	B-Size	LR	LR Sche.	W-Decay	Data Aug.
SL	ResNet-18	100	128	1e-2	CosineAnnealingLR	1e-5	✓
CL	ResNet-18	1000	512	2e-3	CosineAnnealingLR	1e-5	✓
MIM	MAE(ViT-b)	1000	512	2e-4	LambdaLR	5e-2	✓
DM	DDPM(UNet)	1500	128	2e-5	LambdaLR	0	✓
Table 6: Hyper-parameter configuration of linear probing.
Dataset	Epoch	B-Size	LR	LR Sche	W-Decay	Data Aug.
CIFAR10	30	256	0.1	CosineAnnealingLR	5e-4	✓
Tiny/ImageNet-200	50	256	0.01	CosineAnnealingLR	1e-5	✓
A.4 Robustness Evaluation
To replicate the results reported in Ilyas et al. [19], we use  $\ell_{\infty}$  PGD-1000 [25] bounded by  $\varepsilon = 4 / 255$  and  $\ell_2$  PGD-1000 bounded by  $\varepsilon = 0.5$  as baseline evaluation. Besides, we use  $\ell_{\infty}$  AutoAttack bounded by  $\varepsilon = 4 / 255$  for more reliable evaluation. Due to the randomness induced in the encoding process of MAE and DDPM, we use the randomized version of AutoAttack when evaluating their robustness, otherwise, we use the standard version of AutoAttack.
B Ablation Study on Cross-Paradigm Transferability
In Section 5.1, we have shown that adversarial examples generated by attacking contrastive learning (CL) have poor transferability on attacking supervised learning (SL). To further investigate how this happens, we conduct an experiment that gradually ablates each part of the InfoNCE loss.