SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Comparisons of ImageNet-1K clean and robustness accuracies  $(\%)$  between MIM and NIM-MIM, with or without the proposed  $\mathrm{De}^3$  defense.  ${}^{+}\mathrm{De}^{3}$  refers to the defense method that adapts  $\mathrm{De}^3$  to MIM methods (i.e., de-masking by decoder for adversarial defense). Numbers following  $\mathrm{De}^3$  indicate the degradation level used in defense, where  $\gamma$  is the mask ratio and  $\sigma$  is the noise scale. NIM-MAE models that use  $\mathrm{De}^3$  for defense are finetuned on denoised images from the pretrained model.
Model De3	MAE [17]	NIM-MAE	SimMIM [38]	NIM-SimMIM
None	γ=0.75	None	σ=70	σ=140	None	γ=0.6	None	σ=70	σ=140
Clean	83.05	44.96	82.58	78.68	70.69	83.62	43.05	82.76	76.23	68.76
FGSM	28.29	36.38	31.66	53.66	53.12	30.38	34.29	31.90	51.84	51.67
PGD-10	0.25	11.58	0.31	34.61	39.82	1.13	7.39	2.02	33.41	37.45
AA	0.00	2.58	0.00	23.24	33.70	0.00	1.82	0.00	21.70	32.46
(c)
Figure 3: Qualitative results of reconstruction by MAE and NIM-MAE pretrained models. (a) Original images. (b)  $75\%$  masked images. (c) Reconstructed images of (b) by the MAE pretrained model. (d) Images added  $\sigma = 70$  Gaussian noise. (e) Reconstructed images of (d) by our NIM-MAE pretrained model. (f) Images added  $\sigma = 140$  Gaussian noise. (g) Reconstructed images of (f) by our NIM-MAE pretrained model. The images shown are randomly selected from the ImageNet-1K validation set.
With the  $\mathrm{De}^3$  defense provided by the NIM pretrained models, the downstream finetuned classifiers obtain strong robustness for all white-box attacks. Numerically, when the Gaussian noise's  $\sigma$  is 70 (or 140), the magnitude of the added Gaussian noise is about  $70 / 4 = 17.5$  (or 35) times the adversarial noise. Therefore, the adversarial noise can be flooded and removed along the Gaussian noise by  $\mathrm{De}^3$ . As a result, the NIM models with  $\mathrm{De}^3$  outperform their MIM counterparts without defense with up to  $25.37\%$  improvement in terms of robustness against FGSM attacks, while only  $4.37\%$  clean accuracy is decreased. The  $\mathrm{De}^3$  defense is also effective against strong attacks like PGD-10 and AA that almost  $100\%$  successfully make the vanilla models fail.
In contrast, MIM does not benefit from the defense. Columns 2 and 7 are the results of using the MIM pretrained models for adversarial defense. Here, the  $\mathrm{De}^3$  defense is adapted for MIM models: it is performed by first randomly masking some patches and then reconstructing the masked images via the pretrained encoder and decoder, where the mask ratio remains the same as in pretraining (75% for MAE and 60% for SimMIM). It is shown that MIM pretrained models are unable to offer effective adversarial defenses as the clean accuracy drops drastically while robustness improves marginally. Figure 3 shows the qualitative results of using MAE and NIM-MAE to reconstruct masked and noisy images, respectively. Noticeably, the reconstruction of MAE is much worse than NIM-MAE, even when the  $\sigma$  is 140 and the images are unidentifiable for humans. We further quantitatively evaluate the reconstruction quality by computing PSNR (peak signal-to-noise ratio, higher the better) between the original and reconstructed images over the whole ImageNet-1K validation set. As a result, the PSNR of MAE is 20.88 when the mask ratio is 75%, while for NIM-MAE, it is 27.43 when  $\sigma = 70$  and 24.9 when  $\sigma = 140$ .
Table 2: Comparisons of ImageNet-1K clean and robustness accuracies (%) between adversarial trained ViT-base models and NIM-MAE model with the  $\mathrm{De}^3$  defense.
Pretrain	Defense	Clean	FGSM	PGD-10	AA
scratch	Adv. Training	56.97	39.89	24.25	19.50
MAE	Adv. Training	69.67	51.80	39.65	34.56
NIM-MAE	Adv. Training	69.28	52.77	39.97	34.84
NIM-MAE	De3(σ = 140)	70.69	53.12	39.82	33.70
5.3 Comparing NIM+De<sup>3</sup> with Adversarial Training
To further help understand the effectiveness of our NIM models with the  $\mathrm{De}^3$  defense, we compare it with the adversarial training method in this section. Adversarial training for ViTs is not a trivial task and techniques and hyperparameters for CNNs may not be beneficial or applicable for ViTs [1, 30]. Here, we adopt the adversarial training recipe for ViTs provided by a recent work [23]. We do the PGD-5 adversarial training for 20 epochs so the training time is close to ours and adjust the learning rate schedule accordingly. Table 2 shows the results of doing adversarial training from scratch and the MAE and NIM-MAE pretrained models. It is shown that both MAE and NIM-MAE are helpful for the adversarial training of ViTs. Compared to the adversarial models, our NIM-MAE model with  $\sigma = 140\mathrm{De}^3$  defense shows comparable robustness with slightly higher clean accuracy.
Adversarily trained models often compromise the clean accuracy too much. In our experiment, the performance for natural images downgrades by  $13.38\%$  for the MAE model. Although this issue can be mitigated by setting a larger weight for the standard loss term in the objective function [39], the adjustment of the trade-off comes at the cost of starting over another training process. In contrast, our  $\mathrm{De}^3$  defense enables the testing time tunable trade-off, i.e., we do not need to train another network to trade robustness for accuracy (or vice versa). By increasing the scale of noise added in the defense at test time, more adversarial perturbations can be flooded and removed by the denoising pretrained model, at the expense of poorer reconstruction quality and a greater loss of semantic information (Figure 3 (e) vs. (g)), resulting increased robust accuracies and decreased clean accuracy (Figure 4). For example, our model's robustness against the PGD-10 attack can be improved from almost 0 to  $31.25\%$  and only compromise  $2\%$  of clean accuracy  $(79.37\%)$ . Therefore, in practice, our defense can adjust its performance on clean and adversarial images according to the requirement of the context. For example, in autonomous driving, the need for adversarial robustness may increase when the vehicle comes into an adverse environment, while for safer places, the need for high clean accuracy is prioritized.
While we prepare this paper, some more recent studies on adversarial training for ViTs have emerged [28, 31], and the adversarial robustness of ViTs on ImageNet-1K was improved significantly with more delicate implementation and longer training time. Although our NIM with  $\mathrm{De}^3$  approach does not achieve the same level of adversarial robustness as these existing works, we want to highlight that the aim of this work is not to compete with the state-of-the-art methods. Instead, we aim to demonstrate to the research community that NIM can serve as a promising and advantageous self-learning paradigm for enhancing adversarial robustness.
5.4  $\sigma$  in Pretraining
In this section, we present our ablation studies to evaluate how the random distributions of the Gaussian noise scale parameter  $\sigma$  in pretraining influence the fine-tuned models' performance. We adopt MAE as the default implementation of MIM and randomly selected 5,000 images when evaluating robust accuracy for reducing experimental overhead.
First, we present the clean and robust accuracy of models without defense. The left subfigure of Figure 5 shows that NIM pretrained models can achieve competitive accuracy on clean images, especially when the degradation level hyperparameter is set to or concentrated at a fair value (i.e.,  $\sigma = 75$ ,  $\sigma \sim \Gamma(25,3)$ ,  $\sigma \sim U(50,100)$ ). On the other hand, most of the NIM models lead to higher adversarial robustness than MAE, if the variance of  $\sigma$  is not too large (e.g.,  $\sigma \sim U(0,250)$ ,  $\sigma \sim \Gamma(3,25)$ ). Such phenomenon is consistent with the understanding that NIM learns noise-invariant features in Section 3.2.
Figure 4: Clean and robust accuracies of our NIM-MAE with  $\mathrm{De}^3$  of different  $\sigma$ . The clean accuracy decreases and robust accuracies increase as the  $\sigma$  increases, enabling a tunable trade-off in testing time.
Figure 5: Performance of MAE and NIM-MAE with different  $\sigma$  in pretraining. Left: Accuracy on clean images and adversarial images attacked by FGSM of MAE and NIM-MAE fine-tuned models without  $\mathrm{De}^3$  defense. Right: The Pareto frontiers of NIM-MAE models with  $\mathrm{De}^3$  defense between clean accuracy and robust accuracy against PGD-10 attacks. The Pareto frontiers are measured by using different  $\sigma$  in defense (increased by 10 from 0 to 160).
In the right subfigure of Figure 5, we show the empirical Pareto frontier of  $\mathrm{De}^3$  using different NIM-MAE pretrained models, i.e., the points on the figure are the models' best achievable trade-offs between clean and robust accuracy by using different  $\sigma$  in defense. We first observe that the Pareto frontiers of models using globally set  $\sigma$  collapse to very few points because using  $\sigma$  other than the one in pretraining leads to lowering both clean and robust accuracy. Meanwhile, the results suggest that  $\sigma$  sampled from the Gamma distribution generally achieves better accuracy-robustness trade-off than the ones from the Uniform distribution. Among all models,  $\sigma \sim \Gamma(25,3)$  achieves the best performance by being concentrated to a moderate value and having a suitable variance.
In this paper, we investigate noisy image modeling (NIM), a simple variant of masked image modeling where the pretext task is changed from mask prediction to denoising. We discover that NIM exhibits a remarkable ability to reconstruct images even from images with intense noise and are motivated to utilize this denoising capability to remove adversarial perturbations. We propose a simple method that enables the NIM models to provide not only pretrained features but also an adversarial defense. To further achieve a tunable trade-off during test time, we sample the noise level hyperparameter from random distributions rather than setting it globally as MIM. We demonstrate by extensive experimental results that in terms of adversarial robustness, NIM is advantageous over MIM thanks to its strong denoising ability. We also show that the adversarial defense provided by NIM achieves comparable performance to a recent adversarial training method and offers greater flexibility and applicability due to its tunable accuracy-robustness trade-off. We hope that our work will motivate the community to explore NIM and other variants of MIM so the full potential of generative visual pretraining can be realized.
This work was supported in part by the Australian Research Council under Projects DP240101848 and FT230100549, and also in part by the NRF and IITP grants funded by the Korea government (MSIT) (No. 2022R1A2C3012210, 2021-0-01343).
[1] Yutong Bai, Jieru Mei, Alan L Yuille, and Cihang Xie. Are transformers more robust than cnns? Advances in Neural Information Processing Systems, 34:26831-26843, 2021.
[2] Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021.
[3] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part III 13, pages 387-402. Springer, 2013.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[5] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12299-12310, 2021.
[6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[7] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640-9649, 2021.
[8] Yabo Chen, Yuchen Liu, Dongsheng Jiang, Xiaopeng Zhang, Wenrui Dai, Hongkai Xiong, and Qi Tian. Sdae: Self-distillated masked autoencoder. In European Conference on Computer Vision, pages 108-124. Springer, 2022.
[9] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09670, 2020.
[10] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pages 2206-2216. PMLR, 2020.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE, 2009.
[12] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[13] Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre-training of vision transformers. arXiv preprint arXiv:2111.12710, 2021.