SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zunzhi You<sup>1</sup>, Daochang Liu<sup>1</sup>, Bohyung Han<sup>2</sup>, Chang Xu<sup>1</sup>
<sup>1</sup> School of Computer Science, University of Sydney
$^{2}$  ECE & IPAI, Seoul National University
zyou5834@uni.sydney.edu.au, daochang.liu@sydney.edu.au,
bhhan@snu.ac.kr, c.xu@sydney.edu.au
Recent advancements in masked image modeling (MIM) have made it a prevailing framework for self-supervised visual representation learning. The MIM pretrained models, like most deep neural network methods, remain vulnerable to adversarial attacks, limiting their practical application, and this issue has received little research attention. In this paper, we investigate how this powerful self-supervised learning paradigm can provide adversarial robustness to downstream classifiers. During the exploration, we find that noisy image modeling (NIM), a simple variant of MIM that adopts denoising as the pre-text task, reconstructs noisy images surprisingly well despite severe corruption. Motivated by this observation, we propose an adversarial defense method, referred to as  $\mathrm{De}^3$ , by exploiting the pretrained decoder for denoising. Through  $\mathrm{De}^3$ , NIM is able to enhance adversarial robustness beyond providing pretrained features. Furthermore, we incorporate a simple modification, sampling the noise scale hyperparameter from random distributions, and enable the defense to achieve a better and tunable trade-off between accuracy and robustness. Experimental results demonstrate that, in terms of adversarial robustness, NIM is superior to MIM thanks to its effective denoising capability. Moreover, the defense provided by NIM achieves performance on par with adversarial training while offering the extra tunability advantage. Source code and models are available at https://github.com/youzunzhi/NIM-AdvDef.
The idea of fine-tuning pretrained deep neural networks is essential for the success of machine learning to date. A deep neural network that has been well-pretrained on a large-scale dataset can serve as a good initialization for various downstream tasks and improve their performance. Within this paradigm, self-supervised learning plays a vital role as this approach allows the networks to take advantage of vast amounts of unlabeled data. With the emergence of powerful, scalable Transformer-based vision models [14, 21], the masked image modeling (MIM) has rapidly developed recently and become the new dominant paradigm for visual feature pretraining [38, 17, 8, 26], following the success of masked language modeling [12, 4] in NLP. Conceptually, the idea behind MIM is simple. During the pretraining phase, unlabeled training images are randomly masked patch-wise and then fed into an encoder-decoder architecture, where the decoder attempts to recover the original images from the features embedded by the encoder.
Although pretraining methods like MIM leads to improved standard performance on downstream visual tasks, the adversarial robustness of the fine-tuned model is not enhanced. The vulnerability to imperceptible yet maliciously crafted perturbations will significantly limit the real-world deployment of MIM, especially for safety-critical applications. Although the existing state-of-the-art approach,
(b)
Figure 1: (a) Comparison between the NIM framework and the MIM framework. The two frameworks can be almost exactly the same except for the degradation step. (b) An illustration of the proposed  $\mathrm{De}^3$  adversarial defense.  $\mathrm{De}^3$  first adds a certain level of noise to the attacked image, then uses the model pretrained by the NIM framework to denoise the image.
adversarial training [22, 23], can train adversially robust models, it is known to be computationally expensive [22] and can result in a significant drop in accuracy on clean data [34], which undermines the usefulness of the MIM pretrained features. Moreover, compared to standard training, the sample complexity of achieving adversially robust generalization is significantly higher, increasing the importance of label efficiency in practice. Motivated by these concerns, we ask the question: How can the state-of-the-art generative pretraining paradigm provide adversarial robustness in addition to standard accuracy?
As we explore this question, we observe two notable characteristics about the paradigm: (1) Other degradations than masking patches, such as zooming-in, blurring, or masking in the frequency domain, can be effective for the pretext task of self-supervised learning [33, 37]. (2) An attempt to understand MIM revealed that it implicitly learns occlusion-invariant features [19]. Motivated by the observations, we explore Noisy Image Modeling (NIM), a variant of MIM that uses denoising as the pretext task and learns to encode noise-invariant features (Figure 1 (a)).
During the exploration, we find that NIM's denoised images are of surprisingly high quality, even from intensely noisy images (Figure 3 (e) and (g)). However, in the common practice of NIM, only the encoder is retained for further finetuning, while the decoder that performs image reconstruction is discarded. Therefore, the strong denoising capability of NIM will be wasted, which we believe can be utilized for removing adversarial perturbations. To address this, we propose an adversarial defense method named  $\mathrm{De}^3$ : Denoising by Decoder for Adversarial Defense, which first adds some Gaussian noise to the adversarial samples and then tries to reconstruct the original images, as shown in Figure 1 (b). With  $\mathrm{De}^3$ , NIM provides not only pretrained features but also an adversarial defense to its fine-tuned models. We further propose to randomize the hyperparameter that controls the degradation level in pretraining, so the defense provided by NIM can achieve a tunable trade-off between standard accuracy and adversarial robustness, which further enhances its applicability in practice.
To summarize, this work has the following contributions:
- We propose a novel method called  $\mathrm{De}^3$  to utilize the strong denoising ability of NIM models so they can provide defense against adversarial attacks beyond pretrained features.
- Rather than setting the hyperparameter that controls the degradation level globally, as done in existing MIM approaches, we propose to sample it from random distributions, so the defense achieves a flexible accuracy-robustness trade-off that can be tuned at test time.
- Extensive experiments show that our NIM pretraining learns as good visual features as MIM, while being advantageous for it effectively enhances the robustness of fine-tuned models against strong adversarial attacks.
2.1 Self-supervised Pretraining in Vision
Learning visual representations from unlabeled images has been a research direction in machine learning and computer vision that has drawn increasing attention in recent years. While masked language modeling (MLM) has achieved great success in pretraining large-scale language models [12, 4, 20], the mainstream framework for pretraining vision models was contrastive learning (CL) [6, 18, 7] until recently. With the significant emergence of Transformers adapted for vision tasks [14, 5, 21], the counterpart of MLM in vision, MIM, becomes the new prevailing self-supervised pretraining paradigm [26, 13, 17, 38]. Through the process of masking and predicting image patches, the encoder learns to extract meaningful visual features.
Beyond using mask prediction as the pretext task, there have been studies examining the outcomes of employing alternative degradation methods. Tian et al. [33] investigate five methods, including zoom-in, zoom-out, distortion, blurring, and de-colorizing, and discover that all of these methods outperform supervised pretraining. Additionally, Xie et al. [37] explore masking in the frequency domain and demonstrate that similar low-level tasks such as super-resolution and denoising yield competitive performance. These studies inspire us to explore the framework of noisy image modeling. More recently, Fang et al. [15] also modify the masking operation in the generative pretraining framework, where the input images are degraded by an additional generator with a small trainable BEiT [2]. However, instead of solely using the generative pretraining framework for pretrained features, we further propose to leverage it for enhancing adversarial robustness.
2.2 Adversarial Robustness
Since about a decade ago when researchers found that deep neural networks are fragile to imperceptible adversarial perturbations [32, 3], enhancing their robustness to such attacks has been an active research area, as stronger threat models have also been continuously developed [16, 25, 22, 10]. One of the most successful methods, adversarial training (AT), incorporates adversarial examples during training to enhance the model's adversarial robustness [16, 22, 39]. While effective, AT methods present many challenges that limit their applicability. For instance, due to the necessity of searching for effective attacks for each image during training, the computational cost can be too expensive. Moreover, AT achieves high robustness at the cost of compromising standard accuracy [34, 27]. Although Wang et al. [36] mitigate this issue by a model-conditional training approach that enables an in-situ calibration of the trade-off between robustness and accuracy, the training cost is further increased, and the method is limited to CNNs.
Another line of research aims to defend against adversarial attacks in the input space, as opposed to the model space like AT. These methods often use an additional generator to restore clean images from attacked ones, such as GAN [29] or Diffusion Models [24]. As their training does not depend on the threat model like AT, these methods can often defend against unseen threats. However, their performance is usually not comparable with AT methods [10]. Our  $\mathrm{De}^3$  method is distinct from these methods because our defense is provided by pretraining, eliminating the need for an additional generative model. Therefore, our cost of obtaining the defense also brings pretrained features and boosts fine-tuning. More importantly, the goal of this work is mainly to show the advantage of NIM over MIM in terms of adversarial robustness, not to propose a novel adversarial defense method.
3 Noisy Image Modeling
3.1 Preliminary: Masked Image Modeling
We first briefly revisit the MIM pretraining framework. For every image  $\mathbf{x} \in \mathbb{R}^{H \times W \times C}$ , where  $H$ ,  $W$ , and  $C$  denote the height, width, and channel dimensions respectively, sampled from the training dataset  $\Omega$ , MIM first randomly samples a binary mask  $\mathbf{M}$  under the control of the mask ratio
hyperparameter  $\gamma$  so that  $\sum_{i}^{N}m_{i} / N = \gamma$ , where each element  $m_{i}\in 0,1$  of the binary mask  $\mathbf{M}$  corresponds to a pixel in the image, and  $N$  is the total number of pixels. Then, the masked image is given by
$$
\hat {\mathbf {x}} _ {M I M} = \mathbf {x} \odot (\mathbf {1} - \mathbf {M}), \tag {1}
$$
where  $\odot$  denotes the element-wise product. After that, MIM uses its encoder  $f_{\theta}$  and decoder  $g_{\phi}$  parameterized by  $\theta$  and  $\phi$ , respectively, to predict the reconstructed image. In practice, the encoder  $f_{\theta}$  of MIM is usually instantiated by Vision Transformers (ViTs) [14] or its variants [21], where patch-wise masks are employed. Finally, MIM adopts some distance metric  $\mathcal{D}(\cdot ,\cdot)$  such as  $\ell_1$  loss to measure the reconstruction quality of the masked patches. Overall, the objective of MIM is formulated as follows:
$$
\min  _ {\theta , \phi} \underset {\mathbf {x} \in \boldsymbol {\Omega}} {\mathbb {E}} \mathcal {D} \left(g _ {\phi} \left(f _ {\theta} \left(\hat {\mathbf {x}} _ {M I M}\right)\right) \odot \mathbf {M}, \mathbf {x} \odot \mathbf {M}\right). \tag {2}
$$
3.2 Noisy Image Modeling: Learning Noise-invariant Feature
Figure 1 (a) presents the similarity between the process of MIM and NIM pretraining. As a variant of MIM, NIM shares many components including the network architecture design, the reconstruction target, and the training techniques. The only difference is that NIM degrades the original image by adding Gaussian noise as
$$
\hat {\mathbf {x}} _ {N I M} = \mathbf {x} + \sigma \epsilon , \quad \epsilon \sim \mathcal {N} (0, \mathbf {I}), \tag {3}
$$
where  $\sigma$  is the parameter that controls the degradation level, i.e., the noise scale. The objective function of NIM is given by
$$
\min  _ {\theta , \phi} \mathbb {E} _ {\mathbf {x} \sim \boldsymbol {\Omega}} \mathcal {D} \left(g _ {\phi} \left(f _ {\theta} \left(\hat {\mathbf {x}} _ {N I M}\right)\right), \mathbf {x}\right). \tag {4}
$$
As MIM learns to predict masked regions, the encoder-decoder pretrained under NIM is able to denoise corrupted input images. Note that the term "noise" we use has a different meaning from that in the classical Denoising autoencoders (DAE) [35]. In our context, we refer to the Gaussian noise that is added to the image, while DAE corrupts the input by replacing randomly selected pixels with 0, which is actually equivalent to masking..
Following the work trying to understand MIM from an occlusion-invariant feature learning perspective [19], we present how NIM learns noise-invariant features. Assume there exists a decoder network  $g_{\phi'}'$  parameterized by  $\phi'$  that can almost restore the feature embedded by  $f_{\theta}$  to the its input image:
$$
g _ {\phi^ {\prime}} ^ {\prime} \left(f _ {\theta} (\mathbf {x})\right) \approx \mathbf {x}. \tag {5}
$$
Then, the objective function of NIM in Eq. (4) can be rewritten as
$$
\min  _ {\theta , \phi} \mathbb {E} _ {\mathbf {x} \sim \boldsymbol {\Omega}} \mathcal {D} \left(g _ {\phi} \left(f _ {\theta} \left(\hat {\mathbf {x}} _ {N I M}\right)\right), g _ {\phi^ {\prime}} ^ {\prime} \left(f _ {\theta} (\mathbf {x})\right)\right). \tag {6}
$$
By defining a new similarity measurement
$$
\mathcal {D} _ {\phi , \phi^ {\prime}} ^ {\prime} \left(z _ {1}, z _ {2}\right) \triangleq \mathcal {D} \left(g _ {\phi} \left(z _ {1}\right), g _ {\phi^ {\prime}} ^ {\prime} \left(z _ {2}\right)\right), \tag {7}
$$
Eq. (6) is further simplified as
$$
\min  _ {\theta , \phi} \mathbb {E} _ {\mathbf {x} \sim \boldsymbol {\Omega}} \mathcal {D} _ {\phi , \phi^ {\prime}} ^ {\prime} \left(f _ {\theta} \left(\hat {\mathbf {x}} _ {N I M}\right), f _ {\theta} (\mathbf {x})\right). \tag {8}
$$
Hence, the objective function of NIM can be viewed as minimizing the distance of feature embedding between the original image  $\mathbf{x}$  and the noisy image  $\hat{\mathbf{x}}_{NIM}$ , implying that the feature embedding learned by  $f_{\theta}$  should be noise-invariant.
4 De3: Denoising by Decoder for Adversarial Defense
4.1 Formulation of  $\mathbf{De}^3$
Although MIM is effective in representation learning, it is still challenging to learn robust pretrained features. Fine-tuned models are easily compromised by small adversarial perturbations, limiting their use in safety-critical tasks. With the awareness of this problem, we explore the potential of NIM and find that the reconstruction ability of the NIM models is surprisingly good. For example, Figure 3 (e) and (g) illustrate the reconstructed images from the noisy images. We observe that most
of the semantics are preserved in the reconstruction even for the images that are too noisy to be recognized by humans. However, the common practice of the MIM pretraining only uses the encoder for the initialization of downstream models while the decoder that has the outstanding reconstruction capability is simply discarded. Meanwhile, we believe that the remarkable denoising capability of the NIM decoder can be employed to enhance the adversarial robustness of downstream models and propose a simple defense technique referred to as  $\mathrm{De}^3$ : Denoising by Decoder for Adversarial Defense.
As shown in Figure 1 (b),  $\mathrm{De}^3$  works during the test time of downstream tasks by first adding Gaussian noise to an input image and then denoising the image using the pretrained decoder as follows:
$$
y = c _ {\psi} \left(g _ {\phi} \left(f _ {\theta} \left(\left(\mathbf {x} ^ {\prime} + \sigma_ {D e ^ {3}} \epsilon\right)\right)\right)\right), \tag {9}
$$
where  $c_{\psi}$  is the fine-tuned downstream network parameterized by  $\psi$  given the pretrained  $\theta$ ,  $\mathbf{x}'$  is the input image that may be under adversarial attacks, and  $\sigma_{De^3}$  is the noise scale used for the defense. The motivation behind this method is straightforward: since the adversarial perturbation is bounded to a small  $\epsilon$ , it should be easily flooded by the Gaussian noise with a much larger magnitude and then can be removed along with the random Gaussian noise in the decoding process. Also, the noise is randomly generated during testing, which is usually not fully accessible to the threat models even in white-box attack settings, making it more difficult to find a way for an effective attack.
While the reconstruction of NIM models yields realistic results with minimal error, there remains a discernible gap between the reconstructed images and the original input images. Therefore, we propose to fine-tune the downstream model with denoised images, where training examples used for fine-tuning  $c_{\psi}$  also undergo an adding-noise-then-denoising procedure, simulating the conditions encountered during test time. The objective function of fine-tuning is formulated as follows:
$$
\min  _ {\psi} \mathbb {E} _ {\mathbf {x} \in \Omega} \mathcal {L} \left(c _ {\psi} \left(g _ {\phi} \left(f _ {\theta} \left(\mathbf {x} + \sigma_ {f t} \epsilon\right)\right)\right), \bar {y}\right), \tag {10}
$$
where  $\mathcal{L}$  is the loss function,  $\bar{y}$  is the ground-truth label of  $\mathbf{x}$ , and  $\sigma_{ft}$  is the noise scale using in the fine-tuning. Note that  $\theta$  and  $\phi$  are fixed during the fine-tuning process.
4.2 Achieving Tunable Trade-off by Sampling Random  $\sigma$  in Pretraining
Intuitively, the proposed  $\mathrm{De}^3$  method should have a tunable trade-off between clean accuracy and adversarial robustness by adjusting  $\sigma_{De^3}$ . Increasing  $\sigma_{De^3}$  will make adversarial perturbations more likely to be flooded and subsequently removed, resulting in stronger adversarial robustness. Meanwhile, as the degradation becomes more severe, the reconstruction quality diminishes, leading to lower clean accuracy. However, we observe that this assumption does not necessarily hold in practice. Figure 2 (a) demonstrates that reconstruction quality deteriorates when the noise levels imposed on input images during training and testing are different. We further show how the reconstruction loss varies with noise levels in Figure 2 (b) and observe the same phenomenon in the pretrained models with globally set  $\sigma = 25$ , 75, and 150. These observations suggest that if the model is exposed only to a single noise level during training, it struggles to generalize its denoising capability to images with different noise levels.
To address this issue, we propose to sample the noise level parameter  $\sigma$  from a random distribution instead of fixing it globally as existing MIM methods do. Figure 2 illustrates that models pretrained with randomly selected  $\sigma$ 's can effectively reconstruct input images with good quality, even for corrupted images across various noise scales, as long as the scale is not excessively large. Therefore, we can identify a good trade-off between accuracy and robustness in the NIM models.
In this section, we empirically show the effectiveness of NIM as a self-supervised pretraining method that can bring adversarial robustness via the  $\mathrm{De}^3$  method. We describe the settings in Section 5.1 and compare NIM with MIM in Section 5.2. Then, in Section 5.3, the proposed adversarial defense  $\mathrm{De}^3$  using NIM is compared with the adversarial training approach. Finally, Section 5.4 presents the ablation study that discusses how  $\sigma$  in pretraining makes differences to the model performance. More experimental results and details can be found in the supplementary.
Lower loss indicates better reconstruction.
(a) Qualitative comparison of reconstructions by NIM pretrained models with global and random sigma from images of different noise levels. Top: Noisy images of  $\sigma = 0$  (original image), 50, 75, 100, respectively. Middle: Reconstructed images by the NIM model whose  $\sigma$  in pretraining is globally set to 75. Bottom: Reconstructed images by the NIM model whose  $\sigma$  in pretraining is randomly sampled from  $\Gamma(25,3)$ .
(b) Relationship between the reconstruction loss and the noise level  $\sigma$  of models pretrained with  $\sigma$  set to 25, 75, 150 globally and sampled from  $\Gamma(25,3)$ . The standard deviation is also shown.
Figure 2: Qualitative and quantitative comparisons show that NIM pretraining with globally set  $\sigma$  only reconstructs well from the noisy images of noise level close to the  $\sigma$  in pretraining, while NIM pretraining with random  $\sigma$  reconstructs well for any  $\sigma$  in a reasonable span.
5.1 Experimental Setups
Datasets and Backbones We conduct all the experiments on ImageNet-1K [11] dataset. We use the training set (1.28 million images) in pretraining and finetuning and use the validation set (50,000 images) for evaluation. We adopt ViT-Base (ViT-B/16) [14] as the backbone in our main experiments. Results for additional backbones can be found in the supplementary.
Threat Models To evaluate the adversarial robustness, we adopt untargeted  $\ell_{\infty}$ -bounded attacks with radius  $\epsilon = 4 / 255$ , which is the most commonly used setting for adversarial robustness studies on ImageNet. We consider three popular white-box attacks: single-step attack FGSM [16], multi-step attack PGD [22], and the state-of-the-art attack AutoAttack (AA) [10]. For PGD, we set the number of steps  $n = 10$  and step size  $\alpha = 2 / 255$ . For AA, we use its 'rand' version since our defense is a randomized defense. To save the computational cost, we apply the AA attack on a 5000-image subset of the ImageNet1K validation set selected by RobustBench [9].
Training Implementations We adopt two representative MIM methods, MAE [17] and SimMIM [38], as the baseline MIM methods. By modifying only the degradation part, we train our NIM-MAE and NIM-SimMIM: the pretext task is changed from mask prediction (with a globally set mask ratio) to denoising (with randomly sampled  $\sigma$ ), while all the other hyperparameters and training techniques follow the original paper. Our default models are pretrained with  $\sigma \sim \Gamma(25, 3)$  and finetuned on denoised images of  $\sigma \sim U(0, 30)$ . All models are pretrained for 800 epochs and then finetuned for 100 epochs.
5.2 Comparing NIM with MIM
In Table 1, we compare our NIM models to the MIM baselines. First, we show that without any adversarial defense, the NIM-pretrained classifiers are slightly less accurate but more robust than the MIM-pretrained classifiers. Therefore, we believe that even in settings where a defense model is unavailable, NIM is a simple but effective self-supervised visual learning framework and is worth more investigation in the future.