Denoising Masked Autoencoders Help Robust Classification
Authors: QuanLin Wu
Hang Ye
Yuntian Gu
Huishuai Zhang
Liwei Wang
Di He
Venue: ICLR 2023 poster (2023)
Keywords: self-supervised
certified robustness
randomized smoothing

Abstract: In this paper, we propose a new self-supervised method, which is called denoising masked autoencoders (DMAE), for learning certified robust classifiers of images. In DMAE, we corrupt each image by adding Gaussian noises to each pixel value and randomly masking several patches. A Transformer-based encoder-decoder model is then trained to reconstruct the original image from the corrupted one. In this learning paradigm, the encoder will learn to capture relevant semantics for the downstream tasks, which is also robust to Gaussian additive noises. We show that the pre-trained encoder can naturally be used as the base classifier in Gaussian smoothed models, where we can analytically compute the certified radius for any data point. Although the proposed method is simple, it yields significant performance improvement in downstream classification tasks. We show that the DMAE ViT-Base model, which just uses 1/10 parameters of the model developed in recent work (Carlini et al., 2022), achieves competitive or better certified accuracy in various settings. The DMAE ViT-Large model significantly surpasses all previous results, establishing a new state-of-the-art on ImageNet dataset. We further demonstrate that the pre-trained model has good transferability to the CIFAR-10 dataset, suggesting its wide adaptability. Models and code are available at
https://github.com/quanlin-wu/dmae.

Published as a conference paper at ICLR 2023
DENOISING MASKED AUTOENCODERS HELP ROBUST CLASSIFICATION
Quanlin Wu', Hang Ye', Yuntian Gu', Huishuai Zhang", Liwei Wang?; ; Di He Peeknng University Microsoff Research Asia 3 National Key Lab of General AI, School of Artificial Intelligence, Peking University
{quanlin, yehang, dihe) d apku.edu.. en, guyuntiante ionntianneee pku, edu. en.
muzharg@microsoft.. com, wanglw8ole .pk.. en
ABSTRACT
In this paper, we propose new self-supervised method, which is called Denoising Masked AutoEncoders (DMAE), for learning certified robust classifiers of images. In DMAE, we corrupt each image by adding Gaussian noises to each pixel value and randomly masking several patches. A Transformer-based encoder-decoder model is then trained to reconstruct the original image from the corrupted one. In this learning paradigm, the encoder will learn to capture relevant semantics for the downstream tasks, which is also robust to Gaussian additive noises. We show that the pre-trained encoder can naturally be used as the base classifier in Gaussian smoothed models, where we can analytically compute the certified radius for any data point. Although the proposed method is simple, it yields significant perfor- mance improvement in downstream classification tasks. We show that the DMAE ViT-Base model, which just uses 1/10 parameters of the model developed in re- cent work (Carlini et al., 2022). achieves competitive or better certified accuracy in various settings. The DMAE ViT-Large model significantly surpasses all pre- vious results, establishing a new state-of-the-art on ImageNet dataset. We further demonstrate that the pre-trained model has good transferability to the CIFAR- 10 dataset, suggesting its wide adaptability. Models and code are available at https: //tp://gtttt com/ quan.inn Walliinnn//aii wu/dmaae
INTRODUCTION
Deep neural networks have demonstrated remarkable performance in many real applications (He et al., 2016; Devlin et al., 2019; Silver et al., 2016). However, at the same time, several works observed that the learned models are vulnerable to adversarial attacks (Szegedy et al., 2013; Biggio et al., 2013). Taking image classification an example, given an image æ that is correctly classified to label y by a neural network, an adversary can find a small perturbation such that the perturbed image, though visually indistinguishable from the original one, is predicted into a wrong class with high confidence by the model. Such a problem raises significant challenges in practical scenarios.
Given such critical issue, researchers seek to learn classifiers that can provably resist adver- sarial attacks, which is usually referred to as certified defense. One of the seminal approaches in this direction is the Gaussian smoothed model. A Gaussian smoothed model g is defined as g(x) = fx + v7), in which r ~ (O, I²I) and f is an arbitrary classifier, e.g., neural network. Intuitively, the smoothed classifier g can be viewed as ensemble of the predictions of that takes noise-corrupted images x + 17 as inputs. Cohen et al. (2019) derived how to analytically compute the certified radius of the smoothed classifier g, and follow-up works improved the training methods of the Gaussian smoothed model with labeled data (Salman et al., 2019; Zhai et al., 2021; Jeong & Shin, 2020: Horváth al., 2022; Jeong et al., 2021).
Recently, Salman et al. (2020); Carlini et al. (2022) took the first step to train Gaussian smoothed classifiers with the help of self-supervised learning. Both approaches use compositional model ar- chitecture for ff and decompose the prediction process into two stages. In the first stage, denoising
*Correspondence Di He <dihe www.e@kkuu.euue....... and Liwei Wang < @ut.guu.uuuu............

---

Published as a conference paper at ICLR 2023
Certified Accuracy(%) radius
Method
#Params
Extra data
0.5
1.0
1.5
2.0
3.0
RS (Cohen al., 2019)
26M
X
49.0
37.0
29.0
19.0
12.0
SmoothAdv (Salman et al., 2019)
26M
X
54.0
43.0
37.0
27.0
20.0
Consistency (Jeong & Shin, 2020)
26M
X
50.0
44.0
34,0
24.0
17.0
X
MACER (Zhai et al., 2021)
26M
57.0
43.0
31.0
25.0
14.0
Boosting (Horváth et al., 2022)
78M
X
57.0
44.6
38.4
28.6
20.2
X
SmoothMix (Jeong et al., 2021)
26M
50.0
43.0
38.0
26.0
17.0
Diflusion+BEiT (Carlini et al., 2022)
*857M
71.1"
54.3
38.1
29.5
13.1
Ours (DMAH VTT-B)
87M
X
69.6
57.9"
47.8"
35,4"
22.5'
Ours (DMAE VIT-L)
304M
X
73.6
64.6"
53.7*
41,5"
27.5
Table I: Certified accuracy (top-1) of different models on ImageNet. Following Carlini et al. (2022), for each noise level Ø, we select the best certified accuracy from the original papers. denotes the best result, and denotes the second best at each lg radius. &Carlini et al. (2022) uses diffusion model with 552M parameters and BEiT-Large model with 305M parameters. It can be seen that our DMAE ViT-B/ViT-L. models achieve the best performance in most of the settings.
model is used to purify the noise-corrupted inputs. Then in the second stage, a classifier is applied predict the label from the denoised image. Since the first-stage denoising model and the second- stage classification model can be learned or benefited from standard self-supervised approaches, the smoothed classifier g can achieve better performance than previous works. For example. Carlini et al. (2022) achieved 71.1% certified accuracy at l radius r 0.5 and 54.3% at = 1.0 on ImageNet by applying pre-trained denoising diffusion model in the first stage (Nichol & Dhari- wal, 2021) and a pre-trained BEÎT (Bao et al., 2021) in the second stage. Despite its impressive performance, such a two-stage process requires much more parameters and separated training.
Different from Salman et al. (2020); Carlini al. (2022) that use two models trained for separated purposes, we believe that a single compact network (i.e., vision Transformer) has enough expressive power learn robust feature representation with proper supervision. Motivated by the Masked Au- toEncoder (MAE) (He et al., 2022), which learns latent representations by reconstructing missing pixels from masked images, coder (DMAE). Given an unlabeled image, we corrupt the image by adding Gaussian noise to each pixel value and randomly masking several patches. The goal of the task is to train model to recon- struct the clean image from the corrupted one. Similar to MAE, DMAE also intends to reconstruct the masked information; hence, it can capture relevant features of the image for downstream tasks. Furthermore, DMAE takes noisy patches as inputs and outputs denoised ones, making the learned features robust with respect additive noises. We expect that the semantics and robustness of the representation can be leamned simultancously. enabling efficient utilization of the model parameters.
design new self-supervise
task called Denoising Masked AutoEn-
Although the proposed DMAE method simple, yields significant performance improvement on downstream tasks. We pre-train DMAE ViT-Base and DMAE ViT-Large, use the encoder to initialize the Gaussian smoothed classifier, and fine-tune the parameters on ImageNet. We show that the DMAE ViT-Base model with 87M parameters, one-tenth as many as the model used in Carlini al. (2022), achieves competitive or better certified accuracy in various settings. Further- more, the DMAE ViT-Large model (304M) significantly surpasses the state-of of-the-art results in all tasks, demonstrating a single-stage model is enough to learn robust representations with proper self- supervised tasks. We also demonstrate that the pre-trained model has good transferability other datasets. We empirically show that decent improvement can be obtained when applying it to the CIFAR-10 dataset. Model checkpoints will be released in the future.
2 RELATED WORK
Szegedy et al. (2013); Biggio et al. (2013) observed that standardly trained neural networks are vul- nerable to adversarial attacks. Since then, many works have investigated how to improve the robust- ness of the trained model. One of the most successful methods is adversarial training, which adds adversarial examples to the training set to make the learned model robust against such attacks (Madry al., 2018; Zhang al., 2019). However, as the generation process of adversarial examples is pre-
2

---

Published as a conference paper at ICLR 2023
defined during training, the learned models may be defeated by stronger attacks (Athalye et al., 2018). Therefore, it is important to develop methods that can learn models with certified robustness guarantees. Previous works provide certified guarantees by bounding the certified radius layer by layer using convex relaxation methods (Wong & Kolter, 2018; Wong et al., 2018; Weng et al., 2018; Balunovic & Vechev, 2020; Zhang et al., 2021; 2022a;b). However, such algorithms are usually computationally expensive, provide loose bounds, or have scaling issues in deep and large models.
Randomized smoothing. Randomized smoothing is a scalable approach to obtaining certified ro- bustness guarantees for any neural network. The key idea of randomized smoothing is to add Gaus- sian noise in the input and to transform any model into a Gaussian smoothed classifier. As the Lipschitz constant of the smoothed classifier is bounded with respect to the &g norm, we can ana- lytically compute a certified guarantee on small l2 perturbations (Cohen et al., 2019). Follow-up works proposed different training strategies to maximize the certified radius, including ensemble approaches (Horváth et al., 2022), model calibrations (Jeong et al., 2021), ), adversarial training for smoothed models (Salman et al., 2019) and refined training objectives (Jeong & Shin, 2020; Zhai et al., 2021). Yang et al. (2020); Blum et al. (2020); Kumar et al. (2020) extended the method to general lp perturbations by using different shapes of noises.
Self-supervised pre-training in vision. Learning the representation of images from unlabeled data an increasingly popular direction in computer vision. Mainstream approaches can be roughly cate- gorized into two classes. One class is the contrastive learning approach which maximizes agreement between differently augmented views of an image via contrastive loss (Chen et al., 2020; He al., 2020). The other class is the generative learning approach, which randomly masks patches in an im- age and learns to generate the original one (Bao al., 2021; He et al., 2022). Several works utilized self-supervised pre-training to improve image denoising (Joshua Batson, 2019; Yaochen Xie, 2020), and recently there have been attempts to use pre-trained denoisers achieve certified robustness. The most relevant works are Salman et al. (2020); Carlini et al. (2022). Both works first leverage pre-trained denoiser to purify the input, and then use a standard classifier to make predictions. We discuss these two works and ours in depth in Sec. 3.
3 METHOD
3.1 NOTATIONS AND BASICS
Denote x E Rd as the input and y E E (.....,C) ()...(.... as the corresponding label. Denote g : Rd ) as a classifier mapping
y. For any æ. assume that an adversary can perturb by adding an adversarial noise. The goal of the defense methods is to guarantee that the prediction g(x) doesn't 't change when the perturbation is small. Randomized smoothing (Li et al., 2018; Cohen et al., 2019) a technique that provides provable defenses by constructing a smoothed classifier of the form:
g(æ) arg max cEY
P, n) =c], where 1 NN o² Ta)-
The function f is called the base classifier, which is usually parameterized by neural networks, and is Gaussian noise with noise level a. Intuitively, g(x) can be considered as ensemble clas- sifier which returns the majority vote of ff when its input is sampled from a Gaussian distribution (1 La)a) centered at æ. Cohen et al. (2019) theoretically provided the following certified robust- ness guarantee for the Gaussian smoothed classifier g.
Theorem (Cohen el al., 2019) Given I and g defined as above, assume that g classifies x cor- 2 maxy P,, [ + 7) = v²). Then for any xf satisfying
rectly, i.e. P, [z + 1)) ||x' x z112 < R, we always have g(x) = g(x'),, where
R R= R= き (P,
- + v)) -mmxx Pa[ = b'
(2)
is the cumulative distribution function of the standard Gaussian distribution.
The denoise-then-preede predict network structure. Even without knowing the label, one can still evaluate the robustness of model by checking whether it can give consistent predictions when the input is perturbed. Therefore, unlabeled data can naturally be used to improve the model's robustness (Alayrac et al., 2019; Carmon et al., 2019; Najafi al., 2019; Zhai et al., 2019). Recently,
3

---

Published as a conference paper at ICLR 2023
Goussian Noise
Noisy Image x+
Masked Noisy Image Mask (x
Reconstructed Image
Clean Image
Encoder
Decoder
Reconstruction Loss
Figure I:
Illustration of our DMAE pre-training. We first corrupt the image by adding Gaussian noise to each pixel value, and then randomly masking several patches. The encoder and decoder are trained to reconstruct the clean image from the corrupted one.
Salman et al. (2020); Carlini et al. (2022) took steps to train Gaussian smoothed classifiers with the help of unlabeled data. Both them use denoise-then-predict pipeline. In detail, the base classifier consists of three components: denoisers eeneder and ouuput Given any input æ, the classification process of f is defined as below.
Φb Denoise x + 17: denoiser)
(3)
h Encode d;; Gencoder)
(4)
y Predict(h; Qompuu)
(5)
As f takes noisy image as input (see Eq.1), 1), a denoiser with parameter Adeeoiser is first used to purify ae--1 cleaned image ét. After that, & is further encoded into contextual representation h by Oencoder and the prediction can be obtained from the output head Ouupuut Note that denoiser and fncoder can be pre-trained by self-supervised approaches. For example, one can use denoising auto-encoder (Vin- cent et al., 2008; 2010) or denoising diffusion model (Ho et al., 2020; Nichol & Dhariwal, 2021) to pre-train denoiser and leverage contrastive learning (Chen al., 2020; He al., 2020) masked im- age modelling (He et al., 2022; Xie et al., 2022) to pre-train Genadeer Especially, Carlini et al. (2022) achieved state-of- of-th-of-he-arr performance on ImageNet by applying a pre-trained denoising diffusion model as the denoiser and pre-trained BETT (Bao et al., 2021) as the encoder.
3.2 DENOISING MASKED AUTOENCODERS
In the denoise-then denoise-then-predict network structure above, the denoiser is perfect, will be robust to the Gaussian additive noise n). Then the robust accuracy of y can be as high as the standard accuracy models trained on clean images. However, the denoiser requires a huge number of parameters to obtain acceptable results (Nichol & Dhariwal, 2021), limiting the practical usage of the compositional method in real applications.
Note that our goal is learn representation h that captures rich semantics for classification and resists Gaussian additive noise. Using an explicit purification step before encoding is sufficient to achieve it but may not be necessity. Instead of using multiple training stages for different purposes, we aim to adopt single-stage approach to learn robust through self-supervised learning directly. In particular, we extend the standard masked autoencoder with an additional denoising task, which we call the Denoising Masked AutoEncoder (DMAE). The DMAE works as follows: an image æ is first divided into regular non-overlapping patches. Denote Mask(x) as the operation that randomly masks patches with a pre-defined masking ratio. As shown in Fig. 1, we aim to train an autoencoder that takes Mask(æ + 17) as input and reconstructs the original image:
77 Mask( +
Encoder h Decoder a
Like MAE (He et al., 2022), we adopt the asymmetric encoder-decoder design for DMAE. Both encoder and decoder stacked Transformer layers. The encoder takes noisy unmasked patches with positional encoding as inputs and generates the representation h. Then the decoder takes the representation all patches inputs (h for unmasked patches and masked token embedding for
4

---

Published as a conference paper at ICLR 2023
masked patches) and reconstructs the original image. Pixel-level mean square enror is used as the loss function. Slightly different from MAE, the loss is calculated on all patches as the model can also learn purification on the unmasked positions. During pre-training, the encoder and decoder jointly optimized from scratch, and the decoder will be removed while learning downstream tasks.
In order reconstruct the original image, the encoder and the decoder have learn semantics from the unmasked patches and remove the noise simultaneously. To enforce the encoder (but not the decoder) to learn robust semantic features, we control the capacity of the decoder by setting smaller value of the hidden dimension and depth following He et al. (2022).
Robust fine-tuning for downstream classification tasks. As the encoder of DMAE already learns robust features, we can simplify the classification process of the base classifer as
h Encode ?y: 6 encouder)
y = Predict(h:; h: Oomppur)
To avoid any confusion, we explicitly parameterize the base classifier as f(x; O Oencoder: Outpuu) Predic((Encode(::: :lencoder); ouuppu)) ). and denote F(z; eeccuder: Oounuu) as the output of the last soft- max layer of f. i.e., the probability distribution over classes. We aim maximize the certified accuracy of the corresponding smoothed classifier by optimizing Oencoder and Oompuu. where Oencoder initialized by the pre-trained DMAE model. To achieve the best performance, we use the consis- tency regularization training method developed in Jeong & Shin (2020) to learn Bencoder and ouuput The loss is defined below.
[2r y; fl encoderr: (oupuu)
E. [CrossEntropy(F(c) (F(x + 17; eenooders Oouuput): y)]
X E, LyLL (F(x; Gencoders Oourpat) (xx + m7; Pencoders Ooutput) ))
μ (F((x; Bencoders ooupuu))
where F(x;; Dencoder. oouppu)) = Em--Nio.=2L (F - 17; Rencoder, e ooupur)] is the average prediction distribution the base classifier, and X, 1 are hyperparameters. DKI(-1l-) and (-)) denote the Kullback-Leibler (KL) divergence and the entropy, respectively. The loss function contains three terms. Intuitively, the first term aims to maximize the accuracy of the base classifier with perturbed input. The second term attempts regularize F eecoders doupuut) to be consistent with different n. The last term prevents the prediction from low confidence scores. All expectations are estimated by Monte Carlo sampling.
4 EXPERIMENTS
In this section, we empirically evaluate our proposed DMAE on ImageNet and CIFAR-10 datasets. We also study the influence of different hyperparameters and training strategies on the final model performance. All experiments repeated ten times with different seeds. Average performance is reported, and details can be found in the appendix.
4.1 PRE-TRAINING SETUP
Following He al. (2022); Xie et al. (2022), we use ImageNet-Ik as the pre-training corpus which contains 1.28 million images. All images are resized to tixed resolution of 224 x 224. We utilize two vision Transformer variants as the encoder, the Base model (ViT-B) and the Large model (ViT- L) with 16 x 16 input patch size (Kolesnikov et al., 2021). The ViT-B encoder consists of 12 Transfomer blocks with embedding dimension 768, while the ViT-L encoder consists of 16 blocks with embedding dimension 1024. For both settings, the decoder uses 8 Transformer blocks with embedding dimension 512 and linear projection whose number of output channels equals the number of pixel values in patch. All the Transformer blocks have 16 attention heads. The ViT- B/ViT-L encoder have roughly 87M and 304M parameters, respectively.
For the pre-training of the two DMAE models, we set the masking ratio to 0.75 following He et al. (2022). The noise level a is set to 0.25. Random resizing and cropping are used as data augmentation to avoid overfitting. The ViT-B and ViT-L models pre- are-trained for 1100 and 1600 epochs, where the batch size is set to 4096. We use the AdamW optimizer with 31, B2 = 0.9, 0.95. and adjust
the learning rate 1.5 10-1 The weight decay factor is set to 0.05. After pre-training,
5

---

Published as a conference paper at ICLR 2023
Figure 2: Visualization. For each group, the leftmost column shows the original image. The follow- ing two corresoond to the image with Gaussian perturbation (noise level 0.25) and the masked noisy image. The last column illustrates the reconstructed image by our DMAE ViT-L model.
also visualize the model performance DMAE ViT-L in Fig. 2. From the figure, we can see that the trained model recover the masked patches and purify the noisy unmasked patches, which demonstrates its capability of accomplishing both tasks simultaneously.
4.2 FINE-TUNING FOR IMAGENET CLASSIFICATION
Setup. In the fine-tuming stage, add linear prediction head top of the encoder for classifi- cation. The ViT-B model is fine-tumed for 100 epochs, while the ViT-L is fine-tuned for 50 epochs. Both settings use AdamW with B1, B3 0.9, 0.999. The weight decay factor is
0.05. We set the base learning rate to x 10-1 for ViT-B and 1 x 10-3 for ViT-I. Following Bao et al. (2021), we use layer-wise learning rate decay (Kevin Clark & Manning, 2020) with an exponen- tial rate of 0.65 for ViT-B and 0.75 for ViT-L. We apply standard augmentation for training ViT models, including RandAug (Ekin Cubuk & Le, 2020), label smoothing (Szegedy et al., 2016), mixup (Hongyi Zhang & Lopez-Paz, 2018) and cutmix (Yun et al.. 2019). Following most previous works, we conduct experiments with different noise levels E {0.25, 0.5, 1.0]. For the consistency regularization loss terms, we set the hyperparameters À = 2.0 and μ = 0.5 for E {0.25, 0.5), and set À 2.0 and μ 0.1 for o 1.0.
Evaluation. Following previous works, we report the percentage of samples that can be certified to be robust (a.k.a certified accuracy) at radius r with pre-defined values. For a fair comparison, we use the official implementation' of CERTIFY to calculate the certified radius for any data point?, with n = 10, 000, no = 100 and a = 0.001. The result is averaged over 1,000 images uniformly selected from ImageNet validation set, following Carlini et al. (2022).
Results. We list the detailed results our model and representative baseline methods in Table 2. We also provide summarized result that contains the best performance of different methods each radius r in Table 1. It can be seen from Table 2 that our DMAE ViT-B model significantly surpasses all baselines in all settings except Carlini supervised learning. Compared with Carlini et al. (2022), our model achieves better results when 1.0 and is slightly worse when is small. We would like point out that the DMAE ViT-B model only uses 10% parameters compared to Carlini et al. (2022). which suggests our single-stage than the denoise-ther then-predict approach. Although
(2022). This clearly demonstrates the strength of self-
pre-traimet pre-training method is more marameter-efficies the diffusion model used in Carlini et al. (2022) can be applied with different noise levels, the huge number of parameters and long inference time make more difficult to deploy.
Our DMAE ViT-L. model achieves the best performance over all prior works in all settings and boosts the certified accuracy by a significant margin when a and r are large. For example, at r = 1.5, achieves 53.7% accuracy which is 15.3% better than Boosting (Horváth et al., 2022), and it surpasses Diffusion (Carlini et al., 2022) by 12.0% at r = 2.0. This observation is different from the one reported in Carlini al. (2022), where the authors found that the diffusion model coupled with an off-the-shelf BEiT only yields better performance with smaller o and r.
1ttpps: / thub. com/ locus 1 ab/ smocthing
'One may notice that randomized smoothing methods require significant number of samples (e.g.. 10a) for evaluation. Here the samples are used to calculate the certified radius accurately. A much smaller number of samples enough make predictions in practice.

---

Published as a conference paper at ICLR 2023
Certified Accuracy (%) at & radius
Method
0.0
0.5
1.0
1.5
2.0
3.0
RS (Cohen al., 2019)
67.0
49.0
SmoothAdy (Salman ct al., 2019)
63.0
54.0
0
0
0
Consistency (Jeong & Shin, 2020)
MACER (Zhai ct al., 2021)
68.0
57.0
0.25
Boosting (Horváth et al., 2022)
65.6
57.0
0
0
0
SmoothMix (Jeong al., 2021)
Difiusion+BBiT (Carlini et al., 2022)
82.8*
71.1
Ours (DMAB VYT-B)
78.1
69.6
Ours (DMAE ViT-L)
81.7
73.6
RS (Cohen al., 2019)
57.0
46.0
37.0
29.0
SmoothAdy (Salman ct al., 2019)
54.0
49.0
43.0
37.0
0
Consistency (Jcong & Shin, 2020)
55.0
50.0
44.0
34.0
0
MACER (Zhai et al., 2021)
53.0
43.0
31.0
0.5 Boosting (Horváth ct al., 2022)
57.0
52.0
44.6
38.4
0
SmoothMix (Jcong al., 2021)
55.0
50.0
43.0
38.0
Diffusion+BEiT (Carlini et al., 2022)
77.1
67.8
54.3
38.1
Ours (DMAE ViT-B)
73.0
64.8
57.9
47.8
0
Ours (DMAE ViT-L)
77.6
72.4"
64.6*
53.7*
RS (Cohen al., 2019)
44.0
38.0
33.0
26.0
19.0
12.0
SmoothAdy (Salman et al., 2019)
40.0
37.0
34.0
30.0
27.0
20.0
Consistency (Jeong & Shin, 2020)
41.0
37.0
32,0
28.0
24.0
17.0
MACER (Zhai et al., 2021)
48.0
43.0
36.0
30.0
25.0
14.0
1.0 Boosting (Horváth et al., 2022)
44.6
40.2
37.2
34.0
28.6
20.2
SmoothMix (Jeong al., 2021)
40.0
37.0
34.0
30.0
26.0
20,0
Diffusion-BEiT (Carlini et al., 2022)
60.0
50.0
42.0
35.5
29.5
13.1
Ours (DMAE ViT-B)
58.0
53.3
47.1
41.7
35,4"
22.5'
Ours (DMAR ViT-I.)
65.7
59.0
53.0
47.9"
41.5**
27.5
Table 2: Certified accuracy (top-1) of different models on ImageNet with different noise levels. denotes the best result, and denotes the second best at each radius r.
**
4.3 FINE-TUNING FOR CIFAR-10 10 CLASSIFICATION
Setup. We show the DMAE models can benefit not only ImageNet but also the CIFAR-10 classifi- cation tasks, suggesting the nice transferability of our pre-trained models. We use the DMAE VIT-B checkpoint as a showcase. As the sizes of the images ImageNet and CIFAR-10 are different, we pre-process the images CIFAR-10 to 224 x 224 match the pre-trained model. Note that the data distributions of ImageNet and CIFAR- 10 are far different. To address this significant distributional shift, we continue pre-training the DMAE model the CIFAR-10 dataset. We set the continued pre-training stage to 50 epochs, the base learning rate to x 10-5 and the batch size to 512. Most of the fine-tuning details the same as that on ImageNet in Sec. 4.2, except that batch size of 256, apply only the random horizontal flipping as data augmentation, and reduce the number the fine-tuning epochs 50.
smaller
The evaluation protocol is the same as that on ImageNet in Sec. 4.2. We draw n: = 100, 000 noise samples and report results averaged over the entire CIFAR- 10 test set. The results are pre- sented in Table 3. Without continued pre-training, our DMAE ViT-B model still yields comparable performance with Carlini et (2022), and the model outperforms when continued pre-training applied. It is worth noting that the number of parameters of Carlini et al. (2022) is larger, and the diffusion model trained on CIFAR datasets. In comparison, our model only uses a smaller amount of parameters, and the pre-trained checkpoint is directly borrowed from Sec. 4.1. Our model per- formance is significantly better than the original consistent regularization method (Jeong & Shin, 2020), demonstrating the transferability of the pre- trainiinnng model. Specifically, performs the original consistent regularization by 12.0% at = 0.25, and by 9.0% at r = 0.5. We believe pre-trained checkpoint can also improve other baseline methods achieve better results.
Result.
method out-

---

Published as a conference paper at ICLR 2023
Certified Accuracy($$ (%) at & radius
Method
Params
Extra data
0.25
0.5
0.75
1.0
RS (Cohen et al., 2019)
1.7M
X
61.0
43.0
32.0
23.0
SmoothAdv (Salman et al., 2019)
1.7M
74.8
60.8
47.0
37.8
Consistency (Jeong & Shin, 2020)
1.7M
X
68.8
58.1
48.5
37.8
MACER (Zhai al., 2021)
1.7M
X
71.0
59.0
46.0
38.0"
X
Boosting (Horváth et al., 2022)
17M
70.6
60.4
52.4*
38,8
SmoothMix (Jcong et al., 2021)
1.7M
X
67.9
57.9
47.7
37.2
Diffusion (Carlini et al., 2022)
+137M
V
79.3*
65.5*
48.7
35.5
Ours (DMAE VIT-B)
87M
79.2
64.6
47.3
36.1
V
+continued pre-training
87M
80.8
67.1**
49.7"
37.7
Table 3: Certified accuracy (top-1) of different models on CIFAR-10. Each entry lists the certified accuracy of best Gaussian noise level from the original papers. denotes the best result and denotes the second best at each & radius. +(Carlini et al., 2022) uses a 50M-parameter diffusion model and 87M-parameter ViT-B model.
Certified Accuracy(% at lg radius
Method
0.0
0.5
1.0
1.5
2.0
3.0
MAE
32.5
13.3
0
0
0.25
DMAE
58.7(426.2))
45.3(4+32.0)
0
0
MAE
13.3
6.9
2.6
0.1
0
0
0.5
DMAE
26.4(411.11
(6.7(49.8)
10.6
5.1 (-5.0)
0
МАE
3.0
2.2
1.2
0.5
0.4
1.0
DMAE
(-66)).6)
5-4(23.2)
3.5 (-33)
2.4 (-49)1.)
1.6 (.6(21.2)
(.4(4004)
Table 4: DMAE v.s. MAE by linear probing on ImageNet. Our proposed DMAE is signifi- cantly better than MAE the ImageNet classification task, indicating that the proposed pre-training method is effective and learns more robust features. Numbers in (.) the gap between the two meth- ods in the same setting.
4.4 DISCUSSION
In this section, we discuss several design choices our methods.
Whether DMAE learns more robust features than MAE. Compared with MAE, additionally use a denoising objective in pre-training to learn robust features. Therefore, we need to examine the quality of the representation learned by DMAE and MAE to investigate whether the proposed objective helps. For a fair comparison, we compare our DMAE ViT-B model with the MAE ViT-B checkpoint released by He et al. (2022) in the linear probing setting ImageNet. Linear probing is popular scheme to compare the representation learned by different models, where we freeze the parameters of the pre-trained encoders and use linear layer with batch normalization make predictions. For both DMAE and MAF, we train the linear layer for 90 epochs with a base learning rate of 0.1. The weight decay factor set to 0.0. As overfitting seldom occurs in linear probing, we only apply random resizing and cropping data augmentation and use a large batch size of 16,384.
As shown Table 4, our DMAE outperforms MAE by a large margin linear probing. For ex- ample, with Gaussian noise magnitude
0.25, DMAE can achieve 45.3% certified accuracy at 0.5, 32.0 points higher than that of MAE. Note that even our models were pre-trained with small magnitude of Gaussian noise (a 0.25), they still yield much better results than that of MAE under large Gaussian noise (a = 0.5, 1.0). This clearly indicates that our method learns much more robust features compared with MAE.
Effects of the pre-training steps. Many previous works observe that longer pre-trate pre-training steps usu- ally helps the model perform better downstream tasks. To investigate whether this phenomenon
8

---

Published as a conference paper at ICLR 2023
Certified Accuracy(S5) (%) at l2 radius
Epochs
0.0
0.5
1.0
1.5
2.0
3.0
700
78.8
68.8
0.25
1100
78. (-0.7)
(9.6(+0.8)
700
72.0
64.4
55.5
45.3
0
0.5
1100
(3.0
64.8
(7.9(+
(7.8(+2.5)
700
56.4
50.7
46.2
40.4
34.9
23.3
1.0
1100
(8.0(+
(+2.6))
47.1
47.1 (+0.9)
(+1.3)
35.4(+0.5)
22.8(-0.8)
Table 5: Effects of the pre-training steps. From the table, we can see that the 1 1100-epoch model consistently outperforms the 700-epoch model almost all settings, demonstrating that longer pre- training leads to better downstream task performance. Numbers in (.) is the gap between the two methods the same setting.
Certified Accuracy(%) radius
Method
0.0
0.5
1.0
1.5
2.0
3.0
DMAE-L+RS
81.8
72.6
0
0
0.25
DMAE-L+CR
81.7(-0.1)
73.6
0
0
DMAE-L+RS
76.4
69.5
59.3
45.8
0
0
0.5
DMAE-L+CR
17.6(
(2.9(1
(4.6(
(3.7(47.9)
0
0
DMAE-L+RS
63.2
57.0
49.8
42.9
35.9
21.8 (7.5
1.0
DMAE-LACR
65.7 65.7(4
59.0(42.0)
53.0(43.2)
47.9
(7.9(+5.0)
41.5
Table 6: DMAE with different fine-tuning methods. From the table, we can that our pre- trained model is compatible with different fine-tuning methods. Numbers in (.) is the gap between the two methods in the same setting.
happens in our setting, we also conduct experiments to study the downstream performance of model checkpoints at different pre-training steps. In particular, we compare the DMAE ViT-B model (1 100 epochs) trained in Sec. 4.1 with an early checkpoint (700 epochs). Both models are fine-tuned un- der the same configuration. All results on ImageNet are presented in Table 5. It shows that the 1 1100-epoch model consistently outperforms its 700-epoch counterpart in almost all settings.
Other fine-tuning methods. In the main experiment, we use Consistency Regularization (CR) in the fine-tuning stage, and one may be interested in how much the pre-trained model improve with other methods. To study this, we fine-tune our pre-trained DMAE ViT-L model with the RS algorithm (Cohen et al., 2019), where the only loss used in training is the standard cross-entropy classification loss in Eq.7 For this experiment, we use the same configuration as in Sec. 4.2. The results are provided in Table 6. First, we can see that the regularization loss consistently leads to better certified accuracy. In particular, it yields up to 3-5% improvement at a larger & radius (r 2 1.0). Second, can also be seen that the RS model fine-tuned on DMAE ViT-I. significantly surpasses lots of baselines on ImageNet. This suggests that our pre-trained DMAE ViT-I model may be combined with other training methods in the literature to improve their performance.
5 CONCLUSION
This paper proposes a new self-supervised method, Denoising Masked AutoEncoders (DMAE), for learning certified robust classifiers of images. DMAE corrupts each image by adding Gaussian noises to each pixel value and randomly masking several patches. A vision Transformer is then trained to reconstruct the original image from the corrupted one. The pre-trained encoder of DMAE can naturally be used as the base classifier in Gaussian smoothed models to achieve certified ro- bustness. Extensive experiments show that the pre-trained model is parameter-efficiee oftt--of-the-ar performance, and has nice transferability. We believe that the pre- mre-triined model has great potential in many aspects. We plan to apply the pre-trained model to more tasks, including image segmentation and detection, and investigate the interpretability of the models in the future.
achieves

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work is partially supported by the National Key R&D Program of China (2022Z00160304) The work is supported by National Science Foundation of China (NSFC62276005), The Ma- jor Key Project of PCL (PCL2021A12). Exploratory Research Project of Zhejiang Lab (No. 2022RC0AN02), and Project 2020BD006 supported by PKUBaidu Fund. We thank all the anony- mous reviewers for the very careful and detailed reviews as well as the valuable suggestions. Their help has further enhanced our work.
REFERENCES
Jean-Baptist Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli. Are labels required for improving adversarial robustness? Advances Neural Information Processing Systems, 32, 2019.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventingg defenses to adversarial examples. CoRR, abs/1802.00420, 2018. URL
http://arxiv org/ 802 00420.
Mislav Balunovic and Martin Vechev. Adversarial training and provable defenses: Bridging the gap- In International Conference on Learning Representations, 2020.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations, 2021.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Smdić, Pavel Laskov, Gior- gio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. Joint European conference on machine learning and knowledge discovery in databases, pp. 387-402. Springer, 2013.
Avrim Blum, Travis Dick, Naren Manoj, and Hongyang Zhang. Random smoothing might be unable to certify foo robustness for high-dimensional images. arXiv preprint arXiv:2002.03517, 2020.
Nicholas Carlini, Florian Tramer, Zico Kolter, et al. (certified!!) adversarial robustness for free! arXiv preprint arXiv:2206. 2000222066....5 10550, 2022.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. Advances in Neural Information Processing Systems, 32, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning. pp. 1597-1607. PMLR, 2020.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, pp. 1310-1320. PMLR, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. North American Chapter of the Associa- fion for Computational Linguistics, 2019.
Jonathon Shlens Ekin D Cubuk, Barret Zoph and Quoc Le. Randaugment automated data augmentation with reduced search space. CVPR workshops, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He. Haoqi Fan, Yuxin WU, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 9729-9738, 2020.
10

---

Published as a conference paper at ICLR 2023
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Yann N Dauphin Hongyi Zhang, Moustapha Cisse and David Lopez-Paz. Mixup: Beyond empirical risk minimization. ICLR, 2018
Miklós Z Horváth, Mark Niklas Müller, Marc Fischer, and Martin Vechev. Boosting randomized smoothing with variance reduced classifiers. ICLR, 2022.
Jongheon Jeong and Jinwoo Shin. Consistency regularization for certified robustness of smoothed classifiers. Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, viriual, 2020.
Jongheon Jeong, Sejun Park, Minkyu Kim, Heung-Chame Lee, Do-Guk Kim, and Jinwoo Shin. Smoothmix: Training seffidene--caiiraad smoothed classifiers for certified robustness. Ad- vances in Neural Information Processing Systems, 34:30153-30166 2021.
Loic Royer Joshua Batson. Noise2self: Blind denoising by self-supervision networks with 1-inf-dist neurons. International conference on machine learning, 2019.
Quoc V LLe Kevin Clark, Minh-Thang Luong and Christopher D Manning, Electra: Pre-training text encoders as discriminators rather than generators. ICLR, 2020.
Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lueas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Un- terthiner, and Xiaohua Zhai. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Aounon Kumar, Alexander Levine, Tom Goldstein, and Soheil Feizi. Curse of dimensionality on randomized smoothing for certifiable robustness. arXiv preprint rrXiv:2002.032.9, 2020.
Bai Li, Changyou Chen, Wenlin Wang, and Lawrence Carin. Second-order adversarial attack and certifiable robustness. 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. ICLR, 2018.
Amir Najafi, Shin-ichi Maeda, Masanori Koyama, and Takeru Miyato. Robustness to adversar- ial perturbations in learning from incomplete data. Advances in Neural Information Processing Systems, 32, 2019.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171, PMIR, 2021.
Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. Ad- vances Neural Information Processing Systems, 32, 2019.
Hadi Salman, Mingjie Sun, Greg Yang. Ashish Kapoor, and J Zico Kolter. Denoised smoothing: A provable defense for pretrained classifiers. Advances Neural Information Processing Systems, 33:21945-2199 219557, 2020.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and search. nature, 529(7587):4844888, 2016.
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 1312,6/99 2013.

---

Published as a conference paper at ICLR 2023
Christian Szegedy, Vincent Vanhoucke, Sergey loffe. Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. CVPR, 2016.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoing Manzagol, and
Léon Bottou, Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research, 11 11(1)), 2010.
Lily Weng. Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsich, Luca Daniel, Duane Boning, and Inderjit Dhillon. Towards fast computation of certified robustness for ReLU networks. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5276-5285, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, pp. 5286-5295, PMLR, 2018.
Eric Wong, Frank Schmidt, Jan Hendrik Metzen, and J. Zico Kolter. Scaling provable adversarial defenses. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 8400-8409, Curran Asso- ciates, Inc., 2018
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9653-9663, 2022.
Greg Yang, Tony Duan, Edward Hu, Hadi Salman, Ilya Razenshteyn, and Jerry Li. Randomized smoothing of all shapes and sizes. ICML, 2020.
Shuiwang Ji Yaochen Xie, Zhengyang Wang. Noise2same: Optimizing a self-supervised bound for image denoising. Advances in Neural Information Processing Systems, 2020.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Interna- tional Conference on Computer Vision (ICCV), 2019.
Runtian Zhai, Tianle Cai, Di He, Chen Dan, Kun He, John Hopcroft, and Liwei Wang. Adversarially robust generalization just requires more unlabeled data. arXiv preprini arXiv: 2019.
Runtian Zhai, Chen Dan, Di He, Huan Zhang, Boqing Gong, Pradeep Ravikumar, Cho-Jui Hsieh, and Liwei Wang. Macer: Attack-free and scalable robust training via maximizing certified radius. ICLR, 2021.
Bohang Zhang, Tianle Cai, Zhou Lu, Di He, and Liwei Wang. Towards certifying l-infinity robust- ness using neural networks with l-inf-dist neurons. International conference on machine learning, 2021.
Bohang Zhang, Du Jiang, Di He, and Liwei Wang. Boosting the certified robustness of I-infinity distance nets. ICLR, 2022a.
Bohang Zhang, Du Jiang, Di He, and Liwei Wang. Rethinking lipschitz neural networks and certi- fied robustness: A boolean function perspective. In Advances in Neural Information Processing Systems, 2022b.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp. 7472 7482, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL /ttp:://peeet / /ttpe//eocceeee.ssee/epp/.css/se mlr, press/
v97 v97/zhang1 9p. html.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
We present the full settings of pre-training and fine-tuning in Table 7 and Table 8, respectively.
config
value
optimizer
AdamW
base learning rate
1.5e-4
weight decay
0.05
optimizer momentum
31. $2= 0.9, 0.95 4096
batch size
learning rate schedule
cosine decay
warmup epochs
40
augmentation
RandomResizedCrop
gaussian noise
0.25
Table 7: Robust pre-train setting.
config
value
optimizer
AdamW
base learning rate
5e-4 (B), le-3 (L)
weight decay
0.05
optimizer momentum
$1. B2 = 0.9, 0.999 0.65 (B), 0.75 (L)
layer-wise decay
batch size
1024
learning rate schedule
cosine decay
warmup epochs
5
training epochs
100 (B), 50 (L)
augmentation
RandAug (9, 0.5)
label smoothing
0.1
mixup
0.8
cutmix
1.0
drop path gaussian noise
0.1 (B), (L)
a (0.25, 0.0)
== 2, 0.5 E (0.25,0.5) À = 2, 0.1 (a = 1.0)
consistency regularization
Table 8: Fine-tuning setting.
B EVALUATION PROTOCOL
In this section, we describe the details of how estimate the approximate certified radius and certified accuracy.
Recall that in Theorem 1, the certified radius R for datapoint be expressed as,
(Prl(( n) = v)) Φ (max Pa + n) = -y())]
y'Fy
For computational convenience, one can obtain a bit loose, but simpler bound of certified radius by using a upper bound (f + n) vl 2 max (n n) =
R 2
x (P,'(P,f( = - --- p,(f(x v))) = - - (.,,(((P(f(tt))) = v)) =: R.
2
13

---

Published as a conference paper at ICLR 2023
Certified Accuracy(%) & radius
Method
0.0
0.5
1.0
1.5
2.0
3.0
MAE-B+RS
72.2
61.8
51.5
37.7
0
DMAE-B+RS
72.6(+0.4)
63.0
53.2
39.7
0.5
MAE-B+ B+CR
72.0
64.1
55,9
46.2
0
DMAE-B+CR
(..0)
(4.8(+0.7)
(7.9
(41.6)
Table 9: DMAE v.s. MAE by Fine-tuning ImageNet. Our proposed DMAE is consistently bet- ter than MAE on the ImageNet classification task, indicating that the proposed pre-training method is effective and learns more robust features. Numbers in (-) is the gap between the two methods in the same setting.
Certified Accuracy(%) at l radius r
Finetune a
Evaluate o
0.0
0.25
0.5
0.75
1.0
0.25
90.4
81.2
65.3
44.2
0
0.25
0.5
44.5
35.4
26.8
18.8
11.3
0.25
72.2
65.2
54.1
41.6
0.5
0.5
74.5
66.5
57.3
46.3
34.7
0.25
90.2
80.9
66.1
45.2
[0, 0.75)
0.5
75.6
67.4
58.5
45.7
35.5
Table 10: Certified accuracy of DMAE for different evaluation perturbations on CIFAR-10. [0, 0.75) means training our model with uniformly selectly from [0, 0.75). The results show one model is sufficient to tackle all evaluation settings.
We use the official implementation' of CERTIFY to calculate the certified radius for any data point. For one data point æ, the smoothed classifier samples no 100 points from the noisy Gaussian distribution N Vax, o² Ia) and then votes the predicted class, while we draw n 10, 000 samples (for ImageNet) to estimate the lower bound of [x + 7) = y] and certify the robustness. Following previous works, we report the percentage of samples that can be certified be robust (a. certified accuracy) at radius r with pre-defined values.
C SUPPLEMENTARY EXPERIMENTS
In this section, we report the results of several supplementary experiments.
More comparison with MAE. For fair comparison, we compare our DMAE VIT-B with MAE VTT-B in the fine-tuning setting ImageNet, in addition to the linear probing. The checkpoints are fine-tuned by the RS and CR method described 4.4. Table 9, we can see that DMAE outperforms MAE all radii, which indicates the effectiveness of the denoising pre-training task.
Fine-tuning with various levels of noise. the previous experiments, all the models are fine-tuned and evaluated under specific level of Gaussian noise. One may wonder whether single model can perform well under various levels of noise. To investigate this, we' ve conducted a tiny experiment on the CIFAR- 10 dataset (we draw fewer noise samples 1000 images) and reported the results in Table 10. Specifically, we sample the noise scale a from a uniform distribution over interval (a [0, 0.75) during fine-tuning, resulting in robust classifier under different magnitudes of adversarial perturbations. And we compute the certified radius with this single model. The evaluation results show that it even outperforms the original model trained with a fixed noise scale when r = 1.0, which suggests that we can indeed use one DMAE across different settings without retraining.
10, 000 and report results averaged over
14


Reviews:
[{"review_id": "jrigT2DPxd1", "reviewer": "Reviewer_cA4Y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a variant of masked autoencoders (MAE), which is specifically designed for certified robustness tasks. Specifically, they add Gaussian noise in the pretaining process and use the consistency regularization method for finetuning. With a much smaller computational complexity, they obtain better results than the previous best results obtained by diffusion models.", "summary_of_the_review": "This paper proposes a simple yet effective self-supervised pretraining framework based on MAE. Although the key techniques are not new, I still would like to accept this paper because they are the first to come up with these ideas and make them work for certified robustness on large-scale datasets.", "strength_and_weaknesses": "Strengths:\n1. Writing is easy to follow.\n2. The method is simple yet effective.\n3. Experiments and ablations are solid to support their arguments.\n\nWeaknesses:\n1. The method is highly established on MAE and previously proposed CR loss. The noise in pertaining can also be viewed as Gaussian noise data augmentation. There are not so many novel points.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "imOc7cY_Yme", "reviewer": "Reviewer_E8Qx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed Denoising Masked Autoencoders (DMAE). DMAE added denoising training objective to MAE, and shows superior performance against previous works on $\\ell_2$ certified robustness tasks.", "summary_of_the_review": "This paper proposed a simple and effective method to train certifiably robust classifiers. Despite the simplicity of the proposed method, it outperforms previous works. More principled explanations of why the method works and more ablation studies are desired. ", "strength_and_weaknesses": "Strength: The proposed method is simple and effective. \n1) It seems to be a natural fit for randomized smoothing, and I agree with the authors that there should be a more compact architecture than \"denoise + predict\". \n2) The results on $\\ell_2$ certified robustness surpasses previous works on ImageNet.\n\nWeaknesses:\n1) Explanation of why the proposed method achieves such good performance is desired.\nI am not sure if there is a more principled explanation on why a more compact architecture is sufficient to learn robust classifiers than than \"denoise + predict\". The explanation could be from a theory or experimental perspective. For instance, in Figure 2, the visualization of the reconstructed images seems to have some artifacts (the grid pattern). If one feeds the reconstructed image to a pretrained BEiT, I suspect the classification performance is not going to be good. However, it seems that directly using the encodings that learned from the encoder (the proposed method) gives good performance. Are there more principled explanations for that?\n2) More ablation studies are needed.\n-  In table 2, the proposed method needs to be retrained for different noise level. However, Carlini 2022 uses one model across all different evaluation settings. It'd be good to see the certified acc of DMAE for one model trained with one noise level but evaluated on different perturbation radii. \n-  Inference time is a key consideration in randomized smoothing (RS) since the large number of samples needed in RS. It'd be good to provide some quantitative evaluations for the inference time comparing with methods that use pretrained diffusion models. Comparing with standard diffusion denoising process, Carlini 2022 uses one-shot denoising instead of iterative denoising, which improves the inference speed. This paper directly uses a ViT based architecture, I am wondering if the inference time will be faster. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "cO8X2lLtMo0", "reviewer": "Reviewer_xxcH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed Denoising Masked AutoEncoders (DMAE), a vision-transformer based neural network model, and showed that the certified robustness using randomized smoothing can be either comparable or better than state-of-the-art denoised randomized smoothing methods, such as Carlini et al., 2022.", "summary_of_the_review": "There are two major issues that I perceived and prevented me from giving a better recommendation.\n\n1. Limited applicability: The DMAE appears to be only applicable to vision transformer-based encoder-decoder and not directly extendible to other neural network models. On the other hand, denoised randomized smoothing applies to any pre-trained classifier and can certify different architectures. My one-sentence summary of this work's major contribution would be \"If one applies randomized smoothing on DMAE, it will get better or comparable certified accuracy than other (general-purpose) denoised smoothed models\". However, this argument ignores the fact that denoised smoothing applies to any arbitrary classifier while the proposed model does not. \n\n2. Unfair evaluation: Continue on my point #1, given that there is no fixed classifier in the proposed DMAE setting, I don't think comparing certified accuracy or model size makes any sense because it boils down to simply comparing the certified robustness a model (DAME) to different models (e.g., a fine-tuned vision transformer classifier). The results are not compared in a common setting and I don't see any new insights other than DAME shows good certified performance. If the goal is to show DAME is easier to certify, then more discussion on why this architecture is preferable to certification is required. But this aspect is lacking in the current manuscript. Overall, I would suggest the authors put some deep thoughts into making the comparison meaningful (e.g., trying to certify the same \"classifier\" while allowing the denoiser to vary), or making a deeper analysis of the source of better-certified robustness from the proposed model (again, proper baseline models/architectures would be needed here).", "strength_and_weaknesses": "Strength: Paper is easy to follow. \n\nWeakness: The framework does not seem to be easily extendible to any pre-trained classifier, while in general denoised randomized smoothing methods can apply to any arbitrary classifier. Therefore, the scope is limited and the comparison is not entirely fair.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WFlCaX9zPz6", "reviewer": "Reviewer_Gj4g", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors study the provably/certifiable robust classification problem. Specifically, the authors propose a self-supervised framework called Denoising Masked Auto-encoders to learn robust representations (or to pre-train encoders) by reconstructing images from noisy and masked inputs. The authors further fine-tune the model with the consistency regularization technique to achieve optimal robust classification for the model.", "summary_of_the_review": "The paper studies an interesting problem and is overall easy to follow. Experiments done in the work can indicate the effectiveness of DMAE to some degree. However, there are some concerns about the novelty, significance, and insufficient evaluation of the work. I believe the paper would be a good one if the authors can show its capability in more tasks with noisy input (that would be a lot of empirical contribution), but the current form of this paper may not be good enough.", "strength_and_weaknesses": "### Strengths\n\n(+) Results on existing tasks are good, showing the effectiveness of the proposed method\n\n(+) The work is well-motivated and the paper is easy to follow.\n\n(+) In addition to the certified accuracy evaluation, the authors include empirical studies to justify that DMAE learns more robust features than MAE.\n\n### Weaknesses\n\n(-) (major) The significance of the work may be limited and it is not well-positioned against existing works (i.e., MAE and Randomized Smoothing). The authors formulate their problem as to study the “robust vision learner” and propose the representation learning (with pre-training & fine-tuning) paradigm to learn robust encoder. The pre-trained encoder should be general-purposed and be evaluated in multiple different downstream tasks, e.g., the robust segmentation on noisy inputs, following the MAE work. However, the current evaluation is only performed on the image classification problem with only 2 datasets. This could limit the significance of the method as a “vision learner“.\n\n(-) (major) The proposed framework that adds noise to the training image and performs the masked reconstruction is straightforward given the desire of learning noise/attack robust encoders. Technically, it seems to be incremental to the MAE work and adapts MAE to the certified robust problem setting. Other techniques used in the paper such as consistency regularization are also from existing work. This may limit the novelty of this work.\n\n(-) (major) The proposed method is not sufficiently justified. Even focusing on the robust classification problem, the experiments may still not be sufficient to show the generalizability of the proposed approach. Current experiments only consider the Gaussian additive noise in both training and evaluation. The certified radius evaluation can cover those cases by selecting a large enough radius, but cannot fully justify or compare the model behavior under certain noise/attack types. In real scenarios, the noise/attack might be more diverse and complicated, e.g., Poisson, impulse, combined, or even dedicated to attacking certain models. It would be better to include more evaluation on some of those cases.\n\n(-) (minor) Does the robust fine-tuning (with regularizations) also be applied to baseline methods so that the comparison is fair? If not, it would be better to make comparisons consistently with/without the robust fine-tuning in order to show that the improvement comes from the DMAE pre-training framework.\n\n(-) (minor) To be more self-contained, it is better to include a better formulation of the certified robust classification problem and the evaluation protocol with the certified radius.\n\n(-) (minor) I also suggest the author discuss the self-supervised (blind-spot) image denoising paper [1, 2] since they are relevant and the frameworks are actually similar. The only difference is that the denoising approaches take the final images as outcomes and this work takes the representation or pre-trained encoder as outcomes. In particular, when considering the masked&noise reconstruction framework and the robust fine-tuning term (KL divergence between two outputs), the framework is very close in its looking to the objective in [2]. It would be better to discuss any connections and highlight differences between those works.\n\n[1] Batson et al. Noise2Self: Blind Denoising by Self-Supervision. ICML 2019.\n\n[2] Xie et al. Noise2Same: Optimizing A Self-Supervised Bound for Image Denoising. NeurIPS 2020.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster