Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors study the provably/certifiable robust classification problem. Specifically, the authors propose a self-supervised framework called Denoising Masked Auto-encoders to learn robust representations (or to pre-train encoders) by reconstructing images from noisy and masked inputs. The authors further fine-tune the model with the consistency regularization technique to achieve optimal robust classification for the model.
strength_and_weaknesses: ### Strengths

(+) Results on existing tasks are good, showing the effectiveness of the proposed method

(+) The work is well-motivated and the paper is easy to follow.

(+) In addition to the certified accuracy evaluation, the authors include empirical studies to justify that DMAE learns more robust features than MAE.

### Weaknesses

(-) (major) The significance of the work may be limited and it is not well-positioned against existing works (i.e., MAE and Randomized Smoothing). The authors formulate their problem as to study the “robust vision learner” and propose the representation learning (with pre-training & fine-tuning) paradigm to learn robust encoder. The pre-trained encoder should be general-purposed and be evaluated in multiple different downstream tasks, e.g., the robust segmentation on noisy inputs, following the MAE work. However, the current evaluation is only performed on the image classification problem with only 2 datasets. This could limit the significance of the method as a “vision learner“.

(-) (major) The proposed framework that adds noise to the training image and performs the masked reconstruction is straightforward given the desire of learning noise/attack robust encoders. Technically, it seems to be incremental to the MAE work and adapts MAE to the certified robust problem setting. Other techniques used in the paper such as consistency regularization are also from existing work. This may limit the novelty of this work.

(-) (major) The proposed method is not sufficiently justified. Even focusing on the robust classification problem, the experiments may still not be sufficient to show the generalizability of the proposed approach. Current experiments only consider the Gaussian additive noise in both training and evaluation. The certified radius evaluation can cover those cases by selecting a large enough radius, but cannot fully justify or compare the model behavior under certain noise/attack types. In real scenarios, the noise/attack might be more diverse and complicated, e.g., Poisson, impulse, combined, or even dedicated to attacking certain models. It would be better to include more evaluation on some of those cases.

(-) (minor) Does the robust fine-tuning (with regularizations) also be applied to baseline methods so that the comparison is fair? If not, it would be better to make comparisons consistently with/without the robust fine-tuning in order to show that the improvement comes from the DMAE pre-training framework.

(-) (minor) To be more self-contained, it is better to include a better formulation of the certified robust classification problem and the evaluation protocol with the certified radius.

(-) (minor) I also suggest the author discuss the self-supervised (blind-spot) image denoising paper [1, 2] since they are relevant and the frameworks are actually similar. The only difference is that the denoising approaches take the final images as outcomes and this work takes the representation or pre-trained encoder as outcomes. In particular, when considering the masked&noise reconstruction framework and the robust fine-tuning term (KL divergence between two outputs), the framework is very close in its looking to the objective in [2]. It would be better to discuss any connections and highlight differences between those works.

[1] Batson et al. Noise2Self: Blind Denoising by Self-Supervision. ICML 2019.

[2] Xie et al. Noise2Same: Optimizing A Self-Supervised Bound for Image Denoising. NeurIPS 2020.

clarity,_quality,_novelty_and_reproducibility: Clarity and Reproducibility: The authors study an interesting and well-motivated problem. The paper is well-written and easy to follow. Sufficient implementation details are provided for reproduction.

Quality: The experiments done in the work indicate the effectiveness of DMAE to some degree. However, additional evaluations on more tasks and datasets may make the method more convincing.

Novelty: The key idea is very straightforward given the problem setting and the proposed approach seems to be an adaptation of MAE in a stricter setting. Hence the novelty is limited.
summary_of_the_review: The paper studies an interesting problem and is overall easy to follow. Experiments done in the work can indicate the effectiveness of DMAE to some degree. However, there are some concerns about the novelty, significance, and insufficient evaluation of the work. I believe the paper would be a good one if the authors can show its capability in more tasks with noisy input (that would be a lot of empirical contribution), but the current form of this paper may not be good enough.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed Denoising Masked AutoEncoders (DMAE), a vision-transformer based neural network model, and showed that the certified robustness using randomized smoothing can be either comparable or better than state-of-the-art denoised randomized smoothing methods, such as Carlini et al., 2022.
strength_and_weaknesses: Strength: Paper is easy to follow. 

Weakness: The framework does not seem to be easily extendible to any pre-trained classifier, while in general denoised randomized smoothing methods can apply to any arbitrary classifier. Therefore, the scope is limited and the comparison is not entirely fair.
clarity,_quality,_novelty_and_reproducibility: Clarity: clear presentation
Quality: Applicability and comparison to denoised randomized smoothing need more justification
Novelty: mediocre
Reproducibility: code not provided; unable to verify
summary_of_the_review: There are two major issues that I perceived and prevented me from giving a better recommendation.

1. Limited applicability: The DMAE appears to be only applicable to vision transformer-based encoder-decoder and not directly extendible to other neural network models. On the other hand, denoised randomized smoothing applies to any pre-trained classifier and can certify different architectures. My one-sentence summary of this work's major contribution would be "If one applies randomized smoothing on DMAE, it will get better or comparable certified accuracy than other (general-purpose) denoised smoothed models". However, this argument ignores the fact that denoised smoothing applies to any arbitrary classifier while the proposed model does not. 

2. Unfair evaluation: Continue on my point #1, given that there is no fixed classifier in the proposed DMAE setting, I don't think comparing certified accuracy or model size makes any sense because it boils down to simply comparing the certified robustness a model (DAME) to different models (e.g., a fine-tuned vision transformer classifier). The results are not compared in a common setting and I don't see any new insights other than DAME shows good certified performance. If the goal is to show DAME is easier to certify, then more discussion on why this architecture is preferable to certification is required. But this aspect is lacking in the current manuscript. Overall, I would suggest the authors put some deep thoughts into making the comparison meaningful (e.g., trying to certify the same "classifier" while allowing the denoiser to vary), or making a deeper analysis of the source of better-certified robustness from the proposed model (again, proper baseline models/architectures would be needed here).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed Denoising Masked Autoencoders (DMAE). DMAE added denoising training objective to MAE, and shows superior performance against previous works on $\ell_2$ certified robustness tasks.
strength_and_weaknesses: Strength: The proposed method is simple and effective. 
1) It seems to be a natural fit for randomized smoothing, and I agree with the authors that there should be a more compact architecture than "denoise + predict". 
2) The results on $\ell_2$ certified robustness surpasses previous works on ImageNet.

Weaknesses:
1) Explanation of why the proposed method achieves such good performance is desired.
I am not sure if there is a more principled explanation on why a more compact architecture is sufficient to learn robust classifiers than than "denoise + predict". The explanation could be from a theory or experimental perspective. For instance, in Figure 2, the visualization of the reconstructed images seems to have some artifacts (the grid pattern). If one feeds the reconstructed image to a pretrained BEiT, I suspect the classification performance is not going to be good. However, it seems that directly using the encodings that learned from the encoder (the proposed method) gives good performance. Are there more principled explanations for that?
2) More ablation studies are needed.
-  In table 2, the proposed method needs to be retrained for different noise level. However, Carlini 2022 uses one model across all different evaluation settings. It'd be good to see the certified acc of DMAE for one model trained with one noise level but evaluated on different perturbation radii. 
-  Inference time is a key consideration in randomized smoothing (RS) since the large number of samples needed in RS. It'd be good to provide some quantitative evaluations for the inference time comparing with methods that use pretrained diffusion models. Comparing with standard diffusion denoising process, Carlini 2022 uses one-shot denoising instead of iterative denoising, which improves the inference speed. This paper directly uses a ViT based architecture, I am wondering if the inference time will be faster. 
clarity,_quality,_novelty_and_reproducibility: The paper is mostly well written and the proposed method is clear and novel.
There is one point that could confuse people. The authors mentioned in both Intro and Method that Intro: "such a two-stage process requires much more parameters and separated training" and "Carlini et al. (2022) took steps to train Gaussian smoothed classifiers with the help of unlabeled data". This is not precise because Carlini 2022 only uses pretrained model and does not need any training for certified robustness. I understand that what the authors really want to say is that the off-the-shelf models used by Carlini 2022 need to be trained for two objectives, but the authors should make it clearer.

summary_of_the_review: This paper proposed a simple and effective method to train certifiably robust classifiers. Despite the simplicity of the proposed method, it outperforms previous works. More principled explanations of why the method works and more ablation studies are desired.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a variant of masked autoencoders (MAE), which is specifically designed for certified robustness tasks. Specifically, they add Gaussian noise in the pretaining process and use the consistency regularization method for finetuning. With a much smaller computational complexity, they obtain better results than the previous best results obtained by diffusion models.
strength_and_weaknesses: Strengths:
1. Writing is easy to follow.
2. The method is simple yet effective.
3. Experiments and ablations are solid to support their arguments.

Weaknesses:
1. The method is highly established on MAE and previously proposed CR loss. The noise in pertaining can also be viewed as Gaussian noise data augmentation. There are not so many novel points.
clarity,_quality,_novelty_and_reproducibility: This work presents their methods very clearly. The detailed training process and parameters are listed, so it should not be hard to reproduce. Evaluation of novelty has been given in 'Strength And Weaknesses'.
summary_of_the_review: This paper proposes a simple yet effective self-supervised pretraining framework based on MAE. Although the key techniques are not new, I still would like to accept this paper because they are the first to come up with these ideas and make them work for certified robustness on large-scale datasets.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper investigates the potential of Masked Autoencoder (MAE) for certified robustness. Specifically, by training MAE with Gaussian noise corrupted images, this work shows that the MAE's encoder can be used as a base classifier in Gaussian smoothed models and yields significant improvements in certified robustness. 

Overall, the reviewer enjoyed reading the paper and appreciated its simplicity and strong empirical results. However, they also had some concerns about: (1) the overclaiming of "robust vision learner", as only the image classification task is studied here; (2) the need for more ablations to justify the effectiveness and efficiency of the proposed method; (3) the comparisons are somewhat unfair; (4) the technical novelty over MAE may be limited; and (5) its applicability beyond vanilla ViTs is unclear. The rebuttal well addressed the concerns (1)-(3). Concern (4)-(5) was discussed in the AC-reviewer meeting and concluded to be minor. As a result, we recommend accepting this paper.

In the final version, the authors are strongly encouraged to elaborate on the possibility of generalizing the proposed method beyond vanilla ViT (either as a future plan or a discussion about current limitations), which can help future readers better understand this paper. In addition, the title and abstract in the OpenReview should be updated to be consistent with the pdf file.
justification_for_why_not_higher_score: We still have concerns (though minor) about its limited technical contribution and generalization beyond vanilla ViTs. Therefore, we cannot recommend it for a higher score.


justification_for_why_not_lower_score: This paper well demonstrates the potential of MAE in certified robustness, with strong empirical results. Therefore, we believe that this work is worthy of a poster presentation at ICLR.
Decision: Accept: poster