Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a regularization method for achieving robustness to noisy inputs, with relatively less computation compared to standard data augmentation approaches. Specifically, the authors analyze the analytic expression of the loss on the noisy inputs, and using Jensen’s inequality, propose to minimize a surrogate loss over the expectation of noisy inputs. To minimize the loss over the expectation, the authors impose a regularization over the first moment of the network weights. The authors validate the model with the proposed regularization technique for its robustness against Gaussian attack and other types of attacks, whose results show that the model is robust. 

Pros
- The general idea of the regularization that replaces the generation of noisy samples and optimization over it is conceptually appealing and seems practically useful.
- The derivation of the moment-based regularization makes sense. 
- The proposed regularizer seems to be effective to a certain degree, on the sets of experiments done by the authors.

Cons
- Experimental validation seems highly inadequate due to lack of baselines. Thus it is difficult to assess the degree of robustness the proposed model achieves. The authors should perform extensive evaluation against state-of-the-art techniques against multiple types of attacks, in order to demonstrate the effectiveness of the proposed method.
- While the authors emphasize the computational efficiency of the method, the authors do not report computational cost or actual runtime.  
- The types of non-Gaussian attacks should be better described. Which ones use L-infinity attacks and which use L2 attacks?
- Figure 3 doesn’t seem like a very favorable result to the proposed model, since we are generally more concerned with adversarial examples generated with small perturbations, as large perturbations may change the input semantics.

In sum, while I like the overall idea and find the work novel and potentially practical, it is difficult to properly evaluate the work due to lack of comparison against state-of-the-art data augmentation methods for achieving robustness. Therefore I temporarily give this paper a weak reject, but may change the rating with more experimental results provided in the rebuttal. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: I am not fully convinced by the robustness result in Figure 3. In MNIST, the proposed method is worse than data augmentation. In CIFAR-10, the proposed method does perform better, however, \tilde{N} for data augmentation is chosen as 2, which is too small in my opinion. The data augmentation's robustness is similar to the baseline. I don't know if there are any issues in the training but data augmentation with \tilde{N}=2 is expected to be ineffective to improve robustness. For CIFAR-100, the improvement is marginal and can only be achieved when \sigma is large.

Is the GNR score in Table 1 calculated on all examples or only on correctly classified examples? If it is calculated on all examples, I think it would be better to also report the result of the correctly classified examples. Because at the end we care how the models can correctly and robustly make the classification.

I think it would be better if the authors can have some discussion about the potential connections between the proposed method and the method in [1]. [1] proved that if a model can classify well under Gaussian noise, it is possible to turn it into a classifier that is certifiably robust (I don't think the proposed method is certifiable) to adversarial perturbations in l_2 ball. The training method in [1] is Gaussian data augmentation. An experimental comparison between the proposed method and the method in [1] is necessary. I am not sure how faster the proposed method can be. 

[1] Cohen, Jeremy M., Elan Rosenfeld, and J. Zico Kolter. "Certified adversarial robustness via randomized smoothing." ICML 2019

---------------------------
update after rebuttal:
I appreciate the authors' feedback. However, I am not convinced that the proposed method is highly effective (Table 1 and Figure 3) and I still think the contribution is kind of marginal (especially given [1]). So it is hard to recommend acceptance.


review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper proposes a regularizer to encourage the robustness to the random contamination for training deep neural networks. The idea is straightforward and intuitive, but not that exciting. The experiments show some improvement. However, I have a few serious concerns:

(1) Why do we care about the random noise, especially Gaussian noise? There has been a large amount of literature on training robust network, but they are also for the robustness to adversarial examples. The Gaussian noise is too simple and easy to defend. We can even apply a simple denoiser to preprocess the data, which does not even involve training a sophisticated neural network.

(2) The proposed moment regularizer is very delicate. I do not think it can generalize to other noises or contaminations. This is because for other noises, the moment approximation can be fairly loose.

(3) The Alexnet was proposed in 2011. Consider that it is already late 2019, the authors INDEED need to do experiments using more advanced and recent models, e.g., ResNet34/50 or even powerful ones, e.g, ResNeXt, DenseNet, Wide ResNet.
review_title: Official Blind Review #2

Meta-review: This paper received two weak and one strong reject from the reviewers.  The major issues cited were 1) a lack of strong enough baselines or empirical results, 2) Novelty with respect to "Certified adversarial robustness via randomized smoothing" and 3) a limitation to Gaussian noise perturbations.  Unfortunately, as a result the reviewers agreed that this work was not ready for acceptance.  Adding stronger empirical results and a careful treatment of related work would make this a much stronger paper for a future submission.
Decision: Reject