Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary:

This paper introduces a new smoothing algorithm which produces classifiers with certified robustness against adversarial perturbations. In particular, the authors are interested in settings where the input vectors need not be robust simply to \ell_2 perturbations, for which previous authors (e.g., Cohen et al.) have already introduced a "randomized smoothing" classifier which possesses the desired properties. Instead, motivated by the desire to certify robustness to other families of transformations, the authors propose to introduce a smoothing transform that operates in an additive way on the underlying parameter space (rather than on the raw input vectors, as in the case of \ell_2 randomized smoothing). The authors provide details for a few examples from image and audio data processing, and then show experimental results on ImageNet data.

Comments:

My assessment of this paper is that the level of novelty is relatively low. On the technical side, I do not believe this paper makes any significant contributions. The authors themselves note that the proof of the main result, Theorem 3.2, closely parallels the proof of Cohen et al. Algorithmically, the proposals of the authors are also minor tweaks of the algorithms in Cohen et al., since the types of perturbations considered in this paper are essentially single-parameter problems. (As a question: The authors prove Theorem 3.2 where the randomized smoothing operation adds possible nonisotropic Gaussian noise in the parameter space. However, it seems that in the examples, the noise added is isotropic -- it is not clear how the authors are choosing \sigma_i in relation to \sigma_r for the interpolation example. Can the authors clarify this, and give examples where smoothing with nonisotropic noise would actually be useful?)

The only real tweak that the authors introduce when presenting their smoothing algorithm is to introduce a small correction term in order to deal with interpolation/rounding errors. However, I do not think that this change is particularly novel.

As a final comment, I think it would have been interesting for the authors to include some simulations with audio data as well, since this constitutes one of the main motivating applications discussed by the authors.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors generalize the randomized smoothing type of robustness certification to handle many types of attacks beyond norm-based attacks, e.g., geometric perturbation, volume change, pitch shifts on audio data. At the core of the proposed generalization is using some interpolation which I think is quite straightforward.

1. How about the scenario when both rotation and translation exist?

2. In experiments, can the authors consider the accuracy of the certified model under the adversarial setting, i.e., find the transformation that makes the deep nets performs worst?

3. Besides randomized smoothing on the input images, recently Wang et al showed that randomize the deep nets can
also improve the deep nets and they gave it a nice theoretical interpretation. Here is the reference: Bao Wang, Binjie Yuan, Zuoqiang Shi, Stanley J. Osher. ResNets Ensemble via the Feynman-Kac Formalism to Improve Natural and Robust Accuracies, arXiv:1811.10745, NeurIPS, 2019

4. Interpolation idea has been used in improving robustness of deep nets, for example:
1). Bao Wang, Xiyang Luo, Zhen Li, Wei Zhu, Zuoqiang Shi, Stanley J. Osher. Deep Neural Nets with Interpolating Function as Output Activation, NeurIPS, 2018 
2). Bao Wang, Alex T. Lin, Zuoqiang Shi, Wei Zhu, Penghang Yin, Andrea L. Bertozzi, Stanley J. Osher. Adversarial Defense via Data Dependent Activation Function and Total Variation Minimization, arXiv:1809.08516, 2018 
3). B. Wang, S. Osher. Graph Interpolating Activation Improves Both Natural and Robust Accuracies in Data-Efficient Deep Learning, arXiv:1907.06800

In sum, this paper studies the problem of certifying a broad class of adversarial attacks which is decent, however, the novelty is quite limited. Please address my questions during rebuttal.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper applied the framework of randomized smoothing classifier proposed by Cohen et al. to certify the adversarial attacks other than pixel change, including image rotating, brightness change in RGB space, Volume and pitch shifts for audio.

Certifying general adversarial attack is an interesting direction. However, I do not believe this paper is qualified for publishing in ICLR. Below please find my comments:

Different from change-pixel attack, the certification for rotation/brightness change in image classification, volume and pitch change in audio perturbations is much easier. The studied perturbation can be parameterized by a very low dimension vector (i.e. one dimension (angle and volume) for image rotation and volume change, 3 dimensions (brightness for each channel) for brightness perturbation). To certify this attack, given a base classifier, we can simply do, for example, grid search, on the low dimensional space to find the worst case with very good accuracy. And it should be able to give much better than the randomized smoothing method.

One contribution of this paper is in that: Theorem 3.2 is valid for random smoothing using gaussian with general covariance matrix. However, the effectiveness of using a general covariance matrix is not studied. In the experiment section, I find only isotropic ones are used.
review_title: Official Blind Review #4

Meta-review: This paper proposes a smoothing-based certification against various forms of transformations, such as  rotations, translations. The reviewers have concerns on the novelty of the work and several technical issues. The authors have made efforts to address some of issues, but the work may still significantly benefit from a throughout improvement in both presentation and technical contribution.
Decision: Reject