Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: In this paper, the authors propose a novel method for defending against
adversarial attacks on deep neural networks in a black-box setting. The idea is
simple as it is effective: to simply add some small Gaussian noise to the input
prior to passing it through the model. The authors make some heuristic arguments
for why this might be effective, including the difficulty of estimating
gradients (for gradient-based attacks) and robustness against local-search based
attacks that do not account for variability in the model output. They also
present a set of experiments demonstrating the superiority of their defense
against a variety of common attack techniques, and contrast the performance of
Small Noise Defense (SND) against other common defense techniques.  The
experiments demonstrate that SND maintains a higher clean accuracy (i.e.,
accuracy on unperturbed, non-adversarial images) compared to other defense
methods, while also staying robust to a variety of attack techniques.

Strengths of the paper:
  + The performance of the approach, given its simplicity (as the authors
    point out, it only requires one line of code!), is striking and makes
    SND an attractive option. It will likely be widely adopted given its ease
    of implementation, in situations where protection against adversarial
    attacks is critical.
  + The experimental evaluation was rigorous and supports the paper's main
    claims.
  + The paper is well organized, well written and easy to follow.

Areas for improvement/questions for the authors:
  - The theoretical analysis of the SND model, given its simplicity, is somewhat
    lacking. The authors themselves acknowledge this in their conclusion, and
    cite this as a promising future avenue to explore. Currently, the paper's
    arguments are mostly empirical.
  - Given the similarity to Random Smoothing (Sec. 5), I was curious as to why
    that method was not included in the experimental evaluation. Is this because
    RS uses large variance noise, and is thus unlikely to maintain a sufficient
    clean accuracy? Some clarification here would be welcome.

On balance, I think this paper presents some interesting results, and it's rare
to see such simple ideas have such startling payoffs. That alone makes me
positive about this paper. While the theoretical work is somewhat light, I don't
see that as fatal -- I think this paper as is opens up some interesting avenues
for future work, that could strengthen this contribution even further. I thus
vote for ACCEPTANCE.

review_title: A simple yet effective idea for defending against adversarial query-based black-box attacks on deep neural networks.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a new defense against adversarial examples created through query based attacks. The defense is based on adding a small amount of random noise. The authors provide some theoretical arguments and experimental evaluation of their approach.

Overall evaluation
While the presented idea might be promising neither the theoretical nor the experimental evaluation are sufficient to validate this claim. The research on adversarial robustness is very active and many attacks and defenses have been proposed. To ensure good quality of research, proposals have been made on how to correctly evaluate a proposed defense (see below). The paper falls short of these standards. The main shortcoming is the lack of a robust evaluation against an adaptive attack (i.e. an attacker aware of the defense). The majority of the analysis in the paper is restricted to evaluating  the defense against existing attacks, robustness against an attacker that does not take the defense into account is insufficient. 
The paper provides a short analysis of an adaptive attack, however, this new proposed attack is just a slight variation of an existing attack. This type of defense against a "patched" attack is still insufficient. Especially because from the paper it is unclear why this specific attack was chosen, if it is optimal and if it addresses the shortcomings of the original attack. One clear oversight is that the objective of the attacker changes, as it is not looking for a point that gets misclassified, but for a point that gets misclassified even under noise. 
The shortcoming of the analysis goes beyond the lack described above and I recommend the authors to go through the given guidelines.

References
Carlini, Nicholas, et al. "On evaluating adversarial robustness." arXiv preprint arXiv:1902.06705 (2019).
Athalye, Anish, Nicholas Carlini, and David Wagner. "Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples." arXiv preprint arXiv:1802.00420 (2018).

Concrete comments about the unclarity
"the above observation implies that clean images have a relatively long distance to the decision boundary" -- the existence and prevalence of adversarial examples implies exactly the opposite of this statement, if this would be true there would not be any adversarial examples. 

Section 3.2 is very unclear. eta which is defined in Section 3.1. a independent of $x$ is suddenly dependent on $x$. The sentence "this function is very noisy" is just a claim. It is unclear why the random function becomes discontinuous for large Var. 

Section 3.3 is just saying that existing attacks don't take into account the newly proposed defense.

The analysis in Section 3.4 is for a quadratic function. Most current CNN architectures don't have any quadratic component and in fact the Resnet architecture used in the experiments is a piecewise linear function with a final softmax layer. I fail to see the relevance of the presented argument.

**After Rebuttal**

I thank the authors for their detailed remarks and clarification. While it is encouraging to see that the authors have offered some clarifications and assurances regarding the validity of their approach, my main concern is whether the edit distance between the current work and the proposed modifications is just too high. It appears to me that there simply is too much that the authors need to modify in order to obtain an acceptable manuscript (with the other reviewers' concerns and suggestions as well). I believe that the paper can be significantly improved if the authors incorporate the comments from the current round of reviewing. 
review_title: The paper proposes a new defense against adversarial examples created through query based attacks. The defense is based on adding a small amount of random noise. The authors provide some theoretical arguments and experimental evaluation of their approach.

final_score: rating: 6: Marginally above acceptance threshold
review_content: #Summary

This paper introduces a simple but effective method by adding a small Gaussian noise to the input, and shows it can neutralize query-based black-box attacks and achieve strong results on both clean accuracy and attack success rates w.r.t. several SOTA attack/defense methods.


#Pros
- The proposed approach is very simple and seems effective across several query-based black-box attacks. The authors have evaluated their SND defense against both decision-based attacks (Sign-OPT, HSJA, GeoDA) and score-based attacks (SimBA, SimBA-DCT, Bandit-TD), and the results are pretty strong in terms of  lowest attack success rates under a fixed query budget, compared with four defense methods (PGD-AT, R&P, RSE, PNI), over both CIFAR and ImageNet.
- The proposed approach achieves good clean accuracy as well, which seems to be another plus compared to existing defenses that achieve strong defense ability but at the cost of clean accuracy.

#Cons
- I think a bit more detailed analysis of how this simple approach works would further strengthen this paper. In Section 3, the authors have provided some example analysis but it would be better to see a more in-depth analysis, e.g., how this method obfuscates the expectation for more general functions in section 3.4; similarly, for section 3.3., the attack objective loss "might increase" but there is no further study provided.

- The following paper adopts a similar idea via additive Gaussian noise to the input for certified adversarial robustness, and should be cited and discussed as well: [1] Bai Li, Changyou Chen, Wenlin Wang, Lawrence Carin. Certified Adversarial Robustness with Additive Noise. NeurIPS 2019.

- As the authors have mentioned, one natural way to attack random-noise based defense is through adaptively querying the model and taking the expectation of T queries, at the cost of increased query budget. In experiments, the authors performed a study of HSJA against SND. Could the authors show more study on this, like how is the query efficiency from adaptive attacks based on other attack methods? Also, since $\sigma$ is fixed, after the first round of queries around a fixed $x$, is there a way to infer what $\sigma$ is and further improves query efficiency in later rounds?

- I think it would also be interesting to further see how this defense works under transfer-based black-box attacks. Does this defense work in general, or only for query-based black-box attacks?


#Overall recommendation

Overall I think the results are strong compared to the baselines and the method is simple and well-explained. The experiments are thorough over various SOTA attacks and defenses, on both CIFAR and ImageNet datasets.  I would vote for accepting the paper.

review_title: Interesting work on defense against query-based black-box attacks via small random noise

final_score: rating: 3: Clear rejection
review_content: This paper proposes an adversarial defense method against black-box attack by adding random noise on adversarial examples. The method is simple and empirical results verify that the proposed defense can be used to decrease the attack success rate of both optimization-based and local search-based attacks. 

My main concern of this paper is the simply use of randomness for adversarial defense, rather than directly improving the robustness of the target model itself (e.g. adversarial training). In the discussion about the Obfuscated Gradient [1], the authors criticized the randomness-based defense method: “Stochastic Gradients are caused by randomized defenses, where either the network itself is randomized or the input is randomized before being fed to the classifier.” Although this paper is aimed at defending against black-box attacks, it is still essentially a defense by adding random noise to the input data. In addition, the proposed method is not useful to defense against white-box attacks and transfer-based attacks, since basically it does not change the predictive model.

Another problem is that while the defense mechanism in this paper is simple, there are equally simple ways to circumvent it. An example is that when the randomness of model output is found by the adversary based on Boundary Attack, the attacker only needs to change the judgment condition of successful attack to “most of the attacks are successful in the same input for multiple queries” to avoid the decrease of attack success rate.

[1] https://arxiv.org/abs/1802.00420


review_title: Official Blind Review #3

Meta-review: The paper proposes a defense against black-box adversarial example attacks based on adding small Gaussian noise to the inputs. Its evaluation is carried out empirically using CIFAR-10 and ImageNet datasets.

Despite a somewhat complete experimental evaluation (on two datasets) the lack of theoretical justification strongly affects the significance of the proposed method. It can be clearly seen from the experimental results that the proposed level of noise is a trade-off between clean accuracy and attack effectiveness. However, this tradeoff neither implies a substantial degree of security (the attack success rate is roughly halved but this does imply robustness against attacks since the initial success rate is rather high) nor is  the impact to clean accuracy negligible. Furthermore, the robustness of the proposed method against an attack which is aware of such defense (similar to the Kerckhoff's principle in cryptography) is not evaluated. The authors mentioned several directions for addressing this issue in their response but implementation of such improvements is impossible within the level of revisions acceptable in a post-review process.

A major revision of the paper taking into account the feedback provided by the current reviews would certainly improve its acceptance chances.  
Decision: Reject