Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper presents a new attack: Shadow Attack, which can generate imperceptible adversarial samples. This method is based on adding regularization on total variation, color change in each channel and similar perturbation in each channel. This method is easy to follow and a lot of examples of different experiments are shown.
However, I have several questions about motivation and method.

First, the proposed attack method can yield adversarial perturbations to images that are large in the \ell_p norm. Therefore, the authors claim that the method can attack certified systems. However, attack in Wasserstein distance and some other methods can also do so. They can generate adversarial examples whose \ell_p norm is large.
I think the author should have some discussions about these related methods. 

Second, I notice that compared to the result in Table 1, PGD attack can yield better results [1]. I hope to see some discussions about this. Also, Table 1 is really confused. I would not understand the meaning if I am not familiar with the experiment settings.

[1] Salman, Hadi, et al. "Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers." Neuips (2019).
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: The paper presents a new attack, called the shadow attack, that can maintain the imperceptibility of adversarial samples when out of the certified radius. This work not only aims to target the classifier label but also the certificate by adding large perturbations to the image. The attacks produce a 'spoofed' certificate, so though these certified systems are meant to be secure, can be attacked. Theirs seem to be the first work focusing on manipulating certificates to attack strongly certified networks. The paper presents shadow attack, that is a generalization of the PGD attack. It involves creation of adversarial examples, and addition of few constraints that forces these perturbations to be small, smooth and not many color variations. For certificate spoofing the authors explore different spoofing losses for l-2(attacks on randomized smoothing) and l-inf(attacks on crown-ibp) norm bounded attacks. 

Strengths: The paper is well written and well motivated. The work is novel since most of the current work focus on the imperceptibility and misclassification aspects of the classifier, but this work addresses attacking the strongly certified networks. 

Weakness: It would be good to see some comparison to the state of the art 
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper proposes a new way to generate adversarial images that are perturbed based on natural images called Shadow Attach. The generated adversarial images are imperceptible and have a large norm to escape the certification regions. The proposed method incorporates the quantities of total variation of the perturbation, change in the mean of each color channel, and dissimilarity between channels, into the loss function, to make sure the generate adversarial images are smooth and natural. Quantitative studies on CIFAR-10 and ImageNet shows that the new attack method can generate adversarial images that have larger certified radii than natural images. To further improve the paper, it would be great if the authors can address the following questions:

- In Table 1, for ImageNet, Shadow Attach does not always generate adversarial examples that have on average larger certified radii than the natural parallel, at least for sigma=0.5 and 1.0. Could the authors explain the reason?

- In Table 2, it is not clear to me what is the point for comparing errors of the natural images (which measures the misclassification rate of a natural image) and that of the adversarial images (which measures successful attacks rate), and why this comparison helps to support the claim that the attack results in a stronger certificates. In my opinion, to support the above claim, shouldn’t the authors provide a similar table as Table 1, directly comparing the certified radii of the natural images and adversarial images?

- From Figure 9, we see the certificate radii of the natural have at least two peaks. Though on average the certificate radii of the adversarial attacks is higher than that of the natural images, it is smaller than the right peak. Could the authors elaborate more of the results?

- Sim(delta) should be Dissim(delta) which measures the dissimilarity between channels. A smaller dissimilarity suggests a greater similarity between channels. 

- Lambda sim and lambda s are used interchangeably. Please make it consistent. 

- The caption of Table 1 is a little vague. Please clearly state the meaning of the numbers in the table.

review_title: Official Blind Review #1

Meta-review: This work presents a "shadow attack" that fools certifiably robust networks by producing imperceptible adversarial examples by search outside of the certified radius. The reviewers are generally positive on the novelty and contribution of the work. 
Decision: Accept (Poster)