Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper discovers a very interesting phenomenon that adversarial examples are more transferable when the perturbations are obtained by propagating a higher ratio of gradients via shortcuts in ResNets and DenseNets. 
The method is simple and comprehensive experiments are conducted to prove its correctness. 
It is a good empirical paper that can inspire future research on investigating the role of shortcuts when defending adversarial examples.

To improve the paper:
1. The "transferability" (or robustness) of gamma γ should be studied. In real applications, γ should be selected before transferring to new black-box models. The experiments in this paper are done by reporting the best results under the best γ. This is a hindsight. How can you pick up γ without accessing the transferability results?
2. More theoretical explanations. It is unclear to me that why it works so well, since the back propagation is no longer coherent with the forward function.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: Summary: 
This paper proposes a modification to standard Projected Gradient Descent to improve transferability of adversarial examples, when the source model is a ResNet-like model containing skip connections. The method, Skip Gradient Method (SGM) modifies the backwards pass to scale down the gradient computed in each residual branch of the model, before these gradients are combined with the gradient from the skip connection. This thus upweights the gradients from the skip connections as opposed to residual modules. The paper demonstrates significant improvements in the single-model black-box transfer setting, against a variety of undefended and defended models.

Strengths:
- Lots of interesting empirical results here! One result which stands out to me is Table 4, where across a wide range of target models, adding SGM to existing techniques cuts defender accuracy by ~1/2 (e.g. 79.9% to 89.66% for SE154). The results in Table 3, showing that even without additional techniques, SGM results in large improvements and outperforms previous approaches, are also quite nice. 
- It's notable that such a simple approach leads to significant improvements.
- Results are very clearly presented, and writing is clear throughout. 

Suggestions for improvement:

I have 3 major concerns: (1) discrepancies between baselines and previously published results (2) unrealistic threat model (3) framing / conclusions drawn by paper.

I suspect (1) is easily addressed, but was not clear to me from the current paper.

(1) Baselines: 
- For multi-step transfer against undefended models (Table 3), the attack success rates seem low compared to numbers reported in e.g. Liu et al. For example, using ResNet-152 as the source, and VGG-19 as the target, Liu et al reports 19% defender accuracy = 81% attacker success. This is significantly higher than the 65.52% reported for MI (both are non-regularized optimization attacks), also stronger than the 80.68% reported for SGM. In general, considering these are *untargeted* attacks, with eps=16, the transfer rates seem pretty low.
- For multi-step transfer against defended models (Table 5), the numbers are slightly lower than previously reported. E.g. attacking IncV3_ens3, Dong et al 19 reports 46.9% for MI, but here, 44.28% is reported. (I realize these differences are slight.)
- In general, it would be nice if the paper were structured so as to make these comparisons easier. For example, the appendix could include tables comparing the baselines reported here, to values previously reported, and explain any discrepancies. Particularly with black-box transfer, where baseline performance is so sensitive to small choices, it's important to ensure baselines are properly implemented, and the current writing of the paper makes it impossible for the reader to assess this unless they are very familiar with the black-box transfer literature.

(2) Unrealistic threat model:
- For black-box transfer, all the strongest attacks use multiple source models (see e.g. NeurIPS 2017 Adversarial Examples contest, the baselines cited in the paper). While this paper shows significant improvements in the single-source setting, these results are significantly weaker than any multiple-source attack. For instance, Liu et al 17 achieve near 100% untargeted success (and near 100% targeted success) against all undefended models they study (many which overlap), and Dong et al 19 report ~85% accuracy against IncV3_ens3 (compared to <60% here).
- There's no reason in practice that an adversary would not employ an ensemble-based attack if they wanted to fool an unknown model.

(3) Framing / conclusions:
- The paper frames the results as a "security vulnerability of ResNets," but the results don't show this. In particular, they show that ResNets make effective *source* models to be used by *attackers*, but they don't imply which models defenders should use in order to be robust to black-box attacks. In this way, the main message of the paper seems misleading to practitioners.
- The main message of the paper thus seems to be "on the transferability of adversarial examples generated  with resnets" as opposed to the "security of skip connections." I would encourage rewriting of the title/intro/conclusion as such.

Overall, there are several very interesting empirical findings in this paper. I view the two main impacts these results could later have would be leading to (1) improved understanding of what causes transfer of adversarial examples and (2) improved understanding of ResNet-like architectures (for example, the results provide some support for the view of ResNets as ensembles of shallow models, cited in the paper). If the paper were written with this view, then concern (2) above becomes unimportant.


Suggestions I believe could strengthen the paper, but I do not view as weaknesses, or necessities:
- Can you decouple the effect of SGM on optimization and transfer? The paper does a bit of this, but restricts the analysis to the single-step case. For instance, if the main effect of SGM is on optimization, this has interesting implications for optimization of ResNets. If the main effect is on transfer, this has interesting implications for what causes transfer.

Minor:
- I'd suggest moving Table 1 (one-step attacks) to the Appendix. The results are less impressive than multi-step, and the community as a whole favors stronger attacks.

Overall, there are several interesting empirical findings in this paper (modulo concerns about baselines indicated above). I suggest that the paper either consider more realistic threat models to be useful to the adversarial examples field, or focus on the insights revealed by these findings. I hope that the authors can address the concerns outlined here, and I would be happy to adjust my score if so.

Note that my current indicated confidence rating is for the current state of the paper. I would be happy to adjust my evaluation if revisions to the paper can one/several of the concerns indicated above.


__________________

EDIT: Rating adjusted from 3: Weak Reject to 8: Accept, after accounting for revisions and author rebuttal. See reply below.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: The paper is about adversarial attacks and highlights a security weakness of skip connections in ResNet-like CNNs, namely: skip connections make it easier to obtain adversarial examples. This observation leads to new approach to adversarial attacks, named Skip Gradient Method (SGM), which weights the residual gradient w.r.t  the skip connection gradient. The approach is validated on a variety of image classification attack scenarios (e. g. white-box and transfer attacks) using two families of source models (ResNet and DenseNet). The results show the superiority of SGM when comparing to other adversarial attack scenarios.

Strengths:
- Simple approach that seems to be giving good results
- Large number of adversarial attack scenarios tested
- Good related work review

Weaknesses:
- Results are reported without variance information
- There are some details missing on how the decay factor is selected 
- Results are reported only on one dataset (ImageNet)

The paper is well written, the authors have identified a "problem" of ResNet-like models and proposed an approach that can exploit the problem in adversarial attacks scenarios (SGM). To the best of my knowledge, this is the first time someone has identified the skip connections security problem in ResNets. The SGM is compared against many existing adversarial attacks methods achieving making the evaluation section quite detailed. Thus, I'd lean towards paper acceptance.

However, I have some questions with respect to the evaluation section:
1. Although, the paper includes ablation analysis for different values of the decay factor, I could not find details on how the decay factor hyperparamenter is selected. Given that this factor is a hyperparamenter, it feels like it should be selected on a validation set and tested on a test set. The paper comments about 5000 random validation set images that are used to compere methods (test set), however, I could not find any mention about validation set used to select decay factor. Could the authors specify how the decay factor is set?
2. Since all the results are reported with 5000 random imagent images, it would be interesting to see the variance of results if the sampling of images is repeated.
3. Section 3.3, 2nd paragraph: "Another important observation is that when there are more skip connections in a network.... the crafted attacks become more transferable....". Although, this statement seems to be correct when comparing DenseNet to ResNet, it does not necessarily hold when comparing models within the same family (e. g. RN18 to RN 152 for FGSM or RN34 to RN 152 for PGD) suggesting that the best is not only connected to the number of skip connections. Maybe the superiority of DN is rather related to the nature of the DN models and not to the number of skip connections. Could the authors clarify?
4. Section 4, threat models: "... the same architectures but trained separately." Could the authors clarify what does it mean? Are these models re-trained from scratch changing the random seed of model initialization, order of the dataset, or something else? In general are the authors using pertained models or train all the models "from scratch".
5. Fig 2: Since x-axis values are not continuous - it might be better to use bar plot.
6. In some tables, the results are reported just for SGM while in other tables SGM is reported in combination with other method. I guess that when the results are reported for SGM they represent SGM from Eq. 10 that represents SGM+PGD. Is this correct? The authors might consider clarifying this in the paper.

Some typos:
In the following, we exploits an architectural security weakness about....
... the the crafted attacks...
... not only reminds researcher to pay....

review_title: Official Blind Review #3

Meta-review: This paper makes the observation that, by adjusting the ratio of gradients from skip connections and residual connections in ResNet-family networks in a projected gradient descent attack (that is, upweighting the contribution of the skip connection gradient), one can obtain more transferable adversarial examples. This is evaluated empirically in the single-model black box transfer setting, against a wide range of models, both with and without countermeasures.

Reviewers praised the novelty and simplicity of the method, the breadth of empirical results, and the review of related work. Concerns were raised regarding a lack of variance reporting, strength of the baselines vs. numbers reported in the literature,  and the lack of consideration paid to the threat model under which an adversary employs an ensemble of source models, as well as the framing given by the original title and abstract. All of these appear to have been satisfactorily addressed, in a fine example of what ICLR's review & revision process can yield. It is therefore my pleasure to recommend acceptance.
Decision: Accept (Spotlight)