Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents an empirical study of the adversarial robustness of dynamic neural networks (DyNNs) with early-exits. The authors find that DyNNs are more robust than SDNNs, and DyNNs can be used to generate adversarial samples efficiently. The authors also proposes a novel adversarial attack method specifically designed for DyNNs.
strength_and_weaknesses: Strengthes:
1. The adversarial robustness of DyNNs is an important research problem. Some of the findings are new and can be interesting to the community.
2. The paper is overall well-written and easy to follow.
3. The experiments are well conducted and well supports the findings in the paper.

Weaknesses:
1. Missing related work on DyNN with improved adversarial robustness [Zhou et al. 2020]. This work also investigated the adversarial robustness of DyNNs and introduced a novel early-exit criteria for improved robustness. This makes the novelty of some findings less strong. However, [Zhou et al. 2020] does not conduct extensive experiments and the throughout empirical analysis in this paper can still be beneficial.
2. The proposed early-attack method is not very well motivated. It's unclear in which scenario one would expect the prediction of last layer to be unchanged because this will not influence the final prediction anyway.

[Zhou et al. 2020] BERT Loses Patience: Fast and Robust Inference with Early Exit

clarity,_quality,_novelty_and_reproducibility: Clarity: Good
Quality: Good
Novelty: OK
Reproducibility: Good
summary_of_the_review: This paper conducts an interesting empirical study about the adversarial robustness of DyNNs. It misses an important related work and is thus somewhat overclaiming the novelty of the findings. Nevertheless, some findings are still interesting and can be helpful.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The work studies the robustness of dynamic neural networks (DyNNs) from four perspectives: 1) comparison of the black-box robustness between static and DyNNs; 2) The inference time of adversarial samples in DyNNs; 3) Dynamic mechanisms with the best robustness; 4) synthesize samples to reduce the effectiveness of DyNNs. 
strength_and_weaknesses: ### Strength:

1. The paper is well-written with summarized findings in each section (3-6).
2. Experiments are conducted in different architectures and datasets.
3. Sufficient implementation details are provided.

### Weakness:

1. Why do the authors only consider early-exit DyNNs? "more popular" seems not convincing.
2. The logical relationship between these four perspectives of DyNNs' robustness is not clear, which results in this paper being more like a technical report rather than a conference paper.
3. The literature on adversarial attacks missing a lot of important works. e.g., Auto-Attack, C&W attack, etc.
4. Finding 1 in Section 3 "With respect to black-box attacks, early-exit DyNNs are more robust than SDNNs" is really debatable. 1) missing white-box robustness evaluation 2) Only FGSM and PGD attack is not reliable for assessing robustness. With more advanced attacks, e.g., Auto-Attack, does this finding also hold?

clarity,_quality,_novelty_and_reproducibility: None
summary_of_the_review: None
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates the robustness of dynamic neural networks, focusing specifically on early exit networks. It does so by exploring how attacks generated using static networks transfer to dynamic networks and how the structure of dynamic networks provides another attack surface. In their majority most works appearing in the literature dynamic networks have been utilized to improve real-time performance. However, they have not been investigated under the spotlight of robustness. In this regard I find that this work provides some interesting findings which have not been explored previously.  
strength_and_weaknesses: Strengths:
- I like the questions raised in this paper and believe they provide a new direction for research in dynamic neural networks. 
- The experiments use some standard deep learning models like VGG and ResNet, and explore different scenarios e.g., either dynamic or static model as surrogate etc.
- Some interesting findings such as that the more exits increase the robustness of the model.

Weaknesses:
- The related works section needs to be expanded significantly in order to provide the reader with a more complete picture especially with the available works on dynamic neural networks. 
- The list of references is also quite limited considering that adversarial attacks are a very popular topic and dynamic neural networks have been investigated for some years. 
- The evaluation datasets include only cifar-10 and 100. It is not clear of using other benchmark such as ImageNet which have higher resolution would impact the results since now a much larger attack surface is available. 
- It would have been nice to see a theoretical framework specific for dynamic neural network that would provide some additional foundations besides the empirical findings. 

clarity,_quality,_novelty_and_reproducibility: Overall the paper is easy to read and clearly sets the research questions that it addresses. 

The following is not properly justified. Does this follow an established understanding by the community?
“One of the reasons for this behavior is the lower variance of the DyNNs. DyNNs use lower number of parameters, hence the feature space for adversarial samples of DyNNs is smaller than the feature space for adversarial samples of SDNNs.”

It would be interesting to show the adversarial examples generated by DyNNs vs static networks. Are there particular differences that could be visually interesting?

The Appendix contains some useful information that should be in the main manuscript. For instance, Figure 7 and Algorithm 1. I suggest the authors carefully review the structure of the paper in order to try and include the additional content within the page limit. 

summary_of_the_review: The paper investigates the robustness of dynamic neural networks against adversarial attacks. It provides a new focus for dynamic neural networks and highlights the new attack surface that they provide. While this paper has some shortcomings and can be improved in some areas I would recommend it based on the new aspects that it tries to highlight.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: **Summary:** This paper looks at adversarial robustness in the context of early-exit neural networks. The paper entails a number of different adversarial methods, transfer attacks, etc. - the standard battery of things one does when dealing with adversarial robustness.

**Strengths:** The paper is an excellent catalogue of experiments on dynamic neural networks. The experiments are comprehensive and well chosen.

**Weaknesses:** At the end of the day, this is a technical report cataloging existing properties on a new (and pretty niche) artifact. It's not particularly novel or significant. It's more of a "findings" paper than something that will inspire new research directions.

**Recommendation:** I have recommended a very ambivalent acceptance. I would not be disappointed if the paper were rejected on the basis of lack of significance. I leave it to the senior AC to decide whether this meets the threshold.
justification_for_why_not_higher_score: See justification for why not lower score.


justification_for_why_not_lower_score: This paper is "yet another adversarial example paper in a slightly different setting." Other than the usual academic deference given to the potential practical consequences of (1) adversarial examples and (2) dynamic neural networks, I don't think the work is well-motivated. Why do we care about adversarial examples in the context of dynamic neural networks? Is this a science paper that helps us gain some deeper understanding of adversarial robustness by looking at the problem a different way? Is there something distinctly different in practice about dynamic neural networks that motivates a different view of adversarial examples? This paper is a technical report where everything appears to be true, but I don't see a clear motivation for why it matters. It's a strong technical report, so I'm ambivalent about whether it is accepted.
Decision: Reject