Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper studies adversarial attack and defense for adaptive multi-exit network. Adaptive multi-exit network is, by itself, a pretty new and under-studied topic, let alone the adversarial study on top of it. This paper proposes a simple-yet-effective DeepSloth attack based on layerwise loss function. It also proposed an efficacy metric for better evaluation. The experiments are conducted on four multi-exit networks and two datasets: cifar10 and tiny imagenet. In the appendix, there is also evaluation on different norms of attacks. The results on white-box and black-box attack demonstrate the effectiveness: DeepSloth not only hurts the accuracy (also achieved by baselines), but also hurts the efficacy (only achieved by DeepSloth). To reduce the computation burden of performing the attack, the authors did two things: 1) model partitioning in scenario of IOT, and 2) universal attack across a dataset. At the end, the authors adapt AT to multi-exit networks and demonstrate that AT is effective in general and DeepSloth is further helping AT for better robustness.

Overall, I think this paper is pretty complete, containing both attack side and defense side. The proposed method is simple and effective, and the empirical results look good.

My main concern is about the usefulness. The main gain over baseline attack is that DeepSloth also hurts efficacy. In practical use case, two factors lead to bad results to users: 1) inference timeout; 2) wrong prediction. So if we already got wrong prediction, why we care efficacy (which may cause inference timeout)?

Some other minor points.

1. Section 3: metrics. This is novelty of this paper, so I think should be emphasized somewhat. 

2. From Appendix, why DeepSloth is half time versus PGD-(avg)?

review_title: Interesting problem, simple yet effective method, good results.

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
Summary of the paper
=================

This paper studies a new category of adversarial attacks, i.e., attackers that try to slow-down multi-exit DNNs using adversarial examples. The paper extended adversarial attacks to perform the slow-down attack and showed that the attacks could slow-down multi-exit DNNs by 1.5x - 5.0x. Additionally, the paper experimentally answers many questions such as (1) the effectiveness of adversarial training against the attack, (2) input-agnostic attack, and (3) cross-architecture/domain transferability.

Strongness
=========

- The paper suggests a new threat model.
- The paper studies the problem from multiple aspects such as new-attacks, the effectiveness of existing and new defenses, and transferability.

Overall, I think the paper suggested a new and interesting problem and also performed sufficient experiments and analyses for the new threat.

Weakness
========

Some of the motivation of the attack is unclear. I am not convinced that attackers are motivated to use the slow-down attack instead of performing DDoS on cloud applications. And I do not think multi-exit models are used on IoT applications if the target is a real-time system (The reviewer is not a specialist in real-time systems, and please correct it if it is wrong). The reviewer believes that more discussion and motivational examples of the new thread model will increase this paper's impact.

Clarity
=====

This paper is well organized and clear.

Relation to prior work
================

To the best of the reviewer's knowledge, the paper is correctly placed in the literature. However, it is not likely but possible that the reviewer is missing some important prior work.

review_title: Detailed investigation of a new threat model

final_score: rating: 3: Clear rejection
review_content: This paper aims to indicate a new direction to conduct an adversarial attack: to delay the inference time/raise the computational costs of a ''multi-exit'' model.

However, both the motivation and novelty are lacking, I will argue for the rejection if other reviewers wish to accept it.

From the perspective of the industry:
1. We do not actually use the so-called ''multi-exit'' architectures, because they could be difficult to optimize, quantize, assemble, and so on. For mobile devices, we deploy different ''single-exit'' architectures to meet their computing capacity and balance the performance. And it is even trivial to maintenance these architectures because we have developed automation tools.
2. On the cloud servers, the deployment is redundant, such that the service will still work stably, even all the queries require the maximal flops. 
3. After all, it is easy to limit one's Queries Per Second (QPS), and nothing need to be worried about.

From the perspective of academia:
1. This paper only employs the PGD attack and the adversarial training on a specific task, the novelty is lacking.
2. In section 5.2, if one knows the architecture and the training set and the task, how could this configuration being called ''black-box''.

UPDATE:

In the rebuttal, the authors emphasize two facts:
1. There exist such IoT scenarios and real-time systems that employ multi-exit architectures (including those that employ cloud computation.).
2. The slowdown attack is effective in these scenarios.

However, to prove this method works in practice, it is not a simple "1 then 2", you need to show us "1 and 2". That is, you do actually deploy any system described in [1,2,3,4,5,6,7,8], and provide a feasible approach to attack with your method, and report the actual damage caused by your method, and convince the readers the damage is significantly severe compared to the efforts spent for causing the damage. 
Otherwise, it is only an application of PGD with a different loss function.

review_title: Lacking motivation and novelty

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper defines a new problem named slowdown adversarial attack to decrease inference efficiency of early multi-exit networks.

Pros:
1. Slowdown attack on efficient-inference networks in an innovative new problem with wide real-world applications.
2. The proposed method, basically pushing the model prediction to uniform distribution, is straightforward and intuitively correct.
3. The universal/class-universal variances of slowdown attack provide more flexibility of the proposed method, and the blackbox experiments show good generalization ability of slowdown attack across models and datasets.
4. Last but not least, the authors show that traditional adversarial training is not a cure for the newly proposed slowdown attack, showing the challenging character of this new threat model.

Comments:
1. Limitation in application scenario: From the prospective of an attacker, one may want to jointly attack inference efficacy and accuracy. Is it possible to design a more generalized setting, where the adversarial attacker can generate images that both slowdown inference and leads to wrong prediction? The current method pushes model predictions to uniform distribution, so it may be ineffective to generate adversarial images to fool classifiers. This is validated by the results, where the slowdown adversarial images have an 5% or 15% lower accuracy compared with clean images. 
2. Any discussions on how to conduct slowdown attacks on other efficient-inference models (e.g., SkipNet [1])?
3. Please consider releasing your codes.

[1] SkipNet: Learning Dynamic Routing in Convolutional Networks.
review_title: Good paper with an interesting new adversarial threat model

Meta-review: The paper discusses a new threat model for multi-exit DNNs: attacks against efficiency of inference. The proposed attack increases the inference time of such networks by the factor of 1.5-5, while at the same reducing the accuracy of attacked networks. Unlike classical adversarial examples, the new type of attack cannot be thwarted by adversarial training.

Overall, the paper exhibits a novel contribution, is well written and methodically sound. Its practical motivation is somewhat weak, as it is currently unclear for which applications such attacks may be feasible. However, the novelty of the threat model addressed by this paper makes it an interesting methodical contribution. 
Decision: Accept (Spotlight)