Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a framework coined as ‘Robust Dynamic Inference Networks (RDI-Nets)’. The goal is concurrently achieving accuracy, robustness and efficiency via ‘input-adaptive inference’ and ‘multi-loss flexibility’  on a multi-output architecture. The observation is that 
in a deep architecture, the representations in earlier layers can also be used for solving a specific downstream classification task. So by attaching several final classification stages at the intermediate layers and by using the uncertainty of the softmax output as a decision criteria as when to use the current output as the final decision, the authors aim to achieve a triple win.
The paper then proposes some attack criteria for a multi-output network.

The paper is not very well written. The paper has a designated related work section but the entire paper reads from its abstract to conclusions constantly like a literature review. This makes it hard to focus and identify the original contribution. The proposed architecture is only introduced later in detail in 3.3 after the attacks. I found the organization and writing style not very reader friendly.

The authors provide a large experimental section, however the key problem with the paper is that it blurs the evaluation issue. While the observation of using uncertainty of estimates at intermediate levels has some intuitive appeal, the decision criteria that the authors propose requires careful selection of thresholds and a good calibration. But given the thresholds the final decision is just a function of the entire network - as it should be. So a natural attack here is just attacking this decision function (or an approximate differentiable proxy) to see if this model provides extra robustness. Is such an evaluation available? Otherwise the proposed approach provides a false sense of robustness as the proposed attacks are not geared towards the actual underlying model.

Minor: 
The definition of entropy in (6) is missing a minus sign. 

The notation f(\theta| x) for theta as parameters and x as input for a function is in conflict with probability notation of conditional probabilities.

review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper considers the following problem: in a classification setting, it appears that by increasing the model capacity, the model accuracy and robustness seem to be improved, at the expense of model size and latency. Thus, the authors want to design an approach where at the same time accuracy, robustness and efficiency are improved at the same time.

Their idea is "multi-exit networks" with inference that adapts based on the input. Particularly, their proposed "Robust Dynamic Inference Networks" allows each input  -- clean or adversarial -- to choose adaptively one of the multiple output layers to output its prediction. This way, they can do an investigation to new variations of adversarial attacks and adversarial defenses. Their experiments show that indeed via this approach, they can achieve the triple wins of accuracy, robustness, and efficiency.

+ novel idea, promising results,  
- Although I like the discussion of accuracy-robustness tradeoff in par 2 of Introduction, I am not sure about the statement that increasing model capacity both robustness and accuracy are improved, as used in the abstract, is always true.
+ First time adversarial attacks and defenses are studied in a multi-output model. 
+ Interesting connection of multi-output networks with ensemble models.

Overall, I believe that this is an interesting, novel paper, which could be of high interest in the ICLR community, and I would vote for its acceptance. 
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: The paper exploited input-adaptive multiple early-exits, an idea drawn from the efficient CNN inference, to the field of adversarial attack and defense. It is well-motivated by the dilemma between the large model capacity required by accurate and robust classification, and the resulting model complexity as well as inference latency. 

Overall, this paper presents an interesting perspective, with strong results. The usage of input-adaptive inference reduces the average inference complexity, without conflicting the "larger capacity" assumption for co-winning robustness and accuracy. 

Since no literature has discussed the attacks for a multi-exit network, the authors constructed three attack forms, and then utilized adversarial training to defend correspondingly. The design of Max-Average Attack is particularly smart - to balance between "benefiting all" and "maximally boosting one" (its result is also convincingly good).

The authors presented three groups of experiments, from relatively heavy networks (ResNet38), to very compact ones (MobileNet-V2). It is especially meaningful to see their strategy work on MobileNet too (though the computational saving is a bit less, no surprise). The authors also did due diligence in ablation study and comparing with recent alternatives.

Several points that could be addressed to potentially improve the paper:

- The authors want to make it clearer that: their "triple win" is not about constructing a light-weight model that is both accurate and robust. It's instead about given an accurate + robust, yet heavy-weight model, how to reduce its AVERAGE computational load per sample inference, by routing "easier" examples to earlier exits. 

- Can the authors think of and construct more diverse and stronger attacks for RDI-Nets? For example, it would be interesting to attacking RDI-Nets (e.g., defended by Max-Average) with randomized weighted combinations of single attacks? 

Note that, at inference time, the same "randomized combination" cannot be also adopted as defense, because an input always wants to exit the earliest possible for efficiency gains.

- The advantage over ATMC is not obvious: slightly lower TA, slightly higher ATA, and slightly more parameters. Could the authors try to align their parameters more closely (to the extent possible)?

- A missing related work: "Shallow-Deep Networks: Understanding and Mitigating Network Overthinking", ICML 2019. It also discussed how to append early exits to pre-trained backbones.



review_title: Official Blind Review #1

Meta-review: The authors develop a novel technique to train networks to be robust and accurate while still being efficient to train and evaluate. The authors propose "Robust Dynamic Inference Networks" that allows inputs to be adaptively routed to one of several output channels and thereby adjust the inference time used for any given input. They show 

The line of investigation initiated by authors is very interesting and should open up a new set of research questions in the adversarial training literature.

The reviewers were in consensus on the quality of the paper and voted in favor of acceptance. One of the reviewers had concerns about the evaluation in the paper, in particular about whether carefully crafted attacks could break the networks studied by the authors. However, the authors performed additional experiments and revised the paper to address this concern to the satisfaction of the reviewer.

Overall, the paper contains interesting contributions and should be accepted.
Decision: Accept (Poster)