Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes an instance-aware dynamic network, ISBNet, for efficient image classification. The network consists of layers of cell structures with multiple branches within. During the inference, the network uses SelectionNet to compute a "calibration weight matrix", which essentially controls which branches within the cell should be used to compute the output. Similar to previous works in NAS, this paper uses Gumbel Softmax to compute the branch selection probability. The network is trained to minimize a loss function that considers both the accuracy and the inference cost. Training of the network is divided into two stages: First, a high temperature is used to ensure all the branches are sufficiently optimized, and at the second stage, the authors aneal the temperature. During the inference, branches are selected if their probability computed by Gumbel Softmax is larger than a certain threshold.

Overall, the idea of the paper is clearly presented. The methods used in this paper are similar to previous works on neural architecture search (NAS), but this paper can be seen as a meaningful extension to NAS. 

My main concern for this paper is the experiment section.

1) The paper claims that "ISBNet takes only 8.70% parameters and 31.01% FLOPs of the efficient network MobileNetV2 with comparable accuracy on CIFAR-10". Mainstream efficient networks, such as MobileNet and ShuffleNet, are not designed for CIFAR-10 datasets, but ImageNet datasets. Their downsampling strategy is very aggressive, which leads to relatively poor accuracy on CIFAR-10 datasets with 32x32 images. Therefore, it is not fair to compare the MobileNetV2 on CIFAR-10 and claim superiority over it. Compared with other networks customized for the CIFAR-10 dataset, such as NASNet-A, DARTS, and so on, the error rate of ISBNet is significantly worse (>1% higher than DARTS, or >33% relative increase in error rate). 

2) In table 2, the paper compares ISBNet's performance on the ImageNet dataset with other baselines. However, the baseline models are not up to date. For example, MobileNetV2 achieves a 28% error rate with 300M FLOPs, FBNet achieves a 27% error rate with 249M FLOPs. These results, however, are not cited in this paper. Particularly, MobileNetV2 is compared against on the CIFAR-10 dataset, but not the ImageNet dataset. 

3) ISBNet is not the first instance-aware dynamic network. Previous works such as SkipNet, Soft-Conditional computing [1] explored similar ideas. However, in this paper, there is no comparison with previous dynamic networks. 

Overall, I would expect a much stronger experiment section for the paper to be published. 

[1] https://arxiv.org/abs/1904.04971
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Neural architecture search usually aims to find a single fixed architecture for the task of interest. The paper proposes to condition the architecture on the input instances by introducing a "selection network" that learns to retain a subset of branches in the architecture during each inference pass. The intuition is that easier instances require less compute (hence a shallower/sparser architecture) as compared to the more difficult ones. The authors show improved results on CIFAR-10 and ImageNet in terms of accuracy-latency trade-off over some handcrafted architectures and NAS baselines. The method resembles sparsely gated mixture of experts [1] at a high-level, but has been implemented in a way that better fits the context of architecture search (which is still technically interesting).

My major concern is about the comparisons made against existing works:

The authors argue that instance-aware architecture selection is beneficial. However, there seems to be a misalignment between such a claim and the empirical evidences. Apart from the authors' own controlled experiments (ISBNet w or w/o selection network), in Table 1 & 2 the authors are comparing the accuracy-latency tradeoff of their method (which is resource-aware) against either (1) handcrafted architectures in the literature, or (2) a subset of NAS methods that are *not resource-aware at all*. State-of-the-art resource-aware NAS baselines (that are not instance-aware) such as MNASNet [2] (which is not currently cited), ProxylessNAS, FBNets and EfficientNets are completely missing and are left out from both tables for some reason, which are actually the right baselines to compare against in the reviewer's opinion. 

[1] Shazeer, Noam, et al. "Outrageously large neural networks: The sparsely-gated mixture-of-experts layer." arXiv preprint arXiv:1701.06538 (2017).
[2] Tan, Mingxing, et al. "Mnasnet: Platform-aware neural architecture search for mobile." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.

review_title: Official Blind Review #1

Meta-review: This paper proposes a method for finding neural architecture which, through the use of selective branching, can avoid processing portions of the network on a per-data-point basis. 

While the reviewers felt that the idea proposed was technically interesting and well-presented, they had substantial concerns about the evaluation that persisted post-rebuttal, and lead to a consensus rejection recommendation.
Decision: Reject