ISBNet: Instance-aware Selective Branching Networks
Authors: Shaofeng Cai
Yao Shu
Wei Wang
Gang Chen
Beng Chin Ooi
Venue:  (2020)
Keywords: neural networks
neural architecture search
efficient inference

Abstract: Recent years have witnessed growing interests in designing efficient neural networks and neural architecture search (NAS). Although remarkable efficiency and accuracy have been achieved, existing expert designed and NAS models neglect the fact that input instances are of varying complexity and thus different amounts of computation are required. Inference with a fixed model that processes all instances through the same transformations would incur computational resources unnecessarily. Customizing the model capacity in an instance-aware manner is required to alleviate such a problem. In this paper, we propose a novel Instance-aware Selective Branching Network-ISBNet to support efficient instance-level inference by selectively bypassing transformation branches of insignificant importance weight. These weights are dynamically determined by a lightweight hypernetwork SelectionNet and recalibrated by gumbel-softmax for sparse branch selection. Extensive experiments show that ISBNet achieves extremely efficient inference in terms of parameter size and FLOPs comparing to existing networks. For example, ISBNet takes only 8.70% parameters and 31.01% FLOPs of the efficient network MobileNetV2 with comparable accuracy on CIFAR-10.

Under review as a conference paper at ICLR 2020
ISBNET: INSTANCE-AWARE SELECTIVE BRANCHING NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Recent years have witnessed growing interests in designing efficient neural net- works and neural architecture search (NAS). Although remarkable efficiency and accuracy have been achieved, existing expert designed and NAS models neglect the fact that input instances are of varying complexity and thus different amounts of computation are required. Inference with fixed model that processes all in- stances through the same transformations would incur computational resources unnecessarily. Customizing the model capacity in an instance-awart manner is required to alleviate such problem. In this paper, we propose novel Instance- aware Selective Branching Network-ISBie -SBror to support efficient instance-level in- ference by selectively bypassing transformation branches of insignificant impor- tance weight. These weights are dynamically determined by lightweight hy- pernetwork SelectionNer and recalibrated by gumbel-saftmax for sparse branch selection. Extensive experiments show that ISBNet achieves extremely efficient inference in terms of parameter size and FLOPs comparing to existing networks. For example, ISBNel takes only 8. 3.70% parameters and 31.01% % FLOPs of the effi- cient network MobileNetV2 with comparable accuracy on CIFAR-10.
INTRODUCTION
Deep convolutional neural networks (CNNs) (He et al., 2016; Zoph al.) 2018) have revolution- ized computer vision with increasingly larger and more sophisticated architectures. These model architectures have been designed and calibrated by domain experts with rich engineering experi- ence. To achieve good inference results, these models typically comprise hundreds of layers and contain tens of millions of parameters and consequently, consume substantial amounts of compu- tational resources for both training and inference. Recently, there has been growing interest in efficient network design (Howard et al., 2017 landola et al. 2016 Zhang et al. 2018 Sandler et al,, Ia17ooa
2018) and neural architecture search (NAS) (Zoph 20.8 Real et al. 2018 Liu et al. 2018b). respectively with the objective of devising network architectures that are efficient during inference and automating the architecture design process.
Many efficient architectures have indeed been designed in recent years. SqueezeNet (landola et al,, 2016) and MobileNet (Howard al., (017)) substantially reduce parameter size and computation cost in terms of FLOPs on mobile devices. More recent works such as MobileNetV2 (Sandler et al., 2018 and ShuffieNetV2 (Ma al. 2018 further reduce the FLOPs. It is well recognized that devising these architectures is non trivial and requires engineering expertise.
Automating the architecture design process via neural architecture search (NAS) has attracted in- creasing attention in recent years. Mainstream NAS algorithms (Zoph & Le 2016 Zoph et al. 2018 Real et al., 2018) search for the network architecture iteratively. In each iteration, an archi- tecture is proposed by a controller, and then trained and evaluated. The evaluation performance is in turn exploited to update the controller. This process is incredibly slow because both the controller and each derived architecture require training. For instance, the reinforcement learning (RL) based controller NASNet (Zoph et al.) 2018) takes 1800 GPU days and the evolution algorithm based con- troller AmoebaNet (Real et al. 2018) incurs 3150 GPU days obtain the best architecture. Many acceleration methods (Baker et al. 2017: Liu et al. 2018a Bender et al.) 2018 Pham et al.
2018
have been proposed to accelerate the search process, and recent works (Liu et al.
2018h

---

Under review as a conference paper at ICLR 2020
Xie al. 2018 Wu al., 2018 Cai al. 2018 remove the controller and instead optimize the architecture selection and parameters together with mudientt-based optimization algorithms.
While both expert designed and NAS searched models have produced remarkable efficiency and prediction performance, they have neglected critical issue that would affect inference effi- ciency. The architectures of these models are fixed during inference time and thus not adap- tive the varying complexity input instances. However, in real-world applications, there are only small fraction of input instances requiring deep representations (Wang et al. 2018, Huang al. 70)7a). Consequently, expensive computa- tional resources would be wasted if all instances are treated equally. Designing a model with suffi- cient representational power to cover the hard in- stances, and meanwhile finer- greengg control to provide just necessary computation dynamically for instance of varying difficulty is therefore es- sential.
Shiit!
inportasee
adl ispats
cell mippu
Cell
Figure 1: Illustration of the building block Cell of ISBNet. The cell comprises SelectionNer and N interconnected nodes. Fach node connection trans- forms via B diverse branches. The branch selection is guided by importance weights from the Selection- Net which recalibrated with Gumbel-softtax,
In this paper, we propose ISBNet to address the aforementioned issue with its building block Cell as illustrated in Figure Following the widely adopted strategy in NAS (Zoph et al. 20188) Pham al., 2018 Liu et al., 201 8b Xie et al. 2018). the backbone network is a stack of L structurally iden- tical cells, receiving inputs from their two previous cells and each cell contains N coereconneeted computational Nodes. The architecture of ISBNet deviates from the conventional wisdom of NAS which painstakingly search for the connection topology and the corresponding transformation op- eration of each connection. In ISBNe1, each node is instead simply connected to its prescribed preceding node(s) and each connection transforms via a candidate set of operations (branches). To allow for instance-aware inference control in the branch level, we integrate L lightweight hy- pernetworks SelectionNets, one for each cell to determine the importance weight each branch. Gumbel-softmal (Jang et al. 2016 Maddison et al. (016) is further introduced recalibrate these weights, which enables efficient gradient- based optimization during training, and more importantly, leads sparse branch selection during inference for efficiency.
The contributions of ISBNet can be summarized as follows:
ISBNet is a general architecture framework combining advantages from both efficient net- work design and NAS, whose components readily customizable.
ISBNet is novel architecture supporting the instance-level selective branching mechanism by introducing lightweight SelectionNets, which improves inference efficiency significantly by reducing redundant computation.
ISBNet successfully integrates gumbel-soffmat to the branch selection process, which en- ables direct gradient descent optimization and is more tractable than RL-based method.
ISBNet achieves state-of-the-an the---rrt inference efficiency in terms of parameter size and FLOPss and inherently supports applications requiring fine-grained instance-level control.
Our experiments show that ISBNer is extremely efficient during inference and successfully selects only vital branches on a per-input basis. In particular, with minor 1.07% accuracy decrease, ISB- Ner reduces the parameter size and FLOPs by 10x and 11 11.31x respectively comparing to the NAS searched high-peeformance architecture DARTS (Liu et al.) 201,20088b) Furthermore, with a tiny model of 0.57M parameters, ISBNet achieves much better accuracy while with only 8.03% and 30.60% inference time parameter size and FLOPs comparing to the expert-designed efficient network Shuf- fleNetV2 1.5x (Ma et al. 2018). We also conduct ablation studies and visualize the branch selection process to understand the proposed architecture better. The main results and findings are summa- rized in Sec 4.2 and See 4.3
2

---

Under review as a conference paper at ICLR 2020
2 RELATED WORK
Efficient Network Design. Designing resource-aware networks (landola at. 2016 Gholami et al.) 2018 Ma et al. 2018 Sandler et al. 2018 Gholami et al. 2018) has attracted great deal of atten- tion recent years. However, these works mainly focus reducing parameter size and inference FLOPs in many works. For instances, SqueezeNet (Hu et al. 2018) reduces parameters and com- putation with the fire module; MobileNetV2 (Sandler et al., 2018) utilize depth- and point-wise convolution for more parameter-efficien convolutional neural networks; ShuffleNetV2 (Ma et al.) 2018) proposes lightweight group convolution with channel shuffle facilitate the information flow- ing across the channels. To make inference efficient, many of these transformations are introduced to the candidate operation set in ISBNet.
Many recent works explore conditional (Wang et al., 2018) and resource-constrained predic- tion (Huang et al.] 2017a) for efficiency. SkipNet (Wang et al.) (018)) introduces a gating hy- pernetwork to determine whether to bypass each residual layer (He et al., 2016) conditional on the current input instance. Compared with SkipNet. ISBNer provides more efficient and diversi- fied branch selections for the backbone network and the hypernetworks in ISBNet are optimized in an end-to-end training manner instead of generally less tractable policy gradient (Williams) 1992). MSDNet (Huang et al., 2017a) supports budgeted prediction within prescribed computational re- source constraint during inference by inserting multiple classifiers into a 2D multi-scale version of DenseNet (Huang et al. 2017b). By early-exit into a classifier, MSDNet can provide approximate predictions with minor accuracy decrease. Functionally, ISBNet also supports budgeted prediction by dynamically controlling the number of branches selected, therefore per-input inference cost.
Neural Architecture Search. Mainstream NAS (Zoph et al., 2018 Real et al. 2018) treats archi- tecture search as a stand-alone process whose optimization is severed from candidate architecture optimization. Search algorithms such as RL-based NAS (Zoph et al.) 2018) and evolutionary- -based NAS (Real et al. 2018) obtain state-of-the-arr architectures at an unprecedented amount of the GPU- time searching cost. Recently, many works have been proposed to accelerate the search pipeline, e.g., via performance prediction (Baker et al. 2017 Liu et al. 2018a), hypemetworks generating initialization weights (Brock et al. 2017), weight sharing (Bender et al. 20188] Pham et al.) 2018)) These approaches greatly alleviate the search inefficiency while the scalability issue remains un- solved.
A number of proposals (Liu et al., 2018b; Wu et al. 2018 Cai et al. 2018) instead integrate the architecture search process and architecture optimization into the same gradient-based optimization framework. In particular, DARTS (Liu et al. 8018b) relaxes discrete search space to be continu- ous by introducing operation mixing weights to each connection and optimizes these weights di- rectly with gradient back poock-prpggated from validation loss. Similarly, the discrete search space in SNAS (Xie et al., (0088) is modeled with sets of one-hot random variables for each connection, which is made differentiable by relaxing the discrete distribution with continuous concrete distri- bution (Jang et al., 2016) |Mddiisnn et al.] (0166). In terms of architecture optimization, ISBNet also relaxes the discrete branch selection to continuous importance weights optimized by gradient descent; while instead of direct optimization on the weights, SelectionNets are introduced to dy- namically generate these weights which more effective and meanwhile bring about larger model capacity. Further, SelectionNets enable instance- level architecture customization rather than finding fixed model.
3
INSTANCE-AWARE SELECTIVE BRANCHING NETWORK
3.1 THE BACKBONE NETWORK
The backbone network constructed with a stack of cells, each of which is a directed acyelic graph consisting of an ordered sequence of N intermediate nodes. As is illustrated in Figure 1 z/ and of are the cell input nodes from the two preceding cells; each intermediate node (i 2 2) of the 4eh cell forms a latent representation and receives n input node from its preceding nodes:
'n can be larger than 2 for deeper and wider local representation. E.g., i 1 for each ni leads to dense connection, i.c., DenseNet (Huang al. 2017b).
3

---

Under review as a conference paper at ICLR 2020
5) (0,1, (0,1, = jES:
Thereby, each cell contains C' = n connections in total. The connection passes information from node r z after the aggregation of a candidate set of B branches of transformation inspired from widely-adopted transformations in NAS (Pham et al. 2018 Liu et al. 2018b Xie et al. 2018) and
efficient network design (landola et al. 2016 Sandler al. 2018 Zhang et al. 2018):
Fzcld))
Σ w/ Fu(d))
where Wh here represents the importance of the AE branch (operation) of the connection and is dy- namically generated by the cell hypernetwork rather than a fixed learned parameter as is in existing NAS methods (Liu al. 2018b Xie et al. 2018). We shall introduce the hypernetwork in Sec- tion 3.2 Finally, the output of the cell Toou is aggregated by concatenating the output from all the intermediate nodes. We shall use superscript subscript e and index the cell, connection, and branch respectively.
Recent work (Xie et al. 2019) reveals that architectures with randomly generated connection achieve surprisingly competitive results comparing to best NAS models, which is confirmed empirically in our experiments on smaller datasets. In this paper, we thus mainly focus on the branch transforma- tion and selection part and their impact on the inference efficiency instead of specifying detailed connection topology. Under this architecture formulation framework, we can readily adjust the number of candidate branches B and also the specific transformations before training, customiz- ing model capacity and efficiency respectively depending on the difficulty of the task and resource constraints in deployment.
3.2 SelectionNet FOR WEIGHT RECALIBRATION
To support instance level inference control, we introduce lightweight hypernetworks SelectionNet, one for each cell. Each SelectionNet SNer' receives the same input as the lth cell, specifically the two output nodes 2-22 2n11
wott (i.e. z/g. 2²) from the preceding cells, and concurrently produce sets of recalibration weights, one for each connection of the cell:
W SNer (2t) x1)
where W' E RCCB is the recalibration weight matrix for the Leh cell. The SelectionNer SNer' dy- namically generates these weights with pipeline of = convolutional blocks, a global average pooling and finally an affine transformation. For the m convolutional transformation, we adopt sep- arable convolution (Sandler et al. 2018) which contains point-wise convolution and depth-wise convolution of stride 2 and kernel size 5 x 5. The stride reduces the parameter size and computa- tion of SNer'. and the larger kernel size for depth-wise convolution incurs negligible overhead while extracts features for the immediate weight generation with a larger local receptive field.
The recalibration weights given by the SelectionNet is reminiscent of convolutional attention mecha- nism (Hu et al.] 2018) Woo et al. 2018 Newell al. 2016), where attention weights are determined dynamically by summarizing information of the immediate input and then exploited to recalibrate the relative importance of different input dimensions, e.g., channels SENet (Hu et al., 2018). In ISBNel, the recalibration weights are introduced to the branch. Particularly, each candidate operation of the connection coupled with a rescaling weight.
The gufbel-softmax (Jang et al. 2016 Maddison et al. 2016) technique and the reparameterization trick (Kingma & Welling 2013) is introduced to further recalibrate these weights generated by the SelectionNet, to enable efficient gradient-baut based optimization for the whole network during train- ing, and more importantly, ensure a sparse selection of important branches during inference. More specifically, each set of importance weights W! e R for the ct connection of the 44 cell (C)) af- ter the following recalibration of the gunbel-softmax follows concrete distribution (Maddison et al. 2016) controlled by temperature parameter 7:
4

---

Under review as a conference paper at ICLR 2020
exp((w's (2))/T) W Σ exp((weM + Go (//))/)) =
where We,b is then directly used for branch recalibration as in Equation and Ge log(-log((1)) here is a gumbel random variable coupling with Pyg branch by sampling Ud from Uniform(( 1) (Jang et al. (0166). The concrete distribution (Maddison al. 2016 suggests that (1) We.b B. as +00, and more importantly (2):
B
p lim Web = 1) = exp(w',t)/ Σ 7-0
exp(w",r)
(5)
Therefore, high temperature leads to uniform dense branch selection while lower temperature tends sparsely sample branches following categorical distribution parameterized by softmar(W) (W)).
3.3 OPTIMIZATION AND INFERENCE FOR ISBNet
With the continuous relaxation of the gumbel-softmax (Jang al. 2016 Maddison et al. 2016 and the reparameterivation (Kingma & Welling, 2013), the branch selection process of the SelectionNets made directly differentiable with respect to the weight w particular, the gradient aul
ac
backpropagated from the loss function C to Wh:b through the backbone network can be directly backpropagated to w/, with low variance (Maddison et al. 2016), and further to the 4n SelectionNet unimpededly. Therefore, parameters of the whole network can be optimized in an end-to-end manner by gradient descent.
The temperature of Equation 4 regulates the sparsity of the branch selection. A relatively higher temperature forces the weights to distribute more uniformly so that all the branches each connec- tion efficiently trained. While low temperature instead tends to sparsely sample one branch from the categorical distribution parameterized by the importance weights dynamically determined by SelectionNets, thus supporting finer-grained instance-level inference control by bypassing unim- portant branches. To leverage both characteristics, we propose a two stage training scheme for ISBNet: (1) the first stage pretrains the whole network with a fixed relatively high temperature till convergence. (2) the second stage fine-tunes the parameters with 7 steadily annealing to a relatively low temperature. The first stage ensures that branches are sufficiently optimized before the instance- aware selection and the fine-tuning in the second stage helps maintain the performance of ISBNet under sparse branch selection during inference.
To further promote inference efficiency and reduce redundancy, regularization term is explicitly introduced the fine-tuning stage which takes into account the expectation of the resource con- sumption R. in the final loss function L for correctly classified instances:
C = Lor + A1llw| A2 12 log E(R.
c
B
(6)
LoF + Aillwi Ae log ΣΣΣ web (R((F.
where Log and A1 Ar|lw|| denotes the cross-entropy loss and the weight decay term, y the ground truth class label, ÿ' the prediction, X2 controls the regularization strength and R(-) calculates the resource consumption of each operation Fi F... The operation importance weight th represents the probability of the corresponding branch Fi being selected during inference, and therefore the regularization term E[R] corresponds to the expectation of the aggregated resource required for each input instance.
The resource regularizer is readily adjustable depending deployment constraints, which may include the parameter size, FLOPs, and memory access cost (MAC). In this work, we mainly focus on the inference time, specifically FLOPS, which can be calculated beforehand for each branch.
5

---

Under review as a conference paper at ICLR 2020
R(F2,()) is thus constant here, which means that the regularizer R is also directly differentiable with respect to ui We denote ISBNer trained with regularization strength X2 as ISBNet-R-Xxz
During inference, the instance-level selective branching is achieved for each connection C² by se- lecting branches of the top largest recalibration weights whose aggregated weight s just exceeds a threshold T. Denoting Wi sorted in descending order as W c²
then:
min Skk : Sk Σ
w
8k TT
After the selection, the recalibration weight w of the selected branch is rescaled by 전 to stabilize the scale of the representation. Consequently, the SelectionNet will select only necessary branche for each instance depending on the imput difficulty and meanwhile the FLOPs of each branch, i.e., trading off between LcE and R in Equation 6 Furthermore, the resource consumption of each in- stance can be precisely regulated for each connection. In this paper, the same threshold is shared among all connections for simplicity and ISBNet inference with the threshold is denoted as ISBNer-T-t.
finer-grained manner by scheduling the threshold dynamically
Under such an inference scheme, the backbone network comprises up to (2 -) possible candi- date subnets, corresponding to each unique branch selection all L-C connections. For a small ISB- Net of 10 cells, with 5 candidate operations, 8 connections per cell, there are (2 1)8-10 = 2- 10 possible candidate architectures of different branch combination, which is orders of magnitudes larger than the search space of conventional NAS (Pham al., 2018 Liu al., 2018b; Xie al., 2018
Cai et al. 2018: Stamoulis et al. 2019).
EXPERIMENTS
We now compare the performance of ISBNel with the best-performing expert-designed efficient net- works and NAS architectures using benchmark dataset CIFAR-10 10 and ImageNet. The experimental details presented in Sec. [.T] main results reported in Sec. 4.2 followed by the visualizations of the branch selection process of ISBNet in Sec. 4.3
4.1 EXPERIMENTAL SETUP
Dataset CIFAR-IO contains 50,000 training images and 10,000 test images of 32 x 32 pixels in
10 classes. We adopt standard data pre-processing and augmentation pipeline (Liu et al. 2018b; Xie et al.) 2018) and apply AutoAugment (Cubuk al. 2018), cutout (DeVries & Taylor 2017) of length 16. ImageNet contains 1.2 million training and 50,000 validation images in 1000 classes. We adopt standard augmentation scheme following (Liu et al. 2018b; Xie al., 2018) and apply label smoothing of 0.1 and AutoAugment. Results are reported with 224 x 224 center crop.
Candidate Operation Set The following 5 candidate operations (B 5) are adopted for demon- stration and can be readily adjusted in deployment:
x max-pooling
3 separable-conv
3 x 3 avg-pooling
x 5 separable-conv
skip connection
In particular, separable-com stands for two separable convolution (Howard et al. 2017) of ReLU- Com--Con-BN. Skip connection allows for efficient representation forwarding; pooling layers are computationally lightweight with parameter, and separable-conv dominates the parameter size and computation in each connection. The three types of operations support trade-off between rep- resentation power and efficiency for the branch selection of each connection.
In the pre-training stage, the temperature 7 is fixed to 3 till con- vergence. In the fine-tuning stage, is initialized to 1.0 and is further annealed by cxp(-0.0006) x 0.999 every epoch to 0.5 for CIFAR-10. 10. The temperature is throughout for ImageNet.
Temperature Annealing Scheme
At least branch will be selected for each connection.

---

Under review as a conference paper at ICLR 2020
Architecture Details For CIFAR-10, evaluate two ISBNet architectures of different size for demonstration: (1) IS a small network with = cells and 15 initial channels: (2) ISB- Ner(M), a medium network with L 10 cells and 20 initial channels. For ImageNet, we evaluate medium network with I. = 10 cells and 32 initial channels.
All the architectures contain N intermediate nodes in each cell and we adopt simple node connection strategy of connecting to the two preceding nodes (i.e. d/{1 and xy_y) for CIFAR-I0, and connecting to the preceding node (z) and randomly zá or x for ImageNet. Further, nodes directly connected the imput nodes downsampled with stride 2 for 5 -thh and 24 -th cells. An auxiliary classifier with weight 0.4 is connected to the output of 24 th cell for extra regularization.
Optimization Details For CIFAR-I0, we apply SGD with momentum 0.9 and weight decay 3 10- 4 for 1200 epochs for both training stages. The learning rate is initialized to 0.025 and 0.005 for the pre-training and fine-tuning stage respectively. We use batch size 256/128 for CIFAR-10 (SBNeet))/SSNNe((M)) and batch size 128 for ImageNet to fit the whole network into one Titan RTX. For ImageNet, apply SGID with nesterov-momentum 0.9 and weight decay 3 x 10-5 for 250 epochs. We adopt drop-connection and drop-branch linearly to 0.1 and 0.7/0.5 respectively for The learning rate annealed to zero with one cycle of cosine learning rate scheduler (Loshchilov & Hutter 2016).
4.2 ISBNet PERFORMANCE EVALUATION
Architecture
Text Erroe
Training / Inference
Traaiingg / Inference FLOIs
Scurch
Seurch Space
Search Cust days)
Params (M)
ResNetto1
al 2016
6:25
2520
DesseNesBC Huaig <t 201
3.46
25.6
9345
mtanual
MobileeNrt/21 Sundlar al 2015
5.56
2.30
94.42
manual
ShutteNerV22
6.36
NASNet A (2ph et al. 2016
2665
RL
cell
1800
AmazbaNas-A [Rail ml 2013
3.34
3.2
evolution
call
3150
ENAS [ham
2.2
DARTS Liu at 20150
3.00
$42.0
gradient
cell
1.9
SNAS Xie e al [018]
3.10
2.3
gradient
cull
3
IS SelectionNet
gradieni
layer wise
/5.27
0,57/0.55
$4.65 50,99
poalient
laycr-wise
4,57 15.2
0.57/0..8
$4.65/72..0
gradient
layor-wise
3,66 /.007
0.57/0.33
$4.65/47.91
gradient
layer Wise
14.26
0,57/0.31
84.65/45.61
gradient
layer-wise
158Nen8)-R0011-008
3.66//4.21
0,57/
S4.65 04.65/433.82 43..22
gradient
layer wise
3.66//5.85
0,57/0.20
$4.65/29
gowlient
laycr-wise
1.02
267.37 139.5
gradient
layer-wise
(88Nen 0.
/3,44
1.86/0.89
267.3/ 119.6
gradient
layer wise
1S8Nen1)) 0.55
2.84/44.50
1.86
367.3/74.90
gowlient
layen-wise
Table 1:
Comparison of ISBNet with other state-of theee-r architectures on CIFAR-10 dataset. Results of ISBNet reported in fall model / selective branching respectively.
Overall Results and Discussion. Table summarizes the overall performance on CIFAR- 10 of ISBNet under different inference threshold T and resource constraint strength R.. In terms of training efficiency, ISBNet takes only 2.5 and 5.5 GPU training days for [SBNe((S) and ISBNet(M) respec- tively without any architecture searching, which is up three orders of magnitudes less time than conventional evolution-base NAS or RL-based NAS thanks to the efficient network design and the end-to-end gradient- based optimization.
As for inference time performance, ISBNer reduces drastic amount of the parameter size and FLOPS comparing baseline networks. Specifically, with comparable accuracy, ISBNet(S)-R-O.s 7-0.8 only takes 0.20M parameters and 29.28N FLOPS on average during inference, which is only 8.03% and 30.60% efficient network ShuffleNetV2 1.5x; ISBNei( 10x and 11x parameter size and FLOPs reduction than DARTS with 1.07% accuracy decrease. The drastic parameter size and FLOPs reduction demonstrate that the selective branching mechanism in ISBNet enables extremely efficient instance-level prediction. This is also corroborated by the significant reduction of the parameter size and FLOPs from training to inference of ISBNet, ie. from 0.57M parameters and 84.65M FLOPPs to 0.33M and 47.9 4 in (SBNEE())-R-D.0-T-0..8
achieves up to

---

Under review as a conference paper at ICLR 2020
Architecture
Text Eirne
Parame
Search Method
Search Space
Cus
(96)
(GPU
Ziang al
2013
25.1
534
1.0 MakileNet
Honard ot dl
2017
76.4
4.2
569
mannel
035 MokileNet
Howad Hi
2017
31.6
2.6
525
maniel
NASNes3 Zoet
2015
AmoebaNar-4
25.5
evolution
oell
26.9
4.9
595
groliere
cell
8-N11-6-00--1100
25.9
4.9
485
gralian
Leydn-Mri
198-0-8-00--.99
40:0030
39--124
arallen
layar-viis Miie-
308
gridient
13NNa-8-008-7--7
32.1
34:0013
320:
gridenc
wike
Table Statistics and performance of ISBNet compared with other architectures on ImageNet dataset.
(80) cornect
Figure 2: Average Recalibration Weight of the first/last cell during the temperature anncaling stage, and infer- ence time Branch Selection Ratio (last cell) of the 5 branches for each connection of (-NNo0(S---O000---0
Results Table D also validate that the resource regularizer effectively regularizes the network for more efficient inference both parameter size and FLOPs, although only FLOPSs is explicitly regularized. Specifically, the larger the regularization strength X, the more efficient ISBNet is, while at the cost of a minor accuracy decrease. For instance, the inference parameter size of ISBNet(M) 7'-0.8 is reduced from 1.02M, 0.89M to 0.66M, and FLOPP from 139.46M, 119.56M to 74.90M for regularization strength from 0.0, 0.1 to 0.5 respectively.
The results show that a small ISBNet able to achieve competitive accuracy comparable to the best efficient and NAS-searched models, meanwhile with far fewer inference parameters and FLOPs. This raises questions about the necessity of current laborious architecture search of NAS (Zoph al. 2018 Real et al. 2018). In this paper, propose selective branching mechanism evocative of convolutional attention (Hu et al. 2018 Woo et al. 2018) via the introduction of the hypernet- works SelectionNer, which leads larger model capacity and enables selective branching. With 19.30% more parameters and 8.54% more FLOPs, (SBNer(S) integrated with SelectionNets achieves 0.41% noticeably higher accuracy. Further trained with gumbel-softme SelectionNets enables the network to efficiently select necessary branches and customize its architecture per-input basis during inference. Gumbel-softmax is necessary for maintaining accuracy because with plain soft- max trained SelectionNets, JSBNer(S)-Softmee suffers from catastrophic accuracy decrease, from 4.37% to 15.22%, while with limited parameter size and FLOPs reduction with an inference thresh- old 0.6.
summarizes the performance of ISBNet under different thresholds on ImageNet. The results demonstrate that ISBNet achieves quite competitive results compared with expert-designed efficient networks and NAS-searched models even with a simple connection scheme and a preconfigured candidate operation set. Further, with the selective branch- ing of SelectionNet, one single network of ISBNet supports efficiency-accuree trade-off' by simply controlling the importance threshold. In particular, ISBNet reduce the parameter size by 18.37% and FLOPs by 18.14% with threshold 0.9 with minor 0.2 accuracy decrease. With threshold 0.7, ISBNet achieves another 12.24% and 14.02% redundancy reduction respectively. This confirms that networks can support efficient instance-aware inference with the selective branching mechanism.
Accuracy-FLOP, Trade-off. Table 2
4.3 VISUALIZATION OF SELECTIVE BRANCHING
Ratio of Selective Branching. We visualize in Figure the average recalibration weight and branch selection ratio of representative cells in ISBNer R-0.0-T-088 which shows the ratio dur- ing training and in the final model of each branch being selected during inference respectively. An obvious stratified pattern can be observed that one eoparable-convolution branch gradually domi- nates the connection in the first cell while in subsequent cells, the branch selection tends to be more
8

---

Under review as a conference paper at ICLR 2020
uniform and diversified. This pattern demonstrates that features extracted in lower layers share sim- ilar branch transformation where branch pruning can be performed to reduce the parameter size; while the instance-aware efficient inference requires diversified branch selections ascending the lay- ers. Further experiments show that the average number of branches selected in the last cell 1.1, indicating that only small number of branches required for the inference of most instances.
Qualitative Difference between Instances. Denoting instances that the network is confi- dent with prediction as easy instance and un- certain hard instance, we visualize the clus- tering of easy and hard instances in Figure to help understand the selective branching mech- anism. We find that the certainty of the predic- tion made by ISBNer depends mainly on the im- age quality. In general, casy instances are more salient (clear with high contrast) while hard in- stances are more inconspicuous (dark with low contrast). We also compute the accuracy and average FLOPs of each cluster. On average, easy instances achieve much higher classifica- tion accuracy with 11.2% fewer FLOPs com- pared with hard instances. This shows that computation could be greatly reduced without portant branches for relatively easy instances.
(a) easy instances
(h) hard instances
Figure 3: Visualization of easy and hard instances of model (E-Ne((M---000---0. on CIFAR-10. Easy in- stances clearer and brighter in general while hard instances are darker and blurry. sacrificing accuracy by selective bypassing unim-
CONCLUSION
In this paper, we have proposed ISBNet, a novel network framework with the advantages of both efficient network design and neural architecture search. To achieve efficient instance-aware infer- ence, series of lightweight hypernetworss are introduced to cells of the backbone network to determine importance weights for selective branching. We have also integrated gumbel-softmax and the reparameterization trick to the branch selection process, which enables accessible and tractable gradient- end-to-end training, and more importantly, extremely efficient inference. The infer- ence efficiency is further enhanced with the resource-aware regularization. Extensive experiments and visualizations have been conducted, and the results validate the efficiency of instance-aware selective branching inference.
REFERENCES
Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. arXiv preprint arXiv:1705.1082. 2017.
Gabriel Bender, Pjeter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Understand- ing and simplifying one-shot architecture search. International Conference on Machine Learn- ing. pp. 549-558, 2018
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Smash: one-shot model archi- 2017.
tecture search through hypernetworks. arXiv preprint
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search target task and hardware. arXiv preprint 1rXiv:1812.00332. 2018.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint 100.000000000000,,, 2018.
Terrance Vries and Graham W Taylor. Improved regularization convolutional neural networks
with cutout. arXiv preprint arXiv: 077700004555 2017.

---

Under review as a conference paper at ICLR 2020
Amir Gholami, Kiscok Kwon, Bichen Wu, Zizheng Tai, Xiangyu Yue, Peter Jin, Sicheng Zhao, and Kurt Keutzer. Squeezenext; Hardware-awarr neural network design. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 1638-1647, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 17X4::1704.04611, 2017.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitate networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q Wein- berger. Multi-scale dense networks for resource efficient image classification. arXiv preprint arXiv:1703.09844 2017a.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinherger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4708, 2017b.
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. 2016.
arXiv preprint
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizzaioo with gumbel-softmax. arXiv 144, 2016.
preprint
Diederik P Kingma and Max Welling. arXiv:1312.61 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceed- ings of the European Conference on Computer Vision (ECCV), pp. 19-34, 2018a.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.0900 09055, 2018b.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1608 100800960,..3383, 2016.
Ningning Ma, Xiangyu Zhang, Hai-Ta Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 116-131, 2018.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv: 1.00712, 2016.
Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estima- tion. In European Conference on Computer Vision, pp. 483-499. Springer, 2016.
Hieu Pham, Melody Y Guan, Barret Zoph, Quoe V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:7802.03266. 2018.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. arXiv preprint 10X 2018.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018
10

---

Under review as a conference paper at ICLR 2020
Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Single-path nas: Designing hardware-efficient convnets in less than 2019.
hours. arXiv preprint
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph Gonzalez. Skipnet: Learning dy- namic routing in convolutional networks. In Proceedings of the European Conference on Com- puter Vision (ECCV). pp. 409-424, 2018.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinorrcmentt learning, Machine learning, 8(3-4):229-23) 1992.
Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. Cham: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), pp- 3-19, 2018
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang. Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. arXiv preprint arXiv:/812.03444 2018.
Saining Xie, Alexander Kirillov, Ross Girshick, and Kaiming He. Exploring randomly wired neural networks for image recognition. arXiv preprint arXiv:1904.01500 2019.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search.
arXiv preprint arviv: 18/29999, 2018
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6848-6856 2018
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint arX1v:1611.0157,, 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8697-8710, 2018.
11


Reviews:
[{"review_id": "r1eH7iPC9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an instance-aware dynamic network, ISBNet, for efficient image classification. The network consists of layers of cell structures with multiple branches within. During the inference, the network uses SelectionNet to compute a \"calibration weight matrix\", which essentially controls which branches within the cell should be used to compute the output. Similar to previous works in NAS, this paper uses Gumbel Softmax to compute the branch selection probability. The network is trained to minimize a loss function that considers both the accuracy and the inference cost. Training of the network is divided into two stages: First, a high temperature is used to ensure all the branches are sufficiently optimized, and at the second stage, the authors aneal the temperature. During the inference, branches are selected if their probability computed by Gumbel Softmax is larger than a certain threshold.\n\nOverall, the idea of the paper is clearly presented. The methods used in this paper are similar to previous works on neural architecture search (NAS), but this paper can be seen as a meaningful extension to NAS. \n\nMy main concern for this paper is the experiment section.\n\n1) The paper claims that \"ISBNet takes only 8.70% parameters and 31.01% FLOPs of the efficient network MobileNetV2 with comparable accuracy on CIFAR-10\". Mainstream efficient networks, such as MobileNet and ShuffleNet, are not designed for CIFAR-10 datasets, but ImageNet datasets. Their downsampling strategy is very aggressive, which leads to relatively poor accuracy on CIFAR-10 datasets with 32x32 images. Therefore, it is not fair to compare the MobileNetV2 on CIFAR-10 and claim superiority over it. Compared with other networks customized for the CIFAR-10 dataset, such as NASNet-A, DARTS, and so on, the error rate of ISBNet is significantly worse (>1% higher than DARTS, or >33% relative increase in error rate). \n\n2) In table 2, the paper compares ISBNet's performance on the ImageNet dataset with other baselines. However, the baseline models are not up to date. For example, MobileNetV2 achieves a 28% error rate with 300M FLOPs, FBNet achieves a 27% error rate with 249M FLOPs. These results, however, are not cited in this paper. Particularly, MobileNetV2 is compared against on the CIFAR-10 dataset, but not the ImageNet dataset. \n\n3) ISBNet is not the first instance-aware dynamic network. Previous works such as SkipNet, Soft-Conditional computing [1] explored similar ideas. However, in this paper, there is no comparison with previous dynamic networks. \n\nOverall, I would expect a much stronger experiment section for the paper to be published. \n\n[1] https://arxiv.org/abs/1904.04971", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SkxboFl2tB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Neural architecture search usually aims to find a single fixed architecture for the task of interest. The paper proposes to condition the architecture on the input instances by introducing a \"selection network\" that learns to retain a subset of branches in the architecture during each inference pass. The intuition is that easier instances require less compute (hence a shallower/sparser architecture) as compared to the more difficult ones. The authors show improved results on CIFAR-10 and ImageNet in terms of accuracy-latency trade-off over some handcrafted architectures and NAS baselines. The method resembles sparsely gated mixture of experts [1] at a high-level, but has been implemented in a way that better fits the context of architecture search (which is still technically interesting).\n\nMy major concern is about the comparisons made against existing works:\n\nThe authors argue that instance-aware architecture selection is beneficial. However, there seems to be a misalignment between such a claim and the empirical evidences. Apart from the authors' own controlled experiments (ISBNet w or w/o selection network), in Table 1 & 2 the authors are comparing the accuracy-latency tradeoff of their method (which is resource-aware) against either (1) handcrafted architectures in the literature, or (2) a subset of NAS methods that are *not resource-aware at all*. State-of-the-art resource-aware NAS baselines (that are not instance-aware) such as MNASNet [2] (which is not currently cited), ProxylessNAS, FBNets and EfficientNets are completely missing and are left out from both tables for some reason, which are actually the right baselines to compare against in the reviewer's opinion. \n\n[1] Shazeer, Noam, et al. \"Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.\" arXiv preprint arXiv:1701.06538 (2017).\n[2] Tan, Mingxing, et al. \"Mnasnet: Platform-aware neural architecture search for mobile.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for finding neural architecture which, through the use of selective branching, can avoid processing portions of the network on a per-data-point basis. 

While the reviewers felt that the idea proposed was technically interesting and well-presented, they had substantial concerns about the evaluation that persisted post-rebuttal, and lead to a consensus rejection recommendation.
Decision: Reject