Batch-shaping for learning conditional channel gated networks
Authors: Babak Ehteshami Bejnordi
Tijmen Blankevoort
Max Welling
Venue:  (2020)
Keywords: Conditional computation
channel gated networks
gating
Batch-shaping
distribution matching
image classification
semantic segmentation

Abstract: We present a method that trains large capacity neural networks with significantly improved accuracy and lower dynamic computational cost. This is achieved by gating the deep-learning architecture on a fine-grained-level. Individual convolutional maps are turned on/off conditionally on features in the network. To achieve this, we introduce a new residual block architecture that gates convolutional channels in a fine-grained manner. We also introduce a generally applicable tool batch-shaping that matches the marginal aggregate posteriors of features in a neural network to a pre-specified prior distribution. We use this novel technique to force gates to be more conditional on the data. We present results on CIFAR-10 and ImageNet datasets for image classification, and Cityscapes for semantic segmentation. Our results show that our method can slim down large architectures conditionally, such that the average computational cost on the data is on par with a smaller architecture, but with higher accuracy. In particular, on ImageNet, our ResNet50 and ResNet34 gated networks obtain 74.60% and 72.55% top-1 accuracy compared to the 69.76% accuracy of the baseline ResNet18 model, for similar complexity. We also show that the resulting networks automatically learn to use more features for difficult examples and fewer features for simple examples.

Published as a conference paper at ICLR 2020
BATCH-SHAPING FOR LEARNING CONDITIONAL CHANNEL GATED NETWORKS
Babak Ehteshami Bejnordi, Tijmen Blankevoort & Max Welling Qualcomm AI Research' Amsterdam, The Netherlands (behtesha, t jmen, mwel 1 i ing) @qt qua comm. com
ABSTRACT
We present a method that trains large capacity neural networks with significantly improved accuracy and lower dynamic computational cost. We achieve this by gat- ing the deep-learning architecture on fine-graines mnee-rainnd-levell Individual convolutional maps are turned on/off conditionally on features in the network. To achieve this, introduce new residual block architecture that gates convolutional channels in a fine- the-grained manner. We also introduce a generally applicable tool batch-shaping that matches the marginal aggregate posteriors of features in neural network a pre-specified prior distribution. We use this novel technique to force gates to be more conditional on the data. We present results on CIFAR- 10 and ImageNet datasets for image classification, and Cityscapes for semantic segmentation. Our results show that our method can slim down large architectures conditionally, such that the average computational cost on the data is on par with a smaller architecture, but with higher accuracy. particular, on ImageNet, our ResNet50 and ResNet34 gated networks obtain 74.60% and 72.55% top-1 accuracy compared to the 69.76% accuracy of the baseline ResNet 8 model, for similar complexity. We also show that the resulting networks automatically learn to use more features for difficult examples and fewer features for simple examples.
INTRODUCTION
Almost all deep neural networks have a prior that seems suboptimal: All features are calculated all the time. Both from generalization and an inference-time perspective, this is superftuous. For example, there is no reason to compute features that help differentiate between several dog breeds, if there is no dog to be seen in the image. The necessity of specific features for classification performance depends on other features. We can make use of this natural prior information, improve our neural networks. We can also exploit this to spend less computational power on simple and more on complicated examples.
This general idea is commonly encapsulated in the terms conditional computing (Bengio 2013)
or gating architectures (Sigaud et al. (016)). It is known that models with increased capacity, for example increased model depth (He et al.] (016) or width (Zagoruyko & Komodakis (016)), generally help increase model performance when properly regularized. However, models increase in size, so do their training and inference times. This often limits practical applications of deep learning. By conditionally turning parts of the architecture on off we can train networks with very large capacity while keeping the computational overhead small. The hypothesis is that when training conditionally gated networks, we can train models with better accuracy/computaan feed-forward counterparts. In addition, conditional computation with channel gating can potentially models. For example, gating can allow us to identify input patterns that trigger the response of single unit in the network.
cost trade-off than their fully
increase the interpretability
Several works in recent literature that successfully learn conditional architectures, for example, ConvNet-AIO (Veit & Belongie, 2018) and dynamic channel pruning (Gao et al. 2018). However, the conditionality is often very coarse as in ConyNet-AIG (Veit & Belongie] 2018), or the amount of actual conditional features learned is very minimal in Gaternet (Chen al. 2018). We attempt to solve both. Our contributions are as follows:
"Qualcomm Al Research initiative of Qualcomm Technologies Inc.

---

Published as a conference paper at ICLR 2020
We propose a fine-grained gating architecture that turns individual input and output convolutional maps on or off, leading to features that are individually gated. This allows for a better trade-off between computation cost and accuracy than previous work.
We propose a generally applicable tool named batch-shaping that matches the marginal aggregated posterior of feature in the network a specified prior. Depending on the chosen prior, networks can match activation distributions to e.g. the uniform distribution for better quantization, or the Gaussian to enforce behavior similar to nonaaaiormaiiitioonn (loff & Szegedy 2015). Specifically, in this paper, we apply batch-shaping to help the network learn conditional features. We show that this helps performance by controlling the gates to be more conditional on the input data at the end of training.
We show state-of-the- -rrt results compared to other conditional computing architectures such as Convnet-AIG (Veit & Belongie 2018). SkipNet (Wang et al. 2018a), Dynamic Channel Prun- ingo et al. 2018), and soft-guided craptively-dropper neural network (Wang et al. 2018b).
BACKGROUND AND RELATED WORK
Literature on gating connections for deep neural networks dates back to Hinton (198T). Gating can be seen as tri-way connection in neural network (Droniou al. 2015), where output can only be 0 and 1. These connections have originally been used to learn transformations between images with gated Restrieted Boltzmann Machines in Memisevic & Hinton (2007). One of the earliest works to apply this to create sparse network architectures is that of a Mixture of Experts proposed by Jacobs et al. (1991).
Several compression methods exist that statically reduce model complexity. Tensor factorization methods (Jaderberg al. 2014] Zhang al. (016) decompose single layers into more efficient bottleneck layers. Methods such as channel-pruning (He et al. 2017 Molchanov et al., 2017) remove entire input/output channels from the network. Similarly, full channels can be removed during training as in VIBnets (Dai et al. 2018). Bayesian Compression (Louizos et al. 2017) and LO-regularization (Louizos et al. 2018). These methods reduce the overall model capacity while keeping the accuracy as high as possible. Our method allows for higher model capacity while keeping inference times similar to the papers cited above.
Some networks exploit the complexity of each input example to gain performance. Networks such as Branchynet (Teerapittayanon et al. 2016) and Multi-scale dense net (Huang et al. 2018) have early exiting nodes, where complex examples proceed deeper in the network than simpler ones. Our approach also assigns less computation to simpler examples than more complicated examples but has no early-exiting paths. Both methods of inducing less computation can work in tandem, which leave for future work.
Several works focus on adaptive spatial attention for faster inference. Figurnov et al. (2017) proposed a residual network based model that dynamically adjusts the number of executed layers for different regions of the image. Difficulty-awar region convolutions were proposed (Li et al.) 2017) as an irregular convolution, which allow operating the convolution on specific regions of a feature map. Similarly, tiling-based sparse convolution algorithm was proposed (Ren et al.. 2018) which yields significant speed-up in terms of wall-clock time.
Other works exploit similar conditional sparsity properties of networks as this work. ConvNet- AIG (Veit & Belongie| 2018) and SkipNet (Wang et al., 2018a) turn full residual blocks on or off, conditionally dependent on the input. Dynamic Channel Pruning (Gao et al.] (018) turns individual features or off similar to our approach, but they choose the top-k features instead, akin to Outrageously large neural networks (Shazeer et al.) (017). This approach loses the benefit of being able to trade-off compute for simple and complex examples. Gaternet (Chen et al.) 2018) trains completely separate network to gate each individual channel of the main network. The overhead of this network is not necessary for learning effective gates, and we show better conditionality of the gates than is achieved by this paper, at almost no overhead.
BATCH-SHAPING
First, we introduce sathhsshaping, a general method to match the marginal aggregated posterior distribution over any feature in the neural network to a pre-specified prior distribution. In the next paragraph, we will use this to train gates that fire more conditionally, but we see a large potential value of this tool for many other applications such training auto-encoders or network quantization.
2

---

Published as a conference paper at ICLR 2020
1,0
Prku CDF Empeazi CDD
0.6
0.5
07 01
0.2
0.0
(a)
(b)
Figure 1: Patch-shaping loss. (a) Illustration of the computation of the batch-shaping loss. (b) The output distribution for four gates and their shapes over different epochs. We can see that initially gates are firing conditional pattern, but after removing the shaping loss and introducing the Lo-loss they may become fully active, stay conditional or turn off completely.
Consider parameterized feature in a neural network X@). The intention have (+) distributed more according to a chosen probability density function (PDF) f(z). while still being able to differentiate with respect to 0. To do this we consider the Cramér-von-Misse criterion by Anderson (1962), which is a statistical distance between the cumulative distribution function F(x) and an empirical cumulative distribution function Fv (z). The Cramér-von-Mise criterion lends itself naturally to this use-case. Other frequently used statistical distance functions, such as KL-divergence, require calculating histogram of the samples to compare to f(x), which does not allow for gradients to propagate. As we will loss function. The Cramér-von-Minn criterion is given by:
derive gradients with respect to each sample with the proposed
[Fx(x) F(x:)) dF())
We consider batches of N samples U1.N drawn from (0). In order optimize this we follow Anderson 1962). Sorting sort(Z1:N) = zi.N; replacing (z with dFN (x) and normalizing with N gives us the batch-shaping loss to minimize
N Σ 1=1 N+1 F(xt)
S(a* A)
(2)
N
where À a parameter that controls the strength of the loss function. This approach is shown visually in Figure Mh. We can sum this loss for each considered feature in the network to attain full network batch-shaping loss. Note that we can differentiate v7.v with respect to through the sorting operator by keeping the sorted indices. In the backward pass, if a value with index i was sorted to index put the error from position j in position i. This makes the whole loss term differentiable as long as the chosen CDF function is differentiable. The pseudo-code for the implementation of the Batch-Shaping loss is presented the Appendix A
We use this loss to match the marginal aggregated posterior of feature in the network to any PDF. For example, if we want to encourage our activations to be Gaussian, we could use the CDF of the Gaussian in the loss function. This could be useful for purposes similar to batch-normalization. Alternatively, the CDF can be that of the uniform distribution, which could help with fixed- -oint quantization Jacob et al. (018)). We leave this for future work, and only consider a loss function that encourages the conditionality of gates in the next section.
4 CHANNEL GATED NETWORKS
In this section introduce gating network. While the basic structure of gating network could be any kind of CNN structure, we use ResNet (He et al., (016)) as the basic structure. Figure 2 shows overview of channel gated ResNet block. Formally, ResNet building block is defined as:
L141 = ((F(x)) + az),
3

---

Published as a conference paper at ICLR 2020
Gating module
Conv Dm
l.d. Gumbet sampless
Ave. pooling
MLP
XXWXC
1x1xc
1x1XC
Ingut feature map
Figure Illustration of our channel gated ResNet block and the gating module.
where 2ạ Rd xw and Ø1+1 Re xur x denote the input and output of the residual block, and r is the activation function, in this case ReLU (Nair & Hinton 2010) function. The residual function F(xx) is the residual mapping to be learned and is defined as W2 r(Wi x). Here * denotes the convolution operator. W1 Rd xxx 1+1 x is set of c1 1+1 filters, with each filter of size k k. Similarly, W2 R "+1
xx''xxxxxxxx "x x After each convolution layer, batch normalization (loff & Szegedy 2015) used. Our gated residual block is defined as:
#1+1 = r(2 (G(z)) rW1 * xx)) + 21);
where G a gating module and G(x)) = [n 92 93. ... Je, st, is the output of the gating function, where Go E {0, I]: 0 denotes skipping the convolution operation for filter in Wi, and denotes computing the convolution. Here refers to channel-wise multiplication between the output feature map r(W1 z)) and the vector G(z)). The more sparse the output of (zzz), the more computation we can potentially save.
We chose the position of the gate for two specific reasons. Firstly, the gate is applied after the ReLU activation. This prevents the convolution from updating if the gate is off. Placing the gating function before the ReLU caused unstable training behavior. Secondly, the two layers in the residual block. We allow each block to use the full input and update the full output. The network only gales each feature that determines how to update the incoming representation. We tested applying multiple gating setups, including before and after each convolutional layer. The proposed setup performed significantly better.
only gate the representation between
4.1 GATING MODULE
To enable a light-weigh weight gating module design, we squeeze global spatial information in z into a channel descriptor input to our gating module, similar to ConvNet-AIG and Squeeze-and-excine nets /eii & Belongie 2018 Hu et al. 2018). This is achieved via channel-wise global average pooling.
For our gating module, we use a simple fordaforward design comprising of two fully connected layers, with only 16 neurons in the hidden layer. We apply batch normalization and ReLU the output of the first fully connected layer. The second fully connected layer linearly projects the features to unnormalized probabilities ti, k {1,2, cq [11.. We define the logits
In(TT))
Our gating module is computationally inexpensive and has an additional overhead that between 0.018% 0.087% of ResNet block multiply-accummllte (MAC) usage.
To dynamically select a subset of filters relevant for our current input, we need to map the output of gating module binary vector. The task of training binary-valued gates is challenging because we can not directly back propagate through a non- con-differentiable gate. In this paper. we leverage recently proposed approach called Gumbel-Softmax sampling (Jang et al. 2017 Maddissnn al.) 2017) to circumvent this problem. We consider the binary case of the Gumbel-Mar trick, the Binary concrete relaxation BinConcrete((n r). In the forward pass we use the discrete argmax and for the backward pass we use a sigmoid function with temperature: o (x) ( We use 7 2//3 in all of our ex expermments as suggested by Maddison et al. (2017).
4.2 BATCI-SHIAPING BETA DISTRIBUTION PRIOR FOR CONDITIONAL GATES
When initially training the channel-wise gating architecture, many features were trained to be only on off in the first few epochs. Instead, would like feature to be sometimes and sometimes off
4

---

Published as a conference paper at ICLR 2020
for different data points, to exploit the potential for conditionality. We regulate this by applying the batch-shaping loss, with the CDF of Beta distribution (a b), as a prior on each of the gates.
We set 0.6 and 0.4 in experiments, initially inducing 40% sparsity. The Beta distribution will regularize gates towards being sometimes on and sometimes off for different data points, pushing the gates towards the desired batch-wise conditionality. We apply this loss with strong coefficient À at the start of training to encourage activations to be conditional, and gradually anneal À as training progresses. This allows the network to learn different levels of sparsity or even undo the effect of batch-shaping if necessary. Figure 1b presents the output distribution of four gates during training.
4.3 Lo-LOSS
Our batch-shaping loss encourages the network to learn more conditional features. However, our actual intention is to find model that has the best trade-off between (conditional) sparsity and our task loss (e.g., cross-entropy loss). For the second part of our training procedure, we add a loss that regularizes the complexity of the full network explicitly. We use method proposed by Louizos et al.) (2018), which defines a Lg regularization process for neural network sparsification by learning a set of gates with a sparsifying regularizer. Our work can be considered as a conditional version of the Lo gates introduced in this paper, sans stretching parameters. Hence this loss term is natural choice for sparsifying the activations. We use a modified version of the La-loss without stretching, defined as:
de
Lc = Σ (In(xi))
(S)
where A is the total number of gates, is the sigmoid function, and is a parameter that controls the level of sparsification we want to achieve.
It is essential to mention that introducing this loss too early in training can reduce effective network capacity and potentially hurt performance. If the training procedure is not carefully chosen, the procedure often degenerates into training smaller architecture, as full convolutional maps turned off prematurely. Thus, in all our experiments, we introduce this Lo-loss after some delay, and we use warm-up schedule described in [Snndrby et al. (2016)
EXPERIMENTS
We evaluate the performance of our method on two image classification benchmarks: CIFAR-10 (Krizhevsky) 2009) and ImageNet (Russakovsky et al.) 2015). We additionally report preliminary results on the Cityscapes semantic segmentation benchmark (Cordts et al., 2016). For CIFAR-10, we use ResNet20 and ResNet32 architectures (He al. 2016)) our base model. For ImageNet, use ResNet18, ResNet34, and ResNet50. We compare our algorithm with competitive conditional computation methods. We additionally perform experiments to understand the learning patterns of the gates and whether they specialize to certain categories. For semantic segmentation, we employ the pyramid scene parsing network (PSPNet) (Zhao et al. 2017) with ResNet-50 backbone.
The training details and hyperparameters for our gated networks trained on CIFAR1O, ImageNet, and Cityscapes are provided in the appendix B from scratch for all of our models.
The base networks and the gates were trained together
We applied the batch-shaping loss with beta distribution prior from the start of training with coefficient of À = 0.75 and linearly annealed it to zero until epoch 100. Next, the Lo-loss was applied to the output of the gates starting from epoch 100, and the coefficient was linearly increased until epoch 300 where it was kept fixed for the rest of the training. For the Ly-loss we used values of (0, 1, 2, 5, 10, 15. 20)- 10 generate different trade-off points. We also experimented with only using the batch-shaping loss (no Lo-loss) with a fixed X = 0.75 for the entire training.
CIFAR-10:
We compare our batch-shaped channel gated ResNer20 and ResNet32 models, hereafter referred to as ResNer20-BAS and BesNet32-BAS, with other adaptive computation methods: ConvNet- AIG (Veit & Belongie 2018), SkipNet (Wang et al., 2018a), and SGAD (Wang et al.) 2018b). As shown in Figure 3al ResNer20-BAS and ResNet32-BAS outperform SkipNet38, GGAD-ResNet32 and ConvNet-AIC variants of ResNet20 and ResNet32 by a large margin. Our results show that given deep network such as ResNet32, the input) to a value equal or lower than that of a ResNet20 architecture and still achieve better performance. Ideally, gated ResNer32 model should outperform gated ResNer20 model at the
can reduce the average computation (conditioned on
5

---

Published as a conference paper at ICLR 2020
di
Car oal
on 650
HAC (k(0))
MAC (0CP)
VAC 12285
(a) Results on CIFAR-10
(b) Results on ImageNet
(c) Ablation results on CIFAR-10
Figure 3: Comparison of the results of our algorithm and competing methods (a) CIFAR-10 and (b) ImageNet datasets. (c) shows the effect of the batch-shaping loss on our ResNer20 and ResNet32 gated models trained CIFAR-IO. also presents the effect of increasing the network's width.
same average computation. This property is evident in our results. However, the competing methods show performance equal to or lower than their smaller sized counterparts, indicating one may instead train smaller model from scratch.
The highest accuracy obtained by ResNet20-BB BBB and ResNet32-BAS BAS in Figure Sajrelate to the case where only used the batch-shaping loss with fixed X. Despite the high accuracy, this setting leads to a minor reduction in computation costs. We also experimented with changing the batch-shaping distribution be more sparse (e.g. changing prior over time). However, the performance degraded significantly. The Lo-loss, in contrast, provided a better trade-off between accuracy and MAC saving.
larger dataset, applied our gating networks to ImageNet. Similar to CIFAR classification, we introduce the batch-shaping loss from the start of the training with = 0.75 and linearly annealed it zero until epoch 20. Lo-loss was then applied to the output of the gates starting from epoch 30, and the coefficient was linearly increased until epoch 60 where was kept fixed for the rest of the training. We used y values of {0. 2,2,5 5, 10, 15, 20, 30, 40) 10-2 to generate different trade-off points. We also experimented with only using the batch-shaping loss with fixed À 0.75 for the entire training.
ImageNet: To evaluate the performance of our models
Compared to CIFAR-10, the performance difference with the baseline is larger for ImageNet, likely because of the more substantial complexity of the dataset allowing for more conditionality of the features. We also frequently seen in compression literature because of extra regularization, e.g., as in Frankle & Carbin (2019).
increased performance for lower 'compression rates', similar to what is
Figure shows the trade-off between the computation cost and Top-1 1 accuracy for our gated network, ConvNet-AIG, and ConvNet-FBS. FBS (Feature Boosting and Suppression) (Gao al. 2018). The results indicate that our ResNet-BAS models consistently outperform corresponding ConyNet-AIG and ConyNet-FBS models. Similar to the observations CIFAR-10, the performance of ConyNet-AIG34 degrades to level lower than that of ConvNet-AIG188, at the same computation cost. Our models, in contrast, make better use of dynamic allocation of features by learning more conditional features. Subsequently, when the average computation cost is on par with ResNet18, 8, our ResNet50-BAS and ResNet34-BAS BAS gated networks achieved 74, 74.40% and 72.55% top-1 accuracy compared the 70.57% best accuracy of ResNet18- 8-Bee and 69.76% accuracy of the baseline ResNet1 8 model.
Similar to CIFARIO experiments, the highest accuracy obtained by batch-shaper models in Figure relates to the case where we only used the batch-shaping loss with a fixed X.
Semantic segmentation: For this experiment, we only used our batch-shaping loss with a fixed coefficient. The original PSP network achieves an overall IoU (intersection over union) of 0.706 with a pixel-level accuracy 0.929 on the validation set. Our gated PSPNet model was able to accomplish an loU of 0.719 and pixel accuracy of 0.935 while using 76.3% of the MAC count (A = 0.2) of the original PSP model. We additionally compared the models when starting training using imageNet-pretraane pretrained weights initialize the ResNet-50 base network. In this setting, PSPNet achieved an IoU of 0.739 and pixel accuracy of 0.9446. Our gated-model, in comparison, obtained an IoU of 0.744 and pixel accuracy of 0.946 using 76.5% 5 of the PSPNet MAC count (A 0.2). The

---

Published as a conference paper at ICLR 2020
Figure 4: Images with highest MAC usage (top row) and lowest MAC usage (bottom row) from the Cityscapes datasets. The network uses more features for difficult examples and fewer features for simple examples
performance of our gated network further reaches an IoU of 0.747 and pixel accuracy of 0.948 using 95% of the PSPNet MAC count (A = 0.05). Figure shows example images from the cityscapes dataset consuming the highest and lowest MAC counts.
5.1 EFFECT OF INCREASING WIDTH OF THE NETWORK
To assess the effect of increasing the network's width rather than depth, we trained ResNet20 models on CIFARIO with per-block width increased with factors of 10X and 20X. The results are shown in Figure 3c As can be seen, by strongly increasing the number of parameters of the network and allowing for dynamic selection of such parameters, the network consistently achieves higher accuracy.
5.2
MODEL INFERENCE-TIME COMPARISON
We compared the inference time of our models to Convnet-AIG (Veit & Belongie 2018) on CPU in the same setting (see Table II for batch-size of one. For batch-computing, custom convolutional kernel could calculate mask on the fly. We simulated this for the GPU in the same table. Custom hardware would give us benefits from all MACs saved, including energy savings. See appendix Co further details of our timing setup.
Model
GPU (ms)
CPU (ms)
Params (lotal)
MACs (full)
Top Aee
ResNet18
0,46 + 1.0e-5
88.7 8.6e-4
11.6M
L8IG
0.697
ConvnetA1G34
0.71 1: 4.3e
123.7 1: 0.18
19.04M (2L85M)
2.73 (3.66G)
0.722
RosNet34-BAS
5.5e-5
0.22
9.15M (219IM)
L67G* (3.68G)
ResNet34
0.92 +: 3.0e-5
149.9 6.5e-4
21.79M
3,66G
0.733
ConvactA1G34 (Full)
0.89 +: 5.8e-5
137.75
+ 0.24
21.4MM (2L85M)
3,526* (3.66G)
0.732
ResNeL34-BAS BAS (Full)
0.73 1: 1.1e44
111.1
1: 0.36
17.77M (21.9MM)
2.22* (3.68))
0.740
ResNec50
L.55 +: 3.0c-5
184.05 + 1.8c-4
25 35.55M
4,09G
0.761
ConvnetAIG50
1: 4.2e-4
21.97M (26.56M)
3.090* (4.09G)
ResNet50 BAS
1.20 1: 3.4e-4
139.82
1 0.757
15.31M (26.72M)
2.07 (4.11G)
0.757
Table 1: Inference-tine time results. Models compared at roughly the same accuracy. Batch-size cpu: gpu:128. *Average per example usage on ImageNet val set. CPU uses tensor-slicing, GPU uses custom kernel simulation.
5.3 EFFECT OF THE BATCH-SHAPING LOSS
To validate the effectiveness of our proposed batch-shaping loss, we compare the performance of our ResNet-BAS networks in two settings: 1) using both the batch-shaping loss and the Lu complexity loss for training the model similar to the experiments above, or 2) only using the Lo complexity loss.
As can be seen in Figure 3c the models that additionally use the batch-shaping loss consistently outperform the ones using only the Lo complexity loss. ResNer20-LO and ResNet32-LO trained on CIFARIO 10 appear to have more rapid accuracy degradation than the models trained using the batch-shaping loss. However, architectures. This can be attributed to our specific channel gated network architecture which allows for fine- fine-grained dynamic selection of channels or filters in layer, as compared to the models which are designed to skip whole layers. Very importantly, as shown in Figure 33c, by gating the larger non- BOS ResNet32/38 models, do not any improvement over the smaller ResNet20 model
Lo-gated models still outperform Con and SkipNet
7

---

Published as a conference paper at ICLR 2020
666
Alaay en galoo
Neass an pales
100
100
Miree 2012 -lasing MrC
1431 82188-Baaa -aaigg
(a) Gated ResNe(34-LO
(b) Gated ResNet34-BAS
Figure The distribution of different gate activation patterns in our ResNeG3--LO and ResNe(34-BAS models trained on ImageNet while inducing 60% sparsity. Gates are categorized as always on/off, if they are on/off for more than 99% of the inputs.
Block number
14
42
Persian cat
Killer whale
Whale vs cat fiter firing difference
36
35
02
Labrador retriever
Golden retriever
Labrador vs golden fiter firing difference
Figure 6: The histogram shows how often individual filters are executed in each layer of a ResNet block (column). For illustration purposes, filters are sorted by execution frequency over the entire validation set. This histogram is computed for our ResNei34-BAS model.
at similar computation cost. This is in sharp contrast to our ResNet34 BAS model. We observe similar pattern in ImageNet results (See Figure in appendix DD..
5.4 GATE DISTRIBUTION
Analyzing the distribution of the learned gates gives us a better insight into the characteristics of the learned features of the network. Ideally, expect three main gate distributions to appear in network with dynamic computations: 1) Gates that are always on. We expect certain filters in network to be of key importance for all types of inputs. Gates that fire conditionally on/off based on the input. The filters that are more input dependent and hence are more specialized for certain categories can be dynamically selected to executed based on the input. These types of filters are desirable as they contribute to saving computation the inference phase and formation of conditional expert sub-networks inside our network. Therefore, we would like to maximize the learning such filters. 3) Gates that always off. This introduces complete sparsity.
Figure 5 shows the distribution of gates on the ImageNet validation set for our ResNet34-BAS and ResNet34-LO models. We observe that the majority of the gates activate conditionally for our ResNet34-B BAS model. This model prefers conditional sparsity over fully turning gates off. ResNer34-LO0 model achieves the same sparsity level by fully turning off large number of gates.
8

---

Published as a conference paper at ICLR 2020
(a)
(b)
(c)
Figure 7: Illustration of image categories that activate individual gates in different layers of ResNet34-BAS, For visualization, we only considered gates that are barely on they are more specialized and activate for a more specific subset of categories. (a-c) show all possible categories that activate individual gates in the 5th, 10th, and 12th gated ResNet blocks of this model, respectively.
Figure 6 presents an illustration of the fine-grained execution of filters in our gated networks for different image categories. As can be seen, the histograms show small differences in gate execution patterns between similar classes, and large differences between distinct classes.
Figure shows categories that trigger individual gates different layers of ResNet34-BAS network. Overall, we observe that there are gates at different layers of the network that are activated for more specific subset of categories. For example, Figure 7 shows that the activation of a gate in the fifth ResNet Block depends on the appearance of large creatures in the water.
6 CONCLUSION
In this paper, we presented a fine-grained gating architecture that enables conditional computation in deep networks. Our gating network achieves wate-of-theearr accuracy among competing conditional computation architectures on CIFARIO 10 and ImageNet datasets. In both datasets, given model with large capacity, our gating method was the only approach that could reduce the inference computation to value equal or lower than that of lower capacity base network, while obtaining a higher accuracy. On ImageNet, our ResNet50-EE -AS and ResNet34-BB -ANS improve the accuracy by more than 4.8% and 2.8% over a ResNet18 8 model at the same computation cost.
We also proposed a novel patch-shaping loss that can match the marginal aggregated posterior of feature in the network to any prior PDF. We use it to enforce each gate in the network to be more conditionally activated at the start of training and improve performance significantly. We look forward to seeing many novel upplications for this loss in the future, for e.g.. autoencoders, better quantized models, and as an alternative to batch- normalization. Another important future research direction that benefit from our fine-grained gating architecture is continual learning. Designing gating mechanisms that can dynamically decide to allow or prevent the flow of gradients through certain parts of the network could potentially mitigate catastrophic forgetting. Finally, with our gating setup, we could distill smaller sub-networks that work on only subset of the trained classes.
ACKNOWLEDGMENTS
The authors would like to thank Markus Nagel, Pim de Haan, and jakub tomczak for their valuable discussions and feedback.
REFERENCES
T. W. Anderson. On the distribution of the two-sample cramér-von mises criterion. The Annals of Mathematical Statistics, 148-)11148,115, 1962. ISSN 00034851. URL htttt::/wwet jstor
Yoshua Bengio. Deep learning of representations: Looking forward. In Proceedings of the First Inter- national Conference on Statistical Language and Speech Processing, SLSP' 13, pp. 1-37, Berlin, Heidelberg. 2013. Springer Verlag. ISBN 395-2-9-----2-999----5 doi:
URL /tRlT//ttp//ddee.....o///0011110
642 39593-2-2
Liang-Chis Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834-848, 2017.

---

Published as a conference paper at ICLR 2020
Zhourong Chen, Yang Li, Samy Bengio, and Si Si. Gaternet: Dynamic filter selection in convolutional neural network via a dedicated global gating network. arXiv preprint arXiv:1811.11200 2018. URL http:h/atxiiii org/ abs/ /asss1811 1t1//1111111111 11205
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Bin Dai, Chen Zhu, Baining Guo, and David Wipf. Compressing neural networks using the variational information bottleneck. In Proceedings of the 35th International Conference Machine Learning. ICMI. 2018, Stockholmsmässsn, Stockholm, Sweden, July 10-15, 2018, pp- 1143-1152, 2018.
.mr..mmrr..... URL http:: httL:tttpprroeeediniig http://s.........eeee///////////
html
Alain Droniou, Serena Ivaldi, and Olivier Sigaud. Deep unsupervised network for multimodal
perception, representation and classification. Robot. Auton. Syst. 71(C):83-98, September 2015. ISSN 0921 -8800. doi: 10. 10.101666.moon ro..6/..obb...11..11..
htlphttp:/ddd wow..doi..... wtw.o/oo...........0/.....
j.robot 2014 11. 005.
Michael Figurnov, Maxwell Collins, Yukun Zhu, Li Zhang. Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1039-1048, 2017.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https
//openreview wwwp:eeeiwwweee/Eeeoo.eee forum? ? re-iid-------- -
Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert Mullins, and Cheng-Zhong Xu. Dynamic channel
pruning: Feature boosting and suppression. arXiv preprint arxiv:/810.05331, 2018. http?
//axxiv.org/abs..
05331
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778, 2016. doi: 10.1009/CVPR.2016.990 URL
https: //tp::://ooi.o www.ooogooo....... 09 CVPR. 2016.99
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 1398-1400 1406, 2017, doi: 10.1109//CCV.2017.1555. URL /tttss:/doii.oo
1109/ /CCV/CCCV.o 2017. 155
Geoffrey F. Hinton. A parallel computation that assigns canonical object-based frames of reference.
In Proceedings of the 7th International Joini Conference Artificial Intelligence Volume 2, IJCAI'81, pp. 683-685, San Francisco, CA, USA, 1981. Morgan Kaufmann Publishers Inc. URL http:// // acm. org/ cit//iiie citaatiinn www.ttttttt......ooooooooo....
cfm? ctw.immmmmmmmlmmmm 6236622322222 623282)
Jie Hu, Shen, and Gang Sun. gqueeze-and-exciteee conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
networks. In Proceedings of the IEEE
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Weinberger. Multi-scale dense networks for resource efficient image classification. In International Confer- ence on Learning Representations, 2018. URL https: httpss///ppeeeeeiieee net wtw//formmm forum?id
Hk2aImxAD
Sergey loffe and Christian Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on International Conference Machine Learning-Volume 37, pp. 448-456. JMLR. org, 2015.
Benoit Jacob, Skinmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithment arithmetic-only inference. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
10

---

Published as a conference paper at ICLR 2020
Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Comput., 3(1):79-87, March 1991. ISSN 0899-7667. doi: 10. 162/neco. 1991.3.1.79, URL http: dei org/ wwtpp:/pww......oooooooooooo. www.o//enneccc..oo.oo wtt.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. CoRR, abs/ 405.3660,., 2014.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparamettizzation with gumbel-softmax. In Proceedings International Conference on Learning Representations 2017. OpenReviews.net, April 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Xiaoxiao Li, Ziwei Liu, Ping Luo, Chen Change Loy, and Xiaoou Tang. Not all pixels are equal: Difficulty-aware semantic segmentation via deep layer cascade. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3193-3202, 2017.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv:1312.4400, 2013.
Christos Louizos, Karen Ullrich, and Max Welling. Bayesian compression for deep learning. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, Vish- wanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems 30, pp. -228-3299. Curran Associates, Inc., 2017. URL httL:tt::// /papers//pprr....///// wwtpscccppapp.eeeee 6921 bayesian compression for deep learning. .ddf
Christos Louizos, Max Welling, and Diederik Kingma. Learning sparse neural networks through 10 regularization. In International Conference on Learning Representations, 2018. URL https
//openeeveww.o
?idmmidd-Hii H1Y
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In 5th International Conference Learning Representa- tions, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. URL net www.fforrmiidoe SE1151
hitts://openrewye
Roland Memisevic and Geoffrey Hinton. Unsupervised learning image transformations. pp. 1-8, 07 2007. ISBN 1-4244-1180-7, doi: 10.1109/CVPR.200..833000
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In ICLR, 2017.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
Yurii E Nesterov. A method for solving the convex programming problem with convergence (1/k* 2). In Dokl. akad. nauk Sssr, volume 269, pp. 543-547, 1983.
Mengye Ren, Andrei Pokrovsky, Bin Yang, and Raquel Urtasun. Sbnet: Sparse blocks network for fast inference. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 871 1-820, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-252) 2015. doi: 10.1007/s1 12770000122--11
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- gated mixture-of-expers layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24- 26, 2017, Conference Track Proceedings, 2017. URL hollhttps:. http::::///ppeevviii hwtp///pnnreeeeee/op/pp/e. nwt/ um? dd=
11

---

Published as a conference paper at ICLR 2020
Olivier Sigaud, Clément Masson, David Filliat, and Freek Stulp. Gated networks: an inventory. Unpublished manuscript, 17 pages, May 2016. [RRttttps://hal http://aa...cccaeeseeeeeeeee....
Er ft/hal-01113601
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. How train deep variational autoencoders and probabilistic ladder networks. In Proceedings of the 33rd International Conference on Machine Learning (ICML 2016), 2016.
Teerapittayanon, McDanel, and H. T. Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2464-2469, Dec 2016. doi: 10.1 11.1109/CPPP.R 2.CCCP...1..7900000 6,00000.
Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. 2018.
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E. Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In Computer Vision ECCV 2018 15th Euro- pean Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIII, pp. 420- 436, 2018a. doi:
URL
378-3-030-01261-8225
Zhisheng Wang. Fangxuan Sun, Jun Lin, Zhongfeng Wang. and Bo Yuan. Sgad: Soft-guided adaptively-dropped neural network. arXiv preprint arXiv: 1807:01000,004300 2018b.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith (eds.), Proceedings of the British Machine Vision Conference (BMVC), pp. 87.1-87.12. BMVA Press, September 2016. ISBN 1-901725-59-6, doi: 10.5244/C. 30.87. httpss///dx doi www.oogg/o.... 5244//10..24444 5244/2.443.330 30.87
Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional net- works for classification and detection. IEEE Trans. Pattern Anal. Mach. Intell., 18(10))191111111955 2016. doi: 10.110/TPAM.20055550777 URL 101
10.
2015 2502579
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2881-2890, 2017.
12

---

Published as a conference paper at ICLR 2020
7 APPENDIX
A PSEUDO CODE FOR THE IMPLEMENTATION OF THE BATCH-SHAPIIN LOSS
The pseudo code for the forward and backward pass the batch-shaping loss. Note that the terms idr sort: Ppdf+ Peaf= and sort be computed and stored in the forward pass and retrieved in the backward pass.
Batch- Batch-Shaping loss
input :aating output, X. loss factor Beta distribution parameters 8, batch size N
Backward pass of the Batch-Shaping loss
input :Sorted indices id Ppaf Peay, and
Eosy
output: for that gating module
output :hh gradient of the loss with respect to the input
loss =0
for i <- 1 to do
for + to m do
Exort sort(X;
Pedf BetaCDF sort, at,
grad 2pd (Eess p.oay)
Codf Arrange(1 (1, + NN/(N
grad undo-sort(gradd
loss = loss + sum sum(Fodt Codf Deaf)
end
end
return grad
return y loss
B TRAINING DETAILS
B.I CIFARIO
We trained all models using Nesterov's accelerated gradient descent Nesterov (1883) with a mo- mentum 0.9 and weight decay factor 5e 54, No weight decay was applied on the parameters of the gating modules. We used a standard data-augmentation scheme by randomly cropping and horizontally flipping the images Lin et al. (2013). We trained the models for 500 epochs with a mini-batch of 256. The initial learning rate was 0.1 and it divided by 10 epoch 300, 375, and 450.
B.2 IMAGENET
We used similar optimization settings to CIFAR-I0 with a weight decay factor of le-4, We used standard luggenngmnnttoion scheme adopted from He et al. (2016) and trained the model for 150 epochs with mini-batch size of 256. All models were trained using single GPU. The initial learning rate of 0.1 was divided by 10 at epoch 60, 90, and 120.
B.3 CITYSCAPES
Cityscapes Cordts et al. (016) is a dataset for semantic urban scene understanding including 5,000 images with high quality level- annotations and 20,000 additional images with coarse annotations. There are 19 semantic classes for evaluation of semantic segmentation models in this benchmark. For data augmentation,
adopt random mirror and resize with a factor between 0.5 and 2 and use crop-size of 448 x 672 for training. We train and test with only single-scale input and run inference on the whole image. The PSPNet network with the ResNet-50 back-end with a mini- batch size of 6 for 150k iterations. Momentum and weight decay are set to 0.9 and le respectively. For the learning rate we use similar policy to Chen et al. (2017) where the initial learning rate of 2e is multiplied by (1 stereurrend snurret//tter..... /fiter
trained from scratch
U.. We used the same settings for training gated PSPNet. Weight decay for the layers in the gating units was set to 16
C DETAILS MODEL TIMING
The output of the gates are computed before the convolutional operations inside resnet block. The output fed to the preceding and following convolutional layer kernels, to not compute the masked inputs/outputs. In many practical settings, images are fed to network by one (c.g. real-time inference). In this case, the convolutional weights can loaded from DDR memory to local compute memory conditionally. Computation can be done sliced tensors, which implemented in Pytorch.
13

---

Published as a conference paper at ICLR 2020
Top-1 Accuracy
8.77
6.70
Resket baselines
Conuner
148
Conuner AG11
ResNet34
ResNett1 Lo
ReaNet34 nss (curs)
ReaNettE-Be BAs (curs)
(x10P)
Figure 8: Comparison of the effect of the batch-shaping loss and LO loss on our ResNet18 and ResNet34 gated models trained on ImageNet
For batch-computing, custom convolutional kernel could the calculated mask the fly. We simulated this for the GPU the same table. Custom hardware could naturally give us benefits from all MACs saved, including energy savings. All the reported inference times were measured using machine equipped with an Intel Xeon E5-1620 v4 CPU and an Nvidia GTX 1080 Ti GPU.
C.I CPU MEASUREMENTS
Consider W1 R "n xkok and W2 E R Xeg x representing the weight tensors of the first and second layers in ResNet block, where C₁ out c²" For each ResNet block, first use the output of the gates to generate a mask. Using this mask, we slice the original weight tensor of the first layer in the block and apply conv2d on the input featuremap using the sliced weight tensor W1 E Re sellie xkxk, We next apply masking for the first batch normalization layer. The input to the second layer is featuremap with lower number of channels. Using the same mask, we slice the weight tensor of the second layer W2 R Xeg out xkok and apply the conv2d layer using this tensor.
C.2 GPU MEASUREMENTS
For the GPU measurements, we first recorded the gating patterns of the entire images in the validation For each input image, a sparse model (with fewer number convolution kernels in each layer) was defined based on the gating pattern. The computation time was then reported for the sparse model. The overhead caused by the gating modules included the wall-time calculation.
COMPARISON THE EFFECT OF Lo AND BATCH SHAPING LOSS IMAGENET
As can be seen in Figure g the models that additionally use the batch-shaping loss consistently outperform the ones using only the Lo complexity loss. However, similar to CIFAR- observations, our Lo-gated models still outperform AoIG
14


Reviews:
[{"review_id": "rJe55cygqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper studies conditional channel gated networks. The network is designed to disable certain channels depending on the inputs. This can be used to save computation. The idea is built on top of “Convolutional Networks with Adaptive Inference Graphs.” The authors propose the technique of batch shaping to encourage the marginal statistics of the gating to be selective to different inputs. With similar inference time, the gated network can achieve better accuracy it can afford adding more layers in the network.\n\nDetailed comments:\n- The results show quite significance difference compared to ConvNet-AIG, which demonstrates that batch-shaping is very helpful. Table-1 show 3% increase in accuracy compared ResNet-18 by using similar inference time. It is good that the paper reports wall-clock time measurements.\n\n- It’s good to see some visualizations in the paper, including the image samples and gate locations. I recommend to move Figure 7 to the main paper. A regular neural network can also be used to visualize the sensitivity of patterns of specific neurons. What would be the qualitative differences?\n\n- 1-2x reduction in MAC is not super impressive, especially taking into consideration of the overhead for gathering the active channels for convolution. \n\n- Figure 3a) plot is cut off on the right. The baselines only have a single point in the plot, I guess it is also valid to simply add/remove layers in the baseline models to generate a curve in the plot.\n\n- ResNet-50-L0 is missing in Figure 3b). It would be better if the plots can be grouped better. Currently there are too many lines and it is hard to understand the differences.\n\n- It would be good to see comparisons to some other alternatives to batch shaping. For example, one can penalize so that the average value is around 0.5 by using a L1 loss |E(x) – 0.5|.\n\n- The ImageNet experiment has a very complicated set-up, where L0 loss is applied in the middle of the training. Is this necessary? How important is this step? What would happen if L0 loss is not applied in ImageNet? And what would happen if L0 loss is applied from the beginning? Why is L0 loss not applied in other experiments (e.g. CIFAR or Cityscapes), will L0 loss be beneficial on these benchmarks as well?\n\n- There are a number of related works on adaptive spatial attention for faster inference, which can be included in the related work section.\n1) M.  Figurnov,  M.  D.  Collins,  Y.  Zhu,  L.  Zhang,  J.  Huang,D.  P.  Vetrov,  and  R.  Salakhutdinov. Spatially  adaptive computation  time  for  residual  networks. CVPR, 2017.\n2) X. Li, Z. Liu, P. Luo, C. C. Loy, and X. Tang.  Not all pixelsare equal:  Difficulty-aware semantic segmentation via deeplayer cascade. CVPR, 2017.\n3) M. Ren, A. Pokrovsky, B. Yang, R. Urtasun. SBNet: Sparse Blocks Network for Fast Inference. CVPR, 2018.\n\nConclusion: The batch shaping technique introduced in this paper has significant improvement on networks that exploit conditional inference. Further understanding of the effect of L0 loss and other alternative loss function is recommended. My overall rating is weak accept.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SygoXqOR5r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper's focus is on conditional channel-gated networks. Conventional ConvNets process images by computing all the filters, which can be redundant since not all the filters are necessary for a given image. To eliminate this redundancy, this work aims at computing a channel gating on-the-fly, to determine what filters can be turned off. The core contribution of the paper is to propose a \"batch-shaping\" technique that regularizes the channel gating to follow a beta distribution. Such regularization forces channel gates to either switch on or off. Combined with l_0 regularization, the proposed training technique improves the performance of channel gating: ResNet trained with this technique can achieve higher accuracy with lower theoretical MACs. \n\nOverall, the paper proposes a simple yet effective trick for training gated networks. The paper is well written, and experiments are sufficient in demonstrating the effectiveness of the method. \n\nThe main concern for the paper is whether such granular control on the convolution filters can be practically useful. For Conventional ConvNets whose computation is fixed regardless of the input, scheduling the computation on the hardware static and therefore can be easily optimized. When it comes to dynamic networks, especially at such a granular level, it is not clear whether the theoretical complexity reduction can directly translate to actual efficiency (such as latency) improvement. In section 5.2, the author mentions \" We simulated this for the GPU in the same table.\". Can you elaborate on how you \"simulated\" the GPU time? How is the simulation done? How well does it predict the actual implementation? Can you implement an efficient kernel for this and show the actual speedup? For the CPU runtime, can you explain in more detail the experimental setting? Can you report the actual latency improvement against theoretical FLOP reduction? For the result in Table 1, why the result of the original ResNet50 is not reported? ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SJepfyO6Fr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a method to train a network with large capacity, only parts of which are used at inference time in an input-dependent manner. This leads to accuracy gains without an increase in inference cost. Fine-grained conditional selection is done, using gating of Individual convolutional feature maps. A new method termed “batch shaping” to regularize the network to encourage that the features are used conditionally is introduced and combined with additional regularizer adapted from prior work.\n\nThere has been a large body of work along the same research direction. Few of the prior works have focused on fine-grained selection of features, and the ones that have, such as Gao et al, have used a fixed number of features (top-k) across examples instead of dedicating more computation to more difficult examples. In addition, the current work outperforms related prior work through the use of the new regularization technique (batch shaping).\n\nThe paper contains thorough comparison to related prior works on three datasets. It also ablates the contribution of the separate aspects of the method -- the fine-grained gating, the batch shaping regularizer, and the L0 penalty. The results demonstrate the all of these aspects contribute to improvements over prior work and result in good accuracy/efficiency trade-offs.\n\nAlthough this research is not a large departure from prior work, the novelty of the batch shaping regularizer, the thorough empirical study, the experimental gains, and the clarity of the paper makes this a solid contribution.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper describes a method to train a convolutional network with large capacity, where channel-gating (input conditioned) is implemented - thus, only parts of the network are used at inference time. The paper builds over previous work, with the main contribution being a "batch-shaping" technique that regularizes the channel gating to follow a beta distribution, combined with L0 regularization. The paper shows that ResNet trained with this technique can achieve higher accuracy with lower theoretical MACs. Weakness of the paper is that more engineering would be required to convert the theoretical MACs into actual running time - which would further validate the practicality of the approach.

Decision: Accept (Poster)