Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper's focus is on conditional channel-gated networks. Conventional ConvNets process images by computing all the filters, which can be redundant since not all the filters are necessary for a given image. To eliminate this redundancy, this work aims at computing a channel gating on-the-fly, to determine what filters can be turned off. The core contribution of the paper is to propose a "batch-shaping" technique that regularizes the channel gating to follow a beta distribution. Such regularization forces channel gates to either switch on or off. Combined with l_0 regularization, the proposed training technique improves the performance of channel gating: ResNet trained with this technique can achieve higher accuracy with lower theoretical MACs. 

Overall, the paper proposes a simple yet effective trick for training gated networks. The paper is well written, and experiments are sufficient in demonstrating the effectiveness of the method. 

The main concern for the paper is whether such granular control on the convolution filters can be practically useful. For Conventional ConvNets whose computation is fixed regardless of the input, scheduling the computation on the hardware static and therefore can be easily optimized. When it comes to dynamic networks, especially at such a granular level, it is not clear whether the theoretical complexity reduction can directly translate to actual efficiency (such as latency) improvement. In section 5.2, the author mentions " We simulated this for the GPU in the same table.". Can you elaborate on how you "simulated" the GPU time? How is the simulation done? How well does it predict the actual implementation? Can you implement an efficient kernel for this and show the actual speedup? For the CPU runtime, can you explain in more detail the experimental setting? Can you report the actual latency improvement against theoretical FLOP reduction? For the result in Table 1, why the result of the original ResNet50 is not reported? 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Summary: This paper studies conditional channel gated networks. The network is designed to disable certain channels depending on the inputs. This can be used to save computation. The idea is built on top of “Convolutional Networks with Adaptive Inference Graphs.” The authors propose the technique of batch shaping to encourage the marginal statistics of the gating to be selective to different inputs. With similar inference time, the gated network can achieve better accuracy it can afford adding more layers in the network.

Detailed comments:
- The results show quite significance difference compared to ConvNet-AIG, which demonstrates that batch-shaping is very helpful. Table-1 show 3% increase in accuracy compared ResNet-18 by using similar inference time. It is good that the paper reports wall-clock time measurements.

- It’s good to see some visualizations in the paper, including the image samples and gate locations. I recommend to move Figure 7 to the main paper. A regular neural network can also be used to visualize the sensitivity of patterns of specific neurons. What would be the qualitative differences?

- 1-2x reduction in MAC is not super impressive, especially taking into consideration of the overhead for gathering the active channels for convolution. 

- Figure 3a) plot is cut off on the right. The baselines only have a single point in the plot, I guess it is also valid to simply add/remove layers in the baseline models to generate a curve in the plot.

- ResNet-50-L0 is missing in Figure 3b). It would be better if the plots can be grouped better. Currently there are too many lines and it is hard to understand the differences.

- It would be good to see comparisons to some other alternatives to batch shaping. For example, one can penalize so that the average value is around 0.5 by using a L1 loss |E(x) – 0.5|.

- The ImageNet experiment has a very complicated set-up, where L0 loss is applied in the middle of the training. Is this necessary? How important is this step? What would happen if L0 loss is not applied in ImageNet? And what would happen if L0 loss is applied from the beginning? Why is L0 loss not applied in other experiments (e.g. CIFAR or Cityscapes), will L0 loss be beneficial on these benchmarks as well?

- There are a number of related works on adaptive spatial attention for faster inference, which can be included in the related work section.
1) M.  Figurnov,  M.  D.  Collins,  Y.  Zhu,  L.  Zhang,  J.  Huang,D.  P.  Vetrov,  and  R.  Salakhutdinov. Spatially  adaptive computation  time  for  residual  networks. CVPR, 2017.
2) X. Li, Z. Liu, P. Luo, C. C. Loy, and X. Tang.  Not all pixelsare equal:  Difficulty-aware semantic segmentation via deeplayer cascade. CVPR, 2017.
3) M. Ren, A. Pokrovsky, B. Yang, R. Urtasun. SBNet: Sparse Blocks Network for Fast Inference. CVPR, 2018.

Conclusion: The batch shaping technique introduced in this paper has significant improvement on networks that exploit conditional inference. Further understanding of the effect of L0 loss and other alternative loss function is recommended. My overall rating is weak accept.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The paper describes a method to train a network with large capacity, only parts of which are used at inference time in an input-dependent manner. This leads to accuracy gains without an increase in inference cost. Fine-grained conditional selection is done, using gating of Individual convolutional feature maps. A new method termed “batch shaping” to regularize the network to encourage that the features are used conditionally is introduced and combined with additional regularizer adapted from prior work.

There has been a large body of work along the same research direction. Few of the prior works have focused on fine-grained selection of features, and the ones that have, such as Gao et al, have used a fixed number of features (top-k) across examples instead of dedicating more computation to more difficult examples. In addition, the current work outperforms related prior work through the use of the new regularization technique (batch shaping).

The paper contains thorough comparison to related prior works on three datasets. It also ablates the contribution of the separate aspects of the method -- the fine-grained gating, the batch shaping regularizer, and the L0 penalty. The results demonstrate the all of these aspects contribute to improvements over prior work and result in good accuracy/efficiency trade-offs.

Although this research is not a large departure from prior work, the novelty of the batch shaping regularizer, the thorough empirical study, the experimental gains, and the clarity of the paper makes this a solid contribution.

review_title: Official Blind Review #2

Meta-review: The paper describes a method to train a convolutional network with large capacity, where channel-gating (input conditioned) is implemented - thus, only parts of the network are used at inference time. The paper builds over previous work, with the main contribution being a "batch-shaping" technique that regularizes the channel gating to follow a beta distribution, combined with L0 regularization. The paper shows that ResNet trained with this technique can achieve higher accuracy with lower theoretical MACs. Weakness of the paper is that more engineering would be required to convert the theoretical MACs into actual running time - which would further validate the practicality of the approach.

Decision: Accept (Poster)