Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper targets on learning the number of channels across all layers, under computation/model size/memory constraints. The method is simple and the results seems promising. 

However, the following issues need to be resolved:
1. The main method is based on published "slimmable networks," such that the novelty is limited;
2. The method is very simpler to DropPath in [1], which uses DropPath to learn important branches while this paper uses it to learn channels. They are similar.
3. Better ablation studies are required in Table 1. This table should be simplified. As the method cannot learn architectures but channel numbers, the only useful pairs of comparisons are those having the same architecture, such as  a pair of MobileNet vs AutoSlim-MobileNet.
4. an important detail is missing: where does the AutoSlim start from? Does it start from a larger model than the baseline? In the set of "500M FLOPs" experiments, I see the size of "AutoSlim-MobileNet v1" (4.6M) is larger than "MobileNet v1 1.0x" (4.2M), this implies that AutoSlim start from a "MobileNet v1 Nx" and N > 1.0. What is exactly N?
5. If AutoSlim starts from a larger baseline model with N times (N > 1.0) width, then the pruning baseline methods (AMC and ThiNet) should also start from the same larger models for fair comparison. In general, starting from a larger model and pruning it down can achieve a better accuracy vs. size trade-off.
6. "300 epochs with linearly decaying learning rate for mobile networks, 100 epochs with step learning rate schedule for ResNet-50 based models", are baselines trained in the same way?

Minor: 
1. missing captions in a couple of figures, e.g., Figure 5.
2. "the importance of trained weights" vs "the importance of channel numbers" is trivial


[1] Bender, Gabriel, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. "Understanding and simplifying one-shot architecture search." In International Conference on Machine Learning, pp. 549-558. 2018.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper proposes a simple and one-shot approach on neural architecture search for the number of channels to achieve better accuracy. Rather than training a lot of network samples, the proposed method trains a single slimmable network to approximate the network accuracy of different channel configurations. The experimental results show that the proposed method achieves better performance than the existing baseline methods.

- It would be better to provide the search cost of the proposed method and the other baseline methods because that is the important metric for neural architecture search methods. As this paper points out that NAS methods are computationally expensive, it would be better to make the efficiency of the proposed method clear.

- According to this paper, the notable difference between the proposed method and the existing pruning methods is that the pruning methods are grounded on the importance of trained weights, but the proposed method focuses more on the importance of channel numbers. It is unclear to me why such a difference is caused by the proposed method, that is, which part of the proposed method causes the difference? And how does the difference affect the final performance?
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors propose a method to perform architecture search on the number of channels in convolutional layers. The proposed method, called AutoSlim, is a one-shot approach based on previous work of Slimmable Networks [2,3]. The authors have tested the proposed methods on a variety of architectures on ImageNet dataset. 

The paper is well-written and easy to follow. I really appreciate the authors for structuring this paper so well. I have the following questions:

Q1: In figure 4, the authors find that “Compared with default MobileNet v2, our optimized configuration has fewer channels in shallow layers and more channels in deep ones.” This is interesting. Because in network pruning methods, it is found that usually later stages get pruned more [1] (e.g. VGG), indicating that there is more redundancy for deep layers. However, in this case, actually deep layers get more channels than standard models. Is there any justification for this? Is it that more channels in deep layers benefit the accuracy?

Q2: In “Training Optimized Networks”, the authors mentioned that “By default we search for the network FLOPs at approximately 200M, 300M and 500M, and train a slimmable model.” Does this mean that the authors train the final optimized models from scratch as a slimmable network using “sandwich rule” and “in-place distillation” rule? Or are the authors just training the final model with standard training schedule? If it is the first case, can the authors justify why?

Q3: In Table 1, “Heavy Models”, what is the difference between “ResNet-50” and “He-ResNet-50”? Also, why the params, memory and CPU Latency of some networks are omitted?

Q4: In the last paragraph of section 4, the authors tried the transferability of networks learned from ImageNet to CIFAR-10 dataset. I am not sure how the authors transfer the networks from Imagenet to CIFAR-10? Is it the ratio of the number of channels? Can the authors provide the architecture details of MobileNet v2 on CIFAR-10 dataset?

Q5: What is the estimated time for a typical run of AutoSlim? How does it compare to network pruning methods or neural architecture search methods?

Q6: Can the methods be used to search for the number of neurons in fully connected layers? Are there any results?

[1] Rethinking the Value of Network Pruning. Zhuang et al. ICLR 2019
[2] Slimmable neural networks. Yu et al. ICLR 2019.
[3] Universally Slimmable Networks and Improved Training Techniques. Yu et al. Arxiv.

review_title: Official Blind Review #1

Meta-review: The paper presents a simple one-shot approach on searching the number of channels for deep convolutional neural networks. It trains a single slimmable network and then iteratively slim and evaluate the model to ensure a minimal accuracy drop. The method is simple and the results are promising. 

The main concern for this paper is the limited novelty. This work is based on slimmable network and the iterative slimming process is new, but in some sense similar to DropPath. The rebuttal that PathNet "has not demonstrated results on searching number of channels, and we are among the first few one-shot approaches on architectural search for number of channels" seem weak.
Decision: Reject