Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose neural channel expansion (NCE), a neural architecture search (NAS) and quantization method. Existing NAS+Q methods typically search for the architecture of the DNN along with the precision at each layer, maximizing accuracy while respecting some kind of hardware constraint. The result is a DNN with mixed-precision, which is challenging for most existing hardware (which only support one or a few precisions). NCE keeps precision the same in each layer, and instead uses the precision sensitivity signal in the NAS to adjust the width of the layer (expand or shrink). The result is uniform-precision, hardware-friendly DNN.

NCE works by first training normally (with quantization) for 40 warmup epochs, followed by 110 epochs of search. At the end of each search epoch, NCE adjusts the channels search parameter in each layer. A few experiments show convincingly that a wider layer is indeed less sensitive to quantization. Thus using the sensitivity-to-quantization signal to adjust layer width is a good idea.

Experiments on CIFAR-10 show NCE can boost quantized accuracy at 2w2a by up to 0.8%, and in the case of VGG16 trim unnecessary params. On ImageNet there is also some accuracy improvement, though only a little bit over LSQ. And for ResNet-50 NCE again can reduce param size.

The paper is that the idea is fairly simple, and the results are not too impressive. LSQ seems to already do very well and it also uses uniform quantization. One major issue I have with the comparison in Tables 1 and 2 is that, on some of the smaller networks (ResNet-32 for CIFAR and ResNet-18 for ImageNet) the NCE result has more params than the other methods. This is potentially unfair as a larger network is almost always more accurate. I think you should uniformly increase channel widths in at least the "w/o NCE" baseline to see if the accuracy boost is from NCE learning the layer sensitivities or just from a bigger model.

The results on larger networks (VGG and ResNet-50) is much more compelling, showing that NCE can trim unnecessary params while improving accuracy. More results like this would make the paper more convincing. I would also like to see exactly which layers were reduced in size on these networks.

Another issue is that despite being a NAS work, there aren't NAS baselines in the comparison. I understand that NCE only requires one training run while the original NAS required many retrainings. But I believe HAQ (Wang et al 2019) and DARTS (Liu et al 2018) are both NAS techniques for mixed-precision quantization that require only one training run. The authors should include a comparison against such methods or discuss why it isn't needed.

Minor issues:
 - Section 4.3.2 Typo: "results of the 2X case are inferior to the 2X case"
 - Table 2, ResNet-18, you highlighted your own result but LSQ seems to be better in accuracy and param size?

EDIT: Raised score from 4 to 6 after the authors clarified some points and added additional experiments.
review_title: Good idea, but simple. Results comparison has flaws.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Overview
In this paper, the authors proposed selectively channel pruning and expansion via neural architecture search to make an off-the-shelf neural network more robust to quantization. Under 2-bit quantization, the proposed method outperforms the original model at similar or smaller FLOPs and model size.

### Clarity
Overall, the paper shows the concept clearly. The method itself is not complicated. But some definitions are missing to make the paper self-contained. For example, the implementation of channel-wise interpolation (CWI) is not provided, and not referred to existing literature. The presentation of Figure 2(a)(b) makes it hard to see the trend. It would be better to plot the temporally averaged curve (potentially with std) for better visualization.

### Pros
1. The paper is generally well written and clearly expressed.

2. It focuses on an important topic: neural architecture search under (low-bit) model quantization, which has been less explored before. The paper shows that considering the quantization during the search can improve the final accuracy.

3. The proposed method shows good results. On ImageNet, the optimized model outperforms the original version at similar or lower cost after quantization.

### Cons
1. My first concern is the novelty. As acknowledged by the authors, improving quantization with channel expansion has been widely studied [Mishara et al. 2018, Zhao et al. 2019, Park&Choi 2019] and proves to be effective. The authors claimed that "none of the approaches shed light on understanding the benefit of the increased number of channels when a network is trained for quantization", which I think is an overclaim. The reduction of dynamic range from increased channels is already well demonstrated in previous work (e.g., Zhao et al. 2019); while the authors verify the conclusion again in this work. The author did prove that the modified/expanded architecture works well when trained from scratch, but it is not a very surprising finding since networks with higher capacity are more robust to quantization.
As for the NCE method itself, it is like an extension of TAS by allowing expansion in search space and under quantization. The core novelty seems to the extended search space with channel expansion, but the other parts are largely the same. 

2. A trivial alternative would be to apply TAS using the enlarged search space (Sec 4.3.2) with theoretically the same optimal solution. The ablation study shows that the larger search space might hinder successful optimization, leading to worse results. I think using 16 search parameters makes the optimization too difficult; a fair comparison would be to still use 8 search parameters, with channel configurations using larger strides to compare the accuracy, so that we can exclude the factor of architecture search difficulty and focus on the search space design (which is the main contribution of the paper).

3. Regarding the experimental results, it outperforms the original model quantization using various methods. However, the authors did not compare to existing results that modify/expand the network architecture for better quantization accuracy (e.g., Zhao et al. 2019). Without comparing to existing methods on the same topic, it would be difficult to evaluate the contribution.

4. The author mentioned that the expanded architecture itself is beneficial for quantization. For results in Table 2, if we train the optimized architecture from scratch, will we still have the same advantage?

5. Minor question: it is unclear how the batch normalization layers are processed. Are they fused into convolutions or kept as float-point? 


I hope to see the authors' feedback for the final evaluation. Thanks!

EDIT: Raised score from 4 to 6 after the reading authors' response. The response clarifies some of the novelty issues, and it clearly shows the advantage compared to previous methods like TAS. However, I still have concerns about the novelty; the insight why the proposed method is better than TAS is still not very clear to me. I hope the author can further improve the draft for the final version.
review_title: Review for Uniform-Precision Neural Network Quantization via Neural Channel Expansion

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors propose neural channel expansion (NCE) to adjust the network structure to compensate for the performance degradation from uniform-precision quantization. Given a hardware constraint, the proposed NCE selectively expands the width for the quantization sensitive layers. Experiments on CIFAR-10 and ImageNet shows the effectiveness of the proposed NCE. However, the novelty of this paper is limited since the proposed method is just an extension of TAS. Besides, the comparisons between NCE and the existing methods are missing. My detailed comments are as follows. 

Positive points:
1. Based on TAS, the authors propose a neural channel expansion (NCE) method to search for the width across the layers under uniform-precision quantization.

2. The authors provide an analysis of the proposed NCE on the effect of channel expansion for compensation of quantization errors.

3. Experimental results on CIFAR-10 and ImageNet demonstrate that NCE is able to improve the performance of the quantized networks.

Negative points:
1. The novelty of this paper is limited. The authors only make minor modifications to TAS [1] and apply it to network quantization. However, neither of the modification on TAS or the application of channel expansion on network quantization is significant. 

2. This paper is quite similar to the work of Shen et al. [2], which also explores the optimal layer width for quantization. Thus, it would be better to compare the proposed method with Shen et al. ’s. 

3. As mentioned in Section 2, there are several existing works [3][4][5] that reducing the quantization error by increasing the widths of the network. More comparison between the proposed method and the existing works would make this paper more convincing.

4. In Section 4.1, the authors claim that “The substantial increase of STDEV for W2A2 implies that large quantization errors would occur when input activation is quantized”. However, Figure 1a only provides the results of W2A2 and W32A32. Thus, it cannot illustrate whether the large quantization error results from the quantization of activation or not. At least, the results of W32A2 and W2A32 should be provided.

5. In Section 4.1, the authors argue that improved performance is resulted from the reduction of input activation’s dynamic range according to Figures 1a and 1c. However, the STDEV concerning “2X ResNet20 (W32A32)” is missing in Figure 1a. To compare the dynamic range reduction of ResNet20 with 1x and 2x width, more results regarding the STDEV of “2x ResNet W32A32” would make this paper more convincing.

6. As discussed in Section 4.1, the improved performance comes from the reduction of activation’s dynamic range. However, it is not clear how the dynamic range of activation impacts model performance. It is better to give more theoretical analyses and explanations regarding the results.

7. The authors only apply NCE to 2-bit quantization. It will be better if the authors would provide results on more bitwidth quantization, e.g., 4-bit or 1-bit.

8. In Algorithm 1, the threshold T is important to the proposed method. According to the experimental settings on CIFAR-10 and ImageNet, the chosen of threshold T is tricky. More results on different values of T are required. 

9. In the first line of Algorithm 1, the n(D_val) should be n(D_arch).

References

[1] Xuanyi Dong et al. Network pruning via transformable architecture search. NeurIPS 2019.

[2] Mingzhu Shen, Kai Han, et al. Searching for accurate binary neural architectures. ICCV Workshop 2019.

[3] Asit Mishra et al. Wrpn: wide reduced-precision networks. ICLR 2018.

[4] Ritchie Zhao et al. Improving neural network quantization without retraining using outlier channel splitting. ICML 2019.

[5] Hanmin Park et al. Cell division: weight bit-width reduction technique for convolutional neural network hardware accelerators. ASPDAC 2019.

review_title: Official Blind Review #1

Meta-review: This paper suggests a NAS approach for quantization which focuses on expanding the number of channels in problematic layers, given some uniform quantization level for all the layers. The reviewers were initially all negative, but the authors added more experiments and the scores changed to borderline (6/6/5). I think that though the reviewers appreciated the effort made by the authors and clarifying many of the issues. Yet, I think two concerns still remained. First, the method is demonstrated convincingly in only one large scale case (ImageNet) - on ResNet50 W2A2. The other case (ResNet18) is less impressive since it uses more parameters and has a small gap from EdMIPS. I think using more architectures (and possibly also precision levels) is important to convincingly demonstrate the utility of the method (e.g. EdMIPS used 5 architectures). Second, the novelty of the method in comparison to the previous method was not completely clear. Though the authors added more experiments on CIFAR to compare with previous methods, the significance of the results is not clear (small differences in the figures, and no error bars), and also the explanation of why the results of NCE should be better than TAS.
Decision: Reject