Title: No title found

Abstract: No abstract found

:



Figures and Tables:

Peer review (iclr2020):
While network pruning & quantization have been studied intensively in the literature, this paper presents a regularizer that can simultaneously achieve the above two goals. The authors further propose an optimization algorithm based on ADMM for training. Experimental results demonstrate great compression ratios compared with various baseline methods. Overall, I suggest authors have better placement of their novelties in the literature, more justifications on ADMM and stronger baselines in experiments. Q1. *automatically according to a target model size without using any hyper-parameters to manually set the compression ratio for each layer*. - Personally, I do not feel this claim is proper. Indeed, by setting S_{budget} in (1), one does not need to care about hyper-parameter at each level separately. However, this does not mean all other methods have to set such hyper-parameters in a layer-wise manner. For example, one can add a sparse regularizer and then set the overall sparse penalty, like *Sparse Convolutional Neural Networks. ICCV 2015*. - Besides, I do not think the method works in an automatical way. 1) S_{budget} needs to be set and 2) how rho (for ADMM) is tuned is not clear. Q2. It is good that authors organize related work in *pruning*, *quantization* and *AutoML*. It will also be good if authors can organize all tables in this manner as well. For example, in Table 2, which category does BC-GNJ belongs to? In this way, it will be easier for readers to understand why the proposed method can be better than all compared methods. Q3. Could the authors offer model size in all tables? - Personally, I do not feel comfortable to see such a huge compress ratio in, e.g., Table 4. While 205x can be impressive, the real reduction in model size can be small. - Could authors offer STD in tables for small data sets, e.g., Table 4? I am no sure the reduction in memory size is significant, e.g., between *Ye et al 2018b and Ours*. Q4. Since the authors have done a search in methods from three directions, i.e., *pruning*, *quantization* and *AutoML*, could the authors compare with the more recent or stronger state-of-the-art? - While 2120x in Table 2 can be impressive, it is not meaningful if the based network is too weak (i.e., LeNet on MNIST). Perhaps, authors can search on networks identified by NAS methods, e.g., *Efficient Neural Architecture Search via Parameter Sharing*. If authors can achieve the same/or slightly less compressing ratio on these networks, I will definitely give acceptance for this submission. - For Table 4, while MobileNet can be a popular choice, MnasNet can be a better network. Q5. Could the authors offer the learning curves, i.e., epoch v.s training/validation loss? - It is better to show how the proposed algorithm converges, and how fast it converges. Q6. Could the authors show the impact of rho? - There is no convergence guarantee of ADMM on such a problem, the choice of rho can be important. - How the accuracy/sparsity/learning curves change w.r.t. rho? Q7. Could authors offer more discussion with Ye et al 2018b? - A combination of *pruning* and *quantization* using ADMM is already explored in this paper.