Title: No title found

Abstract: No abstract found

:



Figures and Tables:

Peer review (iclr2020):
This paper introduces an approach to jointly pruning and quantizing a deep network so as to satisfy a global budget constraint. This is formulated as an optimization problem with a constraint involving the number of nonzero elements in the network weights and the bitwidth to encode these nonzero elements. A solution to this problem based on the ADMM is proposed. Methodology: - In general, I like the idea of considering both pruning and quantizing in a single framework. My concern regarding the method is linked to the fact that several approximations are used to derive the solution: * The problem is non-convex, and thus the ADMM does not offer much in terms of convergence guarantees (as acknowledged by the authors below Eq. 7); * When updating W, the authors use a quadratic approximation of the true objective function (Eq. 5); * Because the resulting problem remains NP hard, a greedy algorithm is used to approximately solve it (Section 3.3); * Solving for the auxiliary variables V also yields an NP hard problem, which is solved approximately (Section 3.4). It is unclear what these successive approximation entail when it comes to the final solution. As a matter of fact, as acknowledged by the authors at the end of Section 3.4, the final W and V do not truly satisfy the constraints, and thus W is then quantized in a post-processing stage. Experiments: - The experiments show the good behavior of the proposed algorithm. However, the comparison to existing methods is quite incomplete, in the sense that different baselines appear in different tables. In particular, considering that CLIP-Q is initially described as the closest method, I would appreciate seeing its results in more than one table. - It also seems to me that a natural baseline would consist of fixing the number of bits, e.g., to 2 or 3, and then perform pruning in a similar manner as done here so as to satisfy the budget. I wold highly appreciate seeing the results of this baseline, to get a better understanding of the importance of jointly pruning and quantizing, and the benefits of allowing different bitwidths for different parameters. - Considering that the authors also need to perform a post-processing quantization step, it would also be interesting to provide the results of a baseline doing pruning only, and then applying quantization with the same post-processing step. - Detail: From the text, it is not entirely clear to me how the budget was set in the experiments. The number in the tables also do not suggest a very clear budget. Related work: - Table 1 only shows 3 baselines, with a single one performing joint pruning and quantization. In the experiments, however, the authors mention 3 additional joint pruning and quantization methods (Han et al., 2015, Louizos et al., 2017, Ye et al., 2018b). Why aren*t these methods in Table 1 and discussed in more detail? - Note that other methods have proposed to use budget constraints and could serve as baselines (maybe not the 3rd one as the constraints are on the energy): * Chen et al., Constraint-aware Deep Neural Network Compression, ECCV 2018; * Chai et al., Towards the Limit of Network Quantization, ICLR 2017; * Yang et al., Energy-constrained Compression..., ICLR 2019; Summary: I have mixed feelings about this paper. While I like the idea, the proposed solution relies on many approximations, the influence of which is not studied. The experiments show that the method works reasonably well, but several important baselines are missing. I therefore feel that the paper is not yet ready for publication.