Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The authors proposed a matrix factorization-based method for deep neural network weight compression. The weight tensors are factorized as two low precision quantized matrices, out of which one is sparse. 
main_review: 1. I request the authors to discuss a significant low-rank factorization technique used for deep neural network gradient compression---PowerSGD by Vogels et al. NeuRIPS, 2019. PowerSGD uses power iteration to decompose the original gradient matrix $M$ into two $r$-rank matrices $P$ and $R$. This work is relevant and similar to the present work and requires mentioning. 

2. In the expected loss minimization problem, what is $m$? Is it the number of observations in the training set? I did not find it before. 

3. I have questions about solving the problem (10). How in (10), the authors arrive at quantized, $C_q, Z_q$? By using gradient descent, one can reach the factors, $(C, Z)$, and then it requires further projection by using quantization operations (as defined in (7)) to find quantized, $C_q, Z_q$. Running gradient descent algorithm directly on (10) does not guarantee quantized, $C_q, Z_q$. In the present form, it is misleading, and the treatment and evolution of the problem throughout the draft are highly faulty.  

4. **Severely faulty mathematics.** Proposition A.1 is not correct. Please allow me to explain. The projection is not on a convex set; projection onto the set of sparse matrices (by using hard threshold operation) is a nonconvex projection. Moreover, the function that the authors defined in (10) is the same as the function, $f(x)$ defined in (14), which is a nonconvex function. First of all, for nonconvex functions proving convergence in terms of the optimum is impossible because one cannot guarantee the existence of a global minimizer $x_*$. Although the authors applied the projected gradient descent technique to nonconvex optimization, the convergence proof techniques used in the convex case do not apply directly to nonconvex problems. Therefore, the projection claim that the authors provided is mathematically incorrect. Please note that the claim that the authors made about the stationary point is erroneous. Moreover, the convergence rate with the derivation of the L-Lipschitz to justify their faulty analysis in Proposition A.1, is severely flawed. For (local linear) convergence of gradient descent with nonconvex projection, please see [1,2,3]. 

5. SVD stands for Singular Value Decomposition. Therefore, “SVD decomposition” does not make any sense. 

[1] Lewis et al. Local linear convergence for alternating and averaged nonconvex projections. Foundations of Computational Mathematics, 9(4):485–513, 2009.
[2] Lewis and Malick. Alternating projections on manifolds. Mathematics of Operations Research, 33(1):216–234, 2008.
[3] Dutta et al., A Nonconvex Projection Method for Robust PCA, AAAI, 2019. 




summary_of_the_review: I did not read the paper after Section 3.1.1. The article is erroneous and the results are misleading. Please refrain from misleading the readers. This alone makes me recommend a STRONG REJECTION of the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a novel method of weight compression. Weight tensors are stored as sparse, quantized matrix factors, and the underlying matrix factorization problem can be considered as a quantized sparse PCA problem and be solved through iterative projected gradient descent methods. The authors' method  is applicable to both moderate and extreme compression regimes, and is claimed to achieve or be on par with state-of-the-art trade-offs between accuracy and model size.
main_review: Strengths:
The authors empirically demonstrate that their method is the only method to achieve SOTA compression-accuracy trade-offs in both low and high compression regimes. 

Weakness:
1. The method seems to be a simple combination of tensor factorization and vector quantization methods. The idea of using (sparse) PCA in vector quantization methods has also been studied, e.g., in Babenko & Lempitsky (2014). Therefore, the novelty in this submission seems to be quite limited. 
2. The authors should provide more explanations about the motivation and benefits of assuming the factor matrix $\mathbf{Z}$ to be sparse. In addition, the authors should clarify that Z is sparse in what sense? For example, row-sparse, column-sparse, or sparse while considering $\mathbf{Z}$ as a $(kn)$-dimensional vector? In (8), the norm $\||\cdot\||_0$ is used without any explanation. 
3. The authors should be careful to claim that their method for factorizing $\tilde{\mathbf{W}}$ in (1) is sparse PCA (ignoring the quantization). For conventional sparse PCA, the factor matrix $\mathbf{C}$, instead of $\mathbf{Z}$, is assumed to be column-sparse.
4. The gradient descent update on (10) is important and should be clearly presented in this submission, rather than simply citing another paper. 
5. All source codes required for conducting experiments should be included in a code appendix.

Other comments:
1. The terminology "quantized sparse PCA" should be fixed, and the authors should not also use "sparse quantized PCA" arbitrarily (e.g., in the second dot point of the contributions). 
2. The relation $k < d < n$ should be made clear when these parameters first appear in Section 3.1 (instead of mentioning it in Section 3.2). 
3. In Algorithm 1, line 5, equation 10 should be equation (10). In Appendix A, problem 14 should be problem (14).
summary_of_the_review: Although the method proposed by the authors seems to be empirically promising, it seems to be a direct combination of existing methods, and the novelty seems to be limited. The authors provide no theoretical guarantee/intuition for their method. In addition, some terminologies/notations are vague or confusing. My impression is that this submission requires some major revisions.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a method for compression of neural network weights. The proposal turns weight tensors into matrices, factorizes these matrices into a rank-k factorization via PCA, applies quantization to the factor matrices, and additionally makes the right (latent) matrix sparse. An algorithm is presented, with two different thresholding options (per-iteration, or one-shot). In experiments, these ideas lead to an accuracy/compression tradeoff which is either competitive with or better than previous state-of-the-art across all compression ratios investigated.
main_review: # Strenghts
S1. The paper is clearly written and the proposed method and results are convincing. The method is simple yet seems to work well.

S2. The topic of weight compression should be of interest to a large proportion of the deep learning/ML community.

S3. Although I’m not familiar with the literature in this area, the authors seem to do a good job of discussing related work (in Sec. 2) and later on also discussing how their proposal relates to other previous methods (in Sec. 3.1.2).

# Weaknesses
W1. I mentioned the simplicity of the method as a strength. However, the simplicity of the method comes from the fact that it just combines several previous ideas (PCA, sparsity, quantization) into a new method, which could be seen as incremental/less exciting. 

# Questions
Below are some questions I have for the authors. 

Q1. The paper discusses the accuracy/compression trade off, but focuses less on time/computational cost. What can you say about the time it takes to compress a network? Is it negligible compared to, e.g., the time it takes to train the network? 

Q2. Does having the weights in the sparse quantized PCA format increase the inference time compared to the non-compressed network? 

Q3. How is the reshaping of the weight tensor $W$ into $\tilde{W}$ done? Do you always put $f_{\text{out}}$ along the rows, and $f_{\text{in}}$, $h$, $w$ along the rows? You should mention this in the paper.

Q4. What does the $\lfloor \cdot \rceil$ notation in Eqs. (5) and (6) signify? Round to nearest integer?

Q5. Should it be $2^{b_c}$ in Eq. (5) and $2^{b_z}$ in Eq. (6) rather than just $2^b$?

Q6. At end of 1st paragraph in Sec. 4.3.1, you mention doing “one-shot hard-thresholding without data-aware optimization.” What do you mean by “without data-aware optimization”? Are you just doing an SVD of $\tilde{W}$ and then quantizing the $C$ and $Z$ matrices?

Q7. How can quantization be implemented in practice? Do you implement your experiments in Python? 

Q8. What does “MAC operations” stand for (mentioned in the introduction)?

# Typos, other minor things
T1. Above Eq. (11), “while $Z$ encodes the of indices” should be “while $Z$ encodes the indices.”

T2. Sec. 3.2, 2nd paragraph: Should the size of $W$ be $256 \times 256 \times 3 \times 3$ rather than $256 \times 256 \times 9 \times 9$ since $w=h=3$?

T3. Sec. 4.3.2, 2nd paragraph, 1st sentence, there’s a period missing before “In this figure”.

T4. In 3rd sentence after the proof of Proposition A.1, should it be $\Delta_{f}(x) = \alpha \nabla f(x)$, i.e., without a $\star$ on the $x$ on the left hand side?

T5. In Sections B.2 and B.3 in the appendix, it would be nice if you at least added one sentence per section that refers to the relevant table so that we can know which table belongs in which section. Right now, the section headings aren’t really doing much.

summary_of_the_review: The paper is well written and covers a topic that should be of broad interest to the deep learning/ML community. The authors do a good job of putting their proposal in context and explaining previous work. The experiment results are encouraging. Overall, a nice paper that I think is suitable for publication in ICLR.
review_title: null

Meta-review: Reviewer rRp9 expressed concerns regarding the theoretical results included in Appendix A. In the discussion (not visible to the authors), the AC and Reviewer zn4a agree that the exposition in the original manuscript was confusing and could lead readers to assume these results were valid for the proposed algorithm. Also, in the original manuscript the presentation of the theoretical results in the appendix was quite poor (e.g. Proposition A.1). Having said that, the contributions and main points of the work are not affected by these observations as it is mainly an empirical study.

Following from the previous point, Reviewers rRp9 and zn4a pointed out that the overall presentation of the method, particularly the mathematical presentation could be improved. 

Reviewer zn4a points out that the method is not particularly novel, this was also indicated as a weakness by Reviewer iyVU. The main contributions of the work are to simultaneously solve the tensor factorization and vector quantization problems usinga form of projected gradient descent (with hard-thresholding). While the empirical results seem promising, are somewhat limited. The authors could make them stronger by studying other applications on top of image classification (e.g. semi-supervised setting, object detection or segmentation).

In the discussion (not visible to the authors), Reviewer iyVU stated in light of the other reviews, he/she does not oppose rejecting the work.

Overall, the method is technically sound and produces promising results. In its current form, however, the paper is not yet ready for publication. The AC encourages the authors to incorporate the feedback and resubmit the work to a different venue.
Decision: Reject