Quantized sparse PCA for neural network weight compression
Authors: Andrey Kuzmin
Mart Van Baalen
Markus Nagel
Arash Behboodi
Venue: ICLR 2022 Submitted (2022)
Keywords: Model Compression
neural network quantization
sparse principal component analysis
vector quantization

Abstract: In this paper, we introduce a novel method of weight compression. In our method, we store weight tensors as sparse, quantized matrix factors, whose product is computed on the fly during inference to generate the target model's weight tensors. The underlying matrix factorization problem can be considered as a quantized sparse PCA problem and solved through iterative projected gradient descent methods. Seen as a unification of weight SVD, vector quantization and sparse PCA, our method achieves or is on par with state-of-the-art trade-offs between accuracy and model size. Our method is applicable to both moderate compression regime, unlike vector quantization, and extreme compression regime.

Under review as a conference paper at ICLR 2022
QUANTIZED SPARSE PCA FOR NEURAL NETWORK WEIGHT COMPRESSION
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we introduce novel method of weight compression. In our method, we store weight tensors as sparse, quantized matrix factors, whose product com- puted on the fly during inference to generate the target model's weight tensors. The underlying matrix factorization problem can be considered as a quantized sparse PCA problem and solved through iterative projected gradient descent meth- ods. Seen as a unification of weight SVD, vector quantization and sparse PCA, method achieves is par with state-of-the-ard trade-offs between accuracy and model size. Our method is applicable to both moderate compression regime, unlike vector quantization, and extreme compression regime.
INTRODUCTION
Deep neural networks have achieved state-of-the-art results in a wide variety of tasks. However, deployment remains challenging due to their large compute and memory requirements. Neural networks deployed edge devices such mobile or loT devices are subject to stringent compute and memory constraints, while networks deployed in the cloud do not suffer such constraints but might suffer excessive latency power consumption.
To reduce neural network memory and compute footprint, several approaches have been introduced in literature. Methods related to our approach, well as their benefits and downsides, are briefly introduced in this section and described in more detail in the related works section. Tensor factor- ization approaches (Denil et al., 2013) replace layer in a neural network with two layers, whose weights are low-rank factors of the original layer's 's weight tensor. This reduces the number of pa- rameters and multiply-add operations (MACs), but since the factorizations are by design restricted to those that can be realized as individual layers, potential for compression is limited. By pruning a neural network (Louizos et al., 2017; He al., 2017), individual weights are removed. Pruning has shown to yield moderate compression-accural trade-offs. Due to the overhead required to keep track of which elements are pruned, real yield of (unstructured) pruning lower than the pruning ratio. An exception to this is structured pruning, in which entire neurons removed from net- work, and weight tensors can be adjusted accordingly. However, achieving good compression ratios reasonable accuracy using structured pruning has proven difficult. Scalar quantization (Jacob) jet al., 2018;; Nagel et af. 2022)) approximates neural network weights with fixed-point values, i.e., integer values scaled by fixed floating point scalar. Scalar quantization has shown to yield high accuracy at reasonable compression ratios, c.g., 8 bit quantization yields a 4x compression ratio and virtually no accuracy degradation on many networks. However, scalar quantization does not yield competitive compression vs accuracy trade-offs at high compression ratios. Vector quantization (Stock et al.] 2019 Martinez al.] (02T) approximates small subsets of weights, e.g., individual 3 x 3 filters in convolutional weight tensors, by small set of codes. This way, the storage can be reduced by storing the codebook and one code index for each original vector, instead of the individ- ual weights. While vector quantization achieve high compression with moderate accuracy loss, these methods usually struggle to reach the accuracy of uncompressed models in low compression regimes.
In this paper, we provide a novel view on tensor factorization. Instead of restricting factorization to those that can be realized two separate neural network layers, show that much higher compres- sion ratios can be achieved by shifting the order of operations. In our method, we find a factorization C, Z for an original weight tensor W, such that the matrix product CZ closely approximates the

---

Under review as a conference paper at ICLR 2022
and reshape
Quantized Sporse
Weight tensor KAxow
Meight tensor tiled and reshaped to dhoaz 2
C, shape dotd ic
shape 4o0m 30
sparse
Figure I: An illustration application of Quantized Sparse PCA convolutional weight tensor. The weight tensor is reshaped into matrix of shape d x n and factorized into a codebook C' E Raxk and a latent Z &x Both factors are quantized while only sparse. During inference the arrows are followed backwards: the reshaped and tiled matrix is computed from the product of the factors and Z. The result is reshaped into the original weight tensor.
original weight tensor. The compression is then pushed further by obtaining quantized factors and additionally sparse Z. During inference, the product CZ is computed first, and its result is reshaped back into the original weight tensor's shape, and used for following computations. This approach allows the of arbitrary factorization of the original weight tensor.
We show that this approach outperforms or on par with vector quantization in high compres- sion regimes, yet extends to scalar quantization levels of compression-accuracy trade-off's for lower compression ratios.
Our contributions in this paper follows:
We show that the problem of tensor factorization and vector quantization can be formulated in a unified way as quantized sparse principle component analysis (PCA) problem.
We propose an iterative projected gradient descent method to solve the quantized sparse PCA problem.
Our experimental results demonstrate the benefits of this approach. By simultancously solving tensor factorization and vector quantization problem, we can achieve better accu- racy than vector quantization in low compression regimes, and higher compression ratios than scalar quantization approaches at moderate loss in accuracy.
2 RELATED WORK
SVD-based methods and tensor decompositions SVD decomposition was first used to demon- strate redundacy weight parameters neural networks in Denil al. (2013). Later several meth- ods for reducing the inference time based on SVD decomposition were suggested (Denton et al.) 2014 Jaderberg et al. 2014). similar technique was proposed for gradients compression in data- parallel distributed optimization by Vogels al. (2019). The main difference between these methods the way 4D weights of a convolutional layer is transformed into a matrix which leads to different shapes of the convolutional layers in the resulting decomposition. Following similar direction, several works focus on higher-order tensor decomposition methods which lead to introducing of three or four convolutional layers (Lebedev et al., 2014: Kim al. 2015 Su al. 2018).
Weight pruning A straightforward approach reducing neural network model size is removing a percentage of weights. A spectrum of weight pruning approaches of different granularity has been introduced in the literature. Structured pruning approaches such He al. (2017) kill entire channels of the weights, while unstructured pruning approaches (Louizos et al. /2017 & Gupta
2017 Neklyudov al. 2017: Dai et al. 2018) focus individual values. A recent survey unstructured pruning is provided in Gale et al. (2019).
Scalar quantization and mixed precision training By quantizing neural network weights to lower bitwidths, model footprint can be reduced as well, each individual weight requires fewer
2

---

Under review as a conference paper at ICLR 2022
bits to be stored. For example, quantizing 32 bit floating points weight to 8 bit fixed point weights yields a 4x compression ratio. Most quantization approaches use the straight-throug (STE) for training quantized models (Bengio et al. (2013); Krishnamoorrth (2018)). One way to fur- ther improve the accuracy of quantized models is learning the quantization scale and offset jointly
estimator
with the network parameters (Esser et al. (2019) y Bhalgat al. (2020)). A recent survey on practical quantization approaches can be found in Nagel et al. (2021).
In order improve the accuracy of quantized models, several methods suggest using mixed pre- cision quantization. The work by Uhlich et al.] (2019) introduced an approach on learning integer bit-width for each layer using STE. Using non-uniform bit-width allows the quantization method to use lower bit-width for more compressible layers of the network. Several works (van Baalen et al.) 2020 Dong et al., 2019 Wang et al. 2019) improve upon the approach by Uhlich et al. (2019) by using different methods for optimization over the bit-widths.
Vector quantization. Several works use vector quantization approach for compression of weights of convolutional and fully connected layers (Gong et al. (2014); Martinez et al. (2021); Fan et al.) (2020); Stock et al. (2019); Wu et al. (2016)) The convolutional weight tensors are reshaped into matrices, then K-means methods is applied directly on the rows or columns. Besides weight com- pression, the work by Wu et al. (2016) suggests using vector quantization for reducing the inference time by reusing parts of the computation.
Recently several works suggested improvements on the basic vector quantization approach. Data- aware vector quantization which improves the clustering method by considering input activation data is demonstrated to improve the accuracy of the compressed models by |Stock et al. (2019). Another direction is introducing a permutation to the weight matrices which allows find subsets of weights which are more compressible (Martinez et al. (022). An inverse permutation is applied at the output of the corresponding layers to preserve the original output of the model.
Besides weights compression problem, various improved vector quantization methods were applied in image retrieval domain in order to accelerate scalar product computation for image descrip- tors (Chen et al. (2010); Ge et al. (2013); Norouzi & Fleet (013)). In particular, additive quan- tization method which we will show related to quantized sparse PCA was introduced Babenko & Lempitsky (2014). Surveys on vector quantization methods are provided in Matsui et al. (2018);
Gersho & Gray (2122).
Sparse PCA. Introduced in Zou et al. (2066), sparse PCA can be solved by plethora of algorithms. The method proposed in this paper can be considered as an instance of thresholding algorithms (Mal 2013). Although soft-thresholding methods are prevalent in the literature, we adopt an explicit
projection step using hard thresholding to have direct control over the compression ration. Note that
sparse PCA can be extended to include additional structures with sparsity (Jenatton et al. 2010 Lee
et al. 2006).
3
METHOD
In this section, we describe the main algorithm, which can be considered sparse quantized princi- ple component analysis (PCA).
3.1
QUANTIZED SPARSE PCA
Consider a weight tensor of a convolutional layer W E Xfn where fin. fout are the number of input and output feature maps, and h, w are the spatial dimensions of the filter. We reshape it into a matrix W E RRdxn, where is the tile size and n is the number of tiles. For the reshape, we consider the dimensions in the order fout, fin, h, w in our experiments. The goal is to factorize W into the product of two matrices as follows:
W CZ,
(1)
where E Badkk is the codebook, and Z E R Exn is the set of linear coefficients, or a latent variable (k < d < n). Following the standard PCA method, we factorize the zero mean version of W. With this decomposition, every column of W, denoted by W.., is linear combination of k codebook vectors (columns of C):
W
Σ Crai 1=1
(2)
3

---

Under review as a conference paper at ICLR 2022
where C. is the j-th column of C. This decomposition problem is instance of sparse PCA methods (Zou et al. (2006)). For network compression, we are additionally interested in quantized matrices C and Z. The quantization operation for arbitrary C and Z is defined follows:
C = Qc 8e, be).
(3)
Z = (Z; 82. bz),
(4)
where be.b, the quantization bit-widths, and 8. and are the quantization scale vectors for C and Z, respectively. We consider per-channel quantization, i.e., the quantization scale values are shared for each column of C and each row column of Z (see section 3.2 for details on which is used when):
((
Cij
Cg.55 clamp
|s
0, 2
-1
8ir
(5)
213
Zg.53 = clamp
0, 20
852
(6)
8j
where [-] is rounding to nearest integer, and clamp(.) is defined as:
x < a S x bb.
clamp(æ,a,
x b
We refer the problem of finding quantized factors C and Z with sparse Z as quantized sparse PCA problem. Once we obtain the factors C and Z, we get the matrix W = (CZ), which can be reshaped back convolutional layer. The reshaped convolutional layer for factors C, Z is denoted by [CZ]. well known in network compression literature that it is better to find the best factorization on the data manifold (Zhang et al. (2015): He et al. (2017) the following optimization problem:
Stock et al. 2019)). Therefore, we solve
C³ C² = argmin (xX (II - (C,ZZ] * xI(
(8)
C
s.t. C, = Qe C; 8c: bc)
Z, = Qz (Z; 8z, bz)
1 < S, Z E Rxx C E Rd
where the parameter S controls the sparsity ratio Z, X and Y are the stored input output of the target layer in the original model, D is the data distribution, and is the convolution operation. Lo norm is used for the constraint on the number of nonzero elements in the matrix Zg We approximate the expected value of above optimization problem using a subset of the training data:
m
Ex (IX - IC,
Y
[C,Zg] X,11?
m
where is the number samples used for optimization. Following the compression method intro- duced in Zhang et al. 2015)), we use output of the previous compressed layer as X instead of the stored input the layer in the original model. This approach aims at compensating for the error accumulation in deep networks.
3.1.1 PROJECTED GRADIENT DESCENT OPTIMIZATION ALGGRRITHM
The above optimization problem can be solved using an iterative projected gradient descent method. The projection step is hard thresholding operation which project onto the space of sparse matrices. The optimization algorithm is given below.
Initialization. The codebook C is initialized with first left singular vectors of W. Given the SVD decomposition of W as follows:
W
UΣVT
(9)

---

Under review as a conference paper at ICLR 2022
Algorithm Projected Gradient Descent Optimization
Require: W
1: SVD of W UEVT,
2: C - U
3: + 1-U, U.W W
4: while Stopping criteria met do
5: Gradient descent step: C, Z t gradient descent update on equation (IO)
6: M <
binary mask for largest S values of Q2 (Z; s..
7: Hard thresholding step: Z + Z @ M.
8: end while
9: return: Qc(C; bc, Qz bz).
we choose C(0) Ug. where U₁ denotes the top k left singular vectors of W. The latent matrix initialized as the projection of W onto the set first singular vectors, ( = U W.
2. Gradient Descent. At iteration t, the matrices C(t-1), ((t-)) are updated by gradient descent on the following objective w.r.t. C and Z:
Σ 77
(10)
We use the straight through estimator (STE) (Bengio et al. (2013) to compute the gradients the quantization operation, i.e. we use the following gradient estimate for the rounding operation:
Tpp Op
1.
The outcome of this step is denoted by C (
3. Hard thresholding. After the gradient descent, project z (1) onto the space of sparse matrices. The projection is done using entrywise product of z ((d) with binary mask matrix (( namely z
(()), The mask matrix (t)) obtained finding the support of S largest entries of Q= (Z (t), gd $2, bz). This is the hard thresholding step. For the next iteration
we set:
( Z gd
M(1) (( C(
4. Repeat the previous three steps using , (MSE error or fixed iterations number). The final matrices are given be Qo(C(t); se, bc), Qu (z))) s2. bz).
CC+) until termination criteria is met
This iterative projection step has theoretical support for well behaved optimization problems. See Appendix A for more details.
3.1.2 RELATION TO OTHER APPROACHES
In this section, we discuss the connection with other existing methods.
(A) Vector quantization. If replace the quantizers for C and Z by identity functions, i.c. Qe id(.). Q2 id(.). then matrix C in full precision, while Z encodes the indices such that
Zij 6em,
(11)
where my is the centroid index corresponding tile i.
5

---

Under review as a conference paper at ICLR 2022
(B) Additive vector quantization. If Qe ii((),, bg (binary quantization), then
W Σ P..C..j.
(12)
j=1
where Pie E {0. 1} are binary coefficients. Note that in the original work by Babenko & Lempitsky (2014), an additional constraint of Σ, Pji is enforced, i.e. each W., is
sum of tixed number of terms.
(C) Principle component analysis. Qu = id(-), and Q₂ = id(.), then
W Σ C..C..-
(13)
This case corresponds to SVD factorization of zero mean version of weight matrix W. For specific values of n and d, the method is mathematically equivalent to the SVD decom- position methods for neural network compression (Denton el al. (2014); Jaderberg et al. (2014). For example, if n foush, and fin W. then the method equivalent to the spatial SVD decomposition by [aderberg et al. (2014). In contrast to the approaches, we not decompose the convolutional operation into two convolutional lay- ers, which gives extra flexibility in the choice of PCA dimensionality d.
3.2 COMPRESSION RATIO
Without sparsity, the compression ratio is computed as follows. Let L. fout fin xh x32 denote the number of bits required to represent the original 32 bit floating point tensor W, and Le(W) d x k x b + k x n x by the size of the quantized tensor factors and of W. The compression ratio C, is then defined as C, (.(W) L.(
Note that, since ho < d < the latent matrix Z contributes more to the resulting model size than the codebook matrix C. For example, us assume Sout = Sin 256, = w and A = 128. In this scenario, the original weight tensor W has 256 x 256 x 3 x 3 = 589. 824 elements, the codebook matrix C has 256 x 128 = 32, 768 elements and the latent matrix Z has 128 x 2, 034 260, 356 elements, almost times as many as C. For this reason, efforts to compress Z. e.g., by using a lower bitwidth b. or pruning elements, will likely yield a better accuracy vs compression trade-off than focusing similar efforts on C.
256,
Additional compression is achieved by encoding the sparse matrix as follows. We assume the sparsity mask M is stored
binary matrix. Then, only store the nonzero values in Z. At runtime, can be decoded by only reading values in Z for which the corresponding binary mask value equal to The total number of bits required to store M and the nonzero values in Z is k X n (1 r) x k x by, where r = 1 - S/ kn is the sparsity ratio of Z with |Z = S. Note that this method only reduces memory footprint if 1/b..
The size of the compressed tensor must be adjusted for quantization parameters introduced by per- channel quantization. We assume that each quantization scale parameter is encoded using FP1 To account for this we add quantization adjustment term (W) to Le Lc(W), where Lg 2k x 16 is used for per-row quantization of Z.
Finally, taking into account the adjustments for sparsity and quantization parameters, the size of compressed tensor is computed as I. = d lo x b + x (1 (1-r) x k x + (W))
EXPERIMENTS
In this section we describe two sets of experiments: one set to explore the effect of compression hyperparameters on compression-accuue accurrcc trade-offs, and one set of experiments to compare against various baseline methods.
4.1 EXPERIMENTAL SETUP
Initial factorization We start with a pre-trained model and perform the factorization as described in Section 3.1.I We initialize the per-channel quantization scale using min-max quantization (Kr- ishnamoorthi 2018).

---

Under review as a conference paper at ICLR 2022
Per layer data-aware optimization To perform the data-aware optimization, we sample single batch of 64 input and output activations for a layer. We split the data into a training set and a validation set (we keep one eighth of the data the validation set). We keep track of the eπror on the validation set. During this step we use the Adam optimizer (Kingma & Ba 2014 with learning rate 10-4 and weight decay 10-5.
Sparsification Sparsification is applied after per layer data- optimization in a form of hard thresholding. We consider one-shot and iterative thresholding. Specifically, for each layer. the quantized weights are sorted by absolute value, then a desired percentage of values is masked out. We do not prune weights that become zeros after the quantization step, instead, we add extra sparsity on top of accidental sparsity that occurs after the quantization. During the consequent fine-tuning stage, the sparsity mask is frozen.
Fine-tuning We follow| Martinez et al. (202)) and fine-tune our models end-to-end using the target training set. Model-specific finetuning procedures described below.
4.1.1 A NOTE ON COMPRESSION RATIO
In our experiments we leave the first convolutional weight tensor, all bias tensors, all batch nor- malization parameters, and the PCA centering vectors in FP32. We denote these uncompressible Ew (W) parameters as Lu. The sparsity ratio for network is then computed as C, Iu+ (W)
4.2 IMAGENET EXPERIMENTS
We evaluate our method on the ResNet18 and MobileNetV2 achitectures for ImageNet classification. We apply our method on pretrained versions of the model, where we take the pretrained weights from the Py PyTorch model zoo. In all our experiments we keep the tile size d constant at 256, the bitwidth of the codebook matrix C constant at 4, and use per-channel scalar quantization of 4 bits for the fully connected classification layer weights. To achieve different compression ratios we vary the rank with values between 64-256, the bitwidth of the latent matrix Z between 3-6, and the extra sparsity of Z between 0% and 40%. Further experimental details can be found in Section B.1
4.2.1 BASELINE METHODS
We compare our methods against scalar quantization by Esser et al. (2019), binary CNNs by Lin al. (017) (ABC-Net), the vector quantization method by Martinez et al. (2021) (PQF), the vector quantization method by [Stock et al. (2099) (BGD), trained ternary quantization by Zhu et al. (2016) (TTQ). The results for scalar quantization are produced using LSQ [Eser al. (201)), with 4, and 2-bit per-channel weight quantization and FP32 bit activations. For these experiments a weight decay of 10-4 and cosine learning rate decay to 10 of the original learning rate was used.
4.2.2 RESULTS
The Pareto dominating results of our method and comparison to the baselines can be found in Fig- ure 2 The full set of (non Pareto dominating) results can be found in Figure Ba The same results along with the values for rank, bitwidth and sparsity and resulting compression ratios can be found in Table the Appendix. In this figure we see that we outperform BGD, TTQ, ABC-Net and 3 and bit scalar quantization by considerable margins. Furthermore, we outperform or match the strongest baseline at high compression ratios, PQF for compression ratios of 17x and higher, and are on par with b-tit scalar quantization for compression ratios under 17x. Lastly, we are the only method to accuracy trade-offs over the full range from 5x to 40x compression.
achieve SOTA
4.3
ABLATION STUDIES
4.3.1
HARD THRESHOLDING METHODS
In this section, we study the impact of the hard thresholding method and the stopping criterion for data-aware optimization. Algorithm|] applies Japlles the hard-thresholding step at each iteration. However, we can also apply it only after the end of gradient descent iterations. We call this one-shot hard- thresholding. The method has precedence in non-convex sparse optimization (Shen) (2020)). We consider two different stopping criteria. The first is using the fixed number of gradient descent iterations, e.g., 30, 100, or 1000. The second is based achieving the MSE error threshold on the validation set. We split the data used for pay-layer optimization into a training set and a validation set. We stop the iterations soon the emrrr the validation set is not decreasing for more
7

---

Under review as a conference paper at ICLR 2022
Compression ratio vs accuracy (ResNet18)
71
Scalar quantization
70
VQ (Bit goes down)
69
(Permute, quartize, finetune)
ABt-Net (M,15,3,1)
60
TTQ
Ours
67
baseline
1% below FP32 baseline
63
62
35
Compression ratio
Compression ratio vs accuracy (MobileNetV22
72
Scalar quantization
71
VQ four inplementation)
70
Ours
baseline
60
..... 1%, below FP32 baseline
68
67
11
Compression ratio
Figure 2: Results for Resnet18 and MobileNetV2 trained on ImageNet
than two iterations. Finally, we compare both methods with one-shot hard-thresholding without data-aware optimization.
We present the results for & in the Table Three different levels of sparsity are considered for a model with 4 bit latent. We report the validation accuracy of the model before and after fine- tuning. The results suggest that the most important aspect of data-aware optimization is the stopping criterion. The best pre-finetuning results are obtained by using iterative hard thresholding with the least number of iterations, namely 30. The second best method one-shot iterative hard thresholding with the stopping criterion based on the validation set. These results suggest that it is important to limit the number of iterations of per layer optimization in order to prevent the method from over- fitting to the layer input and output data sample. This intuition is further supported by observing the lowest pre- finetinnunnng accuracy for hard thresholding with the highest number of iterations, namely 1000.
However, overall benefits per layer optimization methods become marginal after the compressed model is fine-tuned. In various experiments, we observed an 0. 0.1-0.3% improvement in the valida- tion accuracy compared to using one-shot hard thresholding without any aata-aware optimization. Therefore, in our further full model experiments, ping criterion based on the validation set as a method of low computational complexity, which gives nearly the best results compared to the other methods.
used one-shot hard thresholding with the stop-
4.3.2 QUANTIZATION VS PRUNING VS RANK
In our method, compression can be achieved by lowering rank, lowering the bit-width for latent matrices (hence referred to as 'bit-width"), or increasing extra sparsity. Thus, similar compression ratios can be achieved with different values for rank, bit-width and sparsity. For example, for given model, we can retain the same compression ratio by increasing rank and decreasing bit-width to compensate. In this section we investigate whether we discover patterns that would help guide compression hyperparameter search.
In Figure Bal show scatter plot with the full set of results. In this figure, see that for low compression ratios (below 14x) only minor improvements can be achieved by tweaking compression hyperparameters. For example, around 10x compression even the worst performing compression hy-
8

---

Under review as a conference paper at ICLR 2022
Pre-FT accuracy
Post-FT accuracy
Hard threshold-
Stopping. critetion
10%
20% sparsity
30% sparsity
10% sparsity
20% sparsity
30%
ing
sparsity
sparsity
method
One-shot HT
val. set
63.2
58.9
47.3
69.1
68.7
68.1
One-shot HT
100 iter.
59.4
55.5
45.2
68.8
68.4
68.0
Iterative HT
30 iter.
64.5
63.0
58.8
69.1
68.8
68.0
Iterative HT
100 iter.
62.0
60.7
57.3
69.0
68.7
68.1
Iterative HT
1000 iter.
55.0
55.4
54.1
68.5
68.3
67.9
No data-aware opt.
58.5
53.3
37.8
69.0
68.7
67.9
Table 1: Resnet18 8 ablation on the impact of the hard thresholding method and the stopping criterion on top-1 validation accuracy of the compressed model. Iterative fard-thresholding with 30 iterations shows the best results among all the other methods, however, the benefit vanishes after fine-tuning,
Comoraaa on rollo cccutity (1o18:111
Bank
eth accuracy
Scursity
G4.4
goo dowr
ABC (1
26
(b) Effect rank, bit-width and sparsity models near 1lx (top row) and 25x (bottom row) compres- sion ratios.
(a) All results for ResNet18, including non-Parcto dominating results
Figure Full results and effect of rank, bit-width and sparsity.
perparameters are still on par with quantization to 3 bits (10.67x compression). However, for higher compression ratios the compression hyperparameters have a larger impact on model performance.
In Figure 3b we show the effect of rank, bit-width and sparsity ratios on a models with a low (near 1 1x, top row) and high (near 25x, bottom row) compression ratio. Again we see little effect (roughtly 0.5%) for the lower compression ratio. However, for the more aggressively compressed models in the bottom row, we see that increasing rank while lowering bit-width to compensate has a strong positive effect on model performance. Finally, both plots we see no clearly discernible trend in the effect of compression ratio model accuracy.
Based on these results we conclude that, at high compression ratios, increasing rank and decreasing bit- widthi can improve results. Sparsity can be used to further fine-tun the target compression ratio.
CONCLUSION
We presented a weight compression method based on quantized sparse PCA which unifies SVD- based weight compression, sparse PCA, and vector quantization. We tested our method on Im- ageNet classification and show compression-accurac with strong baselines, and demonstrated that our method is the only method to achieve SOTA compression-accuracy trade-off's in both low and high compression regimes. Lastly, we investi- gated whether low rank approximations, quantization, sparsity yield the best efficiency-accuracy trade-offs, and found that high compression ratios, increasing rank and decreasing bit-width can improve results. For future work, plan to investigate methods for automated rank, bit-width, and sparsity ratio selection in order to further improve the accuracy and reduce the parameer search time.
trade-offs which are otate-of-the-art or on

---

Under review as a conference paper at ICLR 2022
REFERENCES
Artem Babenko and Victor Lempitsky. Additive quantization for extreme vector compression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 931-938, 2014.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint urXiv:1308.35. 3432, 2013.
Yash Bhalgat, Jinwon Lee, Markus Nagel. Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
Yongjian Chen, Tao Guan, and Cheng Wang. Approximate nearest neighbor search by residual vector quantization. Sensors,
11273, 2010.
Bin Dai, Chen Zhu, Baining Guo, and David Wipf. Compressing neural networks using the varia- tional information bottleneck. In International Conference on Machine Learning, pp. 1135-1144. PMLR, 2018.
Misha Denil, Babak Shakibi, Laurent Dinh, Mare' Aurelio Ranzato, and Nando De Freitas. Predict- ing parameters in deep learning. arXiv preprint /306. 0543, 2013.
Emily I. Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in neural informa- tion processing systems, pp. 1269-1277, 2014.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hessian mixed-precision. In Proceedings of the IEEF/CVE aware quantization of neural networks with mixed- precision.
International Conference on Computer Vision, pp. 293-302, 2019.
Steven K. Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra Modha. Learned step size quantization. arXiv preprint arXiv: 08153, 2019.
Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Rémi Gribonval, Herve Jegou, and Armand Joulin. Training with quantization noise for extreme model compression. arXiv preprint arXiv:2004.073220 2020.
Simon Foucart and Holger Rauhut. Mathematical Introduction to Compressive Sensing. Ap- plied and Numerical Harmonic Analysis. Springer New York, New York, NY, 2013. ISBN 978-0-8176-4944 978-0-8176-4948-7 URL http::tttt:///k..ppppt
com/ 10. 1007/
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.0957.. 2019.
Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun. Optimized product quantization. IEEE transac- fions on pattern analysis and machine intelligence, 36(4):744-755, 2013.
Allen Gersho and Robert M Gray. Vector quantization and signal compression, volume 159. Springer Science Business Media, 2012.
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional net- works using vector quantization. arXiv preprint arXiv:1412.6115. 2014.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEFE international conference computer vision, pp. 1389-1397. 2017.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew G. Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-aritmmetic-nnty inference. In 2018 IEEE Conference Computer Vision and
10

---

Under review as a conference paper at ICLR 2022
Pattern Recognition, CVPR 2018, Salt Lake City. UT. USA. June 18-22, 2018, pp. 2704- 2713. Computer Vision Foundation IEEE Computer Society, 2018. doi: 10.1109/CVPR. 2018.00286.
whwceaceceeesee.c....eeeeeeeeee. thecvf. com content evpr 2018/htm1/ Trainniinnnn
Jacob_Quant zat i hrt.aatiii..trr.eeen..aaaainnnno wtw.rriiiinnnccr...__r 2018 2018_paper.._ html.
Laurent Jacques, Jason N Laska, Petros T Boufounos, and Richard G Baraniuk. Robust 1-bit com- pressive sensing via binary stable embeddings of sparse vectors. IEEE Transactions on Informa- fion Theory, 59(4))
2013.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv: 1405.3866, 2014.
Rodolphe Jenatton, Guillaume Obozinski, and Francis Bach. Structured sparse principal component analysis. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 366-373, 2010.
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Com- pression deep convolutional neural networks for fast and low power mobile applications. arXiv preprint 1r1i 1.0630,, 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.688 2014.
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.00 0001210000084.4. 2018.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. arXiv preprint 1rXiv:1412.65555 2014.
Honglak Lee, Alexis Battle, Rajat Raina, and Andrew Y. Ng. Efficient sparse coding algorithms. In Proceedings of the 19th International Conference on Neural Information Processing Systems, pp- 801-808. MIT Press, 2006.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. arXiv preprint arXiv: 171111711.12244 2017.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1.0 regularization. arXiv preprint 172iv:1112.013/2, 2017.
Zongming Ma. Sparse principal component analysis and iterative thresholding. The Annals of Statistics, 41(2):772-801, April 2013. ISSN 0090-5364. doi: 10.1214/13-ASS1097 URL
http://prrojecteucliie org wtt.//uccll.d.. acs/ aos//o/////111
Julieta Martinez, Jashan Shewakramani, Ting Wei Liu, Ioan Andrei Bårsan, Wenyuan Zeng, and Raquel Urtasun. Permute, quantize, and fine-tune: Efficient compression of neural networks. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp-
15708, 2021.
Yusuke Matsui, Yusuke Uchida, Hervé Jégou, and Shin ichi Satoh. A survey of product quantization. ITE Transactions Media Technology and Applications, 6(1):2-10, 2018.
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko. Mart van Baalen, and Tijmen Blankevoort. A white paper on neural network quantization. ArXiv, abs/2106.08295, 2021.
Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Structured bayesian pruning via lor-normal multiplicative noise. arXiv preprint arXiv: 1705.07788 2017.
Mohammad Norouzi and David J Fleet. Cartesian k-means. In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition, pp. 3017-3024, 2013.
Jie Shen. One-bit compressed sensing via one-shot hard thresholding. In Conference on Uncertainty in Artificial Intelligence, pp. 510-519. PMLR, 2020.

---

Under review as a conference paper at ICLR 2022
Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, and Hervé Jégou. And the bit goes down: Revisiting the quantization of neural networks. arXiv preprint 2019.
Jiahao Su, Jingling Li, Bobby Bhattacharjee, and Furong Huang. Tensorized spectrum preserving compression for neural networks. arXiv preprint arXiv: 10352, 2018.
Stefan Uhlich, Lukas Mauch, Kazuki Yoshiyama, Fabien Cardinaux, Javier Alonso Garcia, Stephen Tiedemann, Thomas Kemp, and Akira Nakamura. Differentiable quantization of deep neural networks. arXiv preprini urXiv::905.00 1452, 2(8), 2019.
Mart Baalen, Christos Louizos, Markus Nagel, Rana Ali Amjad, Ying Wang, Tijmen Blankevoort, and Max Welling. Bayesian bits: Unifying quantization and pruning, arXiv preprint arXiv:2005.07099; 2020.
Thijs Vogels, Sai Praneeth Karinireddy, and Martin Jaggi. Powersgd: Practical low-rank gradient compression for distributed optimization. Advances In Neural Information Processing Systems 32 (Nips 2019), 32(CONF), 2019.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quan- tization with mixed precision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8612-8620, 2019.
Jiaxiang Wu, Cong Leng. Yuhang Wang. Qinghao Hu, and Jian Cheng. Quantized convolutional neural networks for mobile devices. In Proceedings of the IEEE Conference Computer Viston and Pattern Recognition, pp. 4820 4828, 2016.
Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. IEEE transactions on pattern analysis and machine intelligence, 38(10):1943-1955 2015.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. arXiv preprint arXiv:1612.0106: 2016.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint 17110:1710.01878, 2017.
Hui Zou, Trevor Hastie, and Robert Tibshirani. Sparse principal component analysis. Journal of computational and graphical statistics, 15(2):265-286. 2006.
A HARD THRESHOLDING FOR SPARSE OPTIMIZATION
The sparse optimization method sketched this paper is based on one-shot or alternating appli- cations of projection step onto the space of sparse vectors. In this section, provide some theoretical supports for this approach. Consider the following optimization problem:
min ((x).
(14)
xED
The set 'D is a generic set. The projection function on D defined as
( arg min | ==|g-
When the set D is the set of sparse vectors denoted by E, (R") with sparsity order the projection function is hard thresholding. For a vector x, the hard thresholding function H₂(x) yields a vector with at most non-zero entries, which are the entries of with largest absolute value.
The following proposition provides a general condition under which the optimization problem [4 be solved using iterative projection methods.
12

---

Under review as a conference paper at ICLR 2022
Proposition A.1. Consider the optimization problem equation 14 and assume that æ, is the mini- mizer of the problem. Consider an iterative optimization algorithm with the initial point Do and the iteration defined by:
I++1 = ( - (x())
where (,(.) is function such that ( is L-Lipschitz, and (x. 0. Then have:
| - x.112 - (2L)* || - x.112.
In particular, L & 1/2, the algorithm converges to the minimizer x, as oc.
Proof. First, for an arbitrary set D R² the projection operation onto the set ll and any æ E D and 2 E IR", we have the following inequality
[[]p(z) 2ا12 - 2 2| 2- 2|1,
We can obtain this inequality in two steps. We first use triangle inequality:
[I 2||2 N For the last term, the definition of projection operation that for each E D, the projection of z to 'D minimizes its distance to the points in D:
[[I[(2) x x112-
[ - 2112 S || - =xll2-
Consider the iteration step t + 1. We have:
[ - (xx) 2=112 - 2 2 Af(x,) - xz112 S 2L || 221122
which implies
| - ( x. S (2I))+1 |120 x. 2..12-
Some remarks are in order. First, the assumption (y(x.) = means that æ, is stationary point of the algorithm. This holds particularly for gradient descent based methods where A (x.) a f and x, a stationary point of f.
The Lipschitz property of x (xx) holds in many situations. As an example, consider the function f(+) as a simple mean squared error (MSE) loss, i.e., (xx = 11y Axl this case, we have: aAA x (I 1I-aAA )x
A
which satisfies L-Lipschitz property with L bounded by the spectral norm I cAA In many situations, when this spectral norm is restricted the set D, it be controlled to be less than 1/2. notable example, closely related to our problem, is the sparse recovery problem where one can prove recovery guarantee for iterative hard thresholding methods when the matrix A sat- isfies restricted isometry property. See Foucart & Rauhut (2013) for a comprehensive exposition of the topic including iterative hard thresholding for linear problems. Hard thresholding methods are widely used in sparse optimization literature. Particularly relevant to our problem is the one-bit bit sparse recovery problem, where iterative hard thresholding is used Jacques al. (2033
B RESNET18 18 EXPERIMENTS
B. EXPERIMENTAL DETAILS
We run a small number of pilot experiments for 5 epochs, to find good settings for the optimizer, learning rate, and learning rate schedule. We then run large number of epoch experiments find good compression-acooe
trade-offs. For the best performing compression-accuss then run longer experiments of 25 epochs, with the two best optimization schedules. We use a batch size of 64 for Resnet18, 8, use cosine learning rate decay to 10 of the initial learning rate, and turn off weight decay.
trade-off's we
B.2 PARETO DOMINATING RESULTS
B.3 ABLATION STUDY RESULTS
13

---

Under review as a conference paper at ICLR 2022
K
mpfd
cr crasee
eval score
256
256
0.00
6.01
70.26
256
256
0.20
6.96
70,06
256
256
0.20
8.91
70.04
192
256
0.20
11,50
69.77
192
256
0.00
13.20
69.47
128
256
0.00
16.95
69.15
128
256
0.00
18.17
68.91
128
256
0.40
20.19
68.73
128
256
0.20
21.70
68.63
92
256
0.15
23.04
68.17
92
256
0.15
25.34
68.11
100
256
0.12
27.49
67.52
64
256
0.00
29.02
67.20
92
256
0.15
30.02
66.94
64
256
0.20
32,62
66.86
64
256
0.00
33.78
66.37
64
256
0.20
39.88
64.48
Table 2: All Pareto dominating results for ResNet18
compression ratios near 11x
compression ratios near 25x
K
D
bw
sparsity
comp. ratio
accuracy
K
D
bw
sparsity
comp. ratio
accuracy
256
256
4
0.3
10.87
69.31
39
256
6
0.00
25.97
54.04
128
256
5
0.1
11.41
69.32
40
256
0.10
25.52
54.06
256
256
3
0.0
11.77
69.45
48
256
5
0.00
25.26
63.03
192
256
5
0.4
11.07
69.49
58
256
4
0.00
25.57
64.89
128
256
5
0.0
11.24
69.57
64
256
5
0.40
25.57
66.03
192
256
4
0.2
11.19
69.77
58
256
0.00
25.57
67.11
192
256
0.1
10.56
69.72
60
256
0.13
25,45
67.18
64
256
3
0.00
26.07
67.40
77
256
3
0.00
25.56
68.00
92
256
3
0.15
25.34
68.11
Table Various hyperparameter settings and resulting compression ratios and accuracies for com- pression ratios near 1x and 25.5x
14


Reviews:
[{"review_id": "Nl2BiAuCLK8", "reviewer": "Reviewer_iyVU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strenghts\nS1. The paper is clearly written and the proposed method and results are convincing. The method is simple yet seems to work well.\n\nS2. The topic of weight compression should be of interest to a large proportion of the deep learning/ML community.\n\nS3. Although I’m not familiar with the literature in this area, the authors seem to do a good job of discussing related work (in Sec. 2) and later on also discussing how their proposal relates to other previous methods (in Sec. 3.1.2).\n\n# Weaknesses\nW1. I mentioned the simplicity of the method as a strength. However, the simplicity of the method comes from the fact that it just combines several previous ideas (PCA, sparsity, quantization) into a new method, which could be seen as incremental/less exciting. \n\n# Questions\nBelow are some questions I have for the authors. \n\nQ1. The paper discusses the accuracy/compression trade off, but focuses less on time/computational cost. What can you say about the time it takes to compress a network? Is it negligible compared to, e.g., the time it takes to train the network? \n\nQ2. Does having the weights in the sparse quantized PCA format increase the inference time compared to the non-compressed network? \n\nQ3. How is the reshaping of the weight tensor $W$ into $\\tilde{W}$ done? Do you always put $f_{\\text{out}}$ along the rows, and $f_{\\text{in}}$, $h$, $w$ along the rows? You should mention this in the paper.\n\nQ4. What does the $\\lfloor \\cdot \\rceil$ notation in Eqs. (5) and (6) signify? Round to nearest integer?\n\nQ5. Should it be $2^{b_c}$ in Eq. (5) and $2^{b_z}$ in Eq. (6) rather than just $2^b$?\n\nQ6. At end of 1st paragraph in Sec. 4.3.1, you mention doing “one-shot hard-thresholding without data-aware optimization.” What do you mean by “without data-aware optimization”? Are you just doing an SVD of $\\tilde{W}$ and then quantizing the $C$ and $Z$ matrices?\n\nQ7. How can quantization be implemented in practice? Do you implement your experiments in Python? \n\nQ8. What does “MAC operations” stand for (mentioned in the introduction)?\n\n# Typos, other minor things\nT1. Above Eq. (11), “while $Z$ encodes the of indices” should be “while $Z$ encodes the indices.”\n\nT2. Sec. 3.2, 2nd paragraph: Should the size of $W$ be $256 \\times 256 \\times 3 \\times 3$ rather than $256 \\times 256 \\times 9 \\times 9$ since $w=h=3$?\n\nT3. Sec. 4.3.2, 2nd paragraph, 1st sentence, there’s a period missing before “In this figure”.\n\nT4. In 3rd sentence after the proof of Proposition A.1, should it be $\\Delta_{f}(x) = \\alpha \\nabla f(x)$, i.e., without a $\\star$ on the $x$ on the left hand side?\n\nT5. In Sections B.2 and B.3 in the appendix, it would be nice if you at least added one sentence per section that refers to the relevant table so that we can know which table belongs in which section. Right now, the section headings aren’t really doing much.\n", "summary_of_the_paper": "The paper proposes a method for compression of neural network weights. The proposal turns weight tensors into matrices, factorizes these matrices into a rank-k factorization via PCA, applies quantization to the factor matrices, and additionally makes the right (latent) matrix sparse. An algorithm is presented, with two different thresholding options (per-iteration, or one-shot). In experiments, these ideas lead to an accuracy/compression tradeoff which is either competitive with or better than previous state-of-the-art across all compression ratios investigated.", "summary_of_the_review": "The paper is well written and covers a topic that should be of broad interest to the deep learning/ML community. The authors do a good job of putting their proposal in context and explaining previous work. The experiment results are encouraging. Overall, a nice paper that I think is suitable for publication in ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DDFOt_weS54", "reviewer": "Reviewer_rRp9", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. I request the authors to discuss a significant low-rank factorization technique used for deep neural network gradient compression---PowerSGD by Vogels et al. NeuRIPS, 2019. PowerSGD uses power iteration to decompose the original gradient matrix $M$ into two $r$-rank matrices $P$ and $R$. This work is relevant and similar to the present work and requires mentioning. \n\n2. In the expected loss minimization problem, what is $m$? Is it the number of observations in the training set? I did not find it before. \n\n3. I have questions about solving the problem (10). How in (10), the authors arrive at quantized, $C_q, Z_q$? By using gradient descent, one can reach the factors, $(C, Z)$, and then it requires further projection by using quantization operations (as defined in (7)) to find quantized, $C_q, Z_q$. Running gradient descent algorithm directly on (10) does not guarantee quantized, $C_q, Z_q$. In the present form, it is misleading, and the treatment and evolution of the problem throughout the draft are highly faulty.  \n\n4. **Severely faulty mathematics.** Proposition A.1 is not correct. Please allow me to explain. The projection is not on a convex set; projection onto the set of sparse matrices (by using hard threshold operation) is a nonconvex projection. Moreover, the function that the authors defined in (10) is the same as the function, $f(x)$ defined in (14), which is a nonconvex function. First of all, for nonconvex functions proving convergence in terms of the optimum is impossible because one cannot guarantee the existence of a global minimizer $x_*$. Although the authors applied the projected gradient descent technique to nonconvex optimization, the convergence proof techniques used in the convex case do not apply directly to nonconvex problems. Therefore, the projection claim that the authors provided is mathematically incorrect. Please note that the claim that the authors made about the stationary point is erroneous. Moreover, the convergence rate with the derivation of the L-Lipschitz to justify their faulty analysis in Proposition A.1, is severely flawed. For (local linear) convergence of gradient descent with nonconvex projection, please see [1,2,3]. \n\n5. SVD stands for Singular Value Decomposition. Therefore, “SVD decomposition” does not make any sense. \n\n[1] Lewis et al. Local linear convergence for alternating and averaged nonconvex projections. Foundations of Computational Mathematics, 9(4):485–513, 2009.\n[2] Lewis and Malick. Alternating projections on manifolds. Mathematics of Operations Research, 33(1):216–234, 2008.\n[3] Dutta et al., A Nonconvex Projection Method for Robust PCA, AAAI, 2019. \n\n\n\n", "summary_of_the_paper": "The authors proposed a matrix factorization-based method for deep neural network weight compression. The weight tensors are factorized as two low precision quantized matrices, out of which one is sparse. ", "summary_of_the_review": "I did not read the paper after Section 3.1.1. The article is erroneous and the results are misleading. Please refrain from misleading the readers. This alone makes me recommend a STRONG REJECTION of the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "6R95Fl3QsHN", "reviewer": "Reviewer_zn4a", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe authors empirically demonstrate that their method is the only method to achieve SOTA compression-accuracy trade-offs in both low and high compression regimes. \n\nWeakness:\n1. The method seems to be a simple combination of tensor factorization and vector quantization methods. The idea of using (sparse) PCA in vector quantization methods has also been studied, e.g., in Babenko & Lempitsky (2014). Therefore, the novelty in this submission seems to be quite limited. \n2. The authors should provide more explanations about the motivation and benefits of assuming the factor matrix $\\mathbf{Z}$ to be sparse. In addition, the authors should clarify that Z is sparse in what sense? For example, row-sparse, column-sparse, or sparse while considering $\\mathbf{Z}$ as a $(kn)$-dimensional vector? In (8), the norm $\\||\\cdot\\||_0$ is used without any explanation. \n3. The authors should be careful to claim that their method for factorizing $\\tilde{\\mathbf{W}}$ in (1) is sparse PCA (ignoring the quantization). For conventional sparse PCA, the factor matrix $\\mathbf{C}$, instead of $\\mathbf{Z}$, is assumed to be column-sparse.\n4. The gradient descent update on (10) is important and should be clearly presented in this submission, rather than simply citing another paper. \n5. All source codes required for conducting experiments should be included in a code appendix.\n\nOther comments:\n1. The terminology \"quantized sparse PCA\" should be fixed, and the authors should not also use \"sparse quantized PCA\" arbitrarily (e.g., in the second dot point of the contributions). \n2. The relation $k < d < n$ should be made clear when these parameters first appear in Section 3.1 (instead of mentioning it in Section 3.2). \n3. In Algorithm 1, line 5, equation 10 should be equation (10). In Appendix A, problem 14 should be problem (14).", "summary_of_the_paper": "This paper introduces a novel method of weight compression. Weight tensors are stored as sparse, quantized matrix factors, and the underlying matrix factorization problem can be considered as a quantized sparse PCA problem and be solved through iterative projected gradient descent methods. The authors' method  is applicable to both moderate and extreme compression regimes, and is claimed to achieve or be on par with state-of-the-art trade-offs between accuracy and model size.", "summary_of_the_review": "Although the method proposed by the authors seems to be empirically promising, it seems to be a direct combination of existing methods, and the novelty seems to be limited. The authors provide no theoretical guarantee/intuition for their method. In addition, some terminologies/notations are vague or confusing. My impression is that this submission requires some major revisions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewer rRp9 expressed concerns regarding the theoretical results included in Appendix A. In the discussion (not visible to the authors), the AC and Reviewer zn4a agree that the exposition in the original manuscript was confusing and could lead readers to assume these results were valid for the proposed algorithm. Also, in the original manuscript the presentation of the theoretical results in the appendix was quite poor (e.g. Proposition A.1). Having said that, the contributions and main points of the work are not affected by these observations as it is mainly an empirical study.

Following from the previous point, Reviewers rRp9 and zn4a pointed out that the overall presentation of the method, particularly the mathematical presentation could be improved. 

Reviewer zn4a points out that the method is not particularly novel, this was also indicated as a weakness by Reviewer iyVU. The main contributions of the work are to simultaneously solve the tensor factorization and vector quantization problems usinga form of projected gradient descent (with hard-thresholding). While the empirical results seem promising, are somewhat limited. The authors could make them stronger by studying other applications on top of image classification (e.g. semi-supervised setting, object detection or segmentation).

In the discussion (not visible to the authors), Reviewer iyVU stated in light of the other reviews, he/she does not oppose rejecting the work.

Overall, the method is technically sound and produces promising results. In its current form, however, the paper is not yet ready for publication. The AC encourages the authors to incorporate the feedback and resubmit the work to a different venue.
Decision: Reject