SPARK: co-exploring model SPArsity and low-RanKness for compact neural networks
Authors: Wanzhao Yang
Miao Yin
Yang Sui
Bo Yuan
Venue: ICLR 2022 Submitted (2022)
Keywords: model compression
low-rankness
sparsity
tensor

Abstract: Sparsification and low-rank decomposition are two important techniques for deep neural network (DNN) compression. To date, these two popular yet distinct approaches are typically used in a separate way; while their efficient integration for better compression performance is little explored. In this paper we perform systematic co-exploration on the model sparsity and low-rankness towards compact neural networks. We first investigate and analyze several important design factors for the joint pruning and low-rank factorization, including operational sequence, low-rank format, and optimization objective. Based on the observations and outcomes from our analysis, we then propose SPARK, a unified DNN compression framework that can simultaneously capture model SPArsity and low-RanKness in an efficient way. Empirical experiments demonstrate very promising performance of our proposed solution. Notably, on CIFAR-10 dataset, our approach can bring 1.25%, 1.02% and 0.16% accuracy increase over the baseline ResNet-20, ResNet-56 and DenseNet-40 models, respectively, and meanwhile the storage and computational costs are reduced by 70.4% and 71.1% (for ResNet-20), 37.5% and 39.3% (for ResNet-56) and 52.4% and 61.3% (for DenseNet-40), respectively. On ImageNet dataset, our approach can enable 0.52% accuracy increase over baseline model with 48.7% fewer parameters.

Under review as a conference paper at ICLR 2022
RASPA: CO-EXPLORING MODEL LOW-RANKNESS AND SPARSITY FOR COMPACT NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Low-rank decomposition and sparsification neural network (DNN) compression. To date, these two popular yet distinct ap- proaches are typically used in a separate way; while their efficient integration for better compression performance is little explored. In this paper we perform sys- tematic co co-exploration on the model low-rankness and sparsity towards compact neural networks. We first investigate and analyze several important design factors for the joint low-rank factorization and pruning. including operational sequence, low-rank format, and optimization objective. Based on the observations and out- comes from our analysis, we then propose RASPA, a unified DNN compression framework that simultancously capture model low-RAnkness and SPArsity in an efficient way. Empirical experiments demonstrate very promising perfor- mance of our proposed solution. Notably, CIFAR-10 dataset, our approach bring 1.25%, 1.02% and 0.16% accuracy increase over the baseline ResNet-20, ResNet-56 and DenseNet-40 models, respectively, and meanwhile the storage and computational costs are reduced by 70.4% and 7 71.1 1 % (for ResNet-20), 37.5% and 39.3% (for ResNet-56) and 52.4% and 61.3% (for DenseNet-40), respectively. On ImageNet dataset, our approach can enable 0.52% accuracy increase over baseline model with 48.7% fewer parameters.
important techniques for deep
INTRODUCTION
Deep neural network (DNN) has served as the backbone machine learning technique in many mod- ern intelligent systems. facilitate the low-cost deployment DNN on the resource-constrained platforms, model compression,
powerful strategy that can efficiently reduce DNN model size, has been extensively studied in recent years. To date, numerous compression approaches have been proposed provide compact DNNs for many practical applications (Han al. (2015b) Gong al. 2019 Liao et al. (2021)).
Among various types of model compression techniques, sparsification (a.k.a., pruning) and low- rank decomposition are two representative and popular solutions (Wang et al. (2021); Gao et al.)
2021 Li et al. (2021a); Xu al. (2020). As revealed by their names, the low-rank and sparse methods aim to explore and leverage the potential low-rankness and sparsity of the uncompressed practice, supported by the widely existed overparameterization phenomenon (Denil et al. 2201); Han et al. (2013b)), such hypothesized structure-level redundancy usually exists and thus it can safely removed while still preserving high model performance.
DNNs, respectively.
Co-exploring Low-rankness & Sparsity: Motivation. Considering the current prosperity of these two methods and their very distinct structural assumptions, an interesting and promising research topic is to explore the efficient integration of low-rank and sparse approaches towards a better model compression solution. As indicated and observed by Yu et al. (2017). DNN models tend to exhibit both low-rankness and sparsity simultancously. For instance, the smooth components in the weight filters can be represented in the low-rank space, and meanwhile some other important information is sparsely scattered. Evidently, fully leveraging such co-existence of these structure-level patterns, being performed properly, can potentially bring a powerful compression solution with attractive performance.
Existing Works. Unlike the current extensive research activities on individual low-rank and sparse methods, the investigations integrating these two approaches, in an efficient and non-trivial way,

---

Under review as a conference paper at ICLR 2022
little explored. To date, only very few efforts study the joint exploration of low-rankness and sparsity for DNN model compression. As the pioneering work along this direction, Yu et al. (2017) develops singular value decomposition (SVD)-free approach to closely approximate original DNN model via combining sparse representation and low-rank matrix factorization. Built on the inter- esting connection between filter decomposition and filter pruning, [ et al. (2020) interprets the decomposition and pruning of convolutional filter in a unified perspective. Most recently, Li et al. (202Ib) proposes a collaborative compression scheme integrate SVD into model sparsification. By adopting a multi-step heuristic removal strategy, this post-training approach achieves promising task and compression performance.
Unanswered Questions. Although these prior works have demonstrated the huge potentials and attractive benefits of jointly decomposing and pruning. the systematic investigation on their effi cient integration is still missing. To be specific, several fundamental and critical questions, whose answers will directly impact the integration scheme and overall compression performance, have not been comprehensively explored yet. For instance, because pruning and decomposition can be jointly performed in several different ways, such as in parallel (Yu et al. (2017)) or in sequence (Li al. (2021b)), which collaborative strategy is the best fit for the target DNN compression task? Also, considering low-rankness can be exploited from different perspectives, which type of low low-rank approach should be adopted? The matrix factorization used in Li et al. (2020) 2 Or even high- order tensor decomposition? In addition, to achieve promising compression performance, what is the most suitable optimization objective that the integration scheme should aim? The approximation error focused in Yu et al. (2017)? The now-ranknesssspariity regularized loss in Yang et al. (020) Or some other new alternative??
Technical Preview and Contributions. To answer these questions and develop efficient integrated model compression solution, this paper we perform systematic co-exploration on the model low- rankness and sparsity towards compact neural networks. To be specific, we first review and analyze several important design factors for the joint low-rank decomposition and pruning. Based on the observations and outcomes from our analysis, we then propose RASPA, unified DNN compression framework that can simultancously capture model low-R -ow-RAnnness and SPArsity in an efficient way. Overall, the contributions of this paper are summarized as follows:
We systematically investigate and analyze the critical design knobs when co-exploring model low-rankness and sparsity, including operational sequence, low-rank format, and optimization objective. Based on our qualitative and quantitative analysis, we propose sev- eral recommended design options for efficient joint low-rank decomposition and pruning.
We develop unified framework that formulates the integration of low-rank decomposi- tion and pruning to an optimization problem with low-tensor-rank and sparse constraints. We then derive a training-aware approach to solve this challenging non-convex high-order tensor-format problem, and thereby leading to efficient exploration of rich low-rankness and sparsity the model.
We empirically evaluate our proposed co-exploration solution for various DNN models different datasets, and the experimental results demonstrate its very promising perfor- mance. Notably, on CIFAR-10 dataset, our solution can bring 1.25%. 1. 1.02% and O. 0.16% ac- curacy increase over the baseline ResNet-20, ResNet-56 and DenseNet-40 models, respec- tively, and meanwhile the storage and computational costs are reduced by 70.4% and 71.1% (for ResNet-20), 37.5% and 39.3% (for ResNet-56) and 52.4% and 61.3% (for DenseNet- 40), respectively. On ImageNet dataset, our approach can enable 0.52% accuracy increase over baseline model with 48.7% fewer parameters.
RELATED WORK
Sparsification.
Sparsification, also known as pruning, has been extensively studied for model com- pression (Han et al. (2015a): Wen et al. (2016); Gao et al. (2019); Guo et al. (2016); Rao et al. (021 ) In general sparsifying a DNN can be realized via two ways. The first one is to use a certain criterion, e.g., weight magnitude (Han et al. (2015a)), to directly remove some part of the model, and then perform fine-tuning to recover the accuracy. The second one is to add the sparsity-induced regularization during the training, such as & or group lasso term (Wen al. (2016)), to enforce the
2

---

Under review as a conference paper at ICLR 2022
sparsity on the model. In addition, in order to achieve good balance between accuracy and complex- ity reduction, Gao et al. 2 Guo et al. (2016); Rao et al. (202I) also proposes dynamic pruning. In such scenario the DNN sparsification is essentially performed in imput-aware way which part of model should be pruned is dynamically determined by each input data.
Low-Rank Decomposition. Low-rank decomposition is another popular DNN compression ap- proach. Based on different interpretation of the neural network models, the low-rank method can be categorized to matrix decomposition and tensor decomposition. Matrix decomposition views the 4-D weight tensor as the folded matrix, and hence it flattens the 4-D objective to 2-D format and decomposes the reshaped matrix to the product of two small matrices (Tai et al. (2016); Li & Shi 22118;; Xu et al.] (2020).. On the other aspect, tensor decomposition directly factorizes the 4-D weight tensor to multiple small tensor cores without flattening operations. Such explicit high-order processing, by its nature, can better preserve the important spatial information and correlation ex- isted in the weight tensors. To date several tensor decomposition techniques, such as tensor train, Tucker and tensor ring etc., have been used for DNN model compression (Kim al. (2016); Novikov ct al. 2015); Wang et al. (2018)).
Joint Pruning and Decomposition. As observed by Yu et al. (0177) }. well-trained DNN tends to exhibit both sparsity and low-rankness simultaneously. Motivated by this observation, some prior efforts propose to co-explore these two complementary properties for model compression. As the pioneering work, Yu et al. (2017) decomposes the weight tensors of a pre-trained DNN model into independent low-rank and sparse parts and minimizes the reconstruction error. Different from this parallel scheme, Dubey et al. (2018); Li et al. (2021b) adopt sequential compression strategy via performing matrix factorization on a pruned model. In addition, [i al. (2020) proposes to the sparse/low-rank regularization term, instead of reconstruction error, to enforce the desired structural patterns. Also, notice that all of the existing works focus on using either SVD-based or SVD-free matrix decomposition to exploit the low-rankness of DNN model.
CO-EXPLORING LOW-RANKNESS AND SPARSITY: ANALYSIS
As outlined in Section 2 the integration of low-rank decomposition and pruning can be specified by several important factors, including operational sequence, low-rankness format and the overall optimization objective. The existence of such large variety of different factors and their combina- tions, by its nature, calls for the systematic investigation on the best-suited co-exploration scheme for DNN compression. Such analysis framework, if being properly developed, can facilitate the optimal selection of various design factors already proposed in the existing literatures. More impor- tantly, the outcome from this systematic study will further guide and provide the better integration choices that have not been discovered before.
Questions to be Answered. Next we analyze the critical design knobs and factors for efficient co- exploration on model low-rankness and sparsity. To that end, three important questions need to be answered.
Question #1: What is the more suitable operational sequence when jointly low-rank decomposing and pruning DNN models?
Analysis. In general, the co-existence of model low-rankness and sparsity can be explored in dif- ferent ways (see Figure []. For instance, as adopted in Yu et al. (2017), a well-trained DNN can be closely approximated as the combination of a low-rank component and sparse component. In other words, the two types of structure-level properties are imposed and leveraged in spatially parallel way, and we denote this strategy as L+S, where and S represent low-rank decomposition and sparsification, respectively. On the other hand, the joint use of factorization and pruning can also be performed in temporully sequential way. As illustrated in Figure I the original model can be first imposed with low-rankness (or sparsity), and the size of the resulting partially compressed model can be further reduced by the second-stage pruning (or low-rank decomposition). Following the similar notation, such sequential operation can be denoted as S(L) and L(S). In practice L(S) is a preferable choice that has been adopted in the prior works (Dubey et al. (2018): Li et al. (2021b).
Our Proposal. Among the above described three general operational schemes, believe L+S is the more suitable choice when considering to integrate pruning and decomposition together for model compression. This is because unlike S(I.) and L((), which ultimately still produce the compressed
3

---

Under review as a conference paper at ICLR 2022
Decompose
B
Sparsify
S(L)
M
A
L
S
Sparsify
Decompose
B2
L(S)
M
A
S
L
Decompose
B3
L L+S
M
Ag
x
Sparsify
S
Figure I: Different operational sequences when jointly performing low-rant decomposition and pruning. Here L and S represent low-rank decomposition and sparsification, respectively.
model a single representation (sparse or low-rank) space, L+S enables the simultaneous represen- tation of rich information of DNN models across different subspace, and thereby better preserving the structural characteristics and reducing the potential information loss. To verify our hypothesis, examine the approximation error incurred by three integration schemes. As shown in Figure 2. with the same compression ratio for the weight tensor of one layer a pre-trained ResNet-20 CIFAR-10 dataset, L+S shows much lower approximation error than its counterparts, especially in high compression ratio region. This experimental phenomenon demonstrates that L+S scheme indeed can capture both the low-rank and sparse characteristics of DNN model in efficient way.
Question #2: What is the best suitable low- rank decomposition approach used when co- exploring low-rankness and sparsity?
Approximation Emrr
Analysis. From the perspective of linear alge-
bra, the low-rankness of a DNN model can
be exploited using different ways. As illus-
trated in Figure 3 for an example convolu- tional layer, imposing the low-rank structure can be realized by performing simple matrix factorization or high-order tensor decomposi- tion. Specifically, Yu et al.] (2017) chooses SVD-free method to factorize DNN model and obtain the low-rank component, and Li et al. (2021b) proposes to use SVD-based decompo- sition serve the second-stage compression approach in its adopted L(S) scheme. Notice that though the weights of convolutional layer essentially form 4-D tensor format, the ex- isting works exploit the low-rankness via using matrix decomposition - the 4-D tensor needs to be first flatten to 2-D matrix and is then fac- torized to two small matrix components.
S(L)
L(S)
Compression Ratio
The approximation error when compress- ing the weight tensor of one layer in ResNet-20 using different operational sequences (L+S, S(L.) and L(S)). Here mean square crror (MSE) is used to measure the difference between the original uncompressed weight tensor and the reconstruction. SVD is adopted the that
Figure 2:
low-rank decomposition method (L). It L+S can bring much smaller approximation error than its counterparts with the same compression ra- tio. More detailed results are reported in Appendix B.]
Our Proposal. We argue that the high-order tensor decomposition, the option that has not been ex- plored in the integration scheme before, is the better choice than the low-order matrix decomposition adopted the existing works. This because as reshaping free technique that can directly factor- ize the tensor-format data multiple tensor cores, tensor decomposition, such as Tensor Train (TT) and Tucker, can naturally capture and preserve the important spatial information and correlation of the original weight tensors in more efficient way. Therefore, less information loss is expected after performing low-rank tensor- based DNN compression. To verify our hypothesis, we compare the feature maps of the compressed convolutional layer of ResNet-20 CIFAR-10 dataset using dif-
4

---

Under review as a conference paper at ICLR 2022
(n x c h)
chw)
(xxr)
(r chw)
Reshape
Matrix Decompasition
X
B
R
A
Weight Tensor W (it x x w)
(xxxXX1) (xxxx(t
( (2xh113)
(63 X
Tensor Train
Decomposition
91
G3
*
94
Weight Tensor
TT-Cores
Figure 3: Exploring low-rankness of convolutional layer via matrix decomposition (Top) and tensor decompo-
sition (Bottom). Here tensor train (TT) decomposition is adopted for illustration.
Baseline
Tensor Decomposition
(TT)
Matrix Decomposition (SVD)
0.3
0.1
6.3
0.0
0.0
0.0
-02
002
-02
6
Figure 4: Output feature maps of one layer of ResNet-20 after non-compression (Left), tensor decomposition (Middle) and matrix decomposition (Right). The visualization shown here is based the information of channel. is seen that high-order tensor decomposition makes feature map the compressed layer more similar to that of the original uncompressed layer. More detailed results are reported in Appendix (
ferent low-rank methods. As visualized in Figure compared with the matrix decomposition-basse approach with the same compression ratio, tensor decomposition can make the output feature map of the compressed layer much more similar to the feature map of the original uncompressed layer. In other words, low-rank tensor method can provide better preservation of the important feature information and thus it can bring potential higher model compression performance.
Question #3: What is the best suitable optimization objective that the integrated compression scheme should aim?
Analysis. To efficiently realize the joint exploration of model low-rankness and sparsity with promis- ing compression performance, different optimization strategies have been proposed in the existing works. For instance, Yu et al. (2017): Ma et al.) 2019) aim to minimize the difference between the original weight matrix/tensor and the approximated reconstruction. In addition, Ma et al. (2019) pro- poses explicitly add the low-rank and sparse regularization terms to the overall objective function, which can guide the maining-wwre procedure to enforce the desired structural patterns.
Our Proposal. Different from the existing approximation crror-centered regularized boss-based lutions, we propose that the efficient co-cxploration scheme should interpreted as the optimization procedure with the low-rank and sparse constraints. Our rationale lies two important observations of the inherent drawbacks of the prior efforts. First, the approximation strategy adopted in Yu et al. (2017); Ma et al. (2019) focuses on making the reconstructed model approach the original model as close as possible. However, since I) the approximation error always exists; and 2) the original model not the only choice achieve the desired accuracy, such strategy inherently can only search the low-rank and sparse components in a limited exploration space, thereby affecting the overall com- pression performance. Second, though adding the regularization terms into loss function indeed fa- cilitates the extraction of low-rank and sparse patterns, the effect of such simple regularizing method still limited, especially considering the efforts of pushing for sparsity and for low-rankness may interfere with each other, thereby potentially causing unexpected conflicts. Instead, by explicitly im- posing the low-rank and sparse constraints on the overall optimization problem, these two structural requirement be simultaneously satisfied with the proper of optimization technique (to be discussed in Section 4p. As reported in Appendix A our proposed constrained optimization strategy successfully impose the desired low-rankness and sparsity onto the DNN models efficiently.
Summary of Our Analysis. Performing joint low-rank decomposition and pruning in a spatially parallel way (L+S) is the preferable operational sequence. @ High-order tensor decomposition is

---

Under review as a conference paper at ICLR 2022
the most suitable choice for the low-rank approach used in the integrated compression scheme. 3 Imposing low-rankness and sparsity as the direct hard constraints on the loss optimization should be adopted to better satisfy the desired structural requirement.
4 CO-EXPLORING LOW-RANKNESS AND SPARSITY: OUR METHOD
Based on the above three key takeaways obtained from Section in this section we propose a unified framework to co-explore low-rankness and sparsity in an efficient way. To be specific, we first formulate the integration of low-rank tensor decomposition and pruning to unified optimization problem, and then develop an efficient algorithm to solve this non-convex high-order tensor-format problem.
4.1 PROBLEM FORMULATION
Recall that the analysis in Section brings three important observations/proooaale using L+S oper- ational sequence, choosing tensor decomposition, and directly imposing hard constraints. Built on such three fundamental principles, we are now ready to formulate the integration of pruning and ten- sor decomposition
unified optimization problem. To be specific, given an uncompressed DNN model with weight tensor W E RoxIIX xK of each layer, our goal is to find another compact model with weight tensors + which consists of low-rank component E ROxIxXXX and sparse component S OXXIX x for each layer, minimize the following loss function:
min f(C,8), E,S
s.t. rank(C) S Yo. -1.-. nd.
(S) SKK.
Low-tensoor-ran constraint
Sparse constraint
where f()) is the loss over the entire training dataset, and Tip T1.-.. Va and are the desired tensor ranks and the number of non-zero entries for L and S, respectively. Notice that without loss of generality, we choose tensor train (TT) decomposition as the component high-order low- rank method in framework. So here d is the number of decomposed tensor cores with TT decomposition.
4.2 OPTIMIZATION
Directly optimizing problem 11 challenging because of the co-existence the non-differentiable rank(-) and card(.) ) as well as its inherent high-order tensor format. To efficiently solve this problem, we propose leverage alternating direction optimization method split these two constraints. To be specific, after introducing two auxiliary variables C and S that represent the desired low-TT- rankness and sparsity in the optimization process, problem ( can be then rewritten as:
min L.S.CEP.BE
((C,S),
s.t. L 2
where P ([)akn((C)) < ..... na) is the set of all tensors that satisfy the low-tensor-ramt constraint, and Q = {S (card(S) < k) is the set all tensors that satisfy the sparse constraint. Then, further relax the hard constraints to the corresponding augmented Lagrangian form and now we only need to optimize the following new constraint-free min-max problem:
f(C,S) À (11 + + SS- - v + 1 + 11)1)
min C.S.CEP,BE. u.
(3)
where 24 and V are the dual multipliers associated to L and S, respectively, and À is the penalty pa- rameters. To solve this minmax problem, we can split it into three separated parts, and independently optimize them in iterative way.
Update and S with SGD. The first independent optimization objective can be formulated as:
f(C,)) X (1 2 + + + || - + +1 2.
min
(4)

---

Under review as a conference paper at ICLR 2022
Since there hard constraints the target variables L and 8, standard DNN optimizer (e.g., stochastic gradient descent (SGD)) can be directly applied with learning rate a as:
L - - /ef X(C - +2), - S+ V)].
(5)
8 8 a-8-aaVsf((.55
(6)
Update with TT Decomposition. To update the introduced Σ, the optimization objective is:
ΣEP - - -
min
Because Σ is strictly constrained stay in the low-tensor-rank set P, the desired update can be performed using an analytical solution via TT-rank truncation, i.e.
2 truncp ( C U))
(8)
To realize such truncating operation, first define temporary tensor = +U and reshape as a new tensor T E g ((XX)))x( x/)xx. x(Oxxl with O I Okr I Ix. Then T can be decomposed to d + TT-cores
T(kk, k2), (ot, i1), (d)) Co C2 01,11;)) Cali, oa, id;
(9)
= 1,... ,d. In this TT-format, the dimensions of TT-ranks in TT-cores are truncated to the desired target, i.e., C' = C,(11 7--1,441 79). After that we use the truncated TT-cores to recover the original tensor via:
where Co E K Cj E 1RB-1x0,xIIxX,,
7' (((kk) k2), (o,, (1), ... (oa, id))) = Co k2)
C'al:, oa, da. (10)
And finally 7' reshaped to the original shape of 2 to serve as the updated 2 Update with Projection. For updating S, the third optimization objective is:
min SeQ
(11)
Similar to the low-tensor-rank Σ, the sparse-constrained S can also be analytically updated as
$ + V),,
(12)
where proj(-) is the projection that removes the smallest values to ensure that the updated S can satisfy the sparse constraint.
Update Multipliers U. Upon the update of 2 and S, the dual multipliers U and can be then directly updated as:
u+ u- -- 2, V V
(13)
Notice that after the iterative update finishes, the low-rank component L is explicitly decomposed TT-cores [C){=n- and the entire compressed model consisting of TT-cores and sparse part S is finally fine-tuned with standard SGD. The overall RASPA algorithm is summarized in Algorithm
5 EXPERIMENTS
5.1 EXPERIMENTAL SETTING
Dataset and Baseline. We evaluate our proposed approach two image classification datasets (CIFAR-10 and ImageNet). For experiments on CIFAR-10 dataset, three CNN models (ResNet-20, ResNet-56 and D40)eNet-40) are compressed and tested. For experiments on ImageNet dataset, we evaluate our approach for ResNet-50 and compare its performance with tfat--oftthe-s model compression methods.

---

Under review as a conference paper at ICLR 2022
Hyperparameter. All the experiments conducted using SGD optimizer with batch size, momentum and weight decay 128, 0.9 and 0.0005, respectively. The learning rates adopted in the optimization and fine-tuning pro- cess are set as 0.1 and 0.005, respectively. Within the total 180 epochs, the learning rate is divided by 5 at epoch 54, 108, 144 and 171 gradually. The entire training procedure is per- formed on RTX 3090 GPUs with Pytorch 1.8.1.
The overall RASPA algorithm for co-exploring model low-rankness and sparsity
Algorithm 1
Pre-trained weight tensor W, target TT- ranks {ng)) sparse target k, training epochs T',
Input:
Output: TT-cores [C] sparse component S. Initialize L, 2, with W:
2: Initialize 41 0, V.V.= :
3: for = 1 to T do
4: Update U. V using Eq. 13
5: Update and S using Eq. and Eq.
6: I/ Update C using TT-truncation
5.2 EVALUATION
7: Ž + trunc +2);
AND COMPARISON ON CIFAR- 10 DATASET
Table the evaluation results of the com- pressed ResNet-20, ResNet-56 and DenseNet- 40 models on CIFAR-10 dataset. For each model, we compare the performance of our proposed RASPA with several types of com- pression methods, including decomposition- only (L), pruning-only (S), first-pruning- then- decomposition (L(S)), and layer-wise either- wwwntru-oonnnnnoononnnee or-decomposition (S/L)).
8: I/ Update 8 using projection
9: S --roojo(S+ +V):
10: end for
11: Decompose C TT-cores (C)
12: Fine-tune model with {C} and S.
ResNet-20. For ResNet-20 model, the proposed RASPA solution can bring 1.25% accuracy in- crease over baseline model with 70.4% and 71.1% model size and FLOPs reductions, respectively. With even more aggressive compression strategy aiming 85.3% smaller model size and 86. % fewer FL.OPs, our approach can still enable 0.88% higher accuracy than the original ResNet-20 model.
ResNet-56. For ResNet-56 model, our approach can bring 1.02% accuracy increase over baseline model with 37.5% and 39.3% model size and FLOPs reductions, respectively. When we perform more aggressive compression with 65.6% and 66.0% fewer parameters and computations, our com- pressed ResNet-56 can still enjoy 0.64% higher accuracy than the original uncompressed model, thereby exhibiting very superior performance than other tfate-ofthhe-a art DNN compression meth- ods.
DenseNet-40. For DenseNet-40 model, our proposed sparstyylow-rankness co-cxploration can bring 0.16% accuracy increase the baseline model with 52.4% and 61.3% model size and FLOPs reductions, respectively. Moreover, with further higher compression effect, our RASPA approach can still enable 0.07% higher accuracy than the original uncompressed model with 65.3% and 74.5% fewer parameters and computations, respectively; while the existing approaches suffer accuracy loss with even lower compression ratio.
5.3 EVALUATION AND COMPARISON IMAGENET DATASET
Table summarizes the compression performance of our approach and other existing works for Re on ImageNet dataset. It is seen that our RASPA solution can bring 0.52% accuracy increase over baseline model with 48.7% fewer parameters. When targeting for generating more compact model, our approach can still achieve high performance it only has 0.25% accuracy drop with 58.6% model size reduction, which means it shows better test accuracy than its counterparts with even higher compression ratio.
5.4 DISCUSSION ANALYSIS
To obtain deep understanding of our proposed approach, we also perform empirical analysis and ablation studies on the co-exploration procedure. The details are referred to Appendix A
CONCLUSION
In this paper we propose to systematically co-explore DNN low-rankness and sparsity for efficient model compression. By performing comprehensive analysis critical design factors, propose
8

---

Under review as a conference paper at ICLR 2022
Table 1: Experimental results on CIFAR-10 dataset. Here "L" denotes low-rank decomposition and "S"
denotes sparsification (pruning). Notice that no prior tensor decompasition work reports performance for compressing ResNet-56 and DenseNet-40.
Decomp. Format
Top-1 Accuracy (%)
Params.
FLOPs
Compression Method
Тype
Bascline
Comp.
A
(%)
(%)
ResNet-20
PSTRN
Li ct al.
2
L
Tensor
91.25
90.80
-0.45
55.6
N/A
PSTRN Li ct al.
2021a
Tensor
91.25
89.30
-1.95
85.2
N/A
TRP (x
et al. 20201
L
Matrix
91.74
90.88
-0.66
48.1
51.0
SVDT Yang et al. (2020)
Matrix
90.93
90.97
+0.04
N/A
54.5
Hinge el al.
(0200)
S/L
Matrix
92.54
91.84
-0.70
55.5
54.5
SCOP
Tang et
2020
N/A
92.22
90.75
-1.47
56.3
55.7
FPGM
He al.
20191)
S
N/A
92.20
90.44
-1.76
51.0
54.0
RASPA
(Ours)
L+S
Tensor
91.25
92.50
+1.25
70.4
71.1
RASPA (Ours)
L+S
Tensor
91.25
92.13
+0.88
85.3
86.1
ResNet-56
TRP (Xu ct al.
2020
L
Matrix
93.14
92.77
-0.37
N/A
56.7
HRank
20200
N/A
93.26
93.52
+0.26
16.8
29.3
HRank Lin ct al.
20200
S
N/A
93.26
93.17
-0.09
42.4
50.0
SVDT Yang
(2020))
Matrix
93.28
93.67
+0.39
N/A
63.0
CC
(Li al.
202Tb)
L.S)
Matrix
93.33
93.87
+0.54
36.5
42.4
CC
(L
2021b
L(S)
Matrix
93.33
93.64
+0.31
48.2
52.0
RASPA (Ours)
L+S
Tensor
93.27
94.29
+1.02
37.5
39.3
RASPA (Ours)
L4S
Tensor
93.27
93.91
+0.64
65.6
66.0
DenseNct-40
HRank Lin
2020
S
N/A
94.81
94.24
-0.57
36.5
40.8
HRank Lin et al.
2020
N/A
94.81
93.68
-1.13
53.8
61.0
Hinge
(ii et al.
(0200)
S/L
Matrix
94.74
94.67
-0.07
27.5
44.4
CC Li ct al.
2021b
L(S)
Matrix
94.81
94.67
-0.14
51.9
47.0
CC
(Li et 20216
L(S)
Matrix
94.81
94.40
-0.41
64.4
60.4
RASPA (Ours)
L+S
Tensor
94.81
94.97
+0.16
52.4
61.3
RASPA (Ours)
L+S
Tensor
94.81
94.88
+0.07
65.3
74.5
Table 2: Experimental results ImageNet dataset. Here "L." denotes low-rank decomposition and "S" de- notes sparsification (pruming). Notice that no prior tensor decomposition work reports performance compressing ResNet-50.
Decomp. Format
Top-1 Accuracy (%)
Top-5 Accuracy (%
Params. ((%)
Compression Method
Type
Base.
Comp.
A
Base.
Comp.
A
ResNet-50
TRP (uu ct al.
2020
I.
Matrix
75.90
74.06
-1.84
92.70
92.07
-0.63
44.4
HRank et al..
2020
76.15
74.98
-1.17
92.87
92.33
-0.54
36.7
SCOP Tang al.
20200
S
N/A
76.15
75.95
-0.20
92.87
92.79
-0.08
42.8
SVDT Yang (2020)
Matrix
N/A
N/A
N/A
91.91
91.97
+0.06
30.6
CC (Li et al 2021b))
L(S)
Matrix
76.15
75.59
-0.56
92.87
92.64
-0.23
48.4
RASPA (Ours)
L+S
Tensor
76.
76.65
+0.52
92.86
93.14
+0.28
48.7
TRP (Xu et al. (2020))
L
Matrix
75.90
72.69
-3.21
92.70
91.41
-1.99
56.3
HRank Lin el (2020)
N/A
76.15
71.98
-4.17
92.87
91.01
-1.66
46.0
SCOP Tang et all
2020
S
N/A
76.15
75.26
-0.89
92.87
92.53
-0.34
51.8
Hinge Li 2020)
S/L
Matrix
76.15
74.70
-1.45
N/A
N/A
CC (Li al. 2021b)
L(S)
Matrix
76.15
74.54
-1.61
92.87
92.25
-0.62
58.6
RASPA (Ours)
L+S
Tensor
76.13
75.29
92.86
92.61
58.6
RASPA, unified compression framework that capture model low-rankness and sparsity simul- taneously and efficiently. Evaluation results show that our proposed approach can bring significant model size and computational cost reductions while still preserving high model accuracy.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Misha Denil, Babak Shakibi, Laurent Dinh, Marc' Aurelio Ranzato, and Nando De Freitas. Predict- ing parameters in deep learning, arXiv preprint arXiv:1306.00 0543, 2013.
Abhimanyu Dubey, Moitreya Chatterjee, and Narendra Ahuja. Coreset-based neural network com- pression. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 454-470, 2018.
Shangqian Gao, Feihu Huang, Weidong Cai, and Heng Huang. Network pruning via performance maximization. In Proceedings of the IEEE/CVE Conference on Computer Vision and Patlern Recognition, pp.
2021.
Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert Mullins, and Cheng-zhone Xu. Dynamic channel pruning: Feature boosting and suppression. In International Conference on Learning Represen- tations, 2019.
Ruihao Gong, Xianglong Liu, Shenghu Jiang. Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 4852-4861, 2019.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dons. arXiv preprint arXiv: 04493,, 2016.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00144 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in Neural Information Processing Systems, 28:1 135-1143, 2015b.
Yang He, Ping Liu, Ziwei Wang. Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4340-4349, 2019.
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Com- pression of deep convolutional neural networks for fast and low power mobile applications. In International Conference on Learning Representations, 2016.
Chong Li and CJ Shi. Constrained optimization based low-rank approximation of deep neural net- works. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 732-747, 2018.
Nannan Li, Yu Pan, Yaran Chen, Zixiang Ding, Dongbin Zhao, and Zenglin Xu. Heuristic rank selection with progressively searching tensor ring network. Complex & Intelligent Systems, pp- 1-15, 2021a.
Yawei Li, Shuhang Gu, Christoph Mayer, Lue Van Gool, and Radu Timofte. Group sparsity: The hinge between filter pruning and decomposition for network compression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8018-8027, 2020.
Yuchao Li. Shaohui Lin, Jianzhuang Liu, Qixiang Ye, Mengdi Wang. Fei Chao, Fan Yang, Jincheng Ma, Qi Tian, and Rongrong Ji. Towards compact cnns via collaborative compression. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6438-6447 2021b.
Siyu Liao, Chunhua Deng, Miao Yin, and Bo Yuan. Doubly residual neural decoder: Towards low-complexity pigh-performance channel decoding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8574-8582, 2021.
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1529-1538, 2020.
10

---

Under review as a conference paper at ICLR 2022
Yuzhe Ma, Ran Chen, Wei Li, Fanhua Shang, Wenjian Yu, Minsik Cho, and Bei Yu. A unified approximation framework for compressing and accelerating deep neural networks. In 2019 JEEE st International Conference Tools with Artificial Intelligence (ICTAI), pp. 376-383. IEEE, 2019.
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. Advances in Neural Information Processing Systems, 28:442-4455 2015.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. arXiv preprini urXiv:2106.0203: 2021.
Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, et al. Convolutional neural networks with low- rank regularization. In International Conference Learning Representations, 2016.
Ychui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing XU, Chao Xu, and Chang Xu. Scop: Scientific control for reliable neural network pruning. In Advances in Neural Information Pro- cessing Systems, volume 33, pp. 10936-10947 2020.
Wenqi Wang, Yifan Sun, Brian Eriksson, Wenlin Wang, and Vaneet Aggarwal. Wide compression: Tensor ring nets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2018.
Zi Wang, Chengcheng Li, and Xiangyang Wang. Convolutional neural network pruning with struc- tural redundancy reduction. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 14913-14922, 2021.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances neural information processing systems, 29:2074-2082, 2016.
Yuhui Xu, Yuxi Li, Shuai Zhang, Wei Wen, Botao Wang, Yingyong Qi, Yiran Chen, Weiyao Lin, and Hongkai Xiong. Trp: Trained rank pruning for efficient deep neural networks. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pp. 977- 983, 2020.
Huanrui Yang, Minxue Tang, Wei Wen, Feng Yan, Daniel Hu, Ang Li, Hai Li, and Yiran Chen. Learning low-rank deep neural networks via singular vector orthogonality regularization and sin- gular value sparsification. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition workshops, pp. 678-679, 2020.
Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7370-7379, 2017.

---

Under review as a conference paper at ICLR 2022
A DETAILS OF EMPIRICAL ANALYSIS IN SECTION 5.4
In this section present the details of empirical analysis and discussion in Section 5.4 Notice that the example model investigated and evaluated here is ResNet-20 on CIFAR-I0 dataset.
Simultaneously Obtaining Low-rankness and Sparsity. Figure shows the loss curves during the model optimization procedure. Notice that here besides overall training loss, the individual low- rank and sparse loss component, which directly reflects the progress of enforcing low-rankness and sparsity. respectively, is also explicitly reported in this figure. It is seen that our proposed approach indeed successfully imposes the desired -eesseerrnnness and sparsity with hard constraints onto the model, and thereby ensuring that the compressed model can fully exhibit both low-rank and sparse characteristics after the optimization.
Effect of Optimization Procedure. We also study the benefits of using our proposed optimization procedure described in Algorithm [ solve problem (2). Here we compare our approach with direct method that performs TT decomposition and pruning on the uncompressed model straight- forwardly with the same TT-ranks and sparsity settings. In addition, the same fine-tuning process that RASPA adopts is also applied in this direct method. Figure shows the comparison results with respect to different compression ratios. is seen that our proposed optimization procedure brings significant accuracy increase.
Visualization. Figure 7 visualizes the weight tensor of one convolutional layer in a pre-trained ResNet-20 model before and after performing our proposed compression approach. Here the visu- alization of the low-rank and sparse components of the compressed layer is also illustrated in this figure. It is seen that most of the weight information is preserved in the low-rank component, and meanwhile the sparse component contains some spatial pattern as well.
Overall loss
Low-rank part of loss
Loss
2
Sparse part of loss
100
150
Epochs
Figure 5: Loss curves a 2esNet-20 trained CIFAR- 10 dataset using RASPA algorithm. Here the curves of the individual low-rank and sparse loss components also illustrated. It is seen that the low-rankness and sparsity are indeed imposed on the model via using RASPA.
Top-1 Accuracy
92.5
Direct SPARK
92.0
2.0
2.5
3.5 Compression Ratio
Figure 6: The effect of optimization procedure for jointly TT decomposing and pruning ResNet-20 on CIFAR- 10. Here RASPA and the direct method the same TT-rank setting and sparsity ratio. However, the direct method does not perform optimization on original model. Instead, first performs TT decomposition and then prunes the difference hetween original model and the low-rank component to obtain the sparse component. The two components generated by this direct method will then be fine-tuned with the same way that RASPA uses. It is seen that our proposed optimization procedure in RASPA brings significantly accuracy increase.
12

---

Under review as a conference paper at ICLR 2022
Baseline
0.00
SPARK compression
0.06
Low-rank component
0.00
Sporse component
0.00
0.04
0.04
0.04
0.00
0.00
Figure 7: Visualization of one layer of ResNet-20 before and after performing proposed RASPA compres- sion. Here the low-rank and sparse components of the compressed layer also visualized. It is that low-rank component preserve most of weight information, and some spatial patterns are contained in the sparse component.
B DETAILS OF ANALYSIS IN SECTION 3
B. MORE RESULTS FOR ANALYSIS IN QUESTION
Table and show the layer-wise approximation errors incurred by three operational sequences (L+S, S(L) and L(S)). Here the baseline models include well-trained ResNet-20, ResNet-56 on CIFAR-10 10 dataset, and ResNet-50 on ImageNet dataset, and the compression ratio 3.0 for all the layers. It seen that L+S scheme always brings the smallest approximation errors.
Table 3: Approximation errors for different layers ResNet-20 with different operational se-
quences. The compression ratio is set as 3.0 for all the layers.
Layer
Weight Shape
L+S
S(L)
I(S)
Name
rank
sparse
rank
sparse
rank
sparse
l1b1.covv
(16, 16,3,3) 3,3)
0.9
1.32
0.1
1.73
4
0.1
1.95
11b1.comv2
(16, 16, 3,3)
0.9
1.31
0.1
1.61
0.1
11 11b2.convl
(16, 16, 3,3)
3
0.9
1.29
5
0.1
1.63
4
0.1
1.78
11b2.conv2
(16, 3,3)
3
0.9
1.33
5
0.1
1.53
4
0.1
1.67
11b3.convl
(16, 16, 3,3)
0.9
1.65
0.1
2.05
4
0.1
2.25
11b3.conv2
(16, 3,
0.9
0.1
0.1
12b1.convi
(16, 32, 3,
6
0.9
1.91
9
0.1
2.22
8
0.1
2.36
12b1.conv2
(32, (2. 3,
6
0.9
2.44
10
0.1
2.84
0.1
2.97
12b2.convl
0.9
2.34
10
0.1
2.81
0.1
2.92
12b2.conv2
32,
0.9
10
0.1
0.1
2.65
con3.convl
(32, 3,
6
0.9
2,36
10
0.1
2,80
9
0.1
2.91
12b3.conv2
32. 32,
6
0.9
2.02
10
0.1
2.34
9
0.1
2.44
13b1.convl
(32, 3,3)
12
0.9
2.98
19
0.1
3.41
17
0.1
3.59
13b1.conv2
(64, 3,
13
0.9
0.1
19
0.1
4.82
13b2.conv]
(64, 3,
13
0.9
4.18
21
0.1
4.86
19
0.1
5.05
cob2.conv2
(64, 64.
13
0.9
3.49
21
0.1
4.01
19
0.1
4.18
13h3.convl
(64, 3,3)
13
0.9
3.86
21
0.1
4.47
19
0.1
4.65
13b3.conv2
(64, 3,
13
0.9
0.40
0.1
0.42
19
0.1
0.44
B.2 MORE RESULTS FOR ANALYSIS IN QUESTION #2
Table/ojshows the difference (in term of mean square error (MSE)) between the output feature maps of original layer and the compressed one in ResNer-20 on CIFAR-10 dataset. Here SVD and tensor train (TT) are adopted for matrix decomposition and tensor decomposition, respectively. seen that, with the same even higher compression ratio, high-order tensor decomposition always brings smaller approximation error than the matrix decomposition.
Table 7 shows the difference between the output feature maps of one original layer and the com- pressed version in ResNet-20 on CIFAR-10 10 dataset and the difference the final accuracy. Here SVD, Tucker and TT are adopted for low-rank decompositions. It is that high-order tensor decomposition always brings smaller approximation error than the matrix decomposition and TT always has the least impact the final accuracy.
13

---

Under review as a conference paper at ICLR 2022
Table 4: Approximation errors for different layers of ResNet-56 with different operational
quences. The compression ratio is set as 3.0 for all the layers.
Layer
Weight
L+S
S(L)
L(S)
Name
Shape
rank
sparse
rank
sparse
rank
sparse
eπror
IIblconv
(16, 16, 3, 3)
3
0.9
1.71
5
0.1
2.20
4
0.1
2.32
11b1.conv2
(16, 16, 3,
0,9
0.1
0.1
2.72
11b2.conv1
(16, 16, 3.3)
3
0.9
2.32
5
0.1
2.81
4
0.1
3.19
11b2.comv2
(16, 3,3)
3
0.9
2.29
5
0.1
2.69
4
0.1
3.00
c1n3.conv
(16, 3,33,
3
0.9
2.03
5
0.1
2.48
4
0.1
2.70
11b3.conv2
(16, 16,3,3)
0.9
2.17
0.1
2.57
0.1
2.81
11b4.conv]
(16, 3,33
0.9
0.1
0.1
1lb4.conv
(16, 3,33
0.9
2.16
5
0.1
2.48
4
0.1
2.70
[1bS.conv
(16, 1,.3,3)
3
0.9
2.24
5
0.1
2.73
4
0.1
3.04
11b5.conv2
(16, 1,,
0.9
1.87
5
0.1
2.25
0.1
2.47
l1b6.conv1
(16, 3,33
0,9
1.84
0.1
2.28
0.1
2.55
l1b6.conv2
(16, 3,3)
3
0.9
1.73
0.1
2.07
0.1
2.33
11b7.convl
(16, 3,33
0.9
2.59
0.1
3.21
4
0.1
3.53
11b7.comv2
(16, 1,33,3)
0.9
2.07
5
0.1
2.38
4
0.1
2.63
11b8.convl
(16, 3,,
0.9
2.52
0.1
2.90
0.1
3.38
11b8.conv2
(16, 3,3)
0.9
0.1
0.1
2.50
11b9.conv1
(16, 3,3)
3
0.9
1.76
0.1
2.14
0.1
2.46
11b9.conv2
(16, 16.
3
0.9
1.45
0.1
1.76
0.1
1.95
12h1.convi
(32, 3,3)
0.9
3.45
0.1
4.12
0.1
4.38
12b1.conv2
3,
0.9
10
0.1
4.89
0.1
5.14
12b2.conv1
(32, 32.
6
0.9
2.72
10
0.1
3.38
0.1
3.55
12b2.conv2
(32, 3,
6
0.9
2.89
10
0.1
3.37
0.1
3.53
12b3.convl
(32, 32, 3,
0.9
3.36
10
0.1
4.08
0.1
4.27
12b3.conv2
(32, 3,
0.9
3.34
10
0.1
3.92
0.1
4.09
12b4.convl
(32, 32,
0.9
3.35
10
0.1
4.05
0.1
4.26
12b4.conv2
(32, 3,
0.9
3.21
10
0.1
3.73
9
0.1
3.90
12b5.convl
3,
0.9
3.39
10
0.1
4.02
9
0.1
4.21
12h5.conv2
0.9
3.00
10
0.1
3.51
0.1
3.67
12b6.conv1
(32,
0.9
10
0.1
4.00
0.1
4.20
12b6.conv2
32,
0.9
3.04
10
0.1
3.55
0.1
3.73
12b7.convl
32.
0.9
3,40
10
0.1
4.02
9
0.1
4.21
12h7.comv2
(2, 3,
0.9
2.83
10
0.1
3.34
0.1
3.51
12h8.convl
3,
0.9
3.31
10
0.1
3.93
0.1
4.12
12b8.conv2
32, 3,
0.9
10
0.1
0.1
3.53
12b9.conv1
32.
6
0.9
3.40
10
0.1
4.03
0.1
4.21
12b9.conv2
(32, 3,
0.9
2.61
10
0.1
3.06
0.1
3.22
[3b].conv]
(64, 32,
12
0.9
5.13
19
0.1
5.92
0.1
6.26
13b1.conv2
(64, 6,,
13
0.9
6.45
0.1
7.63
19
0.1
7.99
13b2.comv1
(64, 6,
13
0.9
5.36
21
0.1
6.63
19
0.1
6.92
13b2.comv2
(64, 3,33
13
0.9
5.94
21
0.1
6.88
19
0.1
7.17
13b3.convl
(64, 3,
13
0.9
5.65
21
0.1
6.77
19
0.1
7.08
13b3.conv2
(64, 3,
13
0,9
5.40
0.1
6.34
19
0.1
6.62
13b4.conv]
(64, 6,
13
0.9
6.38
21
0.1
7.55
19
0.1
7.88
l3b4.conv2
64.
13
0.9
5,43
21
0.1
6.30
19
0.1
6.58
13h5.comvl
6,,
13
0.9
6.43
21
0.1
7.59
19
0.1
7.91
13h5.comv2
(64, 3,33
13
0.9
4.94
0.1
5.73
19
0.1
7.91
13b6.convl
(64, 3,3)
13
0.9
0.1
19
0.1
13b6.conv2
64.
13
0.9
4.25
21
0.1
4.84
19
0.1
5.08
13b7.convi
(64, 6,.
13
0.9
5.31
21
0.1
6.14
19
0.1
6.42
13b7.conv2
(64, 3,
13
0.9
3.32
21
0.1
3.76
19
0.1
3.95
13b8.conv1
(64, 64,
13
0.9
4.12
0.1
4.71
19
0.1
4.98
13b8.conv2
(64, 6,
13
0.9
2.14
21
0.1
2.39
19
0.1
2.55
3eb9.convl
(64, 64, 3,
13
0.9
3.32
21
0.1
3.65
19
0.1
3.87
13b9.conv2
(64, 64, 3,
13
0.9
1.41
21
0.1
1.49
19
0.1
1.60
14

---

Under review as a conference paper at ICLR 2022
Table 5: Approximation errors for different layers of ResNet-50 with different operational
quences. The compression ratio is set as 3.0 for all the layers.
Layer
Weight Shape
L+S
S(L)
L(S)
Name
rank
sparse
еттОr
rank
sparse
егтог
rank
sparse
eтror
Ilblcoovv
(64, 64, I. 1)
7
0.9
1.65
11
0.1
2,43
10
0.1
2,60
11bl.conv
(64, 64, 3, 3)
21
0.1
1.82
19
0.1
2.01
11bl.conv3
(256, 64, 1)
11
0.9
1.62
18
0.1
1.82
17
0.1
1.95
11h2.comv1
(64, 256, 1)
11
0.9
1.49
18
0.1
1.79
17
0.1
1.86
11b2.conv2
(64, 64, 3,
13
0.9
2.34
21
0.1
2.73
19
0.1
2.88
11b2.conv3
(256, 64, 1)
11
0.9
1.70
18
0.1
2.32
17
0.1
2.40
11b3.conv1
(64, 256, I)
1.55
18
1.79
17
0.1
1.85
11b3.conv2
(64, 64, 3,
13
0.9
2.93
21
0.1
3.26
19
0.1
3.44
11b3.conv3
(256, 64, I)
11
0,9
1.56
18
0.1
2.20
17
0.1
2.28
12b1.convl
(128, 256, 1)
0.9
2.63
31
0.1
2.99
28
0.1
3.20
12b1.conv2
(128, 128, 3)
26
4.27
42
4.78
38
0.1
5.07
12b1.conv3
(512, 128, 1)
23
0.9
2.94
37
0.1
3.84
34
0.1
4.06
12b2.conv
(128, 512, 1)
23
0.9
1.04
37
0.1
1.17
34
0.1
1.24
12b2.conv2
(128, 128, 3.
26
0.9
1.90
42
0.1
2.00
38
0.1
2.17
12b2.conv3
(512, 128. 1)
23
1.58
37
0.1
2.04
34
0.1
2.18
12b3.convl
(128, 512, I)
23
0.9
2.48
37
0.1
2.78
34
0.1
2,93
12h3.conv2
(128, 3,3)
26
0.9
3.67
42
0.1
4.15
38
0.1
4.40
12h3.conv3
(512, 128, 1,
23
0.9
3.09
37
0.1
3.63
34
0.1
3.82
12h4.convl
(128,
23
0.9
2.81
37
0.1
3.18
34
0.1
3.33
12b4.conv2
(128, 128. 3.3)
26
4.38
42
4.95
38
0.1
5.22
12b4.conv3
(512, 128, 1)
23
0.9
2.84
37
0.1
3.57
34
0.1
3.73
13bl.convl
512,
39
0.9
4.96
63
0.1
5.72
56
0.1
6.11
13b1.conv2
3.3,
53
0.9
6.18
85
0.1
6.77
76
0.1
7.27
13b1.conv3
(1024, 1.
47
5.55
75
6.42
0.1
6.83
13b2.conv1
(256, 1024, 1)
47
0.9
3.12
75
0.1
3.47
68
0.1
3.71
13b2.conv2
256, 3.3)
53
0,9
5.09
85
0.1
5.56
76
0.1
5,94
13b2.conv3
256.
47
0.9
4.94
75
0.1
5.79
68
0.1
6.08
13b3.convl
(256, 1024,
47
0.9
3.37
75
0.1
3.72
68
0.1
3,95
13b3.conv2
256
53
5.37
85
0.1
5.94
76
0.1
6.32
13n3.comv3
256, 1,
47
0.9
4.54
75
0.1
5.27
68
0.1
5.55
13b4.convi
(256, 1,
47
0.9
4.22
75
0.1
4.78
68
0.1
5.04
13b4.conv2
256.
53
0.9
5.76
85
0.1
6.48
76
0.1
6.85
13b4.conv3
(1024, 256, 1.
47
4.55
75
5.32
68
0.1
5.59
13b5.conv1
(256, 1,
47
0.9
4.66
75
0.1
5.33
68
0.1
5.59
13b5.conv2
(256, 256.
53
0.9
5.77
85
0.1
6.52
76
0.1
6.88
13h5.conv3
256,
47
0.9
4.49
75
0.1
5.29
68
0.1
5.54
13b6.conv1
(256, 1024,
47
0.9
5.31
75
0.1
6.11
68
0.1
6.39
13b6.conv2
(256, 256, 3,3)
53
0.9
5.85
85
0.1
6.61
76
0.1
6.98
13b6.conv3
(1024,256, 1, 1)
47
0.9
4.83
75
0.1
5.66
68
0.1
5.93
14b1.covv
(512, 1024, 1,
79
0.9
8.85
126
0.1
10.13
113
0.1
10.63
14bl.conv2
(512, 512,
107
0.9
9.47
170
0.1
10.11
153
0.1
10.76
14b1.conv3
(2048, I,
9S
8.21
151
9.24
136
0.1
9.78
14h2.comvl
(512, 2048, 1,
95
0.9
7.82
151
0.1
8.74
136
0.1
9.24
14h2.conv2
5,2,
107
0.9
9.94
170
0.1
11.06
153
0.1
11.66
14b2.conv3
(2048, 512, 1,
95
0.9
7.91
151
0.1
8.94
136
0.1
9.43
14b3.conv1
(512, 2048, 1.
95
10.10
151
11.46
136
0.1
12.05
14b3.conv2
(512, 512, 3.3)
107
0.9
8.61
170
0.1
9.02
153
0.1
9.68
14h3.comv3
(2048, 512, 1,
95
0,9
7.11
151
0.1
7.83
136
0.1
8.42
C DISTRIBUTION OF TT-RANK AND SPARSITY RATIO
Table. g
10 and 11 list the layer-wise TT-rank and sparsity ratio of four compressed model.
Here Model and Model II in each table correspond to two reported models of each network with different Top-1 accuracy in Table and]]
15

---

Under review as a conference paper at ICLR 2022
Table 6: Approximation errors for the feature maps of different layers of ResNet-20 on CIFAR-10
dataset with matrix decomposition (SVD) and tensor decomposition (TT).
Compression Ratio
Approximation Error (MSE)
Layer Name
Weight Shape
SVD
TT
SVD
TT
llbl.com
(16, 16, 3, 3)
1.59
1.59
14.16
11.96
1Ib1.conv2
(16, 16, 3)
1.59
1.59
8.37
7.91
11b2.conv
(16, 16, 3,
1.79
1,85
25.35
13.79
11b2.conv2
(16, 16, 3.3)
1.79
1.85
8.49
5.72
11b3.conv1
(16, 16, 3,3)
1.79
1.85
27.20
26.61
11b3.conv2
(16, 16, 3)
1.79
1.85
8.22
6.57
12b1.conv
(16, 3,
2.00
2.11
15.40
10.13
12b1.conv2
32, 3,
1.91
2.03
8.81
5.17
12b2.conv
3,
1.91
2.03
15.12
12.72
12b2.conv2
(3,, 3.
1.91
2.03
4.13
2.99
12b3.conv1
(32, 3,
1.91
2.03
14.88
11.97
12b3.conv2
(32, 32,
1.91
2.03
3,23
2.18
[3b1.conv
(32, 64,3,)
2.01
2.01
10.52
2.95
13b1.conv2
(64, 6,,
1.98
2.01
5.14
3.85
13b2.conv
(64, 64,
1.98
2.01
12.42
8.52
13b2.conv2
(64, 64, 3,
1.98
2.01
2.70
2.04
13b3.conv
(64, 64, 3,
1.98
2.01
12.46
6.72
13b3.conv2
(64, 64, 3)
1.98
2.01
0.21
0.16
Table 7: Feature map approximation error layer3.0.conv2 in ResNet-20 and the corresponding accuracy drop with SVD, Tucker and TT in different compression ratio settings.
SVD
Tucker
TT
Compr. Ratio
Approx. Error
Acc.
Compr. Ratio
Approx. Error
Acc.
Compr. Ratio
Approx. Error
Acc.
A (%)
A (%)
A (%)
1.47
4.29
-.06
1.48x
3.72
-0.43
1.51x
3.16
-0.31
1.98
5.60
-3.0
1.97
4.80
-2.1
2.01x
4.16
-0.8
2.50x
6.57
-5.27
2.50x
5.41
-3.46
2.50x
5.13
-1.86
3.03x
7.35
-7.62
3.09x
6.02
-3.76
3.09
6.06
-3.18
3.38x
7.71
-9.66
3,46x
6.42
-4.54
3,57x
6,40
-4.03
4.11x
8.31
-11.32
4.18x
6.81
-6.04
4.20x
6.94
-5.41
D OTHER EXPERIMENTAL RESULTS
Please check Table 12
14 and 15 for other experimental results including different setting of
compressed ResNet- 20 and comparison with more related works from [I.1] to [A3].
E RELATED WORK
Pruning, Pruning is a very popular compression strategy that explores the sparsity of DNN models. Existing pruning methods can be performed with different granularity, e.g., weight pruning [E1] and channel pruning [F2][ with different pruning criteria, such as magnitude based [E1], rank based [F2], and nuclear norm based [F4].
Low-rank Decomposition. Low-rank compression methods aim to explore the intrinsic linear cor- relations of weights. general, low-rank decomposition can be performed via exploring matrix decomposition [[5][E6] and tensor decomposition [E7]. For compressing convolutional neural net- works, tensor decomposition is typically more suitable choice since high-order tensor decomposition can properly explore multidimensional correlation in the weight tensor; while the existing matrix decomposition based works [E5][E6] suffer the loss important spatial information incurred by inevitable tensor flatten operation.
16

---

Under review as a conference paper at ICLR 2022
Table 8: Layer-wise distribution Ra-Rank and sparsity ratio for the compressed ResNet--2 models on CIFAR-10 dataset.
Model with 92.13% Acc.
Model II with 92.50% Acc.
Layer Name
Weight Shape
TT Ranks
Sparsity (%)
TT Ranks
Sparsity (%)
IIbl.conv
(16, 16, 3, 3)
(1, 10, 10, 1)
90.0
(1, 16, 16, 1)
85.0
11b1.conv2
(16, 16, 3)3)
(1,
90.0
(1, 12, 12, 1)
85.0
11b2.conv
(16, 1,.3,3)
(1,8,8,
90.0
(1, 12, 12. 1)
85.0
11b2.comv2
(16, 16,
11,
90.0
(1, 12, 12, 1)
85.0
11b3.convl
(16, 3,33
(1,5,5,
90.0
11,
85.0
11b3.conv2
(16, 3,3)
(1,5,5,1)
90.0
10,
85.0
12b1.convl
3,333
(1,5,5,1)
90.0
10,
85.0
12b1.conv2
32,
(1,3,3,1)
90.0
(1,20,
85.0
12b2.conv1
32,
11,3,3,1)
90.0
10,
85.0
12b2.conv2
(32, 3,,
(1,3.3.1)
90.0
(1,
85.0
12b3.convl
(32, 32, 3,3)
(1,3,3,1)
90.0
10,
85.0
12b3.conv2
(3,,32.3,3
(1,4,4,
90.0
11, 10,
85.0
[3o1.convi
(32, 6,33
11,4,4,1)
90.0
13,
85.0
13b1.conv?
(64, 6,,
1),4,4.1)
90.0
11,
85.0
13b2.conv1
(64, 3,3)
(1,4.4.1)
90.0
10,
85.0
13b2.conv2
(64, 3,3
51,5,5,
90.0
(1, 10, 1)
85.0
13b3.conv1
(64, 3,3)
51,
90.0
(1. 10, 10,
85.0
13b3.conv2
(64, 3,
(1,5,5,
90.0
(1, 10, 10,
85.0
Co-exploring Sparsity/Low-rankeess for Model Compression. Recently, several papers have been proposed to co-explore sparsity and low-rankness for efficient DNÑ model compression. In particular, additive compression strategy (L+S) have been explored in some prior works [E8][E)][E1]] However, the low-rank component of all of these existing works are based on matrix decomposition. As analyzed before, high-order tensor decomposition
suitable low-rank methods for CNN compression; while this important technique has not been explored by the existing additive compression efforts yet. Also, [[8][[9] aims to closely approximate original models. As we analyze in Question #3, this strategy is not the optimal solution for DNN model compression, since our task goal is generate a compressed model instead of close approximation the original model. Different from these existing works, our approach is built on tensor decomposition with con- strained formulation as the optimization objective, thereby demonstrating better performance than [E8][E9][E10].
Co-exploring Sparsity/Low-rankness for Matrix Approxinaaiio//stimattoi sparsity/lww-rannness for efficient matrix approximation/esimation has also been explored [EI1- E14], However, the feasibility and efficiency of those general methods for DNN model compression not verified and not clear yet. To be specifie, our approach aims perform model compression deep neural networks with high accuracy; while those general methods aim to approximatc/cssimmat Consider modern DNN models consist of many large-size
Co-exploring
matrix with low approximation layers and they need maintain high accuracy on large-scale dataset, our approach is focusing a very different problem that these general methods do work on. Though it is possible that those gen- eral techniques may be applied to DNN model compression after certain unknown modification, the feasibility really verified yet, especially on large-scale dataset such as ImageNet. Also, the corresponding computational costs and accuracy performance are unknown.
ADMM-based Model Compression. recent years ADMM has been used in several model com- pression papers, including single pruning [E15][E16], single low-rank compression [E18], joint quantization and pruning [E17] and additive compressing [E8][E10]. Our approach focuses on ad- ditive compression with distinct difference from [E8][E10] and other ADMM-based works. This is because low-rank approach used in [E8][E10] based on matrix decomposition, which is not the best solution for compression CNNs; while we adopt high-order tensor decomposition in our pro- posed additive compression. From the perspective of ADMM process, the projection towards tensor decomposition is much
complicated than the one for matrix factorization. This is because 1) it is involved with advanced high-order tensor operation instead of straightforward 2-D matrix computation; and 2) tensor decomposition (e.g., TT) outputs multiple 4-D tensor cores; while ma-
17

---

Under review as a conference paper at ICLR 2022
trix factorization only generates two 2-D matrices. Such huge difference makes the corresponding projection on the decomposed tensor cores is significantly different from the projection on low-rank used in [E8][E10] sparse matrix used in [E]5][E[]][E17].
[E1] Song Han et al. "Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding," ICLR'16 16
[E2] Mingbao Lin et al. "HRank: Filter Pruning using High-Rank Rank Feature Map." CVPR 20
[E3] Yehui Tang et al. "SCOP: Scientific Control for Reliable Neural Network Pruning," NeurlPS'
[E4] Yang Sui et al. "CHIP: CHannel Independence-based Pruning for Compact Neural Networks,"
NeurlPS'21
[E5] Huanrui Yang et al. "Learning low-rank deep neural networks via singular vector orthogonality regularization and sin-gulan value sparsification." CVPR Wokrshop" 20
[E6] Yuhui Xu et al, "Trp: Trained rank pruning for efficient deep neural networks," IJCAI'20
[E7] Nannan Li et al, "Heuristic rankselection with progressively searching tensor ring net- work.Complex Intelligent Systems," Complex and intelligent system, 2021.
[E8] Yuzhe Ma al, "A unified approximation framework for compressing and accelerating deep neural networks," ICTAI'19
"iripppossiio,,,,PPPR [E9] Xiyu Yu et al, "On compressing deep models by low-rank and sparse decomposition." CVPR'I 17
[Е10] Yerlan Idelbayev et al, "More General and Effective Model Compression via an Additive Combination of Compressions," BCML'21
[E11] Handbook of Robust Low-Rank and Sparse Matrix Decomposition. Applications in Image and Video Processing (CRC Publishers 2016)
[E12] Emile Richard et al, "Intersecting singularitics for multi-structured estimation," " ICML' 13
[E13] Emile Richard et al, "Estimation of Simultancously Sparse and Low Rank Matrices," ICML' 12
[E14] Xi Luo "Recovering Model Structures from Large Low Rank and Sparse Covariance Matrix Estimation," JSM' 11
[E15] Tianyuan Zhang et al, "A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers," ECCV'18
[E16] Yerlan Idelbayev et al, Seeaning-Compresnt Algorithms for Neural Net Pruning." CVPR'18
[E17] Haichuan Yang et al, "Automatic Neural Network Compression by Quarsit-Qqannizzation Joint Learning: A Constrained Optimization- based Approach," CVPR'20
[E18] Yerlan Idelbayev et al, "Low-Rank Compression of Neural Nets: Learning the Rank of Each
Layer," CVPR 2020
18

---

Under review as a conference paper at ICLR 2022
Table 15: Overall comparison
Method
Model
Top-1 Acc. (%)
Top-5 Acc. (%)
Params.
CIFAR-10
[L4]
90.2
66.67%
ResNet-20
N/A
Ours
92.5
70.40%
[PI]
93.08
85.0%
ResNet-56
N/A
Ours
93.26
85.1%
[P2]
91.29
16.00
ResNet-20
N/A
Ours
92.09
16.00
[A]]
93.34
60.99
VGG16
N/A
Ours
93.38
61.16x
[A3]
86.17
76.09%
VGGNet-7
N/A
Ours
86.44
76.22%
[Ma.
2019]
91.65
77.48%
VGG-16
N/A
Ours
93.39
77.60%
ImageNet
[L]] Ours
88.9
80.00%
VGG-16
N/A
89.03
93.94%
[1.3]
69.29 70.23
88.78
58.68%
ResNet-18
Ours
89.43
58.70%
[L1] Accelerating Very Deep Convolutional Networks for Classification and Detection (IEEE TPAMI 2016)
[1.2] GroupReduce: Block-Wiss Low-Rank Approximation for Neural Language Model Shrinking, NeurIPS2018
[L3] Automated Multi-Stage Compression of Neural Networks (ICCV Workshops 2019)
[L4] Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer (CVPR 2020)
[L5] Factorized Higher-Order CNNs with an Application to Spatio-Temporal Emotion Estimation
(CVPR 2020) Relevant pruning works:
[P1]
Algorithms for Neural Net Pruning (CVPR 2018)
[P2] Automatic Neural Network Compression by Sparsity-Quantizatin Joint Learning: Con-
strained Optimization-based Approach (CVPR 2020) Relevant additive combinations work:
[A1] More General and Effective Model Compression via Additive Combination of Compres- sions (ECMI. 2021)
[A2] Handbook of Robust Low-Rank and Sparse Matrix Decomposition. Applications in Image and Video Processing (CRC Publishers 2016)
[A3] Compressing by Learning in Low-Rank and Sparse Decomposition Form (IEEE Access
2019)
19

---

Under review as a conference paper at ICLR 2022
Table Layer-wise distribution T'-Raak and sparsity ratio for the compressed ResNet-56 models on CIFAR-I0 10 dataset.
Model with 94.29% Acc.
Model II with 93,91 % Acc.
Layer Name
Weight Shape
TT Ranks
Sparsity
TT Ranks
Sparsity
[Ibl.conv
(16, 16, 3, 3)
(1, 16. 16, 1)
0.8
(1, 16. 16, 1)
0.85
11bl.conv2
(16, 16, 3)
(1, 16, 16, 1)
0.8
(1, 16, 16. 1)
0.85
11b2.conv
(16, 16, 3,3)
(1, 16, 16, 1)
0.8
(1, 16, 16, 1)
0.85
11b2.conv2
(16, 3,3)
(1,16,
0.8
(1, 16,16,1 16,
0.85
11b3.convl
(16, 1,,3,3)
(1, (1,16,1,,)
0.8
(1, 10,10,0011
0.85
11b3.conv2
(16, 3,)3,3)
(1,
0.8
(1, 10,
0.85
11b4.conv
(16, 1,,3,3)
16,
0.8
(1, 10,
0.85
11b4.conv2
(16, 16,3,3)
(6,
0.8
(1, 10,
0.85
11b5.conv1
(16, 36,3,3)
16,
0.8
(1, 10,
0.85
11b5.conv2
(16, 16,3,3)
16,
0.8
10,
0.85
11b6.conv
(16, 16,3,3
(1,
0.8
(1, 10, 10,
0.85
11b6.conv2
16,3,33
16,
0.8
(1, 10,
0.85
11b7.convI
(16, 3,,3,3
16,
0.8
(1, 10,
0.85
11b7.conv2
16.3,3
(1, 16,
0.8
(1, 10,
0.85
11b8.convl
(16, 3,,33)
16,
0.8
(1, 10,
0.85
11b8.conv2
(16, 16,3,3)
(1,16,
0.8
(1, 10,
0.85
11b9.conv1
(16, 3,3
(1,
0.8
(1, 10,
0.85
11b9.conv2
(16, 1,,3,3)
16,
0.8
(1, 10,
0.85
12b1.convl
(32, 3,33
(1,32,
0.8
20,
0.85
12b1.conv2
(32, 32,33,3,33
(1,32,
0.8
(1,25,
0.85
12b2.conv1
(32, 32,
20,
0.8
16,
0.85
12b2.conv2
(32, 32, 3,3)
(1, 20, 1)
0.8
(1, 16,
0.85
12b3.convl
(32, 3,
(1, 20,
0.8
(1, 16,11,1))1)
0.85
12b3.conv2
(32, 3,
(1, 20,20,22 20, 1)
0.8
(1, 16,
0.85
12b4.convl
(32, 3,
(1, 20, 20, 1)
0.8
(1, 16,
0.85
12b4.conv2
(32, 3,
(1, 20, 20,
0.8
(1, 16,
0.85
12b5.convl
32,
(1, 20, 20, 1)
0.8
(1, 16,16,16,1)
0.85
12b5.conv2
32,
(1, 20,
0.8
(1, 16, 16,
0.85
12b6.conv
(32, 32,
(1, 20, 1)
0.8
(1, 10, 10,
0.85
12b6.conv2
(32, 32, 3,
(1, 20,
0.8
(1, 10,
0.85
12b7.convl
(32, 32, 32,
(1, 1)
0.8
10,
0.85
12b7.conv2
32,
16, 16,
0.8
(1, 10,
0.85
12b8.conv
(32, 3,33
(1, 16,
0.8
(1, 10,
0.85
12b8.conv2
(32,
(1, 16.
0.8
(1, 10,
0.85
12b9.conv1
(32, 3,3)
(1, 16. 16,
0.8
(1, 10,
0.85
12b9.conv2
32,
(1, 16, 16,
0.8
(1, 10,
0.85
13bl.conv
(6,,
(1,40,
0.8
(I, 10,
0.85
13b1.conv2
(6,, 6,,
(1,40, 30,
0.8
(1, 10,
0.85
13b2.conv
(64, 64,
(1,40,
0.8
(1, 10,
0.85
13b2.conv2
(64, 6,
(1,40,30,
0.8
10,
0.85
13b3.conv
64,
(1, (1,40,30, 30,
0.8
16,
0.85
13b3.conv2
(64,
(1,40, 30,
0.8
16,
0.85
13b4.convl
(64, 6,,33)
(1, (1,40, 30,
0.8
(1, 16,
0.85
13b4.conv2
64,
(1,40.30 30,
0.8
16,
0.85
13b5.convl
64,
(1, 20,
0.8
(1, 16,
0.85
13b5.conv2
(4,
(1,30, 20,
0.8
(1, 16,
0.85
13b6.conv1
(64,
(1,30, 20, 1)
0.8
(1, 16,
0.85
13b6.conv2
64,
(1,30, 20, 1)
0.8
(1,
0.85
13b7.convl
(64, 3,
(1,30, 20, 1)
0.8
20,
0.85
13b7.conv2
64,
(1,30, 20.
0.8
20,
0.85
13b8.convl
64,
(1,30, 20, 1)
0.8
20,
0.85
13b8.conv2
(64, 3,
(1,30, 20,
0.8
20,
0.85
13b9.co convl
(64, 64, 3,3)
(1,30, 20,
0.8
20, (1, 20,
0.85
13b9.conv2
(64, 3,
(1,30,20)
0.8
(1, 20, 20,
0.85
20

---

Under review as a conference paper at ICLR 2022
Table 10: Layer-wise distribution of TT-Rank and sparsity ratio for the compressed DenseNet-40
models on CIFAR-10 dataset.
Model with 94.97% Acc.
Model with 94.88% Acc.
Layer Name
Weight Shape
TT Ranks
Sparsity
TT Ranks
Sparsity
b1l0.conv
(12, 24, 3)
(1, 18, 16, I)
0,9
(1, 18, 16, 1)
0.9
bll1.conv
(12, 36, 3,3)
(1, 20, 18. 1)
0.9
(1, 20, 18, 1)
0.9
b112.conv
(12, 3,33)
(1,30, 18,
0.9
(1, 20, 18, 1)
0.9
b113.conv
(12, 3,3)
(1,30,24,
0.9
20,20,20 1)
0.9
bll4.conv
(12, 3,33
(0,30,25.
0,9
(1,20.20,
0.9
b115.conv1
(12, 3,33
(1, 15,
0.9
15,
0.9
b116.conv1
(1,,9),3,3
(1, 15,
0.9
15,
0.9
b117.conv1
108,
(1, 15, 15,
0.9
(1, 151 15.
0.9
b118.conv1
(12, 320,
(1, 15,
0.9
(1, 15,
0.9
b119.conv1
132,
(1, 15, 15,
0.9
(1, 15,
0.9
b1l10.conv
(12, 3,4,
(1, 25,
0.9
(1, 10, 10,
0.9
billl.conv
156,
(5, 1)
0.9
10,
0.9
b2I0.conv
168,
(1,30,
0.9
25,
0.9
b2ll.conv
180,
(1,30, 30,
0.9
21,
0.9
b212.conv1
192,
(1,30, 30,
0.9
(1, 20,
0.9
b213.conv
(12, 204,
(1,30, 30. 1)
0.9
(1,20, 1)
0.9
b214.conv1
2,6,
(1, 15, 15,
0.9
(1, 10,
0.9
b215.convl
(12, 228,
(1, 15,
0.9
(1, 10,
0.9
b216.conv1
(12, 3,
(1, 15,
0.9
(1, 10, 10,
0.9
b217.conv1
(12, 252,
(1, 15, 15, 1)
0.9
(1, 10,
0.9
b218.conv
(12, 264, 3,3)
(1, 15, 15,
0.9
11, 10,
0.9
convl
(12, 276,
(0, 1)
0.9
10,
0.9
b2110.conv
(12, 288, 3,3)
(1, 20,
0.9
10,
0.9
b21lv.convl
(12, 3,0,
(1, 20, 1)
0.9
(1, 10,
0.9
b310.conv
312,
(1, 30, 1)
0.9
(1,20,
0.9
b311.conv1
(12, 324,
(1,30, 1)
0.9
21,
0.9
b312.conv1
336,
(1,30,30, 1)
0.9
(1,
0.9
b313.conv
(12, (48, 3,3)
(1, 30,
0.9
15,
0.9
b314.conv1
(12, 360,
25,
0.9
15,
0.9
b315.conv
372,
(1, 25,
0.9
15, 15.
0.9
b316.conv1
(12, 3,
(1, (1,25,25.1)
0.9
(1, 15, 1)
0.9
b317.conv
396,
21,
0.9
15, 15,
0.9
b318.conv1
(12, 408,
(1, 25,
0.9
(1, 15,
0.9
b319.convi
(12, 420,
(1, 25,
0.9
(1, 15, 15.
0.9
b3110.comv
(12, 432,
(1, 25, 25,
0.9
(1, 20, 1)
0.9
b3l11.comv/
(12, 4,3,
(1, 25,
0.9
(1. 20,
0.9
21

---

Under review as a conference paper at ICLR 2022
Table 11: Laser-wise distribution of TT-Rank and sparsity ratio for the compressed ResNet-50 mod- els ImageNet dataset.
Model I with 76.65% Acc.
Model II with 75.29% Acc.
Layer Name
Weight Shape
TT Ranks
Sparsity
Ranks
Sparsity
11.0.conv2
(64, 64, 3, 3)
(1, 64, 64, 1)
0.8
(1, 64, 64, 1)
0,85
11.1.conv2
(64, 64, 3,
(1, 61,64 64, 1)
0.8
(1, 64, 64, 1)
0.85
11.2.conv2
(64, 3)
(1,64, 64,
0.8
(1, 64, 64, 1)
0.85
12.0.conv2
(128, 128, 3,
(1,50, 50, 1)
0.8
51,50, 1)
0.85
12.1.conv2
(128, 128. 3.3)
(1, 50, 1)
0.8
(1,50, 1)
0.85
12.2.conv2
(128, 128, 3,
50,
0.8
(1,50, 1)
0.85
12.3.conv2
(128, 128, 3)
50, 1)
0.8
(1,50, 1)
0.85
13.0.conv2
(256, 256, 3,
(1,64, 64,
0.8
(1,40, 1)
0.85
13.0.comv3
(1024, 256, 1.
(1,50, 50, 1)
0.8
(1,30, 1)
0.85
13.1.conv2
(256, 256, 3)
(1, 64,
0.8
(1,40,40, 1)
0.85
13.1.conv3
(1024, 256, 1, 1)
(1, 1)
0.8
(1, 30, 1)
0.85
13.2.conv2
(256, 256, 3)
(1, 64,
0.8
(1,40,40, 1)
0.85
13.2.conv3
(1024, 1,
(1,50, 1)
0.8
(1,30, 30. 1)
0.85
13.3.conv2
(256, 256, 3)
(1, 6,,44,60 64, 1)
0.8
(1,40, 40, 1)
0.85
13.3.conv3
(1024,
(1, 50, 1)
0.8
(1,30.30, 1)
0.85
13.4.conv2
(256, 256, 3)
(1, 64.4,6411 1)
0.8
(1,40,40, 1)
0.85
13.4.conv3
(1024, 256, 1)
(1, 1)
0.8
(1,30, 30. 1)
0.85
13.5.conv2
(256, 3,3)
(1, 64. 1)
0.8
(1,40, 40. 1)
0.85
13.5.comv3
(1024, 256, I)
(1, 1)
0.8
30. 1)
0.85
14.0.conv
(512, 1024, I)
(1, 50, 1)
0.8
(1, 30,
0.85
14.0.conv2
(512, 512, 3,
(1, 80, 80,
0.8
(1,30, 1)
0.85
34.0.conv3
(2048, 512, 1)
(1, 6,,44,64 64, 1)
0.8
(1, 30, 1)
0.85
14.1.conv1
(512, 2048, 1. I)
(1, 64, 1)
0.8
(1,30, 1)
0.85
14.1.conv2
(512, 512, 3)
80, 80, 1)
0.8
(1,30, 1)
0.85
14.1.c3
(2048, 512, 1,
(1, 64, 1)
0.8
(1,30,30, 1)
0.85
14.2.convl
(512, 2048, 1)
(1. 64. 64, 1)
0.8
(1,30, 30. 1)
0.85
14.2.conv2
(512, 3,
(1, 80, 80, 1)
0.8
(1,30, 30, 1)
0.85
14.2.conv3
(2048, 512, 1)
(1, 64, 64, 1)
0.8
10,
0.85
Table 12: Experimental results on CIFAR-10 dataset targeting lower target rank.
Top-1 Accuracy (%)
Params.
Rank Ratio
Sparsity
Baseline
Compressed
A
$ (%)
ResNet-20
91.25
93.15
+1.9
30.8
2.0x
80%
91.25
93.27
+2.02
37.1
2.3x
80%
91.25
93,00
+1.75
43.8
2.7x
80%
91.25
93.17
+1.92
50.8
3.4x
80%
91.25
93.14
+1.89
55.8
4.1x
80%
91.25
92.48
+1.23
61.7
5,4x
80%
91.25
92.74
+1.49
67.1
7.6x
80%
22

---

Under review as a conference paper at ICLR 2022
Table 13: Experimental results on CIFAR-10 dataset targeting higher sparsity.
Top-1 Accuracy (%)
Params.
Rank Ratio
Sparsity
Baseline
Compressed
A
4 (%)
ResNet-20
91.25
93.04
+1.79
35,9
3.4x
65%
91.25
92.99
+1.74
40.9
3.4x
70%
91.25
93.11
+1.86
45.8
3,4x
75%
91.25
93.17
+1.92
50.8
3.4x
80%
91.25
93.02
+1.77
55.7
3.4x
85%
91.25
92.72
+1.47
60.7
3.4x
90%
91.25
92.24
+0.99
65.6
3,4x
95%
Table 14: Experimental results on CIFAR-10 dataset with the same overall compression ratio.
Top-1 Accuracy (%)
Params.
Rank Ratio
Sparsity
Baseline
Compressed
(%)
ResNet-20
91.25
93.13
+1.88
55.6
10.3x
65%
91.25
93.04
+1.79
55.6
6.8x
70%
91.25
93.13
+1.88
55.7
5.1x
75%
91.25
93.14
+1.89
55.8
4.1x
80%
91,25
93.02
+1.77
55.7
3.4
85%
91.25
93.05
+1.80
55.9
2.9x
90%
91.25
92.63
+1.38
55.6
2.5x
95%
23


Reviews:
[{"review_id": "f2KU39e3BPE", "reviewer": "Reviewer_cT6Q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nAuthors argue and empirically demonstrate that an additive combination of compression is a good choice for the neural network compression. Then authors formulate an optimization problem and solve it using alternating directions of multipliers which has three steps: step over nn parameters, step over compression weights, and step over multipliers. Authors justify usefulness of this scheme using small empirical studies as well as by showcasing compression results. By itself, this would have been a good contribution if executed perfectly, however there are several shortcomings that prevent me from accepting the paper.\n\n1. Justifications for the claims. To justify the usefulness of the additive combinations, authors perform several empirical studies over weights of the pretrained NNs (ResNet-20, fig2) and show that L+S scheme is much better than any other compounding (S->L or L->S). While this is an interesting empirical study, I would not be so sure to generalize it over all possible networks and use it as a trump card. In other words, an empirical study on a single network should not be generalized for all networks by saing \"...L+S is the best choice...\". Additionally, authors do not report the rank+sarsity settings used for this experiment, which further questions validity of such studies.  \nHere is one practical argument againts L+S or any additive compression scheme: the overall compression ratio of any additive combination scheme is limited by the compression ratio of the worst term. For any L+S scheme, simply doing just L or S would get higher compression (but worse approximation error). Clearly, there is a compression-error interplay  between using additive scheme vs using any single scheme, which should be studied more formally to make any long standing claims.\n\n2. The formulation and the choice of hyperparamters. The fundamental practical issue for the proposed optimization problem is the fact that users need to give all ranks and cardinality constrains per layer, which involves on selecting over combinatorial number of different settings (rank per each layer $times$ cardinality per each layer)^(number of layers).\n\n3. Missing details on reported quantities. Authors report overall compression ratio, however, do not report how these values were computed. While compressed storage of tensor weights is straightforward to obtain, the storage of sparse matrices might recuire different amount of bits depending on how they are being saved to disk. Please report clearly how the measures are being computed.\n\n4. Authors omit mentioning/comparing to a vast literature of low-rank, pruning, and additive combinations literature. Some missing low-rank/tensor decomposition works:\n- [L1] Accelerating Very Deep Convolutional Networks for Classification and Detection (IEEE TPAMI 2016)\n- [L2]  GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking, NeurIPS2018\n- [L3] Automated Multi-Stage Compression of Neural Networks (ICCV Workshops 2019)\n- [L4] Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer (CVPR 2020)\n- [L5] Factorized Higher-Order {CNNs} with an Application to Spatio-Temporal Emotion Estimation (CVPR 2020)\n\nRelevant pruning works:\n- [P1] “Learning-Compression” Algorithms for Neural Net Pruning (CVPR 2018)\n- [P2] Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach (CVPR 2020)\n\nRelevant additive combinations work:\n- [A1] More General and Effective Model Compression via an Additive Combination of Compressions (ECML 2021)\n- [A2] Handbook of Robust Low-Rank and Sparse Matrix Decomposition. Applications in Image and Video Processing (CRC Publishers 2016)\n- [A3] Compressing by Learning in a Low-Rank and Sparse Decomposition Form (IEEE Access 2019)\n\nImportantly, many compression approaches use variation of ADMM (e.g., L4, P1, P2, above or the work of Ma et al. (2019)). More specifically, additive combinations for model compression have been studied in [A1], and generally, such a combination approach is very well studied in image processing field. See for example the reference [A2]. Authors should mention these works and factor paper's contribution into the existing literature in a more rigorous way.", "summary_of_the_paper": "The authors argue and propose to compress the neural networks using an additive combination of TT decomposed and sparse tensors/matrices. To make this happen, authors formulate an optimizaiton problem and solve it using the ADMM framework. Authors provide compression results that are interesting (improved accuracy for a given compression rate), however, the method has an important shortcoming that requires setting the compression parameters by hand for each layer (ranks, sparsities) that limits its practical applicability. Additionally, authors miss a large body of related work on: a) additive combinations of compressions b) low-rank and tensor decomposition methods.\n", "summary_of_the_review": "My rating of the paper is based on the following issues:\n\n1. Weakly justified claims\n2. Practical difficulties of the proposed formulation\n3. Missing details\n4. Literature review and positioning", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "E24wZOZyXAH", "reviewer": "Reviewer_YjQo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the ideas and overview presented seem solid.\n\nHowever, several omissions make me worry about how well this contribution is situated in the literature.\n\n1. The term Spark is well established in the sparsity literature and defined as something else entirely (see e.g. https://www.pnas.org/content/pnas/100/5/2197.full.pdf). I would heavily encourage the authors not to create more confusion about terms than there already is, especially when working in the same field, and rename this method to something else.\n\n2. Arguably the seminal contribution in the most recent neural network era pertaining to low-rankness of weights is https://arxiv.org/abs/1404.0736 . The fact that it is not mentioned makes me worry about how many other references are missing and potential comparisons. E.g. why is the method of Yu, Liu, Wang, Tao not transposed to the architectures evaluated here and compared in the table?\n\n3. The sparsity literature is vast and includes some extremely relevant contributions, all of which have been omitted. The following can almost be drop-in replacements for the alternating algorithm described in the paper:\n- http://proceedings.mlr.press/v28/richard13.pdf\n- https://arxiv.org/abs/1206.6474\n- https://arxiv.org/abs/1111.1133\nThis list is certainly not exhaustive and it will not do to simply cite these papers. There are more in the references they list and the ones listed are too similar not to be compared.\n\nContrary to the algorithm presented, the above contributions lend themselves to convergence analysis and guarantees due to their convexity.\n", "summary_of_the_paper": "This paper presents a survey of methods for enforcing low-rankness and sparsity in neural network weights and proposes SPARK, an alternating algorithm for creating low-rank and sparse weight tensors from a pre-trained network. Baseline method accuracies are retained or slightly improved while parameter count is reduced by a large factor.", "summary_of_the_review": "Possibly solid contribution, but high uncertainty on the originality of the contribution due to lack of references to the surrounding literature.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "B3daBPSoMip", "reviewer": "Reviewer_Kmv9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n- This is a well-written and well-motivated paper. The algorithm is introduced and derived systematically. I especially appreciated the thorough discussion and analysis in Section 3.\n- The experiments are convincing and impressive.\n\n### Weaknesses\n\nThese are relatively minor:\n\n- While the paper appears to do a reasonable job of covering related work, it would have been good to have pointers in Section 4.2 for how the projection steps are similar to or differ from prior low-rank/sparse compression methods.\n- There are only two compressed versions reported for each architecture/benchmark. I realize that running the optimization+training is computationally expensive. But it would've been good to have a more complete picture of the performance-compression curve. For example, for Resnet-20, there are only models with >= 70% compression --- would've been nice to know how performance degrades as you push for lower target rank/sparsity. Again, this is not a requirement for publication (because these experiments are expensive), but I'd recommend that --- if possible --- authors consider doing a more thorough exploration for Resnet-20 on CIFAR-10 (the lightest model+benchmark).\n- One of the things that's a little hard to glean is which is better at maintaining performance: pushing for lower rank or  higher sparsity. Right now the algorithm requires both to be provided as input, but one can imagine doing a 'meta'-optimization on top to set these parameters to achieve the highest performance for a target FLOPs budget. While that would be a separate algorithm and out of the scope of this paper, it would be helpful to give readers a bit more intuition on which approximation causes a higher performance loss (and if it varies from layer to layer).\n- Finally, it is interesting to observe that for Resnet-20 and -56, the cheaper models actually lead to higher accuracy. I assume this is because of a regularizing effect, but perhaps the compressed models also have better optimization behavior. It would be nice to also report training set performance to verify this.", "summary_of_the_paper": "The paper proposes a novel approach for 'model compression': reducing the size and computational cost of a neural network model by converting weight matrices to (a) be sparse and (b) low-rank.  While prior works have considered both sparsity (i.e., pruning) and low-rank-ness before, the proposed method utilizes both simultaneously: approximating the weight matrix as a sum of two matrices, one that is low-rank and one that is sparse. This leads to an improved performance-compression trade-off: indeed, in some cases, this approach seems to have a regularizing effect and actually improves the accuracy of the model.", "summary_of_the_review": "Overall this is a great paper, and I believe clearly meets the bar for ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "2s73fonxsmC", "reviewer": "Reviewer_WR6A", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the paper addresses an important problem and brings an interesting contribution, even though the ideas (low-rankness and sparsity) are quite straightforward. The key elements of the method are clearly discussed and supported (even too heavily in my opinion). The proposed technique is challenged against state of the art for three common architectures on CIFAR-10 and ImageNet.\n\nIn my opinion this is an interesting contribution and I support its publication. But I feel that the paper could have been more convincing if providing more extensive experiments, especially because the novelty is somewhat limited (the paper combines already existing techniques, even if this is done cleverly). I detail here some criticisms that could be addressed to improve the contribution.\n\n1. Authors state that they exhaustively analyze the \"design knobs and factors\" on how to optimally combine low-rankness and sparsity for DNN compression. And this appears to be one of the main contribution of the paper. While I agree with the three asked questions, I think that the answers could be better supported. For instance, the answer to Question 2 compare SVD and TT on a single experiment, and only in terms of approximation error. Intead, I was expecting more experimental settings, more various decompositions (e.g., Tucker), and other metrics such as the final impact on the network's performance (that might not been exactly correlated to the approximation error, as remarked by the Authors). Similarly, for answer to question #3, I agree that using the loss instead of the approximation error might be preferable. But this should be supported by various experiments. And the Authors do not compare the regularized and the constrained formulations.\n\n2. Experiments and comparison with state of the art. Even though the experiments and the baselines seem to assess the interest of the proposed technique, their could be more experiments and comparisons (in sections 5.2 and 5.3 or in Appendix), in order to show the impact of the proposed \"design knobs\" in the final result, or analyze the parameters, in particular the sparsity level and ranks. The computational load should be properly discussed and given for all the techniques, especially because it could be a drawback of the proposed technique.\n\n3. Minor comments\n- Section 1 & 2 contain several repetitions and redundancies\n- Figure 3: the way the dimensions are reshaped before SVD is not clearly explained\n- Answer to question #3: I would say that the constrained and penalized formulations are very similar, and even equivalent for a given Lagrangian parameter. What makes them really different is the way they are parameterized, and it is often easier to set a constrain parameter than a penalty weight. Could this be better discussed?\n- Optimization: why not considering proximal gradient instead of ADMM?\n", "summary_of_the_paper": "This paper introduces a new DNN compression technique. It consists in approximating the weights of a trained DNN by the sum of a low-rank and a sparse tensor. This is done by adding sparsity and low-rank constraints to the usual loss, the optimization problem being solved with ADMM. Experiments and comparisons with state of the art show the effectiveness of the technique.", "summary_of_the_review": "Summary Of The Review\n- Fairly good paper, but overall quality could be improved\n- Significant contribution, but might be slightly oversell\n- Convincing experimental validation but limited to only 3 experiments, with few additionnal results", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a neural network compression technique based on sparse and low-rank approximations. The paper received mixed reviews, with one accept, one reject, and two borderline accepts. Most reviewers have appreciated the effort conducted for the evaluation. Three reviewers are nevertheless worried about the limited novelty and two of them found the positioning in the literature unclear with many missing references. In particular, one reviewer makes a strong case against the accceptance of the paper.

The authors have made a significant effort to address the issues raised by the reviewers with a very long rebuttal. The area chair has read in details the responses, the points raised by the reviewers, and the paper itself. He/she tend to agree with the issues raised by the reviewers about the positioning of the paper in the literature and the missing baselines. The rebuttal was very helpful and addresses some of the concerns. There are still some remaining issues
 - the discussion about related work is relegated to an appendix. Yet, it is critical for positioning the paper and a discussion within the main paper would be more appropriate.
 - there is no assessment of the statistical significance of the results. Hyper-parameters are fixed to some ad-hoc values and it is unclear what the effect of different hyper-parameter choices is upon the method and other baselines.
 - for reproductibility purposes, providing code with the submission would be very helpful, especially given the empirical nature of the contribution.

Overall, this is a borderline case, which, unfortunately, would require additional work before being ready for acceptance.
Decision: Reject