Year: 2021

Abstract:
Low-rank decomposition and sparsification are two important techniques for deep neural network (DNN) compression. To date, these two popular yet distinct approaches are typically used in a separate way; while their efficient integration for better compression performance is little explored. In this paper we perform systematic co-exploration on the model low-rankness and sparsity towards compact neural networks. We first investigate and analyze several important design factors for the joint low-rank factorization and pruning, including operational sequence, low-rank format, and optimization objective. Based on the observations and outcomes from our analysis, we then propose RASPA, a unified DNN compression framework that can simultaneously capture model low-RAnkness and SPArsity in an efficient way. Empirical experiments demonstrate very promising performance of our proposed solution. Notably, on CIFAR-10 dataset, our approach can bring 1.25%, 1.02% and 0.16% accuracy increase over the baseline ResNet-20, ResNet-56 and DenseNet-40 models, respectively, and meanwhile the storage and computational costs are reduced by 70.4% and 71.1% (for ResNet-20), 37.5% and 39.3% (for ResNet-56) and 52.4% and 61.3% (for DenseNet-40), respectively. On ImageNet dataset, our approach can enable 0.52% accuracy increase over baseline model with 48.7% fewer parameters.

heading: 1 INTRODUCTION
text: Deep neural network (DNN) has served as the backbone machine learning technique in many modern intelligent systems. To facilitate the low-cost deployment of DNN on the resource-constrained platforms, model compression, as a powerful strategy that can efficiently reduce DNN model size, has been extensively studied in recent years. To date, numerous compression approaches have been proposed to provide compact DNNs for many practical applications (Han et al. (2015b); Gong et al. (2019); Liao et al. (2021)).
Among various types of model compression techniques, sparsification (a.k.a., pruning) and lowrank decomposition are two representative and popular solutions (Wang et al. (2021); Gao et al. (2021); Li et al. (2021a); Xu et al. (2020)). As revealed by their names, the low-rank and sparse methods aim to explore and leverage the potential low-rankness and sparsity of the uncompressed DNNs, respectively. In practice, supported by the widely existed overparameterization phenomenon (Denil et al. (2013); Han et al. (2015b)), such hypothesized structure-level redundancy usually exists and thus it can be safely removed while still preserving high model performance.
Co-exploring Low-rankness & Sparsity: Motivation. Considering the current prosperity of these two methods and their very distinct structural assumptions, an interesting and promising research topic is to explore the efficient integration of low-rank and sparse approaches towards a better model compression solution. As indicated and observed by Yu et al. (2017), DNN models tend to exhibit both low-rankness and sparsity simultaneously. For instance, the smooth components in the weight filters can be represented in the low-rank space, and meanwhile some other important information is sparsely scattered. Evidently, fully leveraging such co-existence of these structure-level patterns, if being performed properly, can potentially bring a powerful compression solution with attractive performance.
Existing Works. Unlike the current extensive research activities on individual low-rank and sparse methods, the investigations on integrating these two approaches, in an efficient and non-trivial way,
are little explored. To date, only very few efforts study the joint exploration of low-rankness and sparsity for DNN model compression. As the pioneering work along this direction, Yu et al. (2017) develops a singular value decomposition (SVD)-free approach to closely approximate original DNN model via combining sparse representation and low-rank matrix factorization. Built on the interesting connection between filter decomposition and filter pruning, Li et al. (2020) interprets the decomposition and pruning of convolutional filter in a unified perspective. Most recently, Li et al. (2021b) proposes a collaborative compression scheme to integrate SVD into model sparsification. By adopting a multi-step heuristic removal strategy, this post-training approach achieves promising task and compression performance.
Unanswered Questions. Although these prior works have demonstrated the huge potentials and attractive benefits of jointly decomposing and pruning, the systematic investigation on their efficient integration is still missing. To be specific, several fundamental and critical questions, whose answers will directly impact the integration scheme and overall compression performance, have not been comprehensively explored yet. For instance, because pruning and decomposition can be jointly performed in several different ways, such as in parallel (Yu et al. (2017)) or in sequence (Li et al. (2021b)), which collaborative strategy is the best fit for the target DNN compression task? Also, considering low-rankness can be exploited from different perspectives, which type of low-rank approach should be adopted? The matrix factorization used in Li et al. (2020; 2021b)? Or even highorder tensor decomposition? In addition, to achieve promising compression performance, what is the most suitable optimization objective that the integration scheme should aim? The approximation error focused in Yu et al. (2017)? The low-rankness/sparsity regularized loss in Yang et al. (2020)? Or some other new alternatives?
Technical Preview and Contributions. To answer these questions and develop efficient integrated model compression solution, in this paper we perform systematic co-exploration on the model lowrankness and sparsity towards compact neural networks. To be specific, we first review and analyze several important design factors for the joint low-rank decomposition and pruning. Based on the observations and outcomes from our analysis, we then propose RASPA, a unified DNN compression framework that can simultaneously capture model low-RAnkness and SPArsity in an efficient way. Overall, the contributions of this paper are summarized as follows:
• We systematically investigate and analyze the critical design knobs when co-exploring model low-rankness and sparsity, including operational sequence, low-rank format, and optimization objective. Based on our qualitative and quantitative analysis, we propose several recommended design options for efficient joint low-rank decomposition and pruning.
• We develop a unified framework that formulates the integration of low-rank decomposition and pruning to an optimization problem with low-tensor-rank and sparse constraints. We then derive a training-aware approach to solve this challenging non-convex high-order tensor-format problem, and thereby leading to efficient exploration of rich low-rankness and sparsity in the model.
• We empirically evaluate our proposed co-exploration solution for various DNN models on different datasets, and the experimental results demonstrate its very promising performance. Notably, on CIFAR-10 dataset, our solution can bring 1.25%, 1.02% and 0.16% accuracy increase over the baseline ResNet-20, ResNet-56 and DenseNet-40 models, respectively, and meanwhile the storage and computational costs are reduced by 70.4% and 71.1% (for ResNet-20), 37.5% and 39.3% (for ResNet-56) and 52.4% and 61.3% (for DenseNet40), respectively. On ImageNet dataset, our approach can enable 0.52% accuracy increase over baseline model with 48.7% fewer parameters.

heading: 2 RELATED WORK
text: Sparsification. Sparsification, also known as pruning, has been extensively studied for model compression (Han et al. (2015a); Wen et al. (2016); Gao et al. (2019); Guo et al. (2016); Rao et al. (2021)). In general sparsifying a DNN can be realized via two ways. The first one is to use a certain criterion, e.g., weight magnitude (Han et al. (2015a)), to directly remove some part of the model, and then perform fine-tuning to recover the accuracy. The second one is to add the sparsity-induced regularization during the training, such as `1 or group lasso term (Wen et al. (2016)), to enforce the
sparsity on the model. In addition, in order to achieve good balance between accuracy and complexity reduction, Gao et al. (2019); Guo et al. (2016); Rao et al. (2021) also proposes dynamic pruning. In such scenario, the DNN sparsification is essentially performed in an input-aware way – which part of model should be pruned is dynamically determined by each input data.
Low-Rank Decomposition. Low-rank decomposition is another popular DNN compression approach. Based on different interpretation of the neural network models, the low-rank method can be categorized to matrix decomposition and tensor decomposition. Matrix decomposition views the 4-D weight tensor as the folded matrix, and hence it flattens the 4-D objective to 2-D format and decomposes the reshaped matrix to the product of two small matrices (Tai et al. (2016); Li & Shi (2018); Xu et al. (2020)). On the other aspect, tensor decomposition directly factorizes the 4-D weight tensor to multiple small tensor cores without flattening operations. Such explicit high-order processing, by its nature, can better preserve the important spatial information and correlation existed in the weight tensors. To date several tensor decomposition techniques, such as tensor train, Tucker and tensor ring etc., have been used for DNN model compression (Kim et al. (2016); Novikov et al. (2015); Wang et al. (2018)).
Joint Pruning and Decomposition. As observed by Yu et al. (2017), a well-trained DNN tends to exhibit both sparsity and low-rankness simultaneously. Motivated by this observation, some prior efforts propose to co-explore these two complementary properties for model compression. As the pioneering work, Yu et al. (2017) decomposes the weight tensors of a pre-trained DNN model into independent low-rank and sparse parts and minimizes the reconstruction error. Different from this parallel scheme, Dubey et al. (2018); Li et al. (2021b) adopt a sequential compression strategy via performing matrix factorization on a pruned model. In addition, Li et al. (2020) proposes to use the sparse/low-rank regularization term, instead of reconstruction error, to enforce the desired structural patterns. Also, notice that all of the existing works focus on using either SVD-based or SVD-free matrix decomposition to exploit the low-rankness of DNN model.

heading: 3 CO-EXPLORING LOW-RANKNESS AND SPARSITY: ANALYSIS
text: As outlined in Section 2, the integration of low-rank decomposition and pruning can be specified by several important factors, including operational sequence, low-rankness format and the overall optimization objective. The existence of such large variety of different factors and their combinations, by its nature, calls for the systematic investigation on the best-suited co-exploration scheme for DNN compression. Such analysis framework, if being properly developed, can facilitate the optimal selection of various design factors already proposed in the existing literatures. More importantly, the outcome from this systematic study will further guide and provide the better integration choices that have not been discovered before.
Questions to be Answered. Next we analyze the critical design knobs and factors for efficient coexploration on model low-rankness and sparsity. To that end, three important questions need to be answered.
Question #1: What is the more suitable operational sequence when jointly low-rank decomposing and pruning DNN models?
Analysis. In general, the co-existence of model low-rankness and sparsity can be explored in different ways (see Figure 1). For instance, as adopted in Yu et al. (2017), a well-trained DNN can be closely approximated as the combination of a low-rank component and a sparse component. In other words, the two types of structure-level properties are imposed and leveraged in a spatially parallel way, and we denote this strategy as L+S, where L and S represent low-rank decomposition and sparsification, respectively. On the other hand, the joint use of factorization and pruning can also be performed in a temporally sequential way. As illustrated in Figure 1, the original model can be first imposed with low-rankness (or sparsity), and the size of the resulting partially compressed model can be further reduced by the second-stage pruning (or low-rank decomposition). Following the similar notation, such sequential operation can be denoted as S(L) and L(S). In practice L(S) is a preferable choice that has been adopted in the prior works (Dubey et al. (2018); Li et al. (2021b)).
Our Proposal. Among the above described three general operational schemes, we believe L+S is the more suitable choice when considering to integrate pruning and decomposition together for model compression. This is because unlike S(L) and L(S), which ultimately still produce the compressed
model in a single representation (sparse or low-rank) space, L+S enables the simultaneous representation of rich information of DNN models across different subspace, and thereby better preserving the structural characteristics and reducing the potential information loss. To verify our hypothesis, we examine the approximation error incurred by three integration schemes. As shown in Figure 2, with the same compression ratio for the weight tensor of one layer of a pre-trained ResNet-20 on CIFAR-10 dataset, L+S shows much lower approximation error than its counterparts, especially in high compression ratio region. This experimental phenomenon demonstrates that L+S scheme indeed can capture both the low-rank and sparse characteristics of DNN model in an efficient way.
Question #2: What is the best suitable lowrank decomposition approach used when coexploring low-rankness and sparsity?
Analysis. From the perspective of linear algebra, the low-rankness of a DNN model can be exploited using different ways. As illustrated in Figure 3, for an example convolutional layer, imposing the low-rank structure can be realized by performing simple matrix factorization or high-order tensor decomposition. Specifically, Yu et al. (2017) chooses SVD-free method to factorize DNN model and obtain the low-rank component, and Li et al. (2021b) proposes to use SVD-based decomposition to serve as the second-stage compression approach in its adopted L(S) scheme. Notice that though the weights of convolutional layer essentially form a 4-D tensor format, the existing works exploit the low-rankness via using matrix decomposition – the 4-D tensor needs to be first flatten to a 2-D matrix and it is then factorized to two small matrix components.
Our Proposal. We argue that the high-order tensor decomposition, the option that has not been explored in the integration scheme before, is the better choice than the low-order matrix decomposition adopted in the existing works. This is because as a reshaping-free technique that can directly factorize the tensor-format data to multiple tensor cores, tensor decomposition, such as Tensor Train (TT) and Tucker, can naturally capture and preserve the important spatial information and correlation of the original weight tensors in a more efficient way. Therefore, less information loss is expected after performing low-rank tensor-based DNN compression. To verify our hypothesis, we compare the feature maps of the compressed convolutional layer of ResNet-20 on CIFAR-10 dataset using dif-
ferent low-rank methods. As visualized in Figure 4, compared with the matrix decomposition-based approach with the same compression ratio, tensor decomposition can make the output feature map of the compressed layer much more similar to the feature map of the original uncompressed layer. In other words, low-rank tensor method can provide better preservation of the important feature information and thus it can bring potential higher model compression performance.
Question #3: What is the best suitable optimization objective that the integrated compression scheme should aim?
Analysis. To efficiently realize the joint exploration of model low-rankness and sparsity with promising compression performance, different optimization strategies have been proposed in the existing works. For instance, Yu et al. (2017); Ma et al. (2019) aim to minimize the difference between the original weight matrix/tensor and the approximated reconstruction. In addition, Ma et al. (2019) proposes to explicitly add the low-rank and sparse regularization terms to the overall objective function, which can guide the training-aware procedure to enforce the desired structural patterns.
Our Proposal. Different from the existing approximation error-centered or regularized loss-based solutions, we propose that the efficient co-exploration scheme should be interpreted as the optimization procedure with the low-rank and sparse constraints. Our rationale lies on two important observations of the inherent drawbacks of the prior efforts. First, the approximation strategy adopted in Yu et al. (2017); Ma et al. (2019) focuses on making the reconstructed model approach the original model as close as possible. However, since 1) the approximation error always exists; and 2) the original model is not the only choice to achieve the desired accuracy, such strategy inherently can only search the low-rank and sparse components in a limited exploration space, thereby affecting the overall compression performance. Second, though adding the regularization terms into loss function indeed facilitates the extraction of low-rank and sparse patterns, the effect of such simple regularizing method is still limited, especially considering the efforts of pushing for sparsity and for low-rankness may interfere with each other, thereby potentially causing unexpected conflicts. Instead, by explicitly imposing the low-rank and sparse constraints on the overall optimization problem, these two structural requirement can be simultaneously satisfied with the proper use of optimization technique (to be discussed in Section 4). As reported in Appendix A, our proposed constrained optimization strategy can successfully impose the desired low-rankness and sparsity onto the DNN models efficiently.
Summary of Our Analysis. ¶ Performing joint low-rank decomposition and pruning in a spatially parallel way (L+S) is the preferable operational sequence. · High-order tensor decomposition is
the most suitable choice for the low-rank approach used in the integrated compression scheme. ¸ Imposing low-rankness and sparsity as the direct hard constraints on the loss optimization should be adopted to better satisfy the desired structural requirement.

heading: 4 CO-EXPLORING LOW-RANKNESS AND SPARSITY: OUR METHOD
text: Based on the above three key takeaways obtained from Section 3, in this section we propose a unified framework to co-explore low-rankness and sparsity in an efficient way. To be specific, we first formulate the integration of low-rank tensor decomposition and pruning to a unified optimization problem, and then develop an efficient algorithm to solve this non-convex high-order tensor-format problem.

heading: 4.1 PROBLEM FORMULATION
text: Recall that the analysis in Section 3 brings three important observations/proposals: using L+S operational sequence, choosing tensor decomposition, and directly imposing hard constraints. Built on such three fundamental principles, we are now ready to formulate the integration of pruning and tensor decomposition to a unified optimization problem. To be specific, given an uncompressed DNN model with weight tensor W ∈ RO×I×K×K of each layer, our goal is to find another compact model with weight tensors L + S, which consists of low-rank component L ∈ RO×I×K×K and sparse component S ∈ RO×I×K×K for each layer, to minimize the following loss function:
min L,S
f(L,S),
s.t. rank(L) ≤ γ0, γ1, · · · , γd︸ ︷︷ ︸ Low-tensor-rank constraint , card(S) ≤ κ︸ ︷︷ ︸ Sparse constraint , (1)
where f(·) is the loss over the entire training dataset, and γ0, γ1, · · · , γd and κ are the desired tensor ranks and the number of non-zero entries for L and S, respectively. Notice that without loss of generality, we choose tensor train (TT) decomposition as the component high-order lowrank method in our framework. So here d is the number of decomposed tensor cores with TT decomposition.

heading: 4.2 OPTIMIZATION
text: Directly optimizing problem (1) is challenging because of the co-existence of the non-differentiable rank(·) and card(·) as well as its inherent high-order tensor format. To efficiently solve this problem, we propose to leverage alternating direction optimization method to split these two constraints. To be specific, after introducing two auxiliary variables L̂ and Ŝ that represent the desired low-TTrankness and sparsity in the optimization process, problem (1) can be then rewritten as:
min L,S,L̂∈P,Ŝ∈Q
f(L,S),
s.t. L = L̂,S = Ŝ, (2)
where P = {L|rank(L) ≤ γ1, · · · , γd} is the set of all tensors that satisfy the low-tensor-rank constraint, and Q = {S|card(S) ≤ κ} is the set of all tensors that satisfy the sparse constraint. Then, we further relax the hard constraints to the corresponding augmented Lagrangian form and now we only need to optimize the following new constraint-free min-max problem:
min L,S,L̂∈P,Ŝ∈Q max U ,V f(L,S) + λ 2 (‖L− L̂ + U‖2F + ‖S − Ŝ + V‖2F + ‖U‖2F + ‖V‖2F ), (3)
where U and V are the dual multipliers associated to L and S, respectively, and λ is the penalty parameters. To solve this minmax problem, we can split it into three separated parts, and independently optimize them in an iterative way.
Update L and S with SGD. The first independent optimization objective can be formulated as:
min L,S f(L,S) + λ 2 (‖L− L̂ + U‖2F + ‖S − Ŝ + V‖2F ). (4)
Since there are no hard constraints on the target variables L and S, standard DNN optimizer (e.g., stochastic gradient descent (SGD)) can be directly applied with learning rate α as:
L← L− α[∇Lf(L,S) + λ(L− L̂ + U)], (5) S ← S − α[∇Sf(L,S) + λ(S − Ŝ + V)]. (6)
Update L̂ with TT Decomposition. To update the introduced L̂, the optimization objective is:
min L̂∈P
λ 2 ‖L− L̂ + U‖2F . (7)
Because L̂ is strictly constrained to stay in the low-tensor-rank set P , the desired update can be performed using an analytical solution via TT-rank truncation, i.e.
L̂← truncP(L + U). (8)
To realize such truncating operation, we first define a temporary tensor T = L + U and reshape it as a new tensor T̃ ∈ R(K×K)×(O1×I1)×···×(Od×Id) with O = ∏d k=1Ok, I = ∏d k=1 Ik. Then T̃ can be decomposed to d+ 1 TT-cores as:
T̃ ((k1, k2), (oi, i1), · · · , (od, id))) = C0(k1, k2)C1(:, o1, i1, :) · · ·Cd(:, od, id, :), (9)
where C0 ∈ RK×K ,Cj ∈ RRj−1×Oj×Ij×Rj , j = 1, · · · , d. In this TT-format, the dimensions of TT-ranks in TT-cores are truncated to the desired target, i.e., C′j = Cj(1 : γj−1, :, :, 1 : γj). After that we use the truncated TT-cores to recover the original tensor via:
T̃ ′ ((k1, k2), (oi, i1), · · · , (od, id))) = C0(k1, k2)C′1(:, o1, i1, :) · · ·C ′ d(:, od, id, :). (10)
And finally T̃ ′ is reshaped to the original shape of L̂ to serve as the updated L̂.
Update Ŝ with Projection. For updating Ŝ, the third optimization objective is:
min Ŝ∈Q
λ 2 ‖S − Ŝ + V‖2F . (11)
Similar to the low-tensor-rank L̂, the sparse-constrained Ŝ can also be analytically updated as
Ŝ ← projQ(S + V), (12)
where proj(·) is the projection that removes the smallest values to ensure that the updated Ŝ can satisfy the sparse constraint.
Update Multipliers U ,V . Upon the update of L̂ and Ŝ, the dual multipliers U and V can be then directly updated as:
U ← U + L− L̂, V ← V + S − Ŝ. (13)
Notice that after the iterative update finishes, the low-rank component L is explicitly decomposed to TT-cores {C}dj=0, and the entire compressed model consisting of TT-cores and sparse part S is finally fine-tuned with standard SGD. The overall RASPA algorithm is summarized in Algorithm 1.

heading: 5 EXPERIMENTS

heading: 5.1 EXPERIMENTAL SETTING
text: Dataset and Baseline. We evaluate our proposed approach on two image classification datasets (CIFAR-10 and ImageNet). For experiments on CIFAR-10 dataset, three CNN models (ResNet-20, ResNet-56 and DenseNet-40) are compressed and tested. For experiments on ImageNet dataset, we evaluate our approach for ResNet-50 and compare its performance with state-of-the-art model compression methods.
Algorithm 1 The overall RASPA algorithm for co-exploring model low-rankness and sparsity Input: Pre-trained weight tensor W , target TTranks {γj}dj=0, sparse target κ, training epochs T . Output: TT-cores {C}dj=0, sparse component S.
1: Initialize L, L̂,S, Ŝ with W ; 2: Initialize U := 0,V := 0 3: for t = 1 to T do 4: Update U ,V using Eq. 13; 5: Update L and S using Eq. 5 and Eq. 6; 6: // Update L̂ using TT-truncation 7: L̂← truncP(L + U); 8: // Update Ŝ using projection 9: Ŝ ← projQ(S + V);
10: end for 11: Decompose L to TT-cores {C}dj=0; 12: Fine-tune model with {C}dj=0 and S.
Hyperparameter. All the experiments are conducted using SGD optimizer with batch size, momentum and weight decay as 128, 0.9 and 0.0005, respectively. The learning rates adopted in the optimization and fine-tuning process are set as 0.1 and 0.005, respectively. Within the total 180 epochs, the learning rate is divided by 5 at epoch 54, 108, 144 and 171 gradually. The entire training procedure is performed on RTX 3090 GPUs with Pytorch 1.8.1.
5.2 EVALUATION AND COMPARISON ON CIFAR-10 DATASET
Table 1 shows the evaluation results of the compressed ResNet-20, ResNet-56 and DenseNet40 models on CIFAR-10 dataset. For each model, we compare the performance of our proposed RASPA with several types of compression methods, including decompositiononly (L), pruning-only (S), first-pruning-then-
decomposition (L(S)), and layer-wise either-pruning-or-decomposition (S/L)).
ResNet-20. For ResNet-20 model, the proposed RASPA solution can bring 1.25% accuracy increase over baseline model with 70.4% and 71.1% model size and FLOPs reductions, respectively. With even more aggressive compression strategy aiming 85.3% smaller model size and 86.1% fewer FLOPs, our approach can still enable 0.88% higher accuracy than the original ResNet-20 model.
ResNet-56. For ResNet-56 model, our approach can bring 1.02% accuracy increase over baseline model with 37.5% and 39.3% model size and FLOPs reductions, respectively. When we perform more aggressive compression with 65.6% and 66.0% fewer parameters and computations, our compressed ResNet-56 can still enjoy 0.64% higher accuracy than the original uncompressed model, thereby exhibiting very superior performance than other state-of-the-art DNN compression methods.
DenseNet-40. For DenseNet-40 model, our proposed sparsity/low-rankness co-exploration can bring 0.16% accuracy increase over the baseline model with 52.4% and 61.3% model size and FLOPs reductions, respectively. Moreover, with further higher compression effect, our RASPA approach can still enable 0.07% higher accuracy than the original uncompressed model with 65.3% and 74.5% fewer parameters and computations, respectively; while the existing approaches suffer accuracy loss with even lower compression ratio.

heading: 5.3 EVALUATION AND COMPARISON ON IMAGENET DATASET
text: Table 2 summarizes the compression performance of our approach and other existing works for ResNet-50 on ImageNet dataset. It is seen that our RASPA solution can bring 0.52% accuracy increase over baseline model with 48.7% fewer parameters. When targeting for generating more
compact model, our approach can still achieve high performance – it only has 0.25% accuracy drop with 58.6% model size reduction, which means it shows better test accuracy than its counterparts with even higher compression ratio.

heading: 5.4 DISCUSSION & ANALYSIS
text: To obtain deep understanding of our proposed approach, we also perform some empirical analysis and ablation studies on the co-exploration procedure. The details are referred to Appendix A.

heading: 6 CONCLUSION
text: In this paper we propose to systematically co-explore DNN low-rankness and sparsity for efficient model compression. By performing comprehensive analysis on critical design factors, we propose
RASPA, a unified compression framework that can capture model low-rankness and sparsity simultaneously and efficiently. Evaluation results show that our proposed approach can bring significant model size and computational cost reductions while still preserving high model accuracy.

heading: A DETAILS OF EMPIRICAL ANALYSIS IN SECTION 5.4
text: In this section we present the details of our empirical analysis and discussion in Section 5.4. Notice that the example model investigated and evaluated here is ResNet-20 on CIFAR-10 dataset.
Simultaneously Obtaining Low-rankness and Sparsity. Figure 5 shows the loss curves during the model optimization procedure. Notice that here besides overall training loss, the individual lowrank and sparse loss component, which directly reflects the progress of enforcing low-rankness and sparsity, respectively, is also explicitly reported in this figure. It is seen that our proposed approach indeed successfully imposes the desired low-tensor-rankness and sparsity with hard constraints onto the model, and thereby ensuring that the compressed model can fully exhibit both low-rank and sparse characteristics after the optimization.
Effect of Optimization Procedure. We also study the benefits of using our proposed optimization procedure described in Algorithm 1 to solve problem (2). Here we compare our approach with a direct method that performs TT decomposition and pruning on the uncompressed model straightforwardly with the same TT-ranks and sparsity settings. In addition, the same fine-tuning process that RASPA adopts is also applied in this direct method. Figure 6 shows the comparison results with respect to different compression ratios. It is seen that our proposed optimization procedure brings significant accuracy increase.
Visualization. Figure 7 visualizes the weight tensor of one convolutional layer in a pre-trained ResNet-20 model before and after performing our proposed compression approach. Here the visualization of the low-rank and sparse components of the compressed layer is also illustrated in this figure. It is seen that most of the weight information is preserved in the low-rank component, and meanwhile the sparse component contains some spatial pattern as well.
It is seen that our proposed optimization procedure in RASPA brings significantly accuracy increase.

heading: B DETAILS OF ANALYSIS IN SECTION 3
text: B.1 MORE RESULTS FOR ANALYSIS IN QUESTION #1
Table 3,4 and 5 show the layer-wise approximation errors incurred by three operational sequences (L+S, S(L) and L(S)). Here the baseline models include well-trained ResNet-20, ResNet-56 on CIFAR-10 dataset, and ResNet-50 on ImageNet dataset, and the compression ratio is set as 3.0 for all the layers. It is seen that L+S scheme always brings the smallest approximation errors.
B.2 MORE RESULTS FOR ANALYSIS IN QUESTION #2
Table 6 shows the difference (in term of mean square error (MSE)) between the output feature maps of original layer and the compressed one in ResNet-20 on CIFAR-10 dataset. Here SVD and tensor train (TT) are adopted for matrix decomposition and tensor decomposition, respectively. It is seen that, with the same or even higher compression ratio, high-order tensor decomposition always brings smaller approximation error than the matrix decomposition..
Table 7 shows the difference between the output feature maps of one original layer and the compressed version in ResNet-20 on CIFAR-10 dataset and the difference of the final accuracy. Here SVD, Tucker and TT are adopted for low-rank decompositions. It is seen that high-order tensor decomposition always brings smaller approximation error than the matrix decomposition and TT always has the least impact on the final accuracy.

heading: C DISTRIBUTION OF TT-RANK AND SPARSITY RATIO
text: Table. 8, 9, 10 and 11 list the layer-wise TT-rank and sparsity ratio of four compressed model. Here Model I and Model II in each table correspond to two reported models of each network with different Top-1 accuracy in Table 1 and 2.

heading: D OTHER EXPERIMENTAL RESULTS
text: Please check Table 12, 13, 14 and 15 for other experimental results including different setting of compressed ResNet-20 and comparison with more related works from [L1] to [A3].

heading: E RELATED WORK
text: Pruning. Pruning is a very popular compression strategy that explores the sparsity of DNN models. Existing pruning methods can be performed with different granularity, e.g., weight pruning [E1] and channel pruning [E2][E3], or with different pruning criteria, such as magnitude based [E1], rank based [E2], and nuclear norm based [E4].
Low-rank Decomposition. Low-rank compression methods aim to explore the intrinsic linear correlations of weights. In general, low-rank decomposition can be performed via exploring matrix decomposition [E5][E6] and tensor decomposition [E7]. For compressing convolutional neural networks, tensor decomposition is typically more suitable choice since high-order tensor decomposition can properly explore multidimensional correlation in the weight tensor; while the existing matrix decomposition-based works [E5][E6] suffer the loss of important spatial information incurred by inevitable tensor flatten operation.
Co-exploring Sparsity/Low-rankness for Model Compression. Recently, several papers have been proposed to co-explore sparsity and low-rankness for efficient DNN model compression. In particular, additive compression strategy (L+S) have been explored in some prior works [E8][E9][E10]. However, the low-rank component of all of these existing works are based on matrix decomposition. As analyzed before, high-order tensor decomposition is a more suitable low-rank methods for CNN compression; while this important technique has not been explored by the existing additive compression efforts yet. Also, [E8][E9] aims to closely approximate original models. As we analyze in Question #3, this strategy is not the optimal solution for DNN model compression, since our task goal is to generate a compressed model instead of close approximation to the original model. Different from these existing works, our approach is built on tensor decomposition with constrained formulation as the optimization objective, thereby demonstrating better performance than [E8][E9][E10].
Co-exploring Sparsity/Low-rankness for Matrix Approximation/Estimation. Co-exploring sparsity/low-rankness for efficient matrix approximation/estimation has also been explored [E11E14]. However, the feasibility and efficiency of those general methods for DNN model compression is not verified and not clear yet. To be specific, our approach aims to perform model compression on deep neural networks with high accuracy; while those general methods aim to approximate/estimate a matrix with low approximation error. Consider modern DNN models consist of many large-size layers and they need maintain high accuracy on large-scale dataset, our approach is focusing a very different problem that these general methods do not work on. Though it is possible that those general techniques may be applied to DNN model compression after certain unknown modification, the feasibility is not really verified yet, especially on large-scale dataset such as ImageNet. Also, the corresponding computational costs and accuracy performance are unknown.
ADMM-based Model Compression. In recent years ADMM has been used in several model compression papers, including single pruning [E15][E16], single low-rank compression [E18], joint quantization and pruning [E17] and additive compressing [E8][E10]. Our approach focuses on additive compression with distinct difference from [E8][E10] and other ADMM-based works. This is because low-rank approach used in [E8][E10] is based on matrix decomposition, which is not the best solution for compression CNNs; while we adopt high-order tensor decomposition in our proposed additive compression. From the perspective of ADMM process, the projection towards tensor decomposition is much more complicated than the one for matrix factorization. This is because 1) it is involved with advanced high-order tensor operation instead of straightforward 2-D matrix computation; and 2) tensor decomposition (e.g., TT) outputs multiple 4-D tensor cores; while ma-
trix factorization only generates two 2-D matrices. Such huge difference makes the corresponding projection on the decomposed tensor cores is significantly different from the projection on low-rank used in [E8][E10] or sparse matrix used in [E15][E16][E17].
[E1] Song Han et al. ”Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding,” ICLR’16
[E2] Mingbao Lin et al. ”HRank: Filter Pruning using High-Rank Feature Map,” CVPR’20
[E3] Yehui Tang et al. ”SCOP: Scientific Control for Reliable Neural Network Pruning,” NeurIPS’20
[E4] Yang Sui et al. ”CHIP: CHannel Independence-based Pruning for Compact Neural Networks,” NeurIPS’21
[E5] Huanrui Yang et al. ”Learning low-rank deep neural networks via singular vector orthogonality regularization and sin-gular value sparsification,” CVPR Wokrshop’20
[E6] Yuhui Xu et al, ”Trp: Trained rank pruning for efficient deep neural networks,” IJCAI’20
[E7] Nannan Li et al, ”Heuristic rankselection with progressively searching tensor ring network.Complex Intelligent Systems,” Complex and intelligent system, 2021.
[E8] Yuzhe Ma et al, ”A unified approximation framework for compressing and accelerating deep neural networks,” ICTAI’19
[E9] Xiyu Yu et al, ”On compressing deep models by low-rank and sparse decomposition,” CVPR’17
[E10] Yerlan Idelbayev et al, ”More General and Effective Model Compression via an Additive Combination of Compressions,” ECML’21
[E11] Handbook of Robust Low-Rank and Sparse Matrix Decomposition. Applications in Image and Video Processing (CRC Publishers 2016)
[E12] Emile Richard et al, ”Intersecting singularities for multi-structured estimation,” ICML’13
[E13] Emile Richard et al, ”Estimation of Simultaneously Sparse and Low Rank Matrices,” ICML’12
[E14] Xi Luo ”Recovering Model Structures from Large Low Rank and Sparse Covariance Matrix Estimation,” JSM’11
[E15] Tianyuan Zhang et al, ”A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers,” ECCV’18
[E16] Yerlan Idelbayev et al, “Learning-Compression” Algorithms for Neural Net Pruning,” CVPR’18
[E17] Haichuan Yang et al, ”Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach,” CVPR’20
[E18] Yerlan Idelbayev et al, ”Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer,” CVPR 2020
[L1] Accelerating Very Deep Convolutional Networks for Classification and Detection (IEEE TPAMI 2016)
[L2] GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking, NeurIPS2018
[L3] Automated Multi-Stage Compression of Neural Networks (ICCV Workshops 2019)
[L4] Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer (CVPR 2020)
[L5] Factorized Higher-Order CNNs with an Application to Spatio-Temporal Emotion Estimation (CVPR 2020) Relevant pruning works:
[P1] “Learning-Compression” Algorithms for Neural Net Pruning (CVPR 2018)
[P2] Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach (CVPR 2020) Relevant additive combinations work:
[A1] More General and Effective Model Compression via an Additive Combination of Compressions (ECML 2021)
[A2] Handbook of Robust Low-Rank and Sparse Matrix Decomposition. Applications in Image and Video Processing (CRC Publishers 2016)
[A3] Compressing by Learning in a Low-Rank and Sparse Decomposition Form (IEEE Access 2019)

heading: 91.25 92.48 +1.23 61.7 5.4× 80%

heading: 91.25 93.14 +1.89 55.8 4.1× 80%

heading: 91.25 93.17 +1.92 50.8 3.4× 80%

heading: 91.25 93.00 +1.75 43.8 2.7× 80%

heading: 91.25 93.27 +2.02 37.1 2.3× 80%

heading: 91.25 93.15 +1.9 30.8 2.0× 80%

heading: 91.25 93.05 +1.80 55.9 2.9× 90%

heading: 91.25 93.02 +1.77 55.7 3.4 × 85%

heading: 91.25 93.14 +1.89 55.8 4.1× 80%

heading: 91.25 93.13 +1.88 55.7 5.1× 75%

heading: 91.25 93.04 +1.79 55.6 6.8× 70%

heading: 91.25 93.13 +1.88 55.6 10.3× 65%

heading: 91.25 92.24 +0.99 65.6 3.4× 95%

heading: 91.25 92.72 +1.47 60.7 3.4× 90%

heading: 91.25 93.02 +1.77 55.7 3.4× 85%

heading: 91.25 93.17 +1.92 50.8 3.4× 80%

heading: 91.25 93.11 +1.86 45.8 3.4× 75%

heading: 91.25 92.99 +1.74 40.9 3.4× 70%

heading: 91.25 93.04 +1.79 35.9 3.4× 65%

Review summaries:
The authors argue and propose to compress the neural networks using an additive combination of TT decomposed and sparse tensors/matrices. To make this happen, authors formulate an optimizaiton problem and solve it using the ADMM framework. Authors provide compression results that are interesting (improved accuracy for a given compression rate), however, the method has an important shortcoming that requires setting the compression parameters by hand for each layer (ranks, sparsities) that limits its practical applicability. Additionally, authors miss a large body of related work on: a) additive combinations of compressions b) low-rank and tensor decomposition methods. 

The paper proposes a novel approach for 'model compression': reducing the size and computational cost of a neural network model by converting weight matrices to (a) be sparse and (b) low-rank.  While prior works have considered both sparsity (i.e., pruning) and low-rank-ness before, the proposed method utilizes both simultaneously: approximating the weight matrix as a sum of two matrices, one that is low-rank and one that is sparse. This leads to an improved performance-compression trade-off: indeed, in some cases, this approach seems to have a regularizing effect and actually improves the accuracy of the model.

This paper introduces a new DNN compression technique. It consists in approximating the weights of a trained DNN by the sum of a low-rank and a sparse tensor. This is done by adding sparsity and low-rank constraints to the usual loss, the optimization problem being solved with ADMM. Experiments and comparisons with state of the art show the effectiveness of the technique.