SPARK: co-exploring model SPArsity and low-RanKness for compact neural networks

1 INTRODUCTION . Deep neural network ( DNN ) has served as the backbone machine learning technique in many modern intelligent systems . To facilitate the low-cost deployment of DNN on the resource-constrained platforms , model compression , as a powerful strategy that can efficiently reduce DNN model size , has been extensively studied in recent years . To date , numerous compression approaches have been proposed to provide compact DNNs for many practical applications ( Han et al . ( 2015b ) ; Gong et al . ( 2019 ) ; Liao et al . ( 2021 ) ) . Among various types of model compression techniques , sparsification ( a.k.a. , pruning ) and lowrank decomposition are two representative and popular solutions ( Wang et al . ( 2021 ) ; Gao et al . ( 2021 ) ; Li et al . ( 2021a ) ; Xu et al . ( 2020 ) ) . As revealed by their names , the low-rank and sparse methods aim to explore and leverage the potential low-rankness and sparsity of the uncompressed DNNs , respectively . In practice , supported by the widely existed overparameterization phenomenon ( Denil et al . ( 2013 ) ; Han et al . ( 2015b ) ) , such hypothesized structure-level redundancy usually exists and thus it can be safely removed while still preserving high model performance . Co-exploring Low-rankness & Sparsity : Motivation . Considering the current prosperity of these two methods and their very distinct structural assumptions , an interesting and promising research topic is to explore the efficient integration of low-rank and sparse approaches towards a better model compression solution . As indicated and observed by Yu et al . ( 2017 ) , DNN models tend to exhibit both low-rankness and sparsity simultaneously . For instance , the smooth components in the weight filters can be represented in the low-rank space , and meanwhile some other important information is sparsely scattered . Evidently , fully leveraging such co-existence of these structure-level patterns , if being performed properly , can potentially bring a powerful compression solution with attractive performance . Existing Works . Unlike the current extensive research activities on individual low-rank and sparse methods , the investigations on integrating these two approaches , in an efficient and non-trivial way , are little explored . To date , only very few efforts study the joint exploration of low-rankness and sparsity for DNN model compression . As the pioneering work along this direction , Yu et al . ( 2017 ) develops a singular value decomposition ( SVD ) -free approach to closely approximate original DNN model via combining sparse representation and low-rank matrix factorization . Built on the interesting connection between filter decomposition and filter pruning , Li et al . ( 2020 ) interprets the decomposition and pruning of convolutional filter in a unified perspective . Most recently , Li et al . ( 2021b ) proposes a collaborative compression scheme to integrate SVD into model sparsification . By adopting a multi-step heuristic removal strategy , this post-training approach achieves promising task and compression performance . Unanswered Questions . Although these prior works have demonstrated the huge potentials and attractive benefits of jointly decomposing and pruning , the systematic investigation on their efficient integration is still missing . To be specific , several fundamental and critical questions , whose answers will directly impact the integration scheme and overall compression performance , have not been comprehensively explored yet . For instance , because pruning and decomposition can be jointly performed in several different ways , such as in parallel ( Yu et al . ( 2017 ) ) or in sequence ( Li et al . ( 2021b ) ) , which collaborative strategy is the best fit for the target DNN compression task ? Also , considering low-rankness can be exploited from different perspectives , which type of low-rank approach should be adopted ? The matrix factorization used in Li et al . ( 2020 ; 2021b ) ? Or even highorder tensor decomposition ? In addition , to achieve promising compression performance , what is the most suitable optimization objective that the integration scheme should aim ? The approximation error focused in Yu et al . ( 2017 ) ? The low-rankness/sparsity regularized loss in Yang et al . ( 2020 ) ? Or some other new alternatives ? Technical Preview and Contributions . To answer these questions and develop efficient integrated model compression solution , in this paper we perform systematic co-exploration on the model lowrankness and sparsity towards compact neural networks . To be specific , we first review and analyze several important design factors for the joint low-rank decomposition and pruning . Based on the observations and outcomes from our analysis , we then propose RASPA , a unified DNN compression framework that can simultaneously capture model low-RAnkness and SPArsity in an efficient way . Overall , the contributions of this paper are summarized as follows : • We systematically investigate and analyze the critical design knobs when co-exploring model low-rankness and sparsity , including operational sequence , low-rank format , and optimization objective . Based on our qualitative and quantitative analysis , we propose several recommended design options for efficient joint low-rank decomposition and pruning . • We develop a unified framework that formulates the integration of low-rank decomposition and pruning to an optimization problem with low-tensor-rank and sparse constraints . We then derive a training-aware approach to solve this challenging non-convex high-order tensor-format problem , and thereby leading to efficient exploration of rich low-rankness and sparsity in the model . • We empirically evaluate our proposed co-exploration solution for various DNN models on different datasets , and the experimental results demonstrate its very promising performance . Notably , on CIFAR-10 dataset , our solution can bring 1.25 % , 1.02 % and 0.16 % accuracy increase over the baseline ResNet-20 , ResNet-56 and DenseNet-40 models , respectively , and meanwhile the storage and computational costs are reduced by 70.4 % and 71.1 % ( for ResNet-20 ) , 37.5 % and 39.3 % ( for ResNet-56 ) and 52.4 % and 61.3 % ( for DenseNet40 ) , respectively . On ImageNet dataset , our approach can enable 0.52 % accuracy increase over baseline model with 48.7 % fewer parameters . 2 RELATED WORK . Sparsification . Sparsification , also known as pruning , has been extensively studied for model compression ( Han et al . ( 2015a ) ; Wen et al . ( 2016 ) ; Gao et al . ( 2019 ) ; Guo et al . ( 2016 ) ; Rao et al . ( 2021 ) ) . In general sparsifying a DNN can be realized via two ways . The first one is to use a certain criterion , e.g. , weight magnitude ( Han et al . ( 2015a ) ) , to directly remove some part of the model , and then perform fine-tuning to recover the accuracy . The second one is to add the sparsity-induced regularization during the training , such as ` 1 or group lasso term ( Wen et al . ( 2016 ) ) , to enforce the sparsity on the model . In addition , in order to achieve good balance between accuracy and complexity reduction , Gao et al . ( 2019 ) ; Guo et al . ( 2016 ) ; Rao et al . ( 2021 ) also proposes dynamic pruning . In such scenario , the DNN sparsification is essentially performed in an input-aware way – which part of model should be pruned is dynamically determined by each input data . Low-Rank Decomposition . Low-rank decomposition is another popular DNN compression approach . Based on different interpretation of the neural network models , the low-rank method can be categorized to matrix decomposition and tensor decomposition . Matrix decomposition views the 4-D weight tensor as the folded matrix , and hence it flattens the 4-D objective to 2-D format and decomposes the reshaped matrix to the product of two small matrices ( Tai et al . ( 2016 ) ; Li & Shi ( 2018 ) ; Xu et al . ( 2020 ) ) . On the other aspect , tensor decomposition directly factorizes the 4-D weight tensor to multiple small tensor cores without flattening operations . Such explicit high-order processing , by its nature , can better preserve the important spatial information and correlation existed in the weight tensors . To date several tensor decomposition techniques , such as tensor train , Tucker and tensor ring etc. , have been used for DNN model compression ( Kim et al . ( 2016 ) ; Novikov et al . ( 2015 ) ; Wang et al . ( 2018 ) ) . Joint Pruning and Decomposition . As observed by Yu et al . ( 2017 ) , a well-trained DNN tends to exhibit both sparsity and low-rankness simultaneously . Motivated by this observation , some prior efforts propose to co-explore these two complementary properties for model compression . As the pioneering work , Yu et al . ( 2017 ) decomposes the weight tensors of a pre-trained DNN model into independent low-rank and sparse parts and minimizes the reconstruction error . Different from this parallel scheme , Dubey et al . ( 2018 ) ; Li et al . ( 2021b ) adopt a sequential compression strategy via performing matrix factorization on a pruned model . In addition , Li et al . ( 2020 ) proposes to use the sparse/low-rank regularization term , instead of reconstruction error , to enforce the desired structural patterns . Also , notice that all of the existing works focus on using either SVD-based or SVD-free matrix decomposition to exploit the low-rankness of DNN model . 3 CO-EXPLORING LOW-RANKNESS AND SPARSITY : ANALYSIS . As outlined in Section 2 , the integration of low-rank decomposition and pruning can be specified by several important factors , including operational sequence , low-rankness format and the overall optimization objective . The existence of such large variety of different factors and their combinations , by its nature , calls for the systematic investigation on the best-suited co-exploration scheme for DNN compression . Such analysis framework , if being properly developed , can facilitate the optimal selection of various design factors already proposed in the existing literatures . More importantly , the outcome from this systematic study will further guide and provide the better integration choices that have not been discovered before . Questions to be Answered . Next we analyze the critical design knobs and factors for efficient coexploration on model low-rankness and sparsity . To that end , three important questions need to be answered . Question # 1 : What is the more suitable operational sequence when jointly low-rank decomposing and pruning DNN models ? Analysis . In general , the co-existence of model low-rankness and sparsity can be explored in different ways ( see Figure 1 ) . For instance , as adopted in Yu et al . ( 2017 ) , a well-trained DNN can be closely approximated as the combination of a low-rank component and a sparse component . In other words , the two types of structure-level properties are imposed and leveraged in a spatially parallel way , and we denote this strategy as L+S , where L and S represent low-rank decomposition and sparsification , respectively . On the other hand , the joint use of factorization and pruning can also be performed in a temporally sequential way . As illustrated in Figure 1 , the original model can be first imposed with low-rankness ( or sparsity ) , and the size of the resulting partially compressed model can be further reduced by the second-stage pruning ( or low-rank decomposition ) . Following the similar notation , such sequential operation can be denoted as S ( L ) and L ( S ) . In practice L ( S ) is a preferable choice that has been adopted in the prior works ( Dubey et al . ( 2018 ) ; Li et al . ( 2021b ) ) . Our Proposal . Among the above described three general operational schemes , we believe L+S is the more suitable choice when considering to integrate pruning and decomposition together for model compression . This is because unlike S ( L ) and L ( S ) , which ultimately still produce the compressed model in a single representation ( sparse or low-rank ) space , L+S enables the simultaneous representation of rich information of DNN models across different subspace , and thereby better preserving the structural characteristics and reducing the potential information loss . To verify our hypothesis , we examine the approximation error incurred by three integration schemes . As shown in Figure 2 , with the same compression ratio for the weight tensor of one layer of a pre-trained ResNet-20 on CIFAR-10 dataset , L+S shows much lower approximation error than its counterparts , especially in high compression ratio region . This experimental phenomenon demonstrates that L+S scheme indeed can capture both the low-rank and sparse characteristics of DNN model in an efficient way . Question # 2 : What is the best suitable lowrank decomposition approach used when coexploring low-rankness and sparsity ? Analysis . From the perspective of linear algebra , the low-rankness of a DNN model can be exploited using different ways . As illustrated in Figure 3 , for an example convolutional layer , imposing the low-rank structure can be realized by performing simple matrix factorization or high-order tensor decomposition . Specifically , Yu et al . ( 2017 ) chooses SVD-free method to factorize DNN model and obtain the low-rank component , and Li et al . ( 2021b ) proposes to use SVD-based decomposition to serve as the second-stage compression approach in its adopted L ( S ) scheme . Notice that though the weights of convolutional layer essentially form a 4-D tensor format , the existing works exploit the low-rankness via using matrix decomposition – the 4-D tensor needs to be first flatten to a 2-D matrix and it is then factorized to two small matrix components . Our Proposal . We argue that the high-order tensor decomposition , the option that has not been explored in the integration scheme before , is the better choice than the low-order matrix decomposition adopted in the existing works . This is because as a reshaping-free technique that can directly factorize the tensor-format data to multiple tensor cores , tensor decomposition , such as Tensor Train ( TT ) and Tucker , can naturally capture and preserve the important spatial information and correlation of the original weight tensors in a more efficient way . Therefore , less information loss is expected after performing low-rank tensor-based DNN compression . To verify our hypothesis , we compare the feature maps of the compressed convolutional layer of ResNet-20 on CIFAR-10 dataset using dif- ferent low-rank methods . As visualized in Figure 4 , compared with the matrix decomposition-based approach with the same compression ratio , tensor decomposition can make the output feature map of the compressed layer much more similar to the feature map of the original uncompressed layer . In other words , low-rank tensor method can provide better preservation of the important feature information and thus it can bring potential higher model compression performance . Question # 3 : What is the best suitable optimization objective that the integrated compression scheme should aim ? Analysis . To efficiently realize the joint exploration of model low-rankness and sparsity with promising compression performance , different optimization strategies have been proposed in the existing works . For instance , Yu et al . ( 2017 ) ; Ma et al . ( 2019 ) aim to minimize the difference between the original weight matrix/tensor and the approximated reconstruction . In addition , Ma et al . ( 2019 ) proposes to explicitly add the low-rank and sparse regularization terms to the overall objective function , which can guide the training-aware procedure to enforce the desired structural patterns . Our Proposal . Different from the existing approximation error-centered or regularized loss-based solutions , we propose that the efficient co-exploration scheme should be interpreted as the optimization procedure with the low-rank and sparse constraints . Our rationale lies on two important observations of the inherent drawbacks of the prior efforts . First , the approximation strategy adopted in Yu et al . ( 2017 ) ; Ma et al . ( 2019 ) focuses on making the reconstructed model approach the original model as close as possible . However , since 1 ) the approximation error always exists ; and 2 ) the original model is not the only choice to achieve the desired accuracy , such strategy inherently can only search the low-rank and sparse components in a limited exploration space , thereby affecting the overall compression performance . Second , though adding the regularization terms into loss function indeed facilitates the extraction of low-rank and sparse patterns , the effect of such simple regularizing method is still limited , especially considering the efforts of pushing for sparsity and for low-rankness may interfere with each other , thereby potentially causing unexpected conflicts . Instead , by explicitly imposing the low-rank and sparse constraints on the overall optimization problem , these two structural requirement can be simultaneously satisfied with the proper use of optimization technique ( to be discussed in Section 4 ) . As reported in Appendix A , our proposed constrained optimization strategy can successfully impose the desired low-rankness and sparsity onto the DNN models efficiently . Summary of Our Analysis . ¶ Performing joint low-rank decomposition and pruning in a spatially parallel way ( L+S ) is the preferable operational sequence . · High-order tensor decomposition is the most suitable choice for the low-rank approach used in the integrated compression scheme . ¸ Imposing low-rankness and sparsity as the direct hard constraints on the loss optimization should be adopted to better satisfy the desired structural requirement .

Summary: This paper introduces a new DNN compression technique. It consists in approximating the weights of a trained DNN by the sum of a low-rank and a sparse tensor. This is done by adding sparsity and low-rank constraints to the usual loss, the optimization problem being solved with ADMM. Experiments and comparisons with state of the art show the effectiveness of the technique.