SPARK: co-exploring model SPArsity and low-RanKness for compact neural networks

1 INTRODUCTION . Deep neural network ( DNN ) has served as the backbone machine learning technique in many modern intelligent systems . To facilitate the low-cost deployment of DNN on the resource-constrained platforms , model compression , as a powerful strategy that can efficiently reduce DNN model size , has been extensively studied in recent years . To date , numerous compression approaches have been proposed to provide compact DNNs for many practical applications ( Han et al . ( 2015b ) ; Gong et al . ( 2019 ) ; Liao et al . ( 2021 ) ) . Among various types of model compression techniques , sparsification ( a.k.a. , pruning ) and lowrank decomposition are two representative and popular solutions ( Wang et al . ( 2021 ) ; Gao et al . ( 2021 ) ; Li et al . ( 2021a ) ; Xu et al . ( 2020 ) ) . As revealed by their names , the low-rank and sparse methods aim to explore and leverage the potential low-rankness and sparsity of the uncompressed DNNs , respectively . In practice , supported by the widely existed overparameterization phenomenon ( Denil et al . ( 2013 ) ; Han et al . ( 2015b ) ) , such hypothesized structure-level redundancy usually exists and thus it can be safely removed while still preserving high model performance . Co-exploring Low-rankness & Sparsity : Motivation . Considering the current prosperity of these two methods and their very distinct structural assumptions , an interesting and promising research topic is to explore the efficient integration of low-rank and sparse approaches towards a better model compression solution . As indicated and observed by Yu et al . ( 2017 ) , DNN models tend to exhibit both low-rankness and sparsity simultaneously . For instance , the smooth components in the weight filters can be represented in the low-rank space , and meanwhile some other important information is sparsely scattered . Evidently , fully leveraging such co-existence of these structure-level patterns , if being performed properly , can potentially bring a powerful compression solution with attractive performance . Existing Works . Unlike the current extensive research activities on individual low-rank and sparse methods , the investigations on integrating these two approaches , in an efficient and non-trivial way , are little explored . To date , only very few efforts study the joint exploration of low-rankness and sparsity for DNN model compression . As the pioneering work along this direction , Yu et al . ( 2017 ) develops a singular value decomposition ( SVD ) -free approach to closely approximate original DNN model via combining sparse representation and low-rank matrix factorization . Built on the interesting connection between filter decomposition and filter pruning , Li et al . ( 2020 ) interprets the decomposition and pruning of convolutional filter in a unified perspective . Most recently , Li et al . ( 2021b ) proposes a collaborative compression scheme to integrate SVD into model sparsification . By adopting a multi-step heuristic removal strategy , this post-training approach achieves promising task and compression performance . Unanswered Questions . Although these prior works have demonstrated the huge potentials and attractive benefits of jointly decomposing and pruning , the systematic investigation on their efficient integration is still missing . To be specific , several fundamental and critical questions , whose answers will directly impact the integration scheme and overall compression performance , have not been comprehensively explored yet . For instance , because pruning and decomposition can be jointly performed in several different ways , such as in parallel ( Yu et al . ( 2017 ) ) or in sequence ( Li et al . ( 2021b ) ) , which collaborative strategy is the best fit for the target DNN compression task ? Also , considering low-rankness can be exploited from different perspectives , which type of low-rank approach should be adopted ? The matrix factorization used in Li et al . ( 2020 ; 2021b ) ? Or even highorder tensor decomposition ? In addition , to achieve promising compression performance , what is the most suitable optimization objective that the integration scheme should aim ? The approximation error focused in Yu et al . ( 2017 ) ? The low-rankness/sparsity regularized loss in Yang et al . ( 2020 ) ? Or some other new alternatives ? Technical Preview and Contributions . To answer these questions and develop efficient integrated model compression solution , in this paper we perform systematic co-exploration on the model lowrankness and sparsity towards compact neural networks . To be specific , we first review and analyze several important design factors for the joint low-rank decomposition and pruning . Based on the observations and outcomes from our analysis , we then propose RASPA , a unified DNN compression framework that can simultaneously capture model low-RAnkness and SPArsity in an efficient way . Overall , the contributions of this paper are summarized as follows : • We systematically investigate and analyze the critical design knobs when co-exploring model low-rankness and sparsity , including operational sequence , low-rank format , and optimization objective . Based on our qualitative and quantitative analysis , we propose several recommended design options for efficient joint low-rank decomposition and pruning . • We develop a unified framework that formulates the integration of low-rank decomposition and pruning to an optimization problem with low-tensor-rank and sparse constraints . We then derive a training-aware approach to solve this challenging non-convex high-order tensor-format problem , and thereby leading to efficient exploration of rich low-rankness and sparsity in the model . • We empirically evaluate our proposed co-exploration solution for various DNN models on different datasets , and the experimental results demonstrate its very promising performance . Notably , on CIFAR-10 dataset , our solution can bring 1.25 % , 1.02 % and 0.16 % accuracy increase over the baseline ResNet-20 , ResNet-56 and DenseNet-40 models , respectively , and meanwhile the storage and computational costs are reduced by 70.4 % and 71.1 % ( for ResNet-20 ) , 37.5 % and 39.3 % ( for ResNet-56 ) and 52.4 % and 61.3 % ( for DenseNet40 ) , respectively . On ImageNet dataset , our approach can enable 0.52 % accuracy increase over baseline model with 48.7 % fewer parameters . 2 RELATED WORK . Sparsification . Sparsification , also known as pruning , has been extensively studied for model compression ( Han et al . ( 2015a ) ; Wen et al . ( 2016 ) ; Gao et al . ( 2019 ) ; Guo et al . ( 2016 ) ; Rao et al . ( 2021 ) ) . In general sparsifying a DNN can be realized via two ways . The first one is to use a certain criterion , e.g. , weight magnitude ( Han et al . ( 2015a ) ) , to directly remove some part of the model , and then perform fine-tuning to recover the accuracy . The second one is to add the sparsity-induced regularization during the training , such as ` 1 or group lasso term ( Wen et al . ( 2016 ) ) , to enforce the sparsity on the model . In addition , in order to achieve good balance between accuracy and complexity reduction , Gao et al . ( 2019 ) ; Guo et al . ( 2016 ) ; Rao et al . ( 2021 ) also proposes dynamic pruning . In such scenario , the DNN sparsification is essentially performed in an input-aware way – which part of model should be pruned is dynamically determined by each input data . Low-Rank Decomposition . Low-rank decomposition is another popular DNN compression approach . Based on different interpretation of the neural network models , the low-rank method can be categorized to matrix decomposition and tensor decomposition . Matrix decomposition views the 4-D weight tensor as the folded matrix , and hence it flattens the 4-D objective to 2-D format and decomposes the reshaped matrix to the product of two small matrices ( Tai et al . ( 2016 ) ; Li & Shi ( 2018 ) ; Xu et al . ( 2020 ) ) . On the other aspect , tensor decomposition directly factorizes the 4-D weight tensor to multiple small tensor cores without flattening operations . Such explicit high-order processing , by its nature , can better preserve the important spatial information and correlation existed in the weight tensors . To date several tensor decomposition techniques , such as tensor train , Tucker and tensor ring etc. , have been used for DNN model compression ( Kim et al . ( 2016 ) ; Novikov et al . ( 2015 ) ; Wang et al . ( 2018 ) ) . Joint Pruning and Decomposition . As observed by Yu et al . ( 2017 ) , a well-trained DNN tends to exhibit both sparsity and low-rankness simultaneously . Motivated by this observation , some prior efforts propose to co-explore these two complementary properties for model compression . As the pioneering work , Yu et al . ( 2017 ) decomposes the weight tensors of a pre-trained DNN model into independent low-rank and sparse parts and minimizes the reconstruction error . Different from this parallel scheme , Dubey et al . ( 2018 ) ; Li et al . ( 2021b ) adopt a sequential compression strategy via performing matrix factorization on a pruned model . In addition , Li et al . ( 2020 ) proposes to use the sparse/low-rank regularization term , instead of reconstruction error , to enforce the desired structural patterns . Also , notice that all of the existing works focus on using either SVD-based or SVD-free matrix decomposition to exploit the low-rankness of DNN model . 3 CO-EXPLORING LOW-RANKNESS AND SPARSITY : ANALYSIS . As outlined in Section 2 , the integration of low-rank decomposition and pruning can be specified by several important factors , including operational sequence , low-rankness format and the overall optimization objective . The existence of such large variety of different factors and their combinations , by its nature , calls for the systematic investigation on the best-suited co-exploration scheme for DNN compression . Such analysis framework , if being properly developed , can facilitate the optimal selection of various design factors already proposed in the existing literatures . More importantly , the outcome from this systematic study will further guide and provide the better integration choices that have not been discovered before . Questions to be Answered . Next we analyze the critical design knobs and factors for efficient coexploration on model low-rankness and sparsity . To that end , three important questions need to be answered . Question # 1 : What is the more suitable operational sequence when jointly low-rank decomposing and pruning DNN models ? Analysis . In general , the co-existence of model low-rankness and sparsity can be explored in different ways ( see Figure 1 ) . For instance , as adopted in Yu et al . ( 2017 ) , a well-trained DNN can be closely approximated as the combination of a low-rank component and a sparse component . In other words , the two types of structure-level properties are imposed and leveraged in a spatially parallel way , and we denote this strategy as L+S , where L and S represent low-rank decomposition and sparsification , respectively . On the other hand , the joint use of factorization and pruning can also be performed in a temporally sequential way . As illustrated in Figure 1 , the original model can be first imposed with low-rankness ( or sparsity ) , and the size of the resulting partially compressed model can be further reduced by the second-stage pruning ( or low-rank decomposition ) . Following the similar notation , such sequential operation can be denoted as S ( L ) and L ( S ) . In practice L ( S ) is a preferable choice that has been adopted in the prior works ( Dubey et al . ( 2018 ) ; Li et al . ( 2021b ) ) . Our Proposal . Among the above described three general operational schemes , we believe L+S is the more suitable choice when considering to integrate pruning and decomposition together for model compression . This is because unlike S ( L ) and L ( S ) , which ultimately still produce the compressed model in a single representation ( sparse or low-rank ) space , L+S enables the simultaneous representation of rich information of DNN models across different subspace , and thereby better preserving the structural characteristics and reducing the potential information loss . To verify our hypothesis , we examine the approximation error incurred by three integration schemes . As shown in Figure 2 , with the same compression ratio for the weight tensor of one layer of a pre-trained ResNet-20 on CIFAR-10 dataset , L+S shows much lower approximation error than its counterparts , especially in high compression ratio region . This experimental phenomenon demonstrates that L+S scheme indeed can capture both the low-rank and sparse characteristics of DNN model in an efficient way . Question # 2 : What is the best suitable lowrank decomposition approach used when coexploring low-rankness and sparsity ? Analysis . From the perspective of linear algebra , the low-rankness of a DNN model can be exploited using different ways . As illustrated in Figure 3 , for an example convolutional layer , imposing the low-rank structure can be realized by performing simple matrix factorization or high-order tensor decomposition . Specifically , Yu et al . ( 2017 ) chooses SVD-free method to factorize DNN model and obtain the low-rank component , and Li et al . ( 2021b ) proposes to use SVD-based decomposition to serve as the second-stage compression approach in its adopted L ( S ) scheme . Notice that though the weights of convolutional layer essentially form a 4-D tensor format , the existing works exploit the low-rankness via using matrix decomposition – the 4-D tensor needs to be first flatten to a 2-D matrix and it is then factorized to two small matrix components . Our Proposal . We argue that the high-order tensor decomposition , the option that has not been explored in the integration scheme before , is the better choice than the low-order matrix decomposition adopted in the existing works . This is because as a reshaping-free technique that can directly factorize the tensor-format data to multiple tensor cores , tensor decomposition , such as Tensor Train ( TT ) and Tucker , can naturally capture and preserve the important spatial information and correlation of the original weight tensors in a more efficient way . Therefore , less information loss is expected after performing low-rank tensor-based DNN compression . To verify our hypothesis , we compare the feature maps of the compressed convolutional layer of ResNet-20 on CIFAR-10 dataset using dif- ferent low-rank methods . As visualized in Figure 4 , compared with the matrix decomposition-based approach with the same compression ratio , tensor decomposition can make the output feature map of the compressed layer much more similar to the feature map of the original uncompressed layer . In other words , low-rank tensor method can provide better preservation of the important feature information and thus it can bring potential higher model compression performance . Question # 3 : What is the best suitable optimization objective that the integrated compression scheme should aim ? Analysis . To efficiently realize the joint exploration of model low-rankness and sparsity with promising compression performance , different optimization strategies have been proposed in the existing works . For instance , Yu et al . ( 2017 ) ; Ma et al . ( 2019 ) aim to minimize the difference between the original weight matrix/tensor and the approximated reconstruction . In addition , Ma et al . ( 2019 ) proposes to explicitly add the low-rank and sparse regularization terms to the overall objective function , which can guide the training-aware procedure to enforce the desired structural patterns . Our Proposal . Different from the existing approximation error-centered or regularized loss-based solutions , we propose that the efficient co-exploration scheme should be interpreted as the optimization procedure with the low-rank and sparse constraints . Our rationale lies on two important observations of the inherent drawbacks of the prior efforts . First , the approximation strategy adopted in Yu et al . ( 2017 ) ; Ma et al . ( 2019 ) focuses on making the reconstructed model approach the original model as close as possible . However , since 1 ) the approximation error always exists ; and 2 ) the original model is not the only choice to achieve the desired accuracy , such strategy inherently can only search the low-rank and sparse components in a limited exploration space , thereby affecting the overall compression performance . Second , though adding the regularization terms into loss function indeed facilitates the extraction of low-rank and sparse patterns , the effect of such simple regularizing method is still limited , especially considering the efforts of pushing for sparsity and for low-rankness may interfere with each other , thereby potentially causing unexpected conflicts . Instead , by explicitly imposing the low-rank and sparse constraints on the overall optimization problem , these two structural requirement can be simultaneously satisfied with the proper use of optimization technique ( to be discussed in Section 4 ) . As reported in Appendix A , our proposed constrained optimization strategy can successfully impose the desired low-rankness and sparsity onto the DNN models efficiently . Summary of Our Analysis . ¶ Performing joint low-rank decomposition and pruning in a spatially parallel way ( L+S ) is the preferable operational sequence . · High-order tensor decomposition is the most suitable choice for the low-rank approach used in the integrated compression scheme . ¸ Imposing low-rankness and sparsity as the direct hard constraints on the loss optimization should be adopted to better satisfy the desired structural requirement .

Summary: The authors argue and propose to compress the neural networks using an additive combination of TT decomposed and sparse tensors/matrices. To make this happen, authors formulate an optimizaiton problem and solve it using the ADMM framework. Authors provide compression results that are interesting (improved accuracy for a given compression rate), however, the method has an important shortcoming that requires setting the compression parameters by hand for each layer (ranks, sparsities) that limits its practical applicability. Additionally, authors miss a large body of related work on: a) additive combinations of compressions b) low-rank and tensor decomposition methods.