Scalable Deep Neural Networks via Low-Rank Matrix Factorization

1 INTRODUCTION . As part of the great progress made in deep learning , deep neural network ( DNN ) models with higher performance have been proposed for various machine-learning tasks ( LeCun et al. , 2015 ) . However , these performance improvements require a higher number of parameters and greater computational complexity . Therefore , it is important to compress them without sacrificing the performance for running the models on resource-constrained devices . Han et al . ( 2016 ) reduced the memory requirement for devices by pruning and quantizing weight coefficients after training the models . Howard et al . ( 2017 ) ; Sandler et al . ( 2018 ) ; Howard et al . ( 2019 ) used factorized operations called depth-wise and point-wise convolutions in a proposal for light-weight models suited to mobile devices . However , these methods require pre-defined network structures and pruning the model weights after training . Recently , automated frameworks , such as the so-called neural architecture search ( NAS ) ( Zoph & Le , 2017 ) , have been proposed . Tan et al . ( 2019 ) proposed a NAS method to accelerate the inference speed on smartphones by incorporating resource-related constraints into the objective function . Stamoulis et al . ( 2019 ) significantly reduced the search costs for NAS by applying a gradient-based search scheme with a superkernel that shares weights for multiple convolutional kernels . However , the models trained by these methods are dedicated to specific devices , and thus do not possess the ability to be reconfigured for use on different devices . In order to change the model size , it is necessary to re-train them according to the resources of the target devices . For example , it has been reported that the inference speed when operating the same model on different devices differs according to the computing performance and memory capacity of the hardware accelerator ( Ignatov et al. , 2018 ) . Therefore , it is desirable that the model size can be flexibly changed according to the resources of the target devices without re-training the model , which we refer to as scalability in this paper . To this end , Yu et al . ( 2019 ) introduced switchable batch normalization ( BN ) ( Ioffe & Szegedy , 2015 ) , which switches BN layers according to pre-defined widths , and proposed “ slimmable ” networks whose width can be changed after training . Moreover , Yu & Huang ( 2019 ) proposed universally slimmable networks ( US-Nets ) that extend slimmable networks to arbitrary widths . However , since these methods directly reduce the width ( i.e. , dimensionality ) in each layer , the principal components are not taken into account . In addition , they reduce the width uniformly across all layers , which ignores differences in the importance of different layers . In this paper , we propose a novel method that enables DNNs to flexibly change their size after training . We factorize a weight matrix in each layer into two low-rank matrices after training the DNNs via singular value decomposition ( SVD ) . By changing the rank in each layer , our method can scale the model to an arbitrary size ( Figure 1 ( a ) ) . Our contributions are as follows . • We do not directly reduce the width but instead reduce the redundant basis in the column space of the weight matrix , which prevents the feature map in each layer from losing important features . • We introduce simple criteria that characterize the importance of each basis and layer , namely , the error- and complexity-based criteria . These enable to effectively compress the error and complexity of the models as little as possible . • We facilitate the performance of low rank networks with the following methods : a learning procedure that simultaneously minimizes losses for both the full and low rank networks ( Figure 1 ( b ) ) , and the mean & variance correction for each BN layer according to the given rank . In the experiments on image-classification tasks of the CIFAR-10/100 ( Krizhevsky , 2009 ) datasets using deep convolutional neural networks ( CNNs ) , our method exhibits better performance for up to approximately 75 % compressed models than slimmable networks and US-Nets . In the following , we first describe the details of our method ( Section 2 ) and briefly review related works ( Section 3 ) . Then , we give some experimental results ( Section 4 ) and conclude the paper ( Section 5 ) . 2 METHODS . In this section , we first give an overview then describe the details of the inference and learning methods . 2.1 OVERVIEW . For a layer in the network , let y = W⊤x ∈ Rn be an output vector given by linear transformation of an input vector x ∈ Rm with a weight matrix W ∈ Rm×n , where m and n are the numbers of input and output nodes , respectively . Let R be the rank of the weight matrix , with 1 ≤ R ≤ min ( m , n ) . Given U = ( u1 , . . . , uR ) ∈ Rm×R and V = ( v1 , . . . , vR ) ∈ Rn×R as matrices that have left and right singular vectors ( i.e. , bases ) in columns , and S = diag ( σ1 , . . . , σR ) ∈ RR×R as a matrix composed of singular values in diagonal components , we can formulate the SVD as W = USV ⊤ . An example of our scalable neural networks with fully connected layers is shown in Figure 1 . After the training , each weight matrix in the network is factorized into two matrices of rank R via SVD , and we control this value to change the model size . This can be viewed as inserting a sub-layer between the original layers and changing its width R. For the convolutional tensor W ∈ RKw×Kh×Cin×Cout of kernel width Kw , kernel height Kh , input channels Cin , and output channels Cout , we transform it to the matrix form W ∈ RKwKhCin×Cout , where m = KwKhCin and n = Cout . Then , we apply SVD as in Zhang et al . ( 2016 ) ; Wen et al . ( 2017 ) . This yields two layers with a tensor W1 ∈ RKw×Kh×Cin×R and a tensor W2 ∈ R1×1×R×Cout . The number of parameters in each layer becomes ( m+ n ) R by this factorization . Thus , we can compress the network to an arbitrary size by changing the rank r ( ≤ R ) within the range 1 ≤ r < mn/ ( m+ n ) . Associated with changing the rank , the monotonicity of approximation error holds for each layer . Proposition 2.1 . Let Wr = UrSrV ⊤r be a rank-r approximation using the truncated-SVD for W and let yr = W⊤r x . Then , the squared error between an original y and yr satisfies that ∥y − y1∥2 ≥ · · · ≥ ∥y − yr∥2 ≥ ∥y − yr+1∥2 ≥ · · · ≥ ∥y − yR∥2 = 0 . The proof is given in Appendix A . According to the above , errors between an original output y and its approximation yr monotonically decrease as the rank increases . Hence , it can be expected the performance of the entire network will scale with the model size , which is controlled by the rank in our method . 2.2 INFERENCE . 2.2.1 RANK SELECTION . Given a target size for a model , we select the rank of each layer by reference to the following criteria . Error-based criterion . According to Eq . ( 5 ) in Appendix A , the error associated with a rank-1 decrease is given by |v⊤y| = σ|u⊤x| = σ∥x∥| cos θ| . This implies that the error depends on the singular value σ and the cosine similarity between an input vector x and the corresponding left singular vector u . Based on this , we consider how to compress the model with as little error as possible by reducing the bases that induce lower errors . It has been reported that networks with BN layers and ReLUs ( rectified linear units ) ( Glorot et al. , 2011 ) possess the scale-invariance property ( Arora et al. , 2019 ) . Thus , the error |v⊤y| should be normalized by the scale of y in each layer . Exploiting the fact that ∥y∥ ≤ ∥W ∥2∥x∥ , we normalize it as |v⊤y|/ ( ∥W ∥2∥x∥ ) = σ| cos θ|/∥W ∥2 ∈ [ 0 , 1 ] , where ∥W ∥2 is the spectrum norm of W ( i.e. , the maximum singular value ) . Computing the cosine similarities is costly because each layer requires input x over the whole dataset . Therefore , we omit it and simply use the following criterion for selecting the rank : C1 ( ℓ , k ) = σ ( ℓ ) k /∥W ( ℓ ) ∥2 , ( 1 ) where ℓ is a layer index . This is equivalent to keeping ∥W ( ℓ ) −W ( ℓ ) r ∥2F small in each layer . We consider this is a simple but effective criterion for the following reasons . First , Arora et al . ( 2018 ) have reported that the subspace spanned by each layer ’ s weight vectors and the subspace spanned by their input vectors both become implicitly low rank and correlated after training . In other words , there should be many small singular values in each layer ’ s weight matrix . Second , the principal directions of the weights are correlated with those of the inputs . Thus , by reducing the bases that correspond to smaller singular values , we can reduce by a large number of ranks without significantly increasing the errors . Moreover , the cosine similarities are expected to be higher for large singular values , meaning that our method can reflect the principal directions of the data distribution even if we only use the singular values of the weight matrices as the criterion . Complexity-based criterion . We achieve a high compression rate by reducing the rank in layers that have a large number of parameters and multiply-accumulate operations ( MACs ) . For convolutional layers , the numbers of parameters , excluding biases and the MACs , are given by P = KwKhCinCout and M = PHW for a feature map of height H and width W , respectively . We use the following as a complexity-based criterion : C2 ( ℓ ) = ( 1− P ( ℓ ) / ∑ i P ( i ) ) ( 1−M ( ℓ ) / ∑ i M ( i ) ) , ( 2 ) where P ( ℓ ) and M ( ℓ ) are the numbers of parameters and the MACs in layer ℓ , respectively . By coupling the above two criteria , we reduce the bases with lower values of C ( ℓ , k ) = C1 ( ℓ , k ) C2 ( ℓ ) across the entire network . In practice , we compute the criterion for all bases after training . Then , we sort them in ascending order and store as a list . The only necessary step for selection is to reduce the first d bases in the list , where d is determined by the target model size . The algorithm is given in Appendix B . 2.2.2 BN CORRECTION . As pointed out by Yu et al . ( 2019 ) , the means and variances of the BN layers should be corrected when the model size is changed . Suppose that a BN layer is inserted right after the convolutional layer , and that the mean and variance of y ( = W⊤x ) are normalized in the BN layer . Then , we should correct those values according to the rank-r approximation of y ( i.e. , yr ) . Because yr = VrSrU ⊤ r x , yr lies in the rank-r subspace spanned by the columns of Vr . Hence , letting µ and Σ be , respectively , the population mean and covariance matrix for y , we can exactly compute their projection onto the subspace as µr = VrV ⊤r µ and Σr = VrV ⊤ r ΣVrV ⊤ r ( note that diagonal components are extracted from Σr for the BN layer ) . For practical reasons , we compute µ and Σ for each layer after training ( Ioffe & Szegedy , 2015 ) . Because Σ has n ( n+1 ) /2 extra parameters to store , we keep V ⊤ R̃ ΣVR̃ instead , where R̃ ( 1 ≤ R̃ < R ) is the maximum rank to be used , reducing the number of parameters to R̃ ( R̃ + 1 ) /2 . At the time of inference , we can correct the mean and variance according to the ranks in each layer . On the other hand , if a list of candidate model sizes is available in advance , we can retain the means and variances for those models as Yu & Huang ( 2019 ) . We compare both methods in Section 4 .

Summary: This paper proposes to reshape the weights of the layers of deep neural networks and parametrize them with a low-rank matrix decomposition (SVD). The rank is optimized using two criterion (error-based and complexity-based). Since the decomposition is applied post-hoc, the authors propose to correct the parameters of the batch norm analytically. The authors propose to jointly optimize a loss on the full network and the low-rank version. Experiments are done on CIFAR 10 an 100 with a VGG-15 and ResNet-34 architecture.