Year: 2019

Abstract:
Compressing deep neural networks (DNNs) is important for real-world applications operating on resource-constrained devices. However, it is difficult to change the model size once the training is completed, which needs re-training to configure models suitable for different devices. In this paper, we propose a novel method that enables DNNs to flexibly change their size after training. We factorize the weight matrices of the DNNs via singular value decomposition (SVD) and change their ranks according to the target size. In contrast with existing methods, we introduce simple criteria that characterize the importance of each basis and layer, which enables to effectively compress the error and complexity of models as little as possible. In experiments on multiple image-classification tasks, our method exhibits favorable performance compared with other methods.

heading: 1 INTRODUCTION
text: As part of the great progress made in deep learning, deep neural network (DNN) models with higher performance have been proposed for various machine-learning tasks (LeCun et al., 2015). However, these performance improvements require a higher number of parameters and greater computational complexity. Therefore, it is important to compress them without sacrificing the performance for running the models on resource-constrained devices.
Han et al. (2016) reduced the memory requirement for devices by pruning and quantizing weight coefficients after training the models. Howard et al. (2017); Sandler et al. (2018); Howard et al. (2019) used factorized operations called depth-wise and point-wise convolutions in a proposal for light-weight models suited to mobile devices. However, these methods require pre-defined network structures and pruning the model weights after training. Recently, automated frameworks, such as the so-called neural architecture search (NAS) (Zoph & Le, 2017), have been proposed. Tan et al. (2019) proposed a NAS method to accelerate the inference speed on smartphones by incorporating resource-related constraints into the objective function. Stamoulis et al. (2019) significantly reduced the search costs for NAS by applying a gradient-based search scheme with a superkernel that shares weights for multiple convolutional kernels.
However, the models trained by these methods are dedicated to specific devices, and thus do not possess the ability to be reconfigured for use on different devices. In order to change the model size, it is necessary to re-train them according to the resources of the target devices. For example, it has been reported that the inference speed when operating the same model on different devices differs according to the computing performance and memory capacity of the hardware accelerator (Ignatov et al., 2018). Therefore, it is desirable that the model size can be flexibly changed according to the resources of the target devices without re-training the model, which we refer to as scalability in this paper.
To this end, Yu et al. (2019) introduced switchable batch normalization (BN) (Ioffe & Szegedy, 2015), which switches BN layers according to pre-defined widths, and proposed “slimmable” networks whose width can be changed after training. Moreover, Yu & Huang (2019) proposed universally slimmable networks (US-Nets) that extend slimmable networks to arbitrary widths. However, since these methods directly reduce the width (i.e., dimensionality) in each layer, the principal components are not taken into account. In addition, they reduce the width uniformly across all layers, which ignores differences in the importance of different layers.
In this paper, we propose a novel method that enables DNNs to flexibly change their size after training. We factorize a weight matrix in each layer into two low-rank matrices after training the DNNs via singular value decomposition (SVD). By changing the rank in each layer, our method can scale the model to an arbitrary size (Figure 1(a)). Our contributions are as follows.
• We do not directly reduce the width but instead reduce the redundant basis in the column space of the weight matrix, which prevents the feature map in each layer from losing important features. • We introduce simple criteria that characterize the importance of each basis and layer,
namely, the error- and complexity-based criteria. These enable to effectively compress the error and complexity of the models as little as possible. • We facilitate the performance of low rank networks with the following methods: a learning
procedure that simultaneously minimizes losses for both the full and low rank networks (Figure 1(b)), and the mean & variance correction for each BN layer according to the given rank.
In the experiments on image-classification tasks of the CIFAR-10/100 (Krizhevsky, 2009) datasets using deep convolutional neural networks (CNNs), our method exhibits better performance for up to approximately 75% compressed models than slimmable networks and US-Nets. In the following, we first describe the details of our method (Section 2) and briefly review related works (Section 3). Then, we give some experimental results (Section 4) and conclude the paper (Section 5).

heading: 2 METHODS
text: In this section, we first give an overview then describe the details of the inference and learning methods.

heading: 2.1 OVERVIEW
text: For a layer in the network, let y = W⊤x ∈ Rn be an output vector given by linear transformation of an input vector x ∈ Rm with a weight matrix W ∈ Rm×n, where m and n are the numbers of input and output nodes, respectively. Let R be the rank of the weight matrix, with 1 ≤ R ≤ min (m,n). Given U = (u1, . . . ,uR) ∈ Rm×R and V = (v1, . . . ,vR) ∈ Rn×R as matrices that have left and right singular vectors (i.e., bases) in columns, and S = diag (σ1, . . . , σR) ∈ RR×R as a matrix composed of singular values in diagonal components, we can formulate the SVD as W = USV ⊤.
An example of our scalable neural networks with fully connected layers is shown in Figure 1. After the training, each weight matrix in the network is factorized into two matrices of rank R via SVD, and we control this value to change the model size. This can be viewed as inserting a sub-layer between the original layers and changing its width R. For the convolutional tensor W ∈ RKw×Kh×Cin×Cout of kernel width Kw, kernel height Kh, input channels Cin, and output channels Cout, we transform it to the matrix form W ∈ RKwKhCin×Cout , where m = KwKhCin and n = Cout. Then, we apply SVD as in Zhang et al. (2016); Wen et al. (2017). This yields two layers with a tensor W1 ∈ RKw×Kh×Cin×R and a tensor W2 ∈ R1×1×R×Cout . The number of parameters in each layer becomes (m+ n)R by this factorization. Thus, we can compress the network to an arbitrary size by changing the rank r (≤ R) within the range 1 ≤ r < mn/ (m+ n). Associated with changing the rank, the monotonicity of approximation error holds for each layer.
Proposition 2.1. Let Wr = UrSrV ⊤r be a rank-r approximation using the truncated-SVD for W and let yr = W⊤r x. Then, the squared error between an original y and yr satisfies that ∥y − y1∥2 ≥ · · · ≥ ∥y − yr∥2 ≥ ∥y − yr+1∥2 ≥ · · · ≥ ∥y − yR∥2 = 0.
The proof is given in Appendix A. According to the above, errors between an original output y and its approximation yr monotonically decrease as the rank increases. Hence, it can be expected the performance of the entire network will scale with the model size, which is controlled by the rank in our method.

heading: 2.2 INFERENCE

heading: 2.2.1 RANK SELECTION
text: Given a target size for a model, we select the rank of each layer by reference to the following criteria. Error-based criterion. According to Eq. (5) in Appendix A, the error associated with a rank-1 decrease is given by |v⊤y| = σ|u⊤x| = σ∥x∥| cos θ|. This implies that the error depends on the singular value σ and the cosine similarity between an input vector x and the corresponding left singular vector u. Based on this, we consider how to compress the model with as little error as possible by reducing the bases that induce lower errors. It has been reported that networks with BN layers and ReLUs (rectified linear units) (Glorot et al., 2011) possess the scale-invariance property (Arora et al., 2019). Thus, the error |v⊤y| should be normalized by the scale of y in each layer. Exploiting the fact that ∥y∥ ≤ ∥W ∥2∥x∥, we normalize it as |v⊤y|/ (∥W ∥2∥x∥) = σ| cos θ|/∥W ∥2 ∈ [0, 1], where ∥W ∥2 is the spectrum norm of W (i.e., the maximum singular value). Computing the cosine similarities is costly because each layer requires input x over the whole dataset. Therefore, we omit it and simply use the following criterion for selecting the rank:
C1(ℓ, k) = σ (ℓ) k /∥W (ℓ)∥2, (1)
where ℓ is a layer index. This is equivalent to keeping ∥W (ℓ) −W (ℓ)r ∥2F small in each layer. We consider this is a simple but effective criterion for the following reasons. First, Arora et al. (2018) have reported that the subspace spanned by each layer’s weight vectors and the subspace spanned by their input vectors both become implicitly low rank and correlated after training. In other words, there should be many small singular values in each layer’s weight matrix. Second, the principal directions of the weights are correlated with those of the inputs. Thus, by reducing the bases that correspond to smaller singular values, we can reduce by a large number of ranks without significantly increasing the errors. Moreover, the cosine similarities are expected to be higher for large singular values, meaning that our method can reflect the principal directions of the data distribution even if we only use the singular values of the weight matrices as the criterion.
Complexity-based criterion. We achieve a high compression rate by reducing the rank in layers that have a large number of parameters and multiply-accumulate operations (MACs). For convolutional layers, the numbers of parameters, excluding biases and the MACs, are given by P = KwKhCinCout and M = PHW for a feature map of height H and width W , respectively.
We use the following as a complexity-based criterion:
C2(ℓ) =
( 1− P (ℓ)/
∑ i P (i)
)( 1−M (ℓ)/
∑ i M (i)
) , (2)
where P (ℓ) and M (ℓ) are the numbers of parameters and the MACs in layer ℓ, respectively. By coupling the above two criteria, we reduce the bases with lower values of C(ℓ, k) = C1(ℓ, k) C2(ℓ) across the entire network. In practice, we compute the criterion for all bases after training. Then, we sort them in ascending order and store as a list. The only necessary step for selection is to reduce the first d bases in the list, where d is determined by the target model size. The algorithm is given in Appendix B.

heading: 2.2.2 BN CORRECTION
text: As pointed out by Yu et al. (2019), the means and variances of the BN layers should be corrected when the model size is changed. Suppose that a BN layer is inserted right after the convolutional layer, and that the mean and variance of y ( = W⊤x ) are normalized in the BN layer. Then, we should correct those values according to the rank-r approximation of y (i.e., yr). Because yr = VrSrU ⊤ r x, yr lies in the rank-r subspace spanned by the columns of Vr. Hence, letting µ and Σ be, respectively, the population mean and covariance matrix for y, we can exactly compute their projection onto the subspace as µr = VrV ⊤r µ and Σr = VrV ⊤ r ΣVrV ⊤ r (note that diagonal components are extracted from Σr for the BN layer). For practical reasons, we compute µ and Σ for each layer after training (Ioffe & Szegedy, 2015). Because Σ has n(n+1)/2 extra parameters to store, we keep V ⊤
R̃ ΣVR̃ instead, where R̃ (1 ≤ R̃ < R) is the maximum rank to be used, reducing
the number of parameters to R̃(R̃ + 1)/2. At the time of inference, we can correct the mean and variance according to the ranks in each layer. On the other hand, if a list of candidate model sizes is available in advance, we can retain the means and variances for those models as Yu & Huang (2019). We compare both methods in Section 4.

heading: 2.3 LEARNING
text: Although our scalable neural networks can operate regardless of learning methods, we propose a method to gain better performance. We simultaneously minimize losses for both the full-rank and the low-rank networks as follows.
min Θ
1
B B∑ b=1 { (1− λ)L(Db,W,Θ) + λL(Db, W̃,Θ) } (3)
Here, L(·) is a loss function, Db is a set of training samples in a mini-batch, B is the batch size, and λ ∈ [0, 1] is a hyperparameter for balancing between the two losses. For this,W = {W (ℓ)}Lℓ=1, W̃ , and Θ are sets of L weight matrices, their low-rank approximations, and other trainable parameters (e.g., biases), respectively. W̃ is generated from W via SVD and Θ is shared between the fulland low-rank networks. Therefore, the number of total paramters is not increased in learning. As shown in Figure 1(b), we additionally propagate each mini batch to a low-rank network. Because W (ℓ)r = U (ℓ) r U (ℓ)⊤ r W (ℓ), the gradient with respect to W (ℓ) can be computed as follows: 1
1
B B∑ b=1
{ (1− λ)∂L(Db,W,Θ)
∂W (ℓ) + λU (ℓ)r U (ℓ)⊤ r ∂L(Db, W̃,Θ) ∂W (ℓ) r
} . (4)
The gradients for Θ are simply computed from the λ-weighted average for those of both networks.
Since we need a single model that achieve good performance in multiple sizes which are to be selected at the inference phase, we randomly select the model size for the low-rank network at each iteration step. Specifically, a global rank ratio Z is sampled from a uniform distribution U(αl, αu) with 0 < αl < αu ≤ 1. Then, letting R(ℓ) be the rank of W (ℓ), we reduce (1 − Z) ∑L ℓ=1 R (ℓ) bases across the entire network using the criterion mentioned in subsection 2.2.1. In a later section, we experimentally investigate the effects of the parameters λ, αl, and αu in the experiment.
1In fact, U (ℓ)r depends on W (ℓ), but we treat it as constant for simplicity.
Arora et al. (2018); Suzuki (2019) derived the generalization error bound for DNNs under a condition that the trained network has near low-rank weight matrices. They proved that the condition contributes not only to yield a better generalization error bound for the non-compressed network but also to compress the network efficiently. That motivates our approaches: a learning scheme which aims to facilitate the performance of the low-rank networks as well as that of the full-rank network, and a compression scheme which reduces the redundant basis obtained via SVD.

heading: 3 RELATED WORK
text: Low-rank approximation & regularization. Compression methods based on low-rank approximation have been proposed in the literature. Denil et al. (2013); Tai et al. (2016); Ioannou et al. (2016) trained networks after factorizing the weight matrix into a low-rank form. Ioannou et al. (2016) achieved a high compression rate by factorizing a convolutional kernel of Kw × Kh into Kw × 1 and 1×Kh. Denton et al. (2014); Lebedev et al. (2015); Kim et al. (2016) proposed methods that use tensor factorization without rearranging the convolutional tensor into the matrix form. Yu et al. (2017) further improved the compression rate by incorporating sparseness into the low-rank constraint. Zhang et al. (2016); Li & Shi (2018) took resource-related constraints into account to automatically select an appropriate rank. Each of these methods trains a network with pre-defined ranks or compress redundant networks by applying complicated optimizations under a given target size for the model. That is, those methods would require re-training to reconfigure the models for different devices.
Kliegl et al. (2017); Xu et al. (2019) utilized trace-norm regularization as a low-rank constraint in learning the network. Wen et al. (2017) proposed a novel method called force regularization to obtain the low-rank weights. The performance of these methods depends on a hyperparamter to adjust strength of regularization. It is difficult to decide on an appropriate range for the hyperparameter in advance, meaning that selection requires trial and error to achieve a particular model size.
Scalable neural networks. Chen et al. (2018) represented the data flow in ResNet-type structures (He et al., 2016) as ordinary differential equations (ODEs), and proposed a Neural-ODEs, which can be used to arbitrarily control the computational cost in the depth direction. Zhang et al. (2019) also obtained scalability in the depth direction by allowing pre-defined intermediate layers to be bypassed.
Yu et al. (2019); Yu & Huang (2019) proposed slimmable networks and US-Nets, which are scalable in the width direction. Their works are closely related to ours, but there are differences in some aspects. First, since their methods directly and uniformly reduce the width for every layer, the principal components are not taken into account, and the relative importance of each layer is not considered. Second, for US-Nets in particular, they introduced a “sandwich rule” to keep the performance for an arbitrary width. However, this rule does not guarantee monotonicity of the error with increasing layer width. In the next section, we compare our method with them.

heading: 4 EXPERIMENTS
text: We evaluate our methods on the image-classification tasks of CIFAR-10/100 (Krizhevsky, 2009) datasets using deep CNNs. The CIFAR-10/100 datasets contain 32× 32 images for object recognition including 10 and 100 classes, respectively. Each dataset contains 50K images for training and 10K images for validation. We implement our method with TensorFlow (Abadi et al., 2015).

heading: 4.1 ABLATION STUDY
text: We test each component in our method on the CIFAR-10 dataset. We use the same baseline setup as in Zagoruyko & Komodakis (2016), which is summarized in Table 1 in Appendix C. Unless otherwise specified, we report the average result from 5 trials with different random seeds. We adopt a VGG-like network with 15 layers (Zagoruyko, 2015; Liu et al., 2017) 2, which we refer to as VGG-15 below.
Firstly, we evaluate our learning method for various values of the parameters αl and αu, fixing λ = 0.5. Our method requires SVD at each iteration step, which makes it costly. To address this, we suppose that the weight subspaces are not drastically changed at each step and recompute the SVD after every two steps, reusing the results to speed up the training. We illustrate the validation accuracy of a full-rank model for different values of αl (resp., αu) with αu = 1.0 (resp., αl = 0.01) fixed, on the left (resp., center) of Figure 2. It can be observed that smaller values of αl and larger values of αu are better. This can be interpreted as indicating that it is better for a full-rank model
2Since the VGG-networks are originally designed for classifying the ImageNet dataset (Deng et al., 2009), we use a smaller type than the original for the CIFAR datasets, as used by Liu et al. (2017).
to learn with various low-rank models than to learn with models biased to a specific range of ranks. Thus, we set αl = 0.01 and αu = 1.0 for the other experiments described below. On the right side of Figure 2, we show the maximum singular value for each basis index in a full-rank model 3. We can see that our learning method obtains smaller singular values than the baseline. This implies that our learning method has an effect similar to trace-norm regularization (Kliegl et al., 2017), suggesting that we can suppress the errors produced by reducing the bases.
Next, we evaluated the performance of our inference method for various model sizes. In Figure 3, we illustrate the inference results for validation data with various number of parameters and MACs. In the figure, “infer (uni)” indicate the results obtained by uniformly reducing the basis in each layer. Concretely, with a global rank ratio G ∈ (0, 1], we reduce (1 − G)R(ℓ) bases in order from the one corresponding to the smallest singular value. Despite the method being simple, the accuracy changes almost smoothly, and it can be confirmed that there the accuracy scales against changes in the model size. This can be considered as due to the monotonicity of errors, which is formalized in Proposition 2.1. Additionally, the performance is also improved with our learning method by applying uniform rank selection and by using our BN correction. Furthermore, the performance with respect to the parameters is improved when we apply the error- and complexity-based criteria for rank selection to both inference and learning (in the figure, “c1” and “c1c2”). However, the performance with respect to the MACs is dropped by changing the rank selection from uniform (“uni”) to error-based (“c1”). As shown on the left side of Figure 4, it is more effective for decreasing MACs to reduce the parameters in shallower layers, which have large feature maps. However, the error-based criterion tends to reduce the parameters in deeper layers because those tend to be low rank. When both criteria are applied (in the figure, “c1c2”), the performance is also improved for the MACs. We show the rank-selection results for different criteria on the right side of Figure 4. It can be confirmed that the ranks are decreased for 4, 6, 7, 9, and 10 layers which have large MACs in the case with both criteria (“c1c2”) relative to the case with only the error-based criterion (“c1”). For the BN correction, our method is effective, but it is better with a method that recomputes the means and variances for given ranks (“bnRe”). Because our method is layer-by-layer correction, this probably occurs because our method cannot fully correct for the inter-layer gap, with the statistics of the deep layer changing due to the reduction of rank in the shallow layer.
Additionally, we investigate the effect of a parameter λ. We evaluate the validation accuracy with respect to the number of paramters for λ ∈ {0.1, 0.2, 0.3, 0.4, 0.5} with VGG-15 and ResNet-34 on the CIFAR-10/100 datasets. The results are shown in Figure 7 in Appendix D. We consider that there is a trade-off between the performance of full- and low-rank models, which depends on λ.

heading: 4.2 COMPARISON WITH SLIMMABLE NETWORKS
text: We compare our method with slimmable networks (Yu et al., 2019) and US-Nets (Yu & Huang, 2019) in terms of performance on the CIFAR-10/100 datasets. We adopt VGG-15 and ResNet34 (He et al., 2016). We implement the models based on the Yu’s code, available at https:// github.com/JiahuiYu/slimmable_networks (written in PyTorch (Paszke et al., 2017)).
3We let σij be a singular value for a basis j in a layer i and then compute maxi (σij). For layers with lower ranks, we simply fill the missing part with zeros.
US-Nets is trained with 2 random widths between the lower and upper width and in-place distillation (Yu & Huang, 2019), then BN-calibration (Yu & Huang, 2019) is applied to each of the slimmable networks and US-Nets after training. For our method, we incorporate all components into the comparisons and adopt BN correction with recomputation. We train the models using λ = 0.4 and the same setup as in the previous subsection. In the following, we report the results for models after the last iteration in training.
First, we compare the scalability of ResNet-34 on the CIFAR-100 dataset. We illustrate the inference results over various numbers of parameters for 5 models trained with different random seeds in Figure 5. The results in the figure show that US-Nets are unstable, which is a problem for practical use. This instability is because US-Nets do not have monotonic error changes in each layer, a property that our method ensures. Next, we show the results for comparison of VGG-15 on CIFAR10 and ResNet-34 on CIFAR-100 in Figure 6. The notations “base (Yu’s code)” and “base (our code)” indicate the baseline results obtained by the Yu’s code and our code with the same setup. Our baseline is slightly better than the Yu’s baseline. We consider this to be due to differences in the framework. Comparing the results with those for VGG-15 on CIFAR-10, our method tends to be more accurate in terms of the number of parameters than in terms of the number of MACs. Since deep layers have more parameters than shallow layers, the rank in deep layers tends to be lower than in shallow layers, resulting in more paramters reduced in deep layers by our method. In contrast, US-Nets reduce the width uniformly across layers, which may contribute to reducing the number of MACs. However, reducing the number of MACs does not necessarily lead to cut the inference cost dominantly, depending on the target device (Yang et al., 2018). Although we only consider the number of parameters and MACs as the complexity metrics in this paper, other metrics such as memory footprint, memory access cost, and runtime latency should be taken into account for validating the effectiveness in practical use (Tan et al., 2019; Sandler et al., 2018; Dai et al., 2019).
We can see that the accuracy of our method is lower than that of US-Nets when the compression rate is extremely high. Our method uses SVD and reduces the bases, which means it does not change the number of inputs and outputs (i.e., the in and out dimensionalities). Because the number of parameters in each layer is (m + n)r, it decreases linearly with respect to the rank. US-Nets reduce both input and output dimensionality, meaning that the number of parameters is decreased at a quadratic rate. This makes it easier for US-Nets to achieve extremely high compression. However, our method is better in larger regimes. In particular, for ResNet-34 on CIFAR-100, the performance of slimmable networks and US-Nets on the full-size model are decreased, while our method does not decrease performance much. We illustrate additional comparison results in Figure 8 in Appendix D and give an analysis of per-layer error in Appendix E.

heading: 5 CONCLUSIONS
text: We proposed a novel method that enables DNNs to flexibly change their size after training. We used to factorize the weight matrix for each layer into two low-rank matrices after training the DNNs. By changing the rank in each layer, our method can scale the model to an arbitrary size. We introduced simple criteria for characterizing the importance of each basis and layer; these are the error- and complexity-based criteria. Those criteria enabled effectively compressing models without introducing much error. In experiments on multiple image-classification tasks using deep CNNs, our method exhibited good performance relative to that of other methods.

heading: A PROOF OF PROPOSITION 2.1

heading: As y = yR by definition, ∥y−yR∥2 = 0. To prove ∥y−y1∥2 ≥ · · · ≥ ∥y−yr∥2 ≥ ∥y−yr+1∥2 ≥
text: · · · ≥ ∥y − yR∥2, we show that ∥y − yr∥2 − ∥y − yr+1∥2 ≥ 0 for 1 ≤ r ≤ R− 1.
Proof.
∥y − yr∥2 − ∥y − yr+1∥2 = ∥ ( In − VrV ⊤r ) y∥2 − ∥ ( In − Vr+1V ⊤r+1 ) y∥2
= y⊤ ( In − VrV ⊤r ) y − y⊤ ( In − Vr+1V ⊤r+1 ) y
= y⊤ ( Vr+1V ⊤ r+1 − VrV ⊤r ) y
= y⊤ ( vr+1v ⊤ r+1 ) y
= ( v⊤r+1y )2 ≥ 0 (5) Here, In indicates the identity matrix with size n× n.

heading: B AN ALGORITHM FOR RANK SELECTION
text: Algorithm 1 Rank selection Input: A network with weight matricesW = {W (ℓ)}Lℓ=1 in which W (ℓ) has rank R(ℓ). Input: A criterion C and the target model size T (e.g., # of parameters and MACs). Output: A set of tuples S which contains indices of layer and basis.
1: S← ∅ 2: for ℓ = 1, . . . , L do 3: for k = 1, . . . , R(ℓ) do 4: Compute C (ℓ, k). 5: S← S ∪ {(ℓ, k)}. 6: end for 7: end for 8: Arrange elements in S in ascending order of C (ℓ, k). 9: Delete the first d elements in S to satisfy the model size T .
10: return S

heading: C BASELINE SETUP OF EXPERIMENTS ON CIFAR-10/100 DATASETS
text: D ADDITIONAL RESULTS FOR VGG-15 AND RESNET-34 ON CIFAR DATASETS

heading: E AN ANALYSIS OF PER-LAYER ERROR
text: We train VGG-15 on the CIFAR-10 dataset with US-Nets (Yu & Huang, 2019) and our method. First, we illustrate learned weights in Figure 9. For comparison, we show a weight matrix W for US-Nets and its factorized form US obtained via SVD for our method. The rank of W in the 1st convolutional layer is 27, and thus only 27 convolutional tensors are shown for our method in the upper part of the figure. Since W becomes low-rank after training, there is only a little number of
dominant bases which correspond to higher singular values in S. For US-Nets in particular, we can see that weight coefficients of large absolute values are concentrated in lower channels. It can be considered that US-Nets implicitly attract important features into lower channels because US-Nets reduce channels in order from the one with higher indices. Therefore, it can be expected the errors induced by reducing channels are partially suppressed for US-Nets.
Next, we compute the sum of squared error: ∑N
n=1 ∥Y (ℓ) n − Ỹ (ℓ)n ∥2F / ∑N n=1 ∥Y (ℓ) n ∥2F over the
validation dataset, where N , Y (ℓ)n , and Ỹ (ℓ) n are the number of validation samples, an output feature map of layer ℓ for the full-size network, and that for the compressed network, respectively. With compressing the entire network from 100% to about 5% in terms of the total number of parameters, we compute the squared error with respect to the number of parameters in each layer as shown in Figure 10. For our method, we reduce the bases of weight marices (i.e., columns in U (ℓ)S(ℓ)) according to the criterion C1(ℓ, k) C2(ℓ) and compute Ỹ (ℓ) with a low-rank weight matrix W (ℓ)r in each layer. For US-Nets, since the dimensionality of Ỹ (ℓ) is decreased with reducing channels for every layer, we fill the missing part with zeros. It can be confirmed the squared errors by our method are suppressed more than that by US-Nets in Figure 10. We consider this is because our method do not directly reduce the channel as in US-Nets but instead reduce the redundant basis in the column space of the weight matrix, which prevents the feature map in each layer from losing important features (i.e., principal components).

Review summaries:
This paper proposes to reshape the weights of the layers of deep neural networks and parametrize them with a low-rank matrix decomposition (SVD). The rank is optimized using two criterion (error-based and complexity-based). Since the decomposition is applied post-hoc, the authors propose to correct the parameters of the batch norm analytically. The authors propose to jointly optimize a loss on the full network and the low-rank version. Experiments are done on CIFAR 10 an 100 with a VGG-15 and ResNet-34 architecture.