Title: SCALABLE DEEP NEURAL NETWORKS VIA LOW-RANK MATRIX FACTORIZATION

Abstract: Compressing deep neural networks (DNNs) is important for real-world applications operating on resource-constrained devices. However, it is difficult to change the model size once the training is completed, which needs re-training to configure models suitable for different devices. In this paper, we propose a novel method that enables DNNs to flexibly change their size after training. We factorize the weight matrices of the DNNs via singular value decomposition (SVD) and change their ranks according to the target size. In contrast with existing methods, we introduce simple criteria that characterize the importance of each basis and layer, which enables to effectively compress the error and complexity of models as little as possible. In experiments on multiple image-classification tasks, our method exhibits favorable performance compared with other methods.

1. INTRODUCTION:

As part of the great progress made in deep learning, deep neural network (DNN) models with higher performance have been proposed for various machine-learning tasks (LeCun et al., 2015) . However, these performance improvements require a higher number of parameters and greater computational complexity. Therefore, it is important to compress them without sacrificing the performance for running the models on resource-constrained devices. Han et al. (2016) reduced the memory requirement for devices by pruning and quantizing weight coefficients after training the models. Howard et al. (2017) ; Sandler et al. (2018) ; Howard et al. (2019) used factorized operations called depth-wise and point-wise convolutions in a proposal for light-weight models suited to mobile devices. However, these methods require pre-defined network structures and pruning the model weights after training. Recently, automated frameworks, such as the so-called neural architecture search (NAS) (Zoph & Le, 2017) , have been proposed. Tan et al. (2019) proposed a NAS method to accelerate the inference speed on smartphones by incorporating resource-related constraints into the objective function. Stamoulis et al. (2019) significantly reduced the search costs for NAS by applying a gradient-based search scheme with a superkernel that shares weights for multiple convolutional kernels.

However, the models trained by these methods are dedicated to specific devices, and thus do not possess the ability to be reconfigured for use on different devices. In order to change the model size, it is necessary to re-train them according to the resources of the target devices. For example, it has been reported that the inference speed when operating the same model on different devices differs according to the computing performance and memory capacity of the hardware accelerator (Ignatov et al., 2018) . Therefore, it is desirable that the model size can be flexibly changed according to the resources of the target devices without re-training the model, which we refer to as scalability in this paper.

To this end, Yu et al. (2019) introduced switchable batch normalization (BN) (Ioffe & Szegedy, 2015) , which switches BN layers according to pre-defined widths, and proposed "slimmable" networks whose width can be changed after training. Moreover, Yu & Huang (2019) proposed universally slimmable networks (US-Nets) that extend slimmable networks to arbitrary widths. However, since these methods directly reduce the width (i.e., dimensionality) in each layer, the principal components are not taken into account. In addition, they reduce the width uniformly across all layers, which ignores differences in the importance of different layers. Learning phase of our method. We explicitly minimizes losses for both of full-and low-rank network, which is designed not only to keep the performance of full-rank network but also to improve multiple low-rank networks (to be used at the inference phase). These schemes can also be applied to CNNs.

In this paper, we propose a novel method that enables DNNs to flexibly change their size after training. We factorize a weight matrix in each layer into two low-rank matrices after training the DNNs via singular value decomposition (SVD). By changing the rank in each layer, our method can scale the model to an arbitrary size (Figure 1 (a)). Our contributions are as follows.

• We do not directly reduce the width but instead reduce the redundant basis in the column space of the weight matrix, which prevents the feature map in each layer from losing important features. • We introduce simple criteria that characterize the importance of each basis and layer, namely, the error-and complexity-based criteria. These enable to effectively compress the error and complexity of the models as little as possible. • We facilitate the performance of low rank networks with the following methods: a learning procedure that simultaneously minimizes losses for both the full and low rank networks (Figure 1 (b)), and the mean & variance correction for each BN layer according to the given rank.

In the experiments on image-classification tasks of the CIFAR-10/100 (Krizhevsky, 2009) datasets using deep convolutional neural networks (CNNs), our method exhibits better performance for up to approximately 75% compressed models than slimmable networks and US-Nets. In the following, we first describe the details of our method (Section 2) and briefly review related works (Section 3). Then, we give some experimental results (Section 4) and conclude the paper (Section 5).

2. METHODS:

In this section, we first give an overview then describe the details of the inference and learning methods.

2.1. OVERVIEW:

For a layer in the network, let y = W ⊤ x ∈ R n be an output vector given by linear transformation of an input vector x ∈ R m with a weight matrix W ∈ R m×n , where m and n are the numbers of input and output nodes, respectively. Let R be the rank of the weight matrix, with

1 ≤ R ≤ min (m, n). Given U = (u 1 , . . . , u R ) ∈ R m×R and V = (v 1 , . . . , v R ) ∈ R n×R

as matrices that have left and right singular vectors (i.e., bases) in columns, and S = diag (σ 1 , . . . , σ R ) ∈ R R×R as a matrix composed of singular values in diagonal components, we can formulate the SVD as W = U SV ⊤ .

An example of our scalable neural networks with fully connected layers is shown in Figure 1 .

After the training, each weight matrix in the network is factorized into two matrices of rank R via SVD, and we control this value to change the model size. This can be viewed as inserting a sub-layer between the original layers and changing its width R. For the convolutional tensor W ∈ R Kw×K h ×Cin×Cout of kernel width K w , kernel height K h , input channels C in , and output channels C out , we transform it to the matrix form W ∈ R KwK h Cin×Cout , where m = K w K h C in and n = C out . Then, we apply SVD as in Zhang et al. (2016) ; Wen et al. (2017) . This yields two layers with a tensor W 1 ∈ R Kw×K h ×Cin×R and a tensor W 2 ∈ R 1×1×R×Cout . The number of parameters in each layer becomes (m + n) R by this factorization. Thus, we can compress the network to an arbitrary size by changing the rank r (≤ R) within the range 1 ≤ r < mn/ (m + n).

Associated with changing the rank, the monotonicity of approximation error holds for each layer.

Proposition 2.1. Let W r = U r S r V ⊤ r be a rank-r approximation using the truncated-SVD for W and let y r = W ⊤ r x. Then, the squared error between an original y and y r satisfies that ∥y -

y 1 ∥ 2 ≥ • • • ≥ ∥y -y r ∥ 2 ≥ ∥y -y r+1 ∥ 2 ≥ • • • ≥ ∥y -y R ∥ 2 = 0.

The proof is given in Appendix A. According to the above, errors between an original output y and its approximation y r monotonically decrease as the rank increases. Hence, it can be expected the performance of the entire network will scale with the model size, which is controlled by the rank in our method.

2.2. INFERENCE:



2.2.1. RANK SELECTION:

Given a target size for a model, we select the rank of each layer by reference to the following criteria.

Error-based criterion. According to Eq. (5) in Appendix A, the error associated with a rank-1 decrease is given by |v ⊤ y| = σ|u ⊤ x| = σ∥x∥| cos θ|. This implies that the error depends on the singular value σ and the cosine similarity between an input vector x and the corresponding left singular vector u. Based on this, we consider how to compress the model with as little error as possible by reducing the bases that induce lower errors. It has been reported that networks with BN layers and ReLUs (rectified linear units) (Glorot et al., 2011) possess the scale-invariance property (Arora et al., 2019) . Thus, the error |v ⊤ y| should be normalized by the scale of y in each layer. Exploiting the fact that ∥y∥ ≤ ∥W ∥ 2 ∥x∥, we normalize it as

|v ⊤ y|/ (∥W ∥ 2 ∥x∥) = σ| cos θ|/∥W ∥ 2 ∈ [0, 1],

where ∥W ∥ 2 is the spectrum norm of W (i.e., the maximum singular value).

Computing the cosine similarities is costly because each layer requires input x over the whole dataset. Therefore, we omit it and simply use the following criterion for selecting the rank:

EQUATION

)

where ℓ is a layer index. This is equivalent to keeping ∥W (ℓ) -W (ℓ) r ∥ 2 F small in each layer. We consider this is a simple but effective criterion for the following reasons. First, Arora et al. (2018) have reported that the subspace spanned by each layer's weight vectors and the subspace spanned by their input vectors both become implicitly low rank and correlated after training. In other words, there should be many small singular values in each layer's weight matrix. Second, the principal directions of the weights are correlated with those of the inputs. Thus, by reducing the bases that correspond to smaller singular values, we can reduce by a large number of ranks without significantly increasing the errors. Moreover, the cosine similarities are expected to be higher for large singular values, meaning that our method can reflect the principal directions of the data distribution even if we only use the singular values of the weight matrices as the criterion.

Complexity-based criterion. We achieve a high compression rate by reducing the rank in layers that have a large number of parameters and multiply-accumulate operations (MACs). For convolutional layers, the numbers of parameters, excluding biases and the MACs, are given by P = K w K h C in C out and M = P HW for a feature map of height H and width W , respectively.

We use the following as a complexity-based criterion:

EQUATION

)

where P (ℓ) and M (ℓ) are the numbers of parameters and the MACs in layer ℓ, respectively. By coupling the above two criteria, we reduce the bases with lower values of

C(ℓ, k) = C 1 (ℓ, k) C 2 (ℓ)

across the entire network. In practice, we compute the criterion for all bases after training. Then, we sort them in ascending order and store as a list. The only necessary step for selection is to reduce the first d bases in the list, where d is determined by the target model size. The algorithm is given in Appendix B.

2.2.2. BN CORRECTION:

As pointed out by Yu et al. (2019) , the means and variances of the BN layers should be corrected when the model size is changed. Suppose that a BN layer is inserted right after the convolutional layer, and that the mean and variance of y

( = W ⊤ x )

are normalized in the BN layer. Then, we should correct those values according to the rank-r approximation of y (i.e., y r ). Because y r = V r S r U ⊤ r x, y r lies in the rank-r subspace spanned by the columns of V r . Hence, letting µ and Σ be, respectively, the population mean and covariance matrix for y, we can exactly compute their projection onto the subspace as

µ r = V r V ⊤ r µ and Σ r = V r V ⊤ r ΣV r V ⊤

r (note that diagonal components are extracted from Σ r for the BN layer). For practical reasons, we compute µ and Σ for each layer after training (Ioffe & Szegedy, 2015) . Because Σ has n(n + 1)/2 extra parameters to store, we keep

V ⊤ R ΣV R instead, where R (1 ≤ R < R)

is the maximum rank to be used, reducing the number of parameters to R( R + 1)/2. At the time of inference, we can correct the mean and variance according to the ranks in each layer. On the other hand, if a list of candidate model sizes is available in advance, we can retain the means and variances for those models as Yu & Huang (2019) . We compare both methods in Section 4.

2.3. LEARNING:

Although our scalable neural networks can operate regardless of learning methods, we propose a method to gain better performance. We simultaneously minimize losses for both the full-rank and the low-rank networks as follows.

min Θ 1 B B ∑ b=1 { (1 -λ)L(D b , W, Θ) + λL(D b , W, Θ) } (3)

Here, L(•) is a loss function, D b is a set of training samples in a mini-batch, B is the batch size, and λ ∈ [0, 1] is a hyperparameter for balancing between the two losses. For this, W = {W (ℓ) } L ℓ=1 , W, and Θ are sets of L weight matrices, their low-rank approximations, and other trainable parameters (e.g., biases), respectively. W is generated from W via SVD and Θ is shared between the fulland low-rank networks. Therefore, the number of total paramters is not increased in learning. As shown in Figure 1 (b), we additionally propagate each mini batch to a low-rank network. Because W (ℓ) r = U (ℓ) r U (ℓ)⊤ r W (ℓ) , the gradient with respect to W (ℓ) can be computed as follows:

EQUATION

)

The gradients for Θ are simply computed from the λ-weighted average for those of both networks.

Since we need a single model that achieve good performance in multiple sizes which are to be selected at the inference phase, we randomly select the model size for the low-rank network at each iteration step. Specifically, a global rank ratio Z is sampled from a uniform distribution U(α l , α u ) with 0 < α l < α u ≤ 1. Then, letting R (ℓ) be the rank of W (ℓ) , we reduce (1 -Z) ∑ L ℓ=1 R (ℓ) bases across the entire network using the criterion mentioned in subsection 2.2.1. In a later section, we experimentally investigate the effects of the parameters λ, α l , and α u in the experiment. 2019) derived the generalization error bound for DNNs under a condition that the trained network has near low-rank weight matrices. They proved that the condition contributes not only to yield a better generalization error bound for the non-compressed network but also to compress the network efficiently. That motivates our approaches: a learning scheme which aims to facilitate the performance of the low-rank networks as well as that of the full-rank network, and a compression scheme which reduces the redundant basis obtained via SVD. 2019) proposed slimmable networks and US-Nets, which are scalable in the width direction. Their works are closely related to ours, but there are differences in some aspects. First, since their methods directly and uniformly reduce the width for every layer, the principal components are not taken into account, and the relative importance of each layer is not considered. Second, for US-Nets in particular, they introduced a "sandwich rule" to keep the performance for an arbitrary width. However, this rule does not guarantee monotonicity of the error with increasing layer width. In the next section, we compare our method with them. "uni", "c 1 ", "c 2 ", and "c 1 c 2 " in the bracket indicate rank selection by a uniform method, by Eq. ( 1), by Eq. ( 2), and by both, respectively. We do not apply "c 2 " to learning because it slightly decreases the full-rank accuracy. 1), and by Eq. (1 & 2), respectively. We do not reduce parameters for the last fully connected layer for the uniform method because it significantly decreases accuracy.

3. RELATED WORK:



Low:



4. EXPERIMENTS:

We evaluate our methods on the image-classification tasks of CIFAR-10/100 (Krizhevsky, 2009) datasets using deep CNNs. The CIFAR-10/100 datasets contain 32 × 32 images for object recognition including 10 and 100 classes, respectively. Each dataset contains 50K images for training and 10K images for validation. We implement our method with TensorFlow (Abadi et al., 2015) .

4.1. ABLATION STUDY:

We test each component in our method on the CIFAR-10 dataset. We use the same baseline setup as in Zagoruyko & Komodakis (2016) , which is summarized in Table 1 in Appendix C. Unless otherwise specified, we report the average result from 5 trials with different random seeds. We adopt a VGG-like network with 15 layers (Zagoruyko, 2015; Liu et al., 2017) 2 , which we refer to as VGG-15 below. Firstly, we evaluate our learning method for various values of the parameters α l and α u , fixing λ = 0.5. Our method requires SVD at each iteration step, which makes it costly. To address this, we suppose that the weight subspaces are not drastically changed at each step and recompute the SVD after every two steps, reusing the results to speed up the training. We illustrate the validation accuracy of a full-rank model for different values of α l (resp., α u ) with α u = 1.0 (resp., α l = 0.01) fixed, on the left (resp., center) of Figure 2 . It can be observed that smaller values of α l and larger values of α u are better. This can be interpreted as indicating that it is better for a full-rank model to learn with various low-rank models than to learn with models biased to a specific range of ranks. Thus, we set α l = 0.01 and α u = 1.0 for the other experiments described below. On the right side of Figure 2 , we show the maximum singular value for each basis index in a full-rank model3 . We can see that our learning method obtains smaller singular values than the baseline. This implies that our learning method has an effect similar to trace-norm regularization (Kliegl et al., 2017) , suggesting that we can suppress the errors produced by reducing the bases.

Next, we evaluated the performance of our inference method for various model sizes. In Figure 3 , we illustrate the inference results for validation data with various number of parameters and MACs.

In the figure, "infer (uni)" indicate the results obtained by uniformly reducing the basis in each layer. Concretely, with a global rank ratio G ∈ (0, 1], we reduce (1 -G)R (ℓ) bases in order from the one corresponding to the smallest singular value. Despite the method being simple, the accuracy changes almost smoothly, and it can be confirmed that there the accuracy scales against changes in the model size. This can be considered as due to the monotonicity of errors, which is formalized in Proposition 2.1. Additionally, the performance is also improved with our learning method by applying uniform rank selection and by using our BN correction. Furthermore, the performance with respect to the parameters is improved when we apply the error-and complexity-based criteria for rank selection to both inference and learning (in the figure, "c 1 " and "c 1 c 2 "). However, the performance with respect to the MACs is dropped by changing the rank selection from uniform ("uni") to error-based ("c 1 "). As shown on the left side of Figure 4 , it is more effective for decreasing MACs to reduce the parameters in shallower layers, which have large feature maps. However, the error-based criterion tends to reduce the parameters in deeper layers because those tend to be low rank. When both criteria are applied (in the figure, "c 1 c 2 "), the performance is also improved for the MACs. We show the rank-selection results for different criteria on the right side of Figure 4 . It can be confirmed that the ranks are decreased for 4, 6, 7, 9, and 10 layers which have large MACs in the case with both criteria ("c 1 c 2 ") relative to the case with only the error-based criterion ("c 1 "). For the BN correction, our method is effective, but it is better with a method that recomputes the means and variances for given ranks ("bnRe"). Because our method is layer-by-layer correction, this probably occurs because our method cannot fully correct for the inter-layer gap, with the statistics of the deep layer changing due to the reduction of rank in the shallow layer.

Additionally, we investigate the effect of a parameter λ. We evaluate the validation accuracy with respect to the number of paramters for λ ∈ {0.1, 0.2, 0.3, 0.4, 0.5} with VGG-15 and ResNet-34 on the CIFAR-10/100 datasets. The results are shown in Figure 7 in Appendix D. We consider that there is a trade-off between the performance of full-and low-rank models, which depends on λ.

4.2. COMPARISON WITH SLIMMABLE NETWORKS:

We compare our method with slimmable networks (Yu et al., 2019) and US-Nets (Yu & Huang, 2019) in terms of performance on the CIFAR-10/100 datasets. We adopt VGG-15 and ResNet-34 (He et al., 2016) . We implement the models based on the Yu's code, available at https:// github.com/JiahuiYu/slimmable_networks (written in PyTorch (Paszke et al., 2017) ). US-Nets is trained with 2 random widths between the lower and upper width and in-place distillation (Yu & Huang, 2019) , then BN-calibration (Yu & Huang, 2019) is applied to each of the slimmable networks and US-Nets after training. For our method, we incorporate all components into the comparisons and adopt BN correction with recomputation. We train the models using λ = 0.4 and the same setup as in the previous subsection. In the following, we report the results for models after the last iteration in training.

First, we compare the scalability of ResNet-34 on the CIFAR-100 dataset. We illustrate the inference results over various numbers of parameters for 5 models trained with different random seeds in Figure 5 . The results in the figure show that US-Nets are unstable, which is a problem for practical use. This instability is because US-Nets do not have monotonic error changes in each layer, a property that our method ensures. Next, we show the results for comparison of VGG-15 on CIFAR-10 and ResNet-34 on CIFAR-100 in Figure 6 . The notations "base (Yu's code)" and "base (our code)" indicate the baseline results obtained by the Yu's code and our code with the same setup. Our baseline is slightly better than the Yu's baseline. We consider this to be due to differences in the framework. Comparing the results with those for VGG-15 on CIFAR-10, our method tends to be more accurate in terms of the number of parameters than in terms of the number of MACs. Since deep layers have more parameters than shallow layers, the rank in deep layers tends to be lower than in shallow layers, resulting in more paramters reduced in deep layers by our method. In contrast, US-Nets reduce the width uniformly across layers, which may contribute to reducing the number of MACs. However, reducing the number of MACs does not necessarily lead to cut the inference cost dominantly, depending on the target device (Yang et al., 2018) . Although we only consider the number of parameters and MACs as the complexity metrics in this paper, other metrics such as memory footprint, memory access cost, and runtime latency should be taken into account for validating the effectiveness in practical use (Tan et al., 2019; Sandler et al., 2018; Dai et al., 2019) .

We can see that the accuracy of our method is lower than that of US-Nets when the compression rate is extremely high. Our method uses SVD and reduces the bases, which means it does not change the number of inputs and outputs (i.e., the in and out dimensionalities). Because the number of parameters in each layer is (m + n)r, it decreases linearly with respect to the rank. US-Nets reduce both input and output dimensionality, meaning that the number of parameters is decreased at a quadratic rate. This makes it easier for US-Nets to achieve extremely high compression. However, our method is better in larger regimes. In particular, for ResNet-34 on CIFAR-100, the performance of slimmable networks and US-Nets on the full-size model are decreased, while our method does not decrease performance much. We illustrate additional comparison results in Figure 8 in Appendix D and give an analysis of per-layer error in Appendix E.

APPENDICES:

A PROOF OF PROPOSITION 2.1

As y = y R by definition, ∥y-y R ∥ 2 = 0. To prove ∥y-y

EQUATION

Here, I n indicates the identity matrix with size n × n.

B AN ALGORITHM FOR RANK SELECTION:



Algorithm 1 Rank selection:

Input: A network with weight matrices W = {W (ℓ) } L ℓ=1 in which W (ℓ) has rank R (ℓ) . Input: A criterion C and the target model size T (e.g., # of parameters and MACs). Output: A set of tuples S which contains indices of layer and basis.

1: S ← ∅ 2: for ℓ = 1, . . . , L do 3:

for k = 1, . . . , R (ℓ) do 4:

Compute C (ℓ, k).

5::

S ← S ∪ {(ℓ, k)}.

6::

end for 7: end for 8: Arrange elements in S in ascending order of C (ℓ, k). (He et al., 2015) for weights, 0 for biases BN ϵ = 1.0 × 10 -5 , momentum = 0.9. 

E AN ANALYSIS OF PER-LAYER ERROR:

We train VGG-15 on the CIFAR-10 dataset with US-Nets (Yu & Huang, 2019) and our method. First, we illustrate learned weights in Figure 9 . For comparison, we show a weight matrix W for US-Nets and its factorized form U S obtained via SVD for our method. The rank of W in the 1st convolutional layer is 27, and thus only 27 convolutional tensors are shown for our method in the upper part of the figure. Since W becomes low-rank after training, there is only a little number of

:

In fact, U(ℓ) r depends on W(ℓ) , but we treat it as constant for simplicity.

Since the VGG-networks are originally designed for classifying the ImageNet dataset(Deng et al., 2009), we use a smaller type than the original for the CIFAR datasets, as used byLiu et al. (2017).

We let σij be a singular value for a basis j in a layer i and then compute maxi (σij). For layers with lower ranks, we simply fill the missing part with zeros.

Figures and Tables:

Figure 1: Illustration of our scalable neural networks. (a) Inference phase of our method. The model can scale according to the target size by changing the rank of a weight matrix in each layer. (b)Learning phase of our method. We explicitly minimizes losses for both of full-and low-rank network, which is designed not only to keep the performance of full-rank network but also to improve multiple low-rank networks (to be used at the inference phase). These schemes can also be applied to CNNs.

Figure 2: Training results for VGG-15 on CIFAR-10. (Left) Full-rank validation accuracy by changing α l with α u = 1.0. (Center) Full-rank validation accuracy by changing α u with α l = 0.01. (Right) Maximum singular value for each basis index in a full-rank model. "base" indicates normal learning as our baseline.

-rank approximation & regularization. Compression methods based on low-rank approximation have been proposed in the literature.Denil et al. (2013);Tai et al. (2016);Ioannou et al. (2016) trained networks after factorizing the weight matrix into a low-rank form.Ioannou et al. (2016) achieved a high compression rate by factorizing a convolutional kernel of K w × K h into K w × 1 and 1 × K h .Denton et al. (2014);Lebedev et al. (2015);Kim et al. (2016) proposed methods that use tensor factorization without rearranging the convolutional tensor into the matrix form.Yu et al. (2017) further improved the compression rate by incorporating sparseness into the low-rank constraint.Zhang et al. (2016);Li & Shi (2018) took resource-related constraints into account to automatically select an appropriate rank. Each of these methods trains a network with pre-defined ranks or compress redundant networks by applying complicated optimizations under a given target size for the model. That is, those methods would require re-training to reconfigure the models for different devices.Kliegl et al. (2017);Xu et al. (2019) utilized trace-norm regularization as a low-rank constraint in learning the network.Wen et al. (2017) proposed a novel method called force regularization to obtain the low-rank weights. The performance of these methods depends on a hyperparamter to adjust strength of regularization. It is difficult to decide on an appropriate range for the hyperparameter in advance, meaning that selection requires trial and error to achieve a particular model size.Scalable neural networks.Chen et al. (2018)  represented the data flow in ResNet-type structures(He et al., 2016) as ordinary differential equations (ODEs), and proposed a Neural-ODEs, which can be used to arbitrarily control the computational cost in the depth direction.Zhang et al. (2019) also obtained scalability in the depth direction by allowing pre-defined intermediate layers to be bypassed.Yu et al. (2019);Yu & Huang (

Figure 3: Inference results for VGG-15 on CIFAR-10. (a) # of parameters vs. validation accuracy.(b) # of MACs vs. validation accuracy. "+learn" indicates results with our learning method. "+bn" and "+bnRe" indicate results with our BN correction and those with recomputation, respectively. "uni", "c 1 ", "c 2 ", and "c 1 c 2 " in the bracket indicate rank selection by a uniform method, by Eq. (1), by Eq. (2), and by both, respectively. We do not apply "c 2 " to learning because it slightly decreases the full-rank accuracy.

Figure 4: Results of rank selection for VGG-15 on CIFAR-10. (Left) Layer-wise importance with complexity-based criterion (normalized to sum 1). (Right) Remaining rank ratio per layer by different selection methods. "uni", "c 1 ", and "c 1 c 2 " indicate selection results by a uniform method, by Eq. (1), and by Eq. (1 & 2), respectively. We do not reduce parameters for the last fully connected layer for the uniform method because it significantly decreases accuracy.

Figure 5: Scalability for ResNet-34 on CIFAR-100 dataset when using (a) US-Nets and (b) our method. We show the results for each of 5 trials with different random seeds in this figure.

Figure 6: Comparisons with slimmable networks (Yu et al., 2019) and US-Nets (Yu & Huang, 2019) for VGG-15 and ResNet-34 on CIFAR-10/100 datasets. (Left) # of parameters vs. validation accuracy. (Right) # of MACs vs. validation accuracy.

9: Delete the first d elements in S to satisfy the model size T . 10: return S C BASELINE SETUP OF EXPERIMENTS ON CIFAR-10/100 DATASETS

Figure 7: The effect of a hyper-parameter for balancing the losses (λ). Validation accuracies are evaluated with λ ∈ {0.1, 0.2, 0.3, 0.4, 0.5} for VGG-15 and ResNet-34 on CIFAR-10 / 100 datasets.

Figure 8: Comparisons with slimmable networks (Yu et al., 2019) and US-Nets (Yu & Huang, 2019) for VGG-15 and ResNet-34 on CIFAR-10/100 datasets. (Left) # of parameters vs. validation accuracy. (Right) # of MACs vs. validation accuracy.

Baseline setup of experiments on CIFAR-10/100 datasets.

Table content: <table><tr><td>Preprocess</td><td>Per-channel standardization (mean, std.)</td></tr><tr><td/><td>CIFAR-10 : (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)</td></tr><tr><td/><td>CIFAR-100: (0.5071, 0.4865, 0.4409), (0.2673, 0.2564, 0.2762)</td></tr><tr><td colspan="2">Data augmentation Random cropping 32 × 32 after zero-padding 4 pixels</td></tr><tr><td/><td>Random horizontal flipping (p = 0.5)</td></tr><tr><td colspan="2">Batch size / Epochs 128 / 200</td></tr><tr><td>Optimizer</td><td>SGD with Nesterov momentum (µ = 0.9)</td></tr><tr><td>Learning rate</td><td>Initialized to 0.1, multiplied by 0.2 at 60, 120, and 160 epochs</td></tr><tr><td>L 2 regularization</td><td>0.0005</td></tr><tr><td>Initializer</td><td>He-Normal</td></tr></table>

Peer review (iclr2020):
This paper propose to compress deep neural network with SVD decomposition, which however has been published 6 years ago in this paper to decompose FC layers: Xue, Jian, Jinyu Li, and Yifan Gong. *Restructuring of deep neural network acoustic models with singular value decomposition.* Interspeech. 2013. For convolutional layers, Tucker decomposition, which is high-order SVD, is apparently a better choice. (Kim et al. (2016)) They should at least compare their method with this one. In their deduction of full-rank-low-rank model joint training, W_r^{(l)} = U_r^{(l)}{U_r^{(l)}}^T W^{(l)} is used without any explanation. Since U_r^{(l)}{U_r^{(l)}}^T = I_r, W_r^{(l)} will be first r rows of W^{(l)} or first r cols of W^{(l)}. It can not be treated as approximation of W^{(l)}. In other words, the foundation of their training is wrong, which makes their experimental results unconvincing. They conduct their experiments with CIFAR-10/100, which is too small for VGG-15 and ResNet-34. A larger dataset would be better. The writing of this paper is not good. For example: 1. In proposition 2.1, what does *y* represent. 2. For Error-based criterion, what is its difference with selecting rank according to singular values? 3. What is P^{(i)} and M^{(i)} in complexity-based criterion? In conclusion, i will give a weak reject.