Scalable Deep Neural Networks via Low-Rank Matrix Factorization
Authors: Atsushi Yaguchi
Taiji Suzuki
Shuhei Nitta
Yukinobu Sakata
Akiyuki Tanizawa
Venue:  (2020)
Keywords: Deep Learning
Deep Neural Networks
Low-Rank Matrix Factorization
Model Compression

Abstract: Compressing deep neural networks (DNNs) is important for real-world applications operating on resource-constrained devices. However, it is difficult to change the model size once the training is completed, which needs re-training to configure models suitable for different devices. In this paper, we propose a novel method that enables DNNs to flexibly change their size after training. We factorize the weight matrices of the DNNs via singular value decomposition (SVD) and change their ranks according to the target size. In contrast with existing methods, we introduce simple criteria that characterize the importance of each basis and layer, which enables to effectively compress the error and complexity of models as little as possible. In experiments on multiple image-classification tasks, our method exhibits favorable performance compared with other methods.

Under review as a conference paper at ICLR 2020
SCALABLE DEEP NEURAL NETWORKS VIA Low- RANK MATRIX FACTORIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Compressing deep neural networks (DNNs) is important for real-world applica- tions operating on resource-constraines devices. However, it difficult to change the model size once the training is completed, which needs re-training figure models suitable for different devices. In this paper, we propose a novel method that enables DNNs to flexibly change their size after training. We factor- ize the weight matrices of the DNNs via singular value decomposition (SVD) and change their ranks according to the target size. In contrast with existing methods, we introduce simple criteria that characterize the importance of each basis and layer, which enables to effectively compress the error and complexity of models little possible. In experiments multiple caage-classification tasks, our method exhibits favorable performance compared with other methods.
con-
INTRODUCTION
As part of the great progress made in deep learning, deep neural network (DNN) models with higher performance have been proposed for various machine-learnnn tasks (LeCun al. 2015). However, these performance improvements require a higher number of parameters and greater computational complexity. Therefore, it is important to compress them without sacrificing the performance for running the models on cnoource-consrringg devices.
Han et al. (016) reduced the memory requirement for devices by pruning and quantizing weight coefficients after training the models. [Hwwrd et al. (2017); Sandler et al. (2018); [Hwwrd et al. (2019) used factorized operations called depth-wise and point-wise convolutions in a proposal for light-weight models suited to mobile devices. However, these methods require pre-defined network structures and pruning the model weights after training. Recently, automated frameworks, such as the so-called neural architecture search (NAS) (Zoph & Le. (017), have been proposed. [Tan et al. (019) proposed a NAS method to accelerate the inference speed on smartphones by incorporating resource-related constraints into the objective function. [Stamoulis al. (2019) significantly reduced the search costs for NAS by applying gradient-based search scheme with superkernel that shares weights for multiple convolutional kernels.
However, the models trained by these methods are dedicated to specific devices, and thus do not possess the ability be reconfigured for use on different devices. In order change the model size, it is necessary to re-train them according to the resources of the target devices. For example, it has been reported that the inference speed when operating the same model on different devices differs according to the computing performance and memory capacity of the hardware accelerator (Ignatov) al. 2018). Therefore, is desirable that the model size can be flexibly changed according the resources of the target devices without re-training the model, which we refer to as scalability in this paper.
To this end, Yu et al. (2099) introduced switchable batch normalization (BN) (loffe & Szegedy) 2013), which switches BN layers according to pre-defined widths, and proposed "slimmable" net- works whose width can be changed after training. Moreover, Yu & Huang 2019) proposed univer- sally slimmable networks (US-Nets) that extend slimmable networks to arbitrary widths. However, since these methods directly reduce the width (i.e., dimensionality) in each layer, the principal com- ponents are not taken into account. In addition, they reduce the width uniformly across all layers, which ignores differences in the importance of different layers.

---

Under review as a conference paper at ICLR 2020
min- batti
Loss (Full)
LS1
Accuracy
G,S.
Larg
Loss
(Low)
paremeters/MACS
forward
backward
mil Law-rank approx.
(a)
(b) Learning Phase
Figure 1: Illustration of our scalable neural networks. (a) Inference phase of our method. The model can scale according the target size by changing the rank of a weight matrix in each layer. (b) Learning phase of our method. We explicitly minimizes losses for both of full- and low-rank net- work, which designed not only keep the performance of full-rank network but also improve multiple low-rank networks (to be used the inference phase). These schemes can also be applied to CNNs.
In this paper, propose novel method that enables DNNs flexibly change their size after training. We factorize a weight matrix in each layer into two low-rank matrices after training the DNNs via singular value decomposition (SVD). By changing the rank in each layer, our method can scale the model to an arbitrary size (Figure Ia)). Our contributions are as follows.
We do not directly reduce the width but instead reduce the redundant basis the column space of the weight matrix, which prevents the feature map in each layer from losing im- portant features.
We introduce simple criteria that characterize the importance each basis and layer, namely, the error- and complexity- criteria. These enable to effectively compress the eтror and complexity of the models as little as possible.
We facilitate the performance of low rank networks with the following methods: learning procedure that simultaneously minimizes losses for both the full and low rank networks (Figure [hb), and the mean & variance correction for each BN layer according to the given rank.
In the experiments on image-classffication tasks of the CIFAR-10/100 (Krizhevsky 2(09 datasets using deep convolutional neural networks (CNNs), our method exhibits better performance for up to approximately 75% compressed models than slimmable networks and US-Nets. In the following, we first describe the details of our method (Section 2 and briefly review related works (Section 3. Then, we give some experimental results (Section 4 and conclude the paper (Section
2
METHODS
In this section, we first give an overview then describe the details of the inference and learning methods.
2.1 OVERVIEW
For layer in the network, lety T x E R" be output vector given by linear transformation of an input vector æ E T with a weight matrix W E muxn where m and are the numbers of input and output nodes, respectively. Let R be the rank of the weight matrix, with < R < min (m, n). Given U (uI,..., uz) m and V (01,..., un) R matrices that have left and right singular vectors (i.e., bases) in columns, and 5 diag (o1...., on) E B Rix composed of singular values in diagonal components, can formulate the SVD W USVT.
matrix
2

---

Under review as a conference paper at ICLR 2020
An example of our scalable neural networks with fully connected layers is shown in Figure [ After the training, each weight matrix in the network is factorized into two matrices of rank R via SVD, and we control this value change the model size. This can be viewed inserting
a sub-layer between the original layers and changing its width R. For the convolutional tensor of kernel width Kw. kernel height Kn, input channels Cin. and output channels Cout, we transform to the matrix form W E R R...K, Cas where m = Kn, Cin and n = Cout. Then, we apply SVD as in Zhang et al. (2016); Wen et al. (2017). This yields two layers with tensor W1 E RRK X xCin X and a tensor W2 RIXIIXXCCox. The number of parameters in each layer becomes (m + n) by this factorization. Thus, we can compress the network to an arbitrary size by changing the rank (s R) within the range S r < mn/ (m + n.).
W E x
Associated with changing the rank, the monotonicity of approximation error holds for each layer.
Proposition 2.1. Let Wr V, be a rank-r approximation using the truncated-SVD for W and let y, = W x. Then, the squared enror between an original y and yr satisfies that y yr ly yr+1ll 2 ... yy yR
ly y1
The proof is given in Appendix A According the above, errors between an original output y and its approximation y, monotonically decrease as the rank increases. Hence, it can be expected the performance of the entire network will scale with the model size, which is controlled by the rank in our method.
2.2 INFERENCE
2.2.
RANK SELECTION
Given a target size for model, we select the rank of each layer by reference to the following crite- ria.
Error-based criterion, According to Eq. (5 in Appendix A the error associated with a rank- decrease is given by |u y = ol/ x| = ||||| cos 0j. This implies that the error depends on the singular value and the cosine similarity between input vector æ and the corresponding left sin- gular vector u. Based on this, we consider how to compress the model with as little error as possible by reducing the bases that induce lower errors. It has been reported that networks with BN layers and ReLUs (rectified linear units) (Glorot et al.) 2011) possess the scale-invariance property (Arora ctt al. 2019). Thus, the error Iv y should be normalized by the scale of y in each layer. Exploiting the fact that yyll I [ ær|, we normalize as o / (I) W ||2||=]]] = o cos 0) [ E [0, 1], 2l
where || [W|]2 is the spectrum norm of W (i.e., the maximum singular value).
Computing the cosine similarities is costly because each layer requires input æ over the whole dataset. Therefore, omit it and simply the following criterion for selecting the rank:
C. (1 = o
112
(1)
where é is layer index. This equivalent to keeping ||(0 !0 small in each layer. We consider this is a simple but effective criterion for the following reasons. First, Arora et al. (2018) have reported that the subspace spanned by each layer's weight vectors and the subspace spanned by their input vectors both become implicitly low rank and correlated after training. In other words, there should be many small singular values in each layer's weight matrix. Second, the principal directions of the weights are correlated with those of the inputs. Thus, by reducing the bases that correspond to smaller singular values, we can reduce by a large number of ranks without significantly expected to be higher for large singular
increasing the errors. Moreover, the cosine similarities values, meaning that our method can reflect the principal directions the data distribution even only use the singular values of the weight matrices the criterion.
Complexity-based criterion. We achieve high compression rate by reducing the rank in lay- that have a large number of parameters and caltiply-accummuatt operations (MACs). For convolutional layers, the numbers of parameters, excluding biases and the MACs, are given by P K,, K₁, Cin Cout and M PHW for feature map of height H and width W, respectively.
3

---

Under review as a conference paper at ICLR 2020
We use the following as complexity saased criterion:
C2( = (1 (1- Σ p(o)) >/1t) (1-M119 MNOO Σ Earco)
p(()
(2)
where p(() and ( are the numbers of parameters and the MACs in layer L, respectively. By coupling the above two criteria, we reduce the bases with lower values of C(e, k) = C1 (e, (2())
across the entire network. In practice, we compute the criterion for all bases after training. Then,
we sort them in ascending order and store list. The only necessary step for selection is to reduce the first d bases in the list, where d is determined by the target model size. The algorithm is given in Appendix B
2.2.2 BN CORRECTION
As pointed out by Yu et al. (2019). the means and variances of the BN layers should be corrected when the model size changed. Suppose that a BN layer is inserted right after the convolutional layer, and that the mean and variance of y (= W x) are normalized in the BN layer. Then, we should correct those values according to the rank-r approximation of (i.e., 9r). Because yr- = V,S.U, x. yr lies in the rank-r subspace spanned by the columns of V,. Hence, letting l and Σ be, respectively, the population mean and covariance matrix for y. projection onto the subspace as ,,, V, V.V.T μ. and E, V,V. VV,V, (note that diagonal components are extracted from Σ, for the BN layer). For practical reasons, we compute μ and E for each layer after training (Ioffe & Szegedy (015).. Because Σ has nın + 1)/2 extra parameters to R R) is the maximum rank be used, reducing the number of parameters to R(R + 1)/2. At the time of inference, we can correct the mean and variance according to the ranks in each layer. On the other hand, if list of candidate model sizes is available in advance, we can retain the means and variances for those models as Yu & Huang (2019). We compare both methods in Section 4
exactly compute their
store, we keep Vi VVg instead, where R
2.3 LEARNING
Although our scalable neural networks operate regardless of learning methods, we propose method to gain better performance. We simultaneously minimize losses for both the full-rank and the low-rank networks follows.
1 min B hm1
(3)
Here, L(-) is a loss function, D3 is a set of training samples in a mini-batch, B is the batch size, and E [0, 1] is a hyperparameter for balancing between the two losses. For this, W (#) W. and θ are sets of weight matrices, their low-rank approximations, and other trainable parameters (e.g., biases), respectively. W is generated from W via SVD and θ is shared between the full- and low-rank networks. Therefore, the number of total paramters is not increased in learning. As shown in Figure [bb.. we additionally propagate each mini batch to low-rank network. Because ( = U( T W( the gradient with respect to ( can be computed as follows: ||
(a-)))
B
aC(Do, W,O) + AUT TT OL(D,, W.O) O
(1-
(4)
B
aw!!)
The gradients for θ are simply computed from the X-weighted average for those of both networks.
Since we need a single model that achieve good performance in multiple sizes which are to be selected at the inference phase, we randomly select the model size for the low-rank network at each iteration step. Specifically, a global rank ratio Z is sampled from a uniform distribution U(aa, a.) with 0 < < Oa 1. Then, letting R(() be the rank of (( we reduce (1 2) Ete R(O) bases across the entire network using the criterion mentioned in subsection 2.2.1 later section,
experimentally investigate the effects of the parameters X, on, and an in the experiment.
In fact, U depends ( but we treat as constant for simplicity.
4

---

Under review as a conference paper at ICLR 2020
0.936
0.36
0.9
0.8
Veldatton RccuRA
0.922
0.922
0.7
0.93
0.93
10.
0928
0.01
0928
0.9
Snggler
0.5
base
0.3
0.7
04
0.24
0.222
0.922
base
baso
0.92
0.92
250
100
200
300
400
500
Fpoch
Fpoch
Basis
Figure 2: Training results for VGG-15 on CIFAR-IO. (Left) Full-rank validation accuracy by chang- ing ar with au 1.0. (Center) Full-rank validation accuracy by changing Oy with oy 0.01. (Right) Maximum singular value for each basis index in a full-rank model. "base" indicates normal learning as our baseline.
Arora et al. (2018); Suzuki (2099) derived the generalization error bound for DNNs under a con- dition that the trained network has near low-rank weight matrices. They proved that the condition contributes not only to yield a better generalization error bound for the non-compressed network but also to compress the network efficiently. That motivates our approaches: a learning scheme which aims to facilitate the performance of the low-rank networks as well as that of the full-rank network, and a compression scheme which reduces the redundant basis obtained via SVD.
3 RELATED WORK
Low-rank approximation & regularization. Compression methods based on low-rank approxi- mation have been proposed in the literature. Denil et al. (013); [Ta et al. 62016; Toannou et al. 2016 trained networks after factorizing the weight matrix into low-rank form. Toannou et al. 2016) achieved high compression rate by factorizing convolutional kernel of K X K₁ into Kn X 1 and 1 K₁,- Denton et al.
(2015): Kim et al. (2016) proposed meth- ods that use tensor factorization without rearranging the convolutional tensor into the matrix form. Yu al. (2017) further improved the compression rate by incorporating sparseness into the low-rank constraint. Zhang et al. (2016); Li & Shi (2018) took resource-related constraints into account to automatically select an appropriate rank. Each these methods trains a network with pre-defined ranks compress redundant networks by applying complicated optimizations under given target size for the model. That is, those methods would require re-training to reconfigure the models for different devices.
Kliegl et al. (2017); ; Xu et al. (019) utilized trace-norm regularization as a low-rank constraint in learning the network. (Wen et al. (201)) proposed novel method called force regularization obtain the low-rank weights. The performance of these methods depends on a hyperparamter to adjust strength of regularization. It difficult to decide on an appropriate range for the hyperparameter in advance, meaning that selection requires trial and error to achieve particular model size.
Scalable neural networks. Chen al. 2018 represented the data flow in ResNet-type struc- tures (He et al., 2016) as ordinary differential equations (ODEs), and proposed a Neural-One ODEs, which can be used arbitrarily control the computational cost in the depth direction. Zhhang et al.) 2019) also obtained scalability in the depth direction by allowing pre-defined intermediate layers to be bypassed.
Yu et al.] (2019); Yu & Huang 2019) proposed slimmable networks and US-Nets, which are scal- able in the width direction. Their works are closely related to ours, but there are differences in some aspects. First, since their methods directly and uniformly reduce the width for every layer, the principal components are not taken into account, and the relative importance of each layer is not considered. Second, for US-Nets in particular, they introduced a "sandwich rule" to keep the performance for an arbitrary width. However, this rule does not guarantee monotonicity of the error with increasing layer width. In the next section, compare method with them.
5

---

Under review as a conference paper at ICLR 2020
6.0
0.85
infer (un))
infor (uni)
infer (un)) icam (unl)
Infer (unt)) learn (un))
infer (ur))+leam (uni++on
infer (un)+ (nn)+eemm (um)++bn
inter (c, H,laa (un)
(Ka)ra [un)]
435
(c:c]]+lemm [un)] -n
0.75
(GrC))-kann J-lem (un])ean
(c (0,02) (e
(CyCg) (on
27
07
50
303
150
200
250
GLI depranetesos val XXL
(1) MACS vol soc.
Figure 3: Inference results for VGG-15 on CIFAR-10. (a) of parameters vs. validation accuracy. (b) # of MACs vs. validation accuracy. "+learn" indicates results with our learning method. "+bn" and "+hnRe" indicate results with our BN correction and those with recomputation, respectively. "uni", "c,", "c2", ", and "cyez" in the bracket indicate rank selection by uniform method, by Eq. (. by Eq. (2, and by both, respectively. We not apply "02" to learning because it slightly decreases the full-rank accuracy.
3.08
uni
0.075
Jemsttinggee makk rali
0.8
0.07
0.6
0.065
0.4
0.06
0.055
2
10
8
10
11
13
14
15
Layer
Layer
Figure 4: Results of rank selection for VGG-15 on CIFAR-10. (Left) Layer-wise importance with complexity-bene -based criterion (normalized to sum 1). (Right) Remaining rank ratio per layer by differ- ent selection methods. "uni". "c1" and "cicg" indicate selection results by uniform method, by Eq. (. and by Eq. & 2), respectively. We do not reduce parameters for the last fully connected layer for the uniform method because significantly decreases accuracy.
EXPERIMENTS
We evaluate our methods on the image-classffication tasks of CIFAR-10/100 (Krizhessky 2009) datasets using deep CNNs. The CIFAR-10/100 datasets contain 32 32 images for object recogni- tion including 10 and 100 classes, respectively. Each dataset contains 50K images for training and 10K images for validation. We implement our method with TensorFlow (Abadi al. 2015).
4.1 ABLATION STUDY
We test each component in our method on the CIFAR-10 dataset. We use the same baseline setup as in Zagoruyko & Komodakis (2016), which is summarized in Table [ in Appendix C Unless otherwise specified, we report average result from trials with different random seeds. We adopt VGG-like network with 15 layers (Zagoruyko) 2015 Liu et al. 2017 B which we refer to as VGG-15 below.
Firstly, evaluate learning method for various values of the parameters aj and O... fixing = 0.5. Our method requires SVD at each iteration step, which makes costly. To address this, suppose that the weight subspaces not drastically changed at each step and recompute the SVD after every two steps, reusing the results to speed up the training. We illustrate the validation accuracy of full-rank model for different values of aq (resp., a.)) with a,, 1.0 (resp., on 0.01) fixed, on the left (resp., center) of Figure 2 It can be observed that smaller values of o, and larger values of ag are better. This can be interpreted as indicating that it better for full-rank model
Sinne the VGG-networks originally designed for classifying the ImageNet dataset (Deng et al. 2009 smaller type than the original for the CIFAR datasets, as used by [ al. (017).

---

Under review as a conference paper at ICLR 2020
0.78
078
0.77
0.7
Valiaiioaatouu atouuoo
0.76
0.76
0.75
0.75
0.74
0.4
0.73
0.3
0.22
0.22
0.11
0.1
Params (M)
Params (M)
(a) parameters val. (US-Niets)
parameters vol. (Ours)
Figure 5: Scalability for ResNet-34 on CIFAR-100 dataset when using (a) US-Nets and (b) our method. We show the results for each of 5 trials with different random seeds in this figure.
to learn with various low-rank models than to learn with models biased to a specific range of ranks. Thus, we set 0q = 0.01 and a, = 1.0 for the other experiments described below. On the right side of Figure we show the maximum singular value for each basis index in full-rank model We can see that our learning method obtains smaller singular values than the baseline. This implies that our learning method has an effect similar to trace-norm regularization (Kliegl et al., 2017), suggesting that we can suppress the errors produced by reducing the bases.
Next, we evaluated the performance of our inference method for various model sizes. In Figure 3 we illustrate the inference results for validation data with various number of parameters and MACs. In the figure. "infer (uni)" indicate the results obtained by uniformly reducing the basis in each layer. Concretely, with a global rank ratio G E (0, 1). we reduce (1 (1-G), G) bases in order from the one corresponding to the smallest singular value. Despite the method being simple, the accuracy changes almost smoothly, and it can be confirmed that there the accuracy scales against changes in the model size. This can be considered as due to the monotonicity of errors, which is formalized in Proposition 2.1 Additionally, the performance also improved with our learning method by applying uniform rank selection and by using our BN correction. Furthermore, the performance with respect to the parameters is improved when we apply the error- and complexity-based criteria for rank selection to both inference and learning (in the figure, "c1" and "cicz"). However, the performance with respect to the MACs is dropped by changing the rank selection from uniform ("uni") to error-based ("c, "). As shown on the left side of Figure[4 it is more effective for decreasing MACs to reduce the parameters in shallower layers, which have large feature maps. However, the error-based criterion tends to reduce the parameters in deeper layers because those tend be low rank. When both criteria are applied (in the figure, "CiC2"), "), the performance is also improved for the MACs. We show the rank-selection results for different criteria on the right side of Figure It can be confirmed that the ranks are decreased for 4, 6, 7, 9, and 10 layers which have large MACs in the case with both criteria ("c1c=") relative to the case with only the error-based criterion ("c, "). For the BN correction, our method is effective, but it is better with a method that recomputes the means and variances for given ranks ("bnRe"). Because our method is lnyerbby-layer correction, this probably occurs because our method cannot fully correct for the inter-layer gap, with the statistics of the deep layer changing due to the reduction of rank in the shallow layer.
Additionally, we investigate the effect of parameter X. We evaluate the validation accuracy with respect to the number paramters for E {0.1, 0.3, 0.4, 0.5) with VGG-15 and ResNet-34 on the DIFAR-10/100 datasets. The results are shown in Figure in Appendix [ We consider that there is a trade-off between the performance of full- and low-rank models, which depends on X.
4.2 COMPARISON WITH SLIMMABLE NETWORKS
We compare our method with slimmable networks (uu et al. 2019) and US-Nets (Yu & Huang. 2019) in terms of performance on the CIFAR-10/100 datasets. We adopt VGG-15 and ResNet-
34 (He et al. 2016). We implement the models based on the Yu's code, available at httpps: gi rithub. com/ Jiahui s immable networks (written in PyTorch (Paszke al. (017)).
We let Tiy be singular value for a basis in layer and then compute max; (o.)). ). For layers with lower ranks, we simply fill the missing part with zeros.
7

---

Under review as a conference paper at ICLR 2020
VGG-15 CIRA-110
VGS 25 on CPAR 10
base codee
base (lck
Slimmmtti
Slimmable
mam LS
-en US
ours
2015
base lout codo)
base tour code!
MAC P4
Ran on CITAR-100
Bendlet-44 on CIFAR-100
base (ri'' codk)
(Yuls code)
-Simmable
-Simmable
US
cure
coda)
00L
ours
Figure 6: Comparisons with slimmable networks Yo et al 2019) and US- (Yu & Huang 2019) for VGG-15 and ResNet-34 on CIFAR-10/100 datasets. (Left) # of parameters validation
accuracy. (Right) # of MACs vs. validation accuracy.
US-Nets is trained with random widths between the lower and upper width and in-place distil- lation (Yu & Huang 2019), then BN-calibration YYu & Huang 2019) is applied to each of the slimmable networks and US-Nets after training. For our method, we incorporate all components into the comparisons and adopt BN correction with recomputation. We train the models using À 0.4 and the same setup as in the previous subsection. In the following, we report the results for models after the last iteration in training.
First, we compare the scalability of ResNet-34 on the CIFAR- 100 dataset. We illustrate the inference results over various numbers of parameters for 5 models trained with different random seeds in Figure The results in the figure show that US-Nets are unstable, which is a problem for practical use. This instability is because US-Nets do not have monotonic error changes in each layer, property that our method ensures. Next, we show the results for comparison of VGG-15 on CIFAR- 10 and ResNet-34 on CIFAR-100 in Figure G The notations "base (Yu's code)" and "base (our code)" indicate the baseline results obtained by the Yu's code and our code with the same setup. Our baseline is slightly better than the Yu's baseline. We consider this to be due differences in the framework. Comparing the results with those for VGG-15 on CIFAR-10, our method tends to be more accurate in terms of the number parameters than in terms of the number of MACs. Since deep layers have more parameters than shallow layers, the rank in deep layers tends to be lower than in shallow layers, resulting in more paramters reduced in deep layers by our method. In contrast, US-Nets reduce the width uniformly across layers, which may contribute to reducing the number of MACs. However, reducing the number of MACs does not necessarily lead to cut the inference cost dominantly, depending on the target device (Yang et al., (2018). Although we only consider the number of parameters and MACs as the complexity metrics in this paper, other metrics such as memory footprint, memory access cost, and runtime latency should be taken into account for validating the effectiveness in practical use (Tan et al. 2019 Sandler et al. 20188] 8 Dai et al.) 2019).
We can see that the accuracy of our method is lower than that of US-Nets when the compression rate is extremely high. Our method uses SVD and reduces the bases, which means it does not change the number of inputs and outputs (i.e., the in and out dimensionalities). Because the number of parameters in each layer is (m + n)r, it decreases linearly with respect to the rank. US-Nets reduce both input and output dimensionality, meaning that the number of parameters is decreased at a quadratic rate. This makes it easier for US-Nets to achieve extremely high compression. However, our method is better in larger regimes. In particular, for ResNet-34 34 on CIFAR- 100, the performance of slimmable networks and US- Ne-Nets on the full-size model are decreased, while method does not decrease performance much. We illustrate additional comparison results Figure K in Appendix [ and give analysis of per-layer emor in Appendix F
8

---

.w.g.o///..ro..ooooooo....... Software available from Under review as a conference paper at ICLR 2020
5 CONCLUSIONS
We proposed novel method that enables DNNs to flexibly change their size after training. We used to factorize the weight matrix for each layer into two low-rank matrices after training the DNNs. By changing the rank in each layer, our method can scale the model to an arbitrary size. We introduced simple criteria for characterizing the importance of each basis and layer; these are the emor- and complexity introducing much error. In experiments on multiple method exhibited good performance relative that of other methods.
criteria. Those criteria enabled effectively compressing models without tasks using deep CNNs, our
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving. Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shiens, Benoit Steiner, Ily Sutskever, Kunal Talwar, Paul Tucker, Vin- cent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Watten- berg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. tensorflow.org.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via compression approach. In International Conference on Machine Learning (ICML), pp. 2018.
Sanjeev Arora, Zhiyuan Li, and Kaifeng Lyu. Theoretical analysis of auto rate-tuning by batch normalization. In International Conference on Learning Representations (ICLR), 2019.
Tian Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differen- tial equations. In Advances in Neural Information Processing Systems (NeurIPS), pp. 6571-6588 2018.
Xiaoliang Dai, Peizhao Zhang, Bichen Wu, Hongxu Yin, Fei Sun, Yanghan Wang, Marat Dukhan, Yunqing Hu, Yiming Wu, Yangqing Jia, Peter Vajda, Matt Uyttendaele, and Niraj K. Jha. Cham Net: Towards efficient network design through Mattorm--waa model adaptation. In JEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 11398-11400 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale Hicrarchical Image Database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Misha Denil, Babak Shakibi, Laurent Dinh, Marc' Aurelio Ranzato, and Nando de Freitas. Predicting parameters in deep learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 2148-2156. 2013.
Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in Neural Informa- tion Processing Systems (NeurIPS), pp. 1269-1277. 2014.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In International Conference on Artificial Intelligence and Staristics (AISTATS), pp. 315-323, 2011.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural net- work with, trained quantization and huffman coding, In International Conference on Learning Representations (ICLR), 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In IEEE International Conference on Computer Vision (ICCV), pp. 1026-1034, 2015.

---

Under review as a conference paper at ICLR 2020
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chiee Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoe V. Le, and Hartwig Adam. Searching for MobileNetV3. IEEE International Conference on Computer Vision (ICCV), 2019. 2019.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.04861, 2017.
Andrey Ignatov, Radu Timofte, William Chou, Ke Wang, Max Wu, Tim Hartley. and Luc Van Gool. AI benchmark: Running deep neural networks android smartphones. arXiv preprini arXiv:1810.01109, 2018.
Yani loannou, Duncan Robertson, Jamie Shotton, Roberto Cipolla, and Antonio Criminisi. Training CNNs with low-rank filters for efficient image classification. In International Conference on Learning Representations (ICLR), 2016.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), pp- 448 456, 2015.
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taclim Choi, Lu Yang, and Dongjun Shin. Com- pression of deep convolutional neural networks for fast and low power mobile applications. In International Conference on Learning Representations (ICLR), 2016.
Markus Kliegl, Siddharth Goyal, Kexin Zhao, Kavya Srinet, and Mohammad Shoeybi. Trace norm regularization and faster inference for embedded speech recognition RNNs. arXiv preprint arXiv:1710.09026 2017.
A. Krizhevsky. Learning multiple layers of features from tiny images. Master' s thesis, Department of Computer Science, University of Toronto, 2009.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. In International Conference on Learning Representutions (ICLR), 2015.
Yann LeCun, Yoshua Bengio, and Geoffrey E. Hinton. Deep learning. Nature, 521:436 2015.
Chong Li and C. J. Richard Shi. Constrained optimization based low-rank approximation of deep neural networks. In European Conference Computer Vision (ECCV), pp. 746-761, 2018.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In IEEE International Confer- ence on Computer Vision (ICCV), pp. 2755-2763, 2017.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bileNetV2: Inverted residuals and linear bottlenecks. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 5510-45200 2018.
Dimitrios Stamoulis, Ruizhou Ding, Di Wang. Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Single-Path NAS: Designing hardware-efficient convnets in less than hours. arXiv preprint arXiv: 00tiv:9904.0220 02877, 2019.
Taiji Suzuki. Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network. arXiv preprint arXiv: 1909. 1274, 2019.
10

---

Under review as a conference paper at ICLR 2020
Cheng Tai, Tong Xiao, Yi Zhang, XiaogangWang. and Weinan E. Convolutional neural networks with lowrank regularization. In International Conference on Learning Representations (ICLR), 2016.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. MnasNet: Platform-aware neural architecture search for mobile. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Wei Wen, Cong Xu, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Coordinating filters for faster deep neural networks. In IEEE International Conference on Computer Vision (ICCV) pp- 658-666, 2017.
Yuhui Xu, Yuxi Li, Shuai Zhang, Wei Wen, Botao Wang, Wenrui Dai, Yingyong Qi, Yiran Chen, Weiyao Lin, and Hongkai Xiong. Trained rank pruning for efficient deep neural networks. In Workshop on Energy Efficient Machine Learning and Cognitive Computing (EMC*), 2019.
Tien-Ju Yang, Andrew Howard, Bo Chen, Xiao Zhang, Alec Go, Mark Sandler, Vivienne Sze, and Hartwig Adam. NetAdapt: Platform-awan neural network adaptation for mobile applications. In 2018.
European Conference on Computer Vision (ECCV), pp.
Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques.
IEEE International Conference on Computer Vision (ICCV), 2019.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In International Conference on Learning Representations (ICI.R, 2019.
Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 67-76, 2017.
Sergey Zagoruyko. 92.45% on cifar-10 torch, 2015. URL Ut/hhtp::///orbb.c///o/////01111 07 07/30/cifar html
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In the British Machine Vision Conference (BMVC), 2016.
Linfeng Zhang, Zhanhong Tan, Jiebo Song, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. SCAN: A scalable neural networks framework towards compact and efficient models. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38 (8(10):19) 1903-)9999499999 2016.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In Interna- tional Conference on Learning Representationn (ICLR), 2017.

---

Under review as a conference paper at ICLR 2020
APPENDICES
A PROOF OF PROPOSITION 2.1
As y yR by definition, 1/ yRl To prove 1/y- yi|l ly y ||
y y y-
yy yn 1 show that yy y- ||2 y yr+1
for
R I.
Proof.
11 - yr 1y yr+1 22= = [1 - V.V, y1 (I, V,+ v++1) y/1
=y² (In VV,, y yn (I) Vi V V.+1)
yy (Vr+1 V 1+1 VVV,') y
"y' (0++10,41)
(M,+1 2
(5)
Here, Jn indicates the identity matrix with size
B AN ALGORITHM FOR RANK SELECTION
Algorithm Rank selection
Input: A network with weight matrices W (W in which ( has rank R(O)
Input: A criterion C' and the target model size T (e.g., # of parameters and MACs).
Output: A set of tuples which contains indices of layer and basis.
1: - 0
2: for (or(=10
do
3: for k = 1, ..... R do
4: Compute kL,
5: S I-SU {(),
6: end for
7: end for
8: Arrange elements in S in ascending order of C' (E, k).
9: Delete the first elements satisfy the model size T.
10: return $
C BASELINE SETUP EXPERIMENTS ON CIFAR-10/100 DATASETS
Table 1: Baseline setup of experiments on CIFAR- 10/100 datasets.
Preprocess
Per-channel standardization (mean, std.)
10 (0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616) CIFAR- 100: (0.5071, 0.4865, 0.4409), (0.2673, 0.2564, 0.2762) Random cropping 32 32 after zero-padding 4 pixels Random horizontal flipping (p = 0.5)
Data augmentation
Batch size / Epochs
128 200
Optimizer
SGD with Nesterov momentum (μ1 = 0.9)
Learning rate
Initialized 0.1, multiplied by 0.2 at 60, 120, and 160 epochs
Ly regularization
0.0005
Initializer
He-Normal (H al. 2015) for weights, for biases
BN GPUs
c = 1.0 x 10-5 momentum = 0.9. Initialize 7 = 1 and / = 0
12

---

Under review as a conference paper at ICLR 2020
D ADDITIONAL RESULTS FOR VGG-15 AND RESNET-34 ON CIFAR DATASETS
VGG-15 on CIFAR-10
ResNett34 on CIFAR-10
2305
Wild
0.55
092
0.2
6
10
6
12
15
VGG-15 CIFAR-100
ResNiet-34 on CIFAR-100
0.73
035
Valdation
07
034
Pacaro (M)
Parama (M4
Figure 7: The effect of a haper-yppaaeee parameter for balancing the losses (A). Validation accuracies are evaluated with À (0.1,0.2,00 0.3, 0.4, 0.5) for VGG-15 and ResNet-34 CIFAR-10/ 100 datasets.
VGG-15 on CIFAR-100
VGG-15 on CIFAR-100
(Ya's coda)
codel
Simmabie
Simmabbe
US
US
Gurs
base
code)
RecNeet 34 on CIFAR-10
Res 34 on CIFAR-10
0.40
0333
valluate
code)
Stimmbbl
Stimmbble
0X0
base four code:
base (our code)
Figure 8: Comparisons with slimmable networks Yu ct al. 2019) and US-Nets (Yu & Huang 2019) for VGG-15 and ResNet-34 on CIFAR-10/100 datasets. (Left) # of parameters vs. validation accuracy. (Right) # of MACs vs. validation accuracy.
E AN ANALYSIS OF PER-LAYER ERROR
We train VGG-15 the CIFAR-10 dataset with US-Nets (Yu & Huang 2019) and our method. First, we illustrate learned weights in Figure 9 For comparison, we show a weight matrix W for US-Nets and its factorized form US obtained via SVD for method. The rank of W in the 1st convolutional layer is 27, and thus only 27 convolutional tensors are shown for our method in the upper part of the figure. Since W becomes low-rank after training, there is only little number of
13

---

Under review as a conference paper at ICLR 2020
dominant bases which correspond to higher singular values in S. For US-Nets in particular, see that weight coefficients of large absolute values are concentrated in lower channels. It can be considered that US-Nets implicitly attract important features into lower channels because US-Nets reduce channels in order from the one with higher indices. Therefore, it can be expected the errors induced by reducing channels partially suppressed for US-Nets..
Next, we compute the sum of squared error: Σ Y IO || 1 over the validation dataset, where N, Y and y are the number of validation samples, an output feature map layer é for the full-size network, and that for the compressed network, respectively. With compressing the entire network from 100% to about 5% in terms of the total number of parameters, we compute the squared eπror with respect to the number of parameters in each layer as shown in Figure [0] For our method, we reduce the bases of weight marices (i.e., columns in U g()) according to the criterion Ci(e, k) C2({) and compute (+) with low-rank weight matrix W in each layer. For US-Nets, since the dimensionality of Y is decreased with reducing channels for every layer, we fill the missing part with zeros. It can be confirmed the squared errors by our method are suppressed more than that by US-Nets in Figure [0] We consider this is because method do not directly reduce the channel as in US-Nets but instead reduce the redundant basis in the column space of the weight matrix, which prevents the feature map in each layer from losing important features (i.e., principal components).
outputt aaTT
SpUUNO
Wil
3305 (buseline)
Will e x xox33x0 (US Nets)
tensorize (()))
B53385K27 (aurs)
output
YOOU 11gut cremiiss
W00 R3124512 (baseline)
Will)) gRizKoLz (US-Neta)
(1 g
22122312 (ours)
Figure 9: Visualizations of weight coefficients of VGG-15 trained on the CIFAR- 10 dataset. (Upper) Weight tensors in the 1st convolutional layer. (Lower) Weight matrices in the 14th fully connected layer. "baseline" indicates normal learning.
Layer
Layer 48
Layer 01A
05
0.5
Soardd
Squard
www.iilS
0.25
1003
100000
200000
permenter liyert
#f parmreten Or layer
parmessees O4 layed
Figure 10: # of parameters vs. the sum of squared error for the 1st, 8th, and 14th layers of VGG-15 trained on the CIFAR-10 dataset.
14


Reviews:
[{"review_id": "r1eS584AYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to reshape the weights of the layers of deep neural networks and parametrize them with a low-rank matrix decomposition (SVD). The rank is optimized using two criterion (error-based and complexity-based). Since the decomposition is applied post-hoc, the authors propose to correct the parameters of the batch norm analytically. The authors propose to jointly optimize a loss on the full network and the low-rank version. Experiments are done on CIFAR 10 an 100 with a VGG-15 and ResNet-34 architecture.                                                                                           \n\nSome of the ideas are interesting and would be worth developping further. However, the paper in the current state cannot be accepted for the following reasons: (1) the novelty is low, this very type of decomposition is already widely studied (2) the paper is not clear as to what the contributions are, and why they are justified, theoretically or empirically, (3) the review and comparison with the state-of-the-art is lacking and (4) the experimental setup is simplistic and not convicing. (5) overall the paper is imprecise.\n\n\nMain comments\n\n* Applying SVD to the matricized weights of deep neural networks is not new. Actual contributions need to be separated from existing works.\n* The related word needs to be reviewed. Many references are missing. In particular, the proposed method could be considered as a special case of tensor based methods.  \n* The references that *are* listed in the related work are not properly reviewed: the authors aim to not compare with them claiming that they require re-training. Lebedev et al provide a method that works both for end-to-end training or post-hoc, by applying tensor decomposition to the trained weights. Fine-tuning is optional and done to recover performance.\n* How was the complexity-based criterion obtained? Why use both M and P, since M includes P? How do the proposed criterion compare to simple measures, e.g. explained variance?\n* The authors should compare with other compression techniques: layer-wise compression (e.g. Lebedev et al, Speeding-up convolutional neural networks using fine-tuned cp-decomposition, ICLR 2015) or full network compression (e.g. Kossaifi et al, T-Net: Parametrizing Fully Convolutional Nets with a Single High-Order Tensor, CVPR 2019).\n\nMissing experiments\n\n* The proposed learning loss needs to be compared with the original one to demonstrate any potential performance improvement. Currently, it is not clear whether it is actually helping. In other words, there should also be a comparison with \\lambda = 1 or 0.\n* Does the proposed loss have an effect on the selected rank? On the actual rank of the weights? On the distribution of the eigenvalues?\n* The BN correction needs to be experimentally motivated: since the network is trained with a loss that incorporates the low-rank network, is that needed? Does the propose loss affect performance? How does performance change with and without that BN correction?\n* Experiments on CIFAR 10-100 is not sufficient to be convincing, the authors should ideally try a more realistic, large scale dataset, e.g. ImageNet.\n* VGG-15 is not convincing to show-case compression, as more than 80% of the parameters are in the fully-connected layer\n* The authors assume that the SVD decomposition of the weights does no change significantly at each step: is there any empirical evidence supporting this assumption? This most likely depends on the experimental setup (batch-size, learning rate, etc).\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Skx9EjdpKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper propose to compress deep neural network with SVD decomposition, which however has been published 6 years ago in this paper to decompose FC layers:\nXue, Jian, Jinyu Li, and Yifan Gong. \"Restructuring of deep neural network acoustic models with singular value decomposition.\" Interspeech. 2013.\nFor convolutional layers, Tucker decomposition, which is high-order SVD, is apparently a better choice. (Kim et al. (2016)) They should at least compare their method with this one.\n\nIn their deduction of full-rank-low-rank model joint training,  W_r^{(l)} = U_r^{(l)}{U_r^{(l)}}^T W^{(l)} is used without any explanation.  Since U_r^{(l)}{U_r^{(l)}}^T = I_r, W_r^{(l)} will be first r rows of W^{(l)} or first r cols of W^{(l)}.  It can not be treated as approximation of W^{(l)}.  In other words, the foundation of their training is wrong, which makes their experimental results unconvincing.\n\nThey conduct their experiments with CIFAR-10/100, which is too small for VGG-15 and ResNet-34.  A larger dataset would be better.\n\nThe writing of this paper is not good. For example:\n1. In proposition 2.1, what does \"y\" represent.\n2. For Error-based criterion, what is its difference with selecting rank according to singular values?\n3. What is P^{(i)} and M^{(i)} in complexity-based criterion?\n\nIn conclusion, i will give a weak reject.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgsbqjy5r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to modify the computing requirements of a trained model without compromising the accuracy and without the need for retraining (with new requirements). To this end, the algorithm focuses on factorizing the models and uses a 2-branch training process to train low-rank versions of the original model (to minimize the accuracy drop). \n\nDifferent from other approaches, the algorithm claims to exploit the importance of each layer when reducing the compute. \n\n\n\nComments:\n- Figure 2 and related numbers are slightly misleading. The paper focuses on CNN while these numbers and the figure is for FC. M changes significantly when using convolutions. It would be great to clarify this all over the text as M increases signigicantly when using (at least) 3x3 convolutions. \n\n- One missing thing for me is taking into account the latency rather than the number of parameters. While factorization may reduce the number of parameters (considering the rank is sufficiently low), the number of layers and therefore data movements increases and so does the latency. Some analysis on this can be found in [1] where the paper trains a network with low-rank promoting regularization. I missed having [1] and other similar approaches in the related work and how the proposed method compares to those directly promoting low-rank solutions. \n\n- The approach would be sounded if the algorithm does not need to work on the factorized version of the layer. That would bring direct benefits to inference time. \n\n- On the algorithmic side, it is not clear to me how the \"2-branches\" are trained and what parameters are shared. This seems to involve more compute, right? How is this better than aiming at the lowest-rank possible?\n\n- The complexity-based criterion is interesting although only uses FLOPS as a proxy. How would this translate in practice when the latency is not directly represented by the FLOPS (given the parallelization). \n\n\n- During the learning, it is not clear to me how the process is implemented and the scalability of this approach. The paper suggests computing SVD per iteration is infeasible. How many iterations are between SVD? and how results are reused. How this is different from [1] where the authors used truncated SVD to promote low-rank every epoch?\n\n- I need clarification on the need of training full-rank and low-rank (end of page 4). If full-rank does not actually provide better accuracy (see [1]), then, why do we need to rely on that?\n\n- Section 3 focuses on works relying on retraining. It would be nice to see how the proposed method compares to those not considering retraining to improve accuracy. \n\n- Not clear to me what is the take-home message with Figure 4. Is the resulting rank per layer enough to represent a big compression? Why not compressing the last layer. the compression is limited as the factorization does not promote sparsity when combining the basis (Sparsity on V). Thus, the size after factorization is the same as the original. \n\n- The BN correction does not seem to contribute. Experiments suggest: our method is effective. Not very appealing as a contribution. \n\n\nMinor details:\n- x is used as a single input (page 2) and as an entire dataset (page 3)?\n\n- How do we set the \\alpha values?\n\n\n\n\n\n[1] Compression-aware training of DNN, Alvarez and Salzmann. NeurIPS 2017", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The proposed paper presents low-rank compression method for DNNs. This topic has been around for a while, so the contribution is limited. Lebedev et. al paper in ICLR 2015 used CP-factorization to compress neural networks for Imagenet classification; in 2019, the idea has to be really novel in order to be presented on CIFAR datasets. The latency is not analyzed. 
So, I agree with reviewers.
Decision: Reject