Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition
Authors: Lucas Liebenwein
Alaa Maalouf
Dan Feldman
Daniela Rus
Venue: NeurIPS 2021 Poster (2021)
Keywords: neural networks
compression
low-rank decomposition
matrix factorization
pruning

Abstract: We present a novel global compression framework for deep neural networks that automatically analyzes each layer to identify the optimal per-layer compression ratio, while simultaneously achieving the desired overall compression. Our algorithm hinges on the idea of compressing each convolutional (or fully-connected) layer by slicing its channels into multiple groups and decomposing each group via low-rank decomposition. At the core of our algorithm is the derivation of layer-wise error bounds from the Eckart–Young–Mirsky theorem. We then leverage these bounds to frame the compression problem as an optimization problem where we wish to minimize the maximum compression error across layers and propose an efficient algorithm towards a solution. Our experiments indicate that our method outperforms existing low-rank compression approaches across a wide range of networks and data sets. We believe that our results open up new avenues for future research into the global performance-size trade-offs of modern neural networks.

Compressing Neural Networks: Towards
Determining the Optimal Layer-wise Decomposition
Lucas Liebenwein"
Alaa Maalouf
MIT CSAIL
University of Haifa
lucas@csail.mit. wwtpaaai..mtee......m.mm
alaamalouf 100mmmlaal12000111aa 120gmai1......
com
Oren Gal University of Haifa
Dan Feldman
Daniela Rus
University of Haifa
MIT CSAIL
orengal@aluma www.heen..cnc.. ac.
dannyf www.y/ga..s@@gmmai.....mmm com
rus@csail mitiiim.iteeee edu
Abstract
We present a novel global compression framework for deep neural networks that automatically analyzes each layer identify the optimal per-layer compression ratio, while simultancously achieving the desired overall compression. Our algo- rithm hinges on the idea of compressing each convolutional (or conneccon)) layer by slicing its channels into multiple groups and decomposing each group via low-rank decomposition. At the of our algorithm is the derivation of layer-wise error bounds from the Eckart- Youi---oggg---- Mirsky theorem. We then leverage these bounds frame the compression problem wish to minimize the maximum compression error across layers and propose an efficient algorithm towards solution. Our experiments indicate that our method outperforms existing low-rank compression approaches across a wide range of networks and data sets. We believe that our results open up new avenues for future research into the global performance-sin trade-offss of modern neural networks.
optimization problem where we
Introduction
Neural network compression entails taking existing model and reducing its computational and memory footprint in order enable the deployment of large- scale networks essource-constraing Beyond inference time efficiency, compression can yield novel insights into the design (Liu et al., 2019b), training (Liebenwein et al., 202Ia,b la,b), and theoretical properties (Arora et al., 2018) 8) of neural networks.
resnet18, ImageNet
Delta Top1 Test Accuracy
+1.0%
ALDS (Durs)
environments.
0.0%
L-Rank
-1.0%
PPP
-2.0%
-3.0%
Among existing compression techniques which in- clude quantization (Wu et al., 2016), distillation (Hin- et al., 2015), and pruning (Han al., 2015) low-rank compression aims at decomposing a layer's weight tensor into tuple of smaller low-rank ten- sors. Such compression techniques may build upon the rich literature low-rank decomposition and its numerous applications outside deep learning such as dimensionality reduction (Laparra et al., 2015) spec-
20.0%
40.0% 60.0%
80.0%
Compression Ratio (Parameters)
Figure ALDS, Automatic Layer-wise De- composition Selector, can compress up to 60% of parameters on a ResNet18 (Ima- geNet), 3x more compared to baselines. De- tailed results are described in Section 3.
denotes authors with equal contributions. Code:
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Global Step
Local Step
Compute per-layer budget (# params): b
Initial Step
Input: CR, k., k
Input: b²,.., ,'
Compute ,',,,, I that minimizes
v e [L], compute j that minimizes c'(k', ,1)
Randomly initialize k,,..,
max4e[1] e' (k'j')
s.t. the total number parameters is reduced by CR%
s.t. for every layer & the number of parameters is at most be
kk
Figure 2: ALDS Overview. The framework consists of a global and local step, see Section 2.
tral clustering (Peng et al., 2015). Moreover, low-rank compression can be readily implemented in any machine learning framework by replacing the existing layer with set of smaller layers without the need for, e.g.. sparse linear algebra support.
Within deep learning, we encounter two related, yet distinct challenges when applying low-rank compression. On the one hand, each layer should be efficiently decomposed (the "local step") and, on the other hand, we need to balance the amount of compression in each layer in order to achieve desired overall compression ratio with minimal loss in the predictive power of the network (the "global step"). While the "local step", i.e., designing the most efficient layer-wise decomposition method, has traditionally received lots of attention (Denton et al., 2014; Garipov et al., 2016; Jaderberg et al., 2014; Kim et al., 2015 5b; Lebedev et al., 2015; Novikov et al., 2015), the "global step" has only recently been the focus of attention in research, e.g., see the recent works of Alvarez and Salzmann (2017); Idelbayev and
(2020); Xu al. (2020).
In this paper, we set out to design a framework that simultaneously accounts for both the local and global step. Our proposed solution, termed Automatic Layer-wise Decomposition Selector (ALDS), addresses this challenge by iteratively optimizing for each layer's decomposition method (local step) and the low-rank compression itself while accounting for the maximum (global step). In Figure 1, we show how ALDS outperforms existing approaches on the common ResNet18 (ImageNet) benchmark (60% compression compared to ~0% for baselines).
incurred across layers
Our framework relies on a straightforward SVD-based decomposition of each layer. Inspired by Denton et al. (2014); Idelbayev and Carreira- Perpinán (2020); Jaderberg et al. (2014) and others, we decompose each layer by first folding the weight tensor into matrix before applying SVD and encoding the resulting pair of matrices as two separate layers.
Efficient layer-wise decomposition.
Enhanced decomposition via multiple subsets. A natural generalization of low-rank decompo- sition methods entails splitting the matrix into multiple subsets (subspaces) before compressing each subset individually.
the context of deep learning, this was investigated before for individual layers (Denton et al., 2014), including embedding layers (Chen et al., 2018; Maalouf et al., 2021). We take this idea further and incorporate into our layer-wise decomposition method as additional hyper- parameter in terms of the number of subsets. Thus, our local step, i.e., the layer-wise decomposition, constitutes of choosing the number of subsets (k=5) for each layer and the rank (j²).
Towards a global solution for low-rank compression. We can describe the optimal solution for low-rank compression as the set of hyperparameters (number of subspaces k and rank je for each layer in our case) that minimizes the drop in accuracy of the compressed network. While finding the globally optimal solution is NP-complete, we propose ALDS an efficiently solvable alternative that enables us to search for locally optimal solution in terms of the maximum relative error incurred across layers. To this end, we derive spectral norm bounds based on the Eckhart- Young- Mirsky Theorem for our layer-wise decomposition method to describe the trade-off between the layer compression and the incurred error. Leveraging our bounds we can then efficiently optimize over the set of possible per-layer decompositions. An overview of ALDS is shown in Figure 2.
2 Method
In this section, we introduce our compression framework consisting of layer-wise decomposition method (Section 2.1), ), a global selection mechanism to simultaneously compress all layers of a network (Section 2.2), and optimization procedure (ALDS) to solve the selection problem (Section 2.3).

---

parallel convolutional layers each with fllters
input temsors each with c/k channels
Convolution with f filters, e channels, Kj Kg kernel
Input tensor of c channels
filters, Jk channels. 1x
kernel
C/R
c/R
*
channels
Figure 3: Left: 2D convolution. right: decomposition used for ALDS. For XXXXX X convolution with filters, channels, and x 82 kemel, our per-layer decomposition consists: (1) & parallel x 9/x x Kg convolutions; (2) single x kj x convolution applied on the first layer's (stacked) output.
2.1 Local Layer Compression
We detail our low-rank compression scheme for convolutional layers below and note that it readily applies to fully-connected layers well special case of convolutions with 1 x kernel.
Compressing convolutions via SVD. Given a convolutional layer of ʃ filters, channels, and a K1 x Ri kernel we denote the corresponding weight tensor by W E R x XM2, Following Denton al. (2014); 4); Idelbayev and Carreira- Perppiáá (2020); Wen et al. (2017) and others, we can then interpret the layer linear layer of shape f RK and the corresponding rank j-approximation as two subsequent linear layers shape f x jj and j X CK R2. Mapped back to convolutions, this corresponds to j X X K2 convolution followed by f x x 1 x convolution.
Multiple subspaces. Following the intuition outlined Section we propose to cluster the columns of the layer's s weight matrix into k > 2 separate subspaces before applying SVD to each subset. To this end, we may consider any clustering method, such k-means or projective clustering (Chen et al., 2018; 8; Maalouf et al., 2021). However, such methods require expensive approximation algorithms which would limit our ability to incorporate them into an optimization- as outlined in Section 2.2. In addition, arbitrary clustering may require re-shuffling the input tensors which could lead to significant slow-downs during inference. We instead opted for a simple clustering method, namely channel slicing. where simply divide the input channels of the layer into subsets each containing at most [c/k] consecutive input channels. Unlike other methods, channel slicing is efficiently implementable, e.g., grouped convolutions in PoTch (Paszke et al., 2017) and ensures practical speed-up ups subsequent to compressing the network.
compression framework
Overview of per-layer decomposition. In summary, for given integers j. k 2 1 and a 4D tensor W E / (xexx1xx2 X62 representing convolution the per-layer compression method proceeds follows:
PARTITION the channels of the convolutional layer into subsets, where each subset has at most [e/k] consecutive channels, resulting in convolutional tensors (Wi) where W.
R xxcxx XX2, and Σ c.
2. DECOMPOSE each tensor W;, i E [k]. by building the corresponding weight matrix W; E (x c.f. Figure 3, computing its j-rank approximation, and factoring it into a pair of smaller matrices Ug of f rows and j columns and V, of rows and G columns.
3. REPLACE the original layer in the network by 2 layers. The first consists of k parallel convolutions, where the jth parallel layer, i E [k], is described by the tensor V, E IR be constructed from the matrix V; (j filters, &; channels, Kn X K2 kernel). The second layer is constructed by reshaping each matrix Us, E [k:]. to obtain the tensor Ut E /xjxxxx and then channel stacking all tensors U1, 24 to get single tensor of shape f kjj x x1x1.
xe Xe xx1 Xrz which can
The decomposed layer is depicted in Figure 3. The resulting layer pair has joNi and j.fk parameters, respectively, which implies a parameter reduction from fCR N2 to j(fk cK1
2.2 Global Network Compression
In the previous section, we introduced our layer compression scheme. We note that in practice we usually want to compress an entire network consisting of layers up to a pre-specified relative reduction in parameters ("compression ratio" or CR). However, is generally unclear how much
3

---

each layer & E [L] should be compressed in order to achieve the desired CR while incurring minimal increase in loss. Unfortunately, this optimization problem No-complete as we would have to check every combination of layer compression resulting in the desired CR in order to optimally compress each layer. On the other hand, simple heuristics, e.g., constant per-layer compression ratios, may lead to sub-optimal results, see Section 3. To this end, we propose an efficiently solvable global compression framework based on minimizing the maximum relative error incurred across layers. We describe each component of our optimization procedure in greater detail below.
The layer-wise relative error as proxy for the overall loss. Since the true cost (the additional loss incurred after compression) would result in an NP-complete problem, we replace the true cost by a more efficient proxy. Specifically, we consider the maximum relative error ε == mr) s" across layers, where s denotes the theoretical maximum relative error in the th layer as described in Theorem below. We choose to minimize this particular cost because: (i) minimizing the maximum relative error ensures that no layer incurs an unreasonably large error that might otherwise get propagated or amplified; (ii) relying scaling between layers may arbitrarily change, e.g., due to batch normalization, and thus the absolute scale of layer errors may not be indicative of the increase in loss; and (iii) the per-layer relative error has been shown to be intrinsically linked to the theoretical compression error, e.g., see the works of Arora al. (2018) and Baykal et al. (2019a) thus representing natural proxy for the cost.
relative instead of an absolute error notion is preferred as
Definition of per-layer relative error. Let W E
and W ""efxff denote the weight tensor and corresponding folded matrix of layer 4, respectively. The per-layer relative error ef is hereby defined as the relative difference in the operator norm between the matrix Wh (that corresponds to the compressed weight tensor V4*5) and the original weight matrix W in layer L, i.e.
E = || - W W'///|/"
(1)
Note that while in practice our method decomposes the original layer into a set of separate layers (see Section 2.1), for the purpose of deriving the resulting error we re-compose the compressed layers into the overall matrix operator W i.e., W j ... U. V where U is the factorization of the ith cluster (set of columns) in the 9th layer, for every é [L] and i [K*]. see supplementary material for more details. We note that the operator norm || || for a convolutional layer thus signifies the maximum relative error incurred for individual output patch ("pixel") across all output channels.
Derivation of relative error bounds. We now derive an error bound that enables us to describe the per-layer relative error in terms of the compression hyperparameters jL and kf, i.e., ef = = <k',, This will prove useful later on as we have to repeatedly query the relative error in our optimization procedure. The error bound is described in the following.
Theorem 1. Given a layer matrix W and the corresponding low-rank approximation W the relative error el = I W| is bounded by
ex < Vk/ai
-max 0i.j+1+ (E[k]
(2)
where an.j+ is the j + largest singular value of the matrix ffr every E [k]] and a ||W||| the largest singular value of Wt.
Proof. First, we recall the matrices WT. W and we denote the SVD factorization for each of them by: W Ū.E,V, Now, observe that for every approximation of W.S. Hence, the SVD factorization of W can be writen as W = 0 v. where Σ E R/xd is a diagonal matrix such that its first ftrst--diage j-diagonal entries are equal to the first j-entries on the diagonal of D and the rest are zeros. Hence,
[K),, the matrix W. is the j-rank
W /² [W[ WL, W wi/h = 1U
(3)
= [U] Ü diag (E ED (EL - -
By (3) and by the triangle inequality, we have that
W we s <1 ....... UK] diag (15 S4 ".. -S--)))))
(4)
4

---

Now, we observe that
lu.. of - -5
-
J9 03 || = Idiag(A _ (o,.
k.
(5)
Finally, we show that
|diag (SI - VI ..... ($1 -
||
= max
(6)
[k(
max
max aij+bs
(7)
[E(A]
where the second equality holds since the columns of are orthogonal and the last equality holds according to the Eckhart- Bookhrt-Yooun--- (roung-Mirsky Theorem (Theorem 2.4.8 of Golub and Van Loan (2013)).
Plugging (7) and (5) into (4) concludes the proof.
Resulting network size. Let 0 (W denote the set of weights for the T. layers and note that the number of parameters in layer is given by IW = f te is and |0] Eee(r [W Moreover, note that [W = je f + x(())) decomposed, 0 DW and (0 ElE(l [ The overall compression ratio is thus given by 1 1- -1/1/11 where neglected other parameters for case of exposition. Observe that the layer budget W is fully determined by k j just like the error bound.
Global Network Compression. for the optimal per-layer budget:
Putting everything together we obtain the following formulation
Eopt
min (
max e (K' ((',)") KE[L]
subject to
1 1 ld(k' J',....'' k...k" 5...k"1111//111 S CR.
where CR denotes the desired overall compression ratio. Thus optimally allocating a per-layer budget entails finding the optimal number of subspaces ke and ranks je for each layer constrained by the desired overall compression ratio CR.
2.3 Automatic Layer-wise Decomposition Selector (ALDS)
propose to solve (8) by iteratively optimizing k1 ........ kL and pnd , until convergence akin of an EM-like algorithm as shown in Algorithm and Figure 2.
Specifically, for given set of weights and desired compression ratio CR we first randomly initialize the number of subspaces k. ....,, k² for each layer (Line 2). Based on given values for each k we then solve for the optimal ranks j!.....j+ such that the overall compression ratio is satisfied (Line 4). Note that the maximum error is minimized all errors are equal. Thus solving for the
Algorithm ALDS(O, CR, Nlyood)
Input: network parameters; CR: overall compression ratio; Thent: number random seeds to initialize
Output: ka 6t....,ke ....,k"; k²: number of subspaces for each layer; "h....jj desired rank per subspace for each layer
1: for [Mwedd] do
2: kl K'...,k' &L
RANDOMINITO
3: while not converged do
ja j². OPTIMALRRANSS ,...,,k Global step: choose g²
k2)
g
5:
for E E [L] do
6 + (k' +N(KK) D resulting layer budget
7:
A f OPTIINASSURRPPCSSSSS
Local step: minimize error bound for given layer budget
end for
9: end while
10: E, RECORDERROR(A ,..., 12 j")
11: end for
12: return k., ......,k' .j² ...........................
... j from thea argmin, 6:
5

---

ranks in Line 4 entails guessing a value for e, computing the resulting network size, and repeating the process until the desired CR is satisfied, e.g. via binary search.
Subsequently, we re-assign the number of subspaces ke for each layer by iterating through the finite set of possible values for ke (Line 7) and choosing the one that minimizes the relative error for the current layer budget 6f (computed in Line 6). Note that we can efficiently approximate the relative error by leveraging Theorem 1. We then iteratively repeat both steps until convergence (Lines 3-8). To improve the quality of the local optimum we initialize the procedure with multiple random seeds (Lines 1-11) and pick the allocation with the lowest error (Line 12).
We note that we make repeated calls to our decomposition subroutine (i.e. SVD; Lines 4, 7) highlighting the necessity for it to be efficient and cheap to evaluate. Moreover, we can further reduce the computational complexity by leveraging Theorem as mentioned above.
Additional details pertaining to ALDS are provided in the supplementary material.
Extensions. Here, we use SVD with multiple subspaces as per-layer compression method. However, we note that ALDS can be readily extended to any desired set of low-rank compression techniques. Specifically, we can replace the local step of Line 7 by a search over different methods, e.g., Tucker decomposition, PCA, or other SVD compression schemes, and return the best method for given budget. In general. we may combine ALDS with any low-rank compression as long as we can efficiently evaluate the pay-layer error of the compression scheme. In the supplementary material, we discuss some preliminary results that highlight the promising performance of such extensions.
3
Experiments
Networks and datasets. We study various standard network architectures and data sets. Particularly, we test our compression framework on ResNet20 (He et al.. 2016), DenseNet22 (Huang et al.. 2017), WRN16-8 (Zagoruyko and Komodakis. 2016), and VGG16 (Simonyan and Zisserman, 2015) on CIFAR 10 (Torralba et al., 2008); ResNet18 8 (He et al., 2016), AlexNet (Krizhevsky et al., 2012), and MobileNetV2 (Sandler et al., 2018) on ImageNet (Russakovsky et al., 2015); and on Deeplab- V3 (Chen et al., 2017) with a ResNet50 backbone on Pascal VOC segmentation data (Everingham et al., 2015).
Baselines. We compare ALDS to a diverse set of low-rank compression techniques. Specifically, we have implemented PCA (Zhang et al., 2015b), SVD with energy-based layer allocation (S (SVID-nneryy) following Alvarez and Salzmann (2017); Wen et al. (2017), and simple SVD with constant per-layer compression (Denton et al., 2014). Additionally, we also implemented the recent learned rank selection mechanism (L-Rank) of Idelbayev and Carreira-Perpinde (2020). Finally, we implemented two recent filter pruning methods, i.e., FT of Li et al. (2016) and PFP of Liebenwein et al. (2020), as alternative compression techniques for densely compressed networks. Additional comparisons on ImageNet are provided in Section 3.2.
Retraining. For our experiments, we study one-shot and iterative learning rate rewinding inspired by Renda et al. (2020) for various amounts of retraining. In particular, we consider the following unified compress-retrain pipeline across all methods:
1. TRAIN for epochs according the standard training schedule for the respective network. 2. COMPRESS the network according to the chosen method.
ALDS [Ours]
PCA
+
SVD.Energy
SVD
L-Rank
FT
PFP wm16.6.
Deta Top1 Test Accurbay
dansenat22, CIFARO
Deta Top3 Tasst Accuraty
upg16_br, br. OFAR10
CIFAR1O
+1.0%
+ +..0%
13.5%
0.0%
0.0%
Test
11.0%
1.0%
-1.0%
1.5%%
-2.0%
-2.0%
-2.0%
3.00
-2.5%
20.0% 30 0% 40.0% 50,0 0% 60.0% 70 0%
70.0%
75.0%
80.0% 55.0% 90.0% 95.0%
85.0% 88 0% 90, 90.0% 92. 0% 94, 64.0%5 96 0%
Compressuon (Parametern)
Compresiion
Compression
(a) DenseNet22
(b) VGG16
(c) WRN16-8
Figure 4: One-shot compress+retrain experiments on CIFAR10 with baseline comparisons.

---

ALDS (Ours)
PCA
SVD-Energ En-rEneryy
SVD
L-Rank
PFP resnet20, CIFARIO, 6 1.0%
ResNet20, CIFAR 10
Dalaa Topd Test Accuraty
Dett Topt TTest Accuncc
resnee20, CIAR1C
Comprecsion Raiio (Parmms)
1.00%
0.0%
60.0%
-1.0%
60.0%
60.0%
2.07
880.0%
3.0%
0.0%
20 0%
40.0%
60
80
20.0
40
00.0% (Parametors)
100.0%
0%
18.0%
3.0%
1.0%
(Parameters)
Amount
(a) Compress-only (r=0)
(b) Onc-shot (r=e)
(c) Retrain sweep (A (A-Top1 2-1%
RosNer18, ImageNet
Deta Test Teat accurnty
resnettl, Imagehet
Accuracy
reshet1B, ImageNet
resnet111 ImageWet. o 1.0%
+1.0%
80.0%
0.0%
60.0%
40.0%
-3.0%
0.0% !
095 40,00 8%
30.0%
Comprussion Rabio (Parameturs)
Compression Ratio (Forameturs)
Amount of Retraining
(d) Compress-only (r=0)
(e) One-shot (r=e)
(I) Retrain sweep (A- (A-Tpp12-1%) 2-1%)
Figure 5: The size-accuracy trade-off for various compression ratios, methods, and networks. Com- pression performed after training and networks were re- ve-trained once for the indicated amount (one-shot). (a, b, d, e): the difference test accuracy for fixed amounts retraining. (c. f): the maximal compression ratio with less-than- % accuracy drop for variable amounts of retraining.
3. RETRAIN the network for r epochs using the training hyperparameters from epochs [e T, e). 4. ITERATIVELY repeat 1.-3. after projecting the decomposed layers back (optional).
Reporting metrics. We report Top-1, Top-5, and IoU test accuracy as applicable for the respective task. For each compressed network we also report the compression ratio, i.e., relative reduction, in terms of parameters and floating point operations denoted by CR-P and CR-F, respectively. Each experiment was repeated 3 times and we report mean and standard deviation.
3.1 One-shot Compression on CIFAR10, ImageNet, and VOC with Baselines
We train reference networks on CIFARIO, ImageNet, and VOC, and then compress and retrain the networks once with r e for various baseline comparisons and compression ratios.
CIFAR1O. In Figure 4, we provide results for DenseNet22, VGG16, and WRN16-8 CIFAR1O. Notably, our approach is able to outperform existing baselines approaches across a wide range of tested compression ratios. Specifically, in the region where the networks incur only minimal drop in accuracy (A-Top12 2-p12-1990 1%) ALDS is particularly effective.
ResNets (CIFAR10 and ImageNet). Moreover, we tested ALDS on ResNet20 (CIFAR10) 10) and ResNet18 (ImageNet) as shown in Figure 5, For these experiments, we performed a grid search
mobilenet_vk v2, ImageNat
deeplabv3 deenela VOCSeqmentatton2012
Detta Top1 Test Accuracy
0.0%
Dolta loU Test Accuracy
+2.0%
1.0%
+1.0%
ALDS (Ours)
OVD-Energy
-2.0%
-1.0%
L-Rank
-4.0%
-2.0%
PIP
-5.0%
20.0%
30.0%
40.0%
50.0%
60.00
0%
Compression Ratio (Parameters)
Compression Ratio (Parameters)
(a) MohileNetV2 (ImaggNect
(b) D (VOC)
Figure 6: One-shot compress+retrain experiments on various architectures and datasets with baseline comparisons.
7

---

Table 1: Baseline results for A-Topl 0.5% for one-shot with highest CR-P and CR-F among
tensor decomposition methods bolded for each network. Results coincide with Figures 4, 5, 6b.
Tensor decomposition
Filter pruning
Model
Metric
ALDS (Ours)
PCA
EVD-Energy
SVD
L-Rank
PTP
ResNer20
A-Topl
-0.47
91.39 91.39
CR-F CR-F
CR-P.
74,91. 67.86
49.88, 48.67
49.88, 49.08
39.81, 38.95
28.71, 54,89
39.69, 39.57
40.28, 30.06
CIFAR10 10
VGG16
A-Topl
-0.11
-0.02
-0.08
+0.29
-0.35
-0.47
-0.47
Topli: 92.78
CR-F CR-F
CR-P.
95.77, 86.23
89.72, 85.84
82.57,
81.32
70.35,
70.13
85.38, 75.86
79.13, 78.44
94.87, 84.76
DenseNer22
3-Topl
-0.32
+0.20
-0.29
+0.13
+0.26
-0.24
-0.44
Topl:
CR-P. CR.F
56.84, 61.98
14.67, 34.55
15.16, 19.34
15.00, 15.33
14.98, 35.21
28.33, 29.50
40.24, 43.37
A-Topl
-0.44
Topt: 89.88
CR.P.CR.F
87.77, 79.90
85.33,
64.75,
40,20,
49.86,
82.33,
80.68
ImageNet
ResNet18
A Topp. Top5
-0.40, .0.05
00.95.0.37
49, -0.64
-1.75, 40.72
-0.71, -0.23
+0.10, +0.42
-0.39. -0.08
Topt: 6962. Tope: 89.08
CR-P. CR-F
43.51
50.38,
9.86,
17.96
MobileNetV2
A-Topl, Tops
-1.53, -0.73
-0.87, -0.55
-1.27,
-3665, -2.07
19.08, -13.40
-1.73,
-0.97, -0.40
Topl: 71,45 Topó: 90.33
CR-P.CR-F
32,97. 11.01
20.91, 0.26
20,02, 8.57
20,03, 31,99
20,00, 61.97
21.31, 20.23
20.02, 7.96
VOC
DocplabV3
A-IoU, Topl
.026,
-0.28,
-0.42,
-0.49,
loU: 91.39 Topl: 99.34
CR-P. CR-F
64.38, 64.11
55.68, 55.82
55.82
31.61.
32.27
31.64, 31.51
44.99, 45,02
15.00, 15.06
45.17, 43,93
over both multiple compression ratios and amounts of retraining. Here, we highlight that ALDS outperforms baseline approaches even with significantly less retraining. On Resnet 18 (ImageNet) ALDS can compress over 50% of the parameters with minimal retraining (1% retraining) and less-than-Ie - accuracy drop compared the best comparison methods (40% compression with 50% retraining).
MobileNetV2 (ImageNet). Next, tested and compared ALDS on the MobileNetV2 ar- chitecture for ImageNet shown in Figure 6a. Unlike the other networks, MobileNetV2 network already specifically optimized for effi- cient deployment and includes layer structures such as depth-wise and channel-wise convolu- tional operations. It is thus more challenging to find redundancies in the architecture. We find that ALDS can outperform existing tensor de- composition methods in this scenario well.
Table 2: AlexNet and ResNet18 Benchmarks ImageNet. We report Top- 1, Top-5 accuracy and percentage reduction of FLOPs (CR-F). Best
sults with less than 0.5% accuracy drop are bolded.
Method
A-Top1
A A-Top5
CR-F (%)
AL.D (Ours)
-0.38
+0.04
64.5
ALDS (Ours)
-1.37
-0.56
76.3
MUSCO (Geak 2019)
-0.37
-0,20
58.67
TRPI al. 2020)
-4.18
-2.5
44.70
ResNet8 Topl, 5. 69.64%, 60.90%
TRPI+Nu ex 2020)
-4.25
-2.61
55.15
VOC. Finally, we tested the same setup on DeeplabV3 with ResNet50 backbone trained on Pascal VOC 2012 segmentation data, see Figure 6b. We note that ALDS consistently out- performs other baselines methods in this setting as well (60% CR-P vs. 20% without accuracy drop).
TRP2+Nu (Xust 20200
-4.3
-2.37
68.55
PCA (Zlung ctal. 2013b)
-6.54
-4.54
29.07
Expand (ladorhrg 2014)
-5.26
50.00
PFP (Lichenwein 2009)
-2.26
-1.07
29.30
SoftNet al.
-2.54
-1.2
41.80
Median
-1.23
-0.5
41.80
Slimming (Liu etal, 2017)
-1.77
-1.19
28.05
Low-cost (Dang
-3.55
-2.2
34.64
Tabular results. Our one-shot results are again summarized in Table I where we report CR-P and CR-F for 2-Top12-0.5%. We ob- serve that ALDS consistently improves upon prior work. We note that pruning usually takes on the order of seconds and minutes for CIFAR and ImageNet, respectively, which usually faster than even a single training epoch.
Gating (Hureta), 2018)
-1.52
-0.93
37.88
FT (Beetal. 2017)
-3.08
-1.75
41.86
DCP (Zhonng et al, 2018)
-2.19
-1.28
47.08
FBS etan 2018)
-2.44
-1.36
49.49
ALDS (Ours)
-0.21
-0.36
77.9
AlexNet Top1, 5. 57 30%. W0 21%
ALDS (Ours)
-0.41
-0.54
81.4
Tucker (Ki et 2015a)
N/A
-1.87
62.40
Regularize (hiirtal,
N/A
-0.54
74.35
Coordinate (Wim et 201,
N/A
-0.34
62.82
3.2 ImageNet Benchmarks
Efficient (Kin et 2019)
40.7
-0.3
62.40
L-Rank (Jeclayev al., 2020)
-0.13
-0.13
66.77
Next, we test our framework on two common ImageNet benchmarks, ResNet18 and Nee We follow the compress-retrain pipeline outlined in the beginning of the section and repeat it it-
NISP etr 2018)
-1.43
N/A
67.94
OICSR (Lis al., 20194)
-0.47
N/A
53.70
Oracle (Ding et al., 2019)
-1.13
-0.67
31.97
8

---

eratively to obtain higher compression ratios. Specifically, after retraining and before the next compression step we project the decomposed layers back to the original layer. This way, we avoid recursing on the decomposed layers.
Our results are reported in Table 2 where we compare to wide variety of available compression benchmarks (results were adapted directly from the respective papers). The middle part and bottom part of the table for each network are organized into low-rank compression and filter pruning approaches, respectively. Note that the reported differences in accuracy (A-Topl and A-Top5) are relative our baseline accuracies. On ResNet18 we can reduce the number of FLOPs by 65% with minimal drop in accuracy compared to the best competing method (MUSCO, 58.67% With slightly higher drop in accuracy (-1.37%) we can even compress 76% of FLOPs. On AlexNet, our framework finds networks with -0.21% % and -0,41% 1% difference in accuracy with over 77% and 81% % fewer FI.OPs, This constitutes a more-than- th%- improvement in terms of FL.OPs compared to current state-of-the-ar (L.-Rank) for similar accuracy drops.
3.3 Ablation Study
To investigate the different features of our method we ran com- pression experiments using multiple variations derived from our method, see Figure For the simplest version of our method we consider constant per-layer compression ratio and fix the value of 1 to either or 5 for all layers denoted by ALDS-Simple3 and ALDS-Simples
resnet20, CIFARIO
Top1 Test Accuracy
+1.0%
0.0%
-1.0%
ALUS
respectively. Note that ALDS-Simple with k: = corresponds to the SVD comparison method. For the version denoted by ALDS-Erroo we fix the number of subspaces per layer (k = 3) and only run the global step of ALDS (Line 4 of Algorithm 1) to determine the optimal per-layer compression ratio. The results of our ablation study in Figure 7 indicate that our method clearly benefits from the combination of both the global and local step in terms of the number of subspaces (k) and the rank per subspace ()).
-2.0%
Messi
-3.0%
SVB-Irrr
20.0%
40.0%
60 0%
80. 00.0%
Compression (Parameters)
Figure 7: One-shot ablation study of ALDS for Resnet20 (CIFARIO).
We also compare our subspace clustering (channel slicing) to the clustering technique of Maalouf et al. (2021), which clusters the matrix columns using projective clustering. Specifically, we replace the channel slicing of ALDS-Simple3 with projective clustering (Messi3 in Figure 7). As expected Messi improves the performance over ALDS-Simpl but only slightly and the difference is essentially negligible. Together with the computational disadvantages of Messi-like clustering methods (unstruc- tured, NP-hard; see Section 2. ALDS-based based simple channel slicing is therefore the preferred choice in our context.
4 Related Work
Our work builds upon prior work in neural network compression. We discuss related work focusing on pruning, low-rank compression, and global aspects of compression.
Unstructured pruning. Weight pruning (Lin et al., 2020b; Molchanov et al., 2016, 2019; Singh and Alistarh, 2020; Wang et al., 2021; Yu et al., 2018) techniques aim to reduce the number of individual weights, e.g., by removing weights with absolute values below threshold (Han et al., 2015; Renda et al., 2020), or by using mini-batch of data points to approximate the influence of each parameter on the loss function (Baykal et al., 2019a,b) However, since these approaches generate sparse instead of smaller models they require some form of sparse linear algebra support for runtime speed-ups.
Structured pruning. Pruning structures such filters directly shrinks the network (Chen et al., 2020; Li et al., 2019b; Lin et al., 2020a; Liu et al., 2019a; Luo and Wu, 2020; Ye et al., 2018). Filters can be pruned using a score for each filter, e.g., weight-based (He et al., 2018, 2017) or data-informed (Liebenwein et al., 2020; Yu et al., 2018), and removing those with a score below a threshold. It is worth noting that filter pruning is complimentary to low-rank compression.
Low-rank compression (local step). common approach to low-rank compression entails tensor decomposition including ducker-decomposiiioon (Kim et al., 2015b), CP-decomposition (Lebedev et al., 2015), Pensor-Train (Garipov et al., 2016; Novikov et al., 2015) and others (Denil et al., 2013; Ioamnou et al., 2017; Jaderberg et al., 2014). Other decomposition- approaches include weight

---

sharing, random projections, and feature hashing (Arora et al., 2018; Chen et al., 2015a,b; Shi et al., 2009; Ullrich al., 2017: Weinberger al.. 2009). Alternativly, low-rank compression can be performed via matrix decomposition (e.g., SVD) on flattened tensors as done by Denton et al. (2014); Sainath et al. (2013); Tukan et al. (2020); Xue al. (2013); Yu et al. (2017) among others. Chen et al. (2018), Denton et al. (2014); Maalouf et al. (2021) also explores the use of subspace clustering before applying low-rank compression to each cluster to improve the approximation error. Notably, most prior work relies some form of expensive approximation algorithm even to just solve the per-layer low-rank compression, e.g., clustering or tensor decomposition. In this paper, we instead focus on the global compression problem and show that simple compression techniques (SVD with channel slicing) are advantageous in this context as we can use them as efficient subroutines. We note that can even extend our algorithm to multiple, different types of per-layer decomposition.
Network-aware compression (global step). To determine the rank (or the compression ratio) of each layer, prior work suggests to account for compression during training (Alvarez and Salzmann, 2017; loannou et al., 2016, 2015; Wen et al., 2017; Xu al., 2020), e.g. by training the network with penalty that encourages the weight matrices to be low-rank. Others suggest to select the ranks using variational Bayesian matrix factorization (Kim et al., 2015b). In their recent paper, Chin et al. (2020) suggest to produce Our paper was also inspired by a recent line of work towards automatically choosing or learning the rank of each layer (Gusak et al., 2019; Idelbayev and Carreira- Perpinnnn, 2020; Li and Shi, 2018; Tiwari et al., 2021; Zhang et al., 2015b,c). We take such approaches further and suggest a global compression framework that incorporates multiple dccomposition techniques with more than one per layer (number of subspaces and ranks of each layer). This approach increases the number of local minima in theory and helps improving the performance in practice.
entire set of compressed networks with different accuracy/speed trade-offs.
5 Discussion and Conclusion
Practical benefits. By conducting a wide variety of experiments across multiple data sets and networks we have shown the effectiveness and versatility of our compression framework compared to existing methods. The runtime of ALDS is negligible compared to retraining and it can thus be efficiently incorporated into compress-retrain pipelines.
ALDS as modular compression framework. By separately considering the low-rank compression scheme for each layer (local step) and the actual low-rank compression (global step) we have provided a framework that can efficiently search over a set of desired hyperparameters that describe the low- rank compression. Naturally, our framework can thus be generalized to other compression schemes (such as tensor decomposition) and we hope explore these aspects in future work.
Error bounds lead to global insights. At the core of our contribution is our error analysis that enables to link the global and local aspects of layer-wise compression techniques. We leverage our error bounds in practice to compress networks more effectively via an automated rank selection procedure without additional tedious hyperparameter tuning. However, we also have rely on proxy definition (maximum relative error) of the compression error to enable tractable solution that we can implement efficiently. We hope these observations invigorate future research into compression techniques that come with tight error bounds potentially even considering retraining which can then naturally be wrapped into global compression framework.
Acknowledgments
This research was sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750- 19-00-199---0000 The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. This work was further supported by the Office of Naval Research (ONR) Grant N00014-18- 2830.
10

---

Funding Transparency Statement
Authors declare no competing interests. Funding in direct support of this work: the United States Air Foree Research Laboratory and the United States Air Force Artificial Intelligence Accelerator accomplished under Cooperative Agreement Number 1A8750-19-2-1000 and the Office Naval Research (ONR) Grant N00014-15 10--188--1-00
References
Jose M Alvarez and Mathieu Salzmann. Compression-avee aware training deep networks. In Advances in Neural Information Processing Systems, pages 856 867, 2017.
Sanjeev Arora, Rong Ge, Behnam Neyshabur. and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning, pages 263- 2018.
Cenk Baykal, Lucas Liebenwein, Igor Gilitschenski, Dan Feldman, and Daniela Rus. Data-dependent coresets for compressing neural networks with applications to generalization bounds. In Inter- national Conference on Learning Representations, 2019a. URL https //pperevveeeeeeeee net/ forumiid=HJfuJ2AS.
Cenk Baykal, Lucas Liebenwein, Igor Gilitschenski, Dan Feldman, and Daniela Rus. Sipping neural networks: Sensitivity-informed provable pruning of neural networks. arXiv preprint arXiv:1910.05422, 2019b.
Jianda Chen, Shangyu Chen, and Sinno Jialin Pan. Storage efficient and dynamic flexible runtime channel pruning via deep reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.
Siang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05387, 2017.
Patrick H. Chen, Si Si, Yang Li, Ciprian Chelba, and Cho-jui Hsieh. GroupReduce: Block-Wiss Low-Rank Approximation for Neural Language Model Shrinking. Advances in Neural Information Processing Systems, 3018-December: 10988-10998,, jun 2018. URL http: //axxiv/ org/ abs/ 1806. 06950.
Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In International conference on machine learning, pages 2285-2294, 2015a.
Wenlin Chen, James T. Wilson, Stephen Tyree, Kilian Q Weinberger, and Yixin Chen. Compressing convolutional neural networks. CoRR, 0te//150..4444 2015b. URL http://arxiv. orge abs/ 1506. 04449.
Ting-Wi Wu Chin, Ruizhou Ding. Cha Zhang, and Diana Marculescu. Towards efficient model compres- sion via learned global ranking. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1518-1522 1128,, 2020.
Kenneth L Clarkson and David Woodruff. Input sparsity and hardness for robust subspace approximation. In 2015 IEEE 56th Annual Symposium on Foundations of Computer Science, pages 310-329. IEEE, 2015.
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1): 1-22, 1977.
Misha Denil, Babak Shakibi, Laurent Dinh, Mare Aurelio Ranzato, and Nando de Freitas. Predicting parameters in deep learning. In Advances Neural Information Processing Systems 26, pages 2013.
11

---

Emily Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation. Advances in Neural Information Processing Systems, 20anuary):1269-12 12777 apr 2014. URL 0736.
1404.
Xiaohan Ding, Guiguang Ding, Yuchen Guo, Jungong Han, and Chenggang Yan. Approximated oracle filter pruning for destructive cnn width optimization. In International Conference on Machine Learning, pages 1607-1616. PMLR, 2019.
Xuanyi Dong, Junshi Huang, Yi Yang, and Shuicheng Yan. More less: A more complicated network with less inference complexity. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pages 5840-5848, 2017.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(())98-136, 2015.
Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert Mullins, and Cheng-zhong Xu. Dynamic channel pruning: Feature boosting and suppression. arXiv preprint arXiv: 0t.///100000333 2018.
Timur Garipov, Dmitry Podoprikhin, Alexander Novikov, and Dmitry Vetrov. Ultimate tensorization: compressing convolutional and fe layers alike. arXiv preprint arXiv:1611.03214 2016.
Gene H Golub and Charles F Van Loan. Matrix computations, volume 3. JHU press, 2013.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv: 1rXiv:1706.02677 2017.
Julia Gusak, Maksym Kholiavchenko, Evgeny Ponomarev, Larisa Markeeva, Philip Blagoveschensky. Andrzej Cichocki, and Ivan Oseledets. Automated multi-stage compression of neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. CoRR, abs/1510.00149, 2015. URL http:// http:///axxvv..oo
www.iii.orr//baae. 11100 00149.
Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In 2011 International Conference on Computer Vision, pages 991-998. IEEE, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 770-778, 2016.
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruming for accelerating deep convolutional neural networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 2234-2240. AAAI Press, 2018
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4349, 2019.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 1389-1397,, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv: 505.0033..333. 2015.
Weizhe Hua, Yuan Zhou, Christopher De Sa, Zhiru Zhang, and Edward Suh. Channel gating neural networks. arXiv preprint arXiv:1805.125.0 2018.
12

---

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
Yerlan Idelbayev and Miguel A Carreira- Perpinán. Low-rank compression of neural nets: Learning the rank of each layer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages (049-8059) 2020.
Y Ioamnou, D Robertson, J Shotton, R Cipolla, and A Criminisi. Training cnns with low-rank filters for efficient image classification. In 4th International Conference on Learning Representations, ICLR 2016- Colf-Coneeeence Track Proceedings, 2016.
Yani Ioannou, Duncan Robertson, Jamie Shotton, Roberto Cipolla, and Antonio Criminisi. Training cnns with low-rank filters for efficient image classification. arXiv preprint arXiv: 111.044511,06744, 2015.
Yani Ioannou, Duncan Robertson, Roberto Cipolla, and Antonio Criminisi. Deep roots: Improving cnn efficiency with hierarchical filter groups. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1231-1240, 2017.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. In Proceedings of the British Machine Vision Conference. BMVA Press, 2014.
Hyeji Kim, Muhammad Umar Karim Khan, and Chong-Min Kyung. Efficient neural network compression. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12569-12577, 2019.
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Compres- sion of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications. 4th International Conference on Learning Representationss ICLR 2016 Conference Track Proceed- ings, nov 2015a. URL /ttpt:pppp///.i/ttttottt wttp://..wr//aba/11....ee
Yong- Yong-Deok Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang. and Dongjun Shin. Com- pression of deep convolutional neural networks for fast and low power mobile applications. arXiv preprint /51.5005 2015b.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. C. Burges, L. Bottou, and K. Q Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 1097-1105. Curran Associates, Inc., 2012. URL. http:://apente ht/p//paars.iiip.. cc/paper/ 4824 wetereentttclssssfcaattiiaeeeeee wittoonwwitttheee deeee cwpp-poooooooolll--eeeoeoee neeural-networke .df.
Valero Laparra, Jesús Malo, and Gustau Hamps--alls. Dimensionality reduction via regression in hyperspectral imagery. IEEE Journal of Selected Topics in Signal Processing, 96):1026- 2015.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan V. Oseledets, and Victor S. Lempitsky. Speeding-up up convolutional neural networks using fine-tuned cp-decomposition. In ICLR (Poster), 2015. URL http: / /ttp://///xxii/ wrw. www./oog////////oo.//..oo... /atess//12...e.0 6553.
Chong Li and CJ Shi. Constrained optimization based low-rank approximation of deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 732- 747, 2018.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 1.X.vv66000087000 2016.
Jiashi Li, Qi Qi, Jingyu Wang, Ce Ge. Yujian Li, Zhangzhang Yue, and Haifeng Sun. Oiesr: Out-in- channel sparsity regularization for compact deep neural networks. Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 7046-7055, 2019a.
13

---

Yawei Li, Shuhang Gu, Luc Van Gool, and Radu Timofte. Learning filter basis for convolutional neural network compression. In Proceedings of the IEEE International Conference on Computer Vision, pages 5623-5632, 2019b.
Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks. In International Conference on Learning Representations, 2020.
URL httpttp::///eeeeeviiiee www.:ew..eeeo.eeeeeeeeeeeeeeeeee
Lucas Liebenwein, Cenk Baykal, Brandon Carter, David Gifford, and Daniela Rus. Lost in pruning: The effects of pruning neural networks beyond test accuracy. Proceedings of Machine Learning and Systems, 3, 2021a.
Lucas Liebenwein, Ramin Hasani, Alexander Amini, and Daniela Rus. Sparse flows: Pruning continuous-depth models. arXiv preprint vrXiv22106.17116 2021b.
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pages 1529-1538, 2020a.
Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. International Conference on Learning Representations, 2020b. URL www. www.ooiids..eeeeee.ee.e BBBB.
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3296-33660 -395,-305,10 2019a.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pages 7736-27444 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2019b. URL htat//oormmidd:rrInnne
Jian-Hao Luo and Jianxin Wu. Autopruner: An end-to-end trainable filter pruning method for efficient deep model inference. Puttern Recognition, 107:107461, 2020.
Alaa Maalouf, Harry Lang, Daniela Rus, and Dan Feldman. Deep learning meets projective clustering. In International Conference on Learning Representations, 2021. URL https: //perrevies......... net/forum?dd Setrrumiddd-ffpyyF----
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint 1611v:6611.0644)) 2016.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11264-1 1272, 2019.
Alexander Novikov, Dmitry Podoprikhin, Anton Osokin, and Dmitry Vetrov. Tensorizing neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems- Volume 1, pages 442-450, 2015.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS- W, 2017.
Xi Peng, Zhang Yi, and Huajin Tang. Robust subspace clustering via thresholding ridge regression. In Tiveny-Ninth AAAI Conference on Artificial Intelligence, 2015.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing fine-tuning and rewinding in neural network pruning. In International Conference on Learning Representations, 2020. URL. /ctps://opeee /ttps://oppnrreiee: wwt.vviwwwneeeeeeeee wwee//oorumiicoo fodpfoommmmddddisooooooo
14

---

Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-252, 2015. doi: 10.10/ 1233-015-88666-
Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low- rank matrix factorization for deep neural network training with high-dimensional output targets. In 2013 IEEE international conference on acoustics, speech and signal processing, pages 6655-6659. IEEE, 2013.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
Qinfeng Shi, James Petterson, Gideon Dror, John Langford, Alex Smola, and SVN Vishwanathan. Hash kernels for structured data. Journal of Machine Learning Research, 10(Nov):2115-2637, 2009.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2015.
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximations for model compression. arXiv preprint 2tfi::2000.143444 2020.
Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, al. Convolutional neural networks with low-rank regularization. arXiv preprint arXiv: 2015.
Rishabh Tiwari, Udbhav Bamba, Amav Chavan, and Deepak Gupta. Chipnet: Budget-aware pruning with heaviside continuous approximations. In International Conference Learning Representations, 2021. URL
http:aawwwwwweeeeeeeeeeeeeeeeeee wwwt/eerurrn/// httt/forumidd=xOO0 XxxxxxXXTxxXXI111.1
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence, 30(1 1):1958-1970, 2008.
Murad Tukan, Alaa Maalouf, Matan Weksler, and Dan Feldman. Compressed deep networks: Goodbye svd, hello robust low-rank approximation. arXiv preprint wwXiv:2009.056477 2020.
Karen Ullrich, Edward Meeds, and Max Welling. Soft weight-sharing for neural network compression. arXiv preprint arXiv: I 702.04008. 2017.
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Neural pruning via growing regularization. In International Conference on Learning Representations, 2021. URL htttttttt///perrre//e net/ forum?id=0966 Is _PA.
Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, and Josh Attenberg. Feature hashing for large scale multitask learning. In Proceedings of the 26th annual international conference on machine learning, pages 11 1113-1120, 2009.
Wei Wen, Cong Xu, Chumpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Coordinating Filters for Faster Deep Neural Networks. Proceedings of the IEEE International Conference on Computer Vision, 2017-Octob:658-661 mar 2017. URL http: / aaxiv.iiiivv//..//aaaaaaaaaavvaa . org/ abs/ 1703 09746.
Jiaxiang Wu, Cong Leng, Yuhang Wang. Qinghao Hu, and Jian Cheng. Quantized Convolutional Neural Networks for Mobile Devices. In Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Yuhui Xu, Yuxi Li, Shuai Zhang, Wei Wen, Botao Wang, Yingyong Qi, Yiran Chen, Weiyao Lin, and Hongkai Xiong. TRP: Trained Rank Pruning for Efficient Deep Neural Networks. IJCAI International Joint Conference on Artificial Intelligence, 2021 Ja7aaaa997777 apr 2020. URL http:/ /wwiiv org/ abs/ 14566.
15

---

Jian Xue, Jinyu Li, and Yifan Gong. Restructuring of deep neural network acoustic models with singular value decomposition. In Interspeech, pages 2369, 2013.
Jianbo Ye, Xin Lu, Zhe Lin, and James Z. Wang. Rethinking the smaller-norm-less moorm--les-informatiy assumption in channel pruning of convolution layers. In International Conference on Learning Representations, 2018. URL https::t////. /ttp:///oppeeeeeeeeeeee
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I Morariu, Xintong Han, Mingfei Gao, Ching-Yuue Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9194 9193-9203, 2018.
Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7370-7379, 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint IrXiv:1605.00 2016.
Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating Very Deep Convolutional Networks for Classification and Detection. IEEE Transactions on Putfern Analysis and Machine Intelligence, 38(10); may 2015a. URL http:://rxtiii www//bg.///.oooooooogo...oo... 1t////5550000.. 06798.
Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. IEEE transactions on pattern analysis and machine intelligence, 38(10):1943-1955 2015b.
Xiangyu Zhang, Jianhua Zou, Xiang Ming, Kaiming He, and Jian Sun. Efficient and accurate approximations of nonlinear convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and paltern Recognition, pages 1984-1992, 2015c.
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrimination-ave channel pruning for deep neural networks. arXiv preprint arXiv: 1180, 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contri- butions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 2.3 and 5
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms them?
[Yes]
you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material URL)? [Yes] See supplementary material. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] see supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type GPUs, internal cluster, cloud provider)? [Yes] See supplementary material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes]
16

---

(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material or URL? [Yes] We
include an implementation for our method and many other competing methods.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
17


Reviews:
[{"review_id": "szlYfo4rqTE", "reviewer": "Reviewer_piFe", "summary": "The paper proposes a one-shot approach to compress already trained neural networks using low-rank compression. The authors propose a new scheme of applying low-rank: instead of applying it directly to a layer, authors suggest partitioning the filters into K blocks, and apply low-rank to each of these blocks separately. Haven defined this scheme, authors then propose a one-shot alternating-optimization algorithm to select ranks and to partition matrix columns/filters into K blocks. Once ranks and partitions are determined, the matrices are decomposed and finetuned. Optionally, this entire compress=>finetune process can be repeated multiple times. Authors present numerical results where the suggested approach almost always wins, however some of the comparisons are questionable. In terms of the algorithm itself, the details of alt.opt are missing, as well as theoretical analysis of the proposed compression scheme is missing.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written, and the method's main idea is presented well; the literature review is up to date. The authors present an impressive amount of experimental validations, however, there are several limitations of this work stemming from its main contributions. To remind the reader, there are two main contributions of this paper:\n   1) a new low-rank compression scheme\n   2) a simple one-shot alternating optimization\nwhich in combination allows to achieve a quick and accurate (based on the results) compression. Unfortunately, both of these contributions lack a substantial analysis and might actually lead to a subpar (theoretically) performance when compared to a regular low-rank compression. Let me explain my analysis next.\n\n**Issue 1** Analysis of a new compression scheme. The authors, in the \"multiple subspaces\" paragraph (line 82-92) propose to partition (slice) the columns of the matrix and apply j-rank compression to a clustered subspaces and then combine them. For instance, assuming $k=2$ partitioning, if we have a matrix of size $m\\times n$, authors propose to split it into $m \\times a$ and $m \\times b$ (with $a+b=n$) matrices and apply $j$-rank approximation to both matrices. This allows to achieve a compression ratio of $\\frac{mn}{2mj + 2j(a+b)}$. In a sense, this suggested scheme is equivalent (in required parameter size) to a $kj$-rank decomposition ($2j$ in our example). However, if we were to apply $kj$-rank-decomposition immediately, by Eckart-Young-Minsky theorem, we would have a minimal approximation error out of all $kj$-rank decompositions available. \n\nHere is an important question, whether the authors' suggested scheme achieves better than regular SVD approximation when keeping all parameters ($k,j$) fixed? The authors' bound derived by the same Eckart-Young-Minsky theorem does not answer this question, thus I have written a MATLAB script to empirically quantify the approximation errors achieved by the authors' method and by regular SVD.  For a randomly sampled matrices (from a Gaussian), with $k=2$ partitions, $kj$-rank SVD achieves a better approximation error than the author's suggested scheme. This might suggest that author's scheme is inherently less powerful. Certainly, more investigation is needed. (I'm attaching the script below)\n\n\n**Issue 2** The proposed Algorithm 1 lacks details. The algorithm 1, is being dumped on the reader without a sufficient number of details. First of all, how do OpimalRanks and OptimalSubspaces work? Since details are lacking, it is hard to understand whether this consists of convergent iterations (i.e., each step reduces the objective?). Supplemental materials lack the necessary details as well.\n\n\nBeyond these two issues, there are other important ones:\n**Issue 3** Fairness of some of the experimental evaluations. In section 3.1 authors compare to a bunch of baselines, out of which comparison to L-rank seems unfair. The L-rank method requires running an entire pipeline of training, and not of the \"one-shot\" compression method. However, authors seem to run in in a \"one-shot\" setting, in which the method unsurprisingly yields bad results. This is not an apples-to-apples comparison.  At the same time, in table 2,  in the apples-to-apples setting, the quoted L-rank result is on par with the author's method.\n\n**Issue 4** Relevance of theorem 1? Authors spend sufficient time on stating and proving the theorem 1, however, its results are not being used later in the algorithm (at least the details are missing).\n\n\n**Overall**, my current evaluation is between \"reject\" to \"borderline; tending to reject\".\n\nBelow is the Matlab script to quickly assess the proposed compression scheme.  To verify my experimental conclusions mentioned above (that it has worse approximation error when compared to regular SVD when evaluated on randomly sampled $100\\times50$ matrix  with $k=2$ and $j=20$) you can invoke the function as `analyze_proposed_scheme(randn(100,50), 20)`. These findings urge a more in-depth analysis of the scheme's performance.\n```matlab\nfunction analyze_proposed_scheme(X, r)\n    m = size(X,2);\n    authors_min_error = Inf;\n    split_at = Inf;\n    for s=r:m-r\n        a = X(:,1:s);\n        [u1,s1,v1] = svd(a);\n        \n        b = X(:,s+1:end);\n        [u2,s2,v2] = svd(b);\n        scheme_approx = [u1(:,1:r)*s1(1:r,1:r)*v1(:,1:r)' u2(:,1:r)*s2(1:r,1:r)*v2(:,1:r)'];\n        curr_err = norm(X-scheme_approx, 'fro');\n        if curr_err < authors_min_error\n            authors_min_error = curr_err;\n            split_at =s;\n        end\n    end\n    authors_min_error\n    \n    [u,s,v] = svd(X);\n    % the proposed scheme is equivalent in parameter size to kj-rank\n    % decomposition, here we have k=2, and j=r\n    X_svd = u(:,1:2*r)*s(1:2*r,1:2*r)*v(:,1:2*r)';\n    \n    svd_error = norm(X-X_svd, 'fro')\nend\n```", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not believe that the paper has a societal impact that should be addressed.", "review_text": "", "extra_scores": {"ethical_concerns": "I do not see any ethical concerns associated with this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "rW-og9BJAX", "reviewer": "Reviewer_AgkJ", "summary": "The paper propose a novel tensor decomposition pipeline. The propose automatic layer-wise decomposition (ALDC) method comprises two parts: local layer composition and global compression ratio optimization. For local layer decomposition, the weight tensor is first divided into groups and the optimization is within the group. For global compression ratio optimization, the author propose to calculate the relative error bound for each layer and then allocate a layer-wise computation budgets. The experiments are conducted on a various of datasets on image classification and segmentation. The proposed ALDC algorithm is verified over a wide range of network backbones.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong aspects:\n1. The paper is well organized and the method is easy to follow. \n\n2. The proposed method is well justified theoretically.\n\n3. The experiments are comprehensive on the selected backbones and datasets. The results are promising.\n\n4. More importantly, the proposed method do not need training of the network. This might be the main advantages over the NAS algorithms.\n\nWeakness:\n1. The selected backbones are all large and redundant. Experiments on efficient/mobile backbones (e.g. MBV2/EfficientNet/MobileNeXt) will make the proposed algorithm more convincing.\n\n2. The main metric of the proposed method is based on the number of parameters. As decomposing a weight tensor will result a deeper network, I am concerned with the run time latency of the proposed method.\n\n3. The reference format used by the authors is not correct. \n\nI will increase my score if the author could address my concerns sufficiently.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It is better to include some broader impacts in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "Hwo05vXmZhN", "reviewer": "Reviewer_auNv", "summary": "The paper proposes a new layer-wise compression technique to compress neural networks. The new compression method applies channel slicing method to divide convolution layer into k subsets, followed by using SVD method (similar to Denton et al. (2014) method) to compress each layer. The paper further proposes an automatic layer-wise decomposition selector (ALDS) to compress the whole networks with the proposed techniques by optimizing global network compression (equation (8)). Experiment results on Cifar10, ImageNet and VOC compared to the other ow rank method show that the proposed ALDS has a better prediction performance at the considered compression ratio. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is an extension work of Denton et al. (2014) method, which involves new proposed techniques, including channel slicing, SVD approximation, ALDS algorithm, and iterative fine tuning. The algorithms seem to be some adhoc changes compared to Denton et al. (2014) method, particularly when the slicing method is just random slicing. The experiment results, without comparing to the benchmark tensor compression method, might be difficult to convince the effectiveness of the algorithm. \n\nSome questions to the methods used in this paper are\n- Channel slicing methods (line 88-90) in this paper seem to be a random partition of slice into k groups. How is the performance of channel slicing method compared to clustering algorithm based slicing method in model compression? How to select k when applying the compression method in practice?\n- The paper conduct experiment on variance of datasets using bench mark backbone models. However, convolution layer is a tensor and known to be redundant in parameters, and used to be compressed by low rank methods, such low rank tensor methods (Kim et al, 2015, Novikov et al. 2015). Slicing the matrix into parts leveraged the tensor structure of convolutions layers. Without comparing the algorithm with other tensor compression method, it might be difficult to tell the performance gains.\n- Could the author helps to clarify more details about how to find the optimal subspaces k based on layer budget b (Algorithm 1, line 7)?\n- In line 81, the paper claims after SVD decomposition of convolution layer, one convolution layer becomes two small convolution layers. Are there any non-linear transformation considered between the two convolution layers?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not aware of any negative social impact. ", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper suggests a network compression scheme using low-rank approximation of linear layers (either convolutional, or dense).  So far not much new here, except that this paper deals with the combinatorially hard problem of optimizing the matrix decomposition (compression) parameters in all layers simultaneously, so as to minimally sacrifice accuracy.  They suggest an EM-like strategy for solving this optimization problem.

There has been a considerable amount of discussion between the reviewers and the authors, including exchange of code snippets, which is rare.  After reading through this exchange, I became more inclined to accept the paper.  I also think that the optimization problem discussed by the paper does not get enough attention in the network compression literature, and the approach the authors suggest is very reasonable.  Therefore, I choose to recommend acceptance in spite of the average “borderline/reject” score average.

Decision: Accept (Poster)