Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition
Year: 2022

Abstract:
We present a novel global compression framework for deep neural networks that automatically analyzes each layer to identify the optimal per-layer compression ratio, while simultaneously achieving the desired overall compression. Our algorithm hinges on the idea of compressing each convolutional (or fully-connected) layer by slicing its channels into multiple groups and decomposing each group via low-rank decomposition. At the core of our algorithm is the derivation of layer-wise error bounds from the Eckart–Young–Mirsky theorem. We then leverage these bounds to frame the compression problem as an optimization problem where we wish to minimize the maximum compression error across layers and propose an efficient algorithm towards a solution. Our experiments indicate that our method outperforms existing low-rank compression approaches across a wide range of networks and data sets. We believe that our results open up new avenues for future research into the global performance-size trade-offs of modern neural networks.

text: 1 Introduction
Neural network compression entails taking an existing model and reducing its computational and memory footprint in order to enable the deployment of largescale networks in resource-constrained environments. Beyond inference time efficiency, compression can yield novel insights into the design (Liu et al., 2019b), training (Liebenwein et al., 2021a,b), and theoretical properties (Arora et al., 2018) of neural networks.
Among existing compression techniques – which include quantization (Wu et al., 2016), distillation (Hinton et al., 2015), and pruning (Han et al., 2015) – low-rank compression aims at decomposing a layer’s weight tensor into a tuple of smaller low-rank tensors. Such compression techniques may build upon the rich literature on low-rank decomposition and its numerous applications outside deep learning such as dimensionality reduction (Laparra et al., 2015) or spec-
⇤denotes authors with equal contributions. Code: https://github.com/lucaslie/torchprune
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
tral clustering (Peng et al., 2015). Moreover, low-rank compression can be readily implemented in any machine learning framework by replacing the existing layer with a set of smaller layers without the need for, e.g., sparse linear algebra support.
Within deep learning, we encounter two related, yet distinct challenges when applying low-rank compression. On the one hand, each layer should be efficiently decomposed (the “local step”) and, on the other hand, we need to balance the amount of compression in each layer in order to achieve a desired overall compression ratio with minimal loss in the predictive power of the network (the “global step”). While the “local step“, i.e., designing the most efficient layer-wise decomposition method, has traditionally received lots of attention (Denton et al., 2014; Garipov et al., 2016; Jaderberg et al., 2014; Kim et al., 2015b; Lebedev et al., 2015; Novikov et al., 2015), the “global step” has only recently been the focus of attention in research, e.g., see the recent works of Alvarez and Salzmann (2017); Idelbayev and Carreira-Perpinán (2020); Xu et al. (2020).
In this paper, we set out to design a framework that simultaneously accounts for both the local and global step. Our proposed solution, termed Automatic Layer-wise Decomposition Selector (ALDS), addresses this challenge by iteratively optimizing for each layer’s decomposition method (local step) and the low-rank compression itself while accounting for the maximum error incurred across layers (global step). In Figure 1, we show how ALDS outperforms existing approaches on the common ResNet18 (ImageNet) benchmark (60% compression compared to ⇠20% for baselines). Efficient layer-wise decomposition. Our framework relies on a straightforward SVD-based decomposition of each layer. Inspired by Denton et al. (2014); Idelbayev and Carreira-Perpinán (2020); Jaderberg et al. (2014) and others, we decompose each layer by first folding the weight tensor into a matrix before applying SVD and encoding the resulting pair of matrices as two separate layers.
Enhanced decomposition via multiple subsets. A natural generalization of low-rank decomposition methods entails splitting the matrix into multiple subsets (subspaces) before compressing each subset individually. In the context of deep learning, this was investigated before for individual layers (Denton et al., 2014), including embedding layers (Chen et al., 2018; Maalouf et al., 2021). We take this idea further and incorporate it into our layer-wise decomposition method as additional hyperparameter in terms of the number of subsets. Thus, our local step, i.e., the layer-wise decomposition, constitutes of choosing the number of subsets (k`) for each layer and the rank (j`).
Towards a global solution for low-rank compression. We can describe the optimal solution for low-rank compression as the set of hyperparameters (number of subspaces k` and rank j` for each layer in our case) that minimizes the drop in accuracy of the compressed network. While finding the globally optimal solution is NP-complete, we propose ALDS as an efficiently solvable alternative that enables us to search for a locally optimal solution in terms of the maximum relative error incurred across layers. To this end, we derive spectral norm bounds based on the Eckhart-YoungMirsky Theorem for our layer-wise decomposition method to describe the trade-off between the layer compression and the incurred error. Leveraging our bounds we can then efficiently optimize over the set of possible per-layer decompositions. An overview of ALDS is shown in Figure 2.

heading: 2 Method
text: In this section, we introduce our compression framework consisting of a layer-wise decomposition method (Section 2.1), a global selection mechanism to simultaneously compress all layers of a network (Section 2.2), and an optimization procedure (ALDS) to solve the selection problem (Section 2.3).

heading: 2.1 Local Layer Compression
text: We detail our low-rank compression scheme for convolutional layers below and note that it readily applies to fully-connected layers as well as a special case of convolutions with a 1⇥ 1 kernel. Compressing convolutions via SVD. Given a convolutional layer of f filters, c channels, and a 1 ⇥ 2 kernel we denote the corresponding weight tensor by W 2 Rf⇥c⇥1⇥2 . Following Denton et al. (2014); Idelbayev and Carreira-Perpinán (2020); Wen et al. (2017) and others, we can then interpret the layer as a linear layer of shape f ⇥ c12 and the corresponding rank j-approximation as two subsequent linear layers of shape f ⇥ j and j ⇥ c12. Mapped back to convolutions, this corresponds to a j ⇥ c⇥ 1 ⇥ 2 convolution followed by a f ⇥ j ⇥ 1⇥ 1 convolution. Multiple subspaces. Following the intuition outlined in Section 1 we propose to cluster the columns of the layer’s weight matrix into k 2 separate subspaces before applying SVD to each subset. To this end, we may consider any clustering method, such as k-means or projective clustering (Chen et al., 2018; Maalouf et al., 2021). However, such methods require expensive approximation algorithms which would limit our ability to incorporate them into an optimization-based compression framework as outlined in Section 2.2. In addition, arbitrary clustering may require re-shuffling the input tensors which could lead to significant slow-downs during inference. We instead opted for a simple clustering method, namely channel slicing, where we simply divide the c input channels of the layer into k subsets each containing at most dc/ke consecutive input channels. Unlike other methods, channel slicing is efficiently implementable, e.g., as grouped convolutions in PyTorch (Paszke et al., 2017) and ensures practical speed-ups subsequent to compressing the network.
Overview of per-layer decomposition. In summary, for given integers j, k 1 and a 4D tensor W 2 Rf⇥c⇥1⇥2 representing a convolution the per-layer compression method proceeds as follows: 1. PARTITION the channels of the convolutional layer into k subsets, where each subset has at
most dc/ke consecutive channels, resulting in k convolutional tensors {Wi}ki=1 where Wi 2 Rf⇥ci⇥1⇥2 , and Pk i=1 ci = c.
2. DECOMPOSE each tensor Wi, i 2 [k], by building the corresponding weight matrix Wi 2 Rf⇥ci12 , c.f. Figure 3, computing its j-rank approximation, and factoring it into a pair of smaller matrices Ui of f rows and j columns and Vi of j rows and ci12 columns.
3. REPLACE the original layer in the network by 2 layers. The first consists of k parallel convolutions, where the ith parallel layer, i 2 [k], is described by the tensor Vi 2 Rj⇥ci⇥1⇥2 which can be constructed from the matrix Vi (j filters, ci channels, 1 ⇥ 2 kernel). The second layer is constructed by reshaping each matrix Ui, i 2 [k], to obtain the tensor Ui 2 Rf⇥j⇥1⇥1, and then channel stacking all k tensors U1, · · · ,Uk to get a single tensor of shape f ⇥ kj ⇥ 1⇥ 1.
The decomposed layer is depicted in Figure 3. The resulting layer pair has jc12 and jfk parameters, respectively, which implies a parameter reduction from fc12 to j(fk + c12).

heading: 2.2 Global Network Compression
text: In the previous section, we introduced our layer compression scheme. We note that in practice we usually want to compress an entire network consisting of L layers up to a pre-specified relative reduction in parameters (“compression ratio” or CR). However, it is generally unclear how much
each layer ` 2 [L] should be compressed in order to achieve the desired CR while incurring a minimal increase in loss. Unfortunately, this optimization problem is NP-complete as we would have to check every combination of layer compression resulting in the desired CR in order to optimally compress each layer. On the other hand, simple heuristics, e.g., constant per-layer compression ratios, may lead to sub-optimal results, see Section 3. To this end, we propose an efficiently solvable global compression framework based on minimizing the maximum relative error incurred across layers. We describe each component of our optimization procedure in greater detail below.
The layer-wise relative error as proxy for the overall loss. Since the true cost (the additional loss incurred after compression) would result in an NP-complete problem, we replace the true cost by a more efficient proxy. Specifically, we consider the maximum relative error " := max`2[L] "` across layers, where "` denotes the theoretical maximum relative error in the `th layer as described in Theorem 1 below. We choose to minimize this particular cost because: (i) minimizing the maximum relative error ensures that no layer incurs an unreasonably large error that might otherwise get propagated or amplified; (ii) relying on a relative instead of an absolute error notion is preferred as scaling between layers may arbitrarily change, e.g., due to batch normalization, and thus the absolute scale of layer errors may not be indicative of the increase in loss; and (iii) the per-layer relative error has been shown to be intrinsically linked to the theoretical compression error, e.g., see the works of Arora et al. (2018) and Baykal et al. (2019a) thus representing a natural proxy for the cost.
Definition of per-layer relative error. Let W` 2 Rf`⇥c`⇥`1⇥`2 and W ` 2 Rf`⇥c``1`2 denote the weight tensor and corresponding folded matrix of layer `, respectively. The per-layer relative error "` is hereby defined as the relative difference in the operator norm between the matrix Ŵ ` (that corresponds to the compressed weight tensor Ŵ`) and the original weight matrix W ` in layer `, i.e,.
" ` := kŴ ` W `k/kW `k. (1)
Note that while in practice our method decomposes the original layer into a set of separate layers (see Section 2.1), for the purpose of deriving the resulting error we re-compose the compressed layers into the overall matrix operator Ŵ `, i.e., Ŵ ` = [U `1V `1 · · ·U `k`V ` k` ], where U ` i V ` i is the factorization of the ith cluster (set of columns) in the `th layer, for every ` 2 [L] and i 2 [k`], see supplementary material for more details. We note that the operator norm k · k for a convolutional layer thus signifies the maximum relative error incurred for an individual output patch (“pixel”) across all output channels.
Derivation of relative error bounds. We now derive an error bound that enables us to describe the per-layer relative error in terms of the compression hyperparameters j` and k`, i.e., "` = "`(k`, j`). This will prove useful later on as we have to repeatedly query the relative error in our optimization procedure. The error bound is described in the following. Theorem 1. Given a layer matrix W ` and the corresponding low-rank approximation Ŵ `, the relative error "` := kŴ ` W `k/kW `k is bounded by
" `  p k/↵1 ·max
i2[k] ↵i,j+1, (2)
where ↵i,j+1 is the j + 1 largest singular value of the matrix W `i , for every i 2 [k], and ↵1 = kW `k is the largest singular value of W `.
Proof. First, we recall the matrices W `1 , · · · ,W `k and we denote the SVD factorization for each of them by: W `i = Ũ `i ⌃̃`i Ṽ `i . Now, observe that for every i 2 [k], the matrix Ŵ `i is the j-rank approximation of W `i . Hence, the SVD factorization of Ŵ `i can be writen as Ŵ `i = Ũ `i ⌃̂`i Ṽ ` T
i , where ⌃̂`i 2 Rf⇥d is a diagonal matrix such that its first j-diagonal entries are equal to the first j-entries on the diagonal of ⌃̃`i , and the rest are zeros. Hence,
W ` Ŵ ` = [W `1 Ŵ `1 , · · · ,W `k Ŵ `k ] = [Ũ `1(⌃̃`1 ⌃̂`1)Ṽ `1 , · · · , Ũ `k(⌃̃`k ⌃̂`k)Ṽ `k ]
= [Ũ `1 · · · Ũ `k] diag ⇣ (⌃̃`1 ⌃̂`1)Ṽ `1 , . . . , (⌃̃`k ⌃̂`k)Ṽ `k ⌘ .
(3)
By (3) and by the triangle inequality, we have that W ` Ŵ `  h Ũ `1 · · · Ũ `k i diag ⇣ (⌃̃`1 ⌃̂`1)Ṽ `1 , . . . , (⌃̃`k ⌃̂`k)Ṽ `k ⌘ . (4)
Now, we observe that h Ũ `1 · · · Ũ `k i 2 = h Ũ `1 · · · Ũ `k ih Ũ `1 · · · Ũ `k iT = kdiag(k, . . . , k)k = k. (5)
Finally, we show that diag ⇣ (⌃̃`1 ⌃̂`1)Ṽ `1 , . . . , (⌃̃`k ⌃̂`k)Ṽ `k ⌘ = max i2[k] (⌃̃`i ⌃̂`i)Ṽ `i (6)
= max i2[k]
(⌃̃`i ⌃̂`i) = max i2[k] ↵i,j+1, (7)
where the second equality holds since the columns of V are orthogonal and the last equality holds according to the Eckhart-Young-Mirsky Theorem (Theorem 2.4.8 of Golub and Van Loan (2013)). Plugging (7) and (5) into (4) concludes the proof.
Resulting network size. Let ✓ = {W`}L`=1 denote the set of weights for the L layers and note that the number of parameters in layer ` is given by |W`| = f `c``1`2 and |✓| = P `2[L] |W`|. Moreover, note that |Ŵ`| = j`(k`f ` + c``1`2) if decomposed, ✓̂ = {Ŵ`}L`=1, and |✓̂| = P
`2[L] |Ŵ`|. The overall compression ratio is thus given by 1 |✓̂|/|✓| where we neglected other parameters for ease of exposition. Observe that the layer budget |Ŵ`| is fully determined by k`, j` just like the error bound. Global Network Compression. Putting everything together we obtain the following formulation for the optimal per-layer budget:
"opt = min {j`,k`}L`=1
max `2[L]
" `(k`, j`) (8)
subject to 1 |✓̂(k1,j1,...,kL,jL)|/|✓|  CR,
where CR denotes the desired overall compression ratio. Thus optimally allocating a per-layer budget entails finding the optimal number of subspaces k` and ranks j` for each layer constrained by the desired overall compression ratio CR.

heading: 2.3 Automatic Layer-wise Decomposition Selector (ALDS)
text: We propose to solve (8) by iteratively optimizing k1, . . . , kL and j1, . . . , jL until convergence akin of an EM-like algorithm as shown in Algorithm 1 and Figure 2.
Specifically, for a given set of weights ✓ and desired compression ratio CR we first randomly initialize the number of subspaces k1, . . . , kL for each layer (Line 2). Based on given values for each k ` we then solve for the optimal ranks j1, . . . , jL such that the overall compression ratio is satisfied (Line 4). Note that the maximum error " is minimized if all errors are equal. Thus solving for the
Algorithm 1 ALDS(✓, CR, nseed) Input: ✓: network parameters; CR: overall compression ratio; nseed: number of random seeds to initialize Output: k1, . . . , kL: number of subspaces for each layer; j1, . . . , jL: desired rank per subspace for each layer 1: for i 2 [nseed] do 2: k1, . . . , kL RANDOMINIT() 3: while not converged do 4: j1, . . . , jL OPTIMALRANKS(CR, k1, . . . , kL) . Global step: choose s.t. "1 = . . . = "L 5: for ` 2 [L] do 6: b` j`(k`f ` + c``1`2) . resulting layer budget 7: k` OPTIMALSUBSPACES(b`) . Local step: minimize error bound for a given layer budget 8: end for 9: end while 10: "i = RECORDERROR(k1, . . . , kL, j1, . . . , jL) 11: end for 12: return k1, . . . , kL, j1, . . . , jL from ibest = argmini "i
ranks in Line 4 entails guessing a value for ", computing the resulting network size, and repeating the process until the desired CR is satisfied, e.g. via binary search.
Subsequently, we re-assign the number of subspaces k` for each layer by iterating through the finite set of possible values for k` (Line 7) and choosing the one that minimizes the relative error for the current layer budget b` (computed in Line 6). Note that we can efficiently approximate the relative error by leveraging Theorem 1. We then iteratively repeat both steps until convergence (Lines 3-8). To improve the quality of the local optimum we initialize the procedure with multiple random seeds (Lines 1-11) and pick the allocation with the lowest error (Line 12).
We note that we make repeated calls to our decomposition subroutine (i.e. SVD; Lines 4, 7) highlighting the necessity for it to be efficient and cheap to evaluate. Moreover, we can further reduce the computational complexity by leveraging Theorem 1 as mentioned above.
Additional details pertaining to ALDS are provided in the supplementary material.
Extensions. Here, we use SVD with multiple subspaces as per-layer compression method. However, we note that ALDS can be readily extended to any desired set of low-rank compression techniques. Specifically, we can replace the local step of Line 7 by a search over different methods, e.g., Tucker decomposition, PCA, or other SVD compression schemes, and return the best method for a given budget. In general, we may combine ALDS with any low-rank compression as long as we can efficiently evaluate the per-layer error of the compression scheme. In the supplementary material, we discuss some preliminary results that highlight the promising performance of such extensions.

heading: 3 Experiments
text: Networks and datasets. We study various standard network architectures and data sets. Particularly, we test our compression framework on ResNet20 (He et al., 2016), DenseNet22 (Huang et al., 2017), WRN16-8 (Zagoruyko and Komodakis, 2016), and VGG16 (Simonyan and Zisserman, 2015) on CIFAR10 (Torralba et al., 2008); ResNet18 (He et al., 2016), AlexNet (Krizhevsky et al., 2012), and MobileNetV2 (Sandler et al., 2018) on ImageNet (Russakovsky et al., 2015); and on DeeplabV3 (Chen et al., 2017) with a ResNet50 backbone on Pascal VOC segmentation data (Everingham et al., 2015).
Baselines. We compare ALDS to a diverse set of low-rank compression techniques. Specifically, we have implemented PCA (Zhang et al., 2015b), SVD with energy-based layer allocation (SVD-Energy) following Alvarez and Salzmann (2017); Wen et al. (2017), and simple SVD with constant per-layer compression (Denton et al., 2014). Additionally, we also implemented the recent learned rank selection mechanism (L-Rank) of Idelbayev and Carreira-Perpinán (2020). Finally, we implemented two recent filter pruning methods, i.e., FT of Li et al. (2016) and PFP of Liebenwein et al. (2020), as alternative compression techniques for densely compressed networks. Additional comparisons on ImageNet are provided in Section 3.2.
Retraining. For our experiments, we study one-shot and iterative learning rate rewinding inspired by Renda et al. (2020) for various amounts of retraining. In particular, we consider the following unified compress-retrain pipeline across all methods:
1. TRAIN for e epochs according to the standard training schedule for the respective network. 2. COMPRESS the network according to the chosen method.
3. RETRAIN the network for r epochs using the training hyperparameters from epochs [e r, e]. 4. ITERATIVELY repeat 1.-3. after projecting the decomposed layers back (optional).
Reporting metrics. We report Top-1, Top-5, and IoU test accuracy as applicable for the respective task. For each compressed network we also report the compression ratio, i.e., relative reduction, in terms of parameters and floating point operations denoted by CR-P and CR-F, respectively. Each experiment was repeated 3 times and we report mean and standard deviation.

heading: 3.1 One-shot Compression on CIFAR10, ImageNet, and VOC with Baselines
text: We train reference networks on CIFAR10, ImageNet, and VOC, and then compress and retrain the networks once with r = e for various baseline comparisons and compression ratios.
CIFAR10. In Figure 4, we provide results for DenseNet22, VGG16, and WRN16-8 on CIFAR10. Notably, our approach is able to outperform existing baselines approaches across a wide range of tested compression ratios. Specifically, in the region where the networks incur only minimal drop in accuracy ( -Top1 1%) ALDS is particularly effective. ResNets (CIFAR10 and ImageNet). Moreover, we tested ALDS on ResNet20 (CIFAR10) and ResNet18 (ImageNet) as shown in Figure 5. For these experiments, we performed a grid search
over both multiple compression ratios and amounts of retraining. Here, we highlight that ALDS outperforms baseline approaches even with significantly less retraining. On Resnet 18 (ImageNet) ALDS can compress over 50% of the parameters with minimal retraining (1% retraining) and a less-than-1% accuracy drop compared to the best comparison methods (40% compression with 50% retraining).
MobileNetV2 (ImageNet). Next, we tested and compared ALDS on the MobileNetV2 architecture for ImageNet as shown in Figure 6a. Unlike the other networks, MobileNetV2 is a network already specifically optimized for efficient deployment and includes layer structures such as depth-wise and channel-wise convolutional operations. It is thus more challenging to find redundancies in the architecture. We find that ALDS can outperform existing tensor decomposition methods in this scenario as well.
VOC. Finally, we tested the same setup on a DeeplabV3 with a ResNet50 backbone trained on Pascal VOC 2012 segmentation data, see Figure 6b. We note that ALDS consistently outperforms other baselines methods in this setting as well (60% CR-P vs. 20% without accuracy drop).
Tabular results. Our one-shot results are again summarized in Table 1 where we report CR-P and CR-F for -Top1 0.5%. We observe that ALDS consistently improves upon prior work. We note that pruning usually takes on the order of seconds and minutes for CIFAR and ImageNet, respectively, which is usually faster than even a single training epoch.

heading: 3.2 ImageNet Benchmarks
text: Next, we test our framework on two common ImageNet benchmarks, ResNet18 and AlexNet. We follow the compress-retrain pipeline outlined in the beginning of the section and repeat it it-
eratively to obtain higher compression ratios. Specifically, after retraining and before the next compression step we project the decomposed layers back to the original layer. This way, we avoid recursing on the decomposed layers.
Our results are reported in Table 2 where we compare to a wide variety of available compression benchmarks (results were adapted directly from the respective papers). The middle part and bottom part of the table for each network are organized into low-rank compression and filter pruning approaches, respectively. Note that the reported differences in accuracy ( -Top1 and -Top5) are relative to our baseline accuracies. On ResNet18 we can reduce the number of FLOPs by 65% with minimal drop in accuracy compared to the best competing method (MUSCO, 58.67%). With a slightly higher drop in accuracy (-1.37%) we can even compress 76% of FLOPs. On AlexNet, our framework finds networks with -0.21% and -0.41% difference in accuracy with over 77% and 81% fewer FLOPs. This constitutes a more-than-10% improvement in terms of FLOPs compared to current state-of-the-art (L-Rank) for similar accuracy drops.
3.3 Ablation Study
To investigate the different features of our method we ran compression experiments using multiple variations derived from our method, see Figure 7. For the simplest version of our method we consider a constant per-layer compression ratio and fix the value of k to either 3 or 5 for all layers denoted by ALDS-Simple3 and ALDS-Simple5, respectively. Note that ALDS-Simple with k = 1 corresponds to the SVD comparison method. For the version denoted by ALDS-Error3 we fix the number of subspaces per layer (k = 3) and only run the global step of ALDS (Line 4 of Algorithm 1) to determine the optimal per-layer compression ratio. The results of our ablation study in Figure 7 indicate that our method clearly benefits from the
combination of both the global and local step in terms of the number of subspaces (k) and the rank per subspace (j).
We also compare our subspace clustering (channel slicing) to the clustering technique of Maalouf et al. (2021), which clusters the matrix columns using projective clustering. Specifically, we replace the channel slicing of ALDS-Simple3 with projective clustering (Messi3 in Figure 7). As expected Messi improves the performance over ALDS-Simple but only slightly and the difference is essentially negligible. Together with the computational disadvantages of Messi-like clustering methods (unstructured, NP-hard; see Section 2.1) ALDS-based simple channel slicing is therefore the preferred choice in our context.

heading: 4 Related Work
text: Our work builds upon prior work in neural network compression. We discuss related work focusing on pruning, low-rank compression, and global aspects of compression.
Unstructured pruning. Weight pruning (Lin et al., 2020b; Molchanov et al., 2016, 2019; Singh and Alistarh, 2020; Wang et al., 2021; Yu et al., 2018) techniques aim to reduce the number of individual weights, e.g., by removing weights with absolute values below a threshold (Han et al., 2015; Renda et al., 2020), or by using a mini-batch of data points to approximate the influence of each parameter on the loss function (Baykal et al., 2019a,b). However, since these approaches generate sparse instead of smaller models they require some form of sparse linear algebra support for runtime speed-ups.
Structured pruning. Pruning structures such as filters directly shrinks the network (Chen et al., 2020; Li et al., 2019b; Lin et al., 2020a; Liu et al., 2019a; Luo and Wu, 2020; Ye et al., 2018). Filters can be pruned using a score for each filter, e.g., weight-based (He et al., 2018, 2017) or data-informed (Liebenwein et al., 2020; Yu et al., 2018), and removing those with a score below a threshold. It is worth noting that filter pruning is complimentary to low-rank compression.
Low-rank compression (local step). A common approach to low-rank compression entails tensor decomposition including Tucker-decomposition (Kim et al., 2015b), CP-decomposition (Lebedev et al., 2015), Tensor-Train (Garipov et al., 2016; Novikov et al., 2015) and others (Denil et al., 2013; Ioannou et al., 2017; Jaderberg et al., 2014). Other decomposition-like approaches include weight
sharing, random projections, and feature hashing (Arora et al., 2018; Chen et al., 2015a,b; Shi et al., 2009; Ullrich et al., 2017; Weinberger et al., 2009). Alternatively, low-rank compression can be performed via matrix decomposition (e.g., SVD) on flattened tensors as done by Denton et al. (2014); Sainath et al. (2013); Tukan et al. (2020); Xue et al. (2013); Yu et al. (2017) among others. Chen et al. (2018); Denton et al. (2014); Maalouf et al. (2021) also explores the use of subspace clustering before applying low-rank compression to each cluster to improve the approximation error. Notably, most prior work relies on some form of expensive approximation algorithm – even to just solve the per-layer low-rank compression, e.g., clustering or tensor decomposition. In this paper, we instead focus on the global compression problem and show that simple compression techniques (SVD with channel slicing) are advantageous in this context as we can use them as efficient subroutines. We note that we can even extend our algorithm to multiple, different types of per-layer decomposition.
Network-aware compression (global step). To determine the rank (or the compression ratio) of each layer, prior work suggests to account for compression during training (Alvarez and Salzmann, 2017; Ioannou et al., 2016, 2015; Wen et al., 2017; Xu et al., 2020), e.g, by training the network with a penalty that encourages the weight matrices to be low-rank. Others suggest to select the ranks using variational Bayesian matrix factorization (Kim et al., 2015b). In their recent paper, Chin et al. (2020) suggest to produce an entire set of compressed networks with different accuracy/speed trade-offs. Our paper was also inspired by a recent line of work towards automatically choosing or learning the rank of each layer (Gusak et al., 2019; Idelbayev and Carreira-Perpinán, 2020; Li and Shi, 2018; Tiwari et al., 2021; Zhang et al., 2015b,c). We take such approaches further and suggest a global compression framework that incorporates multiple decomposition techniques with more than one hyper-parameter per layer (number of subspaces and ranks of each layer). This approach increases the number of local minima in theory and helps improving the performance in practice.

heading: 5 Discussion and Conclusion
text: Practical benefits. By conducting a wide variety of experiments across multiple data sets and networks we have shown the effectiveness and versatility of our compression framework compared to existing methods. The runtime of ALDS is negligible compared to retraining and it can thus be efficiently incorporated into compress-retrain pipelines.
ALDS as modular compression framework. By separately considering the low-rank compression scheme for each layer (local step) and the actual low-rank compression (global step) we have provided a framework that can efficiently search over a set of desired hyperparameters that describe the lowrank compression. Naturally, our framework can thus be generalized to other compression schemes (such as tensor decomposition) and we hope to explore these aspects in future work.
Error bounds lead to global insights. At the core of our contribution is our error analysis that enables us to link the global and local aspects of layer-wise compression techniques. We leverage our error bounds in practice to compress networks more effectively via an automated rank selection procedure without additional tedious hyperparameter tuning. However, we also have to rely on a proxy definition (maximum relative error) of the compression error to enable a tractable solution that we can implement efficiently. We hope these observations invigorate future research into compression techniques that come with tight error bounds – potentially even considering retraining – which can then naturally be wrapped into a global compression framework.

heading: Acknowledgments
text: This research was sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. This work was further supported by the Office of Naval Research (ONR) Grant N00014-18-12830.
Funding Transparency Statement
Authors declare no competing interests. Funding in direct support of this work: the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator accomplished under Cooperative Agreement Number FA8750-19-2-1000 and the Office of Naval Research (ONR) Grant N00014-18-1-2830.

Review summaries:
The paper proposes a one-shot approach to compress already trained neural networks using low-rank compression. The authors propose a new scheme of applying low-rank: instead of applying it directly to a layer, authors suggest partitioning the filters into K blocks, and apply low-rank to each of these blocks separately. Haven defined this scheme, authors then propose a one-shot alternating-optimization algorithm to select ranks and to partition matrix columns/filters into K blocks. Once ranks and partitions are determined, the matrices are decomposed and finetuned. Optionally, this entire compress=>finetune process can be repeated multiple times. Authors present numerical results where the suggested approach almost always wins, however some of the comparisons are questionable. In terms of the algorithm itself, the details of alt.opt are missing, as well as theoretical analysis of the proposed compression scheme is missing.

The paper proposes a new layer-wise compression technique to compress neural networks. The new compression method applies channel slicing method to divide convolution layer into k subsets, followed by using SVD method (similar to Denton et al. (2014) method) to compress each layer. The paper further proposes an automatic layer-wise decomposition selector (ALDS) to compress the whole networks with the proposed techniques by optimizing global network compression (equation (8)). Experiment results on Cifar10, ImageNet and VOC compared to the other ow rank method show that the proposed ALDS has a better prediction performance at the considered compression ratio. 

The paper propose a novel tensor decomposition pipeline. The propose automatic layer-wise decomposition (ALDC) method comprises two parts: local layer composition and global compression ratio optimization. For local layer decomposition, the weight tensor is first divided into groups and the optimization is within the group. For global compression ratio optimization, the author propose to calculate the relative error bound for each layer and then allocate a layer-wise computation budgets. The experiments are conducted on a various of datasets on image classification and segmentation. The proposed ALDC algorithm is verified over a wide range of network backbones.