Model Preserving Compression for Neural Networks
Authors: Jerry Chee
Megan Renz
Anil Damle
Christopher De Sa
Venue: NeurIPS 2022 Accept (2022)
Keywords: interpolative decomposition
compression
neural network

Abstract: After training complex deep learning models, a common task is to compress the model to reduce compute and storage demands. When compressing, it is desirable to preserve the original model's per-example decisions (e.g., to go beyond top-1 accuracy or preserve robustness), maintain the network's structure, automatically determine per-layer compression levels, and eliminate the need for fine tuning. No existing compression methods simultaneously satisfy these criteria---we introduce a principled approach that does by leveraging interpolative decompositions. Our approach simultaneously selects and eliminates channels (analogously, neurons), then constructs an interpolation matrix that propagates a correction into the next layer, preserving the network's structure. Consequently, our method achieves good performance even without fine tuning and admits theoretical analysis. Our theoretical generalization bound for a one layer network lends itself naturally to a heuristic that allows our method to automatically choose per-layer sizes for deep networks. We demonstrate the efficacy of our approach with strong empirical performance on a variety of tasks, models, and datasets---from simple one-hidden-layer networks to deep networks on ImageNet.


Model Preserving Compression for Neural Networks
Jerry Chee
Megan Flynn (née Renz)' Department of Physics Cornell University mr mr2268@cornel1..
Department of Computer Science
Cornell University wtwpcchee@ccs.ccnnlnlle
Anil Damle
Christopher De Sa Department of Computer Science
Department of Computer Science
Cornell University damle@cornell. edu
Coreell University edu
Abstract
After training complex deep learning models, a common task is to compress the model to reduce compute and storage demands. When compressing, it desirable to preserve the original model's per-example decisions (e.g., to go beyond top-1 accuracy preserve robustness), maintain the network's s structure, automatically determine per-layer compression levels, and eliminate the need for fine tuning. No existing compression methods simultancously satisfy these criteria introduce a principled approach that does by leveraging interpolative decompositions. Our approach simultaneously selects and eliminates channels (analogously, neurons), then constructs an interpolation matrix that propagates a correction into the next layer, preserving the network's s structure. Consequently, our method achieves good performance even without fine tuning and admits theoretical analysis. Our theoretical generalization bound for one layer network lends itself naturally to a heuristic that allows our method to automatically choose per-layer sizes for deep networks. We demonstrate the efficacy of our approach with strong empirical performance variety of tasks, models, and datasets ffom simple one-hideen- layer networks to deep networks on ImageNet.
Introduction
There has recently been a significant theoretical and empirical effort to understand the necessity of over-parameeeriizatoon for deep learning models and to develop algorithmic techniques that can effectively compress (or prune) them while retaining performance. This work is largely driven by results that show over-parameterizztion may aid in training via stochastic algorithms, but that the and therefore can be compressed. Typically the focus has been on reducing parameter counts or FLOPs while maintaining accuracy- task that often necessitates significant fine tuning of compressed models. However, we argue that further criteria are warranted to enable the adoption of compression methods amongst practitioners: (1) preserving the model's 's per-example decisions ensures the preservation of more fine-grain properties. e.g. fairness 4 [14 17 24 61 or adversarial robustness 1
solutions found are often highly redundant [1 22 2 S
16
68 (2) compressing networks 18
while retaining their computational structure ensures compatibility with the rest of the machine learning pipeline; and (3) automatically determining per-layer sizing and minimizing the need for fine tuning alleviates key challenges when using compression methods in practice.
"qqual contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

We develop a novel compression method that satisfies these three criteria. In particular, it uses unlabeled data to remove redundant neurons/channels while simultaneously updating the remaining network parameters to explicitly correct for the removed neurons/channels. The core technical development in our pruning method the novel use of an interpolative decomposition (see Section B to approximate the post-activation output of layers. The interpolative decomposition (ID) explicitly represents the pruned neurons/channels as linear combination of those the model with little to no fine tuning. Moreover the specific structure of the ID allows to sub-select channels backward through the activation function, and propagate information about the pruned channels forward, fusing it into the adjacent layer and preserving the structure of the network (albeit with smaller layer sizes). Collectively, these properties allow our ID based compression approach to produce structurally similar compressed network that well approximates (to any desired accuracy) the original network without any additional training- feature we show theoretically for simple fully connected networks in Section
us to preserve
4 We then extend the results different layer types and more complex architectures in Section 5 and show how our theoretical results allow us to automatically determine per-layer sizes for compressed model based on the trade-off between FLOPs and the estimated error induced by pruning.
We ensure that our method truly compresses network and preserves the decision boundaries, rather than effectively re-training a smaller network through extensive fine tuning through a new metric we develop in Section 6 Consider pruning pretrained VGG16 model on CIFAR-10 10 with 93.6% accuracy to 50% of its FLOPs. A variety of pruning methods achieve a reasonable accuracy of 93.2-93.6%, including structured magnitude pruning, random channel pruning, and our ID-based method. However, methods which rely extensively on fine-tuning (magnitude and random pruning) do not preserve the decision boundaries of the original network. 'he predictions of these pruned models agree with the original model only (93.2%, 93.2% respectively) of the time on the test set. This level of agreement is barely better than a completely independently trained (fullsize) VGG16 model (93.0%). In contrast, our method produces a compressed model whose predictions are highly correlated with the original model, agreeing 97.3% of the time.
To complement our algorithmic developments and theoretical contributions, in Section 7 we demon- strate the efficacy of our method on Atom3D 772. CIFAR- 10 [3], and ImageNet [I9 The experi-
ments are built to highlight how our ID-based compression method satisfies the desired criteria while competing methods do not. To accomplish this, first introduce metric of model similarity based on prediction correlation this metric allows us to systematically determine how well compressed models are preserving the original model. To highlight the practical efficacy of our method (partic- ularly its ability to automatically chose the size of the compressed network) we show our method successfully extends to Atom3D 722) simulating a real-world scenario with little prior compression work and baseline models with a different architecture (based on 3D convolutions). More generally, on all datasets we show our method has superior accuracy before fine tuming and is competitive with other of-ee-of-thh-rr pruning methods after additional training is performed all while producing compressed models which better correlate with the original. Moreover, because our method only reduces layer sizes but otherwise preserves network structure, it can be easily composed with other compression methods. This allows us to substantially compress models for CIFAR-10 and ImageNet without the use of any global fine tuning.
In summary, the key contributions of this pape are:
A model preserving compression method We introduce an ID-based algorithm for compressing neural networks that preserves the model's decisions and maintains the network's 's layer structure without the need for fine tuning.
Theoretical guarantees We provide theoretical guarantees for the output of our pruned model.
Practical efficacy We demonstrate the efficacy of our method across the ATOM3D, CIFAR-10 and ImageNet data sets and using variety of models. Our method requires no labeled data, automatically determines per-layer sizes, and often requires little to no fine tuning.
Consider model A. Another model B of the same architectures is independently trained if during training no information from is passed to B. Essentially this means retraining the same architecture from scratch twice to obtain models A and B.
Our code is available at hutp:y/ghhhccooeeyeeeeeeolleeeer

---

2
Related Work
There are number of different design choices to be made in the compression and pruning process. Classical pruning involves eliminating either channels (analogously neurons) or individual weights, sometimes in a structured way. Magnitude pruning on both weights and neurons is still considered an effective approach 6 21
23
25
When pruning, the method may incorporate a correction to Some methods that correct the network chose to do 66 52 whereas others do not 48 33
5
46
future layers, though it often does not B1
54
local fine tuning [54
often satisfy the first criteria we desire for compression methods, but typically not the second as they add additional layers. These methods sometimes incorporate local fine tuning after compression 40
Of particular note, matrix approximation methods 20
40
50
42
50,
35
rirl and
sometimes do not [220 44]. In contrast, structured pruning methods [i]
48
can satisfy
the second criteria we outlined, but typically not the first as they do a poor job of preserving the model's i decisions and often require excessive amounts of fine tuning.
Pruning with coresets [64] is the closest in spirit to our own work and provides a way to select subset of neurons in the current layer that can approximate those in the next layer as well as new weight connections. Of note, Mussay et al. [64] provide a sample complexity result, and demonstrate their method on fully connected (but not convolution) layers. The HRank method [51] is also close in spirit to our own, and works by selectively pruning channels that produce low-rank feature maps. However, the method does not propagate updates into the next layer and instead relies on excessive amounts of fine tuning (30 epochs for each layer pruned) to fix the network's 's accuracy.
Recently the literature has started to consider criteria beyond topline accuracy metries, and Lieben- wein et al. 4o use measures of functional approximation to conclude that pruned networks well approximate the original models. Marx al. 660 characterize when linear models can achieve similar accuracy but with competing predictions.
3 Interpolative decompositions
Our pruning strategy relies on structured low-rank approximation known as an interpolative decomposition (ID). Classically, the Singular Value Decomposition (SVD) (see, e.g.. (26)) provides an optimal low-rank approximation. However, because consider matrices that include the non- linear activation function the SVD cannot be directly used to either subselect neurons or generate new ones (since it is unclear how to propagate singular vectors "backwards" through the non-linearity). In contrast, an ID constructs a structured low-rank approximation of a matrix A where the basis used for the upproximation constrained be subset of the columns of A. For a matrix A E R= we let A denote sub-selection of the matrix A using sets J C [n]] to denote the selected rows and I [m] denote the selected columns: denotes selection of all rows or columns.
Definition 3.1 (Interpolative Decomposition). Let A E TRnx e 2 0. An e-accurate interpolative decomposition A ~ 7 is subset of columns of A, denoted with the index subset I c [m], and an associated interpolation matrix T such that || A S Al|Al|/2
Remarks 3.2. When computing an ID, we would like to find the smallest possible k = [Z] such that the accuracy requirement is satisfied. Moreover, we would like T to have entries of reasonable magnitude and approximation error not much larger than the best possible for given k (i.e.,
||A A, T2|2 = t (A) for some small t 2 1). While necessarily sub-optimal, the advantage is that we explicitly use subset of the columns of A to build the approximation.
IDs are well studied 13 59 widely used in the domain of rank-structured matrices
35
58
62]. and are closely related to CUR decompositions [56] 74 and subset selection problems 7
15
73
While these decompositions always exist, finding them optimally a difficult task and in this work we appeal to what are known as (strong) rank rank-revealing QR factorizations 88
28 37].
QR factorization). Let A E RnXm, 6 = min(1. r2, m). and take any QR factorization of A computes a permutation matrix IT E Rmxm, an upper- (i.e. R,,j 0 i > j), and matrix Q E Rxx with orthonormal columns (i.e., Q Q = I) such that AlI QR and Q and R satisfy certain properties. Splitting 11, Q, and R into 11, E pmxk Il, E (m-)m))), Q1 E Ra Q2 gax((-i), Rui E R
Definition 3.3
k s E. A trapezoidal matrix R E ex
3

---

R12 E R and R22 E (R- (m) k)) we can write
R11
R12 R22
111
11l
[Q1
Q2
Remarks 3.4. What makes ① raraa-neeealiinn revealing QR factorization is that the permutation II is computed to ensure that R11 is as well-conditioned as possible and R22 is as small as possible. While more formal statements of these conditions exist, we omit them here as they do not factor into our work.
Critically, any rank-revealing QR factorization yields natural rank-k approximation of A with emror
||A - Q1 [R11 R22] |l2 = ||
While finding the optimal rank-revealing QR factorization (i.e., minimizing the error for a given k) is closely related to a provably hard problem [15], we find the original algorithm of Businger and Golub [ works well in practice. This routine is available in LAPACK 12. 67, can be easily incorporated into existing code, and has computational complexity (mmmk) when for & steps.
Computing interpolative decompositions Given a rank-revealing QR factorization, we can imme- diately construct ID (a formal algorithmic statement is given in the appendix). Let I [m] be such that A. = AlI, and define the interpolation matrix
T= Tx
R R12 11
With the choice A, Q1 it follows that the error of the ID as defined by I and T is ||A4 A.rTila = R22 ll.. Picking K such that || l2 S ell-A|la yields the desired relative error. Notably, since x(A,z those of a rank-revealing QR factorization-it R1 is well conditioned then the basis we use for approximation as well and entries of T are not too large. If maax (Rz2) is not much larger than (A) we get near optimal approximation accuracy.
x(I1) and T I Ri R12 11 the desired criteria for an ID map back to
Accuracy of the matrix approximation A key feature of using a column-pivotee QR factorization to compute an ID is that it allows us to dynamically determine the approximation rank k as function of This can be accomplished by monitoring || |Ra l|a at each step of the column-pivoted QR algorithm. However, repeatedly computing Rzall2 is expensive and often unnecessary in practice. When using the algorithm by Businger and Golub [8] the magnitude of the diagonal entries of R are non-increasing and it is common to use | 1/11,1 as proxy for || | While formal bounds indicate the approximation may be loose in the worst case, it is effective
practice (see appendix Figure D and once candidate & has been identified || [222ll can be computed if desired to certify the result- if the accuracy is unacceptable & can increased until it is. In some settings may be preferable to fix & and simply accept whatever accuracy is achieved.
4 Pruning with interpolative decompositions
The core of our approach is a novel use of IDs to prune neural networks. Here we illustrate the scheme for single fully connected layer and we extend the scheme to more complex layers (e.g.. convolution layers) and deeper networks in Section 5 Consider a simple two layer (one hidden layer) fully connected neural network hrc: IRd Re of width defined as
hFo W,U) U g(WTx>)
with hidden layer W E Rdxmm output layer U E mx and activation function g. We omit bias terms: they may be readily incorporated by adding a row to W and suitably augmenting the data.
To prune the model we will use an unlabeled pruning data set {2;) with 2; E Rd Let X e ddx= be the matrix such that X = I;, Preserving the action of the two layer network to accuracy O on the data with fewer neurons is synonymous with finding an accurate approximation
hac(a; W. U) (a; W, i where W has fewer columns than W. We can do this by computing an ID of the activation output of the first layer.
Concretely, let Z E Rmxx be the first-layer output, i.e., 2 =g(WTX X), and let TT ~ (., be rank-k ID of 2T with [ = k and interpolation matrix T E kx that achieves accuracy in

---

Definition 3.1 (note that this means l: is a function of c), Because the activation function g commutes with the sub-selection operator, if Z T Zr. then
g(W x) ~ T (o(W x)) = =T' (W
Multiplying both sides by U now gives an approximation of the original network by a pruned one, hFC (1; W.U) = U g(W X) x hFc (æ; W.,,, = UTT (W.Ix)
(2) That is, the ID has pruned the network of width m into a dense sub-network of width with W = W Raxk and Ü TU Rx Importanll, the SVD of 2 cannot be used for this task since it is not clear how to map the dominant left singular vectors back through the activation function g to either subset of the existing neurons small set of new neurons. This makes use of the ID essential and we provide additional intuition for this scheme the appendix, specifically in Figure
4.1 A generalization bound for the pruned network
key feature our ID based pruning method is that it can be used to dynamically select the width of the pruned network maintain desired accuracy when compared with the full model. This allows us to provide generalization guarantees for the compressed network in terms of generalization of the full model and the accuracy of the ID. We state the results for single hidden fully connected layer with scalar output (i.e., = 1) and squared loss. They can be extended to more complex networks (at the expense of more complicated dependence on the accuracy each layer is pruned to), more general Lipschitz continuous loss functions, and vector valued output. We defer all proofs to the supplementary material.
Assume (Ir, y) ~ D where x E Rdd, E IR, and the distribution D is supported on a compact domain Og Sgy We let Rs = ((l g(WTz) - y) denote the true risk of the trained full model and R2 (( g(W r))l²) be the risk of the pruned model. (Since 1 let E RR denote the last layer.) We also define the empirical risk RID of approximating the full model with our pruned model as
RID
u g(W 2;) a. g( 2.)
where J are i.i.d. samples from D (note that we do not need labels for these samples). Using this notation, Theorem 4.1 controls the generalization error of the pruned model.
Theorem 4.1 (Single hidden layer FC). Consider model hFc u 9(W z)) with hidden neurons and a pruned model hFc aT g( x) constructed using an e accurate ID with n data points drawn i.i.d from D. The risk of the pruned model R, on a data set (z, y) ~ > assuming 'D is compactly supported on Q2y x Qt is bounded by
R2 < RID + Ro + RIDRRo
(3)
where RID is the risk associated with approximating the full model by pruned one and with probability 1 8 satisfies
(V2cdm log (dmlog(ddm)
log(1/8)
RID 2 M M(1 1(1+1TT1)3> n
en
Here, M suppze. || 1l2 and Ç a universal constant that depends on g-
Theorem is developed by considering the ID as learning algorithm applied to the output of the full model using unlabeled pruning data. This allows us to control the risk of the pruned model in terms of the risk of the original model and the additional risk introduced by the ID. Importantly, here control the additional risk in terms of the empirical risk of the ID and an additive term that decays as additional pruning data is used. Lemma codifies this decomposition.
Lemma 4.2. Under the assumptions of Theorem for any ó E (0, 1), RID satisfies
RID S RID M + 1
log(en/p))) +2 - 1022///og((//0000 /llo(1/8)) x))
with probability 5, where M sup,en, (ul?)l/(WT>>)]1 and p Çdm log(d)) for some universal constant that depends only the activation function.
5

---

Remarks 4.3. We believe that the second part of the bound in Lemma 4.2 is likely loose since it relies on a pseudo neeudo-imennioon bound for fully connected neural networks. However, when pruning with an ID we only consider subsets of existent neurons and it is plausible that in this setting the upper bound for the pseudo-dimension could be improved.
Crucially, an immediate consequence of using an ID for pruning is that we can explicitly control RID in terms of the accuracy parameter. This relation between the ID accuracy and empirical risk is given Lemma 4.4 and what allows us to express the risk of the pruned network Theorem 4.1
Lemma 4.4. Following the notation of Theorem 4.1 an ID pruning to aceuracy e yields a compressed
network that satisfies RID I @ll [lg X) 12 n, where X E Rdxxa a matrix whose columns are the pruning data.
5
Convolutional and deep networks
5.1 Convolution layers
To prune convolution layers with the ID at the channel level reshape the output tensor into matrix where each column represents a single output channel. After this transformation, the key idea is the same as in Section 4 Consider simple two layer convolution neural network defined as heoonv(2r W, U)- conv (U, g (conv (W))))), with the convolution-layer operator conv, weight tensors W and U, and activation function g. The kernel dimension, size, stride, padding, and dilation do not change the form of the ID at the output channel level. Let Z = g(conv (oovoW, (W.X)) be the activation output of the first layer with unlabeled pruning data X, and define reshape as the operator which reshapes tensor into a matrix with the output channels as columns, i.c., reshape(2) Z) R x
where m. is the number of channels and n, the product of all other dimensions (e.g. in the case of a 2d convolution, ni would be width x height x number of examples).
We now compute rank-k ID reshape Z) z reshape( ZT. The activation function g and reshape operator both commute with the sub-selection operator, so
Prshapp(g(conv(W.. (W, = reshape(g(conv (W)X))) reshape(g(conn(WW X)))T,
where Wz... denotes an indexing sub-selection of W along its output-channel dimension. Next, we need to propagate this T into the next layer, which we can do with a "matrix multiply" by the next-layer's weights along its input channel dimension: call this operation matmul With this, little algebraic manipulation of our approximate equality above gives
conv (U, g ((conv x)))) z conv(matmulff U), g g(c.nv(WZ..... X))))
(4)
and so if set Ū matmul (T,U) and W Wz..., we can preserve the action of the two layer network with fewer channels as heony (x; W, U') Pe heonv (x; W. Ú). This gives us a recipe for pruning convolution layers analogous to 2 This recipe can be directly applied to composition of a convolution layer followed by pooling layer (27] (or any other layer that acts independently by channel) by treating the conv layer/ pooling layer pair as single convolution layer with "fancy" activation function g: we just run the ID on the output post-pooling, and use that to sub-select the convolution layer's weights. Flatten layers, for connecting to FC layers, equally straightforward.
5.2
Deep networks
The ID pruning primitives for fully connected and convolution layers can now be composed together to prune deep networks. Algorithm | specifies how chain together the fully connected and convolution primitives to prune feedforward networks, for simplicity we assume for the moment we know the desired layer sizes. A multi-layer neural network is pruned from the beginning the end, where the ID is used to approximate the outputs of the original network. The ID pruning primitives
'When is large 'If E mmx"" and U is a weightttensso with n input channels, then to compute matmul(T', U) we: (1) reshape U to be an p matrix for some (2) multiply the reshaped matrix by T. producing m p matrix, and (3) reshape the result back to tensor with input channels and all other dimensions the same U.
appeal randomized ID algorithms 59
the TSQR 3.

---

Algorithm Pruning a multilayer network with interpolative decompositions
Input: Neural net h(x; W() 1.... (,()) layers prune [7.], pruning set X, pruning fraction
Output: Pruned network h(2; W(1) ..... W())
1: T t
2: for / (1... do
3:
Z + (X; W W() [layer activations]
4:
if layer is layer then
5:
(I. (I,T
aa) if & S else (:.4)
6:
W <- w (sub-select neurons, multiply T of prev layer's ID)
7:
else if layer is Conv layer (or Conv+Pool) then
(I. T()) (D((resaaeee))::e ()) ifl S else (t,
8:
W <- mattmul(Tt-1) w [select channels; multiply T]
9:
10: else if layer l a Flatten layer then
11: T + T(-- @ {expand T to have the expected size}
12: end if
13:
end for
sub-select neurons (or channels) in the current layer and propagate the corrective interpolation matrix to the next layer. There are many ways one could prune a multi-layer network with these ID pruning primitives; we selected the approach in Algorithm throuuhh empirical observations (though do not assert that is optimal). Note that as pre-processing step before running Algorithm | batch normalization layers 41 should be absorbed into their preceding fully-connected convolution layers, and dropout
70
layers should be removed.
Iterative Pruning While Algorithm|] D illustrative, in practice we would often like be able to either specify a desired accuracy or choose layer sizes optimally for a desired compression ratio. Our approach allows us accomplish this by iteratively selecting layers to compress. We introduce score function for layers that is the ratio of the estimated relative error "'+11841/r1.11 introduced by compressing a layer the number of flops fi that would be cut if we pruned layer to size and it is heuristic for the compressability of each
k. We call this score st (k) 1T--1*11/T1a1/ layer-lower scores imply a layer is easier to compress. However, different layers of the network are connected, and compressing layer early in the network can effect how well later layers can be compressed. Therefore, we prune the network iteratively, measuring the score for each layer at a given pruning percentage (or step size), choosing the layer with the lowest score, pruning and then re-calculating the scores for the later layers. We repeat the process until the network reaches a desired compression, or until the network performance degrades unacceptably. We refer to this method as Iterative ID, and refer to cutting a constant fraction of neurons in each layer as Constant Fraction ID. For full details see Appendix F and Algorithm
6 Evaluating compression beyond accuracy
important benefit of our approach that we are actually able to preserve the original model's predictions better than other methods. Traditional pruning methods typically do poor job at preserving the original predictions, due their heavy reliance on fine tuning that effectively retrains the model. Here we explain why we might care about preserving per-example predictions beyond top-line accuracy. We argue that in many situations compression methods must well-approximate the original model, and that accuracy is poor metric for this use case.
Consider pretrained model M, resulting "compressed" model Mc, and an evaluation (X, Y). Accuracy measures the similarity between the true labels Y and the predicted labels from the compressed model (XX. This metric does not directly compare the original model M and compressed model Mc. As we have seen in Section I compressed model can recover the accuracy of the original model, but still differ widely on the predictions. Instead our model correlation measures the similarity between the original model's predictions M(X and the compressed model's predictions MXX)). One can think of this metric as an "accuracy to the original predictions". instead of accuracy to the true labels.
7

---

We propose model correlation as the percent of test example predictions two models agree on de-details of this metric are discussed in Appendix D Model correlation general metric measure the similarity between the learned functions of two models. In other contexts people have explored the related concept of model fidelity, where our correlation metric matches the top-1 agreement of Stanton et al. 77. Our claim is that by better preserving model's s per-example decisions, we can better preserve special properties of the model. In the following section we provide experimental evidence for this claim. Models are now often trained to have properties that go beyond test accuracy example robustness to adversarial attacks, sub-class classification accuracy, fairness, etc., and this measure of model correlation is likely to correlate with many of these criteria. Note that do not believe preserving per ver-example decisions "boosts" any of these properties, we are simply preserving properties of the baseline model.
7
Experiments
The Appendix provides a list of all methods we compare with (including references) and a lookup table to find the relevant experiments for each method (i.e., which figures and tables they appear in).
More careful evaluation of pruning methods We propose range of evaluations that goes beyond simple post-fine tuning test accuracy on standard vision benchmarks. First is our proposed model correlation metric, discussed in detail in Section 6 While certain benchmark problems are well studied and the literature has characterized layer sizes for more efficient networks, that cannot be relied upon for realistic problems practitioners want to solve. Therefore, we demonstrate the usability of our method by applying it to a non-stame ntn-standard problem and automatically discover a significantly more efficient allocation of flops per layer. We also consider extensive pre-fine tuning evaluations. In many cases, we may produce results before fine tuning that are sufficiently strong to remove the need for it. Moreover, fine tuning often washes out differences between different methods and reduces model correlation (particularly if the learning rate is too high for too many epochs). We show that by preserving model correlation our method is able preserve the sub-class accuracy of a class which has been removed from the pruning and fine tuning set. Lastly, we show how maintaining network structure allows us compose our methods with other caarix-decomposition based techniques and provide post-fine tuming results. One hidden layer results can be found in the Appendix.
Broad applicability We demonstrate the versatility of our method on atypical architectures and beyond vision tasks using the ATOM3D Ligand Binding Affinity (LBA) [72 benchmark. This simulates a more typical use case for pruning than benchmark vision models. The LBA task predicts the binding strength between proteins and small molecules which is useful for drug discovery. We train and prune the 3DConv network from 772,, which uses 3-D convolutional layers, and achieves a RMSE of 1.42. We use the same method to achieve baseline RMSE of 1.43, and prune the network using our Iterative ID. The baseline network uses 12.8G FLOPS per example. For comparison VGG-16 on Imagenet uses 30G FLOPs. Figure D shows that Iterative ID is able to prune 95% of the total FLOPS without any significant change in the accuracy of the network and does not require any fine tuning.
1.6
Test Loss
ID
Accuracy
Magnitude Baseline
ALDS
1.4
Messi
PCA
0.2
0.0
0.2
0.4
0.8
1.0
0.0
0.6
SVD
Fraction of FLOPS remaining
Fraction of FLOPs Remaining
Figure I: Pre-fine tuning pruning results on a 3D-Conv network for ATOM3D (left) and MobileNet VI for CIFAR-10 (right). We see that ID either matches or outperforms other methods.
Pre-fine tuning results We analyze the performance of ID compression against other pruning methods using the CIFAR-10 4 and Imagenet [19] data sets with VGG16 [69] and Mobilenet VI 58 The ID uses held-out pruning set of 1000 data points on CIFAR-10 and 5000 data points on Imagenet. Full hyper-parameter details can be found the Appendix and code. Figures and 2 illustrate results before fine tuning for structured and unstructured methods. We see that on CIFAR-10
8

---

and ImageNet, Iterative ID matches or ouiperforms other methods which do not use any global fine tuning. We even dominate unstructured methods which are typically more parameter efficient. Many matrix decomposition based methods struggle compress the
convolution layers in the Mobilenet V1 network. Moreover, Iterative ID is able to choose better per-layer sizes than the default VGG16 configuration on CIFAR-10. In the appendix, we show that the choice of how much to prune per iteration is not particularly sensitive, and demonstrate that pruning up to 35% of the FLOPs results in degradation to the pre-fine tuning accuracy. Results for MobileNet VI on ImageNet can be found the Appendix.
0.8
Uni
60
Thi
Accuracy
PFP
Accuracy
SVD
SiPP
40
StructMag
0.4
Snip
Thi
20
Soft
Thres
0.0
0.2
0.4
0.6
0.8
1.0
Messi
0.5
0.6
0.7
0.8
0.9
Fraction of Parameters Remaining
Fraction of FLOPs Remaining
Figure 2: Pre-fine tuning pruning results on VGG16 for CIFAR-10 (left) and ImageNet (right). ID either matches outperforms other methods, including unstructured methods (dotted lines) which are typically more parameter efficient. Solid lines are structured. Note the "SVD" method changes the structure of the network. Model correlation results can be found in the Appendix.
Model correlation as proxy for preserving fairness We conduct an experiment to demonstrate the connection between model correlation and a simple fairness metric of per-class accuracy. We begin with a VGG16 model M trained on all 10 classes of Cifar10. It performs reasonably well on all 10 classes, so we consider "fair". We then remove a class from the pruning set to simulate an under-represented class (but leave it in the train and test sets). We then prune M using our ID-based method, creating compressed model Mia-we do not perform any fine tuning. Sub-class accuracy for each class can be seen in figure 14a as well as the correlation of Mia with respect to M. Despite not having access to an entire class of data, we can prune the model to 50% FLOPs while only losing few percent accuracy on the missing class and retaining reasonably high model correlation. As comparison we prune the model M with magnitude pruning to create a compressed model Mm The magnitude pruned model loses almost all accuracy the missing class when we prune to 50%, and its model correlation initially only 10%. We fine tune with the abbreviated data set in an attempt to recover accuracy (figure 15a shows the correlation and sub-class accuracies). While Mn, recovers the represented classes, it fails to recover any accuracy for the unrepresented class during fine tuning.
VGG-16 Cifar10
VGG-16 Imagenet
100
% Accuracy
70.0
PCA
80
67.5
ID+PCA
ALDS
60
65.0
ID+ALDS
62.5
LRank
ID+LRank
60.0
0.00
0.05
0.10
0.15
0.20
0.25
0.6
0.8
1.0
Fraction Params Remaining
Fraction of FLOPs Remaining
Figure 3: Composing ID with other compression methods for VGG-16 on CIFAR-10 (left) and ImageNet (right). For each ID+compression method, the original network was pruned using IDD until the correlation degraded on the pruning set. second method was then applied to the compressed network. This composition produced smaller and more accurate networks than either method alone.
Model architecture preservation and composition of methods Because our method preserves network structure and only reduces layer sizes can be easily combined with other methods, see Figure 3 We first apply Iterative ID. pruning until the the model correlation begins to drop on the pruning set. Then we apply a secondary compression method. This composition easy because are simply replacing the original model with ID ID-pruned model that has the same architecture.

---

The secondary algorithms we use insert extra layers with different computational structure into the network, so composing them with each other is non-trivial. each example we see that pruning first with ID and then applying secondary algorithm boosts the performance of the secondary algorithm.
Post-global-fine tuning results In Tables and 2 demonstrate the effectiveness of our method for VGG- 16G-166p on CIFAR-10 and ImageNet after moderate amounts of global fine tuning, both alone and when combined with other methods. Hyperparameter details can be found in the Appendix. Networks pruned with ID attain competitive accuracies while having a much higher prediction correlation with their original models. This evidence suggests that have truly compressed the network rather than retraining a new one. Other methods often do not correlate much more than an independently trained model. Surprisingly, we find that in some settings combining ID with other matrix decomposition methods like LRank 40
can be competitive without any global fine tuning (Tables П
We also find that ID pruning and then globally fine-tuning MobileNet VI architecture results in an accuracy of 89.4% on Cifar-10 and a correlation of 92.6% with the original model after 60 epochs at 42% FLOPs reduction, opposed model trained from scratch which achieves an accuracy of 88.9% and a correlation of 88.6% after 120 epochs.
Method
FLOPs Reduction (%)
Accuracy
A
Correlation
Epochs
ID ID+LRank
50
93.31
0.30
97.29
60
50
93.43
0.31
95.9
0
HRank
93.43
-0.50
480
Polar
54
93.92
0.04
200
NS
93.62
-0.26
200
Mag
50
93.56
-0.04
93.15
200
Independent
93.60
93.02
160
Table 1: Post-fine tuning VGG-16 results on CIFAR-10. ID results are averaged over 5 trials, with a standard deviation of 3..36. "Independent" an independently trained model. the 50% FLOPs region, ID produces the most correlated model, while ID+LRank produces a model with reasonable accuracy without any global fine tuning. = original accuracy pruned accuracy.
Method
FLOPs Reduction (%)
Accuracy
A
Correlation
Epochs
ID+LRank
43
68.84
2.67
85.66
0
ID+LRank
43
70.50
1.01
90.90
10
ID+PCA
54
69.18
2.33
87.13
200
LRank -10pl.
43
70.30
1,21
90.38
ThiNet
46
66.59
0.89
70.15
30
AMC
48
70.49
0.65
76.03
120
CP
53
68.20
2.66
78.47
10
Table 2: Post-fine tuning VGG-16 results on ImageNet. We list methods which release both the original and compressed model, so that we can compute the correlation. ID composed methods produce competitive accuracies at range of FLOPs reductions, and produce better correlated models, even without any fine tuning.
8 Limitations and future work
Our method comes with several limitations and possible extensions for future work. While preserving model correlation suggests that are likely to preserve sub-class loss, our theory does not currently extend to that regime and requires that the unlabeled pruning set be from the same distribution as the test set. More broadly, we do not yet fully understand how trainable the models produced by ID are in different conditions and we cannot make claims about how compressable given model will be. In future work will explore modifying training process to improve prunability, which is common We will also explore ways to refine our iterative pruning approach to work with more complicated architectures. Of particular note, our method typically exposes a sizable "free FLOPs" regime and we explore how this can be leveraged more broadly.
approach 33 76 52 78)
A Acknowledgments and Disclosure of Funding
This work was partially funded by the National Science Foundation under awards DMS- 1830274, DGE-1922551, DMS-214600 146079, and NSF CAREER Award 2046760.
10

---

References
[1] Martín Abadi, Ashish Agarwal, Paul Barham, Bugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, lan Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke. Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large- scale machine learning on heterogencous systems, 2015. URL http:///teensrtloo.oooooo org/ Software available from tensorflow.org.
[2] E. Anderson, Z. Bai, C. Bischof, S. Blackford, J. Demmel, Dongarra, J. Du Croz, A. Greenbaum, S. Hammarling, McKenney, and D. Sorensen. LAPACK Users' Guide. SIAM, Philadelphia, PA, third edition, 1999.
[3] Grey Ballard, James Demmel, Laura Grigori. Mathias Jacquelin, Hong Diep Nguyen, and Edgar Solomonik. Reconstructing householder vectors from tall-skinny qr. 2014 IEEE 28th International Purallel and Distributed Processing Symposium. IEEE, 2014.
[4] Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and Machine Learning. fairmlbook.org. 2019. http://www http://www..airrmbbbor org
[5] Cenk Baykal, Lucas Lichenwein, Igor Gilitschenski, Dan Feldman, and Daniela Rus. Sipping neural networks: Sensitivity-innormmed provable pruning of neural networks. CoRR, abs/1910.05422, 2019. URL
attp://arxiv. wwwp:/wwww..ggg////./eee//e/ 05422
[6] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? arXiv preprint arXiv:2003.0303: 2020.
[7] Christos Boutsidis, Michael W Mahoney, and Petros Drincas, An improved approximation algorithm for the column subset selection problem. In Proceedings of the twentieth annual ACM-SIAM symposium on Discrete algorithms, pages 968-977. SIAM, 2009.
[8] Peter Businger and Gene H Golub. Linear least squares solutions by householder transformations. Nu- merische Mathemarik, (3):269-2766 1965.
[9] Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexcy Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv: / 1902.06705 2019.
[10] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alchéé Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. htttt::///oooceeee htwpe//wwww....ooaaccccccee///ee Paper.pde
[11] Tony F Chan and Per Christian Hansen. Some applications of the rank revealing qr factorization. SIAM Journal on Scientific and Statistical Computing, 13(3):727-741, 1992.
[12] Shivkumar Chandrasckaran and Hlse CF Ipsen. On renk-rev factorisations. SIAM J. on Matrix Anal. and Apps., 15(2):592-666 622, 1994.
[13] H. Cheng, Z. Gimbutas, Per-Gunnar Martinsson, and V. Rokhlin. On the compression of low rank matrices. SIAM Journal on Scientific Computing, 26( (6(4):3399-140 2005.
[14] Alexandra Chouldechova and Aaron Roth. A snapshot of the frontiers fairness machine learning. Communications of the ACM, 63:82-89 89, 2020.
[15] Ali Civril and Malik Magdon-Ismail. On selecting maximum volume uub-matix of matrix and related problems. Theoretical Computer Science, +10(47749))4881-488111 2009.
[16] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, pages 1310-1320. PMLR, 2019.
[17] Sam Corbett-Davie and Sharad Goel. The measure and mismeasure of fairness: A critical review of fair machine learning, arXiv: 1808. 1808.0002, 2018
[18] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter- farameter-free attacks. In International Conference on Machine Learning, pages 2206 2216. PMLR, 2020.

---

[19] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. IEEE, 2009.
[20] Emily Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting lineat structure within convolutional networks for efficient evaluation. Advances in neural information processing systems, 2014.
[21] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2018.
[22] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Lincar mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259-3269. PMLR, 2020.
[23] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Pruning neural networks at initialization: Why are we missing the mark? arXiv preprint arXiv:2009.088 085766 2020.
[24] Sorelle A. Friedler, Carlos Scheidegger, Suresh Venkatasubramaniin, Sonam Choudhary, Evan Hamilton, and Derek Roth. A comparative study of fairness-enhancing interventions in machine learning. In Proceedings of the Conference Fairness, Accountability, and Transparency, FAT '19, page 329-338. Association for Computing Machinery, 2019. ISBN 9781 7781450361255.
[25] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.08 1902.0974, 2019.
[26] G. H. Golub and C. F. Van Loan. Matrix computations. Johns Hopkins Univ. Press, Baltimore, third edition, 1996.
[27] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
[28] M. Gu and S. Eisenstat. Efficient algorithms for computing a strong rank-revealing QR factorization. SIAM J. Sci. Comput., 17(4):848-86) 1996.
[29] Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight VC-dimension bounds for piecewise linear neural networks. In Satyen Kale and Ohad Shamir, editors. Proceedings of the 2017 Conference on Learning Theory, volume 65 of Proceedings of Machine Learning Research, pages 1064-008.. PMLR. 07-10 Jul 2017. URL wtt.tttttttttttte http::/proceeddnng .mr............................. mlr preas/ preas/v65// ww5/harvey/11// html
[30] Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelcrating deep convolutional neural networks. CoRR, 800.088680666, 2018. URL http://////ax org/ wwwgg///////eeeeeo 1808 06866)
[31] Yang He, Ping Liu, Ziwei Wang. Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. Conference on Computer Vision and Patter Recognition, 2019.
[32] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In International Conference on Computer Vision, 2017.
[33] Yihui He, Lin, Zhijian Liu, Hanrui Wang, and Song Li, Li-Jia nad Han. Ame: Automl for model compression and acceleration on mobile devices. In European Conference on Computer Vision, 2018.
[34] Kenneth L and Lexing Ying. Hierarchical interpolative factorization for elliptic operators: differential equations. Communications on Pure and Applied Mathematics, 69(8): 1415-1 2016.
[35] K.I.. Ho and I.. Greengard. fast direct solver for structured linear systems by recursive skeletonization. SIAM Journal on Scientific Computing, 34(5):A2557-A222 A2532, 2012. doi: 10.16637/10866683.
[36] K.L. Ho and L. Ying. Hierarchical interpolative factorization elliptic operators: Integral equations. Communications on Pure and Applied Mathematicx, 2015. ISSN 1097-0312. doi: 10. 10.1002//pa.21577.
URL http:tt:://xx doi. org/ 10. 1002/ 21577
[37] Yoo Pyo Hong and C-T Pan. Rank-revealing qr factorizations and the singular value decomposition. Mathematics of Computation. 58(197):213-33) 1992.
[38] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, (bs/1704.04861. 2017. URL http://arxiv. http:/aaaaiiirirrrraaraa/a org/abs/1704 04861
12

---

[39] Zehao Huang and Naiyan Wan. Data-driven sparse structure selection for deep neural networks. In European Conference on Computer Vision, 2018.
[40] Yerlan Idelbayev and Miguel A. Carreir-Perpinann Perpinan. Low-rank compression of neural nets: Learning the rank of each layer. In 2020 IEFE conference on computer vision and pottern recognition, pages 8049-8059. IEFE, 2020.
[41] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 448 456, Lille, France, 07-09 Jul 2015. PMLR.
[42] Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. In Proceedingy of the British Machine Vision Conference. BMVA Press, 2014.
[43] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report, 2009.
[44] Vadim Lebedev, Yaroslav Gamin, Maksim Rakhuba, Ivan Oscledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. International Conference on Learning Representations, 2015.
[45] Namhoon Lec, Thalaiyasingam Ajanthan, and Philip H.S. Torr. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2019.
[46] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2017.
[47] Edo Liberty, Franco Woolfe, Per-Gunnar Martinsson, Vladimir Rokhlin, and Mark Tygert. Randomized algorithms for the low-rank approximation of matrices. Proceedings of the National Academy of Sciences, (01(51)20166-20012) 2007.
[48] Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks. International Conference on Learning Representations, 2020. URL https://openreviiee net/forummid Bdx
[49] Lucas Lichenwein, Cenk Baykal, Brandon Carter, David Gifford, and Daniela Rus. Lost in pruning: The effects of pruning neural networks beyond test accuracy. In Proceedings of the 4th MLSys Conference, 2021.
[50] Lucas Liebenwein, Alaa Maalouf, Dan Feldman, and Daniela Rus. Compressing neural networks: Towards determining the optimal layer- decomposition. In Advances in Neural Information Processing Systems, volume 34, 2021. URL. atttps//aaziv..or/aab////2
11442
[51] Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 1529-1538, 2020.
[52] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao haung, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In International Conference on Computer Vision, 2017.
[53] Zhuang Liu, Mingjee Sun. Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2019.
[54] Jaan-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pages 5058-5066, 2017.
[55] Alaa Maalouf, Harry Lang, Daniela Rus, and Dan Feldman. Deep leaminng meets projective clustering. ArXiv, abs/2010.04290. 2021.
[56] Michael W Mahoney and Petros Drincas. CUR matrix decompositions for improved data analysis. Proceedings of the National Academy of Sciences, 106(3):977702, 2009.
[57] Per Gunnar. Martinsson. Fast Direct Solvers for Elliptic PDEs. Society for Industrial and Applied Mathematics, Philadelphia, PA, 2019. doi: 10.1137/1.97811111176444 URL httpss//epubs.. siam org/doi/abs/ 1137/1 97111111991111117455555
13

---

[58] Per-Gunnar Martinsson and V. Rokhlin. A fast direct solver for boundary integral equations in two dimensions. J. Comput. Phys.. 205(1):1-23, May 2005. ISSN 0021-9991. doi: http///.jc...0.........
URL http://///dxx...m woeidx.doo.oo . org/10 1016:10..1..... j. 2004. 10.004.0..00 10.
[59]
Per-Gunnar Martinsson, Vladimir Rokhlin, and Mark Tygert. A randomized algorithm for the decomposi- tion of matrices. Applied and Computational Harmonic Analysis. 30(1):47-68, 201
[60] Charles T. Marx, Flaviou du Pin Calmon, and Berk Ustum. Predicitive multiplicity in classification. In International Conference on Machine Learning. PMLR, 2020.
[61] Ninarch Mchrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey bias and fairness in machine learning. ACM Comput. Surv. 54(6), jul 2021. ISSN 0360-0300. doi: 10. 145/3447677 URL httph: org/10. 1145/3457607
[62] Victor Minden, Kenneth I. Ho, Anil Damle, and lexing Ying. A recursive skeletonization factorization based on strong admissibility. Multiscale Modeling & Simulation, 15(2)7688-766, 2017.
[63] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations Machine Learning. MIT press, 2 edition, 2018
[64] Ben Mussay, Margarita Osadchy, Vladimir Braverman, Samson Zhou, and Dan Feldman. Data-independent neural pruning via coresets. In International Conference on Learning Representations. 2020.
[65] Bo Peng, Wenming Tan, Zheyang Li, Shun Zhang. Di Xie, and Shiliang Pu. Extreme network compression via filter group approximation. European Conference on Computer Vision, 2018.
[66] Hanyu Peng, Jiaxiang Wu, Shifeng Chen, and Junzhou Huang. Collaborative channel pruning for deep networks. In International Conference on Machine Learning, 2019.
[67]
Gregorio Quintana-Orti, Xiaobai Sun, and Christian H. Bischof. A blas-3 version of the qr factorization with column pivoting. SIAM J. on Sci. Comp., 1965)1446-1144, 1998. doi: 10.1 00.1137751064427555996722
[68] Samuel Henrique Silva and Peyman Najafirad. Opportunitics and challenges in deep learning adversarial robustness: A survey, ArXiv, abs/2200.00773 2020.
[69] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In International Conference on Learning Representations, 2015.
[70] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1); 1929-1958, 2014.
[71] Samuel Stanton, Pavel Izmailov, Polina Kirichenko, A. Alemi Alexander, and Andrew Gordon Wilson. Does knowledge distillation really work? In Advances in neural information processing systems, 2021.
[72] Raphael J. L. Townshend, Martin Vögele, Patricia Suriana, Alexander Derry Alexander Powers, Yianni Laloudakis, Sidhika Balachandar, Brandon M. Anderson, Stephan Eismann, Risi Kondor, Russ Altman. and Ron O. Dror. ATOM3D: tasks on molecules in three dimensions. CoRR, abs/2012.04035, 2020. URL
https://arxil: org/ abs/ 2012.04035
[73] Joel A Tropp. Column subset selection, matrix factorization, and eigenvalue optimization. In Proceedings of the twentieth annual ACM-SIAM symposium on Discrete algorithms, pages 978-986. SIAM, 2009.
[74] Sergey Voronin and Per-Gunnar Martinsson. Efficient algorithms for cur and interpolative matrix decom- positions. Advances in Computational Mathematics, 43(3):495-516 2017.
[75] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for henchmarking machine learning algorithms. 770.00777 2017.
[76] Huanrui Yang, Wei Wen, and Hai Li. Deephoyer: Learning sparser neural network with differentiable scale-invariant sparsity measures. In International Conference on Learning Representations, 2020.
[77] Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015.
[78] Tao Zhuang, Zhixuan Zhang, Yuhen Huang, Xiaoyi Zeng, Kai Shuang, and Xiang Li. Neuron- structured pruning using polarization regularizer. In Advances in neural information processing systems, 2020.
[79] Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wul, Junzhou Huang. and Jinhui Zhu. Discrimination-awall channel pruning for deep neural networks. In Advances in neural information processing systems, 2018
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Section
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] In the Appendix.
3. If you experiments..
(a) Did you include the code. data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as URL)? [Yes] [Yes] Appendix
(b) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] Where including error bars was computationally feasible we did so. This was not computationally feasible on ImageNet.
(c) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Appendix
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you's re
using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "pNPCaS6Qh5s", "reviewer": "Reviewer_12sE", "summary": "This paper proposes a pruning approach based on interpolative decomposition computed by rank revealing QR. The proposed method can automatically determine the layer size. The authors also provide theoretical analysis for the proposed methods.", "questions": "1. There is no explanation about ID+LRank. LRank decomposes layers into a low-rank format using an alternating Lagrangian algorithm, learning to compression. How does it combine with the proposed ID-based pruning approach? \n2. What is the performance gap between the pruned model and the pre-trained model? All the results miss baselines.\n3. How does this approach perform for SOTA models on ImageNet?", "limitations": "To make the proposed approach convincing, I highly recommend the authors to provide more general experimental results for fair comparison, e.g., ResNet-50, MobileNet and ViT on ImageNet. I also recommend the authors to not recycle to another top-tier conference without significant revision or improvement. ", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Using interpolative decomposition is new to compression and it is interesting to maintain the pretrained model information with ID.\n2. Pruning from a pre-trained model without retraining is practically useful. \n3. The motivation of this paper is good and the theoretical guarantee is significant.\n\nWeaknesses:\n1. The experiments are too weak. VGG is an extremely dated model. Results and comparisons for ResNet-50 and MobileNet on ImageNet are required. If authors could provide results on vision transformers, it could be more convincing.\n2. Even though the theoretical parts are promising, the practical results are weak. It is not sure whether the proposed post-training pruning method really works. Based on Table 2,  the performance is even inferior to AMC.\n3. The metric 'correlation' used in this paper is not a general metric to evaluate the model performance. Better correlation shows nothing.\n4. This paper lacks several SOTA references, e.g., [R1-R4].\n\n[R1] Scop: Scientific control for reliable neural network pruning. NeurIPS 2020.\n\n[R2] ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximations. ICLR 2021.\n\n[R3] Provable filter pruning for efficient neural networks, ICLR 2020.\n\n[R4] Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition, NeurIPS 2021. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hhH6JwABMrM", "reviewer": "Reviewer_s68B", "summary": "The main contribution of the paper is a new model compression method that simultaneously 1) preserves the original model's per-example decisions, 2) maintains the network's structure, 3) determines the per-layer compression levels automatically, and 4) eliminates the need for retraining the compressed model. The authors demonstrate the efficacy of their method through extensive experiments on CIFAR-10 and ImageNet datasets.", "questions": "I listed some questions in the previous section. My biggest concern is that authors use \"preserving the model correlation\" as one of the main criteria in model compression by claiming that this implies improved adversarial robustness, fairness, and sub-class accuracy. This relation is never made clear in the paper and I don't see why this is true. ", "limitations": "The authors addressed the limitations. ", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I generally like the approach, and the goal of the paper, and find the analyses theoretically sound. My only hesitation is their claim that preserving the model's per-example decisions would boost fairness and adversarial robustness. If we look at the \"average\" correlation between the compressed and original model, we will still miss errors in underrepresented data samples. So I am not sure how preserving the correlation brings an advantage over preserving the accuracy of the model in terms of fairness. The authors also mention adversarial robustness. Again, I could not make the connection between preserving the model correlation and adversarial robustness. Can the authors explain the relation and make it more clear in the paper? \n\nI find the paragraph between lines 47-57 a bit confusing. What does \"an independently trained VGG16 model\" refer to? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "67nnTAG7cU4", "reviewer": "Reviewer_eWGv", "summary": "This paper proposes a pruning method that uses interpolative decomposition of matrices to prune neurons/channels of a model. Generalization bound for the pruned model is provided for the two-layer network case. The method shows good pre-finetuning, post-finetuning quality metrics and good correlations with the un-pruned models.", "questions": "To successfully approximate the pre-compression layer, it seems that a good pruning data set is crucial. Would it be possible to show some empirical results on the dependence on the number of pruning sample?", "limitations": "NA", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper proposes a novel approach to use interpolative decomposition to compress learned linear/convolutional layers and propagate compression corrections to other layers. The idea of using interpolative decomposition is new and solid;\n\n2. The authors provided theoretical analysis on generalization bounds of a compressed two-layer network;\n\n3. The experiments are done carefully and multiple metrics are used to compare relative performance of different compression methods;\n\n4. The authors provided theoretical results on dependence on the number of samples for the two-layer network case. \n\nWeaknesses:\n\nFor deeper networks, the paper didn't offer discussion on dependence on the number of samples.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept