TVSPrune - Pruning Non-discriminative filters via Total Variation separability of intermediate representations without fine tuning
Authors: Chaitanya Murti
Tanay Narshana
Chiranjib Bhattacharyya
Venue: ICLR 2023 poster (2023)
Keywords: Structured pruning
model compression

Abstract: Achieving structured, data-free sparsity of deep neural networks (DNNs) remains an open area of research.  In this work, we address the challenge of pruning filters without access to the original training set or loss function. We propose the discriminative filters hypothesis, that well-trained models possess discriminative filters, and any non-discriminative filters can be pruned without impacting the predictive performance of the classifier. Based on this hypothesis, we propose a new paradigm for pruning neural networks: distributional pruning, wherein we only require access to the distributions that generated the original datasets. Our approach to solving the problem of formalising and quantifying the discriminating ability of filters is through the total variation (TV) distance between the class-conditional distributions of the filter outputs. We present empirical results that, using this definition of discriminability, support our hypothesis on a variety of datasets and architectures. Next, we define the LDIFF score, a heuristic to quantify the extent to which a layer possesses a mixture of discriminative and non-discriminative filters. We empirically demonstrate that the LDIFF score is indicative of the performance of random pruning for a given layer, and thereby indicates the extent to which a layer may be pruned. Our main contribution is a novel one-shot pruning algorithm, called TVSPrune, that identifies non-discriminative filters for pruning. We extend this algorithm to IterTVSPrune, wherein we iteratively apply TVSPrune, thereby enabling us to achieve greater sparsity. Last, we demonstrate the efficacy of the TVSPrune on a variety of datasets, and show that in some cases, we can prune up to 60% of parameters with only a 2% loss of accuracy without any fine-tuning of the model, beating the nearest baseline by almost 10%.

Published as a conference paper at ICLR 2023
TVSPRUNE PRUNING NON-DISCRIMINATIVE FILTERS VIA TOTAL VARIATION SEPARABILITY OF INTERMEDI- ATE REPRESENTATIONS WITHOUT FINE TUNING
Chaitanya Murti', Tanay Narshana" and Chiranjib Bhattacharyya 1.3 Robert Bosch Centre for Cyber-Physical Systems, Indian Institute of Science! Observe.AI Department of Computer Science and Automation, Indian Institute of Science. (mchai tanya, chiru)81isc.oo aiisc.ac.in, in, tanay narshana@gmaalaaa com
ABSTRACT
Achieving structured, data-free sparsity of deep neural networks (DNNs) remains open area of research. In this work, we address the challenge of pruning filters without access to the original training set or loss function. We propose the discriminative filters hypothesis, that well- models possess discriminative filters, and any non discriminative filters can be pruned without impacting the predictive performance of the classifier. Based on this hypothesis, we propose a paradigm for pruning neural networks: distributional pruning, wherein only require access to the distributions that generated the original datasets. Our approach to solving the problem of formalising and quantifying the discriminating ability filters is through the total variation (TV) distance between the class-condittonal distributions of the filter outputs. We present empirical results that, using this definition of discriminability. support our hypothesis on a variety of datasets and architectures. Next, define the LDIFF score, heuristic to quantify the extent which a layer possesses a mixture of discriminative and -oo-disrriminttive filters. We empirically demonstrate that the LDIFF score is indicative of the performance of random pruning for a given layer, and thereby indicates the extent to which a layer may be pruned. Our main contribution is novel one-shot pruning algorithm, called TVSPrune, that identifies non-discriminative filters for pruning. We extend this algorithm to SterTVSrrune, wherein we iteratively apply TVSPrune, thereby enabling to achieve greater sparsity. Last, we demonstrate the efficacy of the TVSPrume on a variety of datasets, and show that in some cases, we can prune up to 60% of parameters with only 2% loss of accuracy without any mun-ttning of the model, beating the nearest baseline by almost 10%. Our code is available here
INTRODUCTION
Deep neural networks are, in general, highly overparameterized, leading to significant implementation challenges in terms of reducing inference time, power consumption, and memory footprint. This especially crucial for deployment on real world, resource constrained devices (Molchanov et al.) 2019b; Prakash et al. 2019)). A variety of solutions have been proposed to solve this problem, which broadly be grouped into quantization, sparsification or pruning, knowledge distillation, and neural architecture search (NAS) (Hoefler et al. 2021).
Pruning can be further divided into unstructured pruning, wherein individual parameters are set to zero, or structured pruning, wherein entire filters or channels are removed from the architecture (Hoeller) al. 202T). Structured pruning yields immediate improvements in inference time, memory footprint, and power consumption, without requiring any specialized software frameworks. Unstructured pruning, on the other hand, typically yields models that are significantly more sparse than those
Work done at the Department of Computer Science and Automation, Indian Institute of Science. Link to github: https: / /gi thub. com/ cha imurt /TVSPrune

---

Published as a conference paper at ICLR 2023
obtained with structured pruning, but do not provide the same improvements in inference time without specialized sparse linear algebra implementations (Hoefler et al. [021 Blalock et al. 2020).
In this work, we consider the problem of pruning CNNs without access to the training set or loss function. Data-free pruning is an important problem due to concerns such as privacy and security Yin
et al. 2020), as well as the cost of retraining models (Tanaka al., [020 [Hoefler et al.] ) We offer a new perspective to this problem, which we call distributional pruning, wherein we have no access the training data or the loss function, but have access to data distribution, either through it's
moments, or through additional samples separate from the training set.
To facilitate distributional pruning, two crucial questions need to be answered. First, what makes filter valuable to the classification performance of the model?, and second, how do we characterize which layers can be effectively sparsified? To answer these questions, we first identify discriminative filters, which are filters with candiionniiioaal outputs that are well-separated in terms of the total variation. We propose the discriminative filters hypothesis, which states that well-trained models possess a mix of discriminative and non discriminative filters, and that discriminative filters are useful for generalization of the classifier. Based on this hypothesis, discriminative filters are useful for classification purposes whereas non-discriminative filters are not, thus allowing us to prune the latter. Furthermore, layers that possess a mix of discriminative and non- discriminative filters can be effectively pruned, thereby providing a method to identify difficult-to. torpultttt--ueeeeeee layers. We formally state our contributions below.
1. We begin by proposing a quantitative measure of discriminative ability of filters in terms of the TV distance between their class-conditional outputs. Specifically, we say filter is TV- separable if the pairwise minimum TV distance between class-conditional distributions of the outputs is larger than given threshold. If the class conditional distributions are gaussian a common assumption as noted in Wong et al.) (202T): Wen et al. (2016) we can compute the Hellinger distance based lower bound to estimate whether filters are V-spparaele using easily computed class-conditional moments. We describe this in section 4
2.
We produce the empirical observation that the classwise outputs of at least some filters present in CNNs that generalize well are TV-separable, and that untrained models, or models that generalize poorly do not possess discriminative filters; these are presented in section 7 Based on these observations, in section 2 we propose the discriminative filters hypothesis, which states that well-trainee convolutional neural networks possess a mix of discriminative and non diniimcciiiiaaivee filters, and discriminative filters are useful for classification whereas the latter are not. We use this hypothesis to motivate a distributional approach to pruning.
3. Based on the discriminative filters hypothesis, aim use TV separation to identify which filters to prune in a model. We assume the class- conditoaniiionnl distributions are Gaussian, and, using the Hellinger lower bounds discussed in section 4 compute lower bounds on the TV-separation for each filter. We identify important filters (those that cannot be pruned) those filters with the Hellinger lower bound of the TV-separation that are greater than separation threshold; those filters that are not discriminative with respect to the separation threshold can be pruned.
4. As noted in Hoefler et al. (2021): Liehenwein et al. 2019), some layers are more difficult to prune than others. We address the problem of identifying which layers can be effectively pruned using TV-separability. Based on the discriminative filters hypothesis, a layer can be effectively pruned if it possesses mixture of discriminative and -on-dicrriminatiee filters. Thus, in section 5 we propose an informative heuristic, which we call the LDIFFscore, that quantifies the extent to which a layer possesses mixture of discriminative and non- discriminative filters. We empirically validate this heuristic in section 7
5. We use TV-separability and LDIFF scores to develop TVSPR, a layer-wise, threshold based method for structured pruning, requiring no fine tuning and only the class-conditional moments of the outputs of each filter. We also extend this algorithm to an iterative variant, STERTVSPRUNT SPRUNE, which enables superior sparsification of the model. We formally state these algorithms in section 6. in Algorithms and 2. We show that on the CIFAR-10 dataset, method achieves over 40% sparsification with minimal reduction in accuracy on VGG models without any fine tuning; furthermore, our method outperforms contemporary methods such as (Sui et al. 2021 Molchanov et al. 2019b) in this regime.
2

---

Published as a conference paper at ICLR 2023
2 outline our problem statement. In Section 3 detail the notation used in the paper, and review Total Variation distances and the associated Hellinger distance bounds, and in Section[4] we describe how utilize the TV-distance to measure the discrim- inative capacity of filters. In Section [ we propose the LDIFF score for quantifying the difficulty in pruning a given layer, and in Section 6 present the TVSPRUNE and ITERTV PRUNFalgoorrns
Our paper is organized as follows. In Section
In Section 7 we detail our experimental results. We produce a detailed related work section in Appendix A We also provide additional discussions in B variants of the LDIFF score and the andd D. and a variety of additional empirical results in section
TVSPRUNE algorithms in sections
FILTER PRUNING WITHOUT TRAINING DATA AND FINE TUNING
Structured pruning can be thought of as two specific optimization problems: finding the most accurate model that satisfies a sparsity model, and finding the sparsest network that satisfies an accuracy constraint. We define the latter problem below.
W = arg min (IDWll S(W) I t, t> 03.
W
Our goal is to solve equation Puing structured pruning techniques; that is, we aim to remove entire filters at each pruning iteration. As mentioned previously, typical methods to solve this problem require extensive retraining to overcome the losses in accuracy. In this work, we consider the scenario wherein neither the training data nor the loss function are available during the pruning process.
Motivated by empirical results, we see that models that generalize well tend to have filters with well-separated output features (in the distributional sense), whereas models that do not generalize well possess filters with output features that not well separated. We refer the reader to Section 7.] for these results. This prompts us to formulate the following hypothesis.
Hypothesis (Discriminative Filters Hypothesis). "Models that generalize well possess rich mixture of filters that discriminate between classes and those that don't tt Furthermore, we hypothesize that for generalization purposes, it is sufficient to retain the filtess that discriminate between classes well, in those layers that have large variation in discriminative ability."
Note that we also observe that some layers have filters with output features that cannot discriminate well, and yet cannot be pruned.
3 PRELIMINARIES AND BACKGROUND
3.1 NOTATION
Let [n] (1..... .n N. We the define parameters of neural network with L as W (W1,... ,WL), where each W, 1 is collection of tensors. For convolutional neural network (CNN), each Wi R. x In this formulation, the ith filter of layer is tensor / E N. x x and the jth kernel in this filter is a matrix fig E IR K Note that given a model, we can use the pair (t,j) to identify a filter / denotes the layer, and j the filter index.
We vectorize this model as follows. First, suppose the vectorized input to layer o'(X) E N,n, where Ni. is the number of channels and m is the dimension of the input (say, the number of pixels in each dimension of an image). Note that we ignore the superscript indicating the layer for the sake of brevity. Suppose each filter in layer 1 + 1 has kernel size x K, that is, for each j E [J], the we have the vectorized W E IR K² N... Next, we assume that there are P patches, with (X) being the pth patch in the input. In the sequel, drop the layer superscript for convenience. assume draw samples (X, u). where X is the datum and is the class label, from a distribution D. We write D is the class conditional distribution of class a. Here X E N for some integer N is the input, and [Nelasses] N is the integer class.
[L] is
In this work,
3

---

Published as a conference paper at ICLR 2023
3.2 REVIEW OF TOTAL VARIATION DISTANCE
In this section, we formally define the Total Variation and Hellinger distances. We define this formally in the sequel. The results that follow are discussed in, say, Kraft (1955).
Definition Let Q1 and Qi be two probability measures supported IR We define the Total Variation Distance TV as
TV (Q1.
sup [Q((A) 22((A) ACR"
Unfortunately, closed form expression exists for the Total Variation distance, even when Q1, Q2 are Gaussian. This motivates us to use the Hellinger distance, which we define below.
Definition 2. Let Q1 Q2 be two probability measures supported on RRd, and let qi and q2 be the corresponding densities. We define the squared Hellinger distance as
HELLD (Q1, O2)
2 gg (Vg((=) (Ve(())-ox(())) dr
1 [
Crucially, the Hellinger distance provides us upper and lower bounds on the TV-Distance.
HELLD"(QI, Q2) < TV(Qi, Q2) S /2 HELLD(QU. Qo).
(1)
we have Q1 ~ (N1,, and Q2 ~ (u2,, 13 are Gaussian, we get
20102 of + +
|1| 112 1|
HELLD? (Q), Q22 =
(2)
of 02
4 MEASURING DISCRIMINATIVE ABILITY OF FILTERS
In this work, we take distributional view of the discriminative ability of filters. Previously, we have described the TV-distance, and the Hellinger lower bound that can be efficiently estimated. this section, we apply the T'-distance to arriving at means to quantify the discriminative ability of individual filters by using the class-conditional distributions of the outputs. We begin by recasting the dot products used in the convolution operations with vectorized filters and feature maps.
Define Vi.p (X) to be the dot product of the jth filter and the pth patch of (X=)(X); that is
U1.ipp (X) (0) (x),_((x,10 W?).
(X),
Following from this, write
(x) = (Ø' (xx, W!).
where (x-1(Xx= (x)- o We define the Class Conditional Means and Variances for Y (X), the output of given filter as follows.
yra Ex,a)DD (X))) and oi ExX,u>DDD [[ 1108--Fapp 11]
Assumption 1. We assume that the class conditional distributions of each fither is Gaussian, that (X)()))) ~ NVY otz.a)
This assumption enables us to use the closed form expressions described in equation 2 While this assumption strong one, it is well motivated in the literature. The assumption class conditional distributions are gaussian is used in a variety of settings, including /unn et al. (2020); Lee et al. 2020); Seetharaman et al. (2019). Our goal to quantify the discriminative ability of the outputs of the individual filters, and do so, we use the TV-Distance between the class conditional distributions. Assumption I enables us to estimate lower bounds on the TV distance between any two cons-condiiional distributions of a filter's 's output easily. using only samples from that distribution. With that in mind, we define the Minumum TV Separation (MinTVS) between classes of a given filter.
4

---

Published as a conference paper at ICLR 2023
Definition 3. Suppose Asssumption holds. Let Di. be the class- classcoonitional distribution of (X) where (X,s) ~ -D For each layer and filter and each pair a, B, we define the Minimum TV Separation filter j in layer 1 as
MinTVS(L.j) min TV Df., D(.)) a
min HELLD² a
(P...D..)
(5)
Furthermore, since the harmonic mean in equation 2 is upper bounded by 1, we have
MinTVS (1,5) 2 exp
min a.e
a.8 A
/4) D [Y Yii ( + oiss))
thus allowing us to use 2..8 as surrogate for MinTVS(L.j) during experimentation. Using this quantity, we define TV-Separability with respect to some r > 0, which we call the separation threshold.
Definition 4. For each layer I and filter j. and separation threshold rj > 0 we say the filter is 17- TV Separable, or TVSEP(N) if
MinTVS(L,)) 2
Since we cannot directly compute the pairwise minimum of TV-separations between the class- conditional distributions of filter's output, instead rely on the lower bound proposed above. The value of 17 that indicates a "well separated" filter will vary from dataset to dataset, or architecture to architecture; for instance, for the VGG models trained CIFAR10 used in our experiments, find 0.05 is useful.
5 IDENTIFYING DIFFICULT-TO-PRUU LAYERS AND THE LDIFF SCORE
; Liebenwein et al. 2019), in given model, some layers can be effectively pruned, and others cannot. In this section, investigate the problem of identifying which layers of given neural network can be pruned effectively, through the lens of the discriminative filters hypothesis. Specifically, we argue that discriminative filters contribute generalization performance whereas non-discriminative filters do not, and argue that we can effectively prune layers containing mix of discriminative and non discriminative filters. We desire informative heuristic that can (a) indicate the extent a layer can be randomly pruned, thus informing pruning ratios for random pruning; and (b) indicate whether not layer should be pruned at all. Note that we leverage the latter purpose in our proposed algorithms in subsequent sections.
As noted in Hoefler et al. (221))
We define LDIFF, a heuristic score assigned each layer based on the TV separations of the constituent filters to the aggregate TV-separation and given separation threshold r7. Our goal is to capture the mix of discriminative and non-discriminative filters single heuristic indicator. Thus, given a separation threshold y, suppose the fraction of discriminative filters is (7)). We need a function that penalizes T(1) close to either 1 or (as having no discriminative filters in a layer indicates difficulty pruning). With these requirements in mind, we define
LDIFF(I, 7) = 4T (1-))))
(6)
If this quantity is close to 1, then that layer has mix of discriminative and non-discriminative filters. not, then there either a majority of discriminative filters, or con-discriminttie filters; in either case, pruning that layer would be challenging. Thus, this heuristic can also be used to inform random pruning ratios. For instance, we may choose LLIIFF(1, 17), for suitable ? E (0, 1), to the fraction of filters to be randomly pruned. Further refinements of this heuristic include layerwise thresholds, and using statistics of the TVn scores to determine discriminability. We refer readers to Section C for further discussion such refinements, as well empirical support for the use of LDIFF scores for indicating pruneability of layers.
DISTRIBUTIONAL APPROACHES TO STRUCTURED PRUNING
In this section, we discuss our solution to the problem of structured pruning of neural networks without access to the loss function or the original data, and only with access to the distributions of the
5

---

Published as a conference paper at ICLR 2023
outputs of the individual filters and layers. Our proposed algorithm has three steps. First, for given separation threshold 7/, we estimate the n ----sepaarabiltty of the filter outputs: second, we decide which layers we can prune; third, in those layers which can prune from, we remove all filters with TV-separabilityy less than y
Algorithm 1: TVSPRUNE
Algorithm 2: |TERTVSPRUNE
Input: Dataset D = (X,, u4 J.1. Pretrained
Input: Dataset = {X,, uz) 1-1-- Pretrained
CNN with parameters
CNN W = (Wi, WE). initial
W (Wi.... W., W.V.EE threshold n,
TVSEP threshold 7), LDIFF threshold
layer difficulty threshold
6, 0. accuracy threshold t, M/min < n
Compute P2.0. of.3.. for all J, and
Set A: = 0. W(o) = W
Compute Aq, for each a,
while 17 2 7mmin
W = TVSPRUNE(D, W()):
for I E [z.] do
Compute (in)VS((.j) for all J Compute LDIFF(I)
if f(W) S then
W(++1) W
LDIFF(( < v/ V LDIFF(() = 0 then for E [N do
else
7 + 1j bg
if MinT)S(L, < < then
W(++1) & W(N)
W] t 0
A: -- ++1
Output: W (W1,... ... WL), where (Wi) I sup)(Wi) 1
Output: W (W1, WL).f(M) st, supp(Wi) S supp (hi) 14
return VV
return V
6.1
THE TVSPRUNE ALGORITHM VARTABLE PRUNING RATIOS PER LAYER
The goal of this algorithm is to leverage the discriminative ability of filters to decide whether a given filter should be pruned. Thus far, we have defined the MVSSVS value, which is lower bound on the least classwise TV-separation of a filter's output, as well as the LDIFF score, which, in a sense, quantifies the extent which a layer can be sparsified.
The algorithm begins by computing the TV-separability of all the filters using (MinTVS(L.)) }) as defined in equation 10 Next, we compute the LDIFF values as described in equation o We then check the LDIIF score agains the threshold v. and thereby decide whether or not prune the given layer. Then, all filters in each prunable layer (that satisfies I.IIF(() > v/ that are not TVTVV Separable be pruned. The value of 1) dictates the aggressiveness of the pruning as it is measure of how "well separated" we desire if 1) is small, fewer filters are pruned. This approach is useful since it ensures that the pruning ratio varies from layer to layer, and that we prune those layers that are relatively difficult to sparsify far less aggressively.
} J==1"
Furthermore, we observe that some layers are dramatically more difficult to prune from than oth- ers. Therefore, we use the LDIFF scores to decide which layers to prune. We now present the TVSPRUNE algorithm. We assume that we have a labeled dataset DD of pairs (X, u) that was not used for training. The TVSPRUNE algorithm then uses this dataset to compute the moments Yg of for each layer 1, filter j. and class a. Using this data, TVSPRUNE then computes Min.VS((..) for each L, after this, the mean my and standard deviation or computed for the TVSEPvalues (MiTTVS(1,)) MinTVS(I.j) S 17 are pruned.
for each layer 1. After these are computed, those filters that satisfy
6.2 ITERATIVE PRUNING WITH TV SEPARATION
We motivate this section by noting that the TVSPRUNE algorithm does not give us control over the accuracy of the pruned model, or the extent that a model is sparsified. Therefore, it is unsuitable to solve either equation P In this section, we describe IPERTVSPRUNE, that builds upon TVSPRUNE, and which attempts to solve equation P. We also propose a variant of ITERTVSPRUNG that attempts to solve the problem of finding the most accurate model given a sparsity budget, without training data or access to the loss function. We refer readers to the supplemental material for this algorithm description.
iterative algorithm

---

Published as a conference paper at ICLR 2023
The ITERTVSPRUNE algorithm builds upon the fact that the distribution over (X) changes each time the previous layers are pruned. Thus, we may recompute the TV-Separation values, and iteratively prune the model. Note that as was the case with TVSPRUNE, this algorithm does not require any fine-tuning, and does not even require the training data.
The SPRRNV algorithm takes initial TEP threshold r), convolutional neural network with parameters W, an LDIFF threhsold V, and an accuracy threshold t. At each step k. we run the TVSPRUNE algorithm given the current parameters W(a) and a current TVSEP threshold Ths that is W (W 7k. D). After running TVSPRUNE, we check the accuracy of the pruned model if f(W) S t. then we repeat the pruning step, else we reduce 7) and run TVSPRUNE with Wa) and the new value of The utility of varying the TVSEP threshold is that as we increase the threshold, the number filters whose MinTTVS value exceeds the threshold reduces. Thus, as we reduce 77, we increase the acceptable degree of T V-sepaaration, thus making the pruning more gentle. We state the algorithm formally in Algorithm
EXPERIMENTAL RESULTS
In this section, we detail experimental results. We aim compare our method with various existing structured pruning techniques without any fine-tuning. Our experiments utilize the standard CIFAR-10 KKizhevskyy et al., (209)) and Imagenet (Russakovsky al. (015)) datasets under the MIT license. For computing moments, we partition the test set, which contains 10,000 samples, into two subsets of size 7,500 and 2,500 in the case of CIFARIO; for Imagenet, we partition the validation set into two subsets of size 40000 and 10000. We use the larger subset for the computation of moments, and the smaller subset for testing the accuracy of the pruned model. For our hardware setup, we refer readers to section F
We aim to answer three broad questions with our slate of experiments. First, do well trained CNNs have more filters with well-separated filter outputs than models that are not well trained? Second, using the LDIFF mechanism that utilizes the separability of intermediate features, can we gauge the "difficulty" of pruning a given layer? And last, how does our pruning mechanism compare with other existing methods in the scenario where training data and the loss function are unavailable?
For further experimental results, refer the reader to the supplementary material.
7.1 VALIDATING THE DISCRIMINATIV FILTERS HYPOTHESIS: WELL-TRAINED MODELS HAVE FILTERS WITH WELL SEPARATED OUTPUTS
In this set of experiments, aim show that models that generalize well have filters with well- separated output features, whereas models that generalize poorly do not. To measure separation, we compute
NL
1
MinTVS(I) out j=1
Σ
Min.VS(L
NI
that is, the mean minimum classwise TV-separation of filters in each layer. In order obtain models that do not generalize well, we modify the weights of a model with high test accuracy by adding zero-mean gaussian noise to the weights. In our experiments, we progressively increase the variance of the gaussian noise, in order to increase the test error. Furthermore, measure MinTVVS(( values for untrained models for additional comparison. Our experiments focus on VGG16 and ResNet18 models trained on the CIFAR10 and Imagenet datasets. For both architectures, and for both datasets, we observe that as the variance of the noise increases (and thus, the test error decreases), the average TV-separation per layer decreases. We also observe that the average TV-separation for untrained models remains low, and almost constant. Curiously, we note that even with the addition of Gaussian noise, the separation does not significantly change in the layers close to the input, particularly for VGG models. This therefore supports our hypothesis that models which generalize well possess discriminative filters, whereas models that do not generalize well do not.
7.2 DETERMINING WHICH LAYERS ARE DIFFICULT TO PRUNE WITH LDIFF
In this section, we investigate the utility of the LDIFF score for determining which layers can be extensively pruned, and which can't. The experiment we conduct is follows. First, compute the
7

---

Published as a conference paper at ICLR 2023
Tellec Menc ROHAD
0.15
BON ACH
02335
02333
6/
0.0
000
(a) VGG16 trained on (b) ResNet18 trained on (c) VGGI6 trained on Ima- (d) ResNet18 trained on CIFARIO with Gaussian CIFARIO with Gaussian genet with Gaussian Noise Imagenet with Gaussian Noise
Noise
Noise
Figure I: Comparison of accuracies of models with weights perturbed by Gaussian noise with fully trained and untrained models.
20-
Layar
LA5HT
Layer
Luyar
(a) Random Pruming on (b) Random Pruning on (c) Random Pruning on (d) Random Pruning on
VGG-16, CIFARIO
VGG-16, CIFAR100
VGG-19, CIFAR1O
VGG-19, CIFAR100
TNIT Srom
LEIFT Soom
Score
E11
(e) Layerwise LDIFF scores for VGG-16, CIFARI0 for VGG-16, CIFAR100 for VGG19, CIFARIO for VGG19, CIFARI00
(f) Layerwise LDIFF scores
(g) Layerwise LDIFF scores (h) Layerwise LDIFF scores
Figure 2: Top row: Effect of uniform random unstructured pruning of single layers on VGG16/19 models trained on CIFARIO/ION 'Prn_frac" refers the percent of the weights removed from the layer. Bottom row: LDIFF scores for corresponding unpruned models.
LDDIF scores for the VGG16 -19 models trained on the CIFARIO and CIFAR100 datasets. We use 512 samples for CIFARIO, sampled from the appropriate partition of the test set, and 4096 samples for CIFARI00. Then, we prune weights randomly from each layer in isolation; that is, for each experiment, only prune weights from single layer. We then measure the test accuracies. We observe that the LDIFF scores are small for the initial layers, thus indicating that those layers cannot be randomly pruned. This is borne out by our experimental results for both models trained the CIFARIO and the CIFAR 100 dataset. Last, we infer, based on the LDIFF scores and test accuracies of randomly pruned models, that pruning ratios of yLDIFF((, n)), where 7 < 1/2 would result in negligible loss of accuracy. Collectively, these observations support the use of the LDIFF score for determining the extent which layer can pruned. These results are displayed discussion and experiments are provided in the supplemental material in C
Further
7.3 EFFECTIVENESS OF ITERTVSPRUNE FOR STRUCTURED PRUNING WITHOUT FINE-TUNING
In this section, we compare the ITERTVSPRUNE algorithm with some existing baselines, in the setting where we do not have access to the training set or loss function. We consider VGG16, and -19 models trained on CIFAR1O, and ResNet50 and -56 models trained on Imagenet and CIFARIO respectively. In this set of experiments, and dataset; then, having obtained the pruned model satisfying the accuracy constraint, we compare with three baselines, using the same sparsity pattern obtained by SPRUNE. The baselines
ITERTV
solve equation for each model
8

---

Published as a conference paper at ICLR 2023
Accuracy Drop
Accuracy Drop
Accuracy Drap
-10
-20
-20
-30
-40
Trshure:
Tyshiun: DylerFO
1STurE
mendrr
-00
Parametnic sparsity pencentage
sparsty percentage
motricc Sparssy percentage
(a) VGGII on CIFAR-10
(b) VGG16 on CIFARIO
(c) VGGI9 on CIFAR-10
Figure 3: Comparison of accuracies of ITERTVSPRUNE on VGG models trained on CIFARIO
Table 1: Pruning ResNet Models with no training dataset and no finetuning on CIFAR-10 and Imagenet
Model
Dataset
Param. Sparsity
IPERTVSPRUNE
CHIP
L1
Rand.-3
4.76%
-3.02%
-3.41%
-7.86%
-13,9%
ResNer50
ImageNet
9.98%
-10.21%
-10088
-48.2%
43.1% 1%
24.65%
-31.3%
-34.2%
3.5%
-1,47%
-1.36%
-5.46%
-7.12%
ResNee16
CIFARIO
7.6%
-4.82%
-5.56%
N/A
-9.41%
12.3%
-9.86% %
-10.22%
-17411
-21.30%
chosen were L1 based pruming, CHIP Sui et al. 2021) (for ResNet models) and first order gradient based score based on (Prakash et al. 2019; Molchanov et al. 2019a) (for VGG nets), and uniform random pruning (accuracy is the average of three trials of random pruming); all were run without fine tuning. Modifications implementations are discussed in Appendix
We report our results for VGG nets CIFARIO in Figure and for ResNet models in Table 1 We observe that our method consistently outperforms the baselines in the task of structured pruning of models without fine tuning the model with the training set. Notably, for VGG19 trained CIFARIO, are able to remove more than 60% of parameters with minimal loss in accuracy, far exceeding the nearest baseline.
CONCLUSIONS AND DISCUSSION
In this work, we propose a new paradigm for pruning, which we call distributional pruning, which only requires access the data distribution. We make the observation that models with high predictive performance possess a mixture of filters that discriminate well, and those that discriminate poorly. Motivated by these observations, significant loss of test error. We use the TV Distance to quantify the discriminative ability of filters, using which define the heuristic LDIFF score and derive the TVSPRUNE Algorithm.
argue that
prune disiriimmiieeevee filters to avoid
The TRSPE algorithm's 's drawbacks are as follows. First, since classwise distances must be computed, the number of separations for each filter that the algorithm needs estimate are quadratic in the number of classes; limiting the scalability of the algorithm. Next, the relationship between fully understood, and requires further investigation. Future research directions include (a) improving the scalability of this algorithm to massive datasets, (b) evaluating other measures of similarity between distributions, and (c) analyzing parameter quantization through the lens of distributional separability.
TV-spaarability and generalization in DNNs

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We authors gratefully acknowledge AMD for their support. The authors also thank Ramaswamy Govindarajan (Professor, IISc), Himanshu Jain (ISSc), Raghavendra Prakash (AMD), and Ramasamy Chandra Kumar (AMD) for their insight and assistance in this work.
The authors thank the reviewers for their valuable feedback which has helped us improve our work.
REFERENCES
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6541 16549, 2017.
Cenk Baykal, Lucas Liebenwein, Igor Gilitschenski, Dan Feldman, and Daniela Rus. Data-dependent coresets for compressing neural networks with applications generalization bounds. In Interna- fional Conference on Learning Representations, 2018.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems, 2:129-146, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2018
Thomas Gebhart, Udit Saxena, and Paul Schrater. unified paths perspective for pruning at initialization. arXiv preprint arXiv:2101.10553 2021.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint 150.0:15 00/40 2015.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
Yang He, Guoliang Kang, Xuanyi Dong. Yanwei Fu, and Yi Yang. Soft filter pruming for accelerating deep convolutional neural networks. arXiv preprint 18Xiv: (808,088666 2018.
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4340-4349, 2019.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research,
1-1224, 2021.
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint 1rXvv:6602 2016
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32, 2019.
Saumya Jetley, Nicholas Lord, and Philip Torr. With friends like these, who needs adversaries? Advances in neural information processing systems, 31, 2018.
Donggyu Joo, Fojindl Yi, Sunghyun Back, and Junmo Kim. Linearly replaceable filters for deep network channel pruning. In The 34th AAAI Conference on Artificial (nAeligeene,(AAAA),, 2021.
Charles Kraft. Some conditions for consistency and uniform consistency of statistical procedures. University of California Publication in Statistics, 2:12--14,, 1955.
10

---

Published as a conference paper at ICLR 2023
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Dongha Lee, Sehun Yu, and Hwanjo Yu. Multi-class data description for di-ttiiibbtiion detection. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1362-1370, 2020.
Seungmin Lee, Dongwan Kim, Namil Kim, and Seong-Gyun Jeong. Drop adapt: Learning discriminative features for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 91-100, 2019.
Bailin Li, Bowen Wu, Jiang Su, and Guangrun Wang. Eagleeye: Fast sub-net evaluation for efficient neural network pruning. In European conference on computer vision, pp. 639-654. Springer, 2020.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 1608.0660.008/10 2016.
Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks. International Conference on Learning Representations, 2019.
Mingbao Lin, Rongrong Ji, Yan Wang. Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1529-1538, 2020a.
Mingbao Lin, Rongrong Ji, Yuxin Zhang, Baochang Zhang, Yongjian Wu, and Yonghong Tian. Channel pruming via automatic structure search. arXiv preprint arXiv:2001.08555. 2020b.
Jing Liu, Bohan Zhuang, Zhuangwei Zhuang, Yong Guo, Junzhou Huang, Jinhui Zhu, and Mingkui Tan. Discrimination-awann network pruning for deep model compression. IEEE Transactions on Puttern Anulysis and Machine Intelligence, 2021a.
Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In International Conference on Machine Learning, pp. 7021-7032. PMLR, 2021b.
Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In International Conference on Machine Learning, pp. 6682-6691. PMLR, 2020.
Fanxu Meng, Hao Cheng, Ke Li, Huixiang Luo, Xiaowei Guo, Guangming Lu, and Xing Sun. Pruning filter in filter. Advances in Neural Information Processing Systems, 33:17629-17640, 2020.
Molchanov, Tyree, T Karras, Aila, and J Kautz. Pruning convolutional neural networks for resource efficient inference. In 5th International Conference on Learning Representations, ICLR 2017-Conferenc Track Proceedings, 2019a.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11264 11272, 2019b.
Ben Mussay, Dan Feldman, Samson Zhou, Vladimir Braverman, and Margarita Osadchy. Data- independent structured pruning of neural networks via coresets. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Guillermo Ortiz-limenez, Apostolos Modas, Seyed-Mohsen Moosavi, and Pascal Frossard. Hold me tight! influence of discriminative features on deep network boundaries. Advances in Neural Information Processing Systems, 33:2935-2946 2020.
Shreyas Malakarjun Patil and Constantine Dovrolis. Phew:
Constructing sparse networks that learn fast and generalize well without training data. In International Conference on Machine Learning, pp. 8442. PMLR, 2021.
11

---

Published as a conference paper at ICLR 2023
Ankit Pensia, Shashank Rajput, Alliot Nagle, Harit Vishwakarma, and Dimitris Papailiopoulos. Optimal lottery tickets via subset sum: Logarithmic over-aarametrrizationn sufficient. Advances in Neural Information Processing Systems, 33:2599-2610, 2020.
Prafull Prakash, Chaitanya Murti, Saketha Nath, and Chiranjib Bhattacharyya. Optimizing dnn architectures for high speed autonomous navigation in gps denied environments on edge devices. In Pacific Rim International Conference on Artificial Intelligence, pp. 468 481. Springer, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211- 1-222, 2015.
Prem Seetharaman, Gordon Wichern, Shrikant Venkataramani, and Jonathan Le Roux. Class- conditional embeddings for music source separation. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 301-305, IEEE, 2019.
Suraj Srinivas and R Venkatesh Babu. Data-free parameter pruning for deep neural networks. arXiv preprint reXiv:157..00 49, 2015.
Yang Sui, Miao Yin, Xie, Huy Phan, Saman Aliari Zonouz, and Yuan. Chip: Channel independence-baset pruning for compact neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Xin Sun, Zhenning Yang, Chi Zhang, Keck- Voon Ling, and Guohao Peng. Conditional gaussian distribution learning for open set recognition. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 13480-13489, 2020.
Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks with- out any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems, 33:6377-6389, 2020.
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In European conference on computer vision, pp. 499-515. Springer, 2016.
Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debuggable deep networks. In International Conference on Machine Learning, pp. 1 11205-1110 1116. PMLR, 2021.
Sarthak Yadav and Atul Rai. Learning discriminative features for speaker identification and verifica- tion. In Interspeech, pp. 2237-2241, 2018
Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8715-8724, 2020.
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I Morariu, Xintong Han, Mingfei Gao, Ching-Yung Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 9194-9203, 2018
Shaochen Zhong, Guanqun Zhang, Ningjia Huang, and Shuai Xu. Revisit kernel pruning with lottery regulated grouped convolutions. In International Conference Learning Representations, 2021.
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrimination-aver aware channel pruning for deep neural networks. Advances in neural information processing systems, 31, 2018
Zhen Zuo, Gang Wang, Bing Shuai, Lifan Zhao, Qingxiong Yang, and Xudong Jiang. Learning discriminative and shareable features for scene classification. In European Conference Computer Vision, pp. 552-568. Springer, 2014.
12

---

Published as a conference paper at ICLR 2023
APPENDIX
In this supplemental material, we provide additional detail and empirical results to support the claims of the main paper. We organize this document as follows.
1. In section A for the sake of comprehensiveness, provide further discussion of relevant literature, including a discussion on the discriminative ability of convolutional filters.
2. In section we present additional discussion required for clarification purposes, including a brief discussion adopting the machinery used this paper to feedforward models.
3. In section we provide further details regarding the derivation of the LDIFF scores.
4. In section we provide additional experiments in using the TVSPRUNE and TVSPRUNE al- gorithms for structured pruning, We provide plots showing the number of filters pruned in each layer for the same experiments presented Section 7.3 of the original paper.
5. In section D we state a simple variation of the ITERTVSPRUNE algorithm that aims to solve slight variation of the pruning problem.
In the sequel, we list additional material added to this document.
6. In section El we provide additional experiments illustrating the utility and effectiveness of the ITERTVSPRUNE algorithm.
A ADDITIONAL LITERATURE SURVEY
In this section, we survey contemporary and historic results the area of neural network pruning. We focus filter pruning in this work, but there exists wide variety of works addressing problem of unstructured, or weight pruning. For a more detailed discussion on relevant results unstructured pruning, refer to (Hoefler et al. 2021 Blalock et al. 2020).
Filter Pruning Filter pruning refers to processes by which entire filters or channels are pruned from a model; this can be further extended to individual filters(Zhong et al. (02T) or 'stripes' (Meng) itt al., 2020) within filters. Initial results on structured pruning removed filters based on the LI
norm of the weight tensors (Li ct al. 2016). More recent advances involve using gradient based scores (Molchanov et al., 2019b Liu et al, 2021b; Molchanov et al. 2019a). rank minimization
(Lin et al. 2020a), and linear replaceability and channel independence (Joo al. [202]] Sui et al.)
2021). )), and other geometric properties (He et al. 20188 2019 Li et al.) 2020 Lin et al. 2020b) Yu
2018 developed sampling based methods for structured pruning that also provided generalization bounds. Lastly, in (Zhuang
al. 2018). Other works, such (Liebenwein al. 2019 Baykal
al. 2018 Liu al., 2222aap the discriminative capacity of filters is used to inform the pruning strategy by introducing discrimination-ave -wware loss functions. Our work differs from these in two key respects- first, we rank filters based on the discriminative ability of their outputs, and second, we use a novel metric based on the discriminative ability of filters, based on classwise distances between class-conditional output distributions of filters, to decide which layers can be pruned effectively.
Data-Free Pruning Data free pruning has recently gained significant interest (Hoefler et al., 202 however, while there are a variety of methods that do not use data for the task of pruning, there few methods that do not require fine tuning or retraining. Early works in this regime include (Srinivas) Babu 2015). which measured similarity between neurons, and merged similar neurons together. More recently. (Tnakaa et al.) 2020) proposed the SynFlow method, an unstructured, data free pruning method that relied on preserving gradient flow. This work was expanded upon in (Gebhart al. 1021|Patil & Dovrolis 2021), which utilize the Neural Tangent Kernel to improve upon SynFlow. Other approaches include (Mussay et al. 2021). which uses coresets of weights to perform structured pruning. Our work differs from this since utilize the class conditional distributions of the filter outputs to rank the filters, which we either obtain a priori, or compute using previously unseen data. In this section, we briefly survey works pertaining discriminative ability of neural networks, and unstructured pruning. We do so for the sake of comprehensiveness, and to discuss works that are tangentially relevant to ours.
13

---

Published as a conference paper at ICLR 2023
Discriminative Ability of Convolutional Neural Networks Studying the discriminative ability of features and representation at either the layer level or the filter level is an active area of research,
with upplications in speech identification (Yadav & Rail 2018), scene classification (Zuo et al. 2004))
face recognition (Wen et al., 2016), and domain adaptation (Lee et al.] 2019). Understanding the discriminative capacity of features is also crucial toward understanding adversarial robustness (Jetley et al. 2018 88] [lyas et al., 2019) and model interpretability (Bau et al.) 2017). In (Wong et al. 2021 I)
pretrained set of convolutional layers is given as the input a linear classifier which is trained with
-inducigg regularizer to identify discriminative filters. In (Ortiz-limenez et al. (020), the discriminative capacity of intermediate representations is used to analyze decision boundaries of CNNs. However, to the best of our knowledge, there are no works that utilize the discriminative capacity of filters for pruning models.
Discriminative Filters in Pruning Discriminative filters have also been applied to structured pruning of neural networks. In (Zhuang et al.) 2018 Liu et al., (021a), the discriminative capacity of filters is used to inform the pruning strategy. In particular, the authors define the discriminative ability of a filter as the impact of it's ouptut on the final classifer, and introduce a discrimination aware loss function, with which the model is fine tuned. Once this is done, the authors propose a greedy algorithm for channel selection that makes use of this loss function and the associated gradients. This method differs from this work, as we explicitly define discriminative ability in terms of the TV separation, and not on the impact on the final classifier.
Unstructured Pruning In this section, we provide a brief overview of unstructured pruning techniques. While unstructured pruning is not the focus of this work, provide this summary in the interest of completeness. and Unstructured pruning refers to processes by which individual weights pruned from model (Hoefler et al. 2021). Early works on unstructured pruning include (LeCun) let al. 1989 Hassibi & Stork] 1992), which relied on diagonal approximations of the Hessian of the loss function. However, such methods are not scalable very large models, prompting the creation of unstructured pruning methods that married a variety of techniques to achieve dramatic parametric sparsity (Han et al. 2015) landola et al. 2016). More recently, (Frankle Carbin (018) proposed the Lottery Ticket Hypothesis, which asserts that networks contain subnetworks which, when trained from scratch, achieve similar accuracies to the original model; stronger versions of this result were proved in (Malach et al. 2020; Pensia et al. 2020). This remains an active area of research (Blalock
et al. 2020).
B ADDITIONAL DISCUSSION
In this section, we include additional discussion on certain aspects of this work. In particular, we briefly discuss the extension of our method to linear, feedforward networks, and provide additional discussion on the definitions and use of patches.
B. EXTENSION OF PROPOSED METHODOLOGY TO LINEAR LAYERS
Our proposed methodology can easily be extended to feedforward/linear layers as well. Broadly speaking, we consider the layerwise outputs to be the outputs linear layers, as opposed to convolu- tional layers. In that case, it is possible to derive the same measures of distributional discrepancy. We provide a more rigorous derivation below:
Let (X- be the output of the / - 1-th linear layer, and let W be the weight tensor of the I-th layer. Then,
Y((x) = (x)), W
(7)
If (X) is a vector, let (X) be the jthh element (the jth neuron). Then, we simply define
ya -(X [X)) and andoi,j
Exx,u)=D= [[]((X) [0]
(8)
as we did for convolutional layers. However, in this case, outputs are vectors. Thus, each Y, is single real number (along with the expectations and the variances). Using this, we can then define
HELLD? (Qi, Q2) 1 20102 of
(M1 22)
A
oz
of + oz
and
MinTVS(1 min TV a.9
Di
D
2 min 1 exxx-p 0
exp(
A 0.8 (. /4)
(10)
14

---

Published as a conference paper at ICLR 2023
where
8 Aij (Y1)
(1I)
B.2 DESCRIPTION OF PATCHES
patch is subset of the input datum with which dot product with the convolutional filter taken. The input datum is made up of several patches which may or may not overlap. We describe this formally in the sequel.
For simplicity, assume that the input is in R RM and each convolutional filter is a tensor in I x where K M. In convolution operation, the filter is applied to subsets of size K K of the input; that is, the output is a dot product of the filter and that subset of elements of the input. Each such subset called a patch. We describe this further in Section of the supplemental material. Furthermore, we refer the reader to, for example, (Goodfellow al. 2016).
C THE LDIFF SCORE: ADDITIONAL CONTEXT AND EMPIRICAL ANALYSIS
In this section, provide additional context and information regarding the LDIFF score described in Section 6.1, which is a heuristic used to quantify the extent to which a layer may be randomly pruned.
C.I REFINEMENTS OF THE LDIFF SCORE
In this section, we discuss refinements the LDIFF score presented in section 5 Recall that in this work, focus identifying and pruning non-diseriminative filters from individual layers. Thus, as mentioned previously, our goal is to derive a score, leveraging the aforementioned hypothesis and using the discriminative ability of the filters in given layer, that a layer is difficult to prune.
deduce whether
Assuming the hypothesis is true, good measure of the extent to which model can be pruned could be given by the fraction of filters that are highly discriminative. Thus, if a layer has only few highly discriminative filters, should be possible to sparsify that layer extensively.
In the LDIFF score defined in equation 6 we fix the threshold to identify discriminative filters. We first present methods with fixed thresholds that utilize statistics of the layerwise MinTVS scores here.
C.1.1 MEDIAN- SEEIAN-BASED LDIFF SCORES WITH A FIXED THRESHOLD
We begin by recalling definitions from the original paper.
Let MinTVS(I) be the median value of MinTVS(L, j) for given (that is, with respect j). We recall that a filter is discriminative if it is TVSEP(T) for some 1) > O; furthermore, recall that can vary from model to model, and dataset dataset. With this, we first define the following sets.
Suppose that for least one j, the corresponding filter is TVSEP(1) for some 0. We define the sets
= = - Ai, < nh and S = ( A7, 2 Á..)
The set Sn captures the number of filters that are not TVSEP()), and thus non- discriminative.
The setš; is the set Ai, greater than the median.
Let [S] denote the cardinality of a set S. We then define
[ St
LDIFFo(()
(12)
[&
The set Sy,1 n S captures the number of non-discriminative filters in the top two quantiles. Thus, LDIFFg(() captures the fraction of filters in the top two quantiles that are discriminative. quantity equals 1, we conclude that at least half the filters are TVSEP(n); thus LDIFF, captures the mix between discriminative and non non-discriminative filters. Thus, if LDIFF, (1) is close to I, then the layer is difficult to prune.
this
15

---

Published as a conference paper at ICLR 2023
This is illustrated in Figure 9 where plot the LDIFFo(() scores for each layer for VGG16 6 and VGG19 models trained on the CIFARIO 10 and CIFAR100 datasets. We choose 1 = 0.025, and we that the initial layers of VGG16 and VGG19 models trained CIFARIO appear be very easy to prune, since there should be no discriminative filters. Instead, however, those layers cannot be effectively sparsified.
In order to compensate for this, we chose another way to quantify the mix between discriminative and non-discriminative filters in layer. Specifically, we define LDIFF equation 6 which captures the ratio of the separations that are below the median Å, and those above it. As we will show in the sequel, this is a more effective variant for capturing the difficulty of pruning filters, or indeed weights, from a layer.
02
0.
60
13
Layer
Layer
Layer
(a) Layerwise
L.D (b) Layerwise
LD- (c)
Layerwise
LD- (d)
Layerwise LD-
IFFo scores for VGG-16, IFFo scores for VGG-19, IFFo scores for VGG16, scores for VGG19,
CIFARIO
CIFARIO
CIFARI00
CIFARI00
Here, we observe that the LDIFFo score is effective determining whether layer difficult to prune for models trained on CIFAR100, 100, but is seemingly not as effective for models trained on CIFARIO In particular, for models trained on CIFAR10. we observe that the initial layers receive low scores since filter outputs for those models are not well separated. This lends credence to the notion that if there are no filters the sequel, we discuss variants of the scores defined herein and propose variants, including the score proposed in the final paper.
layer that are TVSEP(T), then we should avoid pruning that layer entirely. In
C.2 FLEXIBLE THRESHOLLD LDIFF SCORES
potential drawback of using fixed thresholds for discriminative ability, however, that the scale of the distributional separation may vary from layer to layer. That is value of that ensures discriminative ability one layer may not apply to another. In this section, we extend this definition more flexible variant, wherein utilize layerwise thresholds. We consider multiple variants, including cases where thresholds are fixed priori and variants where thresholds are fixed based on statistics of the MinTVS values for a given layer. Furthermore, layerwise thresholds for discriminative ability provide the added benefit of being able to capture the 'mix' of discriminative and non- discriminative filters.
C.2.1 COUNTING- BASED LDIFFSCORES USING LAYERWISE MinTVS STATISTICS
In this section, we focus on methods that utilize the statistics of MinTVS values for given layer to determine pruning thresholds. A variant of the method used in the main paper follows.
Define Min Min TVS()) to be the mean MinSVS for a given / (that is, with respect to j), and let > 0
be constant. Then, define
Σ (MinTVSS(..)) < MinTVS())
LDIFF (1)
(13)
Σ 1 (MinTVS((.)) > Min
The LDIFF score utilizes the mean MinT'VS score for layer to capture the relative discriminative ability filters in layer, along with a constant ?r. For each layer, this measure counts the number of filters that are discriminative relative to the MVSS score for that layer. As mentioned previously, useful thresholds of discriminative ability can vary significantly from architecture to architecture, layer to layer, and dataset dataset.
16

---

Published as a conference paper at ICLR 2023
C.2.2 MASS-BASED LDIFFSCORES USING LAYERWISE MVSSVSS STATISTICS
The previous counting-bant bassed measures pruneability, even with flexible thresholds, fail to account for the wide range of values that MinTVS scores can take, particularly in wide networks with small number of very discriminative filters. order to address this issue, we define the following variant of the LDIFF score.
Min (MINTTSS.)) < (100)>>MMimm) MinTVS())
LDIFF(() Σ MinTVS(L,)) (MinTVS(L.) > MinTVV(()
(14)
This variant of LDIFF score compares the ratio of MinTVS((,j) scores that lie below MinTT)
to the sum of MinTVS((,j) scores that lie above the discriminative ability of scores below the (variable) threshold those above it. Hence, we use the term "Mass-basce LDIFF scores. This captures the spread of discriminative ability, and can be shown to more effectively predict which layers can be randomly pruned.
Thus, we compare the 'mass" of
LEMET Stom
LDIT Scom
(a) Layerwise LDIFF scores for VGG-16, CIFARIO
(b) Layerwise LDIFF scores
(c) Layerwise LDIFF scores for VGG-16, CIFAR100 for VGGI9, CIFARIO
(d) Layerwise LDIFF scores for VGG19, CIFARI00
Figure 5: Mass based LDIFF scores. Here, a higher score means it is more difficult to prune.
D VARIANTS OF ITERTV
In this section, we describe variants of the TVSPRUNE algorithm. First, describe method that utilizes flexible thresholds, and second, we describe a variant to solve the pruning problem with fixed sparsity budget.
D.1 TVSPRUNE WITH FLEXIBLE THRESHOLDS
This section describes a variant of the TVSPRUNE and ITERTVSPRUNE algorithms that utilize variable thresholds for discriminative ability. In this case, specifically use the mean MinTVS score to determine layerwise pruning thresholds. Furthermore, we use the bass- LDIFF score described in equation 14 to determine which layers prune.
17

---

Published as a conference paper at ICLR 2023
Algorithm 3: TVSPRUNE meaan
Algorithm 4: ITERTV
Input: Dataset D (Xe, un )a Pretrained
Input: Dataset D = (X., uz 3 Pretrained Wi.). 70 0,
CNN with parameters
CNN W = (W1,
W = (Wi.... Wt...7 7 > 0, layer difficulty threshold v
LDIFF threshold v, 6, > 0, accuracy
threshold 1, Tiain <
Compute v/.0. .iii. for all 5, and
Set k: = 0, W(o) = W
Compute A for each a, B.
while Y 2 Ymiin
W = TVSPRUNEmen (D, W(o),,
for [z] do
Compute MinTVS((,,) for all L
if f(W) Σ then
Compute MinTVS((I
W(++1) W
Nout MinTVS((.5)
else
No 21=1"
y t 9 8y W(K+1) f W(()
Compute LDIFF(() from equation 14 if LDIIF(() < then
à - k ++1
for j E [N do if MinLVS((,)) < MMTTVS() then W +
Output: W (Wi,... WL).s(W) St. supp(Wi) S supp(Wi) VI
return W
Output: W- (W1,.... WL), where
supp(Wi) (Wi) (Wi) V
return W
This algorithm functions in a similar fashion to the TVSPRUNE and ITERTVSPRUNE algorithms, except that the thresholds for discriminative ability are flexible (varying between layers), and are decided by the mean of the MinTVVS scores for each layer. Furthermore, just as the threshold for discriminative ability were reduced at each iteration of SPRRN, we vary the constant 7 thus enabling to flexibly prune each layer even the distribution of MinTVS scores changes with each pruning iteration.
D.2 FIXED-BUDGEY PRUNING
In this section, briefly describe variant of the ITERTV algorithm that, using only the data-generating distribution, prunes a model with a fixed sparsity budget using only the data- generating distribution. This variant attempts solve the problem of finding the most accurate model subject to sparsity budget. Formally speaking, this variant of the algorithm attempts to solve 0}
arg min (fW) [ < K, where || = E, ||Wilo and where K is a sparsity budget satisfying K < ||W/ While simple option to simply increase 7) in the TVSPRUNE algorithm (thus raising the threshold as which filters are discriminative), the proposed variant makes use of the fact that the distributions of the filter outputs change each iteration, owing to the filters pruned in the previous step This algorithm,
(P2)
Algorithm 5:
Input: Dataset 1 = (Xi, ui 1 Pretrained CNN W = (W1, 1. WE), initial TVSEP threshold n). LDIFF threshold v, 5n > 0, sparsity hudget K, Thain <
Set k 0. W(o) W
while 2 Mmii
W- TVSPRUNE(D, W(a),
if [DVVlo 2 K then W(+11 < W
cise
" 7 bn
W + Wik)
k +1
Output: W (W1, ..... WL). I11/ s K.
return W
ITERTVSPRUNE-SB (SB refers sparsity budget), iteratively calls the TVSPRUNE algorithm
18

---

Published as a conference paper at ICLR 2023
prune fraction of filters each step. At the kth iteration, if [|]lo K; that is, the fraction of parameters drops below the sparsity budget K, we rewind to the previous value of Wg. and reduce 17. thereby ensuring that prune more gently. The drawback of using this algorithm is that Mmin needs to be chosen carefully so that we are guaranteed to prune some filters at each iteration. While we do hypothesis that may remove non-diseriminative
not directly control for accuracy, rely filters without suffering catastrophic losses in accuracy.
E ADDITIONAL EXPERIMENTS USING ITERTVSPRUNE
In this section, we provide additional experimental details on the use of the ITERTVSPRUNE algo- rithm.
HARDWARE SETUP
Our experiments were conducted on two systems, one with an NVIDIA GTX1060 GPU and an Intel 17-7700, and one with dual NVIDIA RTX3090 GPUs and Intel 19- 1990F. All experiments done using the PyTorch framework, in particular, for obtaining the output features for each filter. Moments and TV VVVPvaluess were computed using standard numpy packages.
DATA SET SPLITS
We briefly discuss the split of the data used in our experiments. As mentioned in the main paper, we do make use of the training data at all. However, since would be drawn from the class conditional distributions of the CIFARIO and CIFAR 100 datasets (i.e. obtaining a large number of images of cats, planes, horses etc), we utilize the test data. We partition the test data into two, with one partition receiving 7500 images, and the other 2500. We describe this in Table
unable obtain sufficient data that
Table 2: Breakdown of dataset splits used in experiments.
Dataset CIFARIO
Training Set
Mean Computation Set
Test Set
Not used
7500 images from test
2500 images from Test
CIFAR100
Not used
7500 images from test set
2500 images from set
Imagenet
Not used
40000 images from validation set
10000 images from validation set
Hyperparameters For all experiments, the value of v as used in Algorithm 2 of the main manuscript chosen empirically, and varies for each experiment. The value for is chosen to be 0.15. The value of v/ is chosen to be 0.5. We state the hyperparameters used in or experiments using fine-tuning in Table
Table 3: Hyperparameters chosen for fine-tuning pruning model with the CIFARIO and Imagenet datasets.
Hyperparameter Learning Rate
Batch Size
Epochs
Momentum
Scheduler
Weight Decay
ValuR/CIFAR10
0.001
128
50
0.05
Cosine
0.9
Value/Imagenet
0.001
256
70
10-6
Cosine
.0.99
BASELINE METHODS
In this section, we briefly describe our modifications to common baselines.
1. Sui et al. (2021: We ensure that the algorithm only has access to the validation set, which is split into a 40000/10000 pair. When rumning the algorithm, we limit the number of samples seen by the algorithm to 512 (1 batch of 512) for CIFAR1O. For Imagenet results, reduce the number of repetitions 1.
19

---

Published as a conference paper at ICLR 2023
2. Taylor first order methods: produce bespoke implementation of the methods described in Molchanov et al. (2019a/b). We allow the algorithm to see 512 samples for CIFAR 10 to estimate gradients, before pruning. We treat this as one-shot pruning method.
3. Other methods: for L1 and random pruning, we use native PyTorch implementations.
E.1 SUMMARY OF EXPERIMENTAL RESULTS AND METHODOLOGY
In this section, we provide tables summarizing our empirical results.
We apply |SRRTV to achieve accuracy drops of at most 1.5%, 5%, and 10%, variety of architectures and datasets. We then check the accuracy observed when we prune the same networks, on the same datasets, with baseline pruning algorithms. We detail this list below. All experiments do not make use of the training set, except experiments where we apply CHIP Sui et al. (202I) toward pruning the models.
1. 8/CFFA1
Baselines: LI Pruning, Random Pruning, Taylor First Order Pruning
2. ResNec50/CIFAR
Baselines: Pruning, Random Pruning, Taylor First Order
3. ResNee56/CIFARRO
Baselines: LI Pruning, Random Pruning, Taylor First Order, CHIP Sui et al. (2021
4. VGGI6/CIFARI0
Baselines: L1 Pruning, Random Pruning, Taylor First Order, CHIP Sui al. 2021
E.2 CIFARIO RESULTS
In this subsection, we detail our experiments on models trained using the CIFAR10 dataset.
E.2.1 VGG MODELS
In this subsection, we compare VGG models trained on the CIFARIO dataset. Specifically, we compare the efficacy of SPERTVSPRUNE VGGI1, VGG16 and VGG19 models, with standard baselines. In our experiment, we first use SVERTSSPRUNN.... to prune the models to a given accuracy threshold; that is, solve qquation[Pooor different values t. Then, using the same sparsity patterns, we apply L1 pruning, random pruning, and first-order gradient-based pruning respectively. Note that here too, do not fine tune the models after pruning. We plot these results in Figure For clarity, we also present these results in Table
Parametric sparsity We observe that using ITERTVSPRUNE dramatically outperforms several standard baselines. Indeed, we are able to prune more than 60% of parameters in VGG19 trained on the CIFAR10 dataset with only 2% drop in accuracy. We also observe, from table 4 that on VGG16, [TERTV also outperforms contemporary baselines such as CHIFSSi et al. (2021)
Layerwise Structured Sparsity In Section 7.3, we report the parametric sparsity that is, the number of parameters pruned from the model using our proposed algorithms. In this section, consider the layerwise structured sparsity achieved using the ITERT IVERTV SPRUNE algorithm. In our experiments, observe that irrespective of accuracy threshold, most of the pruned filters lie in the final layers of the VGG models, and almost no filters are pruned in the initial layers.
Key Takeaways The key takeaway of this slate of experiments that VGG models trained on CIFAR 10 can be sparsified extensively with ITERTVSPRUNE. We see that our method consistently outperforms the closest baselines, often by double digit percentages (in terms of test accuracy) for the same sparsity.
E.2.2 RESNET EXPERIMENTS
In this section, we detail our experiments with ResNet models trained on the CIFARIO dataset. The models consider the ResNet18 88, -20, -50, and -56 models. Our procedure mirrors that used
20

---

Published as a conference paper at ICLR 2023
90
60
50-
60
40
FesPrann TayorT EN.
IVSPYARS Tolor FC DNL
Toyar BN.
20
rxstion
meden
randon
10
15
20
30
10
50
03
70
Parameniic sparsiy percentage
Paremetric Sparsity percentage
Perametric spershy perce
(a) Parametric Sparsity compari- son for VGG-11 on CIFAR-10
(b) Parametric Sparsity compari- son for VGG-16 on CIFAR-10
(c) Parametric Sparsity compari- son for VGG-19 on CIFAR-10
133
Proned
hiterrs Pruned
X0-022
3
Layer
Layer
Layer
(d) Structural Sparsity compari- son for VGG-11 CIFAR-10
(c) Structural Sparsity compari- son for VGG-16 100
(f) Structural Sparsity comparison for V9G-19 on CIFAR-10
Figure 6: Comparison of accuracies of different pruning algorithms applied to VGG models trained on CIFAR-10
xxc 113
M0--T 40-11 Filters Prunce
ax-(33 Prunee
20+013
200-00
300-13
Layer
Layer
(a) VGG-11 on CIFAR-10
(b) VGG-16 on CIFAR-10
(c) VGG-19 on CIFAR-10
xec 523
30c 43
s0r C3
sc--223
M00-C2
Piltare Promee
ax-523
Filters Prunce.
ac-00
ax-353
204433
200-02
300-533
Layer
Layer
(d) VGG-11 on CIFAR-100
(c) VGG-16 on CIFAR-100
(f) VGG-19 on CIFAR-100
Figure 7: Top row: Structured sparsity pattern after ITERTVSPRUNEfor VGG models trained on CIFAR10. Bottom row: Structured sparsity pattern after VVERTVSPRUNEÉor VGG models trained on CIFAR100.
the previous section, with VGG nets: we use SPERTY to prune models with different accuracy thresholds, and then apply baseline methods with the same sparsity patterns obtained via ITERTVSPRUNE. We present the results of our experiments in Table
Key takeaways The key takeaway from this slate of experiments is that IrERTVSPRUNE is effective on residual networks as well. However, is not as effective on narrow resnet models trained on CIFAR10 10 as it is VGG models trained the same dataset. We observe that the extent to
21

---

Published as a conference paper at ICLR 2023
Table 4: Pruning VGG Models with no training dataset and no finetuning on CIFAR-10
ModelParan Sparsity
Pruning Method
Aec. Drop
Comp. with SYPUUNSPPUUNE
ITEKI VSPaUSE
0.52 %
Kandom
2.12%
-1.6%
VGG-I1/5.999
2.02%
-1.5%
CHIP
ITERTVSPRUNE
3,52 %
Random-3
21.32%
-17.8%
VGG-11/24.0%
24.02%
20.5%
Gradient-FO
CHIP
VSPUUN
Random-3
61.35%
-47.8%
VGG-11/3188
53.12%
-90 656
Gradiem--OO
33.92%
-0.4%
CHIP
VSPUUE
Kandom-3
50.668
48.7%
VGG--6637.68
25.3%
-23.4%
Gradien-FO
YSPRUNE
Random
65.3%
61.4%
VGG-16652.5
L1 35.0%%
-30,7%
Gradiems-FO
26.4%
ISERUVSPRUNN
Kandom
72.7%
09.8%
VGG-16667.5%
L1
47.8
34.9%
Gradient-FO
43.2%
-30,3%
16.0%
Random-3
22.9%
-1.66
F.
25.88
-24.5%
Gradiens-FO FO
6.0%
-4.7%
CHIP
ISPUUNE
Random-3
VOG-18164.59
Li
28.7%
-4.4%
Gradiess-P
8.7%
4.4%
CHIP
VSPRUNE
7016
Random-3
$1.88
VGG--1972.39
33.7%
-2 4%
Gradieas-FO
15.0%
-7.3%
CHIP
which "narrow" ResNet models, such as ResNet20, can be sparsified is significantly less than "wider" ResNet models, such as ResNet18, adapted for CIFAR10.
E.3 CIFAR100 RESULTS
In this section, we detail our experiments models trained on the CIFAR100 dataset. The architec- tures considered were VGGI1, VGG16, and VGG19. The experimental setup was identical the previously presented experiments with the same models trained on the CIFARIO dataset. We plot those results below in Figure S and provided a detailed snapshot Table
E.4 EFFECT OF SIZE OF VALIDATION SET
We conduct brief experiment to illustrate the extent which the size of the set of samples used to compute the class-conditional moments, and the MinTVS scores, affects the ranking in terms of MinTVS. For VGG16 model trained the CIFAR10 dataset, sample 1024, 768, and 512 samples, and compute the min 2 where and 3 classes, I denotes the layer, and j denotes the filter, as defined in Section 5 of the main manuscript. We present plots for three layers below.
22

---

Published as a conference paper at ICLR 2023
Table 5: Pruning ResNet Models with no training dataset and no finetuning on CIFAR-10
ModelPParan Sporsity
Pruning Method
Aec. Drop
Comp. with SYPUUTS
ITEKI VSPaUSE
1.50%
Kandom
13.21%
-11.71%
ResNeet//19.112
7,88%
6.38%
ITERT
Ramdom-3
19.89%
-15,33%
R 8/33.9115
14 42.0%%
10 3696
Gradiens: FO
10.02%
5.46%
ITERTVSPRUNE
10.0%
Random-3
778 909
8/39.22
18.76%
Gradient-FO
18.44%
-8.3%
1.4%
Kandam-3
5.258
-388
ResNer2W/2.44
18.766%
Gradiess-FO
VSPRUNE
Kandom-3
13.29
33.%%
ResNer20/4.89
L1
8.9%
-4.2%
Gradiess-RO
6.23%
-1.63%
IVERTVSPRUNE
9.41%
ResNee20/7.11 1%%
17.3%
-7.8%
Gradiemz-FO FO
14.8
-5.9%
1.50%
Random-3
25.5%
-44.%
ResNer50/24.00%
Gradiens-
9.82%
-7.32%
YSPRUNE
Random-3
26.25
-11 58%
ResNer5t1/2998
17.3%
-14688%
Gradiess-RO
7.2%
ITERT VSPRUNE
9.9%%
Random-3
Res
24.3%
-1.36%
Gradicns FO
14.25%
4.31%
ITERT VSPUUE
147%
Random-3
-5.0%%
ResNet56/3.04k
3.09%
Gradient-FO
4.12%
2.65%
CHIP
1.36
0.11
ITERT VSPUUN
4.83%
Random-3
23.42%
-18600%
ResNer56/7.69
Gradiens: 9.11%
-4.9%
CIIPP
5.56
-0.74
We observe that even when we reduce the number of samples by factor of 2, we see that the ordering of filters according to mina.9 A!d (and thus, MinTVS(I;)) is retained even for comparatively smaller sample sizes.
E.5 EXPERIMENTS WITH FINE TUNING
In this section, we detail our results using [TERTV in the setting wherein we have access the training set and loss function, thereby enabling to fine-tune our models after pruning. Our experiments involve fine-tuning models trained on CIFARI0 and Imagenet. Broadly speaking, our experiments demonstrate two facts.
1. Models pruned by our algorithm are able to return to the accuracy of the original pretrained model,
2. Fine- Fine-tuning our models yield models that are competitive in accuracy with those obtained using current, state-of-the-art methods such as Sui al. (2021).
23

---

Published as a conference paper at ICLR 2023
10
10
accuracy
20
20
Tyshuue Tayle TO SN
Todar th.
10
rander
10
18
29
10
20
30
Porametiicc sparsity percentage
Paremetticc sparsly percentage
Perametric spershy perce percentage
(a) Parametric Sparsity compari- son for VGG11 on CIPAR10
(b) Parametric Sparsity compari- son for VGG16 on CIFAR10
(c) Parametric Sparsity compari- son for VGG19 on CIFARIO
563
Proned
20-422
Iitters
3
Layer
Layer
Layer
(d) Structural Sparsity compari- son for VGG11 CIFAR10
(c) Structural Sparsity compari-
(f) Structural Sparsity comparison for VGG19 CIFARION 100
son for VGG16 CIFAR10
Figure 8: Comparison of accuracies of different pruning algorithms applied to VGG models trained on CIFAR100
512 lamples
lamplas
Samples 512 Gample
3.90
$12 Soe
Sar
36-
1:75
1.44
3.0
8.0
Flar Numoor
(a) mina.9 29.8 for layer of
(b) mina.s A A..8 for
layer of
(c) mina..
A for layer of
VGG16
VGG16
VGG16
Figure 9: Effect of sample size on ranking of min A
9.8
E.5.1 RESULTS OF PRUNING WITH FINE TUNING ON CIFAR10
We now present the results of fine-tuning models trained on the CIFAR10 dataset. We chose to fine-tune VGG16 models pruned using VERTVSPRUNE and CHIP et al. 202T), and ResNer18 model pruned with ITERTVSPRUNE: We detail our results below in Table
We observe that after fine-tuning for epochs, we are able to recover the accuracy of the original model, even when the model is pruned substan- tially. with 67.5% of parameters removed, as is the case with VGG16.
E.5.2 RESULTS OF PRUNING WITH FINE TUNING ON IMAGENET
We now present the results of fine-tuning models trained on the Imagenet dataset. We chose to fine-tune VGG16 models pruned using ITERTVSPRUNE and CHIP Sui et al. (021), and ResNet18 model pruned with ITERTVSPRUNE: We detail our results below in Table
As was the case with models trained on CIFARIO, we observe that our models recover the original accuracy after substantial fine-tur fine-tuning. However, the extent of fine-tuning required to achieve this is substantially greater for models trained on Imagenet than for models trained on CIFAR10.
24

---

Published as a conference paper at ICLR 2023
Table 6: Pruning VGG Models with no training dataset and no finetuning on CIFAR-100
Model Name Parmm. Spacitt
Prasing Method
Acc. Doop
Comp. w/ MSaRTNSPRUNE
ITER VSPRUNE
1.6
Random-3
4.2%
2.6%
VGG-11/5.998
L1
5.7%
-4.%
Gradien--90
CHIP
SPRUTV
5.6
Random-3
30.4%
-24.8%
VGG-11/17.99
L1
21.1%
-1..5
Gradien-FO
CHIP
VSPUUE
Randum-3
45.4%
-3..8%
VGG-11/21.38
F.g 37.2%%
-24.6%
Grddees-PO
CHIP
YSPUUE
Random-3
42. 42.745
55. 18%
VGG-16623.61
L1
34.20%
-2.75
Gradien-FO
LIEKT
Random-3 3
66.35
-51.4%
VGGG-1 /31.9
L1
35,696
-30.5%
Gradiem-40O
31.35
ISPRTV
18.56
Random-3
50.08%
-3.52%
VGG-16 BCG--16/0 44..2%
L1
40.38%
-2.825
Gradien-FO
32.36%
-13.8%
VSPRUE
Random-3
38.
VGG-19 37.8%
F.g
26
-26 46
Gradien: FO
6.2%
6.0%
VSPUNNE
5.2%
VGG-19 59.05
L1
29 39.8%
-24.6%
Gradien-RO
12.1%
4.9%
VSPRUUN
VGG--19 72.3%
L.
34.7%
-16.5%
Gradiem: HD
22.8%
4.0%
Table 7: Accuracy of Fine- Tuned models trained on CIFAR10. "Acc. Drop w/o FT" refers to the accuracy drop obtained without fine-tuning the model post pruning, and "Acc. Drop w/ FT denotes the accuracy drop after retraining the model.
Model Name /Param. Sparsity
Pruning Method
Ace. Drop w/o FT
Acc. Drop w/ FT
VGG- (DGG-16/6..55 .5%
ITERTVSPRUNE
12.9 %
0.21 %
VGG- (666-66/67.59/
CHIP
16.6%
0.64%
BesNee18/339255
ITERTVPPRUNE
10.0%
0.06%
E.6 VALIDATING USING Min'VS FOR PRUNING
In this section, we conduct experiments validating the us MinTVS scores, as well as the key hypothesis of the paper. We show that for models trained on the CIFARI 10 dataset, pruning filters with the smallest MinTVS scores achieves superior accuracy compared pruning filters with the largest MinTVS scores. We detail this experiment below.
E.6.1 EXPERIMENTAL. SETUP
We consider VGG16 and VGG19 models trained on the CIFARIO dataset. We consider models with L.DIFF scores that are greater than 0.4; that is, I.DIF((M.))) > 0.4, where m 0.75MinTVS() is
25

---

Published as a conference paper at ICLR 2023
Table 8: Accuracy of Fine-Tuned models trained on Imagenet. "Acc. Drop w/o FT" refers the accuracy drop obtained without fine-tuning the model post pruning, and "Acc. Drop w/ FT denotes the accuracy drop after retraining the model.
Model Name /Param. Sparsity
Pruning Method
Acc. Drop w/o FT
Acc. Drop w/ FT
ResNee50024..7% ResNNc55024.7%
ITERTVSPRUNE
31.2 %
0.03 %
CHIP
34.2%
0.26%
0.75 times the mean Min score for the layer. The partition the validation set for accuracy measurement and computation is given in 2 We do so since the variation in Min TVS scores for layers with low LDIFF scores minimal; thus, there is little difference between the largest scores and the smallest.
E.6.2 EXPERIMENTAL RESULTS
We now present experimental results. We observe that pruning discriminative filters in general yields mode dramatic drops in accuracy when compared to pruning non-discriminative filters, though the effect is slightly less for deeper layers (those closer the output). This observation supports the hypothesis that, given layers with a mix of discriminative and non-discriminative filters, we can safely prune the non-discriminative filters, but the former. We present these results in Figure TO
(a) VGG16, 10% of filters
(b) VGG16, 25% of filters pruned
(c) VGG16, 50% of filters (d) VGG16, 75% of filters pruned
pruned
pruned
(e) VGG19, 10% of filters (f) VGG19, 25% of filters pruned
(g) VGG19, 50% of filters (h) VGG19, 75% of filters pruned
pruned
pruned
Figure 10: Comparison accuracies of when pruning fractions of filters with least MinT'V scores with pruning fraction of filters with greatest MinTVV scores for VGG16 and VGG19 trained on CIFARIO. "Dist. max" refers to accuracies when most discriminative filters are pruned, and "Dist. min" refers to accuracies when least discriminative filters are pruned
26


Reviews:
[{"review_id": "WE9-ekadtE", "reviewer": "Reviewer_Ghni", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The proposed work uses the hypothesis: Non-discriminative filters do not contribute a lot to the predictive performance of a network and can be removed safely. Keeping this in mind, along with a privacy based setting where the original training data and loss function are unavailable yet the a sample from the original distribution is available, the proposed work uses TotalVariation distance between class conditioned distributions as a measure to identify discriminative filters. In addition, the proposed work offers LDIFF score as a way to ascertain the extent to which a layer possesses a mixture of discriminative and non-discriminative filters. Overall, these two ideas are combined to provide a pruning algorithm.", "summary_of_the_review": "Certain choices in parameters and rationalization behind the problem domain are the main concerns when it comes to the proposed work. Addressing the weaknesses highlighted above should solves these concerns.", "strength_and_weaknesses": "Strengths\n- The technical contribution and its relevant information have been explained well.\n- The overall structure of the manuscript is solid and helps the reader digest the information in a steady manner.\n\nWeaknesses\n- Could the authors comment on how to connect the proposed relaxation back to data-free structured pruning? (Pg. 2, Paragraph 1, Line 2-5)\n- Could the authors contextualize the theoretical setting assumed in the proposed work, of class conditioned distribution from various layers of the network, given that storing such information consumes extra memory? In addition, when iteratively updating the pruning algorithm, how much do the distributions vary, post pruning of the previous layer? Are their effects exaggerated on the distribution of successive layers?\n- Could the authors provide more insight in choosing Random Pruning as a baseline to understand the impact of TV distance? Since due to its weak assumptions it provides loose bounds and a measure that provides tighter bounds could inform the pruning better.\n- Could the authors clarify and define the meaning of the notations used in Equation P? \n- Could the authors describe and discuss in detail the statement \"Note that we also observe that some layers have features that cannot discriminate well, and yet cannot be pruned.\"? \n- Could the authors explain how values of $\\eta$ were obtained?\n- From an experimental perspective, could the authors clarify whether selecting the filter with the highest  TV distance matched with the largest drop in performance? (i.e., the values of TV distance corresponded to the expected ranking of discriminative filters)\n- Could the authors clarify if a batch size of 30000 was used for experiments in Section 7.2?\n- Given the extreme drop in performance, relative to a small amount of parameters being removed (Table 1), could the authors discuss if TV distance and LDIFF could be further highlighted when framed similar to more standard pruning setups?\n\nPost Rebuttal\n- Based on the responses provided by the authors, I have updated the final recommendation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LPGVYU-GFGD", "reviewer": "Reviewer_fFbD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper authors propose a mechanism to prune a convolutional neural network model in a relatively data-free manner i.e., they do not utilize training data or loss function for retraining the pruned model. However unlike the actual data-free pruning techniques they assume the availability of moments of class-conditional distributions of the activations. They make a critical assumption that these distributions are Gaussian to exploit cheaply computable sufficient statistics in deriving tractable bounds which are used to guide the pruning process.\n\nThey propose a pruning method which exploits the proposed metric to both decide the extent of pruning for a given layer and the actual filters to prune in a given layer, without measuring the impact on the down-stream layer outputs either in terms of the deviations in the metric or in terms of actual performance. This is a reasonable operation under the fundamental hypothesis informing this paper, i.e., discriminability of a filter strongly correlates with performance impact on pruning it. However they do design an iterative version of their pruning method which explicitly measures the cumulative impact of pruning at all the layers on the over all model.\n\nThey show appreciable reductions in performance degradations for a given pruning budget compared to other data-free pruning methods. ", "summary_of_the_review": "This paper guides the reader through the design of the algorithm and motivates the design decisions with well validated/referenced hypotheses. It addresses a problem of critical interest to this community. The writing style is clear. The references are more than adequate. The proposed technique is has limited but sufficient experimental validation. Though there are minor possible improvements the paper in its current form is already useful to the readers.", "strength_and_weaknesses": "Strengths\n------------\nThe paper clearly calls out the hypothesis and constantly justifies the algorithmic decisions in the context of this hypothesis. \nIt performs intermediate validation exercises for their hypothesis, which motivate the reader and guide them through the author's intuitions.\nThe supplementary material is very exhaustive and helpful in further clarifying the details of the proposed algorithm/metrics.\n\nWeaknesses\n---------------\nThe motivation for data free pruning is not clearly described in the paper. The readers are forced to rely on the references.\nIt would be very informative to the reader to compare the sparsification potential of this technique to a pruning method which exploits fine-tuning post compression.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Jb8iVcsLru", "reviewer": "Reviewer_gKo4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a training-data free structure pruning method to prune deep neural networks. Specifically, the authors propose to measure the discriminating ability of filters through the total variation (TV) distance between the class-conditional distributions of the feature maps outputs by the filters. Then, based on the above TV-distance, the authors define the LDIFF score to decide the pruning ratio of each layer. Last, the authors proposed IterTVSPrune, which iteratively prunes the model to achieve greater sparsity. Experimental results on CIFAR10/100 and ImageNet demonstrate the effectiveness of the proposed method. However, there are still some issues in the paper. Detailed comments are as follows.", "summary_of_the_review": "The authors propose a training-data free structure pruning method to prune deep neural networks. The proposed new paradigm called distributional pruning for pruning neural networks is novel. The idea of data-free pruning is not very novel. The manuscript is easy to read and provides enough experimental details to reproduce. Experiments on the image classification task demonstrate the effectiveness of the proposed method. However, the manuscript still needs more experimental results and explanations.", "strength_and_weaknesses": "## Strength:\n\n1. The authors propose a new paradigm called distributional pruning for pruning neural networks. It measures the discriminating ability of filters through the total variation (TV) distance between the class-conditional distributions of the feature maps outputs by the filters. \n\n2. Experiments on the image classification task demonstrate the effectiveness of the proposed method. \n\n3. The manuscript is easy to read and provides enough experimental details to reproduce.\n\n## Weakness:\n\n1. The proposed method may be hard to be applied in real data-free scenarios because it needs to assess thousands of images from the test set (3/4 of the training set) to calculate the distributions of feature maps. Can the proposed method use less number of images (or images from datasets with similar distribution) to calculate the distributions of feature maps? More explanations and experiments are required.\n\n2. In Algorithm 1, the authors set the weights of the pruned filters as zero rather than remove them directly. In this way, the pruned model may still cost the same storage space to store these zero weights and consume the same computation cost to forward with the zero weights. How does the pruned model reduce the storage space and the inference time? More explanations and experiments are required.\n\n3. In Section 5, the motivation of proposing LDIFF metric to identify which layers can be pruned is unclear. The authors claim that LDIFF metric tends to prune layers with a mixture of discriminative and non-discriminative filters and avoid pruning layers with a majority of discriminative filters or non-discriminative filters. Why should the LDIFF metric avoid pruning layers with a majority of non-discriminative filters? It would be better to visualize and analyze the fraction of discriminative filters $\\tao(\\eta)$ in each layer.\n\n4. In Definition 1, the meaning of the function \"sup|·|\" in the equation is confusing. It would be better for the authors to explain the meaning of \"sup|·|\".\n\n5. In Section 7, it would be better for the authors to provide the theoretical complexity measure and the time consumption of the proposed method.\n\n6. The authors only show the results on heavy-weight models such as ResNet. It would be better for the authors to conduct more experiments on light-weight models such as MobileNet-V2 [1].\n\n7. More ablations of the $\\eta$ are required because the influence of $\\eta$ is unclear. How does this hyper-parameter affect the performance of the proposed pruning method? \n\n8. The authors only present the experimental results on image classification task. It would be better for the authors to show the experimental results on more computer vision tasks such as object detection and semantic segmentation.\n\n## Minor Issues：\n1. Many references in this paper have been officially published, such as “Baykal et al. (2018)” and “Frankle & Carbin (2018)”. Please reference these papers in the formal format.\n\n2. There are some typos in this paper:\n(1) In Section 7.1, \"in order to reduce the test error\" should be \"in order to increase the test error\"?\n(2) In Section 6.1, \"Therefore, we use the LDIFF scores to decide We now present the TVSPRUNE algorithm.\" is an incomplete sentence.\n\n## Reference:\n[1] MobileNetV2: Inverted Residuals and Linear Bottlenecks. CVPR 2018.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ISYV8lQ_Ul", "reviewer": "Reviewer_tFxW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed to solve the problem of formalizing and quantifying the discriminating ability of filters through the total variation (TV) distance between the class-conditional distributions of the filter outputs.", "summary_of_the_review": "The setting of pruning without finetuning is interesting. But many arguments and experimental settings of this paper are not valid or confusing.", "strength_and_weaknesses": "+The setting of pruning without finetuning is kind of attractive.\n\n-The calculation of MinTVS requires samples and their labels. It's hard to understand why authors emphasize they do not need the loss function. Compared to computing loss and using related channel or filter importance scores, MinTVS may not have advantages in terms of efficiency.\n\n-The claim of 'not using training set' is also confusing. Authors resplit the test set to calculate MinTVS. If the whole test dataset is used for evaluation, then MinTVS must use samples from the training set. \n\n-Although the authors claim that they do not finetune the model, the performance of their method does not have significant advantages compared to regular pruning methods like CHIP.\n\n-The experimental settings are also confusing. It's good to show accuracy without any finetuning process. However, some results are quite meaningless. For example, losing more than 30% accuracy for ResNet-50 on ImageNet when only reducing around 25% parameters. Authors should provide meaningful results by including the finetuning process.\n\n-The proposed algorithm includes tuning several hyperparameters, and how to decide the final pruning rate is not straightforward.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster