A Greedy Hierarchical Approach to Whole-Network Filter-Pruning in CNNs
Authors: Kiran Purohit
Anurag Reddy Parvathgari
Sourangshu Bhattacharya
Venue: Accepted by TMLR ()

Abstract: Deep convolutional neural networks (CNNs) have achieved impressive performance in many computer vision tasks. However, their large model sizes require heavy computational resources, making pruning redundant filters from existing pre-trained CNNs an essential task in developing efficient models for resource-constrained devices. Whole-network filter pruning algorithms prune varying fractions of filters from each layer, hence providing greater flexibility. State-of-the-art whole-network pruning methods are either computationally expensive due to the need to calculate the loss for each pruned filter using a training dataset, or use various heuristic / learned criteria for determining the pruning fractions for each layer. Hence there is a need for a simple and efficient technique for whole network pruning. This paper proposes
a two-level hierarchical approach for whole-network filter pruning which is efficient and uses the classification loss as the final criterion. The lower-level algorithm (called filter-pruning) uses a sparse-approximation formulation based on linear approximation of filter weights. We explore two algorithms: orthogonal matching pursuit-based greedy selection and a greedy backward pruning approach. The backward pruning algorithm uses a novel closed-form error criterion for efficiently selecting the optimal filter at each stage, thus making the whole algorithm much faster. The higher-level algorithm (called layer-selection) greedily selects the best-pruned layer (pruning using the filter-selection algorithm) using a global pruning criterion. We propose algorithms for two different global-pruning criteria: (1) layerwise-relative error (HBGS), and (2) final classification error (HBGTS). Our suite of algorithms outperforms state-of-the-art pruning methods on ResNet18, ResNet32, ResNet56, VGG16, and ResNext101. Our method reduces the RAM requirement for ResNext101 from 7.6 GB to 1.5 GB and achieves a 94% reduction in FLOPS without losing accuracy on CIFAR-10.

Published in Transactions on Machine Learning Research (08/2024)
A Greedy Hierarchical Approach to Whole-Network Filter- Pruning in CNNs
Kiran Purohit
hhnnnppuoohttkkgiinnniikkk oc.in anungporaathhgar778@mmma com sourungshu Sese. itkkp. ac in
Anurag Reddy Parvathgari
Sourangshu Bhattacharya
Deportment of Computer Science and Engineering Indian Institute. of Technology, Kharagpur, India
Reviewed on OpenReview: https: // openreview. net/ forum? id- WzftuebRSg
Abstract
Deep convolutional neural networks (CNNs) have achieved impressive performance many computer vision tasks, However, their large model sizes require heavy computational resources, making pruning redundant filters from existing pre-trained CNNs essential task in developing efficient models for ressuree-costtrained devices. Whole-network filter pruning algorithms prune varying fractions of filters from each layer, hence providing greater flexibility. State-of-the-art whole-network pruning methods are either computationally expensive due to the need to calculate the loss for each pruned filter using a training dataset, or use varions heuristic / learned criteria for determining the pruning fractions for each layer. Hence there need for a simple and efficient technique for whole network pruning. This paper proposes two-level hierarchical approach for whole-network filter pruning which is efficient and uses the classification loss the final criterion. The lower-level algorithm (called filter-pruingg) uses a sparse- aprrxxiiaxinnnonnn formulation based on linear approximation of filter weights. We explore two algorithmss: orthogonal matching pursuit- pursuit-based greedy selection and a greedy backward pruning approach. The backward pruning algorithm uses a novel closed-form error criterion for efficiently selecting the optimal filter at each stage, thus making the whole algorithm much faster. The higher-level algorithm (called laycr-selection) greedily selects the best-pruned layer (pruning using the filter-selection algorithm) using a global pruning criterion. We propose algorithms for two different global-pruning criteria: (1) layerwise- relative error (HBGS), and (2) final classification error (HBGTS). Our suite of algorithms outperforms stato-of-the-art pruming methods on ResNet18, ResNet32, ResNet56, VGG16, and 1ex Our method reduces the RAM requirement for ResNext101 from 7.6 GB to 1.5 GB and achieves a 94% reduction in FLOPS without losing accuracy on CIFAR-10.
1 Introduction
Convolutional neural networks (CNNs) have demonstrated remarkable performance across various applications, such as image classification (Han et al. 2016), object detection (Redmon et al. 2016), and image segmentation [Minace al. 2021). However, the deployment of CNNs on ToT devices for computer vision tasks often
encounters practical bottlenecks related to the model size and computational complexity of inference (FLOPs)
(Goel et al. 2 While neural architecture search (Baker et al. 2017 Zoph & Lej 2017) and efficient
model design Tan & Le 2019) can sometimes lead to highly efficient architectures, they impose substantial requirements in terms of data and computational cost, as well rescarch expertisc. However, pruning of
pre-trained models (Lebedev & Lempitsky 2018 Hoefler et. al. 2021 Vadera & Ameen, 2022 He & Xiao
2023) provides a cheaper alternative where one can avoid re- re-training complicated models on large datasets.
For CNNs, stractured pruning or filter-pruning (FP) (He et al.
2017
Luo et al., 2017 He & Xiao 2023 has
emerged as a preferred alternative since it causes reduction in computation (thus leading power savings) well memory requirements without requiring special hardware re-implementation of operations.

---

Published in Transactions on Machine Learning Research (08/2024)
No
Greedl Layer
destred pruning enttera achieved?
Selection
Filters of selected Layer
Filters to prune using approximation
Remaining Filters
Protrained Model
Layer Selected
Efficient Model
Figure 1:
Hierarchical approach for non-uniform pruning of filters across the network.
Pilter-pruning (FP) techniques can further be classified as (1) layer-wise pruning, which prune filters uniformly from each layer (e.g. score-propagation (Yu et al. 2018) and error in activation reconstruction (Luo et al, 2017 ), and (2) whole-nelwork pruming (WNP), which prunes filters from the entire network. The WNP approach can prune different fractions of filters from each layer, hence providing higher flexibbiitty An important challenge for WNP is to determine the pruning fractions for each layer. (Kuzmin et al. 2019) (section 3.4.1) calculates the accuracy by pruning individual layers to a varying fraction and finds an optimal compromise that the overall pruning ratio is achieved while minimizing the maximum loss in accuracy per layer. The main disadvantage of this approach is that the effect of pruning one layer on the pruning of another layer is not captured. Recent works also include methods based on Taylor-scrics expansion of the loss function, which approximates the influence score of pruning a filter on the overall loss function (Wang el al.) 2019; Molchanov et al. 2019 Peng et al., 2019 Nonnenmacher et al. 2021) with good practical performance (He & Xiao] (023). However, these methods can be expensive for large networks as they require passing over the entire training set to calculate each influence score, which can be costly for large datasets. Additionally, (Dong& & Yang. 2019) applied NAS to search for network with flexible channel and layer sizes, but this method can also be expensive for larger networks. On the other hand, some recent works use approximate criteria to prune filters. For instance, (Murti et al.) 2023) propose discriminative pruming technique based on total variation separation distance (TVS), which is an approximate criterion to prune filters from a network. Similarly, (He et al. 2020) choose different criteria to prune filters from different layers using Gumbel-softmax. However, the main drawback of this procedure is that the Gumbel-softmax smoothing only calculates an approximate output feature map for each layer, thus potentially hurting overall performance. Therefore, there is a need for an efficient and accurate WNP technique that directly optimizes the training data loss.
In this work, propose greedy hierarchical training data loss-based approach for whole-network filter pruning (see fig. D The iterative higher-level algorithm (called layer-selection) evaluates all layers based on outputs from the lower-level algorithm, and greedily selects a layer to prune filters from in each iteration. The lower-level algorithm (called filter-pruning) pruncs filters optimally for the current network configuration. We propose two versions of the ilerative layer-selection algorithm: (1) hierarchical backward greedy search (HBGS), which selects layers based on the relative reconstruction error of the layer outputs, and (2) hicrarchical backward greedy tree search (HBGTS) which selects the layers based on the crror of the final classification layer outputs. The key advantage of our greedy dayer-selection, compared to learned criterion He et al. 2020), or a threshold-based criterion (Kuzmin et al. 2019) is that we utilize the activations from the modified network, which arguably leads to better decisions after a few iterations of pruning. However, since each iteration of the greedy layer-sclection makes many calls to the filter-pruning algorithm (typically, the number of layers many calls, with some possibility of caching old results), an expensive filter-pruning algorithm would be impractical for large networks.
A key contribution of this paper is to propose an efficient filter-pruning algorithms, which can ensure the feasibility of the overall hierarchical scheme for large networks. While LRF (Joo et al. 2021 h demonstrated impressive practical performance for pruning filters, it only prunes one filter at-a-time, hence making it prohibitively expensive for our purpose. We formulate the problem of optimally pruning multiple filters from a layer using linear replaceability criteria as a sparse approximation problem. We study an orthogonal matching pursuit (OMP) (Tropp & Gilbert 2007) based algorithm, FP-OMP (Purohit et al.) (2023) for filter-prunt pruning.. Under the assumption of restricted isometry of the matrix composed of filter weights (Tropp) & Gilbert 2007), FP-OMP selects filters whose línear combinations can represent the pruned filters with
2

---

Published in Transactions on Machine Learning Research (08/2024)
minimal error. However, since FP-OMP follows greedy forward selection algorithm, and is called iteratively to prune a small number of filters, the overall algorithm becomes computationally expensive. To alleviate this inefficiency, we propose FP-Backward approximation problem. A key facilitating factor towards a fast implementation of FP-Backward is the calculation of a closed-form expression for the approximation error incurred by pruning a filter. HBGTS along with FP-Backward (called HBGTS-B) is an efficient algorithm taking only 50% of the running time of HBGTS and with performance comparable it.
backward elimination-based algorithm for solving the sparse
Experimental results a variety of standard pre-trained CNN models, e.g. ResNet18, ResNet32, ResNet56, VGG16, ResNext101 on standard image classification datasets, c-g. CIFARIO, CIFARI00, Tiny-Imagenet show that models pruned with HBGS and HBGTS have higher accuracies compared to recent state-of- the-art pruning methods for similar compression levels (see Table | and Figure 2 At higher parameter reduction (290%), the proposed methods outperform existing methods by ~ 5% (see Figure 2 We also find the optimal pruned model to have a highly non-uniform pruning fraction distribution for each layer (sce Figure 4 hence showing the effectiveness of our layer-selection algorithm. To sunmarize:
1. We propose a novel greedy hierarchicnl framework for non-uniform pruning of filters with fülter-pruming at the lower level and layer-selection at a higher level.
2. We propose a backward-eliminationn based scheme, FP-Backward for filter pruning, which takes advantage of a novel closed-form expression for approximation error.
3. We propose HBGTS which uses an efficient implementation to directly optimize the classification error for layer selection.
2 Related Work
Many pruning methods have been proposed in the literature. provide excellent surveys for pruning techniques. Pruning can be categorised two types: unstructured pruning, involving the removal of individual weights (Han et al.] 2015 and structured pruning or filter-pruiing (FP), in which entire nodes or channels are removed (He et al. 2017 Luo et al.) 2017 He & Xiao 2023). Structured pruning provides efficiently implementable models on wide range of accelerator devices e.g. GPUs. (He &
(Heller et al.
2021 Vadera & Ameen 2022
Xiao 2023) provides recent survey and website for comparing structured pruning techniques for CNNs. Pruning can be done on a pre-trained model or from scratch, which is costly and requires large training data. Therefore we focus on pruming pre-trained model. We further categorise it into the following groups: Weight-Based Pruning- - Weights of filters are examined to determine which ones are essential for the model's performance. These methods do not require input data. (Han et al. 2015 focused on eliminating small-norm weights. (He et al.] 2019) incorporates geometric median (Fletcher et al. 2008 to estimate the importance of each filter. (Joo et al. 2021) prunes linearly replaceable filters.
Activation-Based Pruning- Rather than relying on filter weights, these methods utilize activation maps layer outputs to make pruning decisions. We can utilize information ion from act maps at the currend layer or all searrs//woole-nettee
Some of the current layer activation- 2017) which focuses on minimizing the reconstruction error of sparse activation maps, while HRank (Lin et al. 2020) calculates the average rank of activation maps. CHIP Sui et al. 2021 assesses cross-channel correlation
pruning methods are: CP (He et al.
to evaluate channel importance. ThiNet (Luo et al. 2017 El Halabi et al. 2022 approximates activation maps of layer I+1 using subsets of layer I's activation maps. Current layer activation-based pruning methods do not consider the reconstruction error propagation. Some of the all lagers/whole wwwlll--ooooorrr activation-based pruning methods are: NISP YYu et al. 2018 which assesses the Final Response Layer to calculate the neuron importance, while DCP (Zhuang et al. 2018) aims to retain discriminative channels. Layer-output based methods are computationall expensive, since they need to compute the outputs using a training datasel. Regularization Regularization can aid in learning structured sparse networks by incorporating various sparsity regularizers. These regularizers can be implemented on Batch Normalization (BN) parameters (Liu) et al. 2017 You al.) 2019 Kang & Han| 2020). ; with extra parameters (Huang & Wang 2018 Lin al.
2019 and filters (Wen et al. 2016
Chen al. 2021
3

---

Published in Transactions on Machine Learning Research (08/2024)
Taylor Expansion- Taylor Expansion is employed to approximate the change in the loss by pruning filters. Pirsl-order- Taylor uses the first-order information to calculate the loss change caused by pruning weights (Molchanov et al. 2019: You et al., 2019). Second-onder- Taylor exploits the Hessian matrix, containing
sccond-order information [Peng et al. be expensive for large networks.
2019 Wang et al. 2019 Liu et al. 2021
However, these methods can
Corcset- Baasset-Bssed Pruning- - Recently, several methods (Tukan et al. 2022: Liebenwein et al. 2019 Mussay
iet al., 2021) have utilized the concept of coresets for pruning DNNss These approaches focus on providing bounds on the approximation error incurred at each layer of prediction. However, they do not necessarily achieve optimal pruning in terms of representation error. (Tukan et al. (2022) employs an upper bound on sensitivity for sampling filters, which can result in overly pessimistic sampling weights. On the other hand, our method focuses on removing linearly redundant. filters, which is done optimally for a given layer.
There a different line of work in which pruned models are effectively trained from scratch e.g. Frankle & Carbin (2018); Rachwan et al. (2022). Unlike training- based approaches, our method does not require training from scratch, which is costly and requires large training data.
3 A Hierarchical Greedy Approach to Filter Pruning
We propose a Hierarchical scheme, HBGS/HBB /HBGTS for non-uniform filter prusing from a pre-trained CNN. As shown in Figure [ the proposed scheme operates in a two-level hierarchical manner: (1) filter praming at the lower level, this step identifies the most appropriate filters to be pruned from each layer and (2) layer selection at higher level this step selects the best layer currently prune from. These two steps are applied iteratively to achieve non-uniform pruning from the whole network. We first describe our sparse approximation-basel formulation for optimal filter pruming from each layer, and then describe a faster backward elimination-based backward greedy search
algorithm for the same. For layer selection, we describe a rnverwise-regressionn- strategy. We also incorporate an overall error-based strategy for layer selection.
3.1 Sparse Approximation for Filter Pruning
A convolutional filter used in deep CNN is denoted by a K x K matrix. A convolutional layer is defined as filter weights fajj E TRK where = 1,..., m and j = 1,..,, n are the number of input and output channels. Given the input feature map with mn channels X = [X,,... Xm), the output feature map with re-haanels X, E
Y= (Y,...,,,,,) Y, can be calculated as: Y, =
X
Here, * denotes the convolution operation, and f.j E 2 xm denotes all the filter weights for output channel j. For brevity, we describe the algorithm for output channel pruming throughout the paper. Input channel pruning can be performed analogously, For channel pruming, we follow the idea of linearly replaceable filters (LRF) introduced in (Joo et al., 2021)), which states that any filter f., E RK* K can be pruned if the filter weights can be expressed as linear combination of other filter weights of the same layer which are not pruned. Note that, for linear approximation of a filter with respect to other filters of the same layer, we treat the filter weights, f.j, as a flat mimm-iinnni vector. For LRF, we prune the channel j such that ||s/|l is minimum, where Jaj = Eits AgaSa, + + Here, Λj are the coefficients of 1th filter for approximating the 5th filter, which can be computed by solving the minimization problem: minx,; IIfii E Aj. LRF [Joo et al. 2021) works by iteratively pruning one filter using the above technique, and updating the weights of the retained parameters using one epoch of SGD updates minimizing combination of training loss and knowledge distillation loss (Hinton et al. 2015 w.r.t. to unpruned model outputs.
The above method can be generalized by selecting a set of filters, S, in one go. Given the set of selected filters S, the error for jth output filter, f.j & S is given by G2:
f.z = Σ t
(1)
IES
The problem of estimating \jA1 I E can be posed as a sparse approximation problem:
S* X = |
Elll.s Σ Agaffall
(2)
JE(1,2...m)
LE
4

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm Filter Pruning-OMi OMP (FP-OMP)
Algorithm 2 Layer Selection: HBGS
1: Input: n: Number of filters,
1: Input: C: Number of layers,
3: Pruning fraction,
2:
Fg c= 1,.., C: Filters,
fug E IR RR j I.....:: Filters
3:
D: Training datasot,
4: Initialize:
4:
a: Number of filters pruned in one go,
5: Normalize Jig such that ||
5:
3: Total pruning ratio over all layers
6: R = fis Vj 21, n > Residual error
6: Initialize: y. = = vo6 c 1,..,, C, t t 0
7: S=
b Sct of selected filters
7: while Overall pruning ratio < 3 do
8: while (1-
3) * do for in 5 do
8: c = 0 =c= 1,
9: G + FP-OMP(n, 2 F.) Ve = L.... C
10:
for j in {1, 2,, n.. do Compute Proju = R..
where n |F
11:
for i 1,... [D] do
12:
end for Total projection b-= Projjyl end for
for c= 1,.., C, do
13:
12:
Calculate output vi(1) = F
14:
e c
Ilv. G. o (1)ll
15:
ind = max
14:
end for
16:
S - S SU {ind}
13:
end for
17:
for j in {1,2, .., n do
16:
cman arg Ee
18:
aagmmng I
Revised network params pi+1
emin
G Gomin
19:
R, 5 fij - Eies
F++1 = FF + cmin
20: end for
18:
+
21: end while
19:
Run epoch of finetuning.
22: Output:
20: end while
23: S, Aza V Vj E (1,2..,,m
21: Output: Pruned filters Fg Ve I,.... C
where n is the initial number of output channels in the current layer, and pruming ratio B, is the fraction of channels that are pruned. Algorithm 1 describes an orthogonal matching pursuul (OMP) based approximation Tropp & Gilbert 2007 Cai & Wang 2011) for estimating the S, A3d Vj = 1,.., n2; / E S. Note that, equation donotes a multi-variate regression version of the sparse approximation problem where the predicted variable is a vector Juj.j = I..,, n with corresponding independent parameters λj.- Since the total error the sum of squared errors of the individual components, it is easy see that projections used in standard OMP algorithm can be replaced with the sum of projections for each multivariate regression component (line 13 of Algorithm D. This approach has two advantages: (1) this approach is much faster than LRF since the fine-tuning is performed once for each layer, whereas in LRF it is performed after every pruning step (which is equal to the number of pruned filters in a layer), and (2) this approach provides an optimality guarantee for the selected filters in terms of reconstruction error, under conditions on the incoherence matrix of the features (Cai & Wang 2011]. The overall time complexity of algorithm [ O([S]n²). In a normal application sccnario of uniform pruning, the pruning fraction may be quite high (~ 98%), resulting in the size of the selected set [S] being much smaller than n, this algorithm is fast (O(n)))
LRF also uses 1 X 1 convolution layer 9jkkj,k L,..., to compensate for the loss of channel outputs. The modified output feature map, Zk, h = I,..., n is given by Z₁ = =E== 9j.k = ΣE= X faj * 9j.k, when the output filters Ýj.j = 1,..., n are not pruned. However, after pruning, the output feature map from the original convolutional layer becomes Y = EIes X s.t- Weight compensation. is a method for modifying weights for the x 1 convolutional layer, L. S, l,..,... n such that the final predicted output Z Eies X f.r di. matches Zz. The following result provides formula for calculating gi.x-
Result 1. Given Ze. Zhe gi.k: and gi. defined as above, and AJA.J = l...., L E S estimated using the filter pruning process. Letting P.. 9t,k + Eres Ap grrks VI E S, 1,..,,,, ensures that Zu 7 Evese X ey k. where ey is the error vector for the estimation of removed filter 1' S", and Se denotes the set of all removed filters.
5

---

Published in Transactions on Machine Learning Research (08/2024)
For brevity, the derivation of the result is provided in the appendix. This result provides us with formula for updating the weights of the 1 x 1 filters, thus obviating the need to update them using the SGD update.
3.2 Hierarchical Backward Greedy Search (HBGS)
The algorithm outlined in the previous section selects fixed fraction (1- 33 of filters from each layer. However, as shown in the results, each layer can have different fraction of important filters, depending on the architecture. Hence, determining the fraction of filters 3. to be pruned in layer e is an important problem. untuvvivlly Se should not be determined by the filter-weights since comparing them across layers is not meaningful. For example, the weights in a layer may be scaled by a constant factor, compared to those in another layer. Hence, we use reconstruction error of filter outputs using input training datasct as the criteria.
Let D == {(11; u1), -..; (uN, )N be the training dataset, and ye(i) be the output feature map of layer e when the training datapoint (u,, 04) is input to the CNN. Also, let Uz(i) be the output of the eh layer when the training datapoint (u,, 24) is input to the unpruned CNN. Moreover, let F. = <Eves, (S.." * o (), vk = 1,.... ne] be the composite convolutional map of the pruned filters and 1 x convolution for layer e obtained from a filter pruning method (c.g. FP-OMP described in the previous section). The relative reconstruction error eg for [+0,())l2z 2001, We propose hierarchical backward greedy search (HBGS) technique algorithm 2 to both estimate 3e for each layer c, as well as select the appropriate filters from each layer. Given the number of filters to be pruned in one go, the algorithm proceeds iteratively by performing two broad steps in each iteration: (1) determine the optimal filters to be pruned in each layer e, and (2) calculate the total relative reconstruction error ee described above. Finally, the model parameters are updated to prune filters from the layer that leads to the lowest relative reconstruction error. Algorithm 2 line 9 describes the first step, and lines 10 - 15 describe an efficient implementation of the second step, where errors for all the layers are computed in one forward pass per example. The itcrations continue till an overall pruning criterion, e.g. parameter pruning ratio or percentage FLOP reduction is reached. The parameter a chosen to speed up the overall execution and can be chosen as 1 if the running time not a concern. The overall time complexity of algorithm 2 when using algorithm [ as the filter pruning algorithm is: O(TC(N + n²)), where T is the number of iterations needed to achieve the desired pruning (depends a and the pruning criteria), C is the number of layers, N is the training dataset size, and n is the number of filters in each layer. While HBGS (Algorithm 2 can select variable number of filters from each layer, the sequential search over the layers for the best filter renders this algorithm expensive. In the next section, we develop a faster filter pruning algorithm.
layer is given by: C. = LiDDD
3.3 Backward Elimination Algorithm for Filter Pruning
The time complexity of the HBGS algorithm depends on the training set size N and the average number of filters per layer n. In many cases, when the time complexity of the filter pruning step (O(TCn*) is substantially larger than the error computation step O(TCN), the complexity of the filter pruning algorithm becomes substantially larger than that of the fine-tuning algorithm on the training dataset. The main problem is the OMP- filter pruning algorithm (FP-OMP) adds one filter in each step, which is an efficient strategy if the nunber of filters to be selected is small, compared to the total nunber of filters. However, in the context of HBGS algorithm, FP-OMP is sequentially called many times (Algorithm 2 line-9) with decreasing number of filters to be selected each time. In this context, a backward elimination (Couvreur & Bresler 2000 Ament & Gomes 2021 based approach which iteratively removes the feature which causes a minimal increase in approximation error, is intuitively more appropriate. While the original backward elimination algorithm described in (Couvreur & Bresler 2000 is O(n)), a faster implementation based on matrk-matrix inversion was described in (Reeves) 1999 with time complexity of O(n²). Here, we derive a
similar algorithm for our problem.
For simplicity, we follow the notation in (Reeves 1999 For a given layer with n output channels, m input channels, and K K filter size, we re-define the input filter matrix as A E R kmmxn where each column is a flattened vector of filter weights, A.j = J.j,J I,.... n. We also denote the output of the sparse approximation as B KKmxx which the same as A in this case. We are interested in the approximation B R AX, Ran, where Xÿ is the weight vector for the jth output filter. We note that the least, square
6

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm Filter Pruning- Backward Elimination (FP-Backward)
1: Input: n: Number of filtors, 8: Pruning fraction,
Juj E IK km j = 1, :..:: Filters
2:
Initialize: S nI,..,,
Set of currently retained filters
Bij [fall==,.....
Matrix of predicted filter weights
A= B
> Matrix of retained filter weights
6: while |S| > (1 8)* do A]
E S|x(81
7:
8:
for 1.... [S] do
9:
gk = G (()
10:
7k G(k. kt)
11:
dç A kgk TA
14 B
12:
uk
7M
13:
end for
14:
k- wwgmmnn=- 1..........111111........1
15:
$ t S [S[
b remove original index corresponding to k* b remove selected column
16:
A A= A,11-*)
17: end while
18: Calculate À using equation 3
19: Output: Set of selected filters-S, A
solution for Xr is decoupled from A.g where j # j'. Hence, the least squares solution becomes:
argmin
Σ
| Σ A, = =(A"A) B.s.
Vj =
=1,...
n
33)
jE(1.2.....)
1=1....
Hence, the total least square error is given by:
B)A,B)= = Σ |Ba A(ATA)
Σ BB BT (.)...)) j=h....f
(4)
JE
We are interested in calculating the increase in E(A, B) 1 column of and the corresponding row of X are removed from the input. Let = [A_ agl'I where A- is the sub-matrix of after removing the A column ag, and IT is the permutation matrix which permutes the columns of A that kth column is the last. We also have the following definitions of Gks Iki 7k Dr and dy:
G T
gk 7h
A)
D
GRAT + ghag of A + t/a
(5)
We note from equation 4 that only the second term in E(A, B) is dependent A. Hence, state the following result which connects the above equations to compute the increase in the least square error for the case of Mullimariate Linear Regression.
Result 2. Given the definitions of A and 7 above, the following relation holds: Σ3 BT, A A - = E(A, Ej=1.... 1 B.g.. 7
ATT E, +Td B.g hence, E(Ax.,
This result is a generalization of the result reported in (Reeves 1999). For conciscness, we provide the derivation of this result in the appendix. In light of the above result, FP-Backward (algorithm 3) provides the steps for backward elimination- based filter pruning. Note that line 7 in algorithm is the most expensive step in the while loop (lines - 17), which can be estimated from G the previous time-step using block matrix inversion with the worst-case complexity of O(n?). Also, for most calls the algorithm, the parameter B is very low (typically < 0.05), leading to far fewer iterations of the while loop (lines 6 17), which can be

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm 4 Layer Selection: HBGTS
Input: C:: Number of layers, Fe 1, ... C: Filters, D: Training dataset, Number of filters pruned in one step, 3: Pruning ratio over the network 3: Initialize: v2,0 = P v2-1,0 Ve Vc= == C, <-0
4: while Overall pruning ratio < 3 do
5: =0 = 1, C
Initialize total error for each layer pruned vc = I,... where n |F
6: C. - FP-OMP(Mp, (n, &
n. F2)
7: for 1: D] do
for ei {I,.,,C) do
(1 F2 y
1.0 - (1) Go y c 1.0
Unpruned propagation of unpruned previous layers Pruned propagation of unpruned previous layers
10:
11:
for j = 1,
do
12:
(i) P.c * (1
b Unpruned propagation of previous j"h pruned layer
13:
end for end for
14:
15:
for j = 1. C do
16:
of e e vé. [1)
Error in final layer output when jah-layer is pruned
17:
end for end for
18:
19: Cmin =argmin=-1....o ((e))
20: Revised network params: p++1
G F = Fig Ve & cmin
21:
t 1+1
22: Run 1 epoch of finetuning for network paramcters.
23:
end while
24: Output: Pruned filters Fd, Vc =c=1,-. 1,..,
assumed be constant. Moreover, this cost goes down with iterations as the size the matrix reduces significantly with the iterations, reaching only small fraction for n. Hence, assuming a constant number of loop executions (lines 6 17), the overall complexity of Algorithm is O(n²), which is two orders of magnitude improvement over using algorithm !
3.4 Hierarchical Backward Greedy Tree Search (HBGTS)
A key idea behind the hierarchical backward greedy search (HBGS) algorithm is to select the layer which results in minimal relative error when pruned from. However, the prediction error of layer's output is not always indicative of the ultimate predictive performance of the overall network. On the other hand, computing the error of the final network output involves the re-computation of changes through all the downstream layers, starting with the pruned layer. A naive implementation of this can lead to significant compute overhead since it requires OCCN) forward inferences through the network for cach pruning step, where C is the number of layers and is the number of data points the training set.
Amgorithm presents hierarchical backward greedy tree search (HBGTS), an efficient implementation for determining the best layer to prune from at every stage. A key idea here is to calculate the error in final layer output ey, when layer j E (1...,,C C} is pruned, for each input training example. Hence, need to perform only one forward pass per pruning step for each example E [I,...,N] To implement this algorithm, we utilize a data structure vL,, which stores the output of th layer c= 1,.... C, when the (n - + 1)th layer is pruned 1,... C, in the tth pruning step. Here, y represents the output of the t layer when no filter has been pruned. There are 3 cases while calculating v²., from vi- (lines 9, 10, and 12 in algorithm 4): (1) calculation of next unprumed output v (umpruned propagation of unprumed previous layers), (2) calculation of output corresponding to the current pruned layer v², (pruned propagation of unpruned previous layers), and (3) unpruned propagation of the pruned outputs corresponding to all the previous layers. Here, we only need to store vh. for the current timestamp Hence the space complexity of the modified algorithm increases only by ((nd) where d is the output size for each layer. The overall time complexity of the layer
8

---

Published in Transactions on Machine Learning Research (08/2024)
selection algorithm for T pruning steps becomes O(TC2N). Hence, using the backward elimination strategy, the overall time complexity for the proposed algorithm HBGTS-B is O(TC2N + TCn²). Comparatively, the vanilla backward search version of the algorithm, HBGS-B using backward elimination for filter pruning takes O + TCn²) time. However, as shown in the next section, HBGTS-B marginally outperforms HBGS-B in terms of the accuracy of the pruned models for a given pruning ratio.
4 Experimental Results
In this section, we describe the experimental setup and the datasets used. We compare the performance of the proposed pruning methods against state-of-the-art methods. Furthermore, a comprehensive examination of the working of the proposed Backward Greedy Search methods conducted.
4.1 Experimental setting
For the image classification task, we utilize three datasets: CIFAR10, CIFAR100, and Tiny-lmagenet. CIFARIO consists of 10 classes, with a training set of 50k images and a test set of 10k images, all with resolution of 32 x 32. Each class contains 5k training images and 1 test images. Similarly, CIFAR100 comprises 100 classes, with 500 training images and 100 test images per class. Tiny-Imagenet contains 200 classes and includes total of 0.1 M images. For our experimentation purpose, resize the original 64 x 64 images of Tiny-Imagenet to 224 224.
Dataset Description:
Training Details: Our experiments involve R8sNet18, ResNet32, ResNet56, VGG16, and ResNext models, with various percentages of parameter reduction. We prune a pre-trained model and adopt the training settings from LRF (Joo et. al. 2021). In contrast to LRF, where the model fine-tuned for a single epoch after cach filter removal, we fine-tune the model after pruning the entire , fraction of filters from each layer. After the completion of pruning for the entire model, we fine-tune the pruned model for 300 epochs. For fine-tuning, we set the initial learning rate to le- with decay rate of 14- We use step scheduler that reduces the learning rate by factor of 10 at epoch 150. Baselines were implemented using code provided by the authors and the recommended hyperparameters were used. We also performed hyperparameter search for the number of epochs, pruning ratios, and learning rates and reproduced the best results.
Performance Metric: We report the test accuracy for various pruming methods. The dense model's test accuracy corresponds to the pre-trained model's accuracy. Additionally, we report an accuracy drop (Acc }) from the dense model. We also report. the drop in parameters (param 4) and FLOPs (FLOPs 4) metrics to assess the level of pruning and model efficiency. The reduction in parameters refers to the decrease in the number of parameters/ srramcterss/cighht, weights across all retained filters. FLOPs, on the other hand, refer to the number of operations (convolutions), within the retained filters.
4.2 Performance Comparison: Accuracy and Efficiency
We compare our proposed pruming methods with the state-of-the-art methods in Table 1 We observe that our proposed methods (HBGS, HBGTS, FP-Backward, HBGS-B, and HBGTS-B) exhibit higher pruned accuracy compared to state-of-the-an methods for a comparable drop in the number of parameters. We also observe that our proposed methods consistently report a higher drop in FL.OPs compared to state-of-the-art methods.
ResNet and VGG on CIFAR-100 and Tiny- Imagene::: Figure and Table | provide further insights into the consistently superior performance of our proposed methods. We observe that the test accuracy of the proposed methods is consistently and significantly better than the test accuracy of the baseline methods. The fact that this difference is more pronounced in difficult dataset (CIFAR100 and Tiny-Imagenet) further demonstrates the superiority of the proposed methods. From Figure 2 we notice that, as the percenlage of parameter reduction increases, the difference in test accuracy between our proposed methods and state-of of the- art methods also grows. At higher paramcter reduction (290%), the proposed methods outperform existing methods by ~ 8% (see Figure 2 Maintaining or even improving accuracy at such high parameter reduction (>95%) is very valuable, further highlighting the effectiveness of the proposed methods.
9

---

Published in Transactions on Machine Learning Research (08/2024)
Table 1: Performance comparison between different pruning methods on VGGI6/CIFAR100 98%
parameter reduction, ResNet18/CIFARIO at 95% parameter roduction, Rcs and CRsNee33/CIIFAAI at 63% parameler reduction, averaged over three runs. + represents standard deviation, indicate drop, and bold/unde /addeeeeli denotes the brst/ssecond-best result.
VGG16/ VGG16/CIFAR10 98%
ResNet18/CIEAl @ 95%
Method
Acc
Param
FLOPs
Ace
Param
FLOPs
(%)
(%)
(%)
(%)
(%)
(%)
(%)
(%)
Dense
93.2 0.01
0:0
91.5 0.02
0:0
Random
0.18
6.9
98.0
89.0
86.3 0.06
8.2
85.0
EarlyCmP-S Raclwan
2022
90.0 0.47
3.2 0.47
95.0
91.0
0.52
3.5
95.1
85.8
DERFC [He cb
2724
90.1 0.07
3.1
97.3
76.9
SAP Diaso w.
0723
92. 0.03
3.0
91.9
81.9
Clien
90.2
3.0++0.02 0.02
92.0
LitE
(J00 mi
A121
91.3 0.27
2.21
97.8
93.0
91.5 0.37
3.0 0.33
95.1
85.8
FP Bacaward
91.3 0.08
1.08
97.8
93.0
92.8 0.15
1.15
95.1
85.8
02.7:0222
0.21
94.5
93.9 0.24
0.6
96.3
86.2
HBOS-B
92.6 0.19
0.61
98.1
94.3
93.7 0.22
0.5 1.2.2
95.2
86.0
HBGTS
93.5 = 0.25
98.6
94.8
94.7 0.28
-0.2 0.28
95.6
86.7
HBCTS-B
0.22
-.2
94.6
91.6
-0.1 0.24
86.5
VGG16/CIFAR100 & 98%
ResNet18/ COINNeII//IFAROOO @ 95%
Method
Test Ace
Ace
Param
FLOPs
Test Ace
Ace
Param
PLOPss
(%)
(%)
(%)
(%)
(%)
(%)
(%)
Dense
67.1 0.01
010
68.8
0.0
Random
55.5 0.16
0.16
98.0
86.0
519 0.13
13.9 0.19
93.1
812
EarlyCroP-S Rachwan
2022
62.8 0.52
0.52
97.9
64.1 0.45
4.7+0045
86.5
DLREC [He
2022
63.5
3.56 0.08
53.7
PL Chen et al
2023
0.03
3.0
97.3
87.9
LRF Boo et sl.
A721
64.0 0.31
0.31
97.9
65.5 0.29
3.2
87.3
FP-Backaar
66.2+0011
0.9
88.0
67.9 0.14
0.94-0.14
94.6
87.3
HBCS
67.3 0.17
-.17
98.3
89.6
69.1 09.1.10.119
-.19
95.2
88.5
HBGS-B
67.2 0.15
0.15
98.1
89.4
68.9 0.16
-0.1
88.3
HBQTS
67.8 0.23
40.7
0.23
98.5
89.8
69.7 +
0.26
88.6
HBCTS-B
67.6 1 0.21
0.5 0.221
98.4
80.7
69.5 1 0.23
0.7 0.23
95.3
88.5
ReaNet56/CIIAAl /CIAARIO 63%
MONNet32/CIFARR 63%
Test Are
Are
Param
FLOPs (%)
Test Ace (%)
Ace
Param
FLOPs
Method
(%)
(%)
(%)
(%)
(%)
(%)
Dense
93.45 0.02
0.1.
92.49 0.01
0 0.1:0
SFP
91.91.0.12
41.50
FPGM
Ho
2019
93.14 = 0.21
0.31
0.21
63.21
52.00
91.79 0.91
0.70
0.91
63.14
53:20
HRank
Lan
2012
02.56 0.05
0.89 0.05
63.04
62.43
LFPC
0.06
CHIP
Sui et
2021
92.88 0.18
0.37 0.18
63.12
62.08
A ASymihhhane
Halabi 05
2022
93.27 0.11
0.18+ 0.11
63.28
62.34
Joo
2021
62.56
FP- Bacavard
93.88 0.06
-0.13 10.13=00m 0.06
63.35
62.56
92.85
0,01
-0.36 0.01
63.31
62.35
HBGS
94.15 0.09
-0.70 0.09
63.87
64.91
93.06 06.06-1000 0.07
-0.57 0.07
63.65
64.88
HBGS-B
0.08
64.80
93.03
64.78
HBGTS
94.38 0.13
-0.93 1 0.13
63.93
64..95
93.28
0.12
-0.79
0.12
63.78
64.92
HBGTS-B
94.35 0.11
00.00
0.11
63.89
64.93
93. 26
4: 0.09
-0.77 0.09
63.75
64.90
CeeNet56/CIFARR
98% Param
ResNet32 CeeNet32/CTFARR100 CCTFRR100 98%
Method
Test Arc
Are
FLOPs (%)
Test Ace (%)
Ace
Param
FLOPs
(%)
(%)
(%)
(%)
(%)
(%)
Dense
08.18 0.01
0:0
68.48 0.02
0:0
Random
62.64 0.18
0..54
97.90
87.32
58.61
0.16
9.87
0.16
97.45
86.90
LFPC He et al
2020
62,83 0.14
6.35
0.14
97.25
88.18
61.78
177.8.10 0.13
6.70 0.133
97.13
87.67
DAIS
GUAH et di.
2122
61.23 = 0.16
0.16
97.49
58.36
60.31 1
8.14 0.15
97.25
88.12
GCNP
Jann
62.71 0.15
6.47
6.4750.15
97.37
$8.42
DERFC
Lie ct
62.13 0.19
7.05 0.119
97.51
58.73
LRF JJoo et at.
63.73 0.25
0.25
97.84
58.95
63.17 0.21
5.31 0.21
97.43
88.56
FP-Bacavard
1.52
89.24
06.92
1.56 0.07
69.13
HBGS
68.00 + 0.15
0.19 0.19+00.15
98.34
89.41
0.14 68.25
0.23 0.14
98.28
80.35
HBCS-B
68.82
0.36 0.13
98.27
89.32
68.13
0.33
98.23
89.31
HBGTS
-0.311
89.74
-0.17
89.71
HBCTS-B
60.39 - 0.22
0.21 0.22
98.57
80.68
68.59 + 0.16
-0.11 0.116
98.38
80.63
10

---

Published in Transactions on Machine Learning Research (08/2024)
EarlycroP-S (Early)
FP-OMP
HBGS
HBGTS
LRF
FP-Backward
HBGS-B
HBGTS-B
70
69
69
45
68
68
Test Accuracy
40
Test Accurad,
67
67
35
66
Test
66
30
25
64
95
95
96
97
90
Percent Parameter Reduction
Percent of Parameter Reduction
Percent of Parameter Reduction
(a)
(c)
Figure 2: Test accuracy for (a) CesNNe56/CIFFARIA (b) VGGI6/CIFARI00 and (c) ResNet188TTinn
Imagenet with increasing parameter reduction.
Our backward greedy search methods be used for effectivcly pruming large models that exceed the capacily of commodity GPUs. We use ResNext101 32x16d as our large model, consisting of 193 M parameters and requires 7.62 GB of GPU memory for loading. Additionally, we use ResNext 101 32x8d as our smaller dense model, which has 88 M parameters and requires 3.91 GB for GPU memory. Table 2 shows that when ResNext101 32x16d pruned to 98% parameter reduction using HBGTS-B achieves test accuracy that matches its dense counterpart. Hence, we can efficiently deploy the pruned model on edge devices with GPU memory less than 2GB. Furthermore, the pruned model takes 5.04 times less GPU memory than the larger dense model. Notably, the pruned model even outperforms the smaller dense model, ResNext101 32x8d.
To prunc Large Model:
Table 2: Comparison of pruning methods for ResNext101 32x16d (RN16) and similar sized dense 1esNex1101 32x8d (RN8) on CIFAR10 at 98% parameter reduction.
Test Acc
Acc +
Param +
FLOPs 1
VRAM (GB)
Method
(%)
(%)
(%)
(%)
Dense RN16
92.1
0
7.62
Dense RN8
91.8
0
3.91
FP-Backward
92.9
-0.8
98.5
89.9
1.59
HBGS-B
93.0
-0.9
98.7
92.1
1.55
HBGTS-B
93.2
-1.1
98.8
94.3
1.51
Time Comparison: Figure 3 a) provides comparison of uniform pruning methods in terms of pruning times. We can that our proposed method FP-Backward is faster than the best baseline FP-OMP by a factor of 2 for constant pruning ratio in each layer. This is also fair comparison since the baseline methods also prune a constant fraction of filters from each layer.
Figure 3 b) compares the pruning times various non-uniform pruning methods. Our proposed methods (HBGS, HBGS-B, HBGTS, and HBGTS-B) demonstrate superior computational efficiency compared the bascline EarlyCroP-S, a non-uniform pruning method. Notably, HBGS and HBGTS exhibit higher pruning times relative to HBGS-B and HBGTS-B. Specifically, HBGS-B achieves 54.40% and 56.16% reduction in pruning time compared to HBGS for ResNet32 and ResNet56, respectively. Likewise, HBGTS-B shows a 55.03% and 57.58% reduction in pruning time compared to HBGTS for ResNet32 and ResNet56, underscoring the effectiveness of the backward elimination strategy. Further, as expected HBGTS computationally more expensive compared to HBGS with approximately double the time. The most efficient hierarchical pruning method (HBGS-B) takes hours for ResNet32 (sce Figure 3()) (for a 5, number of filters removed each round) compared to hour taken by FP-OMP. The increase time can be further reduced by pruning a higher number of filters (a) in each round.
11

---

Published in Transactions on Machine Learning Research (08/2024)
200
HBGS
Pruning Time (hr)
2.5
FP-OMP
Pruning Time (hr)
1B65-B
(P-dadwardd
2.0
150
HBGTS
EarlyCroP-S
1.5
100
1.0
50
0.5
0.0
ResNet32
ResNet56
ResNet32
ResNet56
(a) Uniform Pruning
(b) Non-Uniform Pruning
Figure 3: Time comparison on
at 63% parameter reduction.
4.3 Analysis of Backward Greedy Search Algorithms
We analyze the working of the proposed greedy search methods in terms of their pruning quality. Figure illustrates heat map showcasing the relative reconstruction error and the percentage of removed filters for each layer across the pruning rounds, using HBGTS-B method for ResNet32 on the CIFAR-100 dataset 63% parameter reduction. The relative reconstruction error is calculated as l1 where, vć,0 is the output from the final classification layer when pruning was done in any of the layers of the network and ve,, is the output from the final classification layer when pruning was done at layer c. Both the relative reconstruction
and the pruning percentage depicted after every 7t round, each pruning 5 filters. Examining Figure 4 observe that the pruning percentage incrcases with each round, but not uniformly. For example, layers 14 18 have higher pruning compared to layers 1-3. Relative reconstruction error also decreases with pruning rounds but is not uniform across layers. From the heat maps, it is evident that our method selects the layer with the least relative reconstruction error for pruning. For example, layers 14 have moderate relative reconstruction errors the initial pruning rounds, so the pruning percentage is also not so high for the same. As the pruning rounds increase, the relative reconstruction error decreases for layers 14 18 and hence more pruning done from those layers as visible in the latter rounds in Figure ! This is in contrast to uniform pruning approaches, where pruning is uniformly applied across each layer.
To understand the intuition behind the filter choices for pruning using HBGTS-B, present visualization diagram of feature maps for two layers: Layer (pruned by 31.25%) and Layer 10 (pruned by 93.75%). By examining feature maps in Figure 5 (top row), we can observe that, Layer exhibits a diverse range of
Reconsturction Error while pruning
Pruning percentage while pruning
1.0
80
0.8
60
Layers
Layers
40
0.4
20
Rounds
Rounds
Figure 4: Heat map for relative reconstruction error and pruning percentage while pruning ResNet32 on CIFAR100 at 63% parameter reduction.
12

---

Published in Transactions on Machine Learning Research (08/2024)
31.25% pruned
93.75% pruned
Figure 5: Visualisation of output feature map of ResNet32 2nd layer (top row) and 10th layer (bottom row) on CIFAR100
filter outputs, indicating their effectiveness in capturing various input features. Consequently, our proposed method prunes only 31.25% of the filters in Layer 2 (as shown in the last column of pruming percentages in Figure 4) Similarly, Figure 5 (bottom row) displays feature map outputs from Layer 10, which appear very similar, indicating redundancy in filter outputs. This observation aligns with the pruning percentages shown in the last column Figure | where Layer 10 has 93.75% its filters removed. Thus, we can conclude that pruning percentages yielded by HBGTS-B are indicative of the amount of information carried by each filter each layer. Filters with more diverse outputs are retained, while those with redundant outputs are prunod.
5 Conclusion
this paper, we propose HBGS and HBGTS as pruning methods that optimize deep neural networks by reducing the number of parameters while maintaining or even improving their accuracy. The proposed pruning methods are based sparse approximation for non-uniform pruning, enabling the removal of redundant filters regardless of the layer they present in. We demonstrate method's consistent performance across various architectures, kernel sizes, and block lypes through extensive experiments on various datasets. We also propose HBGS-B, and HBGTS-B as efficient filter pruning methods, which offer significant advantages in terms of time efficiency compared to HBGS, and HBGTS. The proposed method can in principle be applied to any linear approximation-Ba based pruning technique, one way of applying to transformer-based models is pruning the filters in the ford-forward network (FFN).
References
Scbastian Ament and Carla Gomes. On the optimality of backward regression: Sparse recovery and subset selection. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5599-5603. IEEE, 2021.
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architectures using reinforcement learning. In International Conference on Learning Represenlations, 2017.
T Tony Cai and Lie Wang. Orthogonal matching pursuit for sparse signal recovery with noise. IEEE Transactions Information theory, 57(7))4680 4688, 2011.
Tianyi Chen, Bo Ji, Tianyu Ding, Biyi Fang, Guanyi Wang, Zhihui Zhu, Luming Liang, Yixin Shi, Sheng Yi, and Xiao Tu. Only train once: A one-shot neural network training and pruning framework. Advances in Neural Information Processing Systems, 34:19637-19651, 2021.
Yixuan Chen, Yubin Shi, Mingzhi Dong, Xiaochen Yang, Dongsheng Li, Yujiang Wang, Robert P Dick, Qin Tw, Yingying Zhao, Fan Yang, et al. Over-parameterizee model optimization with polyak-(VI) ojasiewicz condition. In The Eleventh International Conference on Learning Representations, 2023.
Christophe Couvreur and Yoram Bresler. On the optimality of the backward greedy algorithm for the subset selection problem. SIAM Journal on Matrir Analysis and Applications, 21(3):9978808 2000.
Enmao Diao, Ganghua Wang, Jiawei Zhang, Yuhong Yang, Jie Ding, and Vahid Tarokh. Pruming deep neural networks from sparsity perspective. In The Eleventh International Conference on Learning Representations, 2023.
13

---

Published in Transactions on Machine Learning Research (08/2024)
Xuanyi Dong and Yi Yang. Network pruning via transformable architecture search. Advances in Newral Information Processing Systems, 32, 2019.
Marwa EI Halabi, Suraj Srinivas, and Simon Lacoste-Julien. Data-efficient structured pruning via submodular optimization. Advances in Newral Information Processing Systems, 35:36613 16-66633-3662,, 2022.
P Thomas Fletcher, Suresh Venkatasubrammntian, and Sarang Joshi. Robust statistics on riemannian manifolds via the geometric median. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp.
1-8. IEFE, 2008.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arxiv:180 03635, 2018.
Abhinav Goel, Caleb Tung, Hsingn Lu, and George K Thiruvathukal. A survey of methods for low-power deep learning and computer vision. In 2020 IEEE 6th World Forum on Internet of Things (WF-loT), pp. 1-6. IEEE, 2020.
Yushuo Guan, Ning Liu, Pengyu Zhao, Zhengping Che, Kaigui Bian, Yanzhi Wang, and Jian Tang. Dais: Automatic channel pruming via disferentiable annealing indicator search. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient. neural network. Advances in neural information processing systems, 28, 2015.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In International Conference on Learning Representalions, 2016.
Yang He and Lingao Xiao. Structured pruning for deep convolutional neural networks: A survey. arXiv preprint arxi::233..00 00566,, 2023.
Yang Hc, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. arxie preprint Xiv: 100:18000000 1808.06866, 2018.
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4340-44-0 4349, 2019.
Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. Learning filter pruming criteria for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2009-2018, 2020.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pp. 1389-1397, 2017.
Zhiqiang He, Yaguan Qian, Yuqi Wang, Bin Wang, Xiaohui Guan, Zhaoquan Gu, Xiang Ling, Shaoning Zeng, Haijiang Wang, and Wujie Zhou. Filter pruning via feature discrimination in deep neural networks. In European Conference Computer Vision, pp. 245-261. Springer, 2022.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arviv preprint arXiv:1503 1.X2:01003.0231, 2(7), 2015.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. The Journal of Machine Learning Research, 22(1):10882 11005, 2021.
Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European conference on computer vision. (ECCV), pp. 304 320, 2018.
14

---

Published in Transactions on Machine Learning Research (08/2024)
Di Jiang, Yuan Cao, and Qiang Yang. On the channel pruning using graph convolution network for convolutional neural network acceleration. In IJCAT, pp. 3107-3113, 2022.
Donggyu Joo, Fojindl Yi, Sunghyun Back, and Junmo Kim. Linearly replaceable filters for deep wor channel pruning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8021 8029, 2021.
Minsoo Kang and Bohyung Han. Operation-aware soft channel pruning using differentiable masks. In International Conference Machine Learning, pp. 5122 5131. PMLR, 2020.
Andrey Kuzmin, Markus Nagel, Saurabh Pitre, Sandeep Pendyam, Tijmen Blankevoort, and Max Welling. Taxonomy and evaluation of structured compression of convolutional neural networks. ar preprint arXiv:1912.09802 2019.
Vadim Lebedev and Victor Lempitsky. Speeding-up convolutional neural networks: A survey, Bulletin of the Polish Academy of Sciences. Technical Sciences, 66(6):799 811, 2018.
Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks. In International Conference on Learning Representations, 2019.
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 1529-1538, 2020.
Shaohui Lin, Rongrong Ji, Chenqian Yan, Baochang Zhang, Liujuan Cao, Qixiang Ye, Feiyue Huang, and David Docrmann. Towards optimal structured enn pruning via generative adversarial learning. In Proceedings of the IEEE/CK IEEE/CVF conference on compuler vision and pollern recognilion, pp. 2790 2799, 2019.
Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. International Conference on Machine Learning, pp. 7021 7032. PMLR, 2021.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference computer vision, pp. 2736-2744 2017.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 0058-5066, 2017.
Shervin Minace, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey. IEFE bransactions on pollern analysis and machine intelligence, 44(7):3523 3542-3342,, 2021.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, Turi Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on compuler vision and paldern recognition, pp. 11264-121 11272, 2019.
Chaitanya Murti, Tanay Narshana, and Chiranjib Bhattacharyya. Tysprune-pruning non-discriminative filters via total variation separability of intermediate representations without fine tuning. In The Eleventh International Conference on Learning Representations, 2023.
Ben Mussay, Dan Feldman, Samson Zhou, Vladimir Braverman, and Margarita Osadchy. Data-independdent structured pruning of neural networks via coresets. IEEE Transactions on Neural Networks and Learning Systems, 33(12):7829-7844 7844, 2021.
Manuel Nonnenmacher, Thomas Pfeil, Ingo Steinwart, and David Reeb. Sosp: Efficiently capturing global correlations by second-order structured pruning. In International Conference on Learning Representations, 2021.
15

---

Published in Transactions on Machine Learning Research (08/2024)
Hanyu Peng, Jiaxiang Wu, Shifeng Chen, and Junzhou Huang. Collaborative channel pruning for deep PMLR, 2019.
networks. In International Conference on Machine Learning, pp-
Kiran Purohit, Amurag Parvathgari, Soumi Das, and Sourangshu Bhattacharya. Accurate and efficient channel pruning via orthogonal matching pursuit. In Proceedings of the Second International Conference on AT-ML Systems, AIMLSystens '22, New York, NY, USA, 2023. Association for Computing Machinery. ISBN httts://dooo org/10 1e1111..115/3555 1145/3564121 35641399
9781450398473. doi: 10.1145 0055/356412133364111
John Rachwan, Daniel Zügner, Bertrand Charpentier, Simon Geisler, Morgane Ayle, and Stephan Günnemann. Winning the lottery ahead of time: Efficient early network pruning. In International Conference on Machine Learning, pp. 18293 18309. PMLR, 2022.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognétion, pp.
779 788, 2016.
Stanley Reeves. An efficient implementation of the backward greedy algorithm for sparse signal reconstruction. IEEE Signal Processing Letters, 6(10):266-28) 2266-668,,11 1999.
Yang Sui, Miao Yin, Yi Xie, Huy Phan, Saman Aliari Zonouz, and Bo Yuan. Chip: Channel independence- based pruning for compact neural networks. Advances in Neural Information Processing Systems, 34: 24604 24616, 2021.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp. 6105 6114. PMLR, 2019.
Joel A Tropp and Anna C Gilbert. Signal recovery from random measurements via orthogonal matching pursuit. IEEE Transactions on information theory, 53(12):4635-4666 2007.
Murad Tukan, Loay Mualem, and Alaa Maalouf. Pruning neural networks via coresets and convex geometry: Towards no assumptions. Advances in Newral Information Processing Systems, 55:35003-38019, 2022.
Sunil Vadera and Salem Ameen. Methods for pruming deep neural networks IEEE Access, 10:63280-63388 2022.
Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang, Eigendamage: Structured pruning in the kronecker-factored eigenbasis. In International conference on machine learning, pp. (666-6575. PMLRR 2019.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances in neural information processing systems, 29, 2016.
Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. Advances in neural information processing systems, 32, 2019.
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I Morariu, Xintong Han, Minglei Gao, Ching- Yung Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE conference on compuler vision and pattern recognition, pp. 9194 9203, 2018.
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhon Huang, and Jinhui Zhu. Discrininationnawware channel pruning for deep neural networks. Advances in nearal information processing systems, 31, 2018
Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. In International Conference Learning Representations, 2017.
16

---

Published in Transactions on Machine Learning Research (08/2024)
A Proofs of Results
A.1 Weight compensation for multiple channel pruning
Result 1: Given Zu. k. 9jk: F. and Aj,,J = l,...,n, n. E S estimated using the filter pruning process.
Letting fi. = 91.k + Evese Av * kr AI E S, k = I,..., n, ensures that Z Z = Eres= X * ty * where er is the error vector for the estimation of removed filter I' E 5 and S denotes the set of all removed filters.
Proof. Consider the imput and output of any K K convolution layer be X (X1...,, and = YYm,,.... Y.). Y goes imput to the 1 x convolution. Let the output of the 1 convolution layer be Z = (Zi,.., Z,,,, followed by f E Rmxxn and E Ruxx being the filter weights of K x K and 1 x convolution layer respectively. We can formulate the above setup as:
m Y,=Y
X; fig = X fas
(6)
Z Σ Yj * 9j.k =
Σ j=1
x * füj 9j.k
(7)
Now, let f. : E S be the selected filter weights and similarly, let for : I' E S' be the pruned filter weights.
Dividing Equation into the two sets of filter weights, we get:
Σx X S gl,k +
Σ PES
X Say k
(8)
LES
Following the above terminology, write it as:
f.N = Evasfa t 140' e S'
(9)
IES
Substituting Equation g in Equation
we rewrite Zμ as Z in terms of retained filter weights f.a:
Z/ = X + Σ x Σ
Xx*
Araf.a + cp) gr.k
(10)
LES
F'S
LES
The above can also re-structured as:
Z-= Σ[x fat (g1. + Ev.+ gr
X g
(11)
PES
PES
Once the pruning is performed, Equation reduces to
Σ X f.1
(12)
lES
and Equation 11 reduces to
I * f.1 * (gLk + >v. g) LES
(13)
PES'
Thus, the weight difference after pruning, for Z₂ and Z are Evesy X for gr.kll all and Eves X EN gv all respectively. Because ey < fu1s, the weight difference in using 74 is lesser than that of Zg- Also, the lower the difference in weights, the better the approximation.
17

---

Published in Transactions on Machine Learning Research (08/2024)
Hence, we use Equation 13 for the weight compensation step have lesser weight difference and define the following step:
í1. = 9t.k +
Σ
444 E [1, Ai e s g
(14)
FEST
For output channel pruning, Equation 14
is re-defined
kh. I1::
Σ
* Dj: Ves
(15)
JES
while input channel pruning re-defined as
-
Σ
Na+ Ves
(16)
jES
A.2 Backward Elimination Algorithm for Filter Pruning
Result 2: Given the definitions of A k. da, Σj BTJ A x)) )) B. = E, BT B, A) ATB., E(A, B) Ej=1..... 76 |
and
Tk the following relation holds:
1 E, 2 d B hence, E(A,.. B) =
Proof. Given a matrix A E mmx" m 2 n, with column rank n, and an observation matrix B E Rmxj, The best least-squares solution to AX = B with at most r nonzero components is defined as the solution that minimizes the least-squares criterion.
Err(A)= = -IIBB.s A
(17)
Unconstrained least-squares solution of AA = is Ao = A)-AA-AAT B...- Substituting in Equation 17 we oblain
Err(A) Σ |B.. - A(A BT AA"
(18)
Note that only the second term is a function of A; therefore, maximizing B _ A) B.g with respect to combinations of columns comprising A equivalent minimizing Equation 17 with respect to the combination of nonzero components in the solution.
Let A_ is A with the th column deleted. Erry can be written as
Errs- ΣIBL
AT
(19)
A simple update formula for the second term of Equation 18 can be obtained from [Reeves 1999
Σ B (A" A ,T,
(20)
1 bdl 1²)
A(A'A A
Tkk
ddE B for all & and eliminate the column
From this result, it clear that we only need to compare Σ, whose corresponding value is smallest. Note that 7tk is the ttth diagonal element of (AT A)-1,, and df B. is AF the hth element of the solution vector (AT'A) - B.g.- Hence we can say,
ming (Errk)
(21)
ld B.sl
k- ming
(22)
7/k
18


Reviews:
[{"review_id": "PdvxiGcxdz", "reviewer": "Reviewer_j3tq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nWell presented framework of pruning combined with backward search. The combination of the forward pruning and the backward search seems to be an efficient improvement over the original single filter pruning method. \n\nWeaknesses:\n\nThe relative lack of novelty. While the paper is well written and the results are observable the improvement relatively minor given that the most advanced method has a magnitude larger search time when compared to the previous methods. Adding to this the relatively low improvement i wonder if the justification for such a large computation is worth such a low improvement. This should be better articulated so that specific purpose for this type of optimization is clearly observable by the reader.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an pruning approach based on several optimizations and cost reduction operations. The paper is relatively well written despite using some popularizing terms. The paper basically uses an algorithms that prunes one filter at a time as a function of output accuracy change and generalizes it to the set of $S$. In order to more efficiently search a variable groups of pruned filters in each layer, the authors introduce a backward search that simply evaluates going backward the pruning of various filters per individual layer. The filters re removed using an error reconstruction method allowing to estimate best neurons to prune. In addition in order to determine the best layer to be pruned the authors introduce a similar backward method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "More details in the explanation and justification of the proposed method. In Tables 1 and 2 the GHBTS algorithm is the best performing by providing the most improved accuracy and pruning ratio but it is also the most time-expensive. Therefore it should be clearly explained if this model is considered as the final result and when it is worth using it. This is same in the case of the amount of pruned parameters that is in average quite low compared to previous methods.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "u0yxxTKY78", "reviewer": "Reviewer_Zznz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n- The experimental results are strong. For instance, the authors achieve 94.7% on CIFAR10 using ResNet-18 after pruning more than 95% of the parameters!\n- The authors include useful empirical results that support some of the rationales of their method. For example, the author show empirically (Figures 4 and 5) why layers should not have the same pruning ratio. First, layers do not have the same reconstruction error, as shown in Figure 4. Second, their filters are not of the same diversity (e.g. Layer 2 has many diverse filters but layer 10 has few). For this, HBGS prunes >90% of the filters in Layer 10 but only prunes about 1/3 of the filters in Layer 2. \n- The paper is well-written.\n\n*Weaknesses*\n- My biggest concern is that it seems that the results are not reported entirely. For example, the authors report VGG16 results for CIFAR100 but not for CIFAR10. Also, they report ResNet18 results for CIFAR10 but not CIFAR100 or Tiny ImageNet. In total, the authors have experimented with 4 models and 3 datasets so we would expect a total of 12 results (for each combination) but the authors only report 4! \n- The proposed algorithms are a bit complex and not easy to implement, and the authors do not indicate any plans to release their code.\n- All of the experiments are done on small datasets. The largest is Tiny ImageNet which has 100K examples only. Do we observe gains when going to ImageNet-1k?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces new methods for structured pruning of CNNs, called hierarchical backward greedy search (HBGS) and hierarchical backward greedy tree search (HBGTS). In each method, the authors prune entire filters using a technique similar to orthogonal matching pursuit (OMP), in which the minimize the reconstruction error of the layer's output on a given training dataset. They also add 1x1 filters to compensate for the loss. Because this is based on the reconstruction loss, the number of pruned filters can vary from one layer to another. To improve computational efficiency, teh authors also use a technique borrowed from (Reeves, 1999) using block matrix inversion to reduce the cost by two orders of magnitude from O(n^4) down to O(n^2), where n is the number of filters per layer. \n\nExperimentally, the authors show that HBGS and HBGTS both outperform other pruning techniques in the literature using VGG16 and ResNet architectures, evaluated on CIFAR10/100 and Tiny ImageNet. In particular, the gap seems to increase as the percentage reduction in parameters increases.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please explain why ResNet18 results are reported for CIFAR10 only? Similarly, why are VGG16 results missing for CIFAR10?\n- I think it would be useful to compare to random pruning, where you use the same pruning ratios you identify for each layer. I don't expect this to work well but it will help see the improvement that comes about by selecting the pruned filters. \n- In Page 5, you claim that the algorithm is O(n^3) because |S| is small compared to n, but |S| is always proportional to n so it is O(n^4). Isn't it?", "broader_impact_concerns": "The authors should mention that pruning architectures are known to exacerbate biases in models. See for instance: \n\nHooker, Sara, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. \"Characterising bias in compressed models.\" arXiv preprint arXiv:2010.03058 (2020).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OdrHHCoNlq", "reviewer": "Reviewer_Dnjc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written, and I did enjoy reading it.\n\nStrengths:\n * From a theoretical perspective, the paper does not bring a whole new theory, and most claims can be derived from the cited papers by the authors. However, using such tools does open new avenues in the field of model pruning. \n * The authors started by presenting their vanilla approaches, followed by descriptive analysis and modified approaches handling the computational time issues arising from their vanilla approaches.\n * The authors conducted experimental setting and ablation discussing and showcasing the efficacy of their approach.\n \n\nWeakness:\n * See the next section for writing-related comments.\n * There are other sub-fields of channel/filter pruning that the paper did not even consider -- specifically speaking, importance-based sampling methods that have shown efficiency and efficacy; see [1], [2], [3]. Such methods usually require less than 300 epochs of fine-tuning (used in the paper) and require less time than $5$ hours of pruning. When are the proposed methods preferable to be used upon any of [1,2,3]?\n * Another line of work relies on the use of SVD which can be considered as a \"distant cousin\" to the OMP-based method since SVD-based approaches aim to lower the error associated with the matrix structure (lower-dimensional-based metrics for example). One paper, popping on top of my head is the [4]. Such a paper aimed also to find a suboptimal pruning parameter for each layer. How do the proposed methods compare to this paper?\n\n-------------------------------------------\n    [1] Tukan, Murad, Loay Mualem, and Alaa Maalouf. \"Pruning neural networks via coresets and convex geometry: Towards no assumptions.\" Advances in Neural Information Processing Systems 35 (2022): 38003-38019.\n   \n    [2] Liebenwein, L., Baykal, C., Lang, H., Feldman, D., & Rus, D. (2019, September). Provable Filter Pruning for Efficient Neural Networks. In International Conference on Learning Representations.\n   \n    [3] Mussay, B., Feldman, D., Zhou, S., Braverman, V., & Osadchy, M. (2021). Data-independent structured pruning of neural networks via coresets. IEEE Transactions on Neural Networks and Learning Systems, 33(12), 7829-7841.\n\n    [4] Liebenwein, L., Maalouf, A., Feldman, D., & Rus, D. (2021). Compressing neural networks: Towards determining the optimal layer-wise decomposition. Advances in Neural Information Processing Systems, 34, 5328-5344.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes filter pruning methods of CNN models leveraging the use of orthogonal matching pursuit and greedy-based approaches.  What differs from most papers in this field is that this paper proposes a way to find a different sub-optimal pruning ratio for each layer.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please apply the following:\n\n  * Page 5, Algorithm 1, line 9 -- what is $S^\\prime$?\n  * Page 5, Algorithm 1 and Page 8, Algorithm 4 -- $\\mathcal{D}$ in Algorithm 2 is not used. While I understand the intention, do make sure to update the Algorithm to either use it explicitly or simply remove it. Same note for Algorithm 4.\n  * Page 9, remove space between $0.1$ and $M$.\n  * Page 9, what do mean by \"we warmup the model for 20 epochs.\"?", "broader_impact_concerns": "There is no need to add a broader impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While reviewers found the novelty of the material presented in this paper modest, they found the experiments solid and convincing. The topic is timely and the paper is clear. The authors also addressed the concerns raised by the reviewers in their revision, including additional experiments, adding of missing references, and providing additional discussions. All reviewers recommended acceptance.
Decision: Accept (poster)