Revisit Kernel Pruning with Lottery Regulated Grouped Convolutions
Authors: Shaochen Zhong
Guanqun Zhang
Ningjia Huang
Shuai Xu
Venue: ICLR 2022 Poster (2022)
Keywords: structured pruning
efficient computing
parallel computing
grouped convolution
lottery ticket
weights shifting

Abstract: Structured pruning methods which are capable of delivering a densely pruned network are among the most popular techniques in the realm of neural network pruning, where most methods prune the original network at a filter or layer level. Although such methods may provide immediate compression and acceleration benefits, we argue that the blanket removal of an entire filter or layer may result in undesired accuracy loss. In this paper, we revisit the idea of kernel pruning (to only prune one or several $k \times k$ kernels out of a 3D-filter), a heavily overlooked approach under the context of structured pruning. This is because kernel pruning will naturally introduce sparsity to filters within the same convolutional layer — thus, making the remaining network no longer dense. We address this problem by proposing a versatile grouped pruning framework where we first cluster filters from each convolutional layer into equal-sized groups, prune the grouped kernels we deem unimportant from each filter group, then permute the remaining filters to form a densely grouped convolutional architecture (which also enables the parallel computing capability) for fine-tuning. Specifically, we consult empirical findings from a series of literature regarding $\textit{Lottery Ticket Hypothesis}$ to determine the optimal clustering scheme per layer, and develop a simple yet cost-efficient greedy approximation algorithm to determine which group kernels to keep within each filter group. Extensive experiments also demonstrate our method often outperforms comparable SOTA methods with lesser data augmentation needed, smaller fine-tuning budget required, and sometimes even much simpler procedure executed (e.g., one-shot v. iterative). Please refer to our GitHub repository (https://github.com/choH/lottery_regulated_grouped_kernel_pruning) for code.

Published as a conference paper at ICLR 2022
REVISIT KERNEL PRUNING WITH LOTTERY REGU- LATED GROUPED CONVOLUTIONS
Shaochen (Henry) Zhong', Guanqun Zhang", Ningjia Huang'. and Shuai Xu'
Depaarmeent of Computer and Data Sciences, Case Western Reserve University {sxx2517, nxh2339, sxx21 4 4 edu *Center for Combinatories, Nankai University
edwaaaiiedd... "zhanggg1994 9maiia111111m
ABSTRACT
Structured pruning methods which are capable of delivering densely pruned net- work are among the most popular techniques in the realm of neural network prun- ing, where most methods prune the original network at a filter layer level. Al- though such methods may provide immediate compression and acceleration ben- efits, we argue that the blanket removal of an entire filter or layer may result undesired accuracy loss. In this paper, we revisit the idea of kernel pruning (to only prune one or several & x k kernels out of 3D-filter), heavily overlooked approach under the context of structured pruning. This is because kernel pruning will naturally introduce sparsity to filters within the same convolutional layer thus, making the remaining network no longer dense. We address this problem by proposing a versatile grouped pruning framework where we first cluster filters from each convolutional layer into equal-sized groups, prune the grouped kernels we deem unimportant from each filter group, then permute the remaining filters to form densely grouped convolutional architecture (which also enables the parallel computing capability) for fine-tuning. Specifically, we consult empirical findings from series of literature regarding Lottery Ticket Hypothesis determine the op- timal clustering scheme per layer, and develop simple yet cost-efficient greedy approximation algorithm to determine which group kernels to keep within each fil- ter group. Extensive experiments also demonstrate our method often outperforms comparable SOTA methods with lesser data augmentation needed, smaller fine- tuning budget required, and sometimes even much simpler procedure executed (c.g., one-shot v. iterative). Please refer to our GitHub repository for code.
INTRODUCTION
The applications of convolutional neural networks (CNNs) have demonstrated proven success in various computer vision tasks (Voulodimos et al.) 2018). However, with modern CNN architectures has become one of the major challenges
being increasingly deeper and wider, over-parameterizatie of deploying such models to devices with limited computational resources and memory capacity
(Frankle & Carbin 2019). Therefore, the study of network pruning the technique of removing redundant weights from the originally trained network without significantly sacrificing accuracy has been an important subject both for practical concerns (Mozer Smolensky 1989) and better understanding of the properties and mechanisms of neural networks AAroaa et al. 2018).
In the realm of CNN pruning, a spectrum of techniques have been studied where the two ends are populated by structured pruning and unstructured pruning methods (Mao et al. 2017). Methods from the former end often propose to remove redundant weights in groups while following some geometrical constraints
such as removing certain filter layer. The methods from the latter end, on the other hand, prune the network with a more fine- amn-ggrined view where they evaluate every weight individually. Yet, there many other methods lay in between of the two ends. Methods that are more "unstructured" are believed to be capable of yielding better accuracy retention with commensurate amount of parameters pruned, due to having higher degree of freedom how

---

Published as a conference paper at ICLR 2022
and where to introduce sparsity to the originally dense network. Empirical findings also support this claim (Liu et al. 2019; Mao et al. 2017).
Despite having advantages accuracy retention, the resultant networks from methods closer to the unstructured end will be more sparse and less regulated on where to introduce sparsity. It may not provide actual compression and acceleration without relying on dustom-indexing sparse convolution libraries, or even dedicated hardware devices; thus, limits the deployability of such methods (Mao) et al., 2017). Meanwhile, methods closer the structured end, by preserving a more regulated resultant network, are more likely to be sarrarryhadwwrr-frieeiee pruning method may deliver densely resultant network and therefore gain immediate compression and acceleration benefits. We denote this kind of pruning method as densely structured.
The most deployable structured
Naturally, many scholars want develop new methods within the realm of densely structured prun- ing but with better accuracy retention. The majority of densely structured pruning methods focus on pruning the original network at a filter or layer level. We argue that the blanket removal of an entire filter or layer may harm the representation power of the network and result in undesired ac- curacy loss - as removing filter would consequently remove all feature maps generated by such filter. Even worse, removing layer would eliminate more feature maps and even face the danger of layer-collapse, a phenomenon of having an untrainable pruned network due to premature pruning of an entire layer (Tanaka et al. 2020).
In this paper, we revisit the idea of kernel pruning (to only prune one or several A x k kernels from a 3D-filter, instead of an entire one) an alternative and less aggressive pruning approach with higher degree of freedom. We hypothesize that by not removing the entire filter, the representation power of the original network will be better preserved. Although the idea of kernel pruning is nothing too novel (as it is simply a special case of individual weights pruning with 100% of weights of a kernel pruned), it is mostly applied under the context of unstructured pruning or structured pruning methods which may not deliver dense pruned network (Mao et al. 2017:) Ma et al. 2020). This is because a direct implementation of kernel pruning with no constraint would introduce sparsity across the network and therefore make the pruned network no longer dense. We address this problem by proposing a versatile grouped pruning framework, where we:
1. Cluster similar filters from each convolutional layer into a (predefined) number of equal- sized filter groups.
2. For each filter group, identify a certain portion of grouped kernels to prune according to the required pruning ratio.
3. Permute the remaining filters form densely grouped convolutional architecture accord- ing to the number of groups used in step 1.
Like most other post-train grouped pruning methods, we face the challenge of determining which clustering schemes and which importance metrics to use in step 1 and 2 of the above procedure. Upon investigations and experiments, discovered that a classical clustering scoring system (c.g., the Silhouette score) might not capture the better clustering scheme in regard to accuracy retention. Yet many filter importance metrics require sophisticated procedures, which are not computationally friendly or easy to execute when applied at a kernel level. We address the first challenge by consult- ing model-generated information in this case, the empirical findings on Lottery Ticket Hypothesis to develop scoring system that identifies the
(LTH) and related literature on weights shifting optimal clustering scheme among options per each convolutional layer (Frankle & Carbin 2019] Renda et al., 2020). For the second challenge, we design simple and cost-efficient greedy algo- rithm with multiple restarts to generate multiple candidate kernel selection queues and identify the one queue where the preserved kernels are most "distinctive" from each other yet "similar" to the pruned kernels. The main contributions and advantages of our method are:
Bring back an overlooked approach: We brought attention to the heavily overlooked approach of kernel pruning under the context of densely structured pruning.
Simple but effective - a vanilla adaptation of our framework outperforms sophis- ticated variants of other comparable frameworks (e.g., filter pruning): Even by just applying well- undersstood classical mathematical tools, extensive experiments demonstrate our method outperforms comparable SOTA methods across different networks and datasets.
2

---

Published as a conference paper at ICLR 2022
Additionally, our method often needs less data augmentation, smaller fine-tuning budget, and it executes without requiring any custom retraining, special fine-tuning, or iterative prune-train cycles which is rare for the approaches relying LTH-related studies.
Better longevity: We developed a framework that is compatible with further- developee/disscoveeeed clustering schemes and inductive biases, or more advanced variations upon them. This overcomes one of the major drawbacks of many filter pruning methods: as most of them propose different filter importance metrics that are largely incompatible with each other either in terms of their procedures computational requirements.
Improved deployability: The resultant network of our method is structured as densely grouped convolution, which enables parallel computing capability and greatly increases the practical deployability of our methods: we can now share the required computation and memory footprint across multiple end-user devices, where most of them have very limited said resources individually (e.g., loT devices, mobile phones, and wearable technologies).
General impact on LTH and beyond network pruning: Please refer to Section
RELATED WORK
Many prior arts have explored the possibility of obtaining smaller model with comparable perfor- mance by removing redundant weights (Zhu & Guptal [2018] Han et al.] 2016), filters (Molchanov
let al.
2017
Yu et al. 2018 He al. 2019] Wang al. (2019a), layers (Wang et al., 2019b] Lin
et al.
2019)
image input (Howard et al. 2017 Han et al. (020). or from all three dimensions (Wang
al.
2021).
It is clear that filter pruning attracts the most attention among all structured pruning
approaches.
Our method inspired by grouped convolution, widely adopted convolutional architecture which could be implemented efficiently on common devices (landola el al., 2017). Although kernel prun- ing used together with filter clustering not popular trend, we have seen such a combination in work like Yu et al. (2017). However, the proposed method by Yu requires iterative analysis of many different intermediate feature maps per layer, involves complex knowledge distillation application during the fine-tuning stage, and lacks comparable experiment results to recent pruning literature. Where our method (and concurrent work like Zhang et al. (2022) provides a much cleaner adap- tation of the abovementioned combination that delivers beyond-SOTA performance with a straight- forward one-shot pruning and standard fine-tuning procedure. In addition, our method consults em- pirical findings on the lottery ticket hypothesis and its derived literature regarding weights shifting (Frankle & Carbin| 2019; Renda et al.) 20000; Zhou et al.) 2019),, and we propose a novel greedy kernel pruning algorithm that is again simple, efficient, yet effective more this in Section 3
PROPOSED METHOD
3.1 PRELIMINARIES
Assume convolutional neural network W has L convolutional layers, we denote the W to be the L-th convolutional layer of W (for E E {Z+ LJ)). Therefore, we shall have a 4-D tensor we= x where C represents the number of filters in W (also known as the number of output channels in some literature), Ch represents the number of kernels per filter (a.k.a. number of input channels), and H x Wt represents the size of each kernel.
The overall procedure of our method can mainly divided into four stages: 1) Clustering filters into equal-sized groups, where the best clustering scheme for each convolutional layer is determined using the tickets magnitude increase score derived from prior arts on lottery ticket hypothesis and weight-shifting; 2) Evaluating several candidate grouped kernel pruning strategies generated by greedy approximation algorithm with multiple restarts, where the strategy with preserved grouped kernels that are most distinctive from each other, yet most similar to the pruned grouped kernels gets selected; 3) Permuting the preserved filters to form a grouped convolutional architecture with n groups; 4) Fine-tuning the pruned and grouped network to recover accuracy lost from pruning.
3

---

group-by by-group fashion instead of evaluating Published as a conference paper at ICLR 2022
Cunpur Channes
Filter
Outpul Clunnis
Grouped Filics
Clarnds
Channel
Group
Group
Weights:
Kemel
Group
Group 2
Croup 3
Group 3
2
Grouped Kemdl
5
Inpuc Channcl
Input Channels
Espar Channela
Inpul Charnel
Filerr
Group1
Pruming Within Groups
Group
Filter
Layer
Grouping
Reconstruction
Group
Ghoup 2
Layer:
Group
Group 3
Iaput
Ouiput
Curput
Ingui
Ouiuu
Original
Convolutional Layer
Convolational Layer
(b)
(c)
(d)
Figure 1: Steps of our method
3.2 CLUSTERING FILTERS INTO GROUPS
The first step of method is to cluster filters from the same convolutional layer into n equal-sized groups. Known that for a layer W we have C filters, there shall be / n filters inside each equal-sized filter group. We denote F. to be the i-th filter in W (namely, F. = W (...a) for
i {z- [L, Caul]. Filter clustering is maturely adopted technique in network pruning since is a widely accepted assumption that when similar filters are clustered together, the representation power of some filters can be covered by the rest of the filters in the same group (which therefore enables the potential of pruning). Additionally, this technique drastically decomposes the scope of the problem, as we may now proceed to evaluate all of the filters from W at the same time.
Many prior arts have developed methods on filter clustering with linearity assumptions (Guo al,, 2020). via retraining with custom loss function (Wu et al., 2018), or through an iterative process Yu et al. 2017). We argue that since each filter F. is a tensor C x 11 W considerations regarding non-linearity and high-dimensional relationships should be added. Therefore, utilize the following three combinations of proven mathematical tools on dimensionality reduction and clustering in order to cluster filters from each layer into equal-sized groups.
3.2.1 AVAILABLE CLUSTERING SCHEMES
To reduce the dimension of filters in W and cluster them into equal-sized groups, we utilize the following three sets of methods: K-PCA + k-Means, Spectral Clustering + k-Means, and (-SNE DBSCAN (with slight modifications done to obtain the desired cluster size and number). Since all five methods here have been maturely studied, we will omit the introduction of these methods; please refer to section
A.5 for details. We denote each set of methods clustering scheme.
3.2.2 DECIDE OPTIMAL CLUSTERING SCHEME PER LAYER WITH TMI SCORE
With the three proposed and many other unimplemented clustering schemes, one natural question ask is how are we suppose decide the optimal clustering scheme? And at what scale should we make such decision? The latter question was relatively easier to answer as we already set on clustering filters into groups layer-by-layer fashion, so reasonable enough to find out the most suitable clustering scheme for each layer. But to address the former question, need to find way to quantify the "quality" of proposed clustering results. Mathematicians have provided us with classical tools such the Silhouette score to measure the quality of clusters. However, experiment results suggest such tools can hardly identify the better cluster method under the context of accuracy retention A.I.I.
We hypothesize that this is because tools like the Silhouette score focus on properties regarding clusters themselves such distances between clusters, cohesion within each cluster while
4

---

Published as a conference paper at ICLR 2022
accuracy retention relies filter importance, balance between filter groups under the context of the network. Therefore, we decide to introduce network-induced information to help us quantify the quality of different clustering results: where consult empirical findings from series of literature regarding Lottery Ticket Hypothesis and weights shifting.
Background on Lottery Ticket Hypothesis, Weights Shifting, and Weights Rewinding For a dense convolutional neural network W initialized with weights of Wo. the conventional post-train pruning procedure has three steps:
1. Train the proposed network for epochs. Where we denote the network has weights of W at the k-th epoch for 0 I & Σ t.
2. Prune the network with weights W, according to the defined pruning ratio and achieve subnetwork Wi, where W C W,.
3. Fine-tune the network with W, and f more epochs to recover accuracy.
The Lottery Ticket Hypothesis claims that instead of fine-tuning upon the subnetwork W there exists a subnetwork W (defined as W, but replaced with its counterpart weights from Wa) that, when trained in isolation for f more epochs, may match outperform the test accuracy of the fine- tuned W. achieved from step above. This W, is therefore known as the winning ticket FFannkle & Carbin 2019).
vast amount of research has been done to demonstrate the existence of winning tickets across different networks and datasets, making the lottery ticket hypothesis one of the most tested inductive biases among neural networks (Renda al., 2020). Scholars have additionally investigated the relationship between the winning ticket's s weights and final weights in regard to accuracy retention. Zhou et al. (2019) did one of the most thorough experiments on exploring such relationship by deploying nine different zero mask criteria related to initial weights, final weights, and weights shifting during training. They concluded that ||w,| wull (where wi, Wy respectively represent the same weight in W, and Wo). the magnitude increase of weight its initialization and after training, has demonstrated most significant positive correlation with accuracy retention.
Note that the Zhou al. (2019) experiments were conducted under the assumption that the winning ticket exists at Wo. This is because, for early or even concurrent arts on the lottery ticket hypothesis, scholars have their disagreements on whether to reinitialize the weights of W to their initial values (namely, k = 0 for & in W)) (Frankle & Carbin 2019), to near initialization (0 < k & t) (Renda al. 2020). or just to reset randomly (Liu et al.] 2019). However, with more comprehensive exper- iments conducted, the findings reveal there are multiple winning tickets that exist within a range of epochs starting from near initialization: such that for any & where 0 < kn S k < k2 < t, W can be a winning ticket (with Aı being close to 0). This technique is referred as weights rewinding and denote the range of k2, k2] as the tickets window (Renda et al. 2020
Ticket Magnitude Increase scoring system Based on the fact that winning ticket may occur at near initialization but not necessary exactly at initialization, we slightly modify the magnitude increase criteria from Zhou al. (2019) by replacing the initialization weights as the ticket weights. Thus, we define the ticket magnitude increase score (hereinafter "TMI score") as ||w,| |wall for 0 k << (where w.. we respectively represent the same weight in W, and WR). Since we aim to use such scoring system to govern the clustering quality of filter groups, we further expand this scoring system from individual weight to filter-level. For filter F E x x xw* we denote the weight at (h, w) index from the e-th kernel of F as W w]]. so the TMI score for filter Fÿ in regard to winning ticket the k-th epoch should be defined the following.
C/. H² W Σ
TMI k)
ΣΣX
W
We
(I)
Similarly, for filter group 9f of (FF, F.. ... its TMI score would be:
5

---

Published as a conference paper at ICLR 2022
TMI(g
Σ TMI(F k). D:Eg'
Qualifying cluster results with TMI scores With the TMI score for filter group defined, the next question address is what makes good clustering result in regard to TMI scores? We already observed that tools like the Silhouette score could not capture the better clustering result in terms of accuracy retention, and we hypothesized that this is due the lack of attention on filter impor- tance and balance between filter groups. Since Zhou et al. (2019) has established the relationship between better accuracy retention and weights with larger magnitude increase, we expect such re- lationship will expand to a filter level where filters with large TMI scores may help on accuracy retention and therefore be deemed "more important." " We confirm such assumption in A.4.11
In terms of the balance between multiple filter groups - with each filter's TMI score being an indicator of its importance - we would prefer clustering results where all filter groups have a similar TMI score. As if one filter group's TMI score is significantly larger than another filter group, the former group will presumably contain more "important filters" than the latter group. When we proceed to prune the same ratio of grouped kernels out of both groups, the network will lose more "important kernels" and therefore damage the accuracy retention. In other words, we want to minimize the intervals between the TMI scores of all filter groups. So for clustering result G", assume gf of. ..., 1. 9h are the n filter groups of G sorted in descending order according to their TMI scores. We denote the intervals of Ge with respect to winning ticket k as:
Interval (G =) k)- TMI(gS,)) TMI(G(-1,A)) TMI , k)),
(3)
where the clustering result G arg ming (Sum (Summ[nterva((G', k)]) is preferred. To further identify the clustering result with the most balanced filter groups in terms of TMI scores, we also want to minimize the variance of intervals between the TMI scores of all filter groups. By combining the two heuristics with balancing parameter a, the overall scoring system for clustering result Ge is defined as:
Score clustering
(G R[SSum[Interval(C ,)) 1)) + R(VarlInterva((G' ())), (4)
where the function R denotes the ranking of the current Go against other proposed clustering re- sults (for ranking in an ascending order). For example, clustering result G has the smallest Sum IItervall k)] among all proposed clustering results, then [S[um[[nterval( choose to use rank instead of the raw value as we are only interested in if one clustering result better than another on one particular criterion (interval mean or interval variance), but not by how much. Therefore, by projecting both criteria to rank indices, we avoid the imbalance between the raw values of the two criteria. The arg min of this Equation 4 yields the best G
k)]) = 1. We
Increase robustness with multiple ticket evaluations Now we defined how to identify the pre- ferred clustering scheme from all proposed G in layer W', the last challenge find the winning ticket k for TMI score calculations. Unfortunately, finding the winning ticket requires an extremely computationally intensive process called iterative magnitude pruming, in which multiple training and pruning cycles are involved (Frankle et al. 2020 To make matters worse, since we know that there are often multiple winning tickets available from the training process (Renda al. 2020), picking one ticket but not the other will potentially yield a different set of preferred clustering schemes for the network. This makes the both a hard-to- find task and sensitive parameter to tune.
We address both challenges by relying on the same finding from Renda et al. (2020), which demon- strates that for any k where 0 < k < k < K2 < t, W' can be a winning ticket. This tickets window [k1, kg] is shown to be robust for the similar models across different datasets and pruning methods. Thus, decide which model we will prune on, identify its tickets window by consulting periment results from Renda et al.) (2020), and truncate some epochs in such window to conduct multiple evaluations. The optimal clustering scheme for e will be the one that most often yield

---

Published as a conference paper at ICLR 2022
as the preferred clustering scheme per different k setting | By doing this, first avoid searching for the winning ticket k, which greatly reduces the computational power needed to implement our method. Secondly, it increases the robustness of our method, since the optimal clustering scheme per layer is no longer sensitive to the choice of k. Last, granted the wide range of [ki, Az) observed from most networks, practically proximated guess with a reasonable overlap with the ideal (k1, would often suffice. The kind of tolerance, combined with the proven generalizability of winning tickets across different tasks, mod- els, and datasets (Morcos et al. 2019),
don't even need to search for the exact (R1, k2) pair; ap-
with more efficient methods of finding the winning tickets
become available (Tanaka et al.
2020
You et al., 2020). Our proposed method can also be applied
to experiments outside of Renda et al.
2020).
3.3 EVALUATE CANDIDATE PRESERVED GROUPED KERNEL QUEUES
With filters of W' grouped into filter groups {gf g .... ). we will carry out the pruning proce- dure within each filter group. Let K." be the i-th grouped kernel in filter group g' (for 1 S siso C.)) such that KT" is the collection of the i-th kemels from every filter in gng The pruning of a filter group g can be viewed as the task of identifying set group kernels in g" that needs to be preserved. To start off such identification, we first hypothesize a set of grouped kernels that are most "distinctive" from each other may provide the best help on preserving the representation power of the origi- nal filter group. We define a distance matrix between grouped kernels from g' as D(g")) where D(g") Kg
Kg' J. This D(g")) shall be symmetric matrix with its diagonal full of Os.
With this matrix, we may now translate this problem as a maximum edge- connected subgruph problem: Given a filter group g represented
undirected complete graph G(V, E) with edge weights w(u, v) = Dgg [[,,]] between nodes u and v, find a subset V S V with |V*|= s where the total edge sum of G[V*] == (V*, En (Y")) is maximal (with (1 pruning rate). Ch.).
The brute force solution of this problem requires iterating through every possible subset of grouped kernels with size and calculate their edge sum. With (!)) possible subsets available and each subset having (2) edges, this procedure has time complexity of O(s² IV[*). This can be quite compute-extensive task given each graph G(V, E) deduced from g has a |V] equals to Cha which can be as large as 512 in models like R0sNet-101. To address this, we hereby propose a simple greedy algorithm with multiple restarts that may approximate the V* in question.
Our approximated solution was based on the hypothesis that, assume we already have set of nodes U C V' representing grouped kernels which will be preserved. To find the next node Unext E V* for Unext & U such Umext should have a maximal edge sum to all nodes in U* Namely, Umext arg max max (Euev- w(u, M))x)). With this, for a filter group g" we may simply set the first grouped
kernel Uinitial E U to be every grouped kernel Ky E find the Unet until V* and obtain C₁ pruning strategies. This trick is known as multiple restarts and is widely adopted in many algorithms such k-Means clustering.
We formalize the procedure of our approximation algorithm in pseudocode at A.2.1 The core of our algorithm resides at 1 line 10, where we use the computed result of previous row in M to reduce time complexity of our algorithm. With M E Rexc and filled Ch times, our approximation algorithm therefore has a time complexity of (ssVV(2) (as V =.h)). This provides a sensible improvement upon the O(s2 [V(*) brute force solution. To further and better identify the best grouped kernel pruning strategy among the Ch candidate strategies in PS, we define the score of strategy V on g the following:
Score grouped kernel pruning
(V",
Σ (
w(Su, 8w) -3
Σ Σ
w(p,si))) (5)
PVg"\V"
BI=VEV
For example, suppose kı + 29 = k2. clustering scheme A is chosen as the preferred clustering scheme by Equation 4 for 20 times on layer W yet clustering schemes B and C are chosen 5 times respectively. In this case, A will be the optimal clustering scheme for W
7

---

Published as a conference paper at ICLR 2022
The first term calculates the inner distance sum of V*, where larger value represents greater in- ner heterogeneity within the kept grouped kernels. The second term iterates through every pruned grouped kernel p. finds the number of kept grouped kernels that are closest to each p according to the distance matrix D(g"), then sums over the distance between each p to its corresponding 7y kept grouped kernels. A pruning strategy with a smaller second term has better homogeneity between kept and pruned grouped kerels.. Since strategy with greater inner heterogeneity and smaller outer homogeneity is desired, we further introduce a tunable parameter -8 balance two terms and de- fine the best the pruning strategy of g' to be Viss = arg max (Score(V* .g') for all V*s available
V-
in PS. We have also implemented a toy experiment to verify our grouped kernel pruning method may obtain the optimal or a close-to-optimal solution, please refer to AA.1.2 for details. Ablation studies conducted at A.4.5 also confirm the efficacy of our greedy algorithm.
3.4 LAYER RECONSTRUCTION TO GROUPED CONVOLUTION ARCHITECTURE
The layer reconstruction of our method happens at two places: 1) Given a convolutional layer W permute the output channels of such layer according to the filter clustering result. This makes filters within W form n grouped filters as demonstrated in Figure | (a) to (b). 2) After a certain ratio of grouped kernels were pruned from their corresponding filter groups, permute the input channels of the remaining grouped kernels to remove sparsity from each filter group (Figure to (d)). As the result of the above two channel permutations, we will have dense grouped filters left in the pruned W'. Since every grouped filter has its own set of imput-outpur pathways, the pruned W' is essentially a grouped convolution architecture. This architecture enables parallel computing capabilities, because every grouped filter can be deployed to a different device so that all group filters may compute simultaneously. Please refer to A.2.2 for the general procedure in pseudocode.
4 EXPERIMENTS AND RESULTS
4.1 EXPERIMENT SETTINGS
Network Architectures and Datasets We evaluate the efficacy of our method on popular net- works with various depth and architectures: ResNee-20033256/100 with the Basi clockk im- plementation and ResNet-50/101 with the Bottl leeck implementation (He et al.] 2016). For datasets, we choose CIFAR-10 (Krizhevsky) 2009). Tiny-ImageNet (Wu et al. 2017)) and ImageNet (ILSVRC-12) (Deng et al. 2009) as they vary in complexity and Wang et al, (2021) has carried out fairly large-scale comparative experiments of many different structured pruning algorithms running under the same setting- - which provides us with a rich background to compare against.
Compared Methods and Performance Evaluation We test our method against various existing pruning methods showed in Table 2 All mentioned pruning methods are evaluated against the following criteria: AAcc. 1 Params, and 4 FLOPs which are defined as the difference of top- 1 accuracy, parameters reduction, and FLOPs reduction between the baseline and pruned model (where x represents 2% of parameters/FLOPs were removed/reduced comparing to the original network). In addition, we investigate the pruning procedure and pruning setting of the compared methods in the three following aspects: 1) Does the method require an iterative pruning procedure? 2) Does the method require special fine-tuning setup that deviates from Section 3.2.2 Figure ((a) of Wang et al. (2020)? 3) What is the fine-tuning budget of the method peethod-in-questt in--uestion?? We believe these are crucial information for building a fair understanding of the three aforementioned numerical criteria. Note, method with iterative pruning procedure will automatically require special fine- tuning, as it will munetune/reetran after each pruning operation, which is intrinsically different from the procedure introduced in Section 3.2.2 Our training and pruning settings are fairly simple given our method follows one-shot procedure, please refer to A.3.1 for details.
4.2 RESULTS AND ANALYSIS
As demonstrated in Table 2 our method yields superior accuracy retention than various modern SOTA methods across six tested network-dataset combinations with commensurate amount of parameters pruned with the exception of ResNet-56 on Tiny-ImageNes where our performance is reasonably close (-0.12% behind) to the best offering of 3D by Wang et al. (2021). Specifically,
8

---

Published as a conference paper at ICLR 2022
method achieves such results with simple one-shot pruning procedure and standard fine- tuning setup. Which are usually considered two less udvantageous design as many recent SOTA methods utilize iterative prune- train cycles (Wang et al., 202T] Frankle & Carbin 2019). Yet some simple tricks like dynamic pruning rate, soft-pruning (keep the pruned components updatable until very end), and weight reinitialization before fine-tuning may often provide most algorithms another performance boost (Li et al., [2117] [e et al. [2018] Renda et al. 2220;) Liu et al. (211)). We opt not implement such tricks to provide cleaner delivery of our method. Last, although not universally observed, our method often requires smaller fine-tuning budget than other offerings. Here we include an abbreviated version of our experiment report Table please refer Table 2 for the full one. Please also refer 1o.. and A.3.3 for discussions on hyperparameters and speedup.
Table 1: "IT", "SF" respectively indicate the method-in -uuessiiinnn requires an iterative pruning or special fine-tuning procedure. "FB" is the fine-tuning budget of each method (in terms of # of epochs). "BA" represents the pre-pruned network's accuracy. A cell with "_" implies either such information is inapplicable or we failed to confidently identify such information. Methods noted with are replicated by Wang et al. (2021), the rest are drawn from their original papers. The pruning rates (PR) of our method are adjusted meet the ,PParams or LFLOPs of other methods.
Method
HB
SF
BA
A (9)
Params (%) PLOPS))
CIFAR-IO:
1.278
8.33ES
DBPS Wane et al
2019b
560
V3.69
J 0.42
40
52
FPGM
100
93.59
0.33
52
PSemaeen Waint ot a 2000
#2600)
93.23
J. 0.188
50
SFP He draie 201
100
93.59
1.33
53
560
TMFOKF rouns, PX
13.75%)
300
93.78
0.22
43.49
43.23
ResNet- 110 on CIFAR-IO: FLOPS 2,35E8
Parums: 1.736
DHP (L
FPGM (He dl 2010
100
93.68
0.05
52
GAL Lin crai A019
93.50
0.76
45
49
PSemiel
#:900
93.49
SFP [Hc crai 2013
100
93.68
0.30
41
TMI-GKF [008s, PK
43.75%)
X
300
X
94.26
0.64
43.52
43.31
ResNet-101 Tiny-ImageNett FLOPs:
1.01F100
Params: 4,29K7
DBP
Wang et al. 2019b
420
64.85
3.48
76
77
DHP
Lr 2020
420
64.83
0.01
50
75
GAL
420
31 Wang 0 al 2021
420
64.85
0.44
51
75
TME (0UPS, PR 87.5
50%)
300
65.51
1,38
43.53
43.25
50
ImageNet (ILSVRC-
FLOF::
Params:
2.5617
PSceatch
WWang et al 2000
P::11)
77:20
0.50
29
26.8
Tayler
MMlchaaov dlale 2019
25
76.18
J 0.70
30.0
28.1
1.00
per prune
36.7
FPGM
(e al 2m19
100
75.96
0.92
33.2
33.7
SFP
He alal 2013
100
76.15
1.54
41.8
TMI-OKP
73%)
33.74
4.3 ABLATION STUDY We anatomize the effectiveness of different components of our proposed method. Specifically, we first demonstrate if the idea of our proposed TMI system and greedy approach are sound fun- damental level; then we evaluate if they are better than some common alternative approaches with similar functionalities. With the procedure design evaluated, we also provide some real-word run- time experiments of our proposed pruning procedure. Last, we evaluate the relationship between the hyperparameters and the pruned network's accuracy retention. Please refer to A.4 for details.
5 CONCLUSION
We proposed densely structured pruning framework capable of yielding beyond SOTA perfor- mance with a straightforward one-shot procedure. We believe the power of densely structured kernel pruning may go well beyond our implementation, as a) there will certainly be more and hopefully better - inductive biases and unsupervised clustering schemes made available, and b) our adapta- tion of kernel pruning and grouped convolution rather vanilla, where more sophisticated variants are available to explore: our framework may thrive on these further-din discorrriscovrriss.
In addition, our work serves as a proof of the lottery tickets-induced heuristics can be used to guide structured pruning strategy. This is an often overlooked usage despite the popularity of LT-related research, possibly due to the high cost of winning tickets searching. In such case, we hope the multiple (potential) tickets evaluations trick we introduced in our method may help in levitating such concerns. This might have an impact beyond the field of network pruning, as scholars of other tasks might find the ticket-induced heuristies to be useful but deterred by the high cost of tickets searching. Where our trick provides a form of approximation by just inspecting the weights shifting log saved during training.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
As we advocate our proposed framework is able to shine a new light on kernel pruning under the con- text of densely structured pruning, we have prepared a GitHub repository with checkpoints placed on every stage of our method. We hope this will facilitate the reproduction of our work and invite our fellow scholars to research and optimize on different stages of our pruning framework. To reproduce the exact training and pruning settings of our experiments in Table 2 please refer to A.3
REFERENCES
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 254-263. PMLR, 10-15 Jul 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255, Lece, 2009.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR. OpenReview.net, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings PMLR, 2020.
of Machine Learning Research, pp.
Perry Gibson, José Cano, Jack Turner, Elliot J. Crowley, Michael F. P. P.O' O'Bole,, and Amos J. Storkey. Optimizing grouped convolutions on edge devices. In 31st IEEE International Conference on Application-speecffcc Systems, Architectures and Processors ASAP 2020, Manchester, United Kingdom, July 6-8, 2020, pp. 189-196. IEEE, 2020. doi: 10. 10.11///AAAP99222200000000
Qinghei Guo, X. Wu, J. Kittler, and Zhi quan Feng. Self-grouping convolutional neural networks. Neural networks: the official journal of the International Neural Network Society, 132:491-505, 2020.
Kai Han, Yunhe Wang, Qiulin Zhang, Wei Zhang, Chunjing XU, and Tong Zhang. Model rubik's cube: Twisting resolution, depth and width for tinynets. In H. Larochelle, M. Ranzato, Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 19353- 19364. Curran Associates, Inc., 2020.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruming, trained quantization and huffman coding, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recogmition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10. N0.1109/CVPR.2216.990
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In Jérôme Lang (ed.), Proceedings of the Twenty-Sevntth In- ternational Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden. pp. 2234-2240. ijcai.org, 2018. doi: 20124//3/jcai221188300..
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang- Filter pruning via geometric median for deep convolutional neural networks acceleration. In IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2019, Long Beuch, CA, USA, June 16-20, 2019, pp. 4340-4349. Computer Vision Foundation/ IEEE, 2019. doi: 10. 10.11099CVPR.220900000
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In IEEE International Conference on Computer Vision, 2017 Venice, Italy, October 22-29, 2017, pp. 1398-1406. IEEE Computer Society, 2017. doi: 10.1 1091
10

---

Published as a conference paper at ICLR 2022
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, abs/1704.04861, 2017.
Forrest N. landola, Matthew W. Moskewicz, K. Ashraf, Song Han, W. Dally, and K. Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and ilmb model size. ArXiv, abs/1602.07333 2017.
Elsie Maria Jordaan. Development of robust inferential sensors: Industrial application of support vector machines for regression. 2004.
Alex Krizhevsky. Learning multiple layers of features from tiny images. pp. 32-33, 2009.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for
efficient convnets. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Yawei Li, Shuhang Gu, Kai Zhang, Luc Van Gool, and Radu Timofte. DHP: differentiable meta pruning via hypernetorks. Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jachhel Frahm (eds.), Computer Vision ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII, volume 12353 of Lecture Notes Computer Science, pp. Springer, 2020.
Mingbao Lin, Rongrong Ji, Yan Wang. Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In 2020 IEEÉ/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp- 1526-1535. Computer Vision Foundation TEEE, 2020. doi: 10. 0099/CVPR22600.22000000
Shaohui Lin, Rongrong Ji, Chenqian Yan, Baochang Zhang, Liujuan Cao, Qixiang Ye, Feiyue Huang, and David Doermann. Towards optimal structured enn pruning via generative adversarial learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Zhuang Liu, Mingjic Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2019.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 5068-5076. IEEE Computer Society, 2017. doi: 10.1109/ICCV. 2017.541.
Xiaolong Ma. Fu-Ming Guo, Wei Niu, Xue Lin, Jian Tang, Kaisheng Ma, Bin Ren, and Yanzhi Wang. PCONV: the missing but desirable sparsity in DNN weight pruning for real-time execution on mobile devices. The Thirty-Fourd Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 5117-5124. AAAI Press, 2020.
Huizi Mao, Song Han, Jeff Pool, Wenshuo Li, Xingyu Liu, Yu Wang, and William J. Dally. Ex- ploring the granularity of sparsity in convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning, In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 11264-11272. Computer Vision Foun- dation/ / IEFF, 2019. doi: 10.1 10119//CVP.20019.000 152.

---

Published as a conference paper at ICLR 2022
Ari S. Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. In Hanna M. Wallach, Hugo Larochelle. Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8- 8-14, 2019, Vancouver, BC, Canada, pp. 4933-4943, 2019.
Michael C Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. In D. Touretzky (ed.), Advances in Neural Information Processing Systems, volume 1. Morgan- Kauraann,, 1989.
Zheng Qin, Zhaoning Zhang, Dongsheng Li, Yiming Zhang, and Yuxing Peng- Diagonalwise refac- torization: An efficient training method for depthwise convolutions. In 2018 International Joint Conference on Neural Networks, IJCN 2018, Rio de Janeiro, Brazil, July 8-13, 2018, pp. 1-8.
IEEE, 2018. doi: 10.1 10.1109/1CNN.201 8.89911....
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020.
Erich Schubert and Arthur Zimek. ELKI: A large open-source library for data analysis ELKI release 0.7.5 "heidelberg". CoRR, abs/1902.03316, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs//109// 1556, 2014.
Zhuo Su, Linpu Fang, Wenxiong Kang, Dewen Hu, Matti Pietikäinen, and Li Liu. Dynamic group convolution for accelerating convolutional neural networks. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI, volume 12351 of Lecture Notes in Computer Science, pp. 138 155. Springer, 2020.
Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In H. Larochelle, M. Ranzato, R. Had- sell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, vol- ume 33, pp. 6377-6387 6389. Curran Associates, Inc., 2020.
Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and computing, 17(4):395-41 2007.
Athanasios Voulodimos, Nikolaos Doulamis, Anastasios Doulamis, and Eftychios Protopapadakis. Deep learning for computer vision: A review. Computational Intelligence and Neuroscience, 2018:1-13, 2018.
Wenxiao Wang, Cong Fu, Jishun Guo, Deng Cai, and Xiaofei He. COP: customized deep model compression via regularized corelation-basee filter-level pruning. Sarit Kraus (ed.), Proceed- ings of the TwentyEighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 3785-3791. ijeai.org, 2019a. doi: 10.24963/ijcci.20199 525.
Wenxiao Wang, Shuai Zhao, Minghao Chen, Jinming Hu, Deng Cai, and Haifeng Liu. DBP: discrimination based block-level pruning for deep model acceleration. CoRR, abs/1912, 191.1992..1178. 2019b.
Wenxiao Wang, Minghao Chen, Shuai Zhao, Long Chen, Jinming Hu, Hai Liu, Deng Cai, Xiaofei He, and Wei Liu. Accelerate cnns from three dimensions: A comprehensive pruning framework. In ICML 2021, July 2021.
Yulong Wang, Xiaolu Zhang, Lingxi Xie, Jun Zhou, Hang Su, Bo Zhang, and Xiaolin Hu. Pruning from scratch. In The Chirty-Fourthy AAAI Conference on Artificial Intelligence, AAAI 2020, The Seiroy- Innoaative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 12273-12280. AAAI Press, 2020.
12

---

Published as a conference paper at ICLR 2022
Jiay Wu, Qixiang Zhang, and Guoxi Xu. Tiny imagenet challenge. Technical Report, 2017.
Junru Wu, Yue Wang, Zhenyu Wu, Zhangyang Wang, Ashok Veeraraghavan, and Yingyan Lin. Deep k-means: Re-training and parameter sharing with harder cluster assignments for compressing deep convolutions. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th Inter- national Conference Machine Leurning, ICML 2018, Stockholmsmüssan. Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 5359-5368. PMLR, 2018.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang. Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Toward more efficient training of deep networks. In International Conference on Learning Representations, 2020.
Niange Yu, Shi Qiu, Xiaolin Hu, and Jianmin Li. Accelerating convolutional neural networks by group-wise 2d-filter pruning. In 2017 International Joint Conference Neural Networks (IJCNN), pp. 2502-2509, 2017. doi: 10.1 D..100//NNN.2011..999
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui- Hsin Lai, Vlad I. Morariu, Xintong Han, Mingfei Gao, Ching- Yung Lin, and Larry S. Davis. NISP: pruning networks using neuron importance score propagation. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT. USA. June 18-22, 2018, pp. 9194-9203, Computer Vision Foundation / IEEE Computer Society, 2018. doi: 10.1109/CVRR.2018.000
Guanqun Zhang, Shuai Xu, Jing Li, and Alan X. Guo. Group-based network pruning via nonlinear relationship between convolution filters. Applied Intelligence, January 2022. doi: 10. 10.11077/10489--21--00
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Process- ing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 3592-3602, 2019.
Michael Zhu and Suyog Gupta. To prune, not to prune: Exploring the efficacy of pruning for model compression. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Workshop Track Proceedings. OpenReview.net, 2018
A APPENDIX
A.1 SUPPLEMENTARY MATERIAL TO PROPOSED METHOD
A.1.1 Silhouette score AND ACCURACY RETENTION
We experiment ResNet-32 on CIFAR-I0 (BA 92.82%) with K-PCA Equal-sized k-Means and Spectral + Equal-sized k-Means. The mean Silhouette score of the latter combination is 0.3349/0.0265
12.49 times higher than the previous combination; indicating that if the Sil- houette score may capture the better clustering result with respect to accuracy retention, the latter approach should significantly better. However, the K-PCA approach has pruned accuracy of 92.68% where the Spectral approach has pruned accuracy of 92.41%, 1,, a +0.27% in favor of the method which the Silhouette score strongly suggests against. This demonstrates that tools like the Silhouette score might not be good judge of clustering quality in terms of accuracy retention.
A.1.2 EFFECTIVENESS OF OUR GREEDY APPROXIMATION ALGORITHM WITH TOY EXAMPLE
To test out whether our greedy approximation algorithm may yield solution that is close enough to the optimal one, we conducted a trial of TMI-GKP with 3 = 0 in Equation on ResNet-32. For the first ten convolutional layers (80 filter groups pruned), all approximated solutions within 0.5% in terms of the score difference (as defined in Equation 5) to the optimal solution - which implies the effectiveness of our approximation algorithm.
13

---

Published as a conference paper at ICLR 2022
A.2
ADDITIONAL METHOD PROCEDURE INFORMATION
A.2.1
GREEDY GROUPED KERNEL PRUNING PROCEDURE
Algorithm Generate C grouped kernel pruning strategics
Input:
Filter group g¹
D one of the n filter group in layer W number of grouped kernels to preserve, 8 > for storing grouped kernels pruning strategies number of grouped kernel in g and distance matrix of g
Input:
pruning rate) C
Initialize: Empty list PS
Initialize: C D(g²)
1: for each K; E g" do
2: Initialize: Empty list U
storing preserved grouped kernels per the current strategy the cost matrix
3: Initialize: Zero-filled matrix M E R
4: Ma < w w(K,, ;, K) ) for all K2 E K,
5:
U append(K;)
b make K; the first grouped kernel to keep
6:
for row t- to do
7:
Kisst + [-1]
8:
for each grouped kemel K E gf but K & U do
9:
Kindx gaaax--gggettieea g'.getindex(())
10:
K + (1oo--1,maal) + (aiss K)
U* aapppdd (arg max max(M(roo)) (Mrou))
if U then
13:
V* < U*, PS appeadd)) and break
14:
return PS
A.2.2 GENERAL PROCEDURE
Algorithm General Procedure of TMI-GKP
Input: W. (k1, k2)
trained model, tickets window clustering schemes, pruning ratio, number groups
Input: CS,
PR, CS,
1: for each convolutional layer W E W do
2: Initialize: Empty list G candidates
To store candidate clustering results
3: for K; [k1,k]] k2] do
4: Cluster filters from W into n equal-sized groups according to each scheme in CS
5: Identify the preferred clustering result G arg min(
Score (G', (:))([quatiion filters clustering
G
6:
G -andidaees-ppeen (G+,) Identify G +- the most occurred item in G 8: Apply Gb W
7:
candidates
9:
for each filter group of E G do Get grouped kernel pruning strategies V Algorithm (()
10:
A.2.1
11:
Identify Viest arg max(
Score
(V* g()) [Equation 5
V-
grouped kemnel pruning
12:
Apply Vies of W < pruned 8 reconstructed in grouped convolution architecture
13:
14:
return W
A.2.3 TIME COMPLEXITY ANALYSIS OF THE TMI CLUSTERING PROCEDURE
Our pruning procedure is basically two-stage: filter clustering and grouped kernel pruning. Since have already analyzed the time complexity of the second stage at the end of Section 3.3 we hereby focus on the time complexity of our filter clustering procedure.
Given a clustering result convolutional layer W we first calculate the TMI score of each filter group defined by Equation 2 The TMI score of clustering result in regards to epoch & has
14

---

Published as a conference paper at ICLR 2022
time complexity of (W")) for W E RRuxC
This is because TMI scores essentially achieved by finding out the magnitude increase of individual weights in W from epoch & to epoch I. (the final epoch). TMI scores for filters and filters groups simply of the TMI scores of certain individual weights - where the summing procedure is at most (W²) as we only got these many weights sum.
The TMI score for this clustering result is then determined by Equation 4 Which is (nn) proce- dure we have n groups (thus n intervals). Granted ((n) < O(W)), this term negligible and the overall time complexity is still O(W")
Note we may have multiple clustering results generated by multiple clustering schemes, where we use their TMI scores and Equation A to determine the best clustering result for layer W Thus, we need multiple the time complexity of a single clustering result to the number of clustering schemes available, we denote this number as C.Smmm The overall complexity is now C'S O (''). is hard to analyze the time complexity of each clustering scheme involves various dimension-reduction and clustering combinations. For the ease of expression, we uniformly denote all clustering schemes to have a time complexity of OCCSS((S (W)) on layer W (for CS() > W as a clustering scheme need to read all weights in W produce proper clustering result).
Last, as illustrated in Section 3.2.2 Increase robustness with multiple ticket evaluations, we need to repeat the whole procedure for knum times for Knum being the number of potential ticket epochs we evaluated. Thus, the final "rough" big-O complexity for our filter clustering procedure is:
CSaum O(CS(W")) C.Suum knum O(W)).
In our experiments, such krum is usually set to 35 (unless the network is too large) and CSaum is 3 as we have three different clustering schemes available (see Section 3.2.1). By the "absorption" law of big-O analysis, the theoretical time complexity is only (CC((W))) - which is identical to standard single-shot filter clustering procedure. This implies the theoretical lightweight-nes of our proposed method.
For more information, we included a discussion on how to adjust the pruming procedure to meet a time/computation budget at A.2.4 with real-world runtime experiments available at A.4.7
A.2.4 ADJUST THE PRUNING PROCEDURE TO MEET A TIME/COMPUTATION BUDGET
Although the analysis A.2.3 and experiments in A.4.7 demonstrate our TMI pruning procedure is reasonably fast, we admit that can be slow if given a wide network to prune or were asked to eval- algorithm will have to repetitively evaluate the network
uate many potential ticket epochs- with different TMI scores over and over again. We hereby provide several points for adjustability of method.
1. Reduce the range of ticket window: If a ticket window is defined to be [ki, A2). consider using [K kg] where 41 < ky and K2 < kg- So less potential ticket epochs were evaluated.
2. Add a ticket step: For a ticket window [Ki k2). consider adding ksssp so instead of evaluating k2, k1+1, k1+1+1, ... we now evaluate k1, k1-kum kit - Kuas .... By doing this, less potential ticket epochs were evaluated while a wide range of potential ticket epochs from different stage of the network training are still considered.
3. Relax the granularity of clustering evaluation: The proposed TMI-GKP determines the optimal clustering scheme a per layer manner. For CNN models with block-like structure (such as ResNet), one may opt to determine the clustering scheme for one layer of the block, then proceed to use such clustering scheme on the whole block.
4. Adjust clustering schemes: One may opt to reduce the number of clustering schemes available for the TMI score evaluation. Or one may opt to use clustering schemes which are less computational demanding. Granted the TMI system is likely to capture the "better" clustering scheme among the options, the method would still function, but likely not at its full potential.
15

---

Published as a conference paper at ICLR 2022
A.3 SUPPLEMENTARY MATERIAL TO EXPERIMENTS AND RESULTS
A.3.1 TRAINING AND PRUNING SETTINGS
For experiments done on CIFAR-10 and Tiny-lmageNet, we train the baseline models for 300 epochs with the learning rate starting at 0.1 and dividing by 10 per every 100 epochs. The baseline model is trained using SGD with weight-decay set to 5,-4, momentum set to 0.9, and a batch-size of 64, All data are augmented with random crop and randomly horizontal flip. For the experiments done on ImageNet, we train the ResNet-50 model for 90 epochs with the weight-decay set to 1e-4 and learning rate dividing by 10 per every 30 epochs (while keeping all other settings the same as CIFAR-10 and Tiny-ImageNet experiments). Our pruning settings are largely identical to our training settings except for the learning rate, which is 0.01 at the start.
A.3.2 CHOICE OF HYPERPARAMETERS
There are three main tunable hyperparameters in the mainframe of TMI-GKP, from Equation 4 and B, from Equation 5 Note in practice y is not fixed value but rather value that has fixed proportion to its the convolutional layer's input channels, i.e., ? = 'ratio C for layer W This is because different convolutional layer may have a different Cin- so it makes better sense to let of a layer adjust along with its Cin-
For experiments shown in Table 2, we fix a to 0.5 and Tratio to 0.2 for convenience, For experiments of ResNet-55 on CIFAR- 10 and 101 on Tiny-ImageNet, we set 3 to 2. For the experiment of ResNet-32 on CIFAR-10, we set / to 1. For the rest of the experiments in Table 2 we set (w) / (V* 7) with respect to Equation 5 This basically implics want the two terms in Equation 5 to evaluate an equal amount of grouped kernel pairs.
To further ensure/deeoonstrate the reproducibility of our work, in our codebase we will provide Google Colab notebook that replicates all CIFAR-10 pruning experiments conducted in Table For every completed experiment, codebase may register setting json, cluster, json (upon extraction of a pruned model), and an experimesn includes all hyperparameters' settings, cluster. json includes the clustering scheme per each convolutional layer and its resultant permutation matrix for converting the baseline model from (a) to (b) Figure [ and experimes
log. Where the setting, json
log contains the experiment printouts. addition, we have implemented a set of methods so a fellow researcher may pipeline the cluster: json file to the baseline model and try for different set of pruning strategies. This will save them the work of re-clustering the filters and re- --caalllaaingrr--ealuutii the TMI scores for every layer.
A.3.3 OMITTING SPEED UP ANALYSIS
We purposefully omitted the speedup analysis between the original and the pruned network mainly due lack of optimization of grouped convolution on current ML platforms. As an example, on PyTorch, grouped convolution with groups 8 is much slower than the standard convolution despite the former one has much less parameters and FLOPS.
Please direct to our GitHub repository where we discuss in length of why PyTorch is slowing us done, what's the reason behind it, empirically show that it is indeed PyTorch lacks of optimization that causing this problem. We also discuss why this will not be an issue for long (by presenting ML platforms' commitments optimizing the speed of grouped convolution), why this an achievable goal (as scholars have already accelerated grouped convolution Gibson et al. 2020 Qin et al. 2018)), and how this is not likely to affect the serious implementation of method for its intended purposes
said platforms
a group convolution network can be deployed as standard convolution on multiple edge devices in a parallel fashion (Su et al. 2020).
A.3.4 SPECIFIC EXPERIMENTS REQUESTED BY REVIEWERS
Upon the requests of two reviewers, we have additionally conducted experiments with our proposed method VGG-16 (Simonyan& & Zisserman 2014) and on ResNet-56 on CIFAR-10 against pruning method GAL by Lin et al. (2019) (but with a more aggressive pruning rate as Params and FLOPs 60% instead of zs 43%).
16

---

Published as a conference paper at ICLR 2022
Please refer to OpenReview entry (former experiment latter experiment, context of the latter experiment) for details - as these requests should only make sense under their Q&A context, and have not (or for some literature, we not able to) conduct the pruning procedure investigations like we did in Tableeaas defined in Section 4.1
A.4 ABLATION STUDIES
In this section we will anatomize our method in the following aspects. The general idea is we first show the basic format(s) of our approach works - as if the results achieved by following our approach would be better than the results achieved by going against our approach - then, we show that our approach works better than some common alternative approaches with similar functionalities.
First, we confirm the effectiveness of our TMI scoring system by conducting the following ablation studies:
1. TMIMII scores and filter pruning: where we investigate if filters with higher TMI score or larger magnitude increase will lead to better accuracy retention. This shows our TMI scoring system works at filter level.
2. TMI-driven clustering and accuracy retention: where we compare the TMI-driven clus- tering results with shuffled clustering results terms of accuracy retention. This shows our TMI scoring system works in terms of filter clustering.
3. TMI-driven clustering v. Other filter clustering schemes: where we investigate if the TMI-driven clustering schemes are better than some common alternative clustering schemes.
4. TMI-GKP v. Grouped convolution-only:
where we investigate if the performance of our method from the procedure we proposed, or simply due the adaptation of grouped convolution.
Then, we investigate our greedy grouped kernel pruning algorithm with the following ablation stud- ies:
1. Greedy-induced pruning and accuracy retention: where we compare the grouped kernel pruning results induced by our greedy approach to two approaches that take the "inverse" and "complement" of our greedy approach. This shows the basic principle of our greedy approach works.
2. Greedy-induced pruning v. Other grouped kernel pruning policies: where we compare the greedy approach against some alternative kernel pruning policies applied on grouped kernel pruning. This shows our greedy approach works better than some common alterna- tive kernel pruning policies.
Last, we provide some experiments to evaluate the real-world runtime of the pruning procedure of TMI-GKP and ablation studies on hyperparameters.
A.4.1 TMIMMI SCORES AND FILTER PRUNING
We first address the question of whether the TMI score (or the original "MI" magnitude increase score from Zhou et al. (2019)) has a relationship with accuracy retention when relaxed to filter level. Although the question is straightforward, is up to different interpretations as the TMI score was proposed as a tool to determine which clustering result is the optimal one which requires multiple tickets evaluation. Thus, hard to determine the TMI scores of a filter, as sensitive to the choice of k in Equation
To address such problem, we run Equation I on the same set of ks as our TMI-GKP algorithm, then we rank (sort) all filters according to their TMI scores under each k. Namely, when & = 35, we may have filter to be rank 1, filter to be rank 2... and for & = 36 we may have B to be rank 1 and filter C' to be rank 2. We then sum all ranks of each filter across different ks, where the filter with smallest sum (highest sum ranks) is considered the one that is most preferred by the TMI system. We denote this sum the TMI Filter Ranking Score.
17

---

Published as a conference paper at ICLR 2022
We opt rank- per-each-k, but not simply adding TMI scores of the same filter different is to- gether and rank all filters, because the former approach is a) less sensitive to potential extreme value introduced by certain & and b) similar to the proposed scoring mechanism defined in Equation 4 which is used in TMI-GKP.
We maro-mask different filters according the following five criteria. All experiments conducted on ResNet-32 (baseline accuracy: 92.82%) with pruning rate set to 50% (half the filters per each layer are zero masked).
1. TMI preferred: We kept filters that have the lowest TMI Filter Ranking Scores, and zero- masked the rest.
2. TMI complement:
We kept filters that have the highest TMI Filter Ranking Scores, and zero-masked the rest. Since the pruning rate is set to 50%, this is essentially taking the complement of the above scheme.
3. MI preferred: We kept filters that have the highest magnitude increase since initialization, and zero-masked the rest.
4. MI complement The complement of above scheme.
5. Random: Half of the filters per each layer were randomly zero-masked.
Scheme TMI preferred
TMI complement
MI preterred
MI complement
Random
Acc. (%)
64.48
61.93
63.25
62.06
63.92
The above experiments clearly demonstrate that both better TMI filter ranking scores and larger MI scores are correlated with better accuracy retention. In addition, the 1.23% lead of TMI preferred to MI preferred scheme also implies that our TMI system with multiple tickets evaluation is superior than the original magnitude increase system introduced in Zhou et al. (2019) at the filter level.
A.4.2 TMI-DRIVEN CLUSTERING AND ACCURACY RETENTION
Knowing that TMI preferred filters might lead to better accuracy retention. We are interested in learning if the clustering results produced by the TMI system may also lead to better accuracy retention. Thus, we implement the following criterion:
Take a clustering result determined by the TMI scoring system (argmin of Equation and shuffle its filters across different groups. By "shuffle," we mean that if set of filters were originally the same group, after the shuffle, no two of the above-mentioned filters will be in the same group anymore.
TMI shuffled:
Model
Baseline (%)
TMI-GKP (%)
TMI shuffled (%)
ResNet-20
92.35
1 0.34
0.67
ResNet-32
92.82
1 0.22
0.009
ResNet-56
93.78
1 0.22
J. 0.05
ResNet-101 94.26
1 0.64
0.25
The experiment results confirmed that our TMI system may provide positive contribution even under the filter clustering context. Although the improvement is not significant the one provide by the greedy approach (see A.446), the improvement is consistent. Yet without it, our method may not exceed SOTA performance at all.
We further include a discussion in A.4.6 on why empirical evidence suggests the TMI-driven clus- tering may not guarantee finding the best clustering in terms of accuracy retention, but rather robust policy with more comprehensive considerations done to deliver a "better" and very usable solution.
18

---

Published as a conference paper at ICLR 2022
A.4.3 TMI-DRIVEN CLUSTERING OTHER FILTER CLUSTERING SCHEMES
We the greedy grouped kernel pruning procedure fixed and feed our method with different schemes on filter clustering: three clustering schemes individually and a uniform random assignment of the three clustering schemes. The results with ResNet-32 on CIFAR-10 suggest the TMI-driven clustering may deliver the best accuracy retention.
Method
BA (%)
A Ace (%)
Params (%)
4 (LOPs(%)
ResNet-32 on CIFAR-I0:
FLOPs:
6.95E7
Params: 4.29E7
TMI Clustering w/ Greedy
92.82
t 0.22
43.43
43.09
K-PCA k-Means w/ Greedy
92.82
0.00
43,43
43.09
Spectral k-- 2-Means w/ Greedy
92.82
0.16
43.43
43.09
+-SNE DBSCAN w/ Greedy
92.82
0.14
43.43
43.09
Random Clustering Schemes w/ Greedy
92.82
0.00
43.43
43.09
However, by conducting experiments for the A.4.6 we notice that for the combination of ResNet- 110 on CIFAR-10. 10, forcing the clustering scheme as Spectral Clustering may achieve slightly supe- rior accuracy retention to TMI-GKP (+0.06%). But by running the same clustering scheme other experiment combinations like ResNet-20 and ResNet-32 on CIFAR-IO spectral clustering induces much lower results (-0.33% and -0.16% in comparison driven clustering may not guarantee on finding the best clustering in terms of accuracy retention, but rather robust policy with more comprehensive considerations done deliver "better" solu- tion than many other structured pruning methods when combined with our greedy pruning procedure.
TMI-GKP). This implies the TMI-
We observe similar phenomena on the K-PCA k-Means clustering scheme as it only shows slightly lower performance (-0.21% in comparison TMI-GKP) on ResNet-32 on CIFAR-IO. But the same clustering scheme performs consistently worse than TMI-GKP on ResNet-20/56/100 with CIFAR- 10: coming as respectively -0.38%, 20.29% and 0..44% our TMI-GKP method: which implies TMI-GKP is likely more robust on providing "better" solution across different networks.
A.4.4 TMI-GKP V. GROUPED CONVOLUTION-ONLY
Since the architecture of grouped convolution itself may induce a reduction parameters, we are also interested in learning how much it contributes accuracy retention. We modified the original network as a vanilla grouped convolution, trained it with the same training settings as in A.3. and compared it against TMI-GKP. With a comparable amount of parameters and FLOPs reduction, the results achieved with grouped convolution-only are well below our TMI-GKP.
Method
BA (%)
Pruned Acc (%)
Params (%)
FLOPs(%)
ResNet-2 on CIFAR-10: FLOPs: 4.09E7
Params: 2.70E5
Grouped Conv (n 2)
90.48
49.0
49.5
TMI-GKP
92.35
92.01
43.4
42.9
ResNet-32 on CIFAR-10:
FLOPs: 6.95E7
Params: 4.64E5
Grouped Conv (n 2)
91.75
49,2
49.6
TMI-GKP
92,82
93.04
43.4
43.1
ResNet-56 on
CIFAR-10: FLOPs:
1.27ES Params: 8.53E5
Grouped Conv (n
92.34
49.4
49.7
TMI-GKP
93.78
94.00
43.5
43.2
ResNet-110 on CIFAR-10: FLOPs: 2.55E8 Params: 1.73E6
Grouped Conv (n 2)
92.90
49.5
49.7
TMI-GKP 94.26
94.90
43.5
43.3
A.4.5 GREEDY-INDUCET PRUNING AND ACCURACY RETENTION
Our greedy grouped kernel pruning approach is developed on the assumption of (grouped) kernels that are most distinctive from each other are better. now put this assumption to test with the following two schemes:
19

---

Published as a conference paper at ICLR 2022
1. Greedy complement: where we keep the group kernels that was originally pruned in TMI- GKP.
2. Greedy reverse: where we flip the arg max on line 11 of Algorithm A.2.1 to arg min with 3 = 0 in Equation [5] This means the algorithm is now searching for the next grouped kernel that "most similar" the selected grouped kemells.
Here the experiment results with pruning rate to 43.75%
Model
Baseline (%)
TMI-GKP (%)
Greedy complement (%)
Greedy reverse (%)
ResNet-220
92.35
( 0.34
1.50
1.38
ResNet-32
92.82
f 0.22
4 1.38
0.56
ResNet-56 93.78
1 0.22
+ 0.88
0.54
ResNet-110
94.26
1 0.64
J. 0.76
0.67
The experiment results once again demonstrate the significance of our greedy approach. In addition, we observe Greedy reverse to have better accuracy retention to Greedy complement. We believe this is because the pruning strategy produced by Greedy reverse may have overlaps with the one produced by TMI-GKP; yet the pruning strategy produced Greedy complement is mutually exclu- sive with the one produced by TMI-GKP. This indirectly suggests the grouped kernels selected by our greedy approach are certainly "the better" ones, only a partial overlap with the TMI-GKP's pruning strategy may lead to noticeably better accuracy retention.
Upon request, we further investigated Greedy reverse with / 1 ResNet-20/32/56/11 CIFAR-10, 10, where such scheme is identical to Greedy reverse with the exception of setting 3 to 1. This means the algorithm is still searching for the next grouped kernel that is "most similar" to the selected grouped kernels. But among the C candidate pruning strategies, it will pick the one with best outer homogeneity (pruned and kept filters are most similar). The results are 1.52%, 0.42%, I 0.45%, and & 0.61 % respectively to the baseline of four aforementioned networks.
This is in line with anticipation. As per the design of Algorithm and Equation the set of grouped kernels preserved by greedy reverse with 3 = 1 should be very similar to the set group kernels preserved by greedy reverse with 8 0. This because both of them will preserve a of grouped kernels that are very similar to each other; and 56.25% of grouped kernels per layer are preserved, the two policies might very likely end up on the same (or similar) set of grouped kernels. Empirical evidence supports this hypothesis, as the performance differences between the two policies are marginal.
A.4.6 GREEDY-INDUCED PRUNING V. OTHER GROUPED KERNEL PRUNING POLICIES
We the filter clustering scheme to fix Spectral Clustering and feed method with differ- ent commonly applied grouped kernel pruning policies (T.2 and center pruning). Experiments of ResNet-32/1 10 on CIFAR-I0 have confirmed that our greedy-induced pruning scheme may yield better accuracy retention in a controlled setting.
Method
BA (%)
A Acc (%)
Params (%)
FLOPs(%)
ResNet-32 on CIFAR-10:
10:
FLOPs: 6.9SE7
Params: 4.29E7
Greedy-induced Pruning
92.82
t 0.06
43.4
43.1
L Pruning
92.82
I 0.30
43.4
43.1
Center Pruning
92.82
J. 0.19
43.4
43.1
ResNet-110 CIFAR-I0:
FLOPs: 2.55E8
Params: 1.73E6
Greedy-induced Pruning
94.26
t 0.70
43.3
43.5
L Pruning
93.76
1 0.088
43.3
43.5
Center Pruning
93.76
1 0.05
43.3
43.5
A.4.7 RUNTIME ANALYSIS OF TMI-GKP
The following experiments are conducted on a 2.00GHz 4 core Intel Xeon CPU and Tesla V100.
Evaluated on 35 potential ticket epochs.
20

---

Published as a conference paper at ICLR 2022
In our code implementation, first cluster layer, then prune it, then to the next layer. So the runtime is a mixed product of both the TMI filter clustering procedure and the greedy grouped kernel pruning procedure. Please refer to the table below for the runtime of the pruning procedure of our TMI-GKP method. All experiments are against the CIFAR-10 dataset.
Model
ResNet-20
ResNet-32
ResNet-56
ResNet- 101
2,977 sec
4,741 sec
10,376 sec (172.93 min)
19,818 88 sec (330.3 min)
Clustering and Pruning Runtime
(49.62 min)
(78.18 min)
We also separately analyze the runtime of our greedy grouped kernel pruning procedure. This is done by assigning a pre-determined permutation matrix the network (same effect clustering, as it permutes a convolutional layer from Figure aa) to Figure [b)), so the actual greedy-only approach will be even slightly faster). Also note the runtime of this greedy procedure theoretically related to the choice of y in Equation 5 but the greedy procedure itself is so fast to the point the value of does not matter anymore.
Model
ResNet-20
ResNet-32
ResNet-56
ResNet-100
250 sec
2,205 sec
Greedy Pruning Runtime
47 sec
16 sec
(4.17 min)
(36.75 min)
We consider this sort of runtime is totally tolerable as an overhead. When compared to methods involve iterative prune-train cycles (Wang al. (02T), custom loss function (Wu al. (2088), or feature maps analysis (Yu et al. (2017). our method is significantly more efficient and applicable to a broader set of pre-trained networks.
A.4.8 EFFECTS OF DIFFERENT IYPPEPPARAETERRS CHOICES
As mentioned in A.3.2 the two hyperparameters tuned / and 7 (derived from Yrati)) in Equation 5 We hereby provide set of experiments to show how different choices of such two parameters will affect the accuracy retention of the pruned network. All experiments were done on ResNet-32 on CIFAR-10 with pruning rate set to 43. 75% (baseline: 92.82%). Note the experiments below were done with setting one parameter fixed and adjusting the other, so the two tables should be inspect in a collective manner.
We first fixed Tratio 0.2 namely, for every pruned grouped kernel, 20% of all kept grouped ker- nels which are most similar to such pruned grouped kemel were evaluated and try with different
settings. The term "auto" implies ( (("))(((VVV with respect to Equation to A.3.2 on how we inferred this value.
please refer
Fixed Tuned param 3 92.71
3= 2
4
3=
3 auto
Tratio 0.2
93.04
92.58
92.67
92.8
92.76
92.74
The experiment results suggest a relatively more "balanced" relationship between the two terms in Equation [5] may lead to better accuracy retention. This implies both the inner heterogeneity of kept grouped kermels and outer homogeneity between kept and pruned grouped kernels should be taken into consideration
which is exactly how Equation was designed.
We then fixed 3 to "auto" and try with different Tratio settings:
Fixed Tuned param
Traioo 0.1
Tratio 0.2
Tratio
0.4
Taazo
0.6
Tratio 0.8
Tratio
92.97
92.74
92.7
92.65
92.74
92.53
The experiment results suggest "hato should be a relatively small value as increasing its value may lead to worse accuracy retention of the pruned network. This is rather intuitive result, as when Tratio = 1 every pruned grouped kernel is evaluated against all kept grouped kernels. In such case, the second term of Equation 5 longer reveal if there some kept grouped kernels that
21

---

Published as a conference paper at ICLR 2022
similar pruned one, because all kept kernels evaluated, yet all kept kernels are already distinctive from each others due to the selection procedure introduced in A.2.1
Additionally, we have conducted//disclosed more experiments with specifie settings required by one reviewer (c.g.. with / and Tiatio = 0. on ResNets; choice 3 being verses auto). Please refer to our OpenReview entry for more details these specifically requested experiments should only make sense under the Q&A context, and they do not fit well to the structure of our ablation studies.
A.5
DIMENSIONALITY REDUCTION AND CLUSTERING SCHEMES
A.5.1 KERNEL PRINCIPAL. COMPONENT ANALYSIS (K-PCA)
Kernel PCA is considered an improved version of PCA. The choice of kernel functions will signif- icantly affect the experimental results. Global kemells (e.g. polynomial, Sigmoid) and local kernels (e.g. RBF, Laplacian) are the most commonly used kernel functions for K-PCA. They capture dif- ferent characteristics of the data: the former demonstrates better extrapolation abilities, and the latter one has better interpolation abilities (Jordaan 2004). Since the filters are usually not linearly separable and often there are fewer filters than the number of features, we will mix the global and local kernel functions to benefit both extrapolation and interpolation properties. Specifically, we mixed a polynomial and an RBF kernel. We use a parameter control the balance of these two functions.
Know (zr, v) XKPoly + (1 A) 3)
(-
|| -
= >aax T c) + (1 A) exp
20
where À E (0, 1), parameters a and in polynomial kernel have been fixed to and O. d = 2 is the degree of polynomial kernel and a in RBF kernel the reciprocal of the width the radial basis function. In practice, such À is set fixed to 0.5.
A.5.2 SPECTRAL CLUSTERING
Spectral Clustering is yet another maturely studied and widely adopted clustering method. Unlike in K-PCA, for this method we simply utilize implementation of Von Luxburg (2007). Specifically in TMI-GKP, we utilize the variation of mutual-KNN as the similarity graph and cosine distance as the adjacency matrix based on empirical observations. We rely on Yikun Zhang's version of implementation.
A.5.3 EQUAL-SIZED MEANS
To ensure we may have n equal-sized groups out of each filter group, we utilize the default Same- size k-Means Variation offered by |Schubert & Zimek (2019). Specifically, we rely on Nathan Danielsen's ss version of implementation.
A.5.4 1-SNE AND DBSCAN
We explore the option of t-SNE and DBSCAN because we want to add a density-based clus- tering scheme from the two distance-based offerings. For implementation details; we the perplexityy of i-SNE to be the size of the filter group (namely, C /n in layer W²) and iter- ate through different combinations n_components, e, and min_samples until the DBSCAN algorithm may yield clustering result of > + 1 groups (for n.components E n)). After that, we execute the same data-point reassignment procedure as listed Schubert Zimek (0019) until n equal-sized groups achieved.
A.6 ACKNOWLEDGEMENT
We would like to acknowledge and give our warmest thanks to John Mays for his timely help on analytical writing.
22

---

Published as a conference paper at ICLR 2022
Table 2: Full comparisons of different pruning methods with ResNet-20032/56/110 CIFAR-10, ResNet-56/101 on Tiny-ImageNet, and ResNet-50 on ImageNet (ILSVRC-12). "IT", "SF" respec- tively indicate if the method-in-question requires an iterative pruning or special fine-tuning pro- cedure. "FB" is the fine-tuning budget of each method (in terms of # of epochs). "BA" represents the pre-pruned network's accuracy. cell with "_" implies either such information inapplicable or we failed to confidently identify such information. Methods noted with are replicated by Wang ctt al. (2021), rest are drawn from their original papers. The pruning rates (PR) of our method are adjusted to meet the [Params JFLOPs of other methods.
Method
IT
FB
SF
BA (%)
A Acc (%)
Params (%) FLOPs(%)
ResNett-20 on CIFAR-I0:
FLOIs:
4.09E7
Params: 2.70ES
DHP (Li et al.
2020
92.54
4 1.000
44
48
FPGM (He al. 2019
100
92.20
1.11
42
PSeratch Wang et 2020
600
91.75
1.20
50
Rethink (Liu et 2019
40
92.41
1.34
40
SFP (He et 2018
100
92.20
1.37
42
TMI-GKP (ours, PR
43.75%)
X
300
92.35
0.34
43.35
42.87
ResNet-32
CIFAR-10: FLOPs:
6.95E7
Params: 4.64ES
DBP
Wang
2019b
560
93.18
0.53
FPGM
He at. 2019
100
92.63
0.32
GAL
Lin cl al.
560
93.18
1.46
PSeratch (Wang et al.
2020
560
93.18
1.00
50
SFP (He et al. 2018
100
92.63
0.555
42
3D [Wang et al. 2021
560
93.18
0.09
38
49
TMFGKP (ours, PR
43.75%
300
X
92.82
0.22
43.43
43.09
ResNet-56 on CIFAR-10: 10: FLOPs:
1,27E8
Params: 8,53ES
CP (He et al. 2017
20
X
92.80
1.90
50
DBP Wang
2019b
560
93.69
0..42
40
52
DHP
L al 2020)
560
93.65
1..07
42
49
FPGM He al. 2019
100
93.59
10.33
52
GAL
et 2019
x
X
93.26
0.12
12
38
HRank LLn
2020
30 per layer
93.26
0.099
42
50
PSeratch Wang al.
2020
600
93.23
0.118
50
Rethink [[au ot 2019
40
93.80
0..73
50
SFP (He cf al.
2018
100
93.59
1.33
53
3D Waan 2021
560
93.69
0.07
40
50
TMEGKP (ours,
43.75%
X
300
93.78
0.22
43.49
43.23
ResNet-110 CIFAR-I0: FLOPs: 2.55ES
Params: 1.73E6
[He
2017
20
92.80
1.90
DHP
(L) al. 2020
94.69
0.06
FPGM (He 2019
100
93.68
0.05
GAL
93.50
0.766
PFEC Lr
93.53
0.61
PSeratch Wane
2020
500
93.49
Rethink [Liu 2019
40
93.77
0.15
SFP (He el al. 2018
100
93.68
0.30
TMI-GKP (ours,
43%75%
300
94.26
0.64
43.52
43.31
56 on Tiny- ImageNet: FLOPs: 5.06ES
Params: 8.65E5
DBP Wang et al.
2019b
420
V
$6.55
0.98
25
53
DHP
Li ci al. 2020
420
/
56.55
0.73
46
55
GAL
Lin et 20T9
x 420
X
56.55
0.68
32
52
3D Wang ul.
2021
420
56.55
0.51
34
59
TMEGKP (ours, PR
37.25%
X
300
X
55,59
0.63
36.74
37.05
ResNet-101 Tiny- ImageNet: FLOPs:
1.01E10
Params: 4,29E7
DBP
Wang
2019b
420
64.83
3.48
76
77
DHP
Li
2020
420
64.83
0.011
50
75
G
et
20T9
420
64,83
0.500
WWan et at. 2021
420
64.83
0.44
TMI-GKP (ours,
87.50%
300
65.51
1.38
43.53
43.25
ResNet-50 on ImageNet (ILSVRC-DD
FLOPs:
5.37E9
Params:
2.56E7
FPGM (He et al. 2019
100
75.96
0.19
22.1
22.5
PSeratch Wang el al.
2020
R=409
77.20
0.50
29.8
26.8
TMI-GKI (ours, PR 50%
X
90
X
76.15
0.19
22.1
22.5
Taylor MMlchanov et al. 2019
25
76.18
10.70
30.0
28.1
ThiNet [Lo et al. 2017
per prune
72.88
10.84
36.7
FPGM (He ct al. 2019
100
75.96
10992
33.2
33.7
SFP (He al. 2018
100
76.15
1.54
41.8
TMI- GKP tours, PR 75%)
X
90
X
76.15
0.62
33.21
33.74
23


Reviews:
[{"review_id": "iP0OD9czrH3", "reviewer": "Reviewer_hEhG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:\n\n1. The proposed approach is technically sound in general. Each part of the methodology (i.e., clustering, finding LTH, and pruning) is well motivated. The idea to leverage group convolution for kernel pruning is practical.\n\n2. The paper is well written and easy to follow.\n\nWeakness:\n\n1. The empirical evaluations are not self-contained, somewhat weak, and incomplete:\n  - Table 1 is not referred to. More experimental results and ablation studies should be included in the main text.\n  - The performance gain on CIFAR-10 can be minor compared with existing baselines, e.g., ResNet-32 has 92.82% acc, which is even worse than other baselines.\n  - The accuracies of models before pruning on TinyImageNet are apparently lower than other baselines, which is unfair. \n  - There are no results on ImageNet.\n\n2. More ablations are in need to fully verify the proposed approach.\n  - It is not verified if the findings in Zhou et.al.(2019) still hold in filter pruning. I wonder if the magnitudes of grouped filters still correlate with the final accuracies.\n  - The effect of hyper-parameters $\\alpha$ and $\\beta$ is seldom discussed in the experiment.\n\n3. The proposed pruning pipeline involves multiple steps, which is kind of complex to me. One needs to group the filters and find the optimal group division, identify the lottery tickets and formulate pruning as a maximum edge-weight connected subgraph problem. What is the time consumption for each step in such a pipeline, aside from network fine-tuning after pruning?\n\n\nOther comments:\n\n1. \"we expect such relationship will expand to a filter-level where filters with large TMI scores may help on accuracy retention and therefore be deemed “more important.”: It would be more convincing if such assumptions are empirically verified first before moving on to the next steps.\n\n2. \"we must first hypothesize a subset of grouped kernels that are most “distinctive” from each other, which helps to preserve the representation power of the original filter group.\" : Is this empirically verified? In other words, what if you switch the strategy to keep the least \"distinctive\" grouped kernels?\n\n3. Have you tried the practical speed-up of pruned networks via grouped convolutions?", "summary_of_the_paper": "This paper proposes a new approach for kernel pruning, by leveraging group-wise sparsity to obtain a compact architecture that can be parallelly computed. The approach first clusters the kernels based on the defined measurement via recent findings in the lottery ticket hypothesis (LTH), and then develop a simple and efficient greedy approximation algorithm for filter pruning.", "summary_of_the_review": "Overall this is a technically sound paper. However, my major concerns are two-fold: 1. The proposed methodology is over complicated in my view, yet there is little gain in accuracy. 2. The experiment still needs to be enriched to verify the proposed solution from multiple aspects.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MDPJ0GzGHVR", "reviewer": "Reviewer_rPne", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. Dense Structured pruning: this paper has addressed a well-known and crucial problem of kernel-pruning in a structured way yielding a dense pruned network. They have empirically shown how this approach can help improve the representation power of a pruned network (delta increase in performance) and can be easily deployed (less FLOPs and hardware-friendly due to grouped-convolution).\n\n2. Improved Accuracy without Iterative Fine-tuning: one important contribution of this paper is the exhaustive empirical study of their approach with related works (Table 2) with respect to i) need for iterative pruning or ii) special finetuning. Overall, their results imply that the proposed approach of grouped kernel pruning indeed improves the generalisation, but the FLOPs improvement is still either negligible or at par performance as compared to the baselines ( SFP [He et al., 2018] and 3D [Wang et al., 2021b].\n\n3. Enhanced Clustering Scheme: Another important contribution is the better evaluation of filter-clusters by taking filter importance into account. Before this, it was done using a classical clustering Silhouette metric which doesn't take any network context (filter importance) into account.\n\nCons:\n1. Filter Importance and Accuracy Retention? Authors claim that their method of filter-clustering takes accuracy retention into account by following two themes: i) filter importance and ii) \"balance between the filter groups\" (not all important filters should be aggregated in the same group). However, they have not proven/established any kind of direct relationship between accuracy retention and these aforementioned themes (especially for convolutional filters). For example, any experiment to prove the empirical relation between their defined metric TMI score (to evaluate cluster quality) and accuracy would be enough to prove the point.\n\n2. Loopholes in notations (Algorithm 1): The algorithm 1 on page 7 maintains the matrix M of size (C_in-1 X C_in at Line 3) where C_in represents the number of kernel channels. As per the understanding, it should be of size C_in X C_in. Also, following the proposed algorithm, once a pruning ratio is reached, the algorithm should break the loop instead of continuing (Line 14).\n\n3. Technical issue in algorithm 1: The greedy approach proposed by the algorithm always picks the next best kernel K given the current set of selected kernels (nodes) in a filter-group (graph). \nExample: Suppose a filter-group has 6 kernels A,B,C,D,E,F with the distance-matrix (D) as defined below: D(A,B) = 100, D(C,D) = 100\nD(A,C) = D(A,D) = D(B,C) = D(B,D) = -10\nD(E,F) = 0\nAll other kernels have a distance value of -2. In such a scenario, if we try to select 4 kernels of maximum value set (subgraph), the answer should be (A,B,C,D) while the algorithm's greedy approach will never take these 4 kernels (nodes) together. It would be better if the authors can clarify this corner case.\n\nTypos:\n1. \"Abstract\" at Page 1: \"structured pruning due to it\" looks like an incomplete sentence.\n2. \"Introduction\" at Page 1: \"methods lie in between\" looks grammatically incorrect.\n3. \"Related Work\" at Page 3: Line \"Our method is inspired by group convolution, a is well convolutional architecture\" has issue in sentence formation\n", "summary_of_the_paper": "This paper presents a novel approach in the well-established space of structured pruning methods (kernel-pruning) post-training, which primarily consists of three stages: (i) clusters the filters in a convolution layer into predefined number of groups, (ii) prune the unimportant kernels from each group, and (iii) permute remaining kernels to form a grouped convolution operation and then finetune. This paper is motivated by the idea of achieving grouped convolution which can be efficiently deployed on low-compute end devices, and definitely has huge practical implications.", "summary_of_the_review": "Overall, I vote for borderline rejection (score 5). Specifically, I liked the idea of introducing a new metric of ticket magnitude increase score (TMI score) to cluster the filters in the first stage based on the \"lottery ticket hypothesis\". But there are some unproven claims and inconsistencies as pointed out in the cons section.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "G2vWQKgY-hd", "reviewer": "Reviewer_RqjA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* The paper addresses an important research problem in the field of model compression as most kernel pruning methods suffer from the curse of sparsity and are therefore not computational friendly. \n* The paper is well written. The intuition behind each proposed component of the framework (e.g., the proposed TMI score) is well discussed and most of them seem straightforward.\n* The ablation study on different filter clustering schemes is insightful.\n\nWeaknesses:\n* The experiment setup is somewhat limited. The experiments are conducted on CIFAR10 and TinyImageNet only, so it is unclear whether it would work well on datasets like ImageNet. Moreover, only ResNet experiments are reported. It would be more convincing if the results on other popular CNN architectures are also reported (e.g., densely connected blocks and NAS-searched blocks like EfficientNet).\n* As finding winning tickets is sensitive to the learning rate, is there any empirical justification for the learning rate decision?\n* Although the proposed greedy approximation algorithm seems to be cost-efficient, it would be great to show the training time compared to baselines.\n\n", "summary_of_the_paper": "The paper consults the empirical findings on the Lottery Ticket Hypothesis and proposes a kernel pruning framework with densely structured outputs. The proposed method seems to give noticeable improvement over SOTA methods with a rather elegant design.", "summary_of_the_review": "In summary, I feel that although the proposed method seems to be novel, the evaluation part is not yet convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5Hj0pWsf_w_", "reviewer": "Reviewer_4Wb3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "pros.\n1. The proposed method seems simple yet effective, though it has not been fully evaluated on the large dataset.\n2. Figure 1 clearly illustrates the implementation of TMI-GKP. We may easily deploy it on general-purpose devices like CPU/GPU, which benefits from fine-grained and coarse network pruning schemes simultaneously.\n3. The technique part of this paper is easy to follow. The combination of the Lottery Ticket Hypothesis and structured pruning seems interesting. This paper may shed some light on this new direction.\n\ncons.\n1. The main concern is the experiment part of this manuscript. Since post-training grouped pruning has been frequently discussed in the recent literature (in static mode or dynamic mode) [1,2,3,4,5], a direct comparison with those methods is more appropriate. Besides, ImageNet has been a common setting in recent works. Though I fully understand the authors are stuck with limited computing resources, a solid result on ImageNet can be more convincing than CIFAR-10.\n2. I expect the real speedup reported in Table 1 to verify the superiority of learned group convolutions.\n3. It would be better to include the real time cost of the \"Ticket Magnitude Increase scoring system\" and \"Qualifying cluster results with TMI scores\". It seems that the computing complexity of those steps are $\\mathcal{O}(\\prod_{i=0}^{K-1} \\binom{C_{out}-i\\cdot\\frac{C_{out}}{K}}{\\frac{C_{out}}{K}})$ where $C_{out}$ is the number of filters, $K$ is the number of groups, which is still very large for $C_{out}=512$.\n4. I have checked section A.3.1. The proposed \"TMI Clustering w/Greedy\" with $+0.14$% on CIFAR-10 is somewhat incremental when compared with K-PCA+k-means.\n5. It would be better to verify the effectiveness of Algorithm 1 via a toy experiment, such as setting the channel number to $16$. The brute force solution to this problem can be obtained and serves as a baseline.\n6. Since the authors apply the Lottery Ticket Hypothesis to structured pruning, I expect an ablation study under the same setting as *winning ticket* or *weights rewinding* to prove the existence of the ticket. It is hard to evaluate the correlation between $TMI$ and the Lottery Ticket Hypothesis, given the existing experiments.  \n7. According to Table 1, GAL seems to outperform TMI with $12$% Params and $38$% FLOPs?\n8. There seems to be no ablation study on $\\beta$ (in Eq.(5)). Did I miss anything?\n\nref:\n* [1] Dynamic Group Convolution for Accelerating Convolutional Neural Networks. ECCV2020\n* [2] Fully Learnable Group Convolution for Acceleration of Deep Neural Networks. CVPR2019\n* [3] Building Efficient Deep Neural Networks with Unitary Group Convolutions. CVPR2019\n* [4] Differentiable Learning-to-Group Channels via Groupable Convolutional Neural Networks. ICCV2019\n* [5] Efficient Structured Pruning and Architecture Searching for Group Convolution. ICCVw2019\n", "summary_of_the_paper": "The authors present a new metric to determine the similarity between different grouped kernels and prune the unimportant $k\\times k$ slices out of a 3D filter. They utilize the Lottery Ticket Hypothesis and propose a greedy search strategy to overcome the challenge of a huge search space. The experiment results show that the one-shot scheme can still be comparable to two-stage leading methods on the CIFAR-10 dataset, with a slightly lower training cost. The empirical success of this paper may serve as proof of the existence of the Lottery Ticket Hypothesis.", "summary_of_the_review": "In general, I believe this work is somehow different from existing works but the current draft makes it challenging for these ideas to reach their full potential. The authors are encouraged to address the weaknesses above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "06pMCRzDCOs", "reviewer": "Reviewer_Dc6d", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The core idea and contribution is stated and explained clearly. This helps the reader assimilate the paper quickly.\n\nWeaknesses\n- There are multiple instances of language issues throughout the manuscript e.g., Pg. 3, Section 2, Paragraph 2, first sentence. I encourage the authors to correct them to help the reader process the work seamlessly.\n- Could the authors clarify the meaning of \"Permuting the retained filters\" ? Is it similar to replicating the connections of common input nodes across multiple groups? If so, doesn't it add more FLOPs to the forward pass?\n- The nomenclature of dense, sparse and densely structured have a lot of overloaded meaning. It is a little difficult to parse the exact meaning of these terms. I encourage the authors to rethink the terminology and possibly clarify them.\n- Pg. 3, last bullet point: The term \"cleaner\" and other abstract terms can be avoided to maintain focus on the argument at hand.\n- A common theme across the Related works and other sections  is the highlighting of computational cost involved in complex pruning heuristics. Could the authors discuss the time-complexity involved in a) the TMI score generation at $W_t$ for the winning ticket and b) TMI score generation across the entire DNN (l layers) and c) TMI scores for multiple groupings, which itself is computationally expensive?\n- If we factor in the performance of unsupervised clustering approaches from the Appendices, the Table maintains the same numbers for performance, params and FLOPs across all tested methods. Could the authors clarify these results?\n- Could the authors clarify the magnitude increase formula , from Pg. 5 and subsequently used everywhere?\n- When discussing the choice of windows for lottery tickets, the authors mention they lookup values from Renda et al.(2020). However, this leaves the choice of windows for new dataset/DNN architectures open. Could the authors discuss more on how they would approach such a target?\n- Mathematical notations for k  have been overloaded (Alg. 1 and previously). Also, the variable names have a mix of mathematical notation and pseudo-code. I encourage the authors to reassess their choice of variable names.\n- Could the authors discuss their choice of layer pruning ratios ?\n- Could the authors clarify the special fine-tuning setup discussed in Section 3.2.2 and the deviation of other baselines from it in more detail? This could add more strength to the argument of loss computational overhead.\n- If the pruning and retaining settings mirror the training setup, the terminology of fine-tuning might be incorrectly used. Could the authors clarify if their training and pruning setups are the same?\n- The results posted in Table 1 are lower than R. Yu, A. Li, C.-F. Chen, J.-H. Lai, V. I. Morariu, X. Han, M. Gao, C.-Y.Lin, and L. S. Davis, “Nisp: Pruning networks using neuron importance score propagation,” in IEEE Conference on Computer Vision and PatternRecognition, 2018, N. Gkalelis and V. Mezaris, “Fractional step discriminant pruning: A filter pruning framework for deep convolutional neural networks,” in IEEE International Conference on Multimedia & Expo Workshops, 2020 and M. R. Ganesh, J. J. Corso, and S. Y. Sekeh, “Mint: Deep network compression via mutual information-based neuron trimming,” in IEEEInternational Conference on Pattern Recognition, 2020. Could the authors update their results and compare against methods with high performance.\n- Overall, the amount of results and discussion presented in the main body of the paper seem insufficient. I encourage the authors to re-organize their manuscript to further support their claims and analyze their results.\n\nAfter Rebuttal\n- I appreciate the author's clarifications on a number of the comments posted.\n- After considering the revised version of the paper, I have updated my scores across multiple metrics.\n- The update of Table 1 to include a number of results helps emphasize the applicability of the proposed work to a number of datasets. The inclusion of \"Amc: Automl for model compression and acceleration on mobile devices\" , a top performing method on ResNet50-ILSVRC2012 would help strengthen the comparisons drawn from Table 1. \n- At a higher level, a pruning methodology's performance is based on pushing the limits of performance at extreme sparsity levels. \nIn this context, comparisons to the most sparse models mentioned in works, like GAL, DHP, etc. would help highlight the importance of the current work further. ", "summary_of_the_paper": "The proposed work focuses on kernel pruning. The core idea revolves around grouping filters using a similarity criterion and removing common convolutional kernels with the group. The proposed work explores optimal grouping schemes for filters and after pruning unwanted filters, the retained filters are restructured and the network can be fine-tuned to recoup prediction accuracy. ", "summary_of_the_review": "The method proposed in the work provides an alternative pruning approach that should help decrease the computational complexity. However, overheads involved in repeated formulation of groups and TMI evaluation need to be discussed and compared. The results presented in the manuscript do not possess sufficient depth and are missing analyses. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies structured pruning methods, called kernel-pruning in the paper which is also known as channel pruning for convolutional kernels.  A simple method is proposed that primarily consists of three stages: (i) clusters the filters in a convolution layer into predefined number of groups, (ii) prune the unimportant kernels from each group, and (iii) permute remaining kernels to form a grouped convolution operation and then fine-tune the network. Although the novelty of the method is not high, it is simple and effective in experiments after the supplementary sota results in the long rebuttal. Majority of reviewers increase their ratings after the rebuttal (though one reviewer promised this but forgot to act), while some reviewers have concerns on the fairness to other authors by adding lots of new results in unlimited rebuttal and refuse to check more. In terms of the top end of performance, a reviewer thinks that "the authors haven't quite exceeded the results from existing works ("Discrimination-aware channel pruning for deep neural network" and "Learning-compression” algorithms for neural net pruning" for CIFAR-10 and many others on ImageNet)". In all, this work indeed lies on the boundary. After a discussion with other committee members, we recommend the acceptation of this work, if the authors could incorporate all the new results in rebuttal and get the reproducible codes released in the final version.
Decision: Accept (Poster)