DFPC: Data flow driven pruning of coupled channels without data.
Authors: Tanay Narshana
Chaitanya Murti
Chiranjib Bhattacharyya
Venue: ICLR 2023 poster (2023)
Keywords: Pruning
Data Free
Model Compression

Abstract: Modern, multi-branched neural network architectures often possess complex interconnections between layers, which we call coupled channels (CCs). Structured pruning of CCs in these multi-branch networks is an under-researched problem, as most existing works are typically designed for pruning single-branch models like VGG-nets. While these methods yield accurate subnetworks, the improvements in inference times when applied to multi-branch networks are comparatively modest, as these methods do not prune CCs, which we observe contribute significantly to inference time. For instance, layers with CCs as input or output take more than 66% of the inference time in ResNet-50. Moreover, pruning in the data-free regime, where data is not used for pruning, is gaining traction owing to privacy concerns and computational costs associated with fine-tuning. Motivated by this, we study the problem of pruning CCs in the data-free regime. To facilitate the development of algorithms to prune CCs, we define Data Flow Couplings (DFCs) to enumerate the layers that constitute coupled connections and the associated transformation. Additionally, saliencies for pruning CCs cannot be gauged in isolation, as there may be discrepancies among the layerwise importance of CCs using conventional scoring strategies. This necessitates finding grouped saliencies to gauge the importance of all corresponding coupled elements in a network. We thus propose the Backwards Graph-based Saliency Computation (BGSC) algorithm, a data-free method that computes saliencies by estimating an upper bound to the reconstruction error of intermediate layers; we call this pruning strategy Data Flow driven Pruning of Coupled channels (DFPC). Finally, we show the efficacy of DFPC for models trained on standard datasets. Since we pruned coupled channels, we achieve up to 1.66x improvements in inference time for ResNet-101 trained on CIFAR-10 with a 5% accuracy drop without fine-tuning. With access to the ImageNet training set, we achieve significant improvements over the data-free method and see an improvement of at least 47.1% in speedup for a 2.3% accuracy drop for ResNet-50 against our baselines.

Published as a conference paper at ICLR 2023
DFPC: DATA FLOW DRIVEN PRUNING OF COUPLED CHANNELS WITHOUT DATA.
Tanay Narshana Chaitanya Murti,, and Chiranjib Bhattacharyyaa*
Observe.Al'
Systems, Indian Institute of Science? Department of Computer Science and Automation, Indian Institute of Science
Robert Bosch Centre for
tanay. narshana gma a.g.al.cco...m....... com, {mchaitanya, chiruf0lisc ac. in
ABSTRACT
Modern, multi-branched neural network architectures often possess complex in- terconnections between layers, which we call coupled channels (CCs). Structured pruning of CCs in these multi-branch networks is an under-researched problem, as most existing works are typically designed for pruning single-brande models like VGG-nets. While these methods yield accurate subnetworks, the improvements in inference times when applied to multi-branch networks are comparatively modest, as these methods do not prune CCs, which we observe contribute significantly to inference time. For instance, layers with CCs as input or output take more than 66% of the inference time in ResNet-50. Moreover, pruning in the data-free regime, where data is not used for pruning, is gaining traction owing to privacy concerns and computational costs associated with fine-tuning. Motivated by this, we study the problem of pruning CCs in the data-free regime. To facilitate the development of algorithms prune CCs, we define Data Flow Couplings (DFCs) to enumerate the layers that constitute coupled connections and the associated transformation. Additionally, saliencies for pruning CCs cannot be gauged in isola- tion, as there may be discrepancies among the layerwise importance of CCs using conventional scoring strategies. This necessitates finding grouped saliencies gauge the importance of all corresponding coupled elements in a network. We thus propose the Backwards Graph-based Saliency Computation (BGSC) algorithm, data-free method that computes saliencies by estimating an upper bound to the reconstruction error of intermediate layers; we call this pruning strategy Data Flow driven Pruning of Coupled channels (DFPC). Finally, show the efficacy of DFPC for models trained on standard datasets. Since we pruned coupled channels, we achieve up to 1. 1.66x improvements in inference time for ResNet-101 trained on CIFAR-10 with a 5% accuracy drop without fine-tuning. With access to the ImageNet training set, we achieve significant improvements over the data-free method and see an improvement of at least 47.1 1% in speedup for 2.3% accuracy drop for ResNet-50 against our baselines
INTRODUCTION
As computational resources have become significantly more powerful, deep learning models have become correspondingly larger and more complex as well, with some models possessing billions of parameters (Sevilla et al. 2022). Moreover, many modern architectures are huultt--banched networks due layer skip connections like Residual Connections (He al., (016)) that are used to avoid vanishing gradients. These large, complex architectures enable these models to learn patterns in data
with better performance terms of optimization and generalization (Arora al. 2019 Neyshabur
ct al. 2019 Zhang et al. 2021).
The benefits of overparametcrization in these models come at the cost of increased memory and compute footprint, necessitating the invention of techniques to mitigate them. Techniques such as
Work done at the Department of Computer Science and Automation, Indian Institute of Science. Our code is publicly available at http://githbb.ooommmnnyyaaaa///p

---

Published as a conference paper at ICLR 2023
network pruning Hoefler et al.) 2021), quantization(Gholamii al. 2021) knowledge distillation( et al. 202I). and low-rank Aecoopositionn/adereee et al. 2014) make possible to compress overparameterized models in order to improve real-world performance metrics such as inference time and power consumption. Pruning involves discarding elements of neural networks after gauging the importance saliencies of these elements. Generally, two broad categories of pruning techniques exist in the literature unstructured pruning, which involves removing individual weights from [
the model, such the results in Han et al. (2015); [eCun et al. (1989); Tanaka et al. (2020), and structured pruning (also called channel pruning for CNNs), which involves removing entire neurons or channels from the model (Ding et al. [02T] Luo at. 2017] |Paaaash et |
2019 [Singh et al. 2019)
Wang et al. 2021 He et al. 2017b. In this work, we focus on structured pruning for multi-branched CNNs.
Due to the complicated interconnections that exist in multi-branched networks, pruning multi- branched neural networks such as ResNets and MobileNets, raise unique challenges that do not arise when pruning single branch networks such VGG-nets (Simonyan & Zisserman 2015). These complex connections, such as residual connections in ResNets, require channels fed into the connection to be of the same dimensions, thus coupling the channels. Pruming such coupled channels (CCs) is generally not addressed in current works on structured pruning, such as Ding et al. (202I); Joo et al. (202T); Luo al. (2017); (iigh al. (2019); Wang al. (202T), which are designed for pruning single- single-bracchd networks; for example, in ResNets, only the output channels of the first two layers of a ResNet residual block are pruned, and the channels that feed into the residual connections are ignored when using these methods. Pruning CCs is challenging since not pruning filters from all the associated layers would break the CNN. Furthermore, pruning CCs is crucial as we observe that the layers associated with CCs take up a significant portion of the inference time more than 66% in ResNet-50.
The few methods for pruning CCs currently available generally rely on data-driven statistics of the output layer to infer saliencies and involve heavy finetuning( et al. 202T, [iuu et al. 2021 Luo & Wu 2020 Shen et al. (021). However, situations may arise where models trained on proprietary datasets may be distributed but not the dataset for reasons such as privacy, security, and competitive disadvantaged SinengeeYYnneee et al., 2020). Thus, pruning without data is an important challenge and active area of research(Patil & Dovrolis, 2021 (Sriniaas & Babu 2015 Tanaka et al. 2020). However, these techniques do not address pruning CCs, especially in the one-shot and data-free pruning regime, which is an open problem(Hoefler et 2021).
In this work, aim to prune CCs with the additional challenge of doing so without access to data. Towards answering the posed challenges, our contributions in this work are as follows.
1. Unlike single- branch networks, the CCs in multi-branchan networks provide additional chal- lenge for structured pruning. Identifying the associations between coupled layers, as well as the mappings between them, is nontrivial task. To address this problem, we define Data Flow Couplings (DFCs) to abstract the notion of coupling in a network by enumerating both the layers and transformations involved in the coupling. Two types of layers are associated with a DFC- feed-in and feed-out layers; their outputs and inputs are involved in the coupling.
2. Pruning involves measuring saliencies for elements to be pruned, which, for CCs, is not straight- forward due to interconnections between layers. In Section we investigate whether saliencies of channels in DFC can be inferred in isolation from the feed-in layers. To do so, define Maximum Score Disagreement to quantify disagreement in saliencies among the feed-in layers. We empirically observe significant disagreement among saliencies assigned to channels by the feed-in layers suggesting that the importance of such channels cannot be deduced in isolation. This leads us to propose grouped saliencies, with which we can rank coupled elements of DFC.
Measuring the effect of pruning coupled elements of multi-brande network without data, and thus inferring filter saliencies, is a challenging task. For this, Theorem proposss saliency mechanism, using the transformations enumerated by the DFCs, that bounds the joint reconstruction error of the outputs of the DFC's feed-out layers without data. To compute these saliencies, we propose the Backwards Graph-based Saliency Computation (BGSC) [ Algorithm. To mitigate the computational cost of this algorithm (both time and memory) for CNNs, provide parallelized implementation of the algorithm owing to its embarrassingly parallel nature. On ResNet- 101 for the CIFAR-10 10 dataset, we obtain 1.66x inference time speedup for 5 5% accuracy drop without
2

---

Published as a conference paper at ICLR 2023
HH
Input
Output
Cewon
Sranch
Moach
Cawdo
cawdo
frach
(a) A beed-forward neural network
(b) Residual block in ResNet-50 (c) Residual block in ResNet-50 with no layer in shortcut branch
with downsample layer
Figure I: Examples of data flow (suh)graphs in various networks
retraining. On ResNet-50 for the ImageNet dataset, we obtain inference time speedup of at least 47.1 % against our baselines for a 2.3% % accuracy drop with retraining.
2 NOTATION, PRELIMINARIES AND RELATED WORK
2.1 NOTATION AND PRELIMINARIES
Notation. Let [n] {I.....n} C N for any E N. Let E R" denote a vector, and M denote matrix. z:k) and (i, j) denote the hth and (i,j) element of x and M respectively. ||x||| and ||>||> denote the Ll-norm and max-norm of the vector respectively. xx/[|M denote the element- wise absolute (vector of <)mmatrxx of M). That is, |=](k) = xx)) and |M j) = (i. @ denotes the Hadamard product. The output of element wise transformation F applied matrix is such that the j element of the output depends only on the (i. element of the input. That is, for matrix M, FiM) f(, (M, A similar definition is used in this manuscript when considering element-wise transformations on vector and when the element-wise transformation is function of multiple matrices/vectors. Let |A| denote the cardinality of a set A.
Networks under consideration for analysis. Across definitions and derivations in Sections] 3 and 5] we assume that molly-connected (FC) layers are the only layers in the network that do not perform element-wise wise transformations. The network can have layers like batch-norm and non-linearities like ReLU that perform element- wisee transformations. Let there be L FC layers in the network. We assign each FC layer a unique integer from the set [L]. The order of numbering not important. Consider a layer / (Z]. We denote its weight matrix by W, For a layer / that is given an input x, the corresponding output y is obtained as y W,x.
Terminology for CNNs. We recall the standard definitions from Hoetler et al. (2021): y: Dumoulin Visin (0116). Channels denote the different views of the data (e.g., the red, green and blue channels of color image). Convolutional layers have multiple filters, each comprising multiple kernels. convolutional layer with n input and m output channels has m filters, each comprising kernels.
Data flow graph. The data flow graph of a neural network is directed graph that encapsulates the transformation produced by network. Each node in the graph applies some operation to the data provided. Each edge in the graph denotes the data flow between nodes, with data flowing from tail to head of the edge. The backwards graph of the neural network is similar the data flow graph except for the direction of edges being reversed. Figure La shows the data flow graph of layer
neural network. Such network is said have a single branch.
Terminology for ResNets. ResNets consist of Residual Connections He et al. 2016). A block of layers in these networks with residual connection around them is called a residual block. Figures ib [bl show instances of residual blocks in ResNet-50. These have two branches and thus make ResNets multi-branched. The residual branch contains most of the convolutional layers in the residual block. The other branch is called the shortcut branch. In ResNets, multiple consecutive residual blocks are clubbed together and called layer-block.
2.2 RELATED WORK
Pruning coupled channels (CCs). Existing literature Gao et al., 2019 Liu et al. 2021 Luo &
Wu 2020; Shen et al. 2021) prune CCs by grouping layers whose output channels are coupled.
Liu et al. (2021) propose an algorithm to group such layers. Liu et al. (202T): Luo & Wu (200)) ;
Shen al. (022)) utilize data-driven statisties of the output layer measure saliencies.
Gao
3

---

Published as a conference paper at ICLR 2023
et al. 2 and Chen al. (2021) alter the training objective by introducing prommtting
regularizers. But. Gao et al. (2019) use the Train-Prune- senue--inetune pruning pipeline(Han et al.
2013) whereas Chern et al. (2021) simultancously train and prune the network. The experimental results, particularly of Liu et al.
): Shhn et al. (202)). show that pruning CCs results in a better trade-off between accuracy and inference time speedups when finetuning is possible.
Saliency scores. Techniques exist in structured pruning that derive saliencies of channels from the information extracted from consecutive layers(Joo et al.] 2021 [uo et al. 2017) one layer(Hu et al [i al. (0177) without access to a dataset. Such structured pruning techniques locally measure the saliencies of channels. [ao al. (2019) and Yang al. (2018) utilize joint norms of weights in filters of grouped layers to infer saliencies. Minimizing the Reconstruction Error of the next layer is metric to gauge the saliencies of Channels[Luo et al., 2017 Yu et al. 2018 in structured pruning. However, this metric has only been applied to prune Mon-CCs and is assumed in the literature to not apply to CCs since such a metric requires notion of consecutiveness among layers(Liu et al., 2021I. However, in this work, we leverage DFCs to solve this problem.
Data-free pruning. Early efforts toward data-free pruning include [Srinivas Babu (0115), which measured similarity between neurons and merged similar neurons. Recently, Tanaka et al. (2020) proposed the Synflow method, an unstructured, data-free pruning method that relied on preserving gradient flow. Similar works include Gebhart et al. (202T); Patil & Dovrolis (2021). which use the Neural Tangent Kernel-base techniques to modify SynFlow. Yin et al. (2020) synthesize a dataset from a pre-trained CNN classifier and utilize the synthesized dataset to perform iterative
data-driven pruning, with the drawback that synthesizing the dataset from the classifier is very costly.
3 DATA FLOW COUPLINGS
In this section, motivated by the need to enumerate both the layers involved in CCs as well as the transformations between them, we define Data Flow Couplings (DFCs). We also provide examples of DFCs in ResNet-50.
Motivation for defining DFCs. Studies intending to prune CCs in CNNs either group layers
coupled(Gao al., 2019 Liu et al., 2021 Luo & Wu 2020 Shen iet al. 2021) or group weights across layers if they belong to the same coupled channel Chen al. 2021). However, these groupings do not simultaneously enumerate both the layers and the transformations involved in an instance of coupling. Such descriptions aid us in understanding the end-to-end transformation produced by the instance of coupling. In Section 5 we use these descriptions to derive a data-free mechanism for which DFCs are crucial. A Data Flow Coupling (DFC) abstracts the end-to-end transformation and the associated layers for an instance of coupling in network.
whose output channels
Definition Consider a neural network with FC layers where each FC layer is assigned a unique integer from the set [L]. Consider two sets of layers A = {a1, 02 .., ap), B M
(br, by....bb)) where A, B C [T.]. Let Letz mmm) be an arbitrary input sample from data set {2) ) j==1
that is fed to the network, let ua (m) va denote the input to and the corresponding output of layer (m) denote the same for layer b B. Suppose there exists collection of functions F defined by the data flow graph of the network, such that the input to any layer b E B
E A, and let (m) Yb
(("")) F. where Fy is a function of the outputs of layers E A. Let the function that gives the value of activation to the hthh neuron in Fg be denoted by Fok. Then, we say the tuple =< A, B, > is data flow coupling if
is obtained through map F5, Eue dimil!!"))
+ [ddim(x(
(C₂) F consists of element-wise mappings. For all b E B, E dim(xx (mn))
(m) (k)) = Fox
(w.mm) (v (k), vag (k). .....
(m) (m) (k)) day
(C2) Non-redundant. The subgraph of the data-flow graph consisting of layers in A. B, and the connections between them form a single component.
(Cs) Completeness. There do not exist sets A', B' c [L] and a collection of functions F' defined by the data flow graph of the network where A C A' and B B' and either A + A' or B + B' such that < A', B', F' >> satisfies conditions (C
C2)
4

---

Published as a conference paper at ICLR 2023
U1,00 ConaD.3
13
U1 Comod.J
Garzb
92 brnn
Sawdd_
Darad0
12, 9...00n
12,90 Dinwa
90 Denna
La LLCanoo.os
Figure 2: An instance of a DFC in ResNet-50.
We provide discussion for conditions (C2) and (Ca) of Definition
! Appendix
Some terminology for DFCs. For a DFC < A, B, F > denoted by т, we call the layers in the feed-in layers since their outputs used inputs by layers in B post transformations governed by F. Consequently, we call the layers in B the feed-out layers. Additionally, as consequence of
(C)) it is the case that for all A and b E B, dim(va (m)) dim(x;"") (m) n(T). We call n(T))
as the cardinality of coupling.
DFCs in CNNs. The notion of a DFC for a neural network with FC layers can be extended to CNN by altering the element-wise property of transformations in F channel-wise.
Example of a DFC in ResNet-50. Consider the subnetwork of ResNet-50, as shown on the left in Figure 2 For simplicity, we only show the convolutional layers and residual additions from the data flow graph of the subnetwork. Each convolutional layer in the figure has its assigned unique integer on the top left corner besides its corresponding rectangular box in the diagram. This example focuses on the DFC involving channel coupling due to residual additions. The DFC is A, F. F >, where A (9, 10, 13, 16), and B {11, 14, 17, 18). It is easy to infer the DFC's collection of functions F from the diagram. However, for completeness, we show Fi1.--
(i (m)) (k), v10 (k)) = ReLU (B... (v!"" (mm) (x:)) + ReLU (v (m)) (k))
(mn)
(2)
where (x), شيي ④ * + BL. Here, BN denotes the batchnorm transformation applied on the tth channel output from the (th layer of the neural network.
%
Properties of DFCs. Note that in Figure the tuple < A, F> with A = {11} and B = (12) and F capturing the associated transformation satisfies the definition of a DFC. Thus, the notion of consecutive layers, as in single- single-branch networks, is a special case of a DFC. Thus. DFCs simultaneously capture the transformational effect of coupled and non-CCs during the forward pass. Moreover, it is easy to see that network can be divided into collection of DFCs while preserving the overall transformation it produces.
GROUPED SALIENCIES
In this section, we investigate whether saliencies can be assigned to CCs in isolation using existing saliency mechanisms. To do so, define Maximum Score Disagreement, which quantifies the disagrecment or inconsistency in saliency rankings. When computed in isolation for each feed-in layer, our experiments suggest inconsistency among saliency ranks assigned corresponding channels of DFC. Thus, we propose Grouped Saliencies to rank channels of DFC.
Saliency mechanisms under consideration. Broadly, structured pruning algorithms compute saliencies of non-cont coupled channels and discard the lowest- ranked channels first (Hoefler et al. 2021). Saliency scoring mechanisms exist for structured pruning that statisties of the feed-in
5

---

Published as a conference paper at ICLR 2023
Mock
ayerbookk
Mick block
layer
Namber Chareedd
Chanmeds
Vie Score Disacrmement
Max Score Disecmenntt
Max Scoen Droxpreemet
(a) F.O. Taylor Approximate
norm
(c) Linear Replacibility
Figure 3: Plots illustrating disagreement of scores among the feed-in layers for gauging the importance of channels for ResNet-50 model trained CIFAR-I00 dataset. Each plot is histogram of the MSD values for the three saliency scoring mechanisms. DFC with particular downsampling layer its feed-in layers uniquely determined by the layer block containing it the ResNet-50 architecture. The DFC in figure 2 called layer-block-1 for this experiment. DFC named layer-block-1 have cardinality of coupling equal to 256, 512, 1024, and 2048 respectively.
block-1, 1, layer-block
block 2, laycr-block-3, 3, and layer-block-4
layer (Joo et al. 2021 Li al. 2017 Molchanov al. 2019). We use these saliencies illustrate
the necessity of grouped saliencies. Molchanov et al. (2019) propose to gauge the saliency of a filter convolutional layer by measuring the first-order Taylor approximation error attained on discarding the filter. Li et al. 2017) propose to use the LI norm of the weights in the filter to gauge the corresponding channel's s saliency. Joo et al. (202T) measure the saliency of filter by measuring how linearly dependent the filter is on other filters in the layer. To capture the variation of ranks assigned by various feed-in layers of DFC, we define the Maximum Score Disagreement (MSD) follows.
Definition 2 Maximum Score Disagreement. For DFC < A,B,F >, denoted by let rankia (k) denote the rank assigned by the feed-in layer a to channel k using saliency scoring mechanism. We then define the Maximum Score Disagreement for this channel k
MSD+(k)
max |ranka (k) ramks
A.
Discussion, Given DFC =< A, we compute a saliency score for each filter all the feed-in layers separately, thus giving us a rank for each filter in each feed-in layer from the set [n(T)]. We say that the saliency mechanism is consistent or in agreement if MSD, D (k) = 0 Ak e [n(T)]. When under agreement, ranki (k) ranks (k) Vk E [n(T)], and a, E A. This means that all feed-in layers agree regarding the order in which the corresponding channels should be pruned. However, if MSD, (k) is large for some least two layers regarding the importance of channel k.
[n(r)], there is disagreement among
Experiments. We perform three experiments to investigate the agreement among saliencies computed in isolation through the MSD values. We use the three saliency scoring mechanisms mentioned above in this section. We use all DFCs consisting of coupled channels arising from residual connections in ResNet-50 trained on the CIFAR-100 dataset (MIT License) for this experiment (training specifications in section of the supplementary material).
Observation. Consider llyck-block-4 in the histograms in figure 3d The most frequent bin for this particular histogram lies in the MSD Range of 1000-2000. This shows significant disagreements among saliency ranks computed in isolation by the feed-in layers. Similarly, we can see that the disagreement for all three importance measures is significantly high for all histograms in all plots of figure 3 Similar trends also arise for ResNet-50 trained on CIFAR- 10 and ImageNet datasets.
Grouped Saliencies. These observations show that need to jointly consider coupled elements of a DFC to infer their saliencies. We call saliencies that measure the importance of channels in a DFC using least of all the feed-in layers, all the feed-out layers, and the entire collection of functions as grouped saliencies.
In the following section, we propose an algorithm that computes a grouped saliency using all three elements of the triple of DFC as detailed in Definition 1

---

Let acc be as Published as a conference paper at ICLR 2023
5 A DATA FLOW DRIVEN DATA FREE GROUPED SALIENCY BASED ON THE RECONSTRUCTION ERROR OF INTERMEDIATE OUTPUTS
In this section, we propose an Algorithm called BGSC compute the saliency of all neurons in a DFC. We begin by describing the preliminaries for the Algorithm. We then describe the desired objective function measure our saliency. Finally, through Theorem we show that the saliencies computed using the BGSC Algorithm upper bound the desired objective function.
Setup. Consider a neural network with the DFC A, F denoted by t for which ua, Ve. denote the input to and the output of layer a E A. and by X6, denote the same for layer b E B. Let Pea denote the set of all paths from layer E B layer A in the backwards graph of the network. We aim to remove less important neurons from т. On removing neuron from 7, the output of the feed-out layers in may change. Thus, our goal causes the least perturbation in the output across all feed-out layers of т.
select neuron whose removal
Measuring Saliencies. Let E {0, 1)m(r) be a mask, such that ||s||| = n²()) 1. Here, setting s(k) 0 for any & [n(r)] is equivalent to pruning the hthh neuron from т. Thus, to infer the least salient neuron in т, we would want to solve the following optimization problem.
min kE(n(T)
Σ OPT(b) s.t. ||s||| = n(T) - 1, s(k) =
(4)
BEB
where OPT(b) W xx, the mask s.
Wx(x s)ll is the change in output of layer b B applying
Algorithm BGSC: Backwards Graph based Saliency Computation
Input: A DFC =< A, B, >, the backwards graph G
Output: List Sal.
D Sal(k) salieney of t neuron
1: Sal(k) + 0 for all k n(r)
2: for each a E A, E B do
3: for each path between and in G do
4:
= e for each node x do if v performs residual addition then Do nothing.
6:
7:
8:
else if v performs Lipschitz continuous demmntt-wise transformation then
Find C: matrix consisting tightest Lipschitz constants for the transformation.
10:
acc = Clace
11:
for all & [n(r)] do
12:
2 sAk = 0, s(j) 1 j [m(r)] (K)
13:
acc, (k) = W (e' s) O acc
14:
Sal(k) = Sal(k) + | [1)
Overview. The BGSC Algorithm traverses through all paths the backwards graph that exists between any pair of feed-out and feed-in layers of the DFC under consideration to compute the saliency of neurons. For each path, the Algorithm accumulates scores for each neuron. The saliency of neuron is then obtained by summing up the scores accumulated from every path. This is shown in line 14 of Algorithm While traversing each path म. the accumulated score is initialized as shown in line 4 of the Algorithm. Then
traverse the backwards graph along path π from the feed-out layer, we augment the accumulated score at every node depending on the operation it performs, depicted in lines 7 and 10 of the Algorithm. Once we reach the feed-in in layer, we perform one last augmentation to the accumulated score as depicted on line 13 of the Algorithm.
Theorem Suppose =< A.B.F > DFC as defined in Definition 7 computed in Algorithm for all a E A,b B, and E Poo- Then,
OPT(b) < Σ Σ
(acc,, ha (k))
Ab B
(5)
oE DEAXEP
Proof of Theorem is presented in Section of the Appendix.
7

---

Published as a conference paper at ICLR 2023
Prusang Besuts for Tenheti01 on CPASTO
Prusing Bearts for Reshettoo en CEXC30
Pruning Results for Resilesoo Imagelett Prusing Ranu for Ranheetcd en Imagerle:
Fraction parameters reraainnng
Fradtóon FLOPS remaining
Fracton parameters
Fraction parameters remiiinng
(a) Acc. Vs. params for (b) Acc. vs. FLOPs for ResNet101 on CIFARIO ResNet101 on CIFARIO ResNet50 on ImageNet ResNet101 on ImageNet
(c) Acc. vs. params for (d) Acc. vs. params for
Figure 4: Figures comparing Accuracy (Acc.) versus Sparsity (Sp.) (through FLOPs or parameters) for DFPC and Ll--orm-basee pruning strategies under the data-free regime. Additionally, the plots also compare Acc. vs. Sp. when we choose to prune CCs vs. when we don't. In each figure, the blue and the green plots denote Acc. Sp. for DFPC when choose prune CCs, and when we don't choose prune CCs, respectively. Similarly, the orange and the red plots denote Ace. Sp. LI when we choose to prune CCs, and when we don't 't choose to prune CCs, respectively.
From Theorem D we have
OOPT() Σ Σ (k))
S uulss
Σ Σ
|acci (1)
(6)
BEB
aEA,BEBREPS
&EA,DEB STE
Here, since we do not have access the u.s. and we know that the pixel values of an imput image are bounded, we define y = max { ( [. I the network. Here, the maximization over I denotes the maximization over the set of all possible images. Thus, we infer the saliency of neuron k in by
||]lu]"|oo]] ua (2) denotes the value of ua on feeding input
(k)-(k)
E
Σ
||aco (A:)]l
aEA,BEB PEP
Time complexity of Algorithm I Let n be the number of nodes in the subgraph of the backwards graph consisting of the feed-out layers, the feed-in layers, and the connections between them for a DFC r. Also, let P = Aee 3 Pba denote the set of all paths between the feed-in and feed-out layers of F in the backwards graph of the network. If 74 max dim(u,), and
= m dim)(xs) then, the time complexity of BGSC Algoritim is ((((T)))P)((v + (74))
BGSC Algorithms for CNNs. The BGSC algorithm is defined here for multi-branched neural networks. However, while it is computationally expensive for CNNs. it is embarrassingly parallel. In Appendix D we describe a parallelized implementation of the BGSC algorithm that is faster to execute. We use this parallelized version for experiments.
6 PRUNING EXPERIMENTS
In this section, we present the results of our pruning experiments obtained using DFPC on CNNs. Since our work, to the best of our knowledge, is the first to adopt data-free pruning prune coupled channels (CCs), we baseline our work against an extension of the L1-norm based saliency scores(Li et al.. 2017) (similar to Gao et al. (0099)) and random pruning. In Appendix E show how we measure these saliencies for CCs. Moreover, to strengthen the experiments, we baseline against structured pruning algorithms in the data-driven regime on the ImageNet dataset. We also show that decreases in FLOPs does not yield similar decrease in inference time, as also noted in Yang al.) (018)) Liu al. (2021). highlighting the importance of pruning CCs. Details of experiments and the ablation studies are presented in Appendix [
Compute Platform. Appendix A specifies the platform used for inference time measurements. Data-Free Experiments. Experiments are performed on ResNet-50/101, MobileNet-v2, and VGG16 for the CIFAR-10/100 datasets. We also present results on ResNet-50/101 for the ImageNet dataset.
and more in Appendix F show that DFPC consistently outperforms bI-based scores for given sparsity budget whether choose to prune CCs or not. For CIFAR- 10/100 datasets, the performance the LI based saliency score is quite similar whether we chose prune CCs or not, whereas pruning CCs with DFPC outperforms
Results of Data-Free Experiments. Figures 4a
4c
4d
8

---

Published as a conference paper at ICLR 2023
DFPC when CCs are ignored, in terms of both FLOPs and parametric sparsity. For the ImageNet dataset, both DFPC and Ll-based pruning perform similarly whether CCs are pruned or not. Finally, as noted in Tables [ and 5 both accuracies and inference times improve as we prune CCs. Our experiments suggest that generally, DFPC outperforms the L.1-based saliency score in terms of inference time gained for a particular drop in accuracy.
Data-Driven Experiments and Results. For comparison with contemporary work that finetune models after pruning, we present our results in Table [2] On GPU, for a 0.2% accuracy drop. (3)C(30) attains an inference time speedup of 1 1,53x, similar to that of Greg-2 2(ang et al. 2021), but with 2% higher test accuracy. Additionally, for accuracy drop of 2.3%, similar to GReg-2, DFPC(54) attains a 2.28x speedup which is 49% higher than the nearest basline (2 on GPUs, and 47.1% faster than the nearest baseline (ThiNet-33) on our CPU platform. Moreover, DFPC attains a 52% higher speedup for the same reduction in FLOPs on GPUs, and 60% improvement on CPUs, over GReg-2 and OTO.
Table 1: Pruning Results without using the training dataset and finetuning CIFAR-10. RN is an abhreviation for ResNet; CP denotes if we choose to prune coupled channels; RF denotes the reduction in FL.OPs; RP denotes the reduction in parameters; ITS denotes inference time speedup.
Model Name
CPP
Acc-1(98)
RF
RP
ITS:CPU)
ITS/GIU)
Unpraned RN-50
94.99
Ix
1x
Kandom pruned RN-50
No
90.51
1.09x
1.10k
1,06x
L,8K
I.I-nom prese
2017
RN-50
No
83.33
1.38x
1.32s
1.16s
1.17%
DFPC pruned KN230
No
39.95
1.44
1.82x
1,22s
1.22x
Random prinod RN-50
Yes
$3.39
1.09%
1.09s
1,15x
1.11s
Li-nom prane ei
2017
RN
Yes
90.87
1.28%
1.20x
1,62s
1.31%
DFFC pruned RN:30
90.25
1.58x
Unpruned RN-101
92.09
Random prinod RN-101
No
90.35
1.15k
1.14x
1.14x
1.06x
pranel Liel
2017
RN-101
$7.59
1.22%
1.22%
1.21x
1.0%
DFPC prunod RN-101
No
$9.80
1.53
1.84%
1.56s
1.25%
pruned 101
90:18
1.08x
L.08K
1.33x
1.14k
Li-nom pranelli et al
2017
RN-101
Yes
50.31
1.22X
1.21%
1.31x
1.11%
DEPL pruned IN-
Yes
50,14
1.64x
2.22x
1.66s
1.35%
Unprased VGG-19
93.50
1x
1x
1x
1x
Random prined VGG--1
90.09
1.11x
1.11x
1.54x
1.16x
Li-nom prune Li et al
2017
VGG-19
50.05
1.30%
1.96x
1.76x
1,31X
DEPC pruned 1ou
90.12
1.68%
3,16x
1.95s
1.43%
Table ResNet-50 for ImageNet with finetuning. The number x inside the brackets (x) in the Model Name column denotes the pruned model obtained after x pruning iterations.
Model Name
FLOPP Reduction
Parameter Reduction
Top-1 Accuracy(
Spectlp (GPU)
Spendup (CPU)
FLOP Reduction by Specdup (GPU)
FLOP Reduction by Speedup (CFU)
Ungruned
1.00%
1.00%
76.1
1.00x
1.00s
1,00x
1.00%
GReg Waag et al
2021
3.02x
2:31%
73.9
1.53x
1.36s
1.90
2:22
2866
1.4xx
1,25x
2.29
1.98x
1.84x
1.53x
1,4xx
1.39
ThiNet 30
2017
3.46x
2.9%%
71.6
1.50x
1.38x
2.31
2.51
DFPC
3 3.46x
2,65%
73.8
2.28
2/03c
1.51
1.70
DISCUSSION AND CONCLUSION
This work proposes data-free method to prune networks with coupled channels to obtain a superior accuracy vs inference time trade-off. To do this, we propose data flow couplings that abstract the coupling of channels in network. We also show the necessity of defining grouped saliencies. Finally, we provide an algorithm to compute grouped saliencies on DFCs based on the reconstruction error the output of the feed-out layers. We also provide a parallelized implementation of BGSC for use with CNNs. The algorithm attains superior speedups in both the data-free and data-driven regimes against our baselines. Notably, in the data-driven regime, DFPC pruned ResNet-50 obtains up to 47.1% % faster models for a 2.3% accuracy drop on the ImageNet dataset. In the future, we aim to develop pruning strategies robust enough to prune arbitrary networks and advance the goal of achieving faster inference times.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
The authors gratefully acknowledge AMD for their support. The authors also thank Ramaswamy Govindarajan (Professor, I[Sc). Himanshu Jain (IISc), Mehul Shankhapal (IISc). Rughavendra Prakash (AMD), and Ramasamy Chandra Kumar (AMD) for their insightful discussions pertaining to this paper.
The authors thank the reviewers for their valuable feedback which has helped us improve our work.
REFERENCES
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of opti- mization and generalization for overparameterized two-layer neural networks. International Conference on Machine Learning, pp. 322-332. PMLR, 2019.
Tianyi Chen, Bo Ji, Tianyu DING, Biyi Fang, Guanyi Wang, Zhihui Zhu, Luming Liang, Yixin Shi, Sheng Yi, and Xiao Tu. Only train once: A one-shot neural network training and pruning framework. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL bttps::/openzeeieew..t Corum?id=p5rii jreCzq
Xiaohan Ding, Tianxiang Hao, Jianchao Tan, Ji Liu, Jungong Han, Yuchen Guo, and Guiguang Ding. Resrep: Lossless cnn pruning via decoupling remembering and forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 45 4510-4520, 2021.
Vincent Dumoulin and Francesco Visin. guide to convolution arithmetic for deep learning, 2016. TRLptttss://axxii> wttp://wwww...aa/////110000.2.2.
Susan Gao, Xin Liu, Lung-Sheng Chien, William Zhang, and Jose M Alvarez. Vacl: Variance- aware cross-layer regularization for pruning deep residual networks. In Proceedings of the IEEF/CVF International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Thomas Gebhart, Udit Saxena, and Paul Schrater. A unified paths perspective for pruning at initialization. arXiv preprint arXiv:2 2021.
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. Low-Power Computer Vision, pp. 291 326.. Chapman and Hal//CRC, 2021.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):179-1188) 2021.
Song Han, Jeff Pool, John Tran, and William Dally, Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pp. 1389- 1397, 2017.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1-122 2021.
Hengyuan Hu, Rui Peng, Yu- fa-Wing Tai, and Chi- Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.0322. 2016.
10

---

Published as a conference paper at ICLR 2023
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convo- lutional neural networks with low rank expansions. In Proceedings of the British Machine Vision Conference 2014, pp. 88.1-88.13, 2014. doi: 10.5244/
www.siisdeeeeemeeemeeemoomnono c.28.88. URL http::///appp.mememssoosaaaeeeee pttp://aaaallllallaaaaaapp i lonoonpo/ on/pub 1099426768
Donggyu Joo, Fojindl Yi, Sunghyun Baek, and Junmo Kim. Linearly replaceable filters for deep network channel pruning. In The 34th AAAI Conference on Artificial Intelligence,AAAA), 2021.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference Learning Representations, 2017. URL
https::/openrevview net. Ftt.:/ooommmmdm=FFFFF.....
Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Har Hag Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In International Conference on Machine Learning, pp. 7021-7032. PMLR, 2021.
Jian-Hao Luo and Jianxin Wu. Neural network pruning with residual-conneetion and limited-data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1458-1467, 2020.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference computer vision, pp. 5058-5066, 2017.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1264-11272, 2019.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. The role of pweemearraetiizaiioonn in generalization of neural networks. In International Conference on Learning Representations, 2019. URL https::/ httt::://openreviee... net forum? ?id=
BygfghAcYX
Shreyas Malakarjun Patil and Constantine Dovrolis. Phew: Constructing sparse networks that learn fast and generalize well without training data. In International Conference on Machine Learning, pp. 8432-8442. PMLR, 2021.
Prafull Prakash, Chaitanya Murti, Saketha Nath, and Chiranjib Bhattacharyya. Optimizing dnn architectures for high speed autonomous navigation in gps denied environments on edge devices. In Pacific Rim International Conference on Artificial Intelligence, pp. 468-481. Springer, 2019.
Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalo- bos. Compute trends across three eras of machine learning. arXiv preprint arXiv:2202.05924, 2022.
Maying Shen, Hongxu Yin, Pavlo Molchanov, Lei Mao, Jianna Liu, and Jose M Alvarez. Halp: Hardware-aware latency pruning. arXiv preprint arXiv:2 10. 108/ 1, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015. URL
http:://arivv.org/aaa/////
1556
Pravendra Singh, Vinay Kumar Verma, Piyush Rai, and Vinay P Namboodiri. Play and prune: Adaptive filter pruning for deep model compression. arXiv preprint 1905::1955.04446 2019.
Suraj Srinivas and R Venkatesh Babu. Data-free parameter pruning for deep neural networks.
arXiv preprint arXiv: /507 2015.
11

---

Published as a conference paper at ICLR 2023
Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems, 33:6377 6389, 2020.
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Neural pruning via growing regularization. In International Conference on Learning Representations (ICLR), 2021.
Tien-Ju Yang, Andrew Howard, Bo Chen, Xiao Zhang, Alec Go, Mark Sandler, Vivienne Sze, and Hartwig Adam. Netadapt: Platform-awe aware neural network adaptation for mobile applications. In Proceedings of the European Conference Computer Vision (ECCV), pp. 285-300, 2018.
Hongxu Yin, Pavio Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the /EEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
8715-8724, 2020.
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I Morariu, Xintong Han, Mingfei Gao, Ching- Yung Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recogni- lion, pp. 9194-9203, 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3): 107-115, 2021.
APPENDIX
The appendix is structured as follows.
(a) In Appendix A we specify the setup and the procedure used to measure the inference time of models for the pruning experiments performed throughout the manuscript.
(b) In Appendix Data Flow Coupling as defined in Section 3 of the manuscript.
provide discussion on conditions (C2 and (C3) in the Definition of
(c) In Appendix we present the Proof of Theorem as promised in Section 5 of the manuscript.
(d) In Section of the main paper, we perform our experiments on CNNs. But, our definitions and derivations in Sections 3 and 5cooniddr neural networks with linear/full-cconnetted layers. In Section D we discuss how to apply the BGSC Algorithm(Algoritue | of the manuscript) CNNs compute the saliencies of channels.
As part of our pruning experiments from Section of the manuscript, we compare the efficacy of DFPC against two grouped saliencies extended from the Ll-based and random saliency mechanisms in the data-free regime. Section E shows how we extended the said saliency mechanisms to grouped saliencies.
(f) In Section F we state the experimental procedures and their results in detail for our pruning experiments presented in Section of the manuscript.
A SPECIFICATIONS FOR INFERENCE TIME MEASUREMENTS
Inference time measurements. We define the time taken to inference model on the test set as its inference time. Inference time for a given model is measured as follows in our experiments. The five epochs are warmups, and we discard their results. The inference time is now computed as the average of the next ten epochs. [Shen et al. (202T) use a similar method to measure inference times. Inference time does not include the time taken to load data into memory.
12

---

Published as a conference paper at ICLR 2023
A.1
CPU HARDWARE
Table 3: Specifications of CPU hardware used for inference time measurements
CPU Model Name
AMD EPYC 7763 64-Core
CPU(s)
256
(hread(s) per core
Core(s) per socket
64
Socket(s)
NUMA node(s)
CPU MHz
2445.419
Lid & Lli cache
MiB
L2 cache
64 MiB
1.3 cache
512 MiB
RAM
ITB (DDR4, 3200 MT/s)
The CPU inference time measurements performed part of the pruning experiments in Section are performed using the OS Ubuntu 20.04.3 LTS with kernel 5.13.0-39-gemert on the hardware specified in Table The software stack used for inferencing consisted of Python 3.9.7, PyTorch 1.10.1, and Torchvision 0.11.2.
A.2 GPU HARDWARE
The GPU inference time measurements performed as a part of the pruning experiments in Section are performed using the OS Ubuntu 16.04.7 LTS with kernel 4. 1.150-142-generrie specified in T The GPU is an NVIDIA 1080 Ti with CUDA 10.2 and memory of 12GB. The software stack used for inferencing consisted of Python 3.9.7, PyTorch 1.10.., and Torchvision 0.11.2.
the hardware
B DISCUSSION FOR CONDITIONS FLOW COUPLING
(C2
AND C3) IN DEFINITION OF A DATA
In this section, we discuss the requirement of conditions (C2) and (Ca) in defining Data Flow Coupling through examples. We begin by restating the definition and then providing examples that illustrate the importance of the two conditions.
Setup. Consider neural network with I, FC layers where each FC layer is assigned unique inte- ger from the set [L]. Now, consider two sets of layers
{a1, 42, ...,.., Jp B = {b), b2. b,} where A, C [L]. Let (m) be an arbitrary input sample from the data set {z] ) that is fed to the network. Then, by u. Va denote the input to and the corresponding output of layer E A, and by xx," (m) (m) denote the same for layer b E B. Let A, B be such that there exists collection of functions F defined by the data flow graph of the network. The input to any layer
b E B is obtained through a map F5 E.cs dim(v!"") - i dxx E F. F9 is function of the outputs layers a E A. Let the function that gives the value of activation to the Eth neuron in F, be denoted by Fis-
DFC Definition. The tuple 7 A,B,F a data flow coupling
(C1) F consists of element-wise mappings. For all b E B, k E dim(xx) (mx))
(m)) (k) (k) = Fox (v(Tm) (k), U0z (k).
(m)
(k))
(8)
(C2) Non-redundant. The subgraph of the data-fle flow graph consisting of layers in A, B, and the connections between them form a single component.
(C3) Completeness. There do not exist sets A', B' C [L] and a collection of functions F" defined by the data flow graph of the network where C A' and B & B' such that < A', , F satisfies conditions (C1) and
B' and either # A' or
Discussion for Condition (C.) We include this condition to avoid including redundant channels in DFC. Consider two DFCs network with the same cardinality of coupling and layers in
13

---

Published as a conference paper at ICLR 2023
common between the two DFCs. One might mistakenly club the two DFCs into by taking the union of their feed-in and feed-out layers. respectively. Thus, if condition (C2) were not present,
the combination of the two DFCs would also become DFC. This would create an undesired constraint to prune channels from both DFCs simultaneously.
ULBLONAD d0
(a) A DFC
(b) Not a DFC
Figure 5: Figures illustrating the importance of the completeness condition in the definition of a Data Flow Coupling.
Discussion for Condition (C3) This completeness condition ensures that none of the feed-in or the feed-out layers is left out when considering DFC. Let assume that the set of layers and transformations in Figure 5a satisfies the definition of a DFC. If condition (Ca) were not present, could mistakenly not consider all feed-in or feed-out layers while considering this DFC. An example for such an eтror is shown in Figure 5b
C PROOF OF THEOREM
In this section, we present the proof to Theorem [ posited for the BGSC Algorithm in the main mamuscript. We begin by setting up the mathematical preliminaries and re-stating the Theorem Finally, we present our proof.
Setup. Consider neural network with the DFC A, denoted by for which ua Vo denote the input to and the corresponding output of layer a E A, and by X Ys denote the same for layer b E B. In 7, each function F, captures element-wise transformations from operations like batch-normalization. non-linearities, etc. Thus, we model F; as a composite function. That is, Fù f (...) where each fg an element-wise function of v.s. Let Pho denote the set of all paths from layer b E B to layer E A in the backwards graph of the network.
Assumption We assume that all functions f in map the additive identity of their domain to the additive identity of their co-domain and are Lipschitz continuous.
Optimization Problem. Let (0, () be mask, such that ||s|| n²()) 1. Here, setting *(k) = for any E [r²()) is equivalent to pruning the t neuron from т. Thus, to infer the least salient neuron in 7, we want solve the following optimization problem.
min Σ OPT(b) s.t. ||s||| n(T) s(k) =0 kE(n(T)]
BEB
where OPT(() || the mask s. Theorem 1 Then,
Wx(Xi s)lli is the change in output of layer' b B applying
Let acc (k) be as computed in Algorithm 1 for all a E A,b and π Poa-
OPT(b) < >50
(acc (k)) vo E
(10)
A PEPS
14

---

Published as a conference paper at ICLR 2023
11,48.5 Sm
Canob
Figure 6: Focussing on layer 18, feed-out layer in a DFC.
Proof of Theorem 1
feed-out layer, of the DFC For instance, consider layer 18 Figure Consider OP)(b). Let e, e' be vectors whose element are all 1s. The dimensions of e, e' should be clear from the context. We have
We focus
(b)(b) = e² W (x, @ @s) (W& (Wal"e)".(C
@ xal
(11)
We now start unfolding the cascadation of functions that obtain x₁, from the u,S to prove the theorem. Let us define the accumulated score, acc, as the vector on the left in the inner-produse aeer-prouuct of the right-most term of That is,
acc Wol
(12)
Let us now perform case-wise analysis on f.. Let the accumulated score until unfolding level t be acc.
(a) Residual Connection: If f
12 where both f++1 and f22 are element-wise
functions on Vss. Then, we have
acc ( fol S acc" acc".I(e' s) f + acc" -) 1
(13)
(b) Elementwise Lipschitz continuous transformation: When f is Lipschitz continuous, there exists constant Ck. for each (k),,, such that (k)(())) (1( S Ck/r s for any two scalars and in the domain of fk().. Then, from Assumption
1 we have
acc- [[(e' s) C = acc" (r,
o r I ace" .|Crr,+ @
+ = (C.acc) -) o r- (14)
where C' is diagonal matrix with C as its th diagonal element. Thus, the new accumulated score
acCnew Clace.
(15)
Additionally, to generate tighter upper bound for equations 11 use the smallest constant
Cs that satisfy Lipschitz continuity for (k). As one unfolds F, to attain upper bounds OPT(b) using 13 and 15 in a DFC < F, guaranteed to attain situation where either of p(+1 2/ f fft+1 performs transformation on its only input v. for some a E A. Denote by acc the score accumulated until now by unfolding transformations from This condition should occur by the construction of the network. From here, we perform one more step of unfolding, where have
along the path in the backwards graph of the network.
accip-lle' s) @ val S l/WT s) @ acc }T, [u
(16)
Finally, the accumulated score for one path π in the backwards graph from a is acc (k)
[WT]((e - s) @ ačo,
a feed out layer ó of a DFC measures upper bound to the change in value of the output of b if we were to make output value of layer a to be zero for channel & and only for the path π. Thus, to measure the upper bound of joint perturbations of outputs of all feed-out layers removing a channel k. we take summation of these upper bounds across all paths π from between the feed-out and the feed-in layers.
Gist of proof: The value acc, (A:). uu, = W - s) O ačc |ua| for a feed-in layer a and a
15

---

Published as a conference paper at ICLR 2023
D APPLYING BGSC TO CNNS
Across definitions and derivations in Sections 3 and we consider networks with fully- connected layers as the only layers that do not perform element-wise transformations. But, we demonstrate the efficacy of our method through experiments on CNNs in Section 6 CNNs consist of convolutional layers which do not perform element-wist transformations. In this Section, we show an equivalent linear layer for any convolutional layer and how to use the BGSC Algorithm to compute the saliencies of channels in DFCs that consist of convolutional layers.
D.1 LINEAR LAYER EQUIVALENT TO A CONVOLUTIONAL LAYER
A convolutional layer with m input and n output channels consists of n filters and m kernels per filter. Let the 7t filter be denoted by the weight tensor W; R K x for all [r] where K x K is the size of the kernel. Let the jth kernel in the ith filter be denoted by the matrix Wi E R for all j E [m]. Assuming the bias terms to be zero, the j input channel and the tt output channel are denoted as I and O respectively then for all E [n].
O, = Σ
Wiy @ I
(17)
JELm]
where @ denotes the convolutional operation. Let us denote by Oij Wiy @ Ij. Then the (p.9)th element of the matrix Oiş is given by
K-1 K-1 Σ
Os5(p.9) Σ
(18)
This linear transformation. Thus, we can find an equivalent matrix for a convolutional operation. Thus, if I and Oiy denote the flattened vectors corresponding to the matrices Ij and O respectively, then there exists a matrix W.j such that Ö,, = W. If Ó denotes the flattened vector corresponding to the matrix O., have Ó, İj{(m] W. Then, we can write the transformation of convolutional layer through a linear layer follows.
Wn. W1,,... Wim W2. W22,... .... Wzm
i
O =
I
(19)
On Wna Woz..., Wmm
Im
Finding weight matrix for the equivalent linear layer. A convolutional layer has multiple configurations, such as padding, strides, and dilation. One way to computationally find the equivalent linear layer to a convolutional layer in the presence of all such configurations is to emulate the convolution operation. During the emulation, fill the equivalent linear layer's 's weight matrix if an input contributes to the computation of the output by the corresponding weight in the corresponding kernel.
Observation. The equivalent linear layer's weight matrix is sparse (consisting of many Os). Additionally, the weight matrix stores m. , elements, where Ig- T and Og O,, represent the dimensions of I₁ and O, respectively. This number can grow very large very quickly. Using the BGSC Algorithm for CNNs. To measure the saliencies of channels in a DFC r of CNN, we first need to think in terms of channels. Instead of element-wise transformations, the focus shifts to channel-wiss wise transformations, An output channel of channel-wise transformation depends only on the corresponding input channel. The shape of the output and input channels need not be the same; however, the number of input and output channels must be the same in channel-wise transformation. Additionally, the mask s is changed. Consider a convolutional layer in the set of feed-in layers. If we want to prune the tth channel, the mask is such that s(j) 0 for all (1 - 1)0,0, < j & i0,0,, and s(j) = 1 for all other js.
D.2 PARSING THROUGH CHANNEL- WISE OPERATIONS
CNN BGSC ALGORITIM
In this Section, we discuss how to parse through various stannel-wise transformations in the BGSC Algorithm to compute the saliencies of channels in DFC consisting of convolutional layers. Note that all element-wiss transformations are also channel- wise transformations. But, the converse does not hold.
16

---

Published as a conference paper at ICLR 2023
D.2.1 RELU OPERATION
The tightest Lipschitz constant for ReLU function is 1. This clearly is the case since max # (0, y)| S x yI for any 2, E Thus, the matrix consisting of the tightest Lipschitz constants [line 9 of BGSC Algorithm(T) for ReLU operation is identity matrix.
D.2.2 BATCH NORMALIZATION (2D)
For channel k. a batch norm layer linearly transforms each element of the t channel of the input, I, =ーPE TYe 3x where pky oke Tk: 3x are the parameters in batch norm layer. Thus, the de (i, i)th element of the matrix C is Ie where denotes the channel the jt input/output element belongs to.
D.2.3 MAX-POOLING, AVERAGE-POOLING (2D)
For each feature-map corresponding to every input channel, the pooling operation operates on each patch of the feature-map map to reduce their size. Max-pool computes the maximum value for each patch of a feature map to create the downsampled feature map. Average-poo computes the average value for each patch a feature map to create the downsampled feature map.
Consider a pooling kernel of size K₃ x K2. We assume that for max-pooling, over a sufficiently large number of samples, each element is equally likely be the maximum element in any patch of the image of size K1 x K2. Thus, in the long run, the transformation by the max-pool and average-pool is equivalent to a convolutional layer whose specifications follow. If the number of channels input to the pooling layers is m, then the convolutional layer has m input and output channels with filters such that for every filter E [m]. Wij matrix with all its entries as K,Ka j = and otherwise. The bias term is 0 for each channel, and the remaining configurations, like stride, padding, and dilation, remain the same as that of the pooling layer.
Now, from Section D.I] there exists an equivalent linear layer with weight matrix W, for the convolutional layer that is equivalent to the pooling layers. the accumulated score is ace until the BGSC Algorithm reaches node L. Then update the score as
acc
W/' acc
(20)
This is justified through the following inequality in the analysis presented in Section 5
(acc)a ) W Wi(f) @ @ S ([W,)" ([W,/acci,) acc " -I(c' -s o f(
(21)
D.2.4 ADAPTIVE AVERAGE POOLING (2D)
An adaptive average pooling performs average pooling. Here the pooling operation is specified by the shape of the output feature-map desired. Thus the kernel size for the layer is appropriately selected. Once the kernel size identified, the methodology is the same as that of average- pooling D.2.3).
D.3 MISCELLANEOUS IMPLEMENTATION DETAILS
In this Section, we describe choices made while implementing BGSC to produce the results in Section 6 Moreover, we report the execution times for BGSC algorithms on our hardware.
D.3.1 REDUCING MEMORY USAGE
Consider the second convolutional layer in VGG-19. takes 64 channels of 32x32 images as input and produces an output of the same dimensions. From Section D.I we know that the equivalent linear layer for this convolutional layer will require space to store 242 floating point numbers. Assuming each number takes one byte memory, the memory requirement for the weight matrix is already 4GB. This number jointly grows bi bi-quadratically with the dimensions of the input and output feature maps. Thus, to reduce this memory requirement, we use the sparse representation of matrices to represent the weight matrices corresponding to the equivalent linear layer.
D.3.2 REDUCING TIME TO COMPUTE SALIENCIES OF CHANNELS IN ALL DFCS IN NETWORK THROUGH PAR FARALLELIZATIN
The time complexity of the BGSC Algorithm for a DFC is (()) DFC, YA, 7B, n(T) are generally of the same order. So, define Tmaz max 7B, n(7)).
In
17

---

Published as a conference paper at ICLR 2023
Then, can write the dine-complextty of BGSC Algorithm to be m We know that the "maz for a DFC with convolutional layers grows quadratically with respect to the dimensions of feature maps and linearly with the number of channels. Thus BGSC is quite computationally expensive. However, we reduce the time taken to execute BGSC Algorithm by parallelly computing the acc for each path P. Moreover, since saliency computation of two DFCs can be performed independently, we parallelly compute saliencies for channels of multiple DFCs of the network.
Ymax) )
D.3.3 PRACTICAL RUNTIME OF BGSC FOR RESNET-50 AND MOBILENET-V2 ARCHITECTURES.
For exposition, we presented the BGSC Algorithm in Section 5 This algorithm is computationally expensive if run sequentially as per the pseudocode in Algorithm [ In an attempt to speedup saliency computation of DFC using the BGSC Algorithm on CNNs for our experiments, we exploited the embarrassingly parallel nature of this algorithm. In an attempt to prototype this algorithm for DesNer-50/100, MobileNet-v2, and VGG-19, we were able reduce the saliency computation time. However, there still remains scope for improvement which we discuss in this section case one aims to deploy this algorithm in production. We also report the time it takes to compute saliencies for the ResNet-50 and MobileNet-v2 architectures.
Execution times for BGSC Algorithm. We now report the time taken execute BGSC Algorithm on our CPU hardware (specified in Appendix A for computing saliencies of channels in all DFCs in ResNet-50 and MobileNet-v2 when each method can call upto 10 threads argument in our code).
Table 4: Latency executing BGSC algorithm to compute saliencies all channels specified networks when each method can call upto 10 threads.
Model Name
Dataset
Exccution Time
ResNet-55 ResNet-50
ImageNet
38 minutes
CIFAR-10F 10/100
12 minutes
MobileNet-v2
CIFAR-10/100
68 scconds
Further improvements possible. We now list further improvements to improve the latency of executing BGSC for all DFCs of a network. It is important to note that for the execution times measured for BGSC algorithm above, at one instant, the maximum number of DFCs that can be processed is also 10.
In Python, multithreading is not truly possible due to GIL (Global Interpreter Lock), thus we use multiprocessing. There are associated overheads with multiprocessing that affect the execution time the BGSC algorithm. Thus, dedicated effort to write code in C++ or CUDA may be benefical in reducing this overhead cost.
Due to lack of support, we were unable to leverage parallelism for sparse matrix multiplication. Dedicated effort to parallelise sparse matrix multiplication can further reduce the time taken to execute the BGSC Algorithm.
E EXTENDING L1-NORM BASED AND RANDOM SCORES TO PRUNE COUPLED CHANNELS
In this Section we demonstrate the usage the two saliency scoring mechanisms, Ll-norm and random, prune coupled channels. These have been used as benchmark to compare DFPC against in our Pruning Experiments(6).
Consider CNN with L convolutional layers. Let us assign each convolutional layer in the CNN unique integer in [L]. Additionally, consider a DFC Á, F denoted by in the CNN.
E. EXTENDING L -I-NORM BASED SALIENCY SCORE
For a convolutional layer E [z]. Li et al. 2017) assign the tthh channel a score of [[Wll where W. denotes the weights the am filter in layer L. We extend this saliency score to a grouped saliency score follows.
18

---

Published as a conference paper at ICLR 2023
We assign saliency score channel & [n(r)] the sum of L1 Ll-norms of the corresponding filters across all feed-in layers. That is,
Sal, (k)
I Ak E En().
(22)
This extension is similar to that proposed by Gao et al. 2019).
E.2 EXTENDING RANDOM SALIENCY SCORE
Extending this saliency score is trivial. We assign each channel & E n(T) number sampled from the uniform distribution between and as a saliency score. That is,
(k), 1[ Ak n(T).
(23)
Here, Ul(abb b] denotes uniform distribution between scalars b R.
F EXPERIMENTS IN DETAIL AND ABLATION STUDIES
In this Section, we present a comprehensive version of our experiments that we presented in Section of the main manuscript. We begin by presenting the experiments performed on the CIFAR-IO and CIFAR-100 datasets. Then, we present the experiments performed on the ImageNet dataset.
F.1 CIFAR-10 AND CIFAR- 100 EXPERIMENTS
Experimental Setup. We showcase our results using the CIFAR-10 and CIFAR-100 datasets (MIT License) and ResNet-50, ResNet-101, MobileNet-v2. and VGG-19. these experiments, we maintain a data-free regime. Additionally, we use two settings for our experiments to show the effect of pruning coupled channels and fairly compare DFPC, L1, and random scores for ablation. In the first setting, we prune both the coupled and non-coupled channels in the network. In the second setting, we only prune the non-coupled channels. This helps us understand the gain obtained from pruning coupled channels. These experiments are carried out for three grouped saliencies: DFPC, LI, and Random. Moreover, these experiments are performed two times on ResNets. In this first set of experiments, we prune both coupled and non-coupled channels. But in the second set of experiments, we only prune the non-coupled channels.
Pruning Procedure. Once we obtain grouped saliencies Sal, (k) for each channel of every DFC network, we compare these scores globally to select the channel to prune among all DFCs. To prevent layer collapse, we add check not to prune channel if a DFC has coupling cardinality of 1.
Pretrained Models. We train the models using SGD Optimizer with a momentum factor of 0.9 and weight decay of x 10 4 for 200 epochs using Cosine Annealing step sizes with initial learning rate of 0.1.
Tables 5 and 6 We produce these tables as follows. We prune 1% of the remaining channels
time the network and measure the top-1 accuracy of the pruned model. In these tables, we report the description of pruned models with accuracy closest to 90% for CIFAR-10 and 70% for CIFAR-100 For random saliencies, the tables report the average values obtained after three trials.
we plot the results our pruning experiments to show how accuracy varies with sparsities (with respect to FLOPPs and parameters) when choose to prune coupled channels for various strategies to gauge the importance of coupled channels.
Figures. In figures] 7 [3]
F.1.1 DISCUSSION OF EXPERIMENTAL RESULTT
From figures] 7 1 it is evident that DFPC outperforms LI and Random grouped saliencies in accuracy versus sparsity charts for both sparsity terms of parameters and FL.OPs. The margin of outperformance is significantly higher when pruning coupled channels. We observe that this superiority arises due to the occurrence of more pruning iterations to obtain a similar accuracy drop. Additionally, the gap of outperformance is reduced when sparsity is considered with respect to FLOPs. For all cases, but two, DFPC results in a pruned model with faster inference time despite similar accuracies. It is for ResNet-50 and MobileNet-ve v2 trained CIFAR- 10 that L1- norm-based grouped saliency produces pruned model with faster inference time when pruning coupled channels on CPU platform. Additionally, L--norr-bbac grouped saliency performs
19

---

Published as a conference paper at ICLR 2023
similarly in terms of accuracy versus sparsity charts whether chose to prune coupled channels or not. However, in the same regime, DFPC performs slightly better when pruning coupled channels. Thus, by looking at the trends in figures 7 12 and tables I and 5 it is the case that in general, for a given accuracy, both sparsity (in terms of FLOPs and number of parameters) and inference time speed-ups when pruning coupled channels them.
least good as when not pruning
To conclude, we were able to prune models without having access to the training data set or any statisties derived from We did not use fine-tuning either. We showed that our proposed method almost always improves performance in terms of sparsity and inference time speedups as opposed to readily- approaches to gauge saliencies of coupled channels in the absence of data set.
Table 5: Pruning Results without using the training dataset and no finetuning on CIFAR-100. RN is an abbreviation for ResNet; CP denotes if we choose to prune coupled channels; RF denotes the reduction in FLOPs; RP denotes the reduction in parameters; ITS denotes inference time speedup.
Model Name
CP!
Acc- 1(%)
RF
RP
ITS:CPU)
ITSIGPU)
Unprused RN-50
78.85
1x
1x
Random pruned RN-50
No
70.29
1.08x
L08s.
1,06s
1.0xx
Li-nom prane
2017
RN 50
No
70.24
1.66x
L.02K
1.17%
L.BKK
DFPC pruned RN:30
No
71.75
1,23%
1,20s
1,3x
1.1X
Random pninod RN-30
Yes
69.50
1.07%
1,07%
1,02s
1,025
I.1-omm prine 11
2017
RN-50
1.12s
DFPC prined XN:00
Yes 70.31
1.24x
Unprused RN-101
79.43
Random prined RN-101
No
71.66
1.07%
Li-nomu prine el
2017
RN-101
No
70.07
1.18s
1.32x
1.13%
DFFC pruned RN-101
No
70.01
1.71x
1,53
1.54
1,30%
Random pruned RN-101
Yes
31.68
1.08x
1.08s.
1.05
1,02%
Ll-nom prane Li 2017
RN
101
Yes
71.59
1.25%
1.12%
1.20s
1.66%
DEPC pruned KN-10
70.03
1.53x
1.82
1.34x
Unpraned VGG-19
72.02
1x
1x
1x
Tx
Random praned V9
68.92
1.02x
L.02x
1.00s
L.00s
Li-nom prane Li ci
2017
VGG-11
1.14x
pruned
70.10
1.50x
1.30x
Table 6: Pruning Results without using the training dataset and no finetuning for the MobileNet-v2 architecture. MV2 is abbreviation for MobileNet-v2; RF denotes the reduction in FLOPs; RP denotes the reduction in parameters; ITS denotes inference time speedup.
Model Name
Damaer
Acc-1(55
RP
RF
ITSICPU)
ITS(GPU)
Unpouned MV2
CIFAR-10
92.5
1x
1x
Ix
1x
1.1-norm yrino DFPC proned MVZ
2017 MV2
CIFAR-10
90.36
3.92x
3.60s
362s
2,3ss
CIFAR-10
90.08
5.32
374k
3.54x
2.23
Unpruned MV2
CIFAR 100
72.78
1x
1x
1x
1x
1.I-norm yrioof et DFPC proned MVZ
2017 MV2
CIFAR-100
71.87
2,70c
2.1xx
3.21x
2.12x
69.87
3,61
3.16
3.25k
2.15
20

---

Published as a conference paper at ICLR 2023
Pruning CIFAR10
Pruning CIFARIO
Pruning CIFAR10
Accuracy
Accuracy
Accuracy
dun
pariig
Fraction of parameters remaining
Fraction of FLOPS remainina
Fraction of parameters remainina
(a) Accuracy vs parameters for different pruning strategies when coupled channels are pruned
(b) Accuracy vs FLOPPs for different (c) Accuracy vs parameters plot for pruning strategies when coupled channels pruned
comparing DFPC and I.1 when pruning and not pruning coupled channels.
Figure 7: Plots for pruning experiments on the ResNet-50 architecture trained on CIFAR-10 dataset
Pruning Results ResNet101 an CIFAR10
Pruning Results ResNet101 on CIFAR10
Pruning Resultss for ResNet101 on CIFAR10
Accuracy
Accuracy
Accuracy
DeFt
OFF
Fraction of Darameters remainina
Fraction of FLOPS remainina
Fraction of carameters remainina
(a) Accuracy parameters for different pruning strategies when coupled channels are pruned
(b) Accuracy FLOPs for different (c) Accuracy vs parameters plot for pruning strategies when coupled channels are pruned
comparing DFPC and LI when pruning and not pruning coupled channels.
Figure 8: Plots for pruning experiments on the ResNet-101 architecture trained on CIFAR-I0 dataset
Pruning Results for VGG19 CIFAR10
Pruning Results for VGG19 CIFAR10
Accuracy
sccuracy
DPC Tanderrohl
M4
Frection perameters remainina
Fraction FLOPs remainino
(a) Accuracy vs parameters different pruning strategies
(b) Accuracy FLOPs for different pruning strategies
Figure 9: Plots for pruning experiments the VGG19 architecture trained on CIFAR-10 dataset
21

---

Published as a conference paper at ICLR 2023
Pruning CIFAR100
Pruning en CIFAR100
Pruning CIFAR100
Accuracy
Accuracy
Accuracy
Rordom
Sun
paring
Fraction of parameters remaining
Fraction of FLOPS remainina
Fraction of baramaters remainina
(a) Accuracy vs parameters for different pruning strategies when coupled channels are pruned
(b) Accuracy vs FLOPPs for different (c) Accuracy vs parameters plot for pruning strategies when coupled channels pruned
comparing DFPC and I.1 when pruning and not pruning coupled channels.
Figure 10: Plots for pruning experiments on the ResNet-50 architecture trained on CIFAR-100 dataset
Pruning ResNet101 CIFAR100 Pruning Results ResNet101 CIFARI00 Pruning ResNet102 CIFAR100
Accuracy
Accuracy
Accurat
Rardom al
parting
A4
Fraction of parameters rerniinin
Fraction of FLOPs remainina
Fraction parameters remainina
(a) Accuracy vs parameters for different pruning strategies when coupled channels are pruned
(b) Accuracy vs FLOPs for different (c) Accuracy vs parameters plot for pruning strategies when coupled channels pruned
comparing DFPC and LI when pruning and not pruning coupled channels.
Figure 11: Plots for pruning experiments on the ResNet-101 architecture trained on CIFAR- 100 dataset
Pruning Results for VGG19 on CIFAR100
Pruning Results for VBG19 on CIFAR100
Accuracy
Accuracy
DFPC
DPPC
Konkmokk22
Kambempu22 2
Panden
Frection perameters remeinina
FFaction of FLOPs remainina
(a) Accuracy parameters for different pruning strategies when coupled channels are pruned
(b) Accuracy FLOPs for different pruning strategies when coupled channels are pruned
Figure 12: Plots for pruning experiments the VGG19 architecture trained on CIFAR-100 dataset
22

---

Published as a conference paper at ICLR 2023
Pruning on CIFARIO Pruniin NobileNNttv2 CIFAR10C
Accuracy
BFC
Fraction of parameters remainina
Fraction of parameters remainina
(a) Accuracy vs parameters for DFPC and I.1 hased pruning strategies the CIFAR- 10 dataset. strategies the CIFAR- 100 dataset.
(b) Accuracy vs parameters for DFPC and L.1 hased pruning
Figure 13: Plots for pruning experiments on the MobilcNet-v2 architecture trained on CIFAR-10 and CIFAR-100 datasets.
F.2 IMAGENET EXPERIMENTS
F.2.1 WITHOUT FINETUNING (DATA-FREE REGIME)
In this Section, we present the results of Pruning on the ImageNet dataset. We perform the following set of experiments. For ResNet-50, and ResNet-101 we measure accuracy vs sparsity(in carried out for two grouped
terms of parameters) for the ImageNet dataset. These experiments saliencies: DFPC and LI. Moreover, we performed these experiments two times. We prune both coupled and non-coupled channels in this first set of experiments. But in the second set of experiments, we only prune the non-coupe non-coupled channels.
Pruning Procedure. Once we obtain grouped saliencies Sal,(k) for each channel of every DFC network, we compare these scores globally select the channel prune among all DFCs. Pretrained Models. Pretrained models of ResNet-50 and ResNet- 101 are obtained from the Torchvision library.
Figures. figure [4] we plot the results of our pruning experiments show how accuracy varies with parametric sparsities when we choose to prune coupled channels for the two strategies.
Pruning ResNet50 on ImageNet Pruning Results for ResNet101 on ImageNet
JMF aopporingoo DLS Klypeening Doo
DEFC 13 wis prantig Cc
Accuracy
6m6 parameters remaining
Kin 180
parameters remaining
(a) Accuracy vs parameters ResNet-50 models trained on the ImageNet dataset.
(b) Accuracy vs parameters for ResNet- 101 models trained on the ImageNet dataset.
Figure 14: Plots for Accuracy vs parameters for ResNet-50 models trained on the ImageNet dataset.
The accuracies drop quite quickly for models trained on the ImageNet dataset. However, still find that DFPC obtained better sparsities than the L1 score for both cases when we pruned coupled channels and when we didn't. t. Moreover, we see that the trajectories of pruning quite similar in terms of accuracy vs sparsity, irrespective of choosing to prune coupled channels. This could be attributed to the quick drop in accuracy of this experiment. Due quick drop in accuracies, we could not find suitable accuracy level where we could report speedup fairly.
Discussion.
Comparision with Yin et al. (2020) Yin et al. (2020) is contemporary work in Data-Free pruning that synthesizes the dataset from a pre-trained model. Synthesis of such a dataset is computationally expensive. In this comparison, compare the reduction in FLOPS the
23

---

Published as a conference paper at ICLR 2023
accuracy drop of Yin et al. (2020) and DFPC. For 1.02x FLOP reduction, the Accuracy of DFPC drops to 70.8%. However, Yin et al. 2020) maintain a 76.1% 1% accuracy for a FLOP reduction of 1.3x.
R.2.2 WITH FINETUNING (DATA-DRIVEN REGIME)
In this Section, present the experimental of pruning experiments ResNet-50 trained on ImageNet dataset when we finetune the model as presented in Table 2 in Section of the manuscript.
Experimental Setup. We use the pre-trained model of ResNet-50 available part of Torchvision for pruning. We prune 1% the remaining channels each pruning iteration followed by finetuning of 3 epochs, each with step sizes of 10 100 10 per pruning iteration. The batch size was 256. After the pruning ends, we finally prune the network for 90 epochs with a batch size of 512. We use the SGD Optimizer with momentum factor of 0.9 and weight decay of 1 x 10-4 and Cosine Annealed step sizes with an initial learning rate of 0.1. Here, we normalize the saliency scores of each DFC during each pruning iteration.
Pruned ResNet-50 architectures obtained. In Figure 15 we present the pruned architectures of ResNet-50 obtained on pruning and finetuning when using DFPC. We see that all layers are actually being pruned and that layers within the sets in the following list are being pruned which were not pruned by most structured pruning works. Moreover notice that all members within particular set have the same number of remaining channels.
{cooee)
((ayer110.doonssample),, layer1.0.conv3,, layerI.1.com33, layer1.2.compp
(layer220..oowsssammpe.. layer2.0.comv33, layer2.1.comr3, layer2.2.conv33, layer2.3.comw3)
Phnyer3.0..oumsammpe.. 1ayer3.0.come.. layer3.1.comet layer3.2.como3. lager3.3.come: layer33.5.o locoe3....oone3)
[lager4...doounsssmmll.. layer4.0.comme 1.cver,,..cov33,
24

---

Published as a conference paper at ICLR 2023
Model Architecture ResNet DFPC]
Number Channels
(a) ResNet-50 DFPC(30)
Model Architecture ResNet-50 pruned with DFPC154]
Number of Channels
for anrrition Coreen
BRana MigChaadii Charnll
(b) ResNet-50 DFPC(54)
Figure 15: Visualization of ResNet-50 architecture pruned using DFPC.
25


Reviews:
[{"review_id": "hVq0TnhFue", "reviewer": "Reviewer_gfw2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles an important problem of neural network pruning. Specifically, the authors of the paper propose a novel method to prune coupled channels in neural networks. For instance, the layers with skip connections in the ResNet model are considered to be coupled channels. This is an under-considered problem, and most of the previously proposed pruning methods neglect these coupled channels. Nevertheless, coupled channels consist of a significant portion of many modern neural network architectures. in essence, the proposed approach involves traversing through all the paths between any pair of such couple channels, and aggregating the overall pruning scores for all of these individual paths. Then, an overall score can be determined and such couple channels can be effectively pruned. The authors of the paper conduct careful experiments to demonstrate the effectiveness of the proposed method. ", "summary_of_the_review": "All in all, I have some minor concerns regarding the computational cost of the proposed method. nevertheless, I think this is a good paper and think that it should be accepted. ", "strength_and_weaknesses": "Strength: \n- The problem considered is an important one, and the proposed method is technically sound. \n- Experiments conducted show that the proposed method achieves good empirical performance. \n- The paper is well written in general. \n\nWeaknesses:\n- The proposed BGSC algorithm seems quite computationally expensive. While there is an analysis of the time complexity, it would be nice to also report the practical time taken to run the algorithm in practice. Does the proposed method take up significantly more time?\n- I feel like the BGSC algorithm needs more explanation. While the overview gave a good insight into the algorithm, there are a lot of technical details in the algorithm that needs to be further discussed and elaborated. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NCm9EksPr0h", "reviewer": "Reviewer_oXKf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to prune deep neural networks. The motivation of this work is to consider DATA FLOW DRIVEN PRUNING OF COUPLED CHANNELS  (DFPC), which considers the coupled channels (CCs) in a data-free mode when conducting network pruning. The key part of this work is developing a method called DFCs that abstracts the end-to-end transformation of CCs. Then, a Maximum Score Disagreement mechanism is proposed for group saliency for DFCs. Experiments on VGG and ResNets show promising results across various image classification datasets, including Cifar10/100 and ImageNet.", "summary_of_the_review": "Overall, I vote for borderline acceptance. Specifically, I liked the idea of investigating the disagreement mechanism among the coupled channels and proposing a new method for data-free pruning. ", "strength_and_weaknesses": "Strengths\n1. The paper targets the important problem of coupled channels for residual connections, which improves the performance of several classic networks.\n2. The observation of disagreement for channel importance is interesting. Before this, metrics mainly used L1 or Taylor Approximate which doesn't take any channel dependency into account. \n3. The overall writing is easy to follow.\n\nWeaknesses\n1. The authors claim that their method aims to prune channels like those fed into residual connections. However, they only conduct experiments on residual networks, and other popular networks such as mobilenets, densenets have no comparative experiments.\n2. It's hard to read the text and understand Figure4. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0E10Rbr9tBc", "reviewer": "Reviewer_6irU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focused on coupling the data flow to outputs and inputs, allowing for interconnections between outputs of\nmultiple layers. This led into the concept to prune CC without data to obtain aster latencies.", "summary_of_the_review": "Overall, this paper adds value to the body of knowledge.", "strength_and_weaknesses": "The algorithms and parameters were well outlined in this paper. In total, the contributions made in the paper were well set out.\n\nMore data visualisations could have been done to show the impact of this approach", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster