DEPrune: Depth-wise Separable Convolution Pruning for Maximizing GPU Parallelism
Authors: Cheonjun Park
Mincheol Park
Hyunchan Moon
Myung Kuk Yoon
Seokjin Go
Suhyun Kim
Won Woo Ro
Venue: NeurIPS 2024 poster (2024)
Keywords: Pruning
Depth-wise Separable Convolution
GPU
Primary area: deep_learning_architectures
TL;DR: Hardware-aware Pruning for Fast Depth-wise Separable Convolution

Abstract: Depth-wise Separable Convolution (DSConv) has a powerful representation even with fewer parameters and computation, leading to its adoption by almost all of the state-of-the-art CNN models. 
DSConv models are already compact making it hard to apply pruning, and there are few  previous pruning techniques that target depth-wise convolution (DW-conv).
In this paper, we present Depth-wise Separable Convolution Pruning (DEPrune), a novel pruning method applied to both point-wise and depth-wise convolutions. 
DEPrune is optimized by analyzing the computation of DSConv on GPUs.
DEPrune employs a fine-grained pruning approach, yet it achieves the structured sparsity typically absent in fine-grained pruning, enabling practical hardware acceleration. 
Moreover, this method maintains a high pruning ratio without causing any accuracy drop.
We additionally represent techniques that further enhance DEPrune performance: 1) balanced workload tuning (BWT), and 2) hardware-aware sparsity recalibration (HSR).
Experiment results show that DEPrune achieves up to $3.74\times$ practical speedup in DSConv inference on GPUs while maintaining the accuracy of EfficientNet-B0 on ImageNet.

DEPrune: Depth-wise Separable Convolution Pruning for Maximizing GPU Parallelism
Cheonjun Park', Mincheol Park23 Hyunchan Moon', Myung Kuk Yoon
Scokjin Go Suhyun Kim Won Woo Ro
1 Samsung Electronics Yonsei University 3 Korea Institute of Science and Technology
I.G Electronics Ewha Womans University Georgia Institute of Technology
{cheonjun. parkk,,, mincheol. park, wro}]yonsei.ee acc.seii.c........ ac.kr,
{mhcque92, dr. suhyun. kimala com, myungkuk yoo@ewaa..a ac. seokjin go@gatech. edu
Abstract
Depth-wise Separable Convolution (DSConv) has a powerful representation even with fewer parameters and computation, leading to its adoption by almost all of the state-of sate-offtheearr CNN models. DSConv models are already compact making it hard to apply pruning. and there are few previous pruning techniques that target depth-wise convolution (DW-conv). In this paper, we present Depth-wist Separable Convolution Pruning (DEPrune), novel pruning method applied to both point- wise and depth-wise convolutions. DEPrune is optimized by analyzing the computation of DSConv on GPUs. DEPrune employs fine-grained pruning approach, yet it achieves the structured sparsity typically absent fine-grained pruning, enabling practical hardware acceleration. Moreover, this method maintains high pruning ratio without causing any accuracy drop. We additionally represent techniques that further enhance DEPrunc performance: I) balanced workload tuning (BWT), and 2) hardware-ane aware sparsity recalibration (HSR). Experiment results show that DEPrune achieves up to 3.74x practical speedup in DSConv inference GPUs while maintaining the accuracy of EfficientNet- BO on ImageNet.
Introduction
In computer vision tasks, Convolutional Neural Networks (CNNs) have dramatically gained param- eters and computation [54] to solve complex and varied tasks [8] Such massive computation and memory footprint poses challenges in environments with limited hardware resources, such as mobile devices. Many research efforts have been made to address such problems, and two techniques have been most effective: Depth-wise Separable Convolution (DSConv) 6 and DNN pruning [17]
is composed of Depth-wise Convolution (DW-conv) and Point-wise Convolution (PW-conv), allowing it to have similar representation power to traditional CNNs that standard convolution, even with fewer parameters and computation
DSCony 6 43 45
3 Therefore, modern CNNs primarily
adopt DSConv when designing models 9 46 48)
DNN pruning eliminates redundant weight parameters without compromising representation power. Weight pruning [18] brings significant pruning ratio (PR) due to the fine- ame-grained approach but rarely reduces inference time compared to the unprumed model because of index computation overhead [54). In contrast, structured pruning [16] 30 32 21 38 is coarse- -ranine approach that is GPU-friendly and leads to a practical reduction in inference time.
co-coorespnnding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(4) Structured Pruning
(b) DEPrupe (DCP) [ours]
Weight
Weigh
Dephowis (DW-csm)
Ougeut
Oalei
Diagonal wise
Refactorization (DR)
colum veciee
LEN-GENESt (preaed
Recranged
DM copy
weight
wsing DR
Feature: Fine-grained nothod Sinuclured data paitem
Fine-graized method O, Siruciured data
pruncd
suppmmedd
Figure I: Depth convolution is rearranged to multi sub-GEMM GPU by applying Diagonal- wise Refactorization (DR). The 'X' and 'O" symbols indicate the absence and presence of corre- sponding characteristics for each method. Applying (a) Structured Pruning and (b) DEPrune (DCP) to multi sub- GEMM results in a structured data pattern. But (b) DEPrunc (DCP) is more fine-grained method than (a) Structured Pruning.
In DSConv, despite the DW-conv has only about 1% of the parameters, it spends over 82% of the overall inference time [41] As result, reducing the computation time of DW-conv impact the overall network execution time, and employing pruning on DW-conv can provide an effective solution to accelerate DSConv. Applying existing structured pruning (27 20 37 to PW-conv is not difficult, because PW-conv has the same GPU operation characteristic as standard convolution. Conversely, DW-conv has challenges in applying previous pruning methods. First, DW-conv has particularly fewer parameters, so applying coarse- sausse-rained pruning that creates hurdware-friende structured data pattern causes sigmificant accuracy loss. Second. the operation of DW-conv on GPU underutilizes the parallelism since the input unit is smaller than the operation unit, thus applying previous pruning methods is useless in this condition. Therefore, we propose Depth-wise Separable Convolution Pruning (DEPrune), a hardware-awes pruning approach specialized for DW-conv for fast and memory-efficien DSConv.
inherently
First, to address the aforementioned challenges: accuracy loss and underutilization, we analyze the operation of DW-conv widely used GPU. On GPUs, DW-conv computes by being transformed into multiple-GEMV (GEneral Matrix- Vector Multiplication) [1] and this structure does not fully utilize the GPU parallelism. Thus, for efficient operation on GPU, previous work applies Diagonal- wise Refactorization (DR) [41]. DR is method of rearranging DW-cony to multiple Sub-GEMM (GEneral Matrix-Matrix Multiplication) operations to maximize GPU parallelism (Fig. [T. DR places the weights diagonally and zero padding the rest. Therefore, if one non-zero weight is removed, all elements on the same column line have zero value, so the corresponding line all zero vector (Fig. D. At this point, we apply fine-graine pruning on DW-conv rearranged by DR, as we call the Depth-wise Convolution Pruning (DCP) (Sec. [.T]). DCP's fine mee-ggained approach provides novel PR and since most of the sub-GEMM, is pero-padded, it brings regular sparsity even with fine- grained pruning. This hardware-friendly format results in inference speedup without representation power loss. DÉPrune also applies conventional structured pruning to PW-conv; however, taking into account the computational significance of the DSConv model, we selectively prune PW-conv layers to maximize the pruning ratio without sacrificing accuracy.
Second, for DEPrune enhancement we consider the overall operation flow of DSConv to optimize GPU utilization. When DCP is applied, the PR is different for each sub- GEMM, which executes in different processing units. This results in a workload imbalance problem between processing units which directs to GPU under-utilization. The total execution time is set to the longest GEMM, thereby other idle processing units are forced to wait until the longest GEMM finishes. solve this problem, we propose Balanced Workload Tuning (BWT) that sets the same target PR for each sub-GEMM when applying DCP (Sec. 5.T). Our DFPrune applies existing structured pruning (37 PW-cony, and this approach avoids workload imbalance problem.
20 on

---

Pruning for DSConv
Pruning for DW-conv
Pruning for PW-conv Structured!
DCP
Enhance Method
Enhance Method
Status
[ours]
BWT [ours]
HSR [ours]
Pruning
HSR [ours]
DEPrune
DEPrune-B
DEPrune-BH
Table I: Terminology of DEPrune method. This symbol (t) means "we apply our methodology to determine which PW-conv to prune for better performance (Sec. 4.2)". BWT and HSR are our proposed method to enhance DEPrune. BWT and HSR are described in Sec. 5.T and Sec. 5.2 respectively. This symbol (V) means 'Applied'.
Lastly, for DEPrune enhancement we found that in addition to balancing PR between sub-GEMMs, adjusting the pruning ratio proportional to the GPU execution unit could further lead to significant speed improvements. Due to our technique's 's structured data format, theoretically, the acceleration should increase proportionally in a specific execution unit, the computational workload should scale with the unit of execution maximize the acceleration effect. Considering this unit operation, negligible weight parameters for more pruning. and this results in an additional significant speed improvement of 3-16% for DW-conv and also PW-conv without any accuracy loss. We call this Hardware-aware Sparsity Recalibration (HSR), an algorithm that recalibrates the appropriate target PR for DW-conv and PW-conv considering the execution unit of the GPU (Sec. (.2))
the pruning ratio increases. However, because GPU operates
additionally determine
To further optimize DSConv, various techniques are considered, which led to diverse and some- what complex set of terms being used throughout the paper. Therefore we provide Table I which summarizes the structure and terminology of our DEPrune.
(a) Deplllwwis Convolution (C--c))v) imput
(c) Diagonal-wise Refactorization
Sub-GEMM
weight channels
channcls
GEMV #32
GEMV #2
G muilt GEMM
GEMV R3
Sub-GEMM
GEMV Al
GEMV A2
grouping
M1
:
GEMV KM
GEMM
Sub-GEMM
(GEneral Matrix Multiplication)
AG
Ro Am
(b) Channel-by-CCnne by-Chenel
Figure 2: (a) DW-conv is rearranged to multi GEMV through (b) Channel-by by-nanee- on GPU execution. (c) Diagonal-wise Refactorization (DR) rearranges DW-conv into multiple sub-GEMMs. After DR, due to GPU tile size [14, we group M GEMVs into units of 32, resulting in total of G sub-GEMMs.
2 Preliminary
Prerequisites
DW-conv's weight filter 3D tensor. Given 1th DW-conv layer as D()) R Xkw, where M, kn, and kip are the number of channels, height, and width of the filters, respectively. The and Ta are the height and width of the input, respectively.
As shown in Fig. (a), DW-conv is composed of 3D input (I. (Rxxxx) 3D Weight xh. Xkw), each with channels performing independent 2D convolution operations. GPU rearranges standard convolution to GEMM, using im2col [2 5 to enable data reuse. Similarly, major deep learning frameworks (e.g., Caffe, PyTorch, MXNet, and TensorFlow) rearrange DW-conv to M multiple GEMV operations, using Channel-by- -yChannel (Fig. (b)). GEMV consists of
3

---

weight vector of size kn x kw and an input matrix of size (k), x kw) x (In x Iw). However, this approach suffers from the limitation of weight vector size being too small or 25) to fully utilize the GPU's processing units effectively.
Diagonal-wise Refactorization (DR) To address under-utilization, Diagonal-wise Refactorization (DR) 41 arranges the weight vectors of the GEMVs diagonally and sequentially places the input matrix (Fig. (c)). Next, zero padding is added to the empty spaces to create complete dense GEMM, which consists of M x (M x kn x kw) weight matrix and an (M x ky, x kw) x (In, x (w) input matrix. However, the rearranged GEMM is excessively large (M x k4 X ) so this operation requires significant additional computation on the GPU such as tiling [11] Therefore, DR further divides this dense GEMM into smaller sub- GEMMs of certain size. When executing matrix multiplication GPUs, grouping with a size of 32 channels is found to be the most efficient, resulting in a total of G sub-GEMM operations (G 32
M ). Thus each sub-GEMM is composed of a 32 x (32 ka x kw)
weight matrix and a (32 x kg x k x (Jr x Tw) input matrix. This approach allows for highly optimized GPU execution using specialized cuDNN libraries [5]
3 Related Works
3.1 Hardware-aware Pruning
Among the previous DNN pruning techniques, the following three methods consider hardware characteristics to reduce inference time: structured pruning, balanced pruning, and block pruning. Structured pruning [32 16 [3) determines redundancy at the vector level for pruning, thereby creating regular sparsity. This structured data pattern requires almost no additional index computation
GPU, making it effective in reducing inference time [54]. Block pruning [13] 36 49 that considers
the tiling technique [5). applies structural pruning at the small matrix level and has less representation power loss at the same PR compared to structured pruning. Balanced pruning [58] [1 31 23 37] is technique that divides the weight into consistent ranges and assigns equal PR each segment, ensuring a workload balanced characteristie. Most balanced pruning achieves approximately a 2x speedup when specific GPU with dedicated accelerator (e.g. sparse tensor core) is used at only 50% PR [35]- These pruning methods are based on the optimization technique of lowering for standard convolutions. Therefore, such pruning methods are difficult to apply to DW-conv. However, our DEPrune considers the hardware computation of DW-conv, enabling performance improvement.
3.2 Optimizations for DSConv
Since DSCоnV operates differently from standard convolution, continuous research is rapidly ongoing for the optimization of DSConv through dedicated software and hardware optimization.
On the pruning side, Multi-stage gradual pruning [47] prune the filters on DSConv using gradual pruning principle (57). Probability-based channel pruning [56] considers batch normalization when pruning in DSConv and requires little fine-tune to-ing.ing WN- 42 utilizes fine- nee-griieed pruning in DSConv merely reduce parameters without considering speedup. The filter pruning [34] sorts and prunes the filters according to their variance in each DW-conv. However, existing studies either only apply pruning PW-conv do not lead noticeable substantial inference speed improvement even pruning is also applied to DW-conv.
On the software side, DepthShrinker [10] removes non-linear activation functions after training and merges consccutive linear operations into a single dense operation to maximize hardware efficiency without compromising accuracy.
On the hardware side, GPU optimization 3 proposes a dynamic tile size scheme for GPUs to improve GPU utilization and hide memory access latency in DW-conv. The method (55) suggests loop rescheduling and register tiling on DW-conv, because when executing DW-conv on the parallel processor, traffic overload occurs between the cache, memory, and register. Diagonal-wise Refac- torization (DR) maximizes the parallelism of the GPU by proposing a rearrange method that combines all filters of DW-cony into multi GEMM.

---

12
Inctar (feeco imgth
overnedd 04l
MN
GEMV exocuution Time
wettor linges 1254
Top- aecurnry
InWooee
Wedter beggh
2
Chesed Phaing (wen)
Pruning ratio 180)
50
Weight vector length
Depth wise Corveluttoo Layer
(W)
[6]
(c)
Figure 3: (a) Comparison of accuracy drop between DCP and channel pruning on EfficientNer-B0 using ImageNet. (b) Measurement of the GEMV execution time of DW-conv 6th layer of EfficientNet- BO GPU. (c) Measurement of imbalance overhead of Mobilenet-V2 on ImageNet. The imbalance overhead is the difference between minimum sub-GEMM pruning ratio (PR) and layer's target PR.
Proposed Method: DEPrune
4.1 DCP: Depth-wise Convolution Pruning
(a) Motivation 1: Channel pruning on DW-cony has a large pruning unit size problem As shown in Fig. (b). DW-conv generates NEVti-GEMV format for each channel, on GPUs. DW- cony can also achieve structured data format, by evaluating the significance of each GEMV and eliminating unnecessary weight vector of GEMV. Nevertheless, when compared to the 4D tensor weight of the standard convolution, the DW-conv weight a 3D tensor (R (R xkw) notably fewer parameters. Consequently, eliminating a single channel (RR 1xxxxxkxx from DW-conv can greatly diminish its representation power. As shown in Fig. (a), when pruning is done channel-wise, there's a representation power loss of 1.66% compared to the unpruned model even just a 40% PR (EffficientNet-BO on ImageNet). This indicates that channel-wist pruning on DW-conv is not an appropriate choice.
(b) Motivation 2: Hardvare-unfriendly problem of weight pruning without DR Weight pruning experiences the least representation power loss among DNN pruning techniques with the highest pruning ratio. When applying weight pruning to the multiple GEMVs of DW-conv, the representation power loss due to increased PR is much less than the previously mentioned channel pruning. Looking at Fig. (a)) as our DCP similar to weight pruning, there are very minor representation power losses of 0.94% and 1.15% at 50% and 70% PRs, respectively. However, weight pruning without considering DR does not result in practical speedup from pruning. As shown in Fig. (b). the vector size of DW-cony GEMV is K2 X kw (e.g.. or 25). Since this smaller than the GPU's tile size (32), there is almost no change in inference time (Fig. (b)) since GEMV underutilizes processing units of GPU,
(c) Method: DCP We propose Depth-wise Convolution Pruning (DCP) to address the above two issues. We discover that weight pruning after DR can even achieve a structured sparsity in DW-cony with high PR, and making large matrix multiplication fully utilizes GPU parallelism. As shown in Fig. 4 first, we take the weight matrix rearranged in the form of matrix multiplication by DR. The height of the weight matrix is M, and the width M x ky, kg. As shown in Fig. 4 the unprumed values in the weight matrix are placed diagonally, while the rest are eero-pddded. Second, we sort the unpruned values ascending order and select the threshold value that corresponds to the target pruning ratio. When calculating the target pruning ratio, zero-padded values are not considered. Last, for each unprumed value, if it smaller than the threshold, we change to (i.e., magnitude pruning (18). Since the other values in the same column are already zero values, the column vector becomes a zero column vector, which is hardware- frieedffrddly.
Coorgn JeHEing PILO 5050 Top: Prans
RReerranged Weight Matris (using
2. Ranking (oscending
Prening grane)
BOOD
Calculate thresholl
uspernddd
pabling
prured
XCTO coletnn (structered)
Figure 4: Process of Depth- wise Convolution Pruning (DCP).
5

---

4.2 Methodology for Determining which PW-conv Layer to Prune
When filter pruning is applied to PW-conv, the parameters of the subsequent layers are removed with the same sparsity. DSConv has the following structure: DSConv, if PW-conv is filter pruned, the parameters of DW-conv are also removed at the channel level. The existing PW-conv pruning methods prune all PW-conv layers of DSConv, inadvertently leading to prune DW-conv layers as well. However, the parameters of DW-conv are only 1.34% of those PW-conv 41 so each weight element is more sensitive accuracy, thus for DW-Conv, rather than channel pruning, a more fine- fine-grained pruning is necessary. Therefore, our DEPrune does not directly prune all PW-conv layers. DEPrune applies fine-gra aene-grined pruning directly to DW-conv and does not prune PW-convl directly. We only apply filter pruning to PW-conv2. Pruning only PW-conv2, removes only the parameters the subsequent DSConv's PW-convl, which is less sensitive accuracy drop. Thus, DEPrune effectively prune all the layers of DSConv with high PR and representation power.
In
Enhance DEPrune
We propose the following two techniques to enhance DEPrune performance: BWT and HSR.
caamsnge
Conviiation
Preike
GENOr
156.%
1424)
(00
E'lheigh miveriteebiii
zero colarea rector
grasing untsize R
arpourel
peddiny
peated
Stractured Data Puttern (Gaadlore Gissall)
Balanced (Itrrhwam freedy)
Figure 5: Overview of DCP and Balanced Workload Tuning (BWT). (a) DCP is element-wise pruning method that creates a structured data pattern. (b) BWT equalizes the PR of all sub-GEMMs. The balanced range of BWT is 32 x kn X kw
5.1 DEPrune-B
Imbalance overhead problem of DCP GPUs allocate operations of a certain size streaming multiprocessors (SMs) for massively parallel processing. Therefore, DW-conv's multiple sub-GEMMs are also assigned to SMs, respectively. However, when applying DCP on DW-conw, the pruning ratio (P R) of sub-GEMMA may differ, given the varying importance of weights between sub-GEMMs. In that case, the execution time varies for each sub-GEMM due to the difference in PR. This results in a workload imbalance problem in that the other SMs of the GPU have wait until the SM with the lowest PR finishes. The acceleration effect of DCP is then determined by the minimum sub-GEMM P not by the layer target PR. Referring minimum sub- ab-GGMM PR and the layer target PR is compared for each layer of EfficientNet-BB In DW Layer 13, when the layer target PR 60%, the minimum sub-GEMM PR 50.4%, which varies up to 9.6%, which indicates that it decelerates execution by the amount specified.
(a) Motivation:
Fig. (c), the difference between the
(b) Method: Balanced Workload Tuning (BWT) To address the workload imbalance issue of DCP, we propose a DW-conv-specift Workload Balanced Technique that takes into account the operation structure of DW-conv (Fig. 55. DW-conv is dense matrix where non-zero values are arranged diagonally due to DR, while the remainder consists of zero values. We group all non-zero values within sub- GEMM which we call balanced range as illustrated in Fig. (b)). Within each balanced range, we rank weight elements with redundancy and systematically prune the lower-ranked elements until the target PR reached. As every sub-GEMM achieves Fig. (b)),, this resolves the workload imbalance issue associated with DCP. Since DCP is fine-grained pruning (pruning unit size: R), the representation power loss due to additional BWT almost negligible. A detailed analysis related to this is in Sec. 6.1
same target PR like

---

Inference Time
Speed Up (16)
10
Speed Up
(PCP ie lieel (DCP-6 livel 100 (%)
[DCP-8 time
300
Rotten (no speed up
Stagel DWCone
Stage2 DWVcoe
Stage5 PWccom
200
Groupstic: 52
Inference
100
Dino A0 ppecore Et ceee 12 splen 18 sociin 10 o
ppicon dB paicret
55t
22
22
The number of Pruned vector
(b)
Figure 6: (a) Measurement of speed increase by layer due to HSR. The orange bar the max speedup layer. DW-conv PR is 71% (b) Measurement of DW-conv inference time of EfficientNet-BO on ImageNet dataset. Inference time decreases with additional pruning of 32 or more vectors. GPU tile size is 32 [1].
5.2 DEPrune-BH
Unaligned problem As shown in Fig. (+))) maximize parallelism, GPUs divide GEMM operations into small tiles. In general, the size of the tile depends on the hardware specification of GPUs, but is usually multiple of 32 [14]. However, if the width of the unpruned weight matrix in Fig. (7) is not multiple of 32, some parts of the weight tiles are empty. This can cause an unaligned memory access problem on GPUs
(a) Motivation:
In Fig. (b), the inference time
does not decrease linearly with an increase in the size of the pruned vector. Whenever the number of pruned vectors increases by 32, the inference time decreases significantly like a step function graph. In DW-conv of Stage 2, the inference time decreases by 7% for each removal of only one tile. Thus, by removing a few additional weight vectors for aligned memory access, we can reduce the inference time by 7% if we align the number of pruned vectors with multiple of 32 (Fig. (b))..
(b) Method: Hardware-aware Sparsity Recalibration (HSR) We propose Hardware-aware Sparsity Recalibration (HSR) to solve the unaligned memory access problem and enhance DCP-B. As shown in Fig. (c), DCP-B with HSR operates in the following four steps. The first step. DCP-B is applied to DW-conv. The second step, we measure two essential factors (or and e) within the DCP-B model. (1) a : We measure the speedup obtained by solving the unaligned problem per layer. (2) e: We count the number of unpruned vectors of the unaligned tile matrix for each layer. We refer the result obtained by dividing the two parameters, a and e, for each layer as 3. The 3 refers to the size of speed obtained by removing one overflowed vector. The third step, the values of all layers are ranked by comparing them with each other. The last step, the layer with the /3 value of the top 50% is additionally removed as much as it overflows. The additional removed column vector consists of one non-zero value and prro-padded elements. Thus, there is no significant side effect on the representation power. On the other hand, the layer with the value of the bottom 50% additionally recovers as much as it is unaligned. The reason why the criteria for recovery and removal of HSR are set to 50% is to maintain the total target PR.
(4) Unaligned problem GPU
(b) Harddaare-waare Spansity Rocolibration (HSK) Under- uraliged
(c) DCP-8H (ICP-B HSR)
Tilesize
Stop 1. Preegeasz DCP-B
Unalizned (idic)
Lapeaned Model
Accuncy)
Step Menssennnt Bi
Kotey Nore Time
li
sic
lBce
lte
Over- unalipaal (x)
Recalbestion & Proce
Top 10% layer Frunc.
Wright
Prano repersal water (eoluse idle cvee specdup)
Bononn 50% layer Resover
Marris
Minery Arcms fire
Martoy Ascan
14 lyye for mash aarlgedd resten)
Figure 7: (a) Problem of unaligned pruning ratio GPU. (b) Concept of Hardware-aware Sparsity Recalibration (HSR). (c) Process of DCP-BH (DCP-B + HSR).
7

---

Pruning Ratio
Top-
Accuracy (%)
Speed
Model
Method
DW-conv
Real DW
PW-conv
Bascline
Pruned
Ditt.
DEPrune
78%
71%
50%
71.92
71.32
40.40
2.75x
MobileNet-V2
DEPrune-B
78%
78%
50%
71.92
71.51
-0.41
3.35x
DEPrune
82%
74%
50%
67.67
67.26
-0.41
3.88x
MobileNet- V3--mmlll
DEPrune-B 82%
82%
50%
67.67
67.13
-0.54
5.09x
DEPrune
85%
78%
40%
77.69
77.01
-0.68
4.5Tx
EfficientNet-BO
DEPrune-B 85%
85 %
40%
77.69
77.00
-0.69
5.79x
Table 2: Comparison between DEPrune and DEPrune-B (DEPrune + BWT) on ImageNet dataset.
This symbol (f) means 'DW-conv inference time speedup than unpruned DW-conv'. 'Real DW denotes the minimum pruning ratio among the sub-GEMMs of DW-conv. 'Diff." denotes the difference Top-1 aceuracy between the baseline and pruned models.
Pruning Ratio DW-Pat.
Top-1
Accuracy (%)
Speed
Model
Method
DW-cony
PW-conv
Baseline
Pruned
Diff.
Up
DEPrune-B 77.8%
50.0%
71.92
71.31
-0.41
3.35x
MobileNet-V2
DEPrune-Ble BH
77.9%
9u8o
50.1%
71.92
71.51
-0.41
3.52x
DEPrune-B
81.9%
60.2%
67.67
67.17
-0.50
5.09x
MobileNet--3-Seee
DEPrune-Bil BH
82.1%
6u5o
60.0%
67.67
67.18
-0.49
5.29x
DEPrunc-B 84.8%
51.9%
77.69
77.00
-0.69
5.79x
EfficientNet-BO
DEPrane-BH 84.7%
8u7o
52.0%
77.69
76.84
-0.85
6.15x
Table 3: Comparison between DEPrune-B and DEPrune-Bie (DEPrune-B + DW-conv HSR) on ImageNet dataset. This symbol (f) means 'DW-conv inference time speedup than unpruned DW- conv.' 'DW-Pat.' denotes the HSR pattern for DW-conv layers. 'u' and 'o" denotes under-aligned and over-aligned layers, respectively. "Diff.' denotes the difference in Top-1 accuracy between the baseline and pruned models.
Experiments
We assess the effectiveness of DEPrune using ImageNet ( and CIFAR-10 (25. For the validation of image classification, assess our method with CNN models using DSConv: MobileNet-V2 [3 EfficientNet-Bi [45) and MobileNet-V3 [22]
Experiment setting on ImageNet We utilize pre-trained CNN models sourced from the Pytorch framework (39]. We perform fine-tuning with only 65 epochs after conducting pruning methods. We set batch size of 256. We use SGD optimizer with the weight decay, 1 10 and the momentum as 0.9 for fine-tuning. The initial learning rate set to 0.001 and divided by 10 every 30 epoch. All data augmented with random cropping and horizontal flipping. We evaluate DEPrune using NVIDIA RTX 2080 Ti GPUs [ We measured the inference time using NVIDIA CUTLASS (4 We set the batch size to 32 to measure inference time.
6.1 Effect of BWT (DEPrune vs. DEPrune-B)
We analyze the changes accuracy and speedup resulting from applying the BWT to DEPrune (Table 2). DEPrune has varying pruning ratios among sub-GEMMs, causing the overall speed to be dictated by the sub-GEMM with the smallest pruning ratio. MobileNet-V2. the smallest sub-GEMM pruning ratio of DEPrune is 71%, as described in the Table 2 Therefore, DEPrune-B MobileNet-Va V2 21.8% (2.75x + 3.35x) x) faster in inference time than DEPrune. In MobileNet- V3-Small, DEPrune-B achieves 31.2% (3.88x x + 5.09x) improvement in inference time over DEPrune due to BWT. Since the balanced range of DEPrune-B is significantly large at 32 x kn, X kw- DEPrune-B B has an accuracy drop of within 0.1% than DEPrune across representative models.
6.2 Effect of HSR (DEPrune-B vs. DEPrune-BH)
We analyze the changes accuracy and speedup resulting from the application the HSR technique to DEPrune-B (Table 3. Since GPUs process operations and memory access in tile units, the actual speed of the GPU does not decrease linearly with the pruning ratio but rather decreases in a step-wise manner, shown in Fig. (b)). By adjusting the pruning ratio to fit the tile size, the DW-conv layer can

---

Pruning Rati
Top-1 Accuracy
Speed Up
Time
Method
Pruned FLOPs
DW-cony
PW-conv
Bascline
Pruned
DW-comv
Total
(us)
MobileNet-V2
71.9%
1,0x
1.00%
2306
CafeNet-R
37.1%
37.1%
73.7%
1.44x
AMC [19
30.0%
71.8%
70.8%
-1.0%
CC 24
28.
71.9%:
70.9%
-1.0%%
MetaPruning
30.7%
72.0%
71.2%
-0.8%
Random Pruming
28
29. 1%
71.9%:
70.9%
-1.0%
ATO 50
30.15
71.9%
72.0%
+0.15
RIAIL
12
29.4%
71.8%
71.3%
-0.5%
GFS
52
42.8%
42.8%
72.0%
68 8%
-3.2%
1,58x
1.60x
1448
GFS
37.1%
1.4xx
CafeNet-R
44
22.8%
22.8%
73.7%
71.9%
-1.8%
1.22x
1.23x
1871
CafeNet-F
14.2%
14.2%
73.7%
72.4%
-1.3%
1.15x
1.16X
1992
AMC 19
17.1%:
17.1%
72.0%
70.8%
-1.2%
1.17%
1.20X
1971
GFS
32
22.8%
22.8%
72.0%
71.2%
-0.8%
1.22X
1.23x
1871
CafeNet-R
14.2%
14.2%
73.7%
73.3%
-0.4%
1.5xX
1.16x
1992
DEPrune-BB BH [ours]
77.9%
52.7%
56.1%
71.9%
71.6%
-0.3%
3.52x
2.48
930
DEPrune-Bie [ours]
75.1%
64.8%
66.2%
71.9%
71.0%
-0,9%
3.11x
2.70
853
EhiiieenNet-Bl
1,00X
CafeNat-R
30.2%
30.2%
76.4%
74.5%
-1.9%
1.1XX
1.37x
4848
CafeNet-F
26.4%
26.4%
76.4%
74.6%
-1.8%
1.34x
1.30x
5085
DEPrune-BH [ours]
84.7%
62.0%
77.6%
76.8%
40.8%
6.15x
3,74x
1775
MobieNet-
67.7%
1.0x
1.00x
1857
GFS 52]
20.0%
20.0%
67.5%
65.8%
-1.7%
1.24x
1.23x
1499
DEPrunc-BH [ours]
82.1%
70.0%
67.7%
67.1%
-0.6%
5.29
4.12%
450
MobileNer. V3-1 argee
74.0%
1.00X
1.00x
4892
HPGM 20
33.0%
33.0%
74.0%
73.1%
-0.9%
1.48x
1.47%
3945
DEPrune-BH [ours]
77.0%
43.0%
74.0%
73.7%
-0.3%
4.13x
2.83x
1187
Table 4: Comparison of inference time (us) with DEPrune-BI BH and the latest structured pruning on ImageNet dataset. 'Diff.' denotes the difference in Top-1 accuracy between the baseline and pruned
models. DEPrune-BIs applies filter pruning using f--nrrm to PW-conv [26]. This symbol (+) means "baseline model'.
achieve an average inference time speedup of 6.37%, as illustrated in Fig. (a). applying HSR DEPrune-B shows almost difference in accuracy compared According to Table
applying HSR within 0.16%. Specifically, the accuracy difference is only 0.01 % on MobileNet- -3-Smll. For DEPrune-BEr all models have nearly identical numbers of over-aligned and under-aligned layers. DEPrume- BH achieves 6.2% (5.79x > 6.15x) inference time speedup compared to DEPrune-BB on EfficientNet-BO Additionally, HSR be applied to PW-conv layers well.
6.3 Comparison with Structured Pruning
In T we conduct experiments comparing DEPrune with the latest structured pruning methods across four models. On MobileNet-V2, our DEPrune-BH reduces approximately 26.7% more FLOPs compared to RLAL, while exhibiting a 0.2% smaller accuracy drop. GFS removes up to 42.8% of DSConv parameters, resulting in accuracy drop exceeding 3%. In contrast, DEPrune-BH climinates 75. % and 64.8% of parameters in DW-conv and PW-conv, respectively, with an accuracy drop within 1%. On EfficientNet-BO, while other methods prune around 30% of DW-cony, our method prunes 84.7% with only a 0.8% accuracy drop. On MobileNet- -3-Smalll and MobileNet- -----e-------eeeee DEPrune-BI achieves inference times 3.33 times and 3.32 times faster than GFS and FPGM, with accuracy drops of 1, 1.1% and 0.6% less, respectively.
6.4 Discussion: Various Pruning on PW-conv
We apply four structured pruning techniques to PW-conv layers measure the changes in accuracy (See Table When applying fi-norm pruning and fy-norm pruning to PW-conv layers, the accuracy difference within 0.06% for all models except EfficientNet-BB According to the FP paper [27]. there minimal difference between fi-norm and norm pruning, and this similarity is also observed the case of DEPrune-BH. Conversely, on EfficientNer-BBC FPGM (0 which uses geometric median achieves 0.33%, and 0.09% higher accuracy compared to f--orr and f2-norm pruning. respectively. BCBP (37 is block-wise pruning method that can be applied PW-conv. However, applying BCBP to PW-conv following DW-conv climinates some parameters of DW-conv. Therefore,

---

Method
Pruning Ratio
Top-1
Accuracy (%)
Model
DW-conv
PW-conv
DW-conv
PW-conv
Baseline
Pruned
Diff.
FP (EI-norm)
27
78%
40%
71.92
71.59
-0.33
MobileNet
DEPrune-BH [ours]
FP (12-norm)
27
78%
40%
71.92
71.54
-0.38
V2
FPGM 20
78%
40%
71.92
71.42
-0.50
BCBP 37
78%
40%
71.92
71.23
-0.69
-nomm)
27
82%
50%
67.67
67.11
-0.56
MobileNet
DEPrune-BH [ours]
PP (fg-norm)
27
82%
50%
67.67
67.17
-0.50
V3
FPGM 20
82%
50%
67.67
67.18
-0.49
Small
BCBP 37
82%
50%
67.67
66.09
-1.58
27
85%
52%
77.69
76.60
-1.09
EfficientNet
DEPrune-BH
FP (Ly-norm)
27
85%
52%
77.69
76.83
-0.85
B0
[ours]
FPGM [20]
85%
52%
77.69
76.93
-0.76
BCBP [37
85%
52%
77.69
75.91
-1.78
Table 5: Comparison with various pruning methods [27] 20
37
applied to PW-conv on ImageNet
dataset. 'Diff.' denotes the difference in Top-1 accuracy between the baseline and pruned models.
when applying BCBP to PW-conv the accuracy drops on all models described in Table S compared to FP and FPGM.
Conclusion
In this work, we propose a new Depth-wise Separable Convolution Pruning (DEPrune) method tailored for DW-conv to reduce DSConv inference time and fully leverage GPU features. Extensive experimental results on the ImageNet dataset demonstrate that DEPrune effectively preserves represen- tation power, even with higher PR than structured pruning, achieving regular sparsity. Moreover, two techniques, BWT and HSR, further enhance DEPrune's capabilities. With these combined features, DÉPrune-Bil BH achieves substantial GPU speed gain of up to 4.1 x on MobileNet- MobileNet-V3-Small V3-Smal..
8 Acknowledgements
This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IIP) grant funded by the Korea government (MSIT) based wet---eeeeee-Huuuueeeee 11212068. Artificial Intelligence Innovation Hub), and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (RS-2024-00335003),
Simulation- Data Center Workload/System Analysis Platform), (RS-2021-
References
[1] J. Burgess.
Rix on-the nvidia turing gpu. IEEE Micro, 40(2):36-44, 2020.
[2] K. Chellapilla, S. Puri, and P. Simard. High performance convolutional neural networks for document processing. In Tenth international workshop on frontiers in handwriting recognition. Suvisoft, 2006.
[3] W. Chen, Z. Wang, S. Li, Z. Yu, and H. Li. Accelerating compact convolutional neural networks with multi-threaded data streaming. In 2019 IEEE Computer Society Annual Symposium on VLSI (ISVLSI), pages 519-522, IEEE, 2019.
[4] Y. Chen, X. Dai, D. Chen, M. Liu, X. Dong, L.. Yuan, and Z. Liu. Mobile-former: Bridging mobilenet and transformer. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 5270-5279, 2022.
[5] S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, Tran, B. Catanzaro, and E. Shelhamer. cudnn: Efficient primitives for deep learning. arXiv preprint arXiv: 1410.0759, 2014.
[6] Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
10

---

[7] J. Choquette and W. Gandhi. Nvidia a100 gpu: Performance & innovation for gpu computing. In 2020 IEEE Hot Chips 32 Symposium (HCS), pages 1-43. IEEE Computer Society, 2020.
[8] Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255, leee, 2009.
[9] X. Ding, X. Zhang, J. Han, and G. Ding. Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2022.
[10] Y. Fu, H. Yang, J. Yuan, M. Li, C. Wan, R. Krishnamoorthi, V. Chandra, and Y. Lin. Depthshrinker: a new compression paradigm towards boosting real-hardware efficiency of compact neural networks. In International Conference Machine Learning, pages 6849-6862. PMLR, 2022.
[11] T. Gale, M. Zaharia, C. Young, and E. Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.1090 2020.
[12] A. Ganjdanesh. S. Gao, and H. Huang. Jointly training and pruning cnns via learnable agent guidance and alignment. IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2024.
[13] D. Guide. Cuda c++ programming guide. NVIDIA, July, 2020.
[14] C. Guo, B. Y. Hsueh, J. Leng, Y. Qiu, Y. Guan, Z. Wang, X. Jia, X. Li, M. Guo, and Y. Zhu, Accelerating sparse dnn models without hardware-support via tile-wise sparsity. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-15, 2020.
[15) C. Guo, B. Y. Hsueh, J. Leng, Y. Qiu, Y. Guan, Z. Wang. X. Jia, X. Li, M. Guo, and Y. Zhu. Accelerating sparse dnn models without hardware-support via tile-wise sparsity. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 200 IEEE Press, 2020.
[16] S. Guo, Y. Wang, Q. Li, and J. Yan. Dmep: Differentiable markov channel pruning for neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1539-1547, 2020.
[17] S. Han, H. Mao, and W. J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. International Conference on Learning Representations (ICLR), 2016.
[18] S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pages 1135-1143, 2015.
[19] Y. He, Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings of the European Conference on Computer Vision (ECCV), pages 784 800, 2018
[20] Y. He, P. Liu, Z. Wang, Z. Hu, and Y. Yang. Filter pruming via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4340-4349, 2019.
[21] Z. Hou, M. Qin, F. Sun, X. Ma, K. Yuan, Y. Xu, Y.-K. Chen, R. Jin, Y. Xie, and S.-Y. Kung. Chex: Channel exploration for cnn model compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12287-12298, 2022.
[22] A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang. Y. Zhu, R. Pang. V. Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314-1324, 2019.
[23] I. Hubara, B. Chmiel, M. Island, R. Banner, J. Naor, and D. Soudry. Accelerated sparse neural training: provable and efficient method find n: m transposable masks, Advances in neural information processing systems, 14:2109-221111 2021.

---

[24] A. Kerr, D. Merrill, and J. Tran. Cutlass: Fast linear algebra in cuda c++. NVIDIA Developer Blog, 2017.
[25] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[26] H. Li, A. Kadav, L. Durdanovic. H. Samet, and H. P. Graf. Pruning filters for efficient convnets. arXiv preprint arXiv: 1608 (00.000077660 10, 2016.
[27] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2017.
(28) Y. Li, K. Adamczewski, W. Li, S. Gu, R. Timofte, and L. Van Gool. Revisiting random channel pruning for neural network compression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 191-201, 2022.
[29] Y. Li, S. Lin, Liu, Q. Ye, M. Wang, Chao, Yang, J. Ma, Q. Tian, and R. Ji. Towards compact cnns via collaborative compression. In Proceedings of the IEEE/CVF Conference on 2021.
Computer Vision and Pattern Recognition, pages
[30] M. Lin, R. Ji, Y. Wang. Y. Zhang, B. Zhang, Y. Tian, and L. Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF conference on computer vision and pattem recognition, pages 1529-1533 2020.
[3]] M. Lin, Y. Zhang, Y. Li, B. Chen, F. Chao, M. Wang. S. Li, Y. Tian, and R. Ji. Ixn pattern for pruning convolutional neural networks. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(4):3999
2022.
[32] Z. Liu, H. Mu, X. Zhang, Z. Guo, X. Yang, K.-T. Cheng, and J. Sun. Metapruning: Meta learning for automatic neural network channel pruning. Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3296-3305, 2019.
[33] G. Lu, W. Zhang, and Z. Wang. Optimizing depthwise separable convolution operations on gpus. IEEE Transactions Parallel and Distributed Systems, 33(1):70 87, 2021.
[34] Y. Mao, Z. He, Z. Ma, X. Tang, and Z. Wang. Efficient convolution neural networks for object tracking using separable convolution and filter pruning. IEEE Access, 1164466-164774 2019.
[35] A. Mishra, J. A. Latorre, J. Pool, D. Stosic, D. Stosic, G. Venkatesh, C. Yu, and P. Micikevicius. Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.08378 2021.
[36] S. Narang, E. Undersander, and G. Diamos. Block-sparse recurrent neural networks. arXiv preprint 1rXiv:1711.02788 2017.
[37] C. Park, M. Park, H. J. Oh, M. Kim, M. K. Yoon, S. Kim, and W. W. Ro. Balanced column-wise block pruning for maximizing gpu parallelism. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9398-9407, 2023.
[38] M. Park, D. Kim, C. Park, Y. Park, G. E. Gong, W. W. Ro, and Kim. Reprune: Channel pruning via kernel representative selection. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 14545 14553, 2024.
[39] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, Kopf, E. Yang. Z. DeVito, M. Raison, A. Te- jani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, berf-peeformnnce deep learning library. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, F. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024 8035. Curran Associates, Inc., 2019.
[40] D. Qin, C. Leichner, M. Delakis, M. Fornoni, S. Luo, F. Yang, W. Wang, C. Banbury, C. Ye, B. Akin, Mobilenerv4-univerraa models for the mobile ecosystem. arXiv preprint arXiv:2404.10522 2024.
12

---

[41] Z. Qin, Z. Zhang. D. Li, Y. Zhang, and Y. Peng. Diagonalwise refactorization: An efficient training method for depthwise convolutions. In 2018 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2018
[42] P. K. Rao and S. Chatterjee. Wp-unet: Weight pruning u-net with depthwise separable convolu- tions for semantic segmentation of kidney tumors. Research Square, 2021.
[43] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
[44] X. Su, S. You, T. Huang, F. Wang, C. Qian, C. Zhang, and C. Xu. Locally free weight sharing for network width search. In International Conference on Learning Representations, 2021.
[45] M. Tan and Q Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114 14. PMLR, 2019.
[46] M. Tan and Q. Le. Efficientnetv2: Smaller models and faster training. In International conference on machine learning, pages 10096-10106. PMLR, 2021.
[47] C.-H. Tu, J.-H. Lee, Y.-M. Chan, and C.-S. Chen. Pruning depthwise separable convolutions for mobilenet compression. In 2020 International Joint Conference on Neural Networks (IJCNN) pages 1-8. IEEE, 2020.
[48] P. K. A. Vasu, J. Gabriel, J. Zhu, O. Tuzel, and A. Ranjan. Mobilcone: An improved one millisecond mobile backbone. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7907-7917, 2923,
[49] D. T. Vooturi, D. Mudigere, and S. Avancha. Hierarchical block sparse neural networks. arXiv preprint arXiv: 03420, 2018
[50] X. Wu, S. Gao, Z. Zhang. Z. Li, R. Bao, Y. Zhang, X. Wang, and H. Huang. Auto-train-once: Controller network guided automatic network pruning from scratch. IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2024.
[51] Z. Yao, S. Cao, W. Xiao, C. Zhang, and L. Nie. Balanced sparsity for efficient dan inference on gpu. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5676-5683, 2019.
[52] M. Ye. C. Gong, L. Nie, D. Zhou, A. Klivans, and Q. Liu. Good subnetworks provably exist: Pruming via greedy forward selection. In International Conference on Machine Learning, pages 10820-10830. PMLR, 2020.
[53] J. Yu and T. Huang. Network slimming by slimmable networks: Towards one-shot architecture search for channel numbers. arXiv preprint arXiv: 1728, 3, 2019.
[54] J. Yu, A. Lukefahr, D. Palframan, G. Dasika, R. Das, and S. Mahlke. Scalpel: Customizing dnn pruning the underlying hardware parallelism. In 2017 ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA), pages 548-560. IEEE, 2017.
[55] P. Zhang, E. Lo, and B. Lu. High performance depthwise and pointwise convolutions on mobile devices. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6795-6802, 2020.
[56] H.-I.. Zhao, K.-J. Shi, X.-G. Jin, M.-L. Xu, H. Huang, W.-L. Lu, and Y. Liu. Probability- channel pruning for depthwise separable convolutional networks. Journal of Computer Science and Technology, 37(3):584-600 2022.
[57] M. Zhu and S. Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.0182 2017.
[58] M. Zhu, T. Zhang, Z. Gu, and Y. Xie. Sparse tensor core: Algorithm and hardware co-design for vector-wise sparse neural networks on modern gpus. In Proceedings of the 52nd Annual IEEF/ACM International Symposium on Micrarchitecture, pages 359-371. ACM, 2019.
13

---

A Appendix
A.1 Limitation of Balanced Pruning
number of pruming studies [51 37 58 7 31 have been proposed to balanced pruning. a high pruning ratio, the balanced pruned model has negligible accuracy drop. However, achieving balanced pruned model's speedup difficult without support for specific hardware architecture or kernel. Therefore, NVIDIA has changed its hardware structure for balanced pruning 77]
A.2 Why Balanced Pruning Cannot Be Applied to DW-conv?
Balanced pruning [51] is pruning method that accelerates performance by considering workload balance. This approach divides the weight matrix into consistent vector units and assigns an identical PR to each vector. The designated vector range is referred within the vector are ranked based on their redundancy, and those with the lower ranks are pruned sequentially until the target PR achieved. However, directly applying traditional balanced pruning to DW-conv is not suitable. This is because the balanced pruning method doesn't t take into account the structure of DW-conv. Since non-zero value weights in DW-conv with DR are arranged diagonally when setting balanced range, the majority of the values within that range end up being zeros.
'balanced range'. Elements
A.3 Limitation
The limitation of DEPrune is that it is specialized for DW-conv. Specifically, DEPrune is difficult to apply to the Vision Transformer series. Although some models the Vision Transformer 4 use DW-conv, most of the computations in Vision Transformers performed using self-attention and feed-forward network layers. However, our proposed HSR technique appears to be applicable to these layers.
A.4 Experiments on CIFAR-10 Dataset
We experiment the effectiveness of our proposed method using the CIFAR-10 dataset 25 For the validation of image classification, we experiment our method with CNN models: MobileNet-V2 43 and EfficientNNt-BBl [45) We perform fine-tuning with only 100 epochs after processing pruning methods on CIFAR-10.
A.5 Limitation of Channel Pruning on DW-conv
When channel pruning is applied to depth-wise convolution, the pruned model has structured data pattern. However, the pruning unit size of channel pruning is k4, x kw. Since channel pruning is 9 (32) or 25 (5²) times larger than DEPrune with pruning unit size of 1, thereby channel pruning reduces more representation power than DEPrune. Mobilenet-V2 on CIFAR-10, when PR 50% the difference in accuracy between DEPrune and channel pruning is 0.23% (See Table Fven when PR 70%, the difference of accuracy 0.35%. Therefore, our proposed DEPrune, which has not only structured data pattern but also representation power advantage, is appropriate for DW-conv.
MobileNet-V2 on CIFAR-10
Pruning Ratio
Accuracy
DW-cony
PW-conv
Channel Pruning
DEPrune
50%
30%
93.07%
93.30%
60%
30%
92.95%
92.99%
70%
30%
92.45%
92.80%
Table 6: Comparison of accuracy between DEPrune and Channel Pruning with MobileNet-V2 on
CIFAR-10 dataset.
A.6 Comparison between DCP and Filter Pruning on PW-conv
When filter pruning applied to PW-cony, the PW-conv pruned model has a structured data pattern. If filter pruning is performed on PW-cony, the channel of DW-cony that follows is also removed. Even
14

---

the channel of PW-conv behind DW-conv is removed as well. Therefore, the pruning unit size of filter pruning not simply filter of the corresponding PW-conv. The pruning unit size includes also the parameters of following layers. The larger the pruning unit size, the greater the probability that core parameter is included the removing group, which influences the representation power. the other hand, our proposed DCP does not remove the parameters of other layers. Therefore, when the PR is 70%. our DCP is 0.62% higher representation power than filter pruning of PW-conv (See Fig. 8
95
Accuracy (%6
-8-
DCP [ours]
92
FP on PWccovv
91
Baseline
90
40
Pruning Ratio (%)
Figure 8: Comparison of accuracy (%) with DCP and filter pruning (FP) PW-conv of MobileNet- V2 on CIFAR-10.
A.7 Effect of Balanced Range
Research on n.m sparsity is currently very active topic in the field of pruning. However, this sparsity approach has two major limitations: lack of flexibility and the requirement for specialized hardware. First, it lacks flexibility because it is fixed at a 50% pruning ratio, specifically 2:4 pruning (37]. As seen in Table 7 we conducted comparative experiments between NVIDIA's n:m sparsity and DEPrune on MobileNet-V2 using C1FAR-10. At the same pruning ratio of 50%, DEPrune-B achieves 0.31% higher accuracy than n:m sparsity. This is because DEPrune-B achieves 50% pruning ratio within total parameters, whereas n:m sparsity achieves 50% pruning ratio within parameter size of 4 [23]. Secondly, n:m sparsity, achieving optimal performance requires specialized hardware (NVIDIA A100) that can quickly handle index processing 37). In contrast, our approach requires only a customized GPU kernel for execution.
MobileNet-V2 CIFAR-10
Method
Pruning Ratio
Accuracy
Diff.
MobileNet-V2
93.86%
NVIDIA n:m sparsity
50%
92.99%
-0.87%
DEPrune-B
50%
93.30%
-0.56%
Table 7: Comparison of accuracy (%) with DEPrune-B and NVIDIA n:m pruning on CIFAR-10 dataset. 'Diff.' denotes the difference in accuracy between the baseline and pruned model. NVIDIA n:m pruning's and m size are and DEPrune-B applies filter pruning using fn-nrrm PW-conv.
A.8 Efffec of Balanced Workload Tuning (BWT)
We propose Balanced Workload Tuning (BWT) solve the workload imbalance problem on DCP (Sec. 5.1.. However, the BWT method may slightly reduce the representation power. As shown in Table 8 we compare DCP-B (DCP + BWT) with DCP on EfficientNet-B00 of CIFAR-10. When PR is 50%. the difference between the DCP-B and DCP accuracy is 0.17%. Therefore, there is little representation power loss due to BWT.
A.9 Peak Memory Usage
According to paper 41, the extra overhead in total memory consumption due to zero-padding is approximately 0.3%. To assess impact DEPrune-BH, we measured and presented the peak memory usage of MobileNet-V V2 before and after applying DEPrunc-Bi BH with 50% pruning ratio, as
15

---

EfficientNet-BO on CIFAR-10
Pruning Ratio
Accuracy
DW-conv
PW-conv
DCP
DCP-B
10%
10%
91.25%
%
20%
10%
91.18%
91.31%
50%
10%
91.16%
91.33%
Table 8: Comparison between DCP and DCP-B EfficientNet-Be on CIFAR- 10 dataset.
shown in Table 9 Before applying DEPrune-BH, the peak memory usage is 7.22 MB, whereas after application, it decreases to 3.63 MB, representing a reduction of approximately 49.8%.
Peak Memory Usage (MobileNet-V V2 on ImageNet)
Pruning Method: DēPrune-BH
Pruning Ratio
Pre-pruning After-pruning
GAP
50%
7.22 MB 3.63 MB 3.59 MB
Table 9: Analysis of Peak Memory Usage (MB) with DEPrune-BBi BH on ImageNet dataset. 'GAP'
means the after-pu pruning peak memory usage difference rate compared to pre-pruning. DEPrune-Bie -BH applics filter pruning using mn-mm PW-conv.
16

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification:
We have fully reflected our arguments in abstract section and introduction
section.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification: We write limitation for our study in appendix.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA].
Justification: Our paper does not include theoretical results.
Guidelines:
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes].
Justification: We write information for our study in experiment section.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No].
Justification: We going open source the code later.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification: We write experimental setting for our study in experiment section.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA].
Justification: The paper does not include experiments.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes].
Justification: We write computer resource for study in experiment section.
9. Code Of Ethics
17

---

Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes].
Justification: Our research conform NeurIPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
There is no socictal impact of the work performed.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: The paper poses such risks.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
Justification: We cite the original paper that produced the code package or dataset.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification: The paper does not release new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification: The paper does not involve crowdsoureing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human
Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: The paper does not involve crowdsourcing nor research with human subjects.
18


Reviews:
[{"review_id": "oBiFcGN5N0", "reviewer": "Reviewer_cCXz", "summary": "The paper addresses an important topic of pruning depth-wise separable convolutions (DSConv) called DEPrune. While structural model pruning methods like channel pruning can achieve significant speed-up for regular convolutions, they cannot secure notable speed-up on DSConv layers as they mainly prune the point-wise convolution, which is a small fraction of the compute for DSConvs. Further, naively pruning the depth-wise channels significantly reduces its capacity, leading to severe performance degradation. DEPrune prunes the depth-wise convolutions in a fine-grained manner yet achieves structural sparsity to enable practical speed-up on GPUs. It also introduces two techniques BWT and HSR to further improve the performance.", "questions": "I think the paper is interesting and can be useful in practical applications. Currently, as far as I know, DSConvs are not used in computationally intensive architectures like the U-Net models in diffusion models. Eventually, when more efficient architectures using DSConvs are developed, the proposed DEPrune can further improve the performance. I hope the authors release their implementations so that the community can benefit from them.", "limitations": "The limitations are explained in the supplementary.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper is well-written. The motivations behind the design choices are clearly explained. \n\n2. The problem of pruning DSConvs is an important topic as DSConvs are widely adopted for edge applications like MobileNets. Therefore, further increasing their efficiency is of significant interest to the community.\n\n3. Extensive experimental results validate the design choices and show clear speed-up compared to the original models.", "weaknesses": "I cannot think of a major weakness in the paper. I hope that the authors provide more background for the readers not directly familiar with the concepts of \"alignment in GPUs\" or refer to proper references.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude to the reviewer for their constructive and insightful feedback. Below, we have provided our responses to the comments. If the paper is accepted, we will incorporate these helpful suggestions into the camera-ready version.\n\n**Weakness 1) [I hope that the authors provide more background for the readers not directly familiar with the concepts of \"alignment in GPUs\" or refer to proper references.]**\n\nWe agree with the reviewer’s comment that the concepts of \"alignment in GPUs\" should be presented in our paper for better understanding. Therefore, if given the opportunity to revise our paper, we will include these details, referencing prior work [1, 2].\n\n[1] Sparse GPU kernels for Deep Learning, SC 2020\n\n[2] GUIDE, Design. CUDA C++ programming guide. NVIDIA, July, 2020.\n\n\n**Question 1) [I hope the authors release their implementations so that the community can benefit from them.]**\n\nWe are willing to provide our implementation once the paper is accepted to contribute to the community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n9rCM3PViF", "reviewer": "Reviewer_CSWm", "summary": "This paper presents DEPrune, which prunes depthwise separable convolution models into a structured sparsity pattern that is friendly to depthwise refactorization.", "questions": "1. Why are the efficiency metrics of any methods missing in Table 4?\n2. Can the DCP part be applied to general group convolution (where the number of groups != number of channels)?\n3. Following Q1, some of the structured pruning methods in the GKP [3] series are known to be able to maintain the output shape of a pruned conv layer by leveraging group convolution. Can this be applied to the PWConv part of the DSConv and therefore avoid the concerns introduced in Section 4.2 and Appendix 6?\n4. Is a DEPruned model zero-padded? It looks like the case in Figure 4. If so, what kind of peak memory usage are we seeing pre-and-after pruning?\n5. How would this work compare to general structured sparsity approaches with more pruning freedom? e.g., the N:M sparsity [4]?", "limitations": "The authors addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Much of the motivation for conducting CNN pruning is to pursue the most lightweight — often time edge-deployed — model run under much resource constraint. Depthwise separable convolution (DSConv)-based models have a proven track record in this regard, and it is true that there are very few attempts to investigate how to prune such models. So, the exploration done in this paper is a welcoming addition to the field.\n\n2. The authors clearly demonstrate the further complications caused by filter pruning DSConv models and address them by proposing a structured sparsity pattern that is friendly to depthwise refactorization — a common application for improving the utility of DSConv — along with many other submodules. The DEPruned models show real end-to-end speedup.\n\n3. Tables 3 & 4 show DEPrune delivers better accuracy and runtime results than applying typical structured pruning methods to DSConv models.", "weaknesses": "1. Most of the structured pruning methods compared in Table 4 is a bit old. Although I suspect the conclusion won't change much due to the gap present and the fact that methods like CC [1] are often plenty performant even for today, still, a comparison with more modern structured pruning methods should be included. \n\n2. The authors emphasize that \"there are no previous pruning techniques that even target depth-wise convolution (DW-conv)\" in line 4. What about methods like [2] that address the exact same DSConv?\n\n3. The paper did a good job explaining each of its components in detail but lacks a clear high-level overview of how these components are attached together — some abstract visualization with a detailed walkthrough in the caption, as well as a pseudo-code section, should be added. Similarly, the experiment procedure lacks clarity: only the epoch budget of CIFAR10 experiments is shared, and critical details like the learning rate for finetuning, whether the process is iterative or post-prune, etc, are missing.\n\n4. No throughput or latency (bs = 1) results.\n\n5. Not really a weakness, but DSConv pruning is a relatively niche art with limited familiarity with the pruning community. This paper would benefit from making better contrasts with standard conv pruning to highlight its unique challenges.\n\n6. Is this DSPrune or DEPrune? The proposed method family has so many submodules named with acronyms, and it could really use a richer Table 1 for a more friendly reading experience.\n\n[1] Towards Compact CNNs via Collaborative Compression   \n[2] Pruning Depthwise Separable Convolutions for Extra Efficiency Gain of Lightweight Models   \n[3] Revisit Kernel Pruning with Lottery Regulated Grouped Convolutions   \n[4] Structured Sparsity in the NVIDIA Ampere Architecture and Applications in Search Engines", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s time and effort in providing constructive feedback on this manuscript. We will incorporate all of the suggestions into the final version of the paper. [m-\\#] means manuscript's reference.\n\n\n**Weakness 1)** \n\nAs the reviewer pointed out, it is important to compare our proposed scheme with recent prior work. In Table 4, we have compared our scheme with two of the most recent research papers [m-11 and m-48], both presented in 2024. If you have any recommendations for recent structured pruning methods, we would be eager to compare them with our DEPrune if circumstances allow.\n\n\n**Weakness 2)** \n\nWe appreciate your comment. The study you mentioned [2 and m-44] indeed applies an existing gradual pruning method to DSConv, so it is not accurate to state that there are no previous pruning techniques that target DW-Conv. We will revise this statement in the paper if given the opportunity to make revisions: \"there are hardly any previous pruning techniques that target depth-wise convolution (DW-conv)\" in line 4.\nWe also want to mention that the pruning method proposed in [2 and m-44] is focused on applying gradual pruning [m-55] to DSConv layers, which differs from our approach. \nThe research [2] focuses on removing filters to preserve structural pattern on DW-conv. Whereas our research considers GPU kernel and maintains structural pattern while more fine-grained pruning on DW-conv, to reduce accuracy loss.\n\n\n**Weakness 3)** \n\nThank you for your valuable advice. Based on your feedback, we will make the following improvements:\n1. We will rewrite the captions to provide a more detailed walkthrough, ensuring that readers can easily understand them.\n2. To enhance reproducibility, we will add a pseudo-code section to the appendix and plan to release our implementation in the future.\n3. We will include the ImageNet dataset information in the experiment section to clearly outline the experimental procedure:\n\n_[ImageNet experiment setting]\nlearning rate : 0.001, Fine-tuning epoch : 65epochs, learning rate epoch : dividing 10 per every 30epochs, Process : iterative pruning, Weight decay : 1e-4, Optimizer : SGD, Momentum : 0.9, Pre-trained model : PyTorch, All data are augmented with random crop and randomly horizontal flip._\n\n\n**Weakness 6)** \n\nThank you for pointing out our typo. Our proposed scheme is named DEPrune, and the mention of DSPrune is indeed a typo. We will thoroughly review our paper again and correct this and any other typographical errors in the revised version.\nWe also think that so many names can be very difficult to understand. We will double-check the Table 1 and writing of the paper so that it can provide simple and clear concept of DEPrune. As you suggested, creating a richer table will likely make it easier for readers to understand the paper.\n\n\n**Question 1)** \n\nWe believe there are various efficiency metrics, such as peak memory usage, computation throughput, and computation utilization, that can be measured in our experiments. Among these, we consider peak memory usage to be the most important efficiency metric, thus, we will include it in Table 4 in the revised paper. If the reviewer can suggest other efficiency metrics that should be included, we will measure them and incorporate the results in the revised version.\n\n\n**Question 2)** \n\nApplying DCP to general group convolution can be challenging. When the number of groups and channels differ, the rearrangement method for GPU processing varies, complicating the application of DCP. However, the enhanced method HSR, proposed in our paper, can be effectively applied in these scenarios. We believe that combining conventional structured pruning with our HSR could lead to improved inference time performance on the general group convolution.\n\n\n**Question 3)** \n\nThank you very much for your insightful feedback. The paper you mentioned, GKP [3], seems an excellent work. GKP divides weights into groups and applies vector-wise pruning to each group, effectively addressing the issue described in Section 4.2. As you pointed out, when GKP is applied, it does not impact the subsequent layer, thereby avoiding the problem discussed in Section 4.2. We appreciate you bringing this important paper to our attention, and we will consider incorporating this research (GKP) with our DEPrune as a potential future work.\n\n\n**Question 4)** \n\nAccording to Table 4 of paper [m-38], the extra overhead in total memory consumption due to zero-padding is approximately 0.3\\%. To assess the impact of DEPrune-BH, we measured and presented the peak memory usage of MobileNet-v2 before and after applying DEPrune-BH with a 50\\% pruning ratio, as shown in Author-Rebuttal-Table 1. Before applying DEPrune-BH, the peak memory usage is 7.22 MB, whereas after application, it decreases to 3.63 MB, representing a reduction of approximately 49.8\\%.\n\n\n**Question 5)** \n\nResearch on n:m sparsity is currently very active in the field of pruning. However, this sparsity approach has two major limitations: a lack of flexibility and the requirement for specialized hardware.\nFirst, it lacks flexibility because it is fixed at a 50\\% pruning ratio, specifically 2:4 pruning [m-35]. \nAs seen in Author-Rebuttal-Table.2, we conducted comparative experiments between NVIDIA's n:m sparsity and DEPrune on MobileNet-v2 using CIFAR-10.  \nAt the same pruning ratio of 50\\%, DEPrune-B achieves 0.31\\% higher accuracy than n:m sparsity. \nThis is because DEPrune-B achieves a 50\\% pruning ratio within $32 \\times k \\times k$ parameters, whereas n:m sparsity achieves a 50\\% pruning ratio within a parameter size of 4, which is $8 \\times k \\times k$ times smaller than DEPrune-B, leading to a accuracy drop [m-21].\nSecondly, in n:m sparsity, achieving optimal performance requires specialized hardware (NVIDIA A100) that can quickly handle index processing [m-35]. In contrast, our approach requires only a customized GPU kernel for processing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QbTPgbqNOz", "reviewer": "Reviewer_bd55", "summary": "This paper presents the Depth-wise Separable Convolution Pruning (DEPrune). DEPrune prunes point-wise convolution and depth-wise convolution (DW-conv). And it is optimized by considering and analyzing the computation of Depth-wise Separable Convolution on GPU. Experimental results validate the speedup of DSPrune.", "questions": "1.\tWhat does “X” and “O” mean in Figure 1?\n2.\tThe lower parts of Figure 1 seem misleading, which do not correspond to the DW-conv example in the upper half.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe proposed method supports both point-wise convolution and depth-wise convolution pruning.\n2.\tThe performance of the proposed method seems significant.", "weaknesses": "1.\tSome references are incomplete, for example, [23][39].\n2.\tThe proposed scheme is only evaluated on MobileNet-V2, MobileNet-V3, and EfficientNet-B0. How does this method work for other NN models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time you have taken to review our work and for the constructive feedback. If the reviewer allows us to revise this paper, we will try our best to improve the quality of this paper.\n\n\n**Weakness 1) [Some references are incomplete, for example, [23][39].]**\n\nWe greatly appreciate your guidance on this issue. We will revise and incorporate your suggestions as follows. Additionally, we will double-check if other references are incomplete and address them accordingly.\n\n\"[23] Krizhevsky, G. Hinton, et al. Learning Multiple Layers of Features from Tiny Images. Technical report, University of Toronto, 2009.\"\n\n\"[39] P. K. Rao and S. Chatterjee. Weight Pruning-UNet: Weight Pruning UNet with Depth-wise Separable Convolutions for Semantic Segmentation of Kidney Tumors. Research Square 2021\"\n\n\n**Weakness 2) [The proposed scheme is only evaluated on MobileNet-V2, MobileNet-V3, and EfficientNet-B0. How does this method work for other NN models?]**\n\nAs the reviewer's suggestion, we conducted additional experiments with the ConvNeXt-Base model [1].  In Author-Rebuttal-Table.3, DEPrune-BH achieves a 0.46\\% higher Top-1 accuracy compared to FP even with an additional 20\\% pruning in DW-conv. DEPrune-BH is up to 3.3 times faster than the ConvNeXt-Base while achieving a Top-1 accuracy drop within 1.5\\%.\n\n[1] A ConvNet for the 2020s, CVPR 2022\n\n\n**Question 1) [What does “X” and “O” mean in Figure 1?]**\n\nWe agree with the reviewer’s comment that the caption in Figure 1 is not clearly presented. The X and O symbols indicate the absence and presence of corresponding characteristics for each method.\nIn the case of (a) Structured Pruning, this approach forms well-structured data patterns that can benefit performance when executed on GPUs. However, it is not a fine-grained method, as the pruning unit size can extend up to a 2D matrix, which is significantly larger than the pruning unit in DEPrune, the method we propose. Therefore, in the case of (b) DEPrune, it is marked as O for both fine-grained and structured characteristics. We will include these details in the final version of the paper’s caption to prevent any misunderstandings for readers.\n\n\n**Question 2) [The lower parts of Figure 1 seem misleading, which do not correspond to the DW-conv example in the upper half.]**\n\nWe appreciate the reviewer’s valuable comment that points out the statements that could possibly mislead the readers. The top illustration in Figure 1 depicts a typical depth-wise convolution. The bottom illustration shows the depth-wise convolution rearranged to be more favorable for GPU processing. On the left is an image with (a) structured pruning applied, and on the right is after applying (b) DEPrune. Thus, the relationship between the top and bottom illustrations indicates before and after the Diagonalwise Refactorization is applied, and the figure shows the arrangement of the weight. We will enhance the caption with these details to ensure there is no misunderstanding for readers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ufvUlTQJOB", "reviewer": "Reviewer_VAUi", "summary": "This paper proposed a pruning method target at the previously neglected Depth-wise Convolution layers which are widely used in vision models. This work is based on the Diagonal-wise Refactorization (DR) computation strategy used in GPU, and pruning one weight point in depth-wise convolution means convert the corresponding column into zero vector. To accelerate the inference of pruned models, the authors proposed to enhance the GPU performance with two hardware friendly methods: BWT and HSR, which achieves the load balance and align the memory access with GPU tile size. The evaluation result shows the proposed framework brings significant speedup in MobileNet and EfficientNet.", "questions": "See weakness.", "limitations": "See weakness.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The pruning method is based on the DR computation strategy, which provides the opportunity to convert the point pruning in depth-wise convolution layers into masking the corresponding columns as zeros.\n2. To accelerate the inference processing, the proposed BWT and HSR method achieves loading balance among GPU kernels and align the computation with tile size, speeding up the GPU processing.\n3. The evaluation result breakdowns the pruning method, BWT and HSR, demonstrating the contribution of each part.", "weaknesses": "1. The overhead of pruning method is not analyzed. How much time does it need to decide pruning points, loading balance and the parameters in HSR?\n2. Comparing with the gain in pruning ratio, the accuracy drop seems not negligible. Can we modify the algorithm to set the HSR constraints to align the memory access and computation with tile size in pruning phase, without need of recalibration? Does it help with accuracy?\n3. Can you provide the evaluation on edge GPU?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your support of our paper and the valuable feedback. We respond to your reviews as follows.[m-#] means manuscript's reference.\n\n**Weakness 1) [The overhead of pruning method is not analyzed. How much time does it need to decide pruning points, loading balance and the parameters in HSR?]**\n\nWe would like to clarify few things that might not have been clearly expressed in the paper. The pruning, load balancing, and HSR mentioned are all required only during the pruning process (offline). Pre-processing (Pruning and fine-tuning) is conducted offline, while model inference is performed online. The processes you mentioned are overheads in the offline phase and do not affect model inference time. In Author-Rebuttal-Table.4, still this offline overhead accounts for about 0.6\\% of the total pre-processing time in MobileNet-v3-small on ImageNet. The proportion of this overhead becomes even smaller if the fine-tuning epoch increases. We will provide detailed figures based on MobileNet-v3-small to Author-Rebuttal-Table.4.\n\n\n**Weakness 2) [Comparing with the gain in pruning ratio, the accuracy drop seems not negligible. Can we modify the algorithm to set the HSR constraints to align the memory access and computation with tile size in pruning phase, without need of recalibration? Does it help with accuracy?]**\n\nYes, the algorithm can be modified as you suggested. However, the absence of recalibration does not significantly affect accuracy.  Without the recalibration process, users must set the pruning ratio to match the recalibration results by considering the GPU tile size and each layer's size.  Recalibration is an algorithm that automates this process. The main reason there is no significant change in accuracy is as follows: In Figure 7-(C), you can see that DEPrune-BH performs fine-tuning after step 4, which is the entire pruning setting process. Therefore, whether recalibration is performed or users manually set the pruning ratio by considering the tile size, both are processes before fine-tuning, resulting in almost no change in accuracy.\n\n\n\n**Weakness 3) [Can you provide the evaluation on edge GPU?]**\n\nCertainly. We conducted additional experiments on an edge GPU and measured the inference time before and after pruning for MobileNet-v2. The edge GPU that we used for the experiment is NVIDIA Jetson Orin Nano 8GB. In Author-Rebuttal-Table.5, the DEPrune-BH pruned model inference time shows a 2.48 times speedup in inference time than baseline (unpruned model) on edge GPU.  Compared to GFS [m-50] pruned model, DEPrune-BH pruned model is approximately 1.62 times faster. From an accuracy perspective, there is no change because the weights are the same as those on the laptop GPU.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the reviewers' time and effort in providing constructive and valuable feedback on this manuscript.

In response to the reviewers' questions, we have conducted evaluations:

1. Reviewer CSWm's Question 4: We provided peak memory usage data presented in Author-Rebuttal-Table 1.

2. Reviewer CSWm's Question 5: We provided n:m sparsity data presented in Author-Rebuttal-Table 2.

3. Reviewer bd55's Weakness 2: We provided ConvNeXt-Base data presented in Author-Rebuttal-Table 3.

4. Reviewer VAUi's Weakness 1: We provided pre-processing time results in Author-Rebuttal-Table 4.
 
5. Reviewer VAUi's Weakness 3: We have provided edge GPU exeperiment results in Author-Rebuttal-Table 5.

Meta-review: The authors propose a method of pruning depthwise-seperable convolution layers, widely found in efficient CNN architectures for vision tasks, such as EfficientNet. The proposed pruning method for depthwise seperable layers, while being a fine-grained approach, achieves real-world speedups on GPUs using a hardware-aware structured pruning approach. The results show significant real-world speedup of already efficient CNNs on GPUs, e.g. approx. 4x speedup on MobileNet-V3-Small.

The reviewer's concerns mostly centred around over-claims in the paper, and missing references, which were well-addressed overall by the authors in rebuttal. On the "the first pruning implementation on point wise/depthwise convolution" claim, I would also point out that any existing pruning literature using efficientnet/mobilenet/etc models, of which there is much, would also contradict this statement and I would recommend that they re-word the claim to make it more specific to their actual contributions (i.e. designed specifically to improve performance for these types of layers).

Overall the reviewers agreed that the work is well-written, well-motivated work with significant results.
Decision: Accept (poster)