Title: DEPrune: Depth-wise Separable Convolution Pruning for Maximizing GPU Parallelism
Venue: NeurIPS 2024
Accepted: yes
Keywords: Pruning

Depth-wise Separable Convolution

GPU
Abstract: Depth-wise Separable Convolution (DSConv) has a powerful representation even with fewer parameters and computation, leading to its adoption by almost all of the state-of-the-art CNN models. 
DSConv models are already compact making it hard to apply pruning, and there are few  previous pruning techniques that target depth-wise convolution (DW-conv).
In this paper, we present Depth-wise Separable Convolution Pruning (DEPrune), a novel pruning method applied to both point-wise and depth-wise convolutions. 
DEPrune is optimized by analyzing the computation of DSConv on GPUs.
DEPrune employs a fine-grained pruning approach, yet it achieves the structured sparsity typically absent in fine-grained pruning, enabling practical hardware acceleration. 
Moreover, this method maintains a high pruning ratio without causing any accuracy drop.
We additionally represent techniques that further enhance DEPrune performance: 1) balanced workload tuning (BWT), and 2) hardware-aware sparsity recalibration (HSR).
Experiment results show that DEPrune achieves up to $3.74\times$ practical speedup in DSConv inference on GPUs while maintaining the accuracy of EfficientNet-B0 on ImageNet.

DEPrune: Depth-wise Separable Convolution Pruning for Maximizing GPU Parallelism

Abstract

Depth-wise Separable Convolution (DSConv) has a powerful representation even with fewer parameters and computation, leading to its adoption by almost all of the state-of-the-art CNN models. DSConv models are already compact making it hard to apply pruning, and there are few previous pruning techniques that target depth-wise convolution (DW-conv). In this paper, we present Depth-wise Separable Convolution Pruning (DEPrune), a novel pruning method applied to both point-wise and depth-wise convolutions. DEPrune is optimized by analyzing the computation of DSConv on GPUs. DEPrune employs a fine-grained pruning approach, yet it achieves the structured sparsity typically absent in fine-grained pruning, enabling practical hardware acceleration. Moreover, this method maintains a high pruning ratio without causing any accuracy drop. We additionally represent techniques that further enhance DEPrune performance: 1) balanced workload tuning (BWT), and 2) hardware-aware sparsity recalibration (HSR). Experiment results show that DEPrune achieves up to 3 . 74 × practical speedup in DSConv inference on GPUs while maintaining the accuracy of EfficientNet-B0 on ImageNet.

1 Introduction

In computer vision tasks, Convolutional Neural Networks (CNNs) have dramatically gained parameters and computation [54] to solve complex and varied tasks [8]. Such massive computation and memory footprint poses challenges in environments with limited hardware resources, such as mobile devices. Many research efforts have been made to address such problems, and two techniques have been most effective: Depth-wise Separable Convolution (DSConv) [6] and DNN pruning [17].

DSConv [6, 43, 45] is composed of Depth-wise Convolution (DW-conv) and Point-wise Convolution (PW-conv), allowing it to have a similar representation power to traditional CNNs that use standard convolution, even with fewer parameters and computation [3]. Therefore, modern CNNs primarily adopt DSConv when designing models [9, 40, 46, 48].

DNN pruning eliminates redundant weight parameters without compromising representation power. Weight pruning [18] brings a significant pruning ratio ( PR ) due to the fine-grained approach but rarely reduces inference time compared to the unpruned model because of index computation overhead [54]. In contrast, structured pruning [16, 30, 32, 21, 38] is a coarse-grained approach that is GPU-friendly and leads to a practical reduction in inference time.

(a) Structured Pruning

(b) DEPrune (DCP)[ours]

Weight

Weight

Depth-wise

Convolution

(DW-conv)

Input

Output

Input

Output

Diagonal-wise

rearrange

Refactorization (DR)

zerocolumnvector

sub-GEMM#1

sub-GEMM#1(pruned)

：

Rearranged

DW-conv

weight

sub-GEMM#2

sub-GEMM#2（pruned）

using DR

Feature:

Fine-grainedmethod X,Structured datapattern O

Fine-grained method O,Structureddatapattern O

: unpruned

pruned

:zero padding

In DSConv, despite the DW-conv has only about 1% of the parameters, it spends over 82% of the overall inference time [41]. As a result, reducing the computation time of DW-conv can inherently impact the overall network execution time, and employing pruning on DW-conv can provide an effective solution to accelerate DSConv. Applying existing structured pruning [27, 20, 37] to PW-conv is not difficult, because PW-conv has the same GPU operation characteristic as standard convolution. Conversely, DW-conv has two challenges in applying previous pruning methods. First, DW-conv has particularly fewer parameters, so applying coarse-grained pruning that creates a hardware-friendly structured data pattern causes significant accuracy loss. Second, the operation of DW-conv on GPU underutilizes the parallelism since the input unit is smaller than the operation unit, thus applying previous pruning methods is useless in this condition. Therefore, we propose Depth-wise Separable Convolution Pruning (DEPrune) , a hardware-aware pruning approach specialized for DW-conv for fast and memory-efficient DSConv.

First, to address the aforementioned challenges: accuracy loss and underutilization, we analyze the operation of DW-conv on a widely used GPU. On GPUs, DW-conv computes by being transformed into multiple-GEMV (GEneral Matrix-Vector Multiplication) [41], and this structure does not fully utilize the GPU parallelism. Thus, for efficient operation on GPU, previous work applies Diagonalwise Refactorization (DR) [41]. DR is a method of rearranging DW-conv to multiple sub-GEMM (GEneral Matrix-Matrix Multiplication) operations to maximize GPU parallelism (Fig. 1). DR places the weights diagonally and zero padding the rest. Therefore, if one non-zero weight is removed, all elements on the same column line have zero value, so the corresponding line is all zero vector (Fig. 1). At this point, we apply fine-grained pruning on DW-conv rearranged by DR, as we call the Depth-wise Convolution Pruning (DCP) (Sec. 4.1). DCP's fine-grained approach provides novel PR and since most of the sub-GEMM is zero-padded, it brings regular sparsity even with finegrained pruning. This hardware-friendly format results in inference speedup without representation power loss. DEPrune also applies conventional structured pruning to PW-conv; however, taking into account the computational significance of the DSConv model, we selectively prune PW-conv layers to maximize the pruning ratio without sacrificing accuracy.

Second, for DEPrune enhancement we consider the overall operation flow of DSConv to optimize GPU utilization. When DCP is applied, the PR is different for each sub-GEMM, which executes in different processing units. This results in a workload imbalance problem between processing units which directs to GPU under-utilization. The total execution time is set to the longest GEMM, thereby other idle processing units are forced to wait until the longest GEMM finishes. To solve this problem, we propose a Balanced Workload Tuning (BWT) that sets the same target PR for each sub-GEMM when applying DCP (Sec. 5.1). Our DEPrune applies existing structured pruning [37, 27, 20] on PW-conv, and this approach avoids workload imbalance problem.

Lastly, for DEPrune enhancement we found that in addition to balancing PR between sub-GEMMs, adjusting the pruning ratio proportional to the GPU execution unit could further lead to significant speed improvements. Due to our technique's structured data format, theoretically, the acceleration should increase proportionally as the pruning ratio increases. However, because GPU operates in a specific execution unit, the computational workload should scale with the unit of execution to maximize the acceleration effect. Considering this unit operation, we additionally determine negligible weight parameters for more pruning, and this results in an additional significant speed improvement of 3 -16% for DW-conv and also PW-conv without any accuracy loss. We call this Hardware-aware Sparsity Recalibration (HSR) , an algorithm that recalibrates the appropriate target PR for DW-conv and PW-conv considering the execution unit of the GPU (Sec. 5.2).

To further optimize DSConv, various techniques are considered, which led to a diverse and somewhat complex set of terms being used throughout the paper. Therefore we provide Table 1 which summarizes the structure and terminology of our DEPrune.

(a) Depth-wise Convolution (DW-conv)

(c)Diagonal-wiseRefactorization

GEMV#1

Mchannels

weight

Mchannels

input

Mchannels

output

In×Iw

-

Sub-GEMMi

#1

GEMV#32

.··

GEMV#1

32

32×kn×kw

GEMV#2

multiGEMM

im2col

GEMV#3

Sub-GEMM

··.

#2

GEMV#1

口

GEMV#M

···

GEMV#2

口

M

M

.··

.··.

M×knxkw

grouping

1

GEMV#M

口

(GEneral Matrix Multiplication)

GEMM

Sub-GEMM

#G

knXkw

(b) Channel-by-Channel

2 Preliminary

Prerequisites DW-conv's weight filter is 3D tensor. Given l th DW-conv layer as D ( l ) ∈ R M × k h × k w , where M , k h , and k w are the number of channels, height, and width of the filters, respectively. I h , and I w are the height and width of the input, respectively.

Channel-by-Channel As shown in Fig. 2-(a), DW-conv is composed of 3D input ( R M × I h × I w ), 3D Weight ( R M × k h × k w ), each with M channels performing independent 2D convolution operations. GPU rearranges standard convolution to GEMM, using im2col [2, 5] to enable data reuse. Similarly, major deep learning frameworks (e.g., Caffe, PyTorch, MXNet, and TensorFlow) rearrange DW-conv to M multiple GEMV operations, using Channel-by-Channel (Fig. 2-(b)). GEMV consists of a

weight vector of size k h × k w and an input matrix of size ( k h × k w ) × ( I h × I w ) . However, this approach suffers from the limitation of weight vector size being too small ( 9 or 25 ) to fully utilize the GPU's processing units effectively.

Diagonal-wise Refactorization (DR) To address under-utilization, Diagonal-wise Refactorization (DR) [41] arranges the weight vectors of the GEMVs diagonally and sequentially places the input matrix (Fig. 2-(c)). Next, zero padding is added to the empty spaces to create a complete dense GEMM,which consists of M × ( M × k h × k w ) weight matrix and an ( M × k h × k w ) × ( I h × I w ) input matrix. However, the rearranged GEMM is excessively large ( M × k h × k w ) , so this operation requires significant additional computation on the GPU such as tiling [41]. Therefore, DR further divides this dense GEMM into smaller sub-GEMMs of a certain size. When executing matrix multiplication on GPUs, grouping with a size of 32 channels is found to be the most efficient, resulting in a total of G sub-GEMM operations ( G = M 32 ). Thus each sub-GEMM is composed of a 32 × (32 × k h × k w ) weight matrix and a (32 × k h × k w ) × ( I h × I w ) input matrix. This approach allows for highly optimized GPU execution using specialized cuDNN libraries [5].

3 Related Works

3.1 Hardware-aware Pruning

Among the previous DNN pruning techniques, the following three methods consider hardware characteristics to reduce inference time: structured pruning, balanced pruning, and block pruning. Structured pruning [32, 16, 53] determines redundancy at the vector level for pruning, thereby creating a regular sparsity. This structured data pattern requires almost no additional index computation on GPU, making it effective in reducing inference time [54]. Block pruning [15, 36, 49]that considers the tiling technique [5], applies structural pruning at the small matrix level and has less representation power loss at the same PR compared to structured pruning. Balanced pruning [58, 51, 31, 23, 37] is a technique that divides the weight into consistent ranges and assigns an equal PR to each segment, ensuring a workload balanced characteristic. Most balanced pruning achieves approximately a 2 × speedup when a specific GPU with a dedicated accelerator (e.g. sparse tensor core) is used at only 50% PR [35]. These pruning methods are based on the optimization technique of lowering for standard convolutions. Therefore, such pruning methods are difficult to apply to DW-conv. However, our DEPrune considers the hardware computation of DW-conv, enabling performance improvement.

3.2 Optimizations for DSConv

Since DSConv operates differently from standard convolution, continuous research is rapidly ongoing for the optimization of DSConv through dedicated software and hardware optimization.

On the pruning side, Multi-stage gradual pruning [47] prune the filters on DSConv using gradual pruning principle [57]. Probability-based channel pruning [56] considers batch normalization when pruning in DSConv and requires little fine-tuning. WP-UNet [42] utilizes fine-grained pruning in DSConv to merely reduce parameters without considering speedup. The filter pruning [34] sorts and prunes the filters according to their variance in each DW-conv. However, existing studies either only apply pruning on PW-conv or do not lead to noticeable substantial inference speed improvement even if pruning is also applied to DW-conv.

On the software side, DepthShrinker [10] removes non-linear activation functions after training and merges consecutive linear operations into a single dense operation to maximize hardware efficiency without compromising accuracy.

On the hardware side, GPU optimization [33] proposes a dynamic tile size scheme for GPUs to improve GPU utilization and hide memory access latency in DW-conv. The method [55] suggests loop rescheduling and register tiling on DW-conv, because when executing DW-conv on the parallel processor, traffic overload occurs between the cache, memory, and register. Diagonal-wise Refactorization (DR) [41] maximizes the parallelism of the GPU by proposing a rearrange method that combines all filters of DW-conv into a multi GEMM.

78

Baseline Top-1

GEMV execution Time (us)

32

**GEMV**

Vector:1*vectorlength

Imbalance overhead (%)

[minimum PR-target PR]

12

10

Target PR 40% Target PR 50%  Target PR 60%

%9'6

Top-1 accuracy (%)

77

1.0%

28

Matrix:vectorlength*12544

8

76

-1.5%

(DWconv k : 3)

Vectorlength:1~9

6

4

75

24

2

74

O-Channel Pruning--DCP[ours]

22

0

b

73

40

50

60

70

20

75

65

55

45

35

25

15

5

DW_Layer_

DW_

W_Layer_

DW_

DW

_Layer

DW

V_Layer_

_Layer_

MO

_Layer_

DW

_Layer_

DW

DW_

Layer_Average

Pruning ratio (%)

Weightvectorlength

Depth-wiseConvolutionLayer

(a)

(b)

(c)

4 Proposed Method: DEPrune

4.1 DCP: Depth-wise Convolution Pruning

(a) Motivation 1: Channel pruning on DW-conv has a large pruning unit size problem As shown in Fig. 2-(b), DW-conv generates a multi-GEMV format for each channel, on GPUs. DWconv can also achieve structured data format, by evaluating the significance of each GEMV and eliminating an unnecessary weight vector of GEMV. Nevertheless, when compared to the 4D tensor weight of the standard convolution, the DW-conv weight is a 3D tensor ( R M × k h × k w ), notably fewer parameters. Consequently, eliminating a single channel ( R 1 × k h × k w ) from DW-conv can greatly diminish its representation power. As shown in Fig. 3-(a), when pruning is done channel-wise, there's a representation power loss of 1 . 66% compared to the unpruned model even at just a 40% PR (EfficientNet-B0 on ImageNet). This indicates that channel-wise pruning on DW-conv is not an appropriate choice.

(b) Motivation 2: Hardware-unfriendly problem of weight pruning without DR Weight pruning experiences the least representation power loss among DNN pruning techniques with the highest pruning ratio. When applying weight pruning to the multiple GEMVs of DW-conv, the representation power loss due to increased PR is much less than the previously mentioned channel pruning. Looking at Fig. 3-(a) as our DCP similar to weight pruning, there are very minor representation power losses of 0 . 94% and 1 . 15% at 50% and 70% PR s, respectively. However, weight pruning without considering DR does not result in practical speedup from pruning. As shown in Fig. 2-(b), the vector size of DW-conv GEMV is k h × k w (e.g., 9 or 25). Since this is smaller than the GPU's tile size (32), there is almost no change in inference time (Fig. 3-(b)) since GEMV underutilizes processing units of GPU.

(c) Method: DCP We propose Depth-wise Convolution Pruning (DCP) to address the above two issues. We discover that weight pruning after DR can even achieve a structured sparsity in DW-conv with high PR , and making large matrix multiplication fully utilizes GPU parallelism. As shown in Fig. 4, first, we take the weight matrix rearranged in the form of matrix multiplication by DR. The height of the weight matrix is M , and the width is M × k h × k w . As shown in Fig. 4, the unpruned values in the weight matrix are placed diagonally, while the rest are zero-padded. Second, we sort the unpruned values in ascending order and select the threshold value that corresponds to the target pruning ratio. When calculating the target pruning ratio, zero-padded values are not considered. Last, for each unpruned value, if it is smaller than the threshold, we change it to 0 (i.e., magnitude pruning [18]). Since the other values in the same column are already zero values, the column vector becomes a zero column vector, which is hardware-friendly.

if (target pruning ratio ==50%):

1.Rearranged Weight Matrix (using DR)

2.Ranking (ascending order)

Top1~12→Prune

4.Pruning(fine-grained)

1|24152

M

14221011

17576

32416

3.Calculate

threshold

2091321

8181912

M×kxkw

:unpruned

:zero padding

: pruned

zerocolumnvector

(structured)

4.2 Methodology for Determining which PW-conv Layer to Prune

When filter pruning is applied to PW-conv, the parameters of the subsequent layers are removed with the same sparsity. DSConv has the following structure: PW-conv1 → DW-conv → PW-conv2. In DSConv, if PW-conv1 is filter pruned, the parameters of DW-conv are also removed at the channel level. The existing PW-conv pruning methods prune all PW-conv layers of DSConv, inadvertently leading to prune DW-conv layers as well. However, the parameters of DW-conv are only 1.34% of those in PW-conv [41], so each weight element is more sensitive to accuracy, thus for DW-Conv, rather than channel pruning, a more fine-grained pruning is necessary. Therefore, our DEPrune does not directly prune all PW-conv layers. DEPrune applies fine-grained pruning directly to DW-conv and does not prune PW-conv1 directly. We only apply filter pruning to PW-conv2. Pruning only PW-conv2, removes only the parameters in the subsequent DSConv's PW-conv1, which is less sensitive to accuracy drop. Thus, DEPrune effectively prune all the layers of DSConv with high PR and representation power.

5 Enhance DEPrune

We propose the following two techniques to enhance DEPrune performance: BWT and HSR.

DW-conv weight by DR rearrange

(a) Depth-wise Convolution Pruning (DCP)

(b)DCP+BWT

Sub-GEMM

#1

Pruning

Ratio

Pruning

Ratio

:33.3%

:25.0%

(8/24)

(6/24)

Sub-GEMM

Pruning

XWMW:

Pruning

#2

: 16.7%

Ratio

EMWNIK

: 25.0%

Ratio

(4/24)

(6/24)

If (|weight value|< threshold):

prune weight value

zero column vector

pruning unit size:R

Additional element removal for workload balance

：

Balanced range

(32×K×K)

:unpruned : zero padding : pruned

Structured DataPattern (Hardwarefriendly)

Balanced (Hardwarefriendly)

5.1 DEPrune-B

(a) Motivation: Imbalance overhead problem of DCP GPUs allocate operations of a certain size to streaming multiprocessors (SMs) for massively parallel processing. Therefore, DW-conv's multiple sub-GEMMs are also assigned to SMs, respectively. However, when applying DCP on DW-conv, the pruning ratio ( PR ) of sub-GEMMs may differ, given the varying importance of weights between sub-GEMMs. In that case, the execution time varies for each sub-GEMM due to the difference in PR . This results in a workload imbalance problem in that the other SMs of the GPU have to wait until the SM with the lowest PR finishes. The acceleration effect of DCP is then determined by the minimum sub-GEMM PR , not by the layer target PR . Referring to Fig. 3-(c), the difference between the minimum sub-GEMM PR and the layer target PR is compared for each layer of EfficientNet-B0. In DW Layer 13, when the layer target PR is 60% , the minimum sub-GEMM PR is 50 . 4% , which varies up to 9 . 6% , which indicates that it decelerates execution by the amount specified.

(b) Method: Balanced Workload Tuning (BWT) To address the workload imbalance issue of DCP, we propose a DW-conv-specific Workload Balanced Technique that takes into account the operation structure of DW-conv (Fig. 5). DW-conv is a dense matrix where non-zero values are arranged diagonally due to DR, while the remainder consists of zero values. We group all non-zero values within sub-GEMM, which we call a balanced range as illustrated in Fig. 5-(b). Within each balanced range, we rank weight elements with redundancy and systematically prune the lower-ranked elements until the target PR is reached. As every sub-GEMM achieves the same target PR like Fig. 5-(b), this resolves the workload imbalance issue associated with DCP. Since DCP is fine-grained pruning (pruning unit size: R ), the representation power loss due to additional BWT is almost negligible. A detailed analysis related to this is in Sec. 6.1.

InferenceTime

Speed Up (%)

10

8

7.27

Speed Up=

(DCP-BHtime)-(DCP-B time)

(DCP-Btime)

×100(%)

Inference Time(us)

400

300

flatten(no speedup)

Stage2Dwconv

Stage1DWconv

6

6.37

Stage5PWconv

4

200

Group size:32

2

100

0

0

1

1

8

DWconv_1

DWconv_2

DWconv_4

DWconv_

(a)

DWconv_11

DWconv_13

Average

ThenumberofPrunedvector

(b)

5.2 DEPrune-BH

(a) Motivation: Unaligned problem As shown in Fig. 7-(a), to maximize parallelism, GPUs divide GEMM operations into small tiles. In general, the size of the tile depends on the hardware specification of GPUs, but it is usually a multiple of 32 [14]. However, if the width of the unpruned weight matrix in Fig. 7-(a) is not a multiple of 32 , some parts of the weight tiles are empty. This can cause an unaligned memory access problem on GPUs [11, 13]. In Fig. 6-(b), the inference time does not decrease linearly with an increase in the size of the pruned vector. Whenever the number of pruned vectors increases by 32 , the inference time decreases significantly like a step function graph. In DW-conv of Stage 2, the inference time decreases by 7% for each removal of only one tile. Thus, by removing a few additional weight vectors for aligned memory access, we can reduce the inference time by 7% if we align the number of pruned vectors with a multiple of 32 (Fig. 7-(b)).

(b) Method: Hardware-aware Sparsity Recalibration (HSR) We propose Hardware-aware Sparsity Recalibration (HSR) to solve the unaligned memory access problem and enhance DCP-B. As shown in Fig. 7-(c), DCP-B with HSR operates in the following four steps. The first step, DCP-B is applied to DW-conv. The second step, we measure two essential factors ( α and ϵ ) within the DCP-B model. (1) α : We measure the speedup obtained by solving the unaligned problem per layer. (2) ϵ : We count the number of unpruned vectors of the unaligned tile matrix for each layer. We refer the result obtained by dividing the two parameters, α and ϵ , for each layer as β . The β refers to the size of speed obtained by removing one overflowed vector. The third step, the β values of all layers are ranked by comparing them with each other. The last step, the layer with the β value of the top 50% is additionally removed as much as it overflows. The additional removed column vector consists of one non-zero value and zero-padded elements. Thus, there is no significant side effect on the representation power. On the other hand, the layer with the β value of the bottom 50% additionally recovers as much as it is unaligned. The reason why the criteria for recovery and removal of HSR are set to 50% is to maintain the total target PR .

(a)UnalignedproblemonGPU

(b)Hardware-aware Sparsity Recalibration (HSR)

(c)DCP-BH (DCP-B+HSR)

Tile size

Under-unaligned

Step 1.Pre-prune DCP-B

Unaligned

Matrix

andui

Unpruned Model

Recover prunedvector

Step 2. Measurement β=

(idle)

tile

tile

tile

tile

Memory Access Time: 2

Over-unaligned (t)

Step 3.Ranking β values

#1

#2

#2

Unpruned

Weight

ile

il

#5

Prune unpruned vector

Step4.SparsityRecalibration&Prune

βBottom 50%layer=Recover)

（βTop 50% layer =Prune,

Tile

Matrix

tile

(reduce idle core for speedup)

size

#8

Memory Access Time: 2

(α = /-th layer speedup due to up-unaligned removal &

Unpruned weight valuePruning unit size

Pruned weight value

Memory Access Time: 3

& = /-th layer the number of up-unaligned vectors)

6 Experiments

We assess the effectiveness of DEPrune using ImageNet [8] and CIFAR-10 [25]. For the validation of image classification, we assess our method with CNN models using DSConv: MobileNet-V2 [43], EfficientNet-B0 [45], and MobileNet-V3 [22].

Experiment setting on ImageNet We utilize pre-trained CNN models sourced from the Pytorch framework [39]. We perform fine-tuning with only 65 epochs after conducting pruning methods. We set a batch size of 256 . We use SGD optimizer with the weight decay, 1 × 10 -4 , and the momentum as 0 . 9 for fine-tuning. The initial learning rate is set to 0 . 001 and divided by 10 every 30 epoch. All data are augmented with random cropping and horizontal flipping. We evaluate DEPrune using NVIDIA RTX 2080 Ti GPUs [1]. We measured the inference time using NVIDIA CUTLASS [24]. We set the batch size to 32 to measure inference time.

6.1 Effect of BWT (DEPrune vs. DEPrune-B)

We analyze the changes in accuracy and speedup resulting from applying the BWT to DEPrune (Table 2). DEPrune has varying pruning ratios among sub-GEMMs, causing the overall speed to be dictated by the sub-GEMM with the smallest pruning ratio. In MobileNet-V2, the smallest sub-GEMM pruning ratio of DEPrune is 71%, as described in the Table 2. Therefore, DEPrune-B in MobileNet-V2 is 21.8% (2.75 × → 3.35 × ) faster in inference time than DEPrune. In MobileNetV3-Small, DEPrune-B achieves a 31.2% (3.88 × → 5.09 × ) improvement in inference time over DEPrune due to BWT. Since the balanced range of DEPrune-B is significantly large at 32 × k h × k w , DEPrune-B has an accuracy drop of within 0.1% than DEPrune across representative models.

6.2 Effect of HSR (DEPrune-B vs. DEPrune-BH)

We analyze the changes in accuracy and speedup resulting from the application of the HSR technique to DEPrune-B (Table 3). Since GPUs process operations and memory access in tile units, the actual speed of the GPU does not decrease linearly with the pruning ratio but rather decreases in a step-wise manner, as shown in Fig. 6-(b). By adjusting the pruning ratio to fit the tile size, the DW-conv layer can

achieve an average inference time speedup of 6.37%, as illustrated in Fig. 6-(a). According to Table 3, applying HSR to DEPrune-B shows almost no difference in accuracy compared to not applying HSR within 0.16%. Specifically, the accuracy difference is only 0.01% on MobileNet-V3-Small. For DEPrune-BH, all models have nearly identical numbers of over-aligned and under-aligned layers. DEPrune-BH achieves 6.2% (5.79 × → 6.15 × ) inference time speedup compared to DEPrune-B on EfficientNet-B0. Additionally, HSR can be applied to PW-conv layers as well.

6.3 Comparison with Structured Pruning

In Table 4, we conduct experiments comparing DEPrune with the latest structured pruning methods across four models. On MobileNet-V2, our DEPrune-BH reduces approximately 26.7% more FLOPs compared to RLAL, while exhibiting a 0.2% smaller accuracy drop. GFS removes up to 42.8% of DSConv parameters, resulting in an accuracy drop exceeding 3%. In contrast, DEPrune-BH eliminates 75.1% and 64.8% of parameters in DW-conv and PW-conv, respectively, with an accuracy drop within 1%. On EfficientNet-B0, while other methods prune around 30% of DW-conv, our method prunes 84.7% with only a 0.8% accuracy drop. On MobileNet-V3-Small and MobileNet-V3-Large, DEPrune-BH achieves inference times 3.33 times and 3.32 times faster than GFS and FPGM, with accuracy drops of 1.1% and 0.6% less, respectively.

6.4 Discussion: Various Pruning on PW-conv

We apply four structured pruning techniques to PW-conv layers to measure the changes in accuracy (See Table 5). When applying ℓ 1 -norm pruning and ℓ 2 -norm pruning to PW-conv layers, the accuracy difference is within 0.06% for all models except EfficientNet-B0. According to the FP paper [27], there is minimal difference between ℓ 1 -norm and ℓ 2 -norm pruning, and this similarity is also observed in the case of DEPrune-BH. Conversely, on EfficientNet-B0, FPGM [20] which uses geometric median achieves 0.33%, and 0.09% higher accuracy compared to ℓ 1 -norm and ℓ 2 -norm pruning, respectively. BCBP [37] is a block-wise pruning method that can be applied PW-conv. However, applying BCBP to PW-conv following DW-conv eliminates some parameters of DW-conv. Therefore,

when applying BCBP to PW-conv the accuracy drops on all models described in Table 5 compared to FP and FPGM.

7 Conclusion

In this work, we propose a new Depth-wise Separable Convolution Pruning (DEPrune) method tailored for DW-conv to reduce DSConv inference time and fully leverage GPU features. Extensive experimental results on the ImageNet dataset demonstrate that DEPrune effectively preserves representation power, even with higher PR than structured pruning, achieving a regular sparsity. Moreover, two techniques, BWT and HSR, further enhance DEPrune's capabilities. With these combined features, DEPrune-BH achieves substantial GPU speed gain of up to 4 . 1 × on MobileNet-V3-Small.

References

J. Burgess. Rtx on-the nvidia turing gpu. IEEE Micro , 40(2):36-44, 2020.

K. Chellapilla, S. Puri, and P. Simard. High performance convolutional neural networks for document processing. In Tenth international workshop on frontiers in handwriting recognition . Suvisoft, 2006.

W. Chen, Z. Wang, S. Li, Z. Yu, and H. Li. Accelerating compact convolutional neural networks with multi-threaded data streaming. In 2019 IEEE Computer Society Annual Symposium on VLSI (ISVLSI) , pages 519-522. IEEE, 2019.

Y. Chen, X. Dai, D. Chen, M. Liu, X. Dong, L. Yuan, and Z. Liu. Mobile-former: Bridging mobilenet and transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5270-5279, 2022.

S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer. cudnn: Efficient primitives for deep learning. arXiv preprint arXiv:1410.0759 , 2014.

F. Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , July 2017.

J. Choquette and W. Gandhi. Nvidia a100 gpu: Performance & innovation for gpu computing. In 2020 IEEE Hot Chips 32 Symposium (HCS) , pages 1-43. IEEE Computer Society, 2020.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

X. Ding, X. Zhang, J. Han, and G. Ding. Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11963-11975, 2022.

Y. Fu, H. Yang, J. Yuan, M. Li, C. Wan, R. Krishnamoorthi, V. Chandra, and Y. Lin. Depthshrinker: a new compression paradigm towards boosting real-hardware efficiency of compact neural networks. In International Conference on Machine Learning , pages 6849-6862. PMLR, 2022.

T. Gale, M. Zaharia, C. Young, and E. Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901 , 2020.

A. Ganjdanesh, S. Gao, and H. Huang. Jointly training and pruning cnns via learnable agent guidance and alignment. IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR , 2024.

D. Guide. Cuda c++ programming guide. NVIDIA, July , 2020.

C. Guo, B. Y. Hsueh, J. Leng, Y. Qiu, Y. Guan, Z. Wang, X. Jia, X. Li, M. Guo, and Y. Zhu. Accelerating sparse dnn models without hardware-support via tile-wise sparsity. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , pages 1-15, 2020.

C. Guo, B. Y. Hsueh, J. Leng, Y. Qiu, Y. Guan, Z. Wang, X. Jia, X. Li, M. Guo, and Y. Zhu. Accelerating sparse dnn models without hardware-support via tile-wise sparsity. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , SC '20. IEEE Press, 2020.

S. Guo, Y. Wang, Q. Li, and J. Yan. Dmcp: Differentiable markov channel pruning for neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1539-1547, 2020.

S. Han, H. Mao, and W. J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. International Conference on Learning Representations (ICLR) , 2016.

S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems , pages 1135-1143, 2015.

Y. He, J. Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han. Amc: Automl for model compression and acceleration on mobile devices. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 784-800, 2018.

Y. He, P. Liu, Z. Wang, Z. Hu, and Y. Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4340-4349, 2019.

Z. Hou, M. Qin, F. Sun, X. Ma, K. Yuan, Y. Xu, Y.-K. Chen, R. Jin, Y. Xie, and S.-Y. Kung. Chex: Channel exploration for cnn model compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12287-12298, 2022.

A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1314-1324, 2019.

I. Hubara, B. Chmiel, M. Island, R. Banner, J. Naor, and D. Soudry. Accelerated sparse neural training: A provable and efficient method to find n: m transposable masks. Advances in neural information processing systems , 34:21099-21111, 2021.

J. A. Kerr, D. Merrill, and J. Tran. Cutlass: Fast linear algebra in cuda c++. NVIDIA Developer Blog , 2017.

A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto , 2009.

H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 , 2016.

H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations , 2017.

Y. Li, K. Adamczewski, W. Li, S. Gu, R. Timofte, and L. Van Gool. Revisiting random channel pruning for neural network compression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 191-201, 2022.

Y. Li, S. Lin, J. Liu, Q. Ye, M. Wang, F. Chao, F. Yang, J. Ma, Q. Tian, and R. Ji. Towards compact cnns via collaborative compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6438-6447, 2021.

M. Lin, R. Ji, Y. Wang, Y. Zhang, B. Zhang, Y. Tian, and L. Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1529-1538, 2020.

M. Lin, Y. Zhang, Y. Li, B. Chen, F. Chao, M. Wang, S. Li, Y. Tian, and R. Ji. 1xn pattern for pruning convolutional neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(4):3999-4008, 2022.

Z. Liu, H. Mu, X. Zhang, Z. Guo, X. Yang, K.-T. Cheng, and J. Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3296-3305, 2019.

G. Lu, W. Zhang, and Z. Wang. Optimizing depthwise separable convolution operations on gpus. IEEE Transactions on Parallel and Distributed Systems , 33(1):70-87, 2021.

Y. Mao, Z. He, Z. Ma, X. Tang, and Z. Wang. Efficient convolution neural networks for object tracking using separable convolution and filter pruning. IEEE Access , 7:106466-106474, 2019.

A. Mishra, J. A. Latorre, J. Pool, D. Stosic, D. Stosic, G. Venkatesh, C. Yu, and P. Micikevicius. Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.08378 , 2021.

S. Narang, E. Undersander, and G. Diamos. Block-sparse recurrent neural networks. arXiv preprint arXiv:1711.02782 , 2017.

C. Park, M. Park, H. J. Oh, M. Kim, M. K. Yoon, S. Kim, and W. W. Ro. Balanced column-wise block pruning for maximizing gpu parallelism. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 9398-9407, 2023.

M. Park, D. Kim, C. Park, Y. Park, G. E. Gong, W. W. Ro, and S. Kim. Reprune: Channel pruning via kernel representative selection. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 14545-14553, 2024.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019.

D. Qin, C. Leichner, M. Delakis, M. Fornoni, S. Luo, F. Yang, W. Wang, C. Banbury, C. Ye, B. Akin, et al. Mobilenetv4-universal models for the mobile ecosystem. arXiv preprint arXiv:2404.10518 , 2024.

Z. Qin, Z. Zhang, D. Li, Y. Zhang, and Y. Peng. Diagonalwise refactorization: An efficient training method for depthwise convolutions. In 2018 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2018.

P. K. Rao and S. Chatterjee. Wp-unet: Weight pruning u-net with depthwise separable convolutions for semantic segmentation of kidney tumors. Research Square , 2021.

M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4510-4520, 2018.

X. Su, S. You, T. Huang, F. Wang, C. Qian, C. Zhang, and C. Xu. Locally free weight sharing for network width search. In International Conference on Learning Representations , 2021.

M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pages 6105-6114. PMLR, 2019.

M. Tan and Q. Le. Efficientnetv2: Smaller models and faster training. In International conference on machine learning , pages 10096-10106. PMLR, 2021.

C.-H. Tu, J.-H. Lee, Y.-M. Chan, and C.-S. Chen. Pruning depthwise separable convolutions for mobilenet compression. In 2020 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2020.

P. K. A. Vasu, J. Gabriel, J. Zhu, O. Tuzel, and A. Ranjan. Mobileone: An improved one millisecond mobile backbone. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7907-7917, 2023.

D. T. Vooturi, D. Mudigere, and S. Avancha. Hierarchical block sparse neural networks. arXiv preprint arXiv:1808.03420 , 2018.

X. Wu, S. Gao, Z. Zhang, Z. Li, R. Bao, Y. Zhang, X. Wang, and H. Huang. Auto-train-once: Controller network guided automatic network pruning from scratch. IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR , 2024.

Z. Yao, S. Cao, W. Xiao, C. Zhang, and L. Nie. Balanced sparsity for efficient dnn inference on gpu. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 5676-5683, 2019.

M. Ye, C. Gong, L. Nie, D. Zhou, A. Klivans, and Q. Liu. Good subnetworks provably exist: Pruning via greedy forward selection. In International Conference on Machine Learning , pages 10820-10830. PMLR, 2020.

J. Yu and T. Huang. Network slimming by slimmable networks: Towards one-shot architecture search for channel numbers. arXiv preprint arXiv:1903.11728 , 3, 2019.

J. Yu, A. Lukefahr, D. Palframan, G. Dasika, R. Das, and S. Mahlke. Scalpel: Customizing dnn pruning to the underlying hardware parallelism. In 2017 ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA) , pages 548-560. IEEE, 2017.

P. Zhang, E. Lo, and B. Lu. High performance depthwise and pointwise convolutions on mobile devices. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 6795-6802, 2020.

H.-L. Zhao, K.-J. Shi, X.-G. Jin, M.-L. Xu, H. Huang, W.-L. Lu, and Y. Liu. Probability-based channel pruning for depthwise separable convolutional networks. Journal of Computer Science and Technology , 37(3):584-600, 2022.

M. Zhu and S. Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878 , 2017.

M. Zhu, T. Zhang, Z. Gu, and Y. Xie. Sparse tensor core: Algorithm and hardware co-design for vector-wise sparse neural networks on modern gpus. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture , pages 359-371. ACM, 2019.

A Appendix

A.1 Limitation of Balanced Pruning

A number of pruning studies [51, 37, 58, 7, 31] have been proposed to balanced pruning. In a high pruning ratio, the balanced pruned model has negligible accuracy drop. However, achieving a balanced pruned model's speedup is difficult without support for specific hardware architecture or kernel. Therefore, NVIDIA has changed its hardware structure for balanced pruning [7].

A.2 Why Balanced Pruning Cannot Be Applied to DW-conv?

Balanced pruning [51] is a pruning method that accelerates performance by considering workload balance. This approach divides the weight matrix into consistent vector units and assigns an identical PR to each vector. The designated vector range is referred to as a 'balanced range'. Elements within the vector are ranked based on their redundancy, and those with the lower ranks are pruned sequentially until the target PR is achieved. However, directly applying traditional balanced pruning to DW-conv is not suitable. This is because the balanced pruning method doesn't take into account the structure of DW-conv. Since non-zero value weights in DW-conv with DR are arranged diagonally when setting a balanced range, the majority of the values within that range end up being zeros.

A.3 Limitation

The limitation of DEPrune is that it is specialized for DW-conv. Specifically, DEPrune is difficult to apply to the Vision Transformer series. Although some models in the Vision Transformer [4] use DW-conv, most of the computations in Vision Transformers are performed using self-attention and feed-forward network layers. However, our proposed HSR technique appears to be applicable to these layers.

A.4 Experiments on CIFAR-10 Dataset

We experiment the effectiveness of our proposed method using the CIFAR-10 dataset [25]. For the validation of image classification, we experiment our method with CNN models: MobileNet-V2 [43] and EfficientNet-B0 [45] We perform fine-tuning with only 100 epochs after processing pruning methods on CIFAR-10.

A.5 Limitation of Channel Pruning on DW-conv

When channel pruning is applied to depth-wise convolution, the pruned model has a structured data pattern. However, the pruning unit size of channel pruning is k h × k w . Since channel pruning is 9 ( 3 2 ) or 25 ( 5 2 ) times larger than DEPrune with a pruning unit size of 1, thereby channel pruning reduces more representation power than DEPrune. In Mobilenet-V2 on CIFAR-10, when PR is 50% , the difference in accuracy between DEPrune and channel pruning is 0 . 23% (See Table 6). Even when PR is 70% , the difference of accuracy is 0 . 35% . Therefore, our proposed DEPrune, which has not only a structured data pattern but also representation power advantage, is appropriate for DW-conv.

A.6 Comparison between DCP and Filter Pruning on PW-conv

When filter pruning is applied to PW-conv, the PW-conv pruned model has a structured data pattern. If filter pruning is performed on PW-conv, the channel of DW-conv that follows is also removed. Even

the channel of PW-conv behind DW-conv is removed as well. Therefore, the pruning unit size of filter pruning is not simply a filter of the corresponding PW-conv. The pruning unit size includes also the parameters of following layers. The larger the pruning unit size, the greater the probability that a core parameter is included in the removing group, which influences the representation power. On the other hand, our proposed DCP does not remove the parameters of other layers. Therefore, when the PR is 70% , our DCP is 0 . 62% higher in representation power than filter pruning of PW-conv (See Fig. 8).

95

(%)

94

93

—DCP[ours]

92

O-FPonPWconv

91

Baseline

90

10

20

30

40

50

60

70

80

Pruning Ratio (%)

A.7 Effect of Balanced Range

Research on n:m sparsity is currently very active topic in the field of pruning. However, this sparsity approach has two major limitations: a lack of flexibility and the requirement for specialized hardware. First, it lacks flexibility because it is fixed at a 50% pruning ratio, specifically 2:4 pruning [37]. As seen in Table 7, we conducted comparative experiments between NVIDIA's n:m sparsity and DEPrune on MobileNet-V2 using CIFAR-10. At the same pruning ratio of 50%, DEPrune-B achieves 0.31% higher accuracy than n:m sparsity. This is because DEPrune-B achieves a 50% pruning ratio within total parameters, whereas n:m sparsity achieves a 50% pruning ratio within a parameter size of 4 [23]. Secondly, in n:m sparsity, achieving optimal performance requires specialized hardware (NVIDIA A100) that can quickly handle index processing [37]. In contrast, our approach requires only a customized GPU kernel for execution.

A.8 Effect of Balanced Workload Tuning (BWT)

We propose Balanced Workload Tuning (BWT) to solve the workload imbalance problem on DCP (Sec. 5.1). However, the BWT method may slightly reduce the representation power. As shown in Table 8, we compare DCP-B (DCP + BWT) with DCP on EfficientNet-B0 of CIFAR-10. When PR is 50% , the difference between the DCP-B and DCP accuracy is 0 . 17% . Therefore, there is little representation power loss due to BWT.

A.9 Peak Memory Usage

According to paper [41], the extra overhead in total memory consumption due to zero-padding is approximately 0.3%. To assess the impact of DEPrune-BH, we measured and presented the peak memory usage of MobileNet-V2 before and after applying DEPrune-BH with a 50% pruning ratio, as

shown in Table 9. Before applying DEPrune-BH, the peak memory usage is 7.22 MB, whereas after application, it decreases to 3.63 MB, representing a reduction of approximately 49.8%.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes] .

Justification: We have fully reflected our arguments in abstract section and introduction section.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes] .

Justification: We write a limitation for our study in appendix.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA] .

Justification: Our paper does not include theoretical results.

Guidelines:

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes] .

Justification: We write a information for our study in experiment section.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No] .

Justification: We are going to open source the code later.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes] .

Justification: We write a experimental setting for our study in experiment section.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA] .

Justification: The paper does not include experiments.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes] .

Justification: We write a computer resource for our study in experiment section.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes] .

Justification: Our research conform NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA] .

Justification: There is no societal impact of the work performed.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: The paper poses no such risks.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes] .

Justification: We cite the original paper that produced the code package or dataset.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA] .

Justification: The paper does not release new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: The paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] .

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
comments: SUMMARY:
This paper proposed a pruning method target at the previously neglected Depth-wise Convolution layers which are widely used in vision models. This work is based on the Diagonal-wise Refactorization (DR) computation strategy used in GPU, and pruning one weight point in depth-wise convolution means convert the corresponding column into zero vector. To accelerate the inference of pruned models, the authors proposed to enhance the GPU performance with two hardware friendly methods: BWT and HSR, which achieves the load balance and align the memory access with GPU tile size. The evaluation result shows the proposed framework brings significant speedup in MobileNet and EfficientNet.

STRENGTHS:
1. The pruning method is based on the DR computation strategy, which provides the opportunity to convert the point pruning in depth-wise convolution layers into masking the corresponding columns as zeros.
2. To accelerate the inference processing, the proposed BWT and HSR method achieves loading balance among GPU kernels and align the computation with tile size, speeding up the GPU processing.
3. The evaluation result breakdowns the pruning method, BWT and HSR, demonstrating the contribution of each part.

WEAKNESSES:
1. The overhead of pruning method is not analyzed. How much time does it need to decide pruning points, loading balance and the parameters in HSR?
2. Comparing with the gain in pruning ratio, the accuracy drop seems not negligible. Can we modify the algorithm to set the HSR constraints to align the memory access and computation with tile size in pruning phase, without need of recalibration? Does it help with accuracy?
3. Can you provide the evaluation on edge GPU?

QUESTIONS:
See weakness.

LIMITATIONS:
See weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

comments: SUMMARY:
This paper presents the Depth-wise Separable Convolution Pruning (DEPrune). DEPrune prunes point-wise convolution and depth-wise convolution (DW-conv). And it is optimized by considering and analyzing the computation of Depth-wise Separable Convolution on GPU. Experimental results validate the speedup of DSPrune.

STRENGTHS:
1.	The proposed method supports both point-wise convolution and depth-wise convolution pruning.
2.	The performance of the proposed method seems significant.

WEAKNESSES:
1.	Some references are incomplete, for example, [23][39].
2.	The proposed scheme is only evaluated on MobileNet-V2, MobileNet-V3, and EfficientNet-B0. How does this method work for other NN models?

QUESTIONS:
1.	What does “X” and “O” mean in Figure 1?
2.	The lower parts of Figure 1 seem misleading, which do not correspond to the DW-conv example in the upper half.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents DEPrune, which prunes depthwise separable convolution models into a structured sparsity pattern that is friendly to depthwise refactorization.\n\nSTRENGTHS:\n1. Much of the motivation for conducting CNN pruning is to pursue the most lightweight — often time edge-deployed — model run under much resource constraint. Depthwise separable convolution (DSConv)-based models have a proven track record in this regard, and it is true that there are very few attempts to investigate how to prune such models. So, the exploration done in this paper is a welcoming addition to the field.\n\n2. The authors clearly demonstrate the further complications caused by filter pruning DSConv models and address them by proposing a structured sparsity pattern that is friendly to depthwise refactorization — a common application for improving the utility of DSConv — along with many other submodules. The DEPruned models show real end-to-end speedup.\n\n3. Tables 3 & 4 show DEPrune delivers better accuracy and runtime results than applying typical structured pruning methods to DSConv models.\n\nWEAKNESSES:\n1. Most of the structured pruning methods compared in Table 4 is a bit old. Although I suspect the conclusion won't change much due to the gap present and the fact that methods like CC [1] are often plenty performant even for today, still, a comparison with more modern structured pruning methods should be included. \n\n2. The authors emphasize that \"there are no previous pruning techniques that even target depth-wise convolution (DW-conv)\" in line 4. What about methods like [2] that address the exact same DSConv?\n\n3. The paper did a good job explaining each of its components in detail but lacks a clear high-level overview of how these components are attached together — some abstract visualization with a detailed walkthrough in the caption, as well as a pseudo-code section, should be added. Similarly, the experiment procedure lacks clarity: only the epoch budget of CIFAR10 experiments is shared, and critical details like the learning rate for finetuning, whether the process is iterative or post-prune, etc, are missing.\n\n4. No throughput or latency (bs = 1) results.\n\n5. Not really a weakness, but DSConv pruning is a relatively niche art with limited familiarity with the pruning community. This paper would benefit from making better contrasts with standard conv pruning to highlight its unique challenges.\n\n6. Is this DSPrune or DEPrune? The proposed method family has so many submodules named with acronyms, and it could really use a richer Table 1 for a more friendly reading experience.\n\n[1] Towards Compact CNNs via Collaborative Compression   \n[2] Pruning Depthwise Separable Convolutions for Extra Efficiency Gain of Lightweight Models   \n[3] Revisit Kernel Pruning with Lottery Regulated Grouped Convolutions   \n[4] Structured Sparsity in the NVIDIA Ampere Architecture and Applications in Search Engines\n\nQUESTIONS:\n1. Why are the efficiency metrics of any methods missing in Table 4?\n2. Can the DCP part be applied to general group convolution (where the number of groups != number of channels)?\n3. Following Q1, some of the structured pruning methods in the GKP [3] series are known to be able to maintain the output shape of a pruned conv layer by leveraging group convolution. Can this be applied to the PWConv part of the DSConv and therefore avoid the concerns introduced in Section 4.2 and Appendix 6?\n4. Is a DEPruned model zero-padded? It looks like the case in Figure 4. If so, what kind of peak memory usage are we seeing pre-and-after pruning?\n5. How would this work compare to general structured sparsity approaches with more pruning freedom? e.g., the N:M sparsity [4]?\n\nLIMITATIONS:\nThe authors addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses an important topic of pruning depth-wise separable convolutions (DSConv) called DEPrune. While structural model pruning methods like channel pruning can achieve significant speed-up for regular convolutions, they cannot secure notable speed-up on DSConv layers as they mainly prune the point-wise convolution, which is a small fraction of the compute for DSConvs. Further, naively pruning the depth-wise channels significantly reduces its capacity, leading to severe performance degradation. DEPrune prunes the depth-wise convolutions in a fine-grained manner yet achieves structural sparsity to enable practical speed-up on GPUs. It also introduces two techniques BWT and HSR to further improve the performance.\n\nSTRENGTHS:\n1. The paper is well-written. The motivations behind the design choices are clearly explained. \n\n2. The problem of pruning DSConvs is an important topic as DSConvs are widely adopted for edge applications like MobileNets. Therefore, further increasing their efficiency is of significant interest to the community.\n\n3. Extensive experimental results validate the design choices and show clear speed-up compared to the original models.\n\nWEAKNESSES:\nI cannot think of a major weakness in the paper. I hope that the authors provide more background for the readers not directly familiar with the concepts of \"alignment in GPUs\" or refer to proper references.\n\nQUESTIONS:\nI think the paper is interesting and can be useful in practical applications. Currently, as far as I know, DSConvs are not used in computationally intensive architectures like the U-Net models in diffusion models. Eventually, when more efficient architectures using DSConvs are developed, the proposed DEPrune can further improve the performance. I hope the authors release their implementations so that the community can benefit from them.\n\nLIMITATIONS:\nThe limitations are explained in the supplementary.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-17"}