S2HPruner: Soft-to-Hard Distillation Bridges the Discretization Gap in Pruning
Authors: Weihao Lin
Shengji Tang
Chong Yu
Peng Ye
Tao Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: model pruning
knowledge distillation
model compression
Primary area: deep_learning_architectures
TL;DR: A novel differentiable mask pruning method  to alleviate discretization gap

Abstract: Recently, differentiable mask pruning methods optimize the continuous relaxation architecture (soft network) as the proxy of the pruned discrete network (hard network) for superior sub-architecture search. However, due to the agnostic impact of the discretization process, the hard network struggles with the equivalent representational capacity as the soft network, namely discretization gap, which severely spoils the pruning performance. In this paper, we first investigate the discretization gap and propose a novel structural differentiable mask pruning framework named S2HPruner to bridge the discretization gap in a one-stage manner. In the training procedure, SH2Pruner forwards both the soft network and its corresponding hard network, then distills the hard network under the supervision of the soft network. To optimize the mask and prevent performance degradation, we propose a decoupled bidirectional knowledge distillation. It blocks the weight updating from the hard to the soft network while maintaining the gradient corresponding to the mask. Compared with existing pruning arts, S2HPruner achieves surpassing pruning performance without fine-tuning on comprehensive benchmarks, including CIFAR-100, Tiny ImageNet, and ImageNet with a variety of network architectures. Besides, investigation and analysis experiments explain the effectiveness of S2HPruner. Codes will be released soon.

S2HPruner: Soft-to-Hard Distillation Bridges the Discretization Gap in Pruning
Weihao Lin't Shengji Tang"t, Chong Yu² Peng Ye' Tao Chen
'chool of Information Science and Technology, Fudan University, Shanghai. China, Academy for Engineering and Technology, Fudan University, Shanghai, China. Shanghai AT Laboratory, Shanghai, China eetchenmfuudan. edu.cn
Abstract
Recently, differentiable mask pruning methods optimize the continuous relaxation architecture (soft network) as the proxy of the pruned discrete network (hard net- work) for superior sub-architecture search. However, due to the agnostic impact of the discretization process, the hard network struggles with the equivalent repre- sentational capacity as the soft network, namely discretization gap, which severely spoils the pruning performance. In this paper, we first investigate the discretization gap and propose novel structural differentiable mask pruning framework named S2HPruner to bridge the discretization gap in one-stage manner. In the training procedure, S2HPruner forwards both the soft network and its corresponding hard network, then distills the hard network under the supervision of the soft network. To optimize the mask and prevent performance degradation, we propose a decoupled bidirectional knowledge distillation. It blocks the weight updating from the hard to the soft network while maintaining the gradient corresponding to the mask. Com- pared with existing pruning arts, S2HPruner achieves surpassing pruning perfor- mance without fine-tuning on comprehensive benchmarks, including CIFAR-I00, Tiny ImageNet, and ImageNet with variety of network architectures. Besides, in- vestigation and analysis experiments explain the effectiveness of S2HPruner. Codes are publicly available on GitHub: sttps://gu /gtthub........//gg conn/oppss///e opposss//HPrrrrerr
Introduction
As deep neural networks (DNN) have achieved success in substantial fields (20 58 35
61 49
the increasing computation and storage cost of DNN impedes practical implementation. Model pruning [39 57 62], which aims at removing the less informative in a cumbersome network, has been a widespread technique for model compression. Pioneer pruning methods utilize regularization to remove less important weights directly. However, due to the latent correlations between weights, simply eliminating the weights in an over- wer-parameter model will hinder the integrality of structure, especially structural pruning, where grouped filters are removed.
terms 63 to sparsify the network or introduce importance metrics [30,
I9
18
Recently, has been pointed out that the structure of the pruned network is essential for the final pruning performance [40]. Inspired by the differentiable architecture search (DARTS) [36] 67 77. emerging works [7] 16). namely differentiable mask pruning (DMP), introduce learnable parameters to generate the weight mask and impose the task-aware gradient to guide the structure search of the pruned network. In the training procedure, DMP introduces the learnable mask into the gradient graph by coupling the mask with the activation feature or weights, e.g., directly multiplying
"Corresponding Author (eetchen @ findan.edu.cc)) 'Equal Contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Importance-basee Pruning
Differentiable Mask Pruning
Soft Network
Hard Network
Dense Network
Pruncd Network
D eose Network
Discrniraaton
Compute importance
1.2
Prime low scures
12
0.8
0.8
Differentiahle Mask
0.6
0.9
0.6
0.3
High Performenee
Low Performance
Discretization Gap
Figure 1: Comparison of different typical pruning methods and illustration of discretization gap. The darker color represents the higher relative magnitude scale of weights or masks. @ denotes Hadamard product. For ease of demonstration, we use one layer represent the entire network.
the mask with the feature or weights. Through gradient descent, DMP can jointly optimize the weights and mask parameters for bespoke structure and parameter distribution, thus causing better performance. The search procedure essentially regards the mask-coupled network (soft network) as the performance proxy of the final discretized compact pruned network (hard network). Whereas, considering the aim of pruning is to obtain a capable hard network, natural question is whether superior soft network implies corresponding high-performance hard network.
In DARTS, there a problem known as the discretization gap [66, 60 7 which refers to the discrepancy between the continuous relaxation architecture and the discrete architecture due the discretization process. Since DMP follows a similar modeling format to DARTS, it also faces a comparable discretization gap problen that the hard network struggles from having the semblable representational capacity as the soft network. A specific manifestation is that the hard network performs significantly poorer in the evaluation metries than the soft network. Fig. D visuullly exhibits the different pruning methods and discretization gap. The discretization gap severely impacts pruning performance but has been long overlooked DMP. There are potential techniques that may alleviate the discretization gap in previous works, e.g., gradually facilitating the steepness of the Sigmoid function via decaying temperature [26 27 44 50] and optimizing the binary mask via the straight- through estimator (STE) (65 15). However, these methods lead to certain side effects: the decaying temperature results in difficult mask optimization because of the vanishing gradient, and STE causes suboptimal mask due to the coarse gradient.
To alleviate the discretization gap in DMP without influencing mask optimization, we formulate the mask pruning in a soft-to-hard paradigm and propose a structured differentiable mask pruning framework named Soft-to-Hard Pruner (S2HPruner). Specifically, in the training procedure, not only forward the soft network for the structural search but also forward the corresponding hard network and distill under the supervision of the soft network reduce the discretization gap. Meanwhile, we discover that even with the same corresponding hard network, the distribution of the mask parameters influences the discretization gap essentially. However, the common unidirectional knowledge distillation (KD) cannot optimize mask parameters directly, but bidirectional KD causes unbearable performance degradation. Therefore, we propose a decoupled bidirectional KD, which blocks the weight updating from the hard to the soft network while keeping the gradient corresponding to the mask. Exhaustive experiments on three mainstream classification datasets, including CIFAR- 100, Tiny ImageNet, and ImageNet, demonstrate the effectiveness of S2HPruner.
Our contributions are summarised follows:
We first study and reveal the long-standing overlooked discretization gap problem in dif- ferentiable mask pruning. To alleviate it, we propose a soft-to-hard distillation paradigm, which distills the hard network under the supervision of the soft network.
Based on the soft-to-hard knowledge distillation paradigm, we propose novel differentiable mask pruning framework named Soft-to-Hard Pruner (S2HPruner). To further reduce the
'To avoid confusion, the discretization gap discussed following is in the context of DMP.

---

discretization gap and avoid performance degradation, we propose decoupled bidirectional KD which blocks and allows the gradient of model weights and mask parameters selectively,
Extensive experiments on three mainstream datasets and five architectures verify the supe- riority of S2HPruner, e.g., maintaining 96.17 10.17%(Top-- accuracy 73.23% in 76.15% with around 15% FLOPs. Additional ablation and investigation experiments demonstrate the underlying mechanism of the effectiveness.
2 Related works
2.1 Differentiable mask pruning
Considering the network structure has decisive impact on the pruning performance 44). numerous works [15 [2] train a binary mask for an optimal selection of sub-architecture. However, because of the con-differrentiale property, directly optimizing the binary mask is very challenging and even impairs the performance [26). Differently. differentiable mask pruning (DMP) methods [17 9 4
27 44
adopt differentiable continuous relaxation performance proxy of the hard network for
structure search, which can be easily optimized by task-aware loss end-to-end DMCP [17 regards the channel pruning as Markov process and builds a differentiable mask based on the transitions between states. AutoPruner [44) proposes to construct meta-network to generate the differentiable mask according the activation responses, and scaled temperature facilitates the sigmoid function approaching step function to obtain an approximate binary mask. GAL (34] learns differentiable mask by optimizing generative adversarial learning task in a label-free and end-to-end manner. However, the task-aware loss can ensure the high performance of the soft network but not the final hard network. There a discretization gap limiting the target hard network during the discretization process. Different from previous DMP methods, which only focus optimizing the soft network, our approach aims to achieve a high-pent gegh-performance hard network by reducing the discretization gap through soft-to-hard distillation.
2.2 Pruning with distillation
As network compression technique orthogonal to pruning, knowledge distillation 22
28 52
(KD)
transfers the dark knowledge from a large teacher network to enhance a compact student network.
Recently, there have been substantial works 446 47
I0] introducing KD into model pruning to further boost the pruned network. JMC [10] proposes a structured pruning based on the magnitude of weights and many-to-one layer mapping strategy distill the dense model the pruned one. KD ticket 4o exploits the dark knowledge in the early stage of iterative magnitude pruning to boost the lottery tickets in the dense model. DIPNet [72 improves the ability of the pruned model by the supervision of high-resolution output. The above methods treat KD as an independent plug-in technique enhance pruning performance without tight coupling with the selection of weights. Differently, in the proposed method, KD contributes to mask optimization directly as an integral part of the core pruning procedure. Moreover, in contrast to the typical unidirectional KD, we propose novel decoupled bidirectional KD to alleviate the discretization gap between soft and hard networks, due to the distinct attributes of mask and weights.
3
32
3 Method
3.1
Problem formulation
Given a network with parameters 0, a pruning algorithm generates a binary mask m via solving the following constraint optimization:
min L (0 (m)) S. R (m, T) 0.
0.m
The (m) are the remaining parameters after pruning. The and are the task-specific perfor- mance loss and resource regularization, respectively. The T is manually assigned resource budget. Intuitively, pruning algorithm attains slimmed subnet that optimally balances the performance and the resource consumption.
3

---

m
Hard Network
Slloe
Matmol
7h
Network
60
Malmal
Dense Network
Forward
Barkward
The proposed pruner's forward and backward flows, illustrated via exemplary linear layer with parameters 0. The u are the additional learnable parameters normalized by softmax. The denotes the relaxed mask. The estimated binary pruning mask is the m.. The input is denoted by The output of the soft and hard networks are the and h, respectively. The L, 9, and R are the performance loss, gap measure, and resource regularization, respectively.
Figure 2:
3.2 Overview
Directly optimizing the problem T is almost intractable due to the discreteness of m. To get around, introduce relaxation of in w represents the probability of the i-th parameter being retained. Consequently, differentiable representative for 0 (m) can be constructed as O w. where the denotes the Hadamard product. Based on this relaxation, the problem I can be reformulated as two parts:
w, which is continuous and bounded to [0, 1]. The i-th element
Part 1: min (C (0 @ w) + aR (w, T)),
0,w
Part 2: min G (0 (m). @ w)
8.w
The a is a Lagrangian multiplier, regarded as hyperparameter. The G is gap measure, reflecting the difference between (m. and 0 @ un. The m an estimated pruning mask, derived from as [[r.1] (w), where the I is an indicator function, and the is threshold. In the problem 2 the first part searches for nigh-performance soft network that satisfies the resource constraint, and the second part reduces the gap between the hard network and the soft one. Similar to [36] 33), to avoid alternate optimization, we combine the two parts with two additional hyperparameters 3 and 7
min (3C (0 O w) + aBR (w, 7) + 76 (0 (m 0 e w)). 0.w
The problem is differentiable w.r.t. both f and w, thus can be optimized by gradient-based methods
AO = ~2o ( 0 Au - Aw (8g
a 79g 796--c
The do and ARo are learning rates for 0 and w, respectively. The 9x denotes the gradient obtained via a backward path X. Note that the term -G-0o implies aligning the soft network towards the hard one, which would severely deteriorate the performance of the soft network (see Section 4.2 for details). Consequently, the update of 0 is modified to:
AO ~>o (#gc- + --c--(())
The essence of the above optimization lies in two aspects: 1) the joint optimization of the entire parameters 0 @ w and dynamic subset of parameters (m) benefits from stimulative training (68). where the entire parameters transfer knowledge to the partial ones, and the improvement of the partial parameters can, in turn, enhance the entire ones; 2) the optimization of involves the soft-to-hard gap, which provides new dimension to bridge the gap besides adjusting the parameters. The pseudo- code describing the whole training process can be referred to in Algorithm I and a visualization of the forward/backward passes is provided in Fig.

---

Algorithm 1: The training pseudo-code based on Pytorch automatic differentiation
Input: Initialized 0º and w" iteration limit imaz, dataset D, network forward function N, resource budget T, performance metric L. resource regularization R, gap measure G, pruning threshold i, gradient-based optimizer O, hyperparameters 3, and Output: O'nux and vi fmaz = [c.1] (uffux)
i < 0:
2 while < imaz do
Fetch a sample with its label y from D;
Ys N (0' ④ w²):
// The forward pass of the soft network // The forward pass of the hard network
n + N (e' (Inr.1) (w') )));
I + L (y,, y); r R w² T):
di < (yn, ys detach ()): d2 + 9 (yn. detach D,).)
(gc-0ow -0 + vw w gR-Ww f (1 (I+r) backward (); (m)) - - d1. backward ():
Oowww- d2. backward (inputs w²): 0
0
( c0w 0 -0
// Eq.
12
u1 k- O
(i, "", 3g cowo = + 19g - -ww-w
// Eq.
13
1+1
3.3
Implementation details
We focus dependency-grood -o-oo--ouuueeee structural pruning 66 14], where layers in the group share a single mask and are pruned as a whole. Besides, the pruning mask is channel-wise to comply with the structural pattern. The performance metric is the cross-entropy for classification. The Kullback-Leibler divergence is selected as the gap measure G.
Acquisition of and Consider a linear layer parameterized by 0 RCC.uxC... The correspond- ing binary pruning mask is denoted as m E IBC, To generate we define learnable parameters Cous, which can be normalized [0, 1] via softmax function. After softmax, the i-th element in u can be interpreted as the probability of retaining the first channels. Consequently, the probabil- ity of the i-th channel being retained, i.e., W4, can be calculated Σ k=j ug. With the obtained, the pruning threshold is derived as J.... -Een Wk.
Resource regularization We utilize floating point operations per second (FLOPs) to evaluate resource consumption. Given a target T (in percentage), the resource regularization R is defined as (FP soft / /Pau 2T) The FP au is the FLOPs of the entire network. The FP ft is the summation of layer-wise differentiable FLOPs. To differentiable, the output channel number of a layer is calculated as Coui (ux k). The up is softmaxed parameter introduced in the previous section.
4 Experiments
In this section, we begin by validating the effectiveness of the proposed pruner using three bench- mark datasets: CIFAR-100 (29). Tiny ImageNet (II), and ImageNet [[I]. For CIFAR-100 and
Tiny ImageNet, evaluate three common CNN architectures, i.e.. ResNet-50 20 MobileNetV3 (MBV3) 24] and WRN28-10 10 73], and two Transformer architectures, i.e., VIT 61 and Swin Trans- former 37,, across various pruning ratios including 15% 35% and 55%. For ImageNet, 5e serves as the backbone model, and we compare the proposed pruner with several structural pruning methods in terms of Top-1 accuracy and FLOPs. After the benchmarking, investigative experiments performed on CIFAR-100 using ResNet-50 to elucidate the influence of each gradient term in Algorithm| T and the gap-narrowing capacity of the proposed pruner. Detailed training configurations provided in the Appendix.
5

---

Table 1: The comparison of different pruning methods on CIFAR-100. racy(%) of dense and pruned networks with different remaining FLOPs.
We report the Top-1 accu-
ResNet-50 (Acc: 78. 14)
MBV3 (Acc:
78.09)
WRN28- 10 (Acc: 82.
Method
15%
35%
55%
15%
35%
55%
15%
35%
53%
RST-S 1
75.02
76.38
76.48
72.90
76.78
77.30
78.56
81.18
82.19
Group-SI. 14
49.04
77.90
78.37
1.43
4.90
26.24
42.41
67.71
79.59
OTOv2 6
77.04
77.65
78.35
76.29
77.35
78.39
77.26
80.61
80.84
Refill [5]
75.12
77.43
78.19
69.57
75.91
76.96
75.98
79.25
79.56
Ours
79.77
79.87
80.10
77.28
78.17
78.87
80.88
81.81
82.55
Table 2: The comparison of different pruning methods on Tiny ImageNet. accuracy(%) of dense and pruned networks with different remaining FLOPs.
We report the Top-1
ResNet-50 (Acc: 64.28)
MBV3 (Acc: 63.91)
WRN28- 10 (Ace: 61.72)
Method
15%
35%
55%
15%
35%
55%
15%
35%
55%
RST-S
63.03
63.24
64.78
55.13
61.26
62.76
58.03
61.41
62.
Group-SL [4]
0.95
19.94
55.49
0.56
2.35
53.43
0.85
25.74
57.64
OTOv2 6
60.38
63.45
65.16
57.61
59.25
60.16
57.19
61.23
61.70
Refill [5
61.05
64.14
65.02
53.87
61.84
62.49
56.64
61.83
62.22
Ours
67.02
67.38
67.64
62.49
65.11
65.54
61.83
62.46
63.44
Table 3: Verifications of transformers on CIFAR-100. report the Top-1 accuracy (%) of dense and
pruned networks with different remaining FLOPs.
ViT' (Acc: 76.49)
Swin (Acc: 77.16)
Method
15%
35%
55%
15%
35%
55%
RST-S MI
70.74
72.05
74.65
70.53
72.98
75.25
Ours
72.61
75.53
76.49
75.29
75.79
76.69
4.1 Benchmarking
Results on CIFAR-100 and Tiny ImageNet To assess the performance of the proposed pruner and demonstrate its adaptability to various networks, we conduct experiments using CIFAR-100 and Tiny ImageNet datasets, with ResNet-50. MBV3, and WRN28- 10 serving as the backbone architectures. For each dataset-network combination, we test three different FLOPs: 15%, 35% and 55%. We compare the proposed pruner against structured RST [ (referred to as RST-S), Group-SL [14), OTOv2 [6]. and Refill (5) All methods are evaluated under consistent training settings for a fair comparison. The results, presented in Table and Table reveal that the proposed pruner consistently outperforms other methods, particularly
low FLOPS. For instance, when constraint with 15% FLOPs, the proposed pruner maintains high accuracy, with gains of up to 2.73% CIFAR-100 and 3.99% on Tiny ImageNet over the next best method.
To further validate the generalizability of the proposed pruner, we apply it to two typical Transformer models, ViT [6] and Swin Transformer [37. Similar to the CNN experiments, we test these models on CIFAR- 100 with FLOPs targets of 15%, 35%, and 55%. The results, shown in Table 3 indicate that the proposed pruner outperforms RST-S for both Transformer models across all FLOPs targets. Notably, at 55% FLOPs, the ViT pruned by the proposed method does not suffer any performance loss, and the Swin Transformer merely experiences slight performance drop of 0.47%. The results demonstrate that while the proposed pruner is not explicitly designed for Transformers, it still achieves competitive results, highlighting its significant potential for pruning Transformer models.
Results on ImageNet We further assess the performance of the proposed pruner on the prevalent ImageNet- IK benchmark. The ResNet-50 is chosen as the baseline network. Table ssows that, for similar FL.OPs, the proposed pruner consistently suffers the least accuracy drop compared to others, underscoring the effectiveness of the proposed pruner. In the particularly challenging low FLOPs range of 10% to 20%, the proposed pruner stands out, achieving top-1 I accuracy of 73.23%, which is 3. 13% higher than OTOv2, while maintaining nearly the same FLOPs (around 15%).
4.2 Gradient analysis
To investigate the influence of each gradient term in Algorithm T we conduct experiments with some of the terms disabled to observe the impact the final performance. The results are shown in Table

---

Table 4: Results of ResNet-50 on Imagenet. We report the Top-1 accuracy(%) of dense and pruned networks with different remaining FLOPs. The Eyr denotes the pruning epochs. The Eez denotes the epochs for extra stages (such pretraining and finetuning). The pruning epochs can be undetermined
due to dynamic termination conditions, and corresponding terms are marked as "-".
Method
Unprumed top- (%) Pruned top-1 (%)
Top-1 drop (%)
FLOPs (%)
Eyg
Pxr
OTOv2
76. 10
70.10
6.00
14.50
120
Refill
75.84
66.83
9.01
20.00
190
Ours
76.15
73.23
2.92
15.14
200
MetaPruning 38
76.60
73.40
3.20
24.39
32
1288"
Slimmable
76.10
72.10
4.00
26.63
100
0
GAL 84
76.15
69.31
6,84
27.14
32
122
DMCP
76.60
74.40
2.20
26.80
40
100
ThiNet 45
72.88
68.42
4,46
28.50
110
90
OTOv2
76.10
74.30
1.80
28.70
120
GReg-1
76.13
73.75
2.38
32.68
180
GReg-2
76.13
73.90
2.23
32.68
CAIE
76.13
72.39
3.74
32.90
120
Ours
76.15
74.43
1.72
25.31
200
CHIP 58
76.15
75.26
0.89
37.20
270
O1'Ov2
76.10
75.20
0.90
37.30
120
0
GReg-1
62
76.13
74.85
1,28
39.06
180
GReg-2
62
76.13
74.93
1,20
39.06
180
Refill
75.84
72.25
3,59
40.00
190
ThiNet
45
72.88
71.01
1.87
44.17
110
GBN
75.85
75.18
0.67
10
130
GAL 34
76.15
71.80
4.35
45.00
32
122
SCOP 59
76.15
75.26
0.89
45.40
140
90
AutoPrune
65
74.90
74.50
0.40
45.46
60
90
SCP 27
75.89
75.27
0,62
45.70
100
100
FPGM 21
76.15
74.83
1.32
46.50
100
0
LeGR X
76.10
75.30
0.80
47.00
150
AutoSlim
70
76.10
75.60
0.50
48.43
50
100
AutoPruner
44
76.15
74.76
1.39
48.78
120
MetaPruning
1X
76.60
75.40
1.20
48.78
128
CHEX
77.80
77.40
0.40
50.00
250
Ours
76.15
75.81
0.34
34.28
200
CAIE 64
76.13
75.622
0.5T
54.77
120
CHIP 53
76.15
76.30
-0.15
55.20
270
Slimmable 111
76.10
74.90
1.20
55.69
100
TAS 13
77.46
76.20
1,26
56.50
120
120
SSS [26
76.12
71.82
4.30
56.96
FPGM 21
76.15
75.59
0.56
57.80
100
LeGR 8
76.10
75.70
0.40
58.00
150
GBN 69
75.88
76.19
-0.31
59.46
10
130
Refill 5
75.84
74.46
1.38
60.00
95
190
ThiNet 451
72.88
72.04
0.84
63.21
110
90
GReg-1
02
76.13
76.27
-0.14
67.11
180
MetaPruning
38
76.60
76.20
0.40
73.17
32
128
Ours
76.15
77.01
-0.86
54.38
200
SSS [6
76.12
75.44
0.68
84.94
100
0
Ours
76.15
77.53
-1.38
76.19
200
Note that the term RR- is omitted from Table it is essential to satisfy the resource constraint and is always enabled.
The addition of the term 9g-06w-0 severely degrades the accuracy by 14.22%, indicating that the gradient that aligns the soft network towards the hard one is detrimental to the final performance. Intuitively, from the perspective of parameter capacity, the hard network practically pruned, resulting lower capacity than the soft network. Enforeing the soft network moving towards less capable one not plausible.
Both of the term fc- and -6- contribute to improve the accuracy. For the term 9 --- implies searching for a mask that maximizes the performance of the soft network. The term 9g 96- encourages the alignment of the soft and hard networks, Different from the term
7

---

Table 5: The influence of different gradient components in the proposed pruning method. The FLOPs target set to 15% for all experiments.
9 w
0
9o- -c-
Top-1 1 Acc (%)
65.55
X
79.77
X
3,95
X
X
V
1.73
X
78.30
X
X
78.77
X
X
77.69
Table 6: Gap comparison with alternative formulations of the problem [ The symbols 0, C w and
O (rm) represent the top-1 accuracy of the original, soft and hard networks, respectively.
Top-1
Acc (%)
Method
JS
L2
0
w
0 (m
Alt
2.06e-00
2.74e-03
77.13
Alt
5.17e-01
8.58e-04
78.35
77.78
Ours
1.93e-01
1.60e-04
80.14
79.77
-0- which directly imposes on massive parameters, the term -G-00w-3u merely affects the learnable masks, and thus would not drastically deteriorate the soft network while improving the hard one.
The gradient term (g-0 and 9c-0Gw-00 directly optimize the parameters of the hard and soft networks, respectively, leading to crucial roles in maintaining the performance. Removing either of the two terms results in accuracy plummet of above 75%.
4.3 Investigation into gap
According to Section 3 we formulate the pruning problem into two parts: 1) find a superior soft network, i.e., the network parameterized by 0 O w, that satisfies the resource constraint; 2) reducing the gap between the soft network and the practically pruned one, which is referred to as hard network in this manuscript and parameterized by 0 (m In this section, we first provide possible alternatives to formulate the problem T and then compare them with our proposed one on the gap-nan nar-nanoo capacity to demonstrate the superiority of our method.
The first alternative attempts to directly optimize the hard network on its performance, i.e., the 2
straight-through estimators
Alt 1 min (C w) aR (w,T)),
(6)
min L (0 (m)).
The second alternative substitutes the soft network with the original while calculating the gap measure, which conforms to self-distiillaion--ased pruners [70]
Alt 2 min (C (0 O w) a'R (w, T)),
min (C (0) + g (0 (m) 0))
Comparative experiments are conducted on CIFAR-100, using ResNet-50 as the baseline. The FLOPSs target set to 15%. The gap metrics, i.e., the averaged over the entire validation set. We measure the gap between the hard network and its direct supervision. For "Alt 1", the gap metrics are calculated between the 0. label smoothed [56] ground truth and the output of the hard network. For "Alt 2", the outputs of the original network and the hard one are utilized to calculate the gap metrics. For "Ours", the outputs of the soft network and the hard one are selected to analyze the gap.
divergence (JS) and L2 distance, are
Table shows the comparison results. can observed that lower gap between hard network and its direct supervision renders the hard network better performance. With the JS reduced from 2.06 ("Alt 1") to 0.193 ("Ours"), the top-1 accuracy of the hard network increases from 77. 13% to 79. 99%,;; 2) Our proposed soft-to-hard formulation achieves the lowest gap both JS and L2,

---

Table 7: The top-1 accuracy of the hard network different fine-tuning epochs. The top-1 accuracy of the solely trained soft network before fine-tuning is 79.41% %. The symbols C w and @ (m)
represent the top-1 accuracy of the soft and hard networks, respectively.
Epoch
10
50
100
250
500
④ w
79.91
80.00
80.14
79.82
79.31
Top-1 Acc (%)
i (m)
76.42
78.89
79.07
79.49
79.46
Table 8: The top-1 accuracy of different networks pruned from ResNet-50 with a 15% FLOPs constraint and then trained from scratch without bells and whistles.
Network Top-1 Ace (%)
Rand Rand 2 Rand 3 Ours
76.46
76.64
76.96
77.65
obtaining a hard network with the highest performance. The two observations imply that the soft-to- hard formulation is relatively better scheme to narrow the gap, and the lower gap between the hard network and its direct supervision helps improve the hard network's ' performance.
Can fine-tuning reduce the gap? It might be questioned whether the coupled training of the soft and hard networks is necessary. In Section 3 we entangle the two optimizations in the problem P to avoid alternate optimization, which turns out be efficient yet effective scheme according to [6 33 Without the entanglement, multi-stage optimization is required. A soft network that satisfies the resource constraint firstly trained solely, and then fine-tuning stage attempts narrow the gap between the soft network and the hard one. To explore the effect of fine-tuning, we train ResNet-50 on CIFAR- constraint to 15% FLOPs, and merely optimize the soft network for 500 epochs. With this pretrained soft network, we perform fine-tuning via Algorithm [ with a 0.1x learning rate and different epochs. The results can be referred to in Table 7 The fine-tuning does reduce the gap to some extent, costing 250 epochs to align the soft network and the hard one (accuracy difference drops from 3.49% to 0.33%). However, compared with our coupled training, the best accuracy of fine-tuning is still 0.28% lower at the cost of an additional 250 epochs. Consequently, the adopted coupled training turns out to be better choice.
4.4 Architectural superiority
To demonstrate the architectural superiority of our pruned network, we conduct experiments on CIFAR-100, prune a ResNet-50 to 15% FLOPS via our proposed method, and then train it from scratch without bells and whistles. Three networks that are randomly pruned to 15% FLOPSs are selected for the comparison. The results are shown in Table g The network pruned by our method achieves the highest accuracy, verifying that the pruning mask optimized via Algorithm possesses architectural superiority.
Conclusion and limitations
In this paper, we reveal and study the long-standing omitted discretization gap problem in differen- tiable mask pruning. To bridge the discretization gap, we propose a structured differentiable mask pruning framework named Soft-to- tort Pruner (S2HPruner), using the soft network to distill the hard network and optimize the mask. To further optimize the mask and avoid performance degradation, a decoupled bidirectional KD is proposed to alternatively maintain and block the gradient of weights and the mask. Extensive experiments verify and explain that S2HPruner can obtain high-performance hard networks with extraordinarily low resource constraints.
essential to acknowledge the limitations of our method. Therefore, we identify the following limitations: 1) The proposed method merely considers a single dimension, pruning feature channels of layer. However, block containing layers might be redundant and could be pruned which is regarded as another pruning dimension that we do not consider in this manuscript; 2) We only validate our method on the task of image classification. It is left to explore our method's 's capability on other tasks, such as detection, segmentation, or natural language processing; 3) We choose FLOPS as the resource indicator, which might not ensure hardware-friendly architecture. is promising to consider the inference time on a specific hardware an indicator. Above all, the identified limitations present opportunities for future research and development, and we remain committed to further exploration and refinement to overcome these challenges.
whole,

---

Acknowledgement
This work is supported by National Natural Science Foundation of China (No. 62071127); National Key Research and Development Program of China (No. 20227D0060101, Shanghai Natural Science Foundation (No. 1 402900), Shanghai Municipal Science and Technology Major Project NN..221SS2DZX0100 The computations in this research were performed using the CFFF platform of Fudan University.
References
[1] Yue Bai, Huan Wang. ZHIQIANG TAO, Kunpeng Li, and Yun Fu. Dual lottery ticket hypothesis. In International Conference on Learning Representations, 2021.
[2] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint 1308.33300.33332 2013.
[3] Liyang Chen, Yongquan Chen, Juntong Xi, and Xinyi Le. Knowledge from the original network: restore a better pruned network with knowledge distillation. Complex & Intelligent Systems, pages 1-10, 2021.
[4] Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao, and Ping Luo. Diffrate: Differentiable compression rate for efficient vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17164- 17174, 2023.
[5] Tianlong Chen, Xuxi Chen, Xiaolong Ma, Yanzhi Wang, and Zhangyang Wang. Coarsening the granularity: Towards structurally sparse lottery tickets. In International Conference on Machine Learning, pages 3025-3039. PMIR, 2022.
[6] Tianyi Chen, Luming Liang, DING Tianyu, Zhihui Zhu, and Ilya Zharkov. Otov2: Automatic, generic, user-friendly. In International Conference on Learning Representations, 2023.
[7] Xin Chen, Lingxi Xic, Jun Wu, and Qi Tian. Progressive darts: Bridging the optimization gap for in the wild. International Journal of Computer Vision, 129:638-655, 2021.
[8] Ting- Wu Chin, Ruizhou Ding, Cha Zhang, and Diana Marculescu. Towards efficient model compression via learned global ranking. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1518-1528, 1528, 2020.
Minsik Cho, Saurabh Adya, and Devang Naik. Pdp: Parameter-free diffcrentiable pruning you need. Advances Neural Information Processing Systems, 36, 2024.
[10] Baiyun Cui, Yingming Li, and Zhongfei Zhang. Joint structured pruning and dense knowledge distillation for efficient transformer model compression. Neurocomputing, 458:56-69, 2021.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai L.,, and Li Fei-Fei. Imagenet: large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[12] Xiaohan Ding, Guiguang Ding, Yuchen Guo, Jungong Han, and Chenggang Yan. Approximated oracle filter pruning for destructive width optimization. In International Conference on Machine Learning. pages 1607-1616. PMLR, 2019.
[13] Xuanyi Dong and Yi Yang. Network pruning via transformable architecture search. Advances in Neural Information Processing Systems, 32, 2019.
[14] Gongfan Fang, Xinyin Ma. Mingli Song. Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16091-16101, 2023.
[15] Shangqian Gao, Feihu Huang. Jian Pei, and Heng Huang. Discrete model compression with resource constraint for deep neural networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1899 1908, 2020.
[16] Shangqian Gao, Zeyu Zhang. Yanfu Zhang, Feihu Huang. and Heng Huang. Structural alignment for network pruning through partial regularization. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 17402 17412, 2023.
10

---

[17] Shaopeng Guo, Yujie Wang, Quanquan Li, and Junjie Yan. Dmep: Differentiable markov channel pruning for neural networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recogmition, pages 1539-1547, 2020.
[18] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
[19] Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-776 2016.
[21] Yang He, Ping Liu, Ziwei Wang. Zhilan Hu, and Yi Yang. Filter pruning via geometric median deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 4340 2019.
[22] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, stat, 1050:9, 2015.
[23] Zejiang Hou. Minghai Qin. Sun, Xiaolong Ma. Kun Yuan, Xu, Yen Ken-Kuang Chen, Rong Jin, Yuan Xie, and Sun- Yuan Kung. Chex: Channel exploration for enn model compression. Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 12287-12298, 2022.
[24] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chiele Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314 1324, 2019.
[25] Gao Huang, Zhuang Liu. Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[26] Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In Proceedings of the Earopean conference on computer vision (ECCV), pages 304-320, 2018.
[27] Minsoo Kang and Bohyung Han. Operation-awars soft channel pruning using differentiable masks. In International conference on machine learning, pages 5122-5131. PMI.R, 2020.
[28] Jangho Kim, ScongUk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. Advances in neural information processing systems, 31, 2018.
[29] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[30] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 1989.
[31] Scung Hoon Lee, Scunghyun Lec, and Byung Cheol Song. Vision transformer for small-size datasets, 2021.
[32] Tianhong Li, Jianguo Li, Zhuang Liu, and Changshui Zhang. Few sample knowledge distillation for efficient network compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14639 -96477 2020.
[33] Yumqiang Li, Jan C van Gemert, Torsten Hoefler, Bert Moons, Evangelos Eleftheriou, and Bram-Ersst Verhoef. Differentiable transportation pruning. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16957-16967, 2023.
[34] Shaohui Lin, Rongrong Ji, Chenqian Yan, Baochang Zhang, Liujuan Cao, Qixiang Ye, Feiyue Huang, and David Doermann. Towards optimal structured enn pruning via generative adversarial learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2790-2799. 2019.
[35] Weihao Lin, Tao Chen, and Chong Yu. Spvos: Efficient video object segmentation with triple sparse convolution. IEEE Transactions on Image Prucessing, 2023.
[36] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In International Conference on Learning Representations, 2018.

---

[37] Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[38] Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang. Kwang- -ing Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEFF/CVF international conference on computer vision, pages 3296-3305, 2019.
[39] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pages 2736-2744, 2017.
[40] Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2018.
[41] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:171..51001, 2017.
[42] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
[43] Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_0 regularization. In International Conference on Learning Representations, 2018
[44] Jian-Hao Luo and Jianxin Wu. Autopruner: An end-to-end trainable filter pruning method for efficient. deep model inference. Pattern Recognition, 107:107461, 2020.
[45] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pages 5058-5066, 2017.
[46] Haoyu Ma, Tianlong Chen, Ting-Kuei Hu, Chenyu You, Xiaohui Xie, and Zhangyang Wang. Good students play big lottery better. arXiv preprint arXiv:2101.0225 2021.
[47] James O' Neill, Sourav Dutta, and Haytham Assem. Deep neural compression via concurrent pruning and self-distillation. arXiv preprint arXiv:2109 15014, 2021.
[48] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle. A. Beygelzimer. F. d'Alché-Bii Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[49] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 779-788, 2016.
[50] Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification. Advances in neural information processing systems, 33:1 13:11380-11390 2020.
[51] Yiqing Shen, Liwu Xu, Yuzhe Yang, Yaqian Li, and Yandong Guo. Self-distillation from the last mini-batch for consistency regularization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11943-11952 11952, 2022.
[52] Zhiqiang Shen and Eric Xing. A fast knowledge distillation framework for visual recognition. In Computer Vision ECCV 2022: 17th European Conference, Tel Israel, October 27, 2022, Proceedings, Part XXIV, pages 673 690. Springer, 2022.
[53] Yang Sui, Miao Yin, Yi Xic, Huy Phan, Saman Aliari Zonouz, and Bo Yuan. Chip: Channel independence- based pruning for compact neural networks, Advances in Neural Information Processing Systems, 34:24604- 24616, 2021.
[54] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139-1147. PMLR, 2013.
[55] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
12

---

[56] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and puttern recognition. pages 2818-2826, 2016.
[57] Shengji Tang, Weihao Lin, Hancheng Ye. Peng Ye. Chong Yu. Baopu Li, and Tao Chen. Enhanced sparsification via stimulative training. arXiv preprint arXiv:2403.0641 2024.
[58] Shengji Tang, Peng Ye, Baopu Li, Weihao Lin, Tao Chen, Tong He, Chong Yu, and Wanli Ouyang. Boosting residual networks with group knowledge, arXiv preprint arXiv:2308. 13772, 2023.
[59] Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu. and Chang Xu. Scop: Scientific control for reliable neural network pruning. Advances in Neural Information Processing Systems. 33, 2020.
[60] Yunjie Tian, Chang Liu, Lingxi Xie, Qixiang Ye, et al. Discretization aware architecture search. Pattern Recognition, 120: 1881100,220000 2021.
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[62] Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Neural pruning via growing regularization. In Interna- tional Conference on Learning Representations (ICLR), 2021.
[63] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances in neural information processing systems, 29, 2016.
[64] Yu-Cheng Wu, Chih-Ting Liu, Bo- Ying Chen, and Shao-Yi Yi Chien. Constraint-awar importance estimation for global filter pruning under multiple resource constraints. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pages 686- (66-667,77 2020.
[65] Xia Xiao, Zigeng Wang. and Sanguthevar Rajasekaran. Autoprune: Automatic network pruning by regularizing auxiliary parameters. Advances in neural information processing systems, 32, 2019.
[66] Peng Ye, Baopu Li, Tao Chen, Jiayuan Fan, Zhen Mei, Chen Lin, Chongyan Zuo, Qinghua Chi, and Wanli Ouyang. Efficient joint- cintt-iimnsional search with solution space regularization for real- semantic segmentation. International Journal of Computer Vision, 130(11):2774 2694, 2022.
[67] Peng Ye. Baopu Li, Yikang Li, Tao Chen, Jiayuan Fan, and Wanli Ouyang. b-darts: Beta-decay regulariza- tion for differentiable architecture search. proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10874-10883, 2022.
[68] Peng Ye, Shengji Tang, Baopu Li, Tao Chen, and Wanli Ouyang. Stimulative training of residual networks: A social psychology perspective of loating. Advances in Neural Information Processing Systems, 35:3596- 3608, 2022.
[69] Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. Advances in neural information processing systems, 32, 2019.
[70] Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers. arXiv preprint arXiv: 1903. 1728, 2019.
[71] Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang. and Thomas Huang. Slimmable neural networks. arXiv preprint arXiv: 222.8822888889 08928, 2018
[72] Lei Yu, Xinpeng Li. Youwei Li, Ting Jiang, Qi Wu, Haoqiang Fan, and Shuaicheng Liu. Dipnet: Efficiency distillation and iterative pruning for image super resolutiso. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1692 1701, 2023.
[73] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
13

---

Appendix A: Details of experiments
In this section, we provide the detailed specific training settings in the main manuscript. All experiments conducted under the deep learning framework Pytorch [48], versioned 2.0.1 with Python versioned 3. 10. The CUDA version 1 11.8. A cluster equipped with 8 NVIDIA A 100 GPUs, 1024 GB memories, and 120 CPUs is used run experiments, A single GPU is used for experiments on CIFAR-100 and Tiny ImageNet. For Imagenet, four GPUs are paralleled to run the task.
A1. Implementation details of CIFAR-100
The CIFAR-106 dataset 29 is classical classification dataset, which consists of 100 categories with 50,000 training images and 10,000 testing images. For ResNet-50 (20) and MBV3 [24), we follow the training settings in 688 In detail, the whole training epoch number is 500, and the input batch size is 64. We utilize the original SGD as the optimizer with a 0.05 initial learning rate and 0.0003 weight decay. The cosine decay schedule is utilized to adapt the learning rate throughout the training process. For WRN28- 10 773, we follow the training settings of 73 In detail, the epoch number and batch size are 200 and 128, respectively. The SGD is chosen as the optimizer with a 0.1 initial learning rate and a 0.0005 weight decay. The learning rate scheduler is also the cosine decay schedule. For ViT [1] and Swin Transformer [37], we use an image size of 32x32 and a patch size of 4. The epoch number and batch size are 200 and 128, respectively. The optimizer is AdamW [42 with an initial learning rate 0.001/0.003 for Swin/ViT and a 0.05 weight decay. The learning rate is warmed up for 10 epochs. The data augmentations are the same as the ones in [31]. Different from CNNs, where we regard the channel numbers of convolutional and linear layers as the width dimension, to prune the width of Transformers, dimensions (Swin) of attention layers and the channel numbers of linear layers into account.
take the head numbers (ViT) or head feature
A2. Implementation details of Tiny ImageNet
The Tiny ImageNet dataset is derived from the renowned ImageNet dataset 11 comprising 200 categories, 100,000 training images, and 10,000 test images. For the ResNet-50 20 and MBV3 [24] models, employ 500 epochs and batch size of 64. The optimization is performed using SGD with an initial learning rate of 0.1 and a weight decay of 0.0003. We utilize a step-wise learning rate scheduler, reducing the learning rate to 0.1 and 0.01 of the original at the 250th and 375th epochs, respectively. For the WRN28-10 10 (73 architecture, we adopt the training settings from [51), with 200 epochs and batch size of 128. The SGD optimizer is used with initial learning rate of 0.2 and weight decay of 0.0001. The learning rate is decreased in a step-wise manner, dropping to 0.1 and 0.01 of the initial value the 100th and 150th epochs, respectively.
A3. Implementation details of ImageNet
The ImageNet dataset is a widely used classification benchmark, containing 1,000 categories, 1.2 million training images, and 50,000 testing images. For the evaluated ResNet-50 (20) the epoch number and batch size are 200 and 512, respectively. We utilize SGD as the optimizer. The learning rate is initialized 0.2 and is controlled by cosine decay schedule. The weight decay is 0.0001. Besides, we apply the commonly used data augmentations according to 25 55
A4. Hyperparameters o, 3, and
To determine the hyperparameters in Algorithm we utilize dynamic balancing scheme based on the T-2 norm of gradients. Specifically, the -c- and 9G- are firstly normalized by their own La norms before being added together. The addition result then aligned with gR-w via being scaled to the L2 norm of gR-W For -c- and 9g-0 no balancing is applied. The two terms are added with fixed coefficients. For CNNs, the coefficients are 0.5 and for -c-00w--0 and (g--o(m))-o respectively. For Transformers, the coefficients are 1 and I for 9 - and ( -. respectively. The coefficient for gRo is set to 5.
14

---

Appendix B: Trajectory of FLOPs and accuracy
In this section, the FLOPS and accuracy trajectory is provided display the pruning procedure of S2HPruner visually. We conduct experiments on five different models, including ResNet-50 [20). MobileNetV3 (MBV3) (24), WideResNet28-100 10 [73], ViT [6]], and Swin Transformer [37] on
CIFAR-100. The results are shown in Fig. 3 and as the training epoch increases, our methods can fast converge the capacity of the hard network to the target FLOPs. However, it does not mean the mask optimization is finished. It can be seen that the performance of the robust network is steadily improving. It suggests that after entering the feasible region, S2HPruner consistently explores the possible structure and exploits the optimal architecture. Moreover, although applied to five unique architectures, S2HPruner obtains similar trajectories, which demonstrates the generalization of S2HPruner.
15-
033
PEDAC
242
PIGS
0.24
32
333
0.32
200-
33-
55
kotch
(a) ResNet-50
(b) MBV3
(c) WRN28-11 10
140%
Accure
8.49
3.37
8:08
660
53
383
150
203
24
108
150
330
(d) VIT
(e) Swin
Figure 3: The trajectory of FLOPs and accuracy. We report the accuracy and FLOPs of the hard network during the training of different models, including (a) ResNet-50 (b) MobileNetV3 (c) WideResNet28-100 10 (d) VIT (e) Swin Transformer on CIFAR-100.
Appendix C: Visualization of pruning process
We report the detailed output channel variation of different five networks during pruning visually. The results are shown in Fig 5 6 9 The target FLOPs is set to 15%. It is worth noting that because the mask is dependent on the dependencies groups where layers all have the same output channels, we report the index of dependencies groups as the index of layers, which does not correspond to the raw definition completely. can be observed that the channel variation disparate between different layers. which implies our method is not restricted to trivial solutions such as uniform channel distribution. Combined analysis with Fig. 3 we can observe that although the FLOPs satisfies the constraints, our method is not caught in loafing but can consistently explore the structure space to find the optimal architecture. A similar phenomenon also exists in all five networks, which demonstrates the generalization of the proposed method.
Appendix D: The architecture of the pruned network
We provide the architecures of our pruned networks in Fig. 4 The pruned networks are obtained via using Algorithm on CIFAR-100 with a 15% FLOPs target. It can be observed from Fig. 4 that different pruned network varies in architecture pattern. For example, convolutional neural networks (CNNs), i.e., ResNet-50, MBV3, and WRN28- 10 may prefer deeper layers. The retained channels are
15

---

2000
Rerained
Dulput Chemnel Number
1750
Prunes
1000
33
34
Layer Index
(a) ResNet-50
QuAput Channel Number
1200
Prumes
Beraimed
1005
800
500
400
SCOCE STAIN
3249
30
11
12
13
14
15
16
19
29
21
22
23
24
25
26
27
29
Layer Index
(b) MBV3
Retained
JEAVO
272996
273946
24444
J12V43
Output Chaannel Number
Pruneid
500
400
300
3
5
16
12
19
14
15
Layer Index
(c) WRN28-10 10
400
Ouiput Chanme Number
350
1.5
100
10
11
12
13
34
35
1.6
17
18
14
Lay Index
(d) VIT
Qutpu: Channel Number
303
600
800
300-
200-
102
2000
15
Layer Index
(e) Swin
Figure The architectures networks, including (a) ResNet-50 (b) MobileNerV3 (c) NedeRes 10 (d) ViT (e) Swin Transformer, pruned via proposed method on 1000 The target FLOPs set to 15%.
16

---

concentratively distributed in the post-half layers. Different from CNNs, Transformers, i.e., VIT, and Swin seem exhibit an obvious preference for layer depth. The pruning pattern of the shallow layers is almost uniform with that of the deep layers.
Table 9:
The pruning results obtained via training a ResNet-50 on CIFAR-100 100 with different random
seeds using our proposed method. We report the Top-1 accuracy and FLOPs.
Exp
#1
#2
#3
#4
Top-1 Ace (%)
79.77
79.68
79.80
79.75
FLOPs (%)
15.36
15.22
15.94
15.21
Table 10:
Training efficiency comparison with different methods. For fair comparison, double-epoch training results of other methods are included.
RST-S
Depgraph
OTO v2
IMP-Refill
Ours
Top-I Acc (%) (Ix training schedule)
75.02
49.07
77.04
75.12
79.77
Top-1 Acc (%) (2x training schedule)
75.54
50.83
77.21
75.66
GPU time per epoch (s)
44.50
70.97
79.36
74.12
50.13
Peak GPU memory (MB) (training)
4329
4319
4221
4261
4710
Peak GPU memory (MB) (inference)
1351
1365
1262
1329
1279
Appendix E: Robustness against randomness
To assess the consistency of our proposed pruning method, we target a 15% reduction in FLOPs using ResNet-50 as the base model the CIFAR- 100 dataset. Four independent runs with varying random seeds are conducted, and the results are presented in Table 9 The pruned networks consistently achieved comparable performance, with negligible variations in Top-1 accuracy (less than 0.1 % deviation) and FLOPs (less than 1% deviation). These findings validate the robustness of our proposed method, indicating that the resource consumption of the pruned network is expected and its performance is reliable.
Appendix F: Training efficiency
To investigate the training efficiency of the proposed method, we compare its training time to other established pruning methods in Table TO Using ResNet-50 the CIFAR- 100 dataset, our experiments reveal that the proposed method achieves exceptional performance while maintaining a competitive training time, ranking second-shortest among the tested methods. This efficiency stems from the inherent parallelism the soft and hard networks. The forward and backward passes the soft and hard networks can be executed simultaneously, leveraging the power of CUDA streams or multi-GPU parallelism. Furthermore, our method operates in a single stage, eliminating the need for sequential fine-tuning or iterative pruning, further contributing to its time efficiency. To isolate the pass counts, we extended the training epochs of other methods two-fold match our method's counts. Despite this, the performance of these methods plateaued, indicating that simply increasing training time does not guarantee improved pruning results. This underscores the inherent advantages of our method.
impact of forward/backww
Besides, the GPU memory costs during training and inference are also reported in Table 10 During training, method costs bearable (about 10%) more GPU memories than the average of other methods due to the additional learnable masks and the mask state buffers in the optimizer. During inference, the GPU memory target set to I 15% for all the methods, there is no significant difference in GPU memory costs.
merely depend the scale of the pruned network. As the FLOPs
17

---

Layer
ONred mumbr
Layer
Layer
Layer
Reraton
60008
20099
naralion
10000
2x00
4000
200900
10000
Chama Tumaitunee number
Layer
Layer
Layer
Layer
Layer
Layer
200000
0000
20000
40009
20t 10
40000
2000
6000
200000
400
2000
6000
Chunid Chun-drnnao nanioa
Layer
Laver
Layer
Layer
Layer
Chanred numbrr
Layer
Beradon
20099
Daralion Layer
Ieration
20099
20:00
Layer
Layer
Layer
Layer
Layer
Ohamee numbrr
100000
40008
20000
40090
2000 80 Lay
48000
2000
4000
200000
40X
24000
40000
Layer
Laver
Layer
Layer
Chanvvl mumber
Layer
of Sunmn
Chunrel
Bersdon
Ioration Layer
200999
Relaion
Layer
Layer
Chame Sanee
Chamno Lunte
Layer
Channee number
Chanve
250
Ioeation
Rcroden
Re000n
Figure 5: The detailed channel variation of ResNet-50 on CIFAR-100 during training. The target FLOPs is set to 15%. The horizontal axis represents the training iterations. The vertical axis represents the output channel number.
18

---

CAume number
Cumeel number
Chame
ler on L
Randion
liaratin
licrat on
Boraton Layer
Chaste number
Chann rambee
herat on
Eenttin
fiero Layer
Flerston on
Cumma nurberrrrr
12
Cumme mumbrr
Curmel
1S000ppooe Ficraiio Laver 20
flsradinn
Layer 16
Layer 17
Laver 18
number
Lover 19
130000 40000
Cuuma nurbar
Layer 21
Layer
Layer 23
Layer 24
Layer 25
13:00 socoon indroo atite
12 coneerseee 1000140000
Layer 26
Layer 27
Laver
Layer 29
130000 20000000000000 20000000000000000000000000000000 R0000
123000 10100000000000 (0003-000)
13:0000000 12.00 300000 60000
Figure The detailed channel variation of MobileNetV3 on CIFAR-100 during training. The target FLOPs is set to 15%. The horizontal axis represents the training iterations. The vertical axis represents the output channel number.
19

---

Layer
Layer
Laver
Chiree number
33303
1000
8000
23903
12:03
1000
10000
2000
19009
2003
60000
18090
Charno number
150
159
19:03
20:23
12:03
$900
80000
10000
1900
2000
Doratio
Layer 9
Layer 10
Layer 11
Layer 12
Charce Surber
3 146
Clrre numbe
Bar 10c
Der08 on
Itsradin
Diration
Layer 13
Layer 14
Layer 15
290
Bora bocntion
Ilaradion
Figure 7: The detailed channel variation of WideResNee28-10 on CIFAR-I00 during training. The target FLOPs is set to 15%. The horizontal axis represents the training iterations. The vertical axis represents the output channel number.
20

---

Layer
Layer
Layer
Layer
Charma: numbur
Chaome numbrr
103
Cherte
1880 1eats 2005 2X00
5300 toxao 13R019 200000
5005 10000 100002200 teration
30519 DRBES 2000E 2000
Layer
Layer
Laver
Laver
rumber
Chanme rumber
CTT Humberr
5080
Itaration
Beratio
Berstion
Eteration
Layer
Layer
Layer 11
Layer 12
5,000 2000
50099 1/08
Layer
number
Coattel number
100
19009 yohoot porene Tar abion
vin Toaan 200E922000E Ranaton
1000ee 1000022 Boration
TUOOO 1200 Diertion
Layer 17
Layer 18
Layer 19
Channel numberr
Iteration
Ranaton
Eanation
Figure 8: The detailed channel variation of ViT on CIFAR-100 during training. The target FLOPs set to 15%. The horizontal axis represents the training iterations. The vertical axis represents the output channel number.
21

---

Layer
Layer
Layerr
Laver
Laver
20006 Layer 10
Charoe santer
Layer
Chareel sumrer
Layer
Lyyer
Layer
Heerttonn
Rerne
Rerstion
ltenation
Feration
Charte rumber
Layer
Layer
Layer
Layer
Layer
Haraiion
Reration
on
lteration
Itration
Chrree murteeeneee
Laver
Laver
Iteration
Rerotion
to ootin
Iteration
Ieeraion
Cha
Iarazion
L0009 233003
lteration
Ecratin
Cherre number
Layer
Charren mumber
Laver
150
30000
2003
Figure 9: The detailed channel variation of Swin Transformer on CIFAR-100 during training. The target FLOPs 15%. The horizontal axis represents the training iterations. The vertical axis represents the output channel number.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Check the abstract and Section for details.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Check Section 5 for details.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Detailed methods and configurations be queried in Section 3nd Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [No]
Justification:
The code will be released soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuia wwwiicccguie//eeeeeeeeeis CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Check Section and Section for details. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide results under different random seeds. See Section 5 for details.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
25

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section and Section 5 for details.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://wwwrr..coooao/lo/cccccaaa
Answer: [Yes]
Justification: We have carefully read the code ethics and ensure that the research conducted in the paper conforms with it in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
No societal impact involved in this work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper is not relevant to such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets and code frameworks are mentioned and properly respected. See Section details.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "C1CLg6a7Rq", "reviewer": "Reviewer_MZh4", "summary": "Discretization in pruning poses a huge threat to network performance. To alleviate this issue, the paper proposes S2HPruner, a pruning method that leverages distillation. In details, the pruning process involves two networks that share the same architecture. The difference is that the teacher network has a weight covered with soft, differentiable masks, while the pruned student network weight is covered with non-differentiable binary masks. The optimization target involves a distillation loss that narrows the gap between the differentiable and non-differentiable network. The proposed method achieves good performance on various datasets.", "questions": "1. Baselines reflect the actual training capability of a certain codebase. Why do you use 76.15 instead of 76.8 as the ResNet-50 baseline? Please justify.\n\n2. From a high-level perspective, the method is a combination of distillation and pruning. I wonder if the method could outperform the simple combination of pruning and knowledge distillation (at equal training costs)?", "limitations": "Limitations are addressed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper highlights the harm of discretization in pruning, which I think is the core problem.\nThe paper distill knowledge from a soft-mask network to a hard-mask one, which I think is novel.", "weaknesses": "1. Line 9 in Abstract: SH2->S2H\n\n2. Notation problem: the authors should unify the gradient notations in the pseudocode and the equations.\n\n3. For Table 4: as different methods use different training settings, the number of the training epochs should be indicated.\n\n4. The baselines in Table 4 are a bit out-of-date. The proposed S2HPruner should be compared against latest channel pruning methods, like SCOP [1] and CHEX [2].\n\n[1] github.com/yehuitang/Pruning/tree/master/SCOP_NeurIPS2020\n\n[2] github.com/zejiangh/Filter-GaP", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The typos in Abstract and the inconsistency of gradient notations in the pseudocode and the equations.**\n\n*Answer:* Thanks for the suggestion. We will fix the typos and align the notation of gradients in the pseudo code and the equations.\n\n&nbsp;\n\n**Q2: In Table 4, the epochs should be indicated and additional baselines, SCOP and CHEX, are required.**\n\n*Answer:* Following the advice, we will report the epochs required to obtain a pruned network from scratch in Table 4 like Table t1. For methods pruning from a pretrained model, we report the pretraining and pruning epochs, respectively. In Table 4, we aim to select widely used pruning methods with a large range of time for comprehensive comparison, and Table 4 also includes some of the latest methods such as OTOv2 (2023) [2] and Refill (2022) [4]. Following R1's suggestions, we also compare S2HPruner with the mentioned baselines SCOP [1] and CHEX [5]. Specifically, for SCOP, because it has a similar unpruned top-1 accuracy as ours, we directly report the results from its original literature. For CHEX, its unpruned top-1 accuracy is significantly larger than ours. Thus, we additionally conduct an experiment that deploys the standard CHEX on our training schedule, named CHEX*. The results are shown in Table t1. It can be observed that, under different FLOPs constraints, our method consistently suffers the minimal Top-1 drop, demonstrating the superiority of our method. Moreover, under the same training schedule, S2HPruner can outperform CHEX with higher accuracy and lower FLOPs. The above experiments and comparisons will be included in Table 4 in the revision.\n\n&nbsp;\n\n**Q3: Why is the baseline of ResNet-50 76.15\\%?**\n\n*Answer:* The 76.15\\% Top-1 accuracy is obtained via training a ResNet-50 baseline on ImageNet with the recipe of our codebase, which is reported in Appendix A3. Besides, it is hard to compare different pruning methods fairly due to their distinct training schedule, and the Top-1 accuracy around 76.15\\% is a mainstream and typical benchmark adopted by recent pruning works, such as the mentioned SCOP [1], OTOv2 [2], and Greg-2 [3]. Considering it, in this manuscript, we adopt the ResNet-50 with 76.15\\% Top-1 accuracy on ImageNet as the dense model for pruning for fair comparison among different pruning methods.\n\n&nbsp;\n\n**Q4: Whether the proposed method could outperform the simple combination of pruning and knowledge distillation (at equal training costs).**\n\n*Answer:* To compare with the simple combination of pruning and knowledge distillation at equal training costs, we carry out experiments using ResNet-50 on CIFAR-100. Firstly, an STE pruner (see Eq. 6 in the main manuscript) is utilized to prune a network under 15\\% FLOPs constraint for half of the total epochs reported in Appendix A1. Then, vanilla knowledge distillation is applied for the remaining half of the total epochs. The teacher network is a dense ResNet-50 with full training on CIFAR-100, and the student network is the network pruned by STE pruner. The results are shown in Table t2. Our method outperforms the simple combination in Top-1 accuracy by 2.13\\%. It demonstrates that knowledge distillation is tightly integrated into our pruning process, and the joint optimization renders the pruned network better performance. Detailed statements can be found in lines 103-108 in the main manuscript.\n\n&nbsp;\n\n[1]: Tang, Y., Wang, Y., Xu, Y., Tao, D., Xu, C., Xu, C., Xu, C.: Scop: Scientific control for reliable neural network pruning. Advances in Neural Information Processing Systems 33 160 (2020)\n\n[2]: Chen, T., Liang, L., Tianyu, D., Zhu, Z., Zharkov, I.: Otov2: Automatic, generic, user-friendly. In: International Conference on Learning Representations (2023)\n\n[3]: Wang, H., Qin, C., Zhang, Y., Fu, Y.: Neural pruning via growing regularization. In: International Conference on Learning Representations (ICLR) (2021)\n\n[4]: Chen, T., Chen, X., Ma, X., Wang, Y., Wang, Z.: Coarsening the granularity: Towards structurally sparse lottery tickets. In: International Conference on Machine Learning. pp.142 3025–3039. PMLR (2022)\n\n[5]: Hou, Z., Qin, M., Sun, F., Ma, X., Yuan, K., Xu, Y., Chen, Y.K., Jin, R., Xie, Y., Kung, S.Y.: Chex: Channel exploration for cnn model compression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp.152 12287–12298 (2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TL0TAGs0Gs", "reviewer": "Reviewer_aPhZ", "summary": "In this article, the author proposes using a 0-1 mask (hard network) and a differentiable mask (soft network) with an accuracy gap as a starting point for network distillation, where the distillation function selects Kullback Leibler divergence as the gap measure（S2HPruner）. This method was tested on datasets including CIFAR-100, Tiny ImageNet, and ImageNet for Resnet-50, ViT Swin, and other models.", "questions": "Please see the weakness part", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper is easy to follow.\n* The experiments and investigation into gaps and gradients are well-developed.", "weaknesses": "**Q1:**  The abstract section mentions the concept of 'bidirectional', but it's incorrect. My understanding is that S2HPruner only uses sparsity in forward flows and does not involve sparsity in backward flows. Please explain the meaning of this \"bidirectional\" in detail.\n\n**Q2:** Is there a reason to utilize Kullback Leibler divergence when selecting a gap measure? Is it possible to add corresponding experiments or proof explanations to the article that does not provide an explanation?\n\n**Q3:** How does S2HPruner control network's sparsity? The method section of the article lacked any description. From the description in Figure 2, it appears that the mean is calculated from a differentiable soft mask as a threshold, and if this is the case, it seems that the purpose of controlling the sparsity rate will not be achieved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Explain the meaning of \"bidirectional\" in detail.**\n\n*Answer:* We use \"bidirectional\" to describe that the knowledge transfer in our method is bidirectional. The soft network transfers knowledge to the hard one, improving the performance of the hard network. Simultaneously, the hard network provides guidance for the soft network to better optimize the mask. Both directions of knowledge transfer can benefit the reduction of the discretization gap in pruning. Detailed explanations can be referred to in lines 59-63 in the main manuscript.\n\n&nbsp;\n\n**Q2: Why is Kullback-Leibler divergence selected as the gap measure?**\n\n*Answer:* The Kullback-Leibler divergence has been proved advantageous to reduce the gap between two distributions [1,2]. To verify its superiority, we keep other training settings unaltered and compare the Kullback-Leibler divergence with two well-known metrics, L1 and L2 distance. The results are reported in Table t3. The Kullback-Leibler divergence exhibits a distinct advantage over the other two metrics, turning out that it is suitable to be selected as a gap measure.\n\n&nbsp;\n\n**Q3: How does the network control sparsity?**\n\n*Answer:* To control sparsity, we introduce a resource regularization $\\mathcal{R}=\\left(FP_{soft}/FP_{all}-T\\right)^2$ as stated in lines 153-157 in the main manuscript. Given a FLOPs target, the resource regularization controls the soft output channel number of each layer in a differentiable manner to gradually meet the target. In line 157, the soft output channel number is presented as $C_s=\\sum_{k=1}^{C_{out}}\\left(u_{k}*k\\right)$. Note that, the \"differentiable soft mask\" mentioned in the question is the $w_i$, derived from $\\sum_{k=i}^{C_{out}}u_k$ according to line 151. As a result, the mean of the $w_i$ can be derived as $C_{s}/C_{out}$. After thresholding, the hard output channel number is $C_h=\\left|\\left\\\\{j\\in\\left[1,C_{out}\\right]\\left|w_j\\geq\\left(C_{s}/C_{out}\\right)\\right.\\right\\\\}\\right|$. According to [3], the $C_h$ is close to $C_s$ in most cases and can be finally converged to satisfy the FLOPs target of the hard network. Moreover, we provide the difference of $C_s$ and $C_h$ during training in Figure t1. It can be observed that the difference is negligible all over the training process.\n\n&nbsp;\n\n[1]: Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)\n\n[2]: Tang, S., Ye, P., Li, B., Lin, W., Chen, T., He, T., Yu, C., Ouyang, W.: Boosting residual networks with group knowledge. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 5162–5170 (2024)\n\n[3]: Chen, M., Shao, W., Xu, P., Lin, M., Zhang, K., Chao, F., Ji, R., Qiao, Y., Luo, P.: Diffrate: Differentiable compression rate for efficient vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp.138 17164–17174 (2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IbwYMwK3WL", "reviewer": "Reviewer_ECuu", "summary": "The paper titled \"S2HPruner: Soft-to-Hard Distillation Bridges the Discretization Gap in Pruning\" introduces a novel framework designed to address the challenges associated with the discretization gap in neural network pruning techniques. The authors propose a method that bridges the gap between the representation capacities of a continuously relaxed network (soft network) and its pruned counterpart (hard network), aiming to improve the pruning performance without requiring post-pruning fine-tuning.", "questions": "See weaknesses above", "limitations": "It is recommanded to discuss the training time and GPU RAM cost during training and inference. The comparisons between the proposed one and existing SOTAs are necessary.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The S2HPruner framework is innovative in that it directly addresses the issue of the discretization gap, which is often overlooked. By incorporating soft-to-hard distillation, the method ensures that the hard network maintains a similar level of performance to the soft network, which is a significant improvement over traditional pruning methods.\n2. The overall writing is good and the work is easy to follow.", "weaknesses": "1. The figures can be further improved in this work. For instance, figure 2 is not good engouth to illustrated the overall pipeline of the work. It is recommanded to explictly show soft net and hard net instead of thier outputs only.\n2. The formats of the tables are not consistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Figure 2 is recommended to explicitly show soft and hard networks.**\n\n*Answer:* Thanks for providing the suggestion. We will emphasize the concept of \"soft\" and \"hard\" in Figure 2 and try our best to make it more comprehensible. The revised version can be referred to in Figure t2.\n\n&nbsp;\n\n**Q2: The formats of the tables are not consistent.**\n\n*Answer:* Following the advice, we will check all the tables and try our best to align their formats in the revised manuscript.\n\n&nbsp;\n\n**Q3: It is recommended to discuss the training time, and GPU RAM cost during training and inference.**\n\n*Answer:* The discussion about training time can be found in Appendix F, where we compare our method with four SOTA methods that we can reproduce. To further discuss GPU memory costs, we expand the table in Appendix F with additional metrics, as shown in Table t4. During training, our method costs bearable (about 10\\%) more GPU memories than the average of other methods due to the additional learnable masks and the mask state buffers in the optimizer. During inference, the GPU memory costs merely depend on the scale of the pruned network. As the FLOPs target is set to 15\\% for all the methods, there is no significant difference in GPU memory costs. Detailed discussion about the efficiency of our method can be found in Appendix F.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thanks for the valuable feedback provided by all reviewers. We appreciate the reviewers MZh4 (R1), aPhZ (R2), and ECuu (R3) for approving our contributions: (1)  innovative method (R1, R3), (2) well-developed experiments (R2), (3) good writing and easy to follow (R2, R3). Besides, the concerns are mainly concentrated on (1) format problems (R1, R3), (2) the figure of the framework (R3), (3) some declaration on settings and definitions (R1, R2, R3). Under the NeurIPS policy, we will follow reviewers’ suggestions to refine the figures and format of the paper at our discretion. Below, please find our detailed answers to address your concerns. All tables and figures tagged by "tx" are represented in the appended pdf file.

Meta-review: This paper introduces a novel framework to address the challenges associated with the discretization gap in neural network pruning techniques. The reviewers identify the novelty, the experimental performance, and the writing of the paper and point out several minor issues at the same time. In rebuttal, the authors carefully addressed the issues mentioned by the reviewers and the current version looks ready for publication.
Decision: Accept (poster)