S2HPruner: Soft-to-Hard Distillation Bridges the Discretization Gap in Pruning

Authors: weihao  lin
shengji  tang
chong  yu
peng  ye
tao  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 10:57:17

Abstract: Recently, differentiable mask pruning methods optimize the continuous relaxation architecture (soft network) as the proxy of the pruned discrete network (hard network) for superior sub-architecture search. However, due to the agnostic impact of the discretization process, the hard network struggles with the equivalent representational capacity as the soft network, namely discretization gap, which severely spoils the pruning performance. In this paper, we first investigate the discretization gap and propose a novel structural differentiable mask pruning framework named S2HPruner to bridge the discretization gap in a one-stage manner. In the training procedure, SH2Pruner forwards both the soft network and its corresponding hard network, then distills the hard network under the supervision of the soft network. To optimize the mask and prevent performance degradation, we propose a decoupled bidirectional knowledge distillation. It blocks the weight updating from the hard to the soft network while maintaining the gradient corresponding to the mask. Compared with existing pruning arts, S2HPruner achieves surpassing pruning performance without fine-tuning on comprehensive benchmarks, including CIFAR-100, Tiny ImageNet, and ImageNet with a variety of network architectures. Besides, investigation and analysis experiments explain the effectiveness of S2HPruner. Codes will be released soon.

Main paper: 
 introduction: 
 As deep neural networks (DNN) have achieved success in substantial fields [20, 58, 35, 61, 49] , the increasing computation and storage cost of DNN impedes practical implementation. Model pruning [39, 57, 62] , which aims at removing the less informative in a cumbersome network, has been a widespread technique for model compression. Pioneer pruning methods utilize regularization terms [63, 43] to sparsify the network or introduce importance metrics [30, 19, 18] to remove less important weights directly. However, due to the latent correlations between weights, simply eliminating the weights in an over-parameter model will hinder the integrality of structure, especially in structural pruning, where grouped filters are removed.Recently, it has been pointed out that the structure of the pruned network is essential for the final pruning performance [40] . Inspired by the differentiable architecture search (DARTS) [36, 67, 7] , emerging works [15, 17, 16] , namely differentiable mask pruning (DMP), introduce learnable parameters to generate the weight mask and impose the task-aware gradient to guide the structure search of the pruned network. In the training procedure, DMP introduces the learnable mask into the gradient graph by coupling the mask with the activation feature or weights, e.g., directly multiplying DifferentiableHigh Performance Low PerformanceFigure 1 : Comparison of different typical pruning methods and illustration of discretization gap. The darker color represents the higher relative magnitude scale of weights or masks. ⊙ denotes Hadamard product. For ease of demonstration, we use one layer to represent the entire network.the mask with the feature or weights. Through gradient descent, DMP can jointly optimize the weights and mask parameters for a bespoke structure and parameter distribution, thus causing a better performance. The search procedure essentially regards the mask-coupled network (soft network) as the performance proxy of the final discretized compact pruned network (hard network). Whereas, considering the aim of pruning is to obtain a capable hard network, a natural question is whether a superior soft network implies a corresponding high-performance hard network.In DARTS, there is a problem known as the discretization gap [66, 60, 7] , which refers to the discrepancy between the continuous relaxation architecture and the discrete architecture due to the discretization process. Since DMP follows a similar modeling format to DARTS, it also faces a comparable discretization gap problem † that the hard network struggles from having the semblable representational capacity as the soft network. A specific manifestation is that the hard network performs significantly poorer in the evaluation metrics than the soft network. Fig. 1 visually exhibits the different pruning methods and discretization gap. The discretization gap severely impacts pruning performance but has been long overlooked in DMP. There are potential techniques that may alleviate the discretization gap in previous works, e.g., gradually facilitating the steepness of the Sigmoid function via decaying temperature [26, 27, 44, 50] and optimizing the binary mask via the straightthrough estimator (STE) [65, 15] . However, these methods lead to certain side effects: the decaying temperature results in difficult mask optimization because of the vanishing gradient, and STE causes a suboptimal mask due to the coarse gradient.To alleviate the discretization gap in DMP without influencing mask optimization, we formulate the mask pruning in a soft-to-hard paradigm and propose a structured differentiable mask pruning framework named Soft-to-Hard Pruner (S2HPruner). Specifically, in the training procedure, we not only forward the soft network for the structural search but also forward the corresponding hard network and distill it under the supervision of the soft network to reduce the discretization gap. Meanwhile, we discover that even with the same corresponding hard network, the distribution of the mask parameters influences the discretization gap essentially. However, the common unidirectional knowledge distillation (KD) cannot optimize mask parameters directly, but bidirectional KD causes unbearable performance degradation. Therefore, we propose a decoupled bidirectional KD, which blocks the weight updating from the hard to the soft network while keeping the gradient corresponding to the mask. Exhaustive experiments on three mainstream classification datasets, including CIFAR-100, Tiny ImageNet, and ImageNet, demonstrate the effectiveness of S2HPruner.Our contributions are summarised as follows:• We first study and reveal the long-standing overlooked discretization gap problem in differentiable mask pruning. To alleviate it, we propose a soft-to-hard distillation paradigm, which distills the hard network under the supervision of the soft network.• Based on the soft-to-hard knowledge distillation paradigm, we propose a novel differentiable mask pruning framework named Soft-to-Hard Pruner (S2HPruner). To further reduce the discretization gap and avoid performance degradation, we propose a decoupled bidirectional KD which blocks and allows the gradient of model weights and mask parameters selectively.• Extensive experiments on three mainstream datasets and five architectures verify the superiority of S2HPruner, e.g., maintaining 96.17%(Top-1 accuracy 73.23% in 76.15%) with around 15% FLOPs. Additional ablation and investigation experiments demonstrate the underlying mechanism of the effectiveness.2 Related works 
differentiable mask pruning: 
 Considering the network structure has a decisive impact on the pruning performance [40] , numerous works [15, 12] train a binary mask for an optimal selection of sub-architecture. However, because of the non-differentiable property, directly optimizing the binary mask is very challenging and even impairs the performance [26] . Differently, differentiable mask pruning (DMP) methods [17, 9, 4, 27, 44] adopt differentiable continuous relaxation as a performance proxy of the hard network for structure search, which can be easily optimized by task-aware loss end-to-end. DMCP [17] regards the channel pruning as a Markov process and builds a differentiable mask based on the transitions between states. AutoPruner [44] proposes to construct a meta-network to generate the differentiable mask according to the activation responses, and a scaled temperature facilitates the sigmoid function approaching step function to obtain an approximate binary mask. GAL [34] learns a differentiable mask by optimizing a generative adversarial learning task in a label-free and end-to-end manner. However, the task-aware loss can ensure the high performance of the soft network but not the final hard network. There is a discretization gap limiting the target hard network during the discretization process. Different from previous DMP methods, which only focus on optimizing the soft network, our approach aims to achieve a high-performance hard network by reducing the discretization gap through soft-to-hard distillation.As a network compression technique orthogonal to pruning, knowledge distillation [22, 28, 52] (KD) transfers the dark knowledge from a large teacher network to enhance a compact student network. Recently, there have been substantial works [46, 47, 3, 32, 10] introducing KD into model pruning to further boost the pruned network. JMC [10] proposes a structured pruning based on the magnitude of weights and a many-to-one layer mapping strategy to distill the dense model to the pruned one. KD ticket [46] exploits the dark knowledge in the early stage of iterative magnitude pruning to boost the lottery tickets in the dense model. DIPNet [72] improves the ability of the pruned model by the supervision of high-resolution output. The above methods treat KD as an independent plug-in technique to enhance pruning performance without tight coupling with the selection of weights. Differently, in the proposed method, KD contributes to mask optimization directly as an integral part of the core pruning procedure. Moreover, in contrast to the typical unidirectional KD, we propose a novel decoupled bidirectional KD to alleviate the discretization gap between soft and hard networks, due to the distinct attributes of mask and weights.3 MethodQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Check Section 5 for details.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem formulation: 
 Given a network with parameters θ, a pruning algorithm generates a binary mask m via solving the following constraint optimization:EQUATIONThe θ ⟨m⟩ are the remaining parameters after pruning. The L and R are the task-specific performance loss and resource regularization, respectively. The T is a manually assigned resource budget. Intuitively, a pruning algorithm attains a slimmed subnet that optimally balances the performance and the resource consumption. The proposed pruner's forward and backward flows, illustrated via an exemplary linear layer with parameters θ. The u are the additional learnable parameters normalized by softmax. The w denotes the relaxed mask. The estimated binary pruning mask is the m. The input is denoted by i. The output of the soft and hard networks are the s and h, respectively. The L, G, and R are the performance loss, gap measure, and resource regularization, respectively.Directly optimizing the problem 1 is almost intractable due to the discreteness of m. To get around, we introduce a relaxation of m as w, which is continuous and bounded to [0, 1]. The i-th element in w represents the probability of the i-th parameter being retained. Consequently, a differentiable representative for θ ⟨m⟩ can be constructed as θ ⊙ w, where the ⊙ denotes the Hadamard product.Based on this relaxation, the problem 1 can be reformulated as two parts:EQUATIONThe α is a Lagrangian multiplier, regarded as a hyperparameter. The G is a gap measure, reflecting the difference between θ ⟨ m⟩ and θ ⊙ w. The m is an estimated pruning mask, derived from w as I [t,1] (w), where the I is an indicator function, and the t is a threshold. In the problem 2, the first part searches for a high-performance soft network that satisfies the resource constraint, and the second part reduces the gap between the hard network and the soft one. Similar to [36, 33] , to avoid alternate optimization, we combine the two parts with two additional hyperparameters β and γ:EQUATIONThe problem 3 is differentiable w.r.t. both θ and w, thus can be optimized by gradient-based methods [41, 54] :EQUATIONThe λ θ and λ w are learning rates for θ and w, respectively. The g X denotes the gradient obtained via a backward path X. Note that the term g G→θ⊙w→θ implies aligning the soft network towards the hard one, which would severely deteriorate the performance of the soft network (see Section 4.2 for details). Consequently, the update of θ is modified to:EQUATIONThe essence of the above optimization lies in two aspects: 1) the joint optimization of the entire parameters θ ⊙ w and a dynamic subset of parameters θ ⟨ m⟩ benefits from stimulative training [68] , where the entire parameters transfer knowledge to the partial ones, and the improvement of the partial parameters can, in turn, enhance the entire ones; 2) the optimization of w involves the soft-to-hard gap, which provides a new dimension to bridge the gap besides adjusting the parameters. The pseudocode describing the whole training process can be referred to in Algorithm 1, and a visualization of the forward/backward passes is provided in Fig. 2 . i ← i + 1We focus on dependency-group-based structural pruning [6, 14] , where layers in the same group share a single mask and are pruned as a whole. Besides, the pruning mask is channel-wise to comply with the structural pattern. The performance metric L is the cross-entropy for classification. The Kullback-Leibler divergence is selected as the gap measure G.Acquisition of w and t Consider a linear layer parameterized by θ ∈ R Cout×Cin . The corresponding binary pruning mask is denoted as m ∈ B Cout . To generate w, we define learnable parameters u ∈ R Cout , which can be normalized to [0, 1] via a softmax function. After softmax, the i-th element in u can be interpreted as the probability of retaining the first i channels. Consequently, the probability of the i-th channel being retained, i.e., w i , can be calculated as Cout k=i u k . With the w obtained, the pruning threshold t is derived as1 Cout Cout k=1 w k .Resource regularization We utilize floating-point operations per second (FLOPs) to evaluate resource consumption. Given a target T (in percentage), the resource regularization R is defined as(FP sof t / FP all -T ) 2 .The FP all is the FLOPs of the entire network. The FP sof t is the summation of layer-wise differentiable FLOPs. To be differentiable, the output channel number of a layer is calculated as Cout k=1 (u k * k). The u k is a softmaxed parameter introduced in the previous section.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Justification: We have carefully read the code of ethics and ensure that the research conducted in the paper conforms with it in every respect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experiments: 
 In this section, we begin by validating the effectiveness of the proposed pruner using three benchmark datasets: CIFAR-100 [29] , Tiny ImageNet [11] , and ImageNet [11] . For CIFAR-100 and Tiny ImageNet, we evaluate three common CNN architectures, i.e., ResNet-50 [20] , MobileNetV3 (MBV3) [24] , and WRN28-10 [73] , and two Transformer architectures, i.e., ViT [61] and Swin Transformer [37] , across various pruning ratios including 15%, 35%, and 55%. For ImageNet, ResNet-50 serves as the backbone model, and we compare the proposed pruner with several structural pruning methods in terms of Top-1 accuracy and FLOPs. After the benchmarking, investigative experiments are performed on CIFAR-100 using ResNet-50 to elucidate the influence of each gradient term in Algorithm 1 and the gap-narrowing capacity of the proposed pruner. Detailed training configurations are provided in the Appendix. Results on CIFAR-100 and Tiny ImageNet To assess the performance of the proposed pruner and demonstrate its adaptability to various networks, we conduct experiments using CIFAR-100 and Tiny ImageNet datasets, with ResNet-50, MBV3, and WRN28-10 serving as the backbone architectures.For each dataset-network combination, we test three different FLOPs: 15%, 35%, and 55%. We compare the proposed pruner against structured RST [1] (referred to as RST-S), Group-SL [14] , OTOv2 [6] , and Refill [5] . All methods are evaluated under consistent training settings for a fair comparison. The results, presented in Table 1 and Table 2 , reveal that the proposed pruner consistently outperforms other methods, particularly at low FLOPs. For instance, when constraint with 15% FLOPs, the proposed pruner maintains high accuracy, with gains of up to 2.73% on CIFAR-100 and 3.99% on Tiny ImageNet over the next best method.To further validate the generalizability of the proposed pruner, we apply it to two typical Transformer models, ViT [61] and Swin Transformer [37] . Similar to the CNN experiments, we test these models on CIFAR-100 with FLOPs targets of 15%, 35%, and 55%. The results, shown in Table 3 , indicate that the proposed pruner outperforms RST-S for both Transformer models across all FLOPs targets. Notably, at 55% FLOPs, the ViT pruned by the proposed method does not suffer any performance loss, and the Swin Transformer merely experiences a slight performance drop of 0.47%. The results demonstrate that while the proposed pruner is not explicitly designed for Transformers, it still achieves competitive results, highlighting its significant potential for pruning Transformer models.We further assess the performance of the proposed pruner on the prevalent ImageNet-1K benchmark. The ResNet-50 is chosen as the baseline network. Table 4 shows that, for similar FLOPs, the proposed pruner consistently suffers the least accuracy drop compared to others, underscoring the effectiveness of the proposed pruner. In the particularly challenging low FLOPs range of 10% to 20%, the proposed pruner stands out, achieving a top-1 accuracy of 73.23%, which is 3.13% higher than OTOv2, while maintaining nearly the same FLOPs (around 15%).To investigate the influence of each gradient term in Algorithm 1, we conduct experiments with some of the terms disabled to observe the impact on the final performance. The results are shown in Table 5 . Note that the term g R→w is omitted from Table 5 since it is essential to satisfy the resource constraint and is always enabled.The addition of the term g G→θ⊙w→θ severely degrades the accuracy by 14.22%, indicating that the gradient that aligns the soft network towards the hard one is detrimental to the final performance. Intuitively, from the perspective of parameter capacity, the hard network is practically pruned, resulting in a lower capacity than the soft network. Enforcing the soft network moving towards a less capable one is not plausible.Both of the term g L→θ⊙w→w and g G→θ⊙w→w contribute to improve the accuracy. For the term g L→θ⊙w→w , it implies searching for a mask that maximizes the performance of the soft network. The term g G→θ⊙w→w encourages the alignment of the soft and hard networks. Different from the term g G→θ⊙w→θ , which directly imposes on massive parameters, the term g G→θ⊙w→w merely affects the learnable masks, and thus would not drastically deteriorate the soft network while improving the hard one.The gradient term g G→θ⟨ m⟩→θ and g L→θ⊙w→θ directly optimize the parameters of the hard and soft networks, respectively, leading to crucial roles in maintaining the performance. Removing either of the two terms results in an accuracy plummet of above 75%.According to Section 3, we formulate the pruning problem into two parts: 1) find a superior soft network, i.e., the network parameterized by θ ⊙ w, that satisfies the resource constraint; 2) reducing the gap between the soft network and the practically pruned one, which is referred to as a hard network in this manuscript and parameterized by θ ⟨ m⟩. In this section, we first provide possible alternatives to formulate the problem 1 and then compare them with our proposed one on the gap-narrowing capacity to demonstrate the superiority of our method.The first alternative attempts to directly optimize the hard network on its performance, i.e., the straight-through estimators [2] :EQUATIONThe second alternative substitutes the soft network with the original one while calculating the gap measure, which conforms to self-distillation-based pruners [70] :EQUATIONComparative experiments are conducted on CIFAR-100, using ResNet-50 as the baseline. The FLOPs target is set to 15%. The gap metrics, i.e., the Jensen-Shannon divergence (JS) and L 2 distance, are averaged over the entire validation set. We measure the gap between the hard network and its direct supervision. For "Alt 1", the gap metrics are calculated between the 0.1 label smoothed [56] ground truth and the output of the hard network. For "Alt 2", the outputs of the original network and the hard one are utilized to calculate the gap metrics. For "Ours", the outputs of the soft network and the hard one are selected to analyze the gap. obtaining a hard network with the highest performance. The two observations imply that the soft-tohard formulation is a relatively better scheme to narrow the gap, and the lower gap between the hard network and its direct supervision helps improve the hard network's performance.Can fine-tuning reduce the gap? It might be questioned whether the coupled training of the soft and hard networks is necessary. In Section 3, we entangle the two optimizations in the problem 2 to avoid alternate optimization, which turns out to be an efficient yet effective scheme according to [36, 33] . Without the entanglement, multi-stage optimization is required. A soft network that satisfies the resource constraint is firstly trained solely, and then a fine-tuning stage attempts to narrow the gap between the soft network and the hard one. To explore the effect of fine-tuning, we train a ResNet-50 on CIFAR-100, constraint to 15% FLOPs, and merely optimize the soft network for 500 epochs. With this pretrained soft network, we perform fine-tuning via Algorithm 1 with a 0.1x learning rate and different epochs. The results can be referred to in Table 7 . The fine-tuning does reduce the gap to some extent, costing 250 epochs to align the soft network and the hard one (accuracy difference drops from 3.49% to 0.33%). However, compared with our coupled training, the best accuracy of fine-tuning is still 0.28% lower at the cost of an additional 250 epochs. Consequently, the adopted coupled training turns out to be a better choice.To demonstrate the architectural superiority of our pruned network, we conduct experiments on CIFAR-100, prune a ResNet-50 to 15% FLOPs via our proposed method, and then train it from scratch without bells and whistles. Three networks that are randomly pruned to 15% FLOPs are selected for the comparison. The results are shown in Table 8 . The network pruned by our method achieves the highest accuracy, verifying that the pruning mask optimized via Algorithm 1 possesses architectural superiority. 
conclusion and limitations: 
 In this paper, we reveal and study the long-standing omitted discretization gap problem in differentiable mask pruning. To bridge the discretization gap, we propose a structured differentiable mask pruning framework named Soft-to-Hard Pruner (S2HPruner), using the soft network to distill the hard network and optimize the mask. To further optimize the mask and avoid performance degradation, a decoupled bidirectional KD is proposed to alternatively maintain and block the gradient of weights and the mask. Extensive experiments verify and explain that S2HPruner can obtain high-performance hard networks with extraordinarily low resource constraints.It is essential to acknowledge the limitations of our method. Therefore, we identify the following limitations: 1) The proposed method merely considers a single dimension, pruning feature channels of a layer. However, a block containing layers might be redundant and could be pruned as a whole, which is regarded as another pruning dimension that we do not consider in this manuscript; 2) We only validate our method on the task of image classification. It is left to explore our method's capability on other tasks, such as detection, segmentation, or natural language processing; 3) We choose FLOPs as the resource indicator, which might not ensure a hardware-friendly architecture. It is promising to consider the inference time on a specific hardware as an indicator. Above all, the identified limitations present opportunities for future research and development, and we remain committed to further exploration and refinement to overcome these challenges. concentratively distributed in the post-half layers. Different from CNNs, Transformers, i.e., ViT, and Swin seem not to exhibit an obvious preference for layer depth. The pruning pattern of the shallow layers is almost uniform with that of the deep layers. Justification: Check the abstract and Section 1 for details.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: No societal impact is involved in this work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The paper is not relevant to such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All datasets and code frameworks are mentioned and properly respected. See Section 5 for details.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: † To avoid confusion, the discretization gap discussed following is in the context of DMP. 


Decision: This paper introduces a novel framework to address the challenges associated with the discretization gap in neural network pruning techniques. The reviewers identify the novelty, the experimental performance, and the writing of the paper and point out several minor issues at the same time. In rebuttal, the authors carefully addressed the issues mentioned by the reviewers and the current version looks ready for publication.

Reviews: review: paper_summary: paper_summary: Discretization in pruning poses a huge threat to network performance. To alleviate this issue, the paper proposes S2HPruner, a pruning method that leverages distillation. In details, the pruning process involves two networks that share the same architecture. The difference is that the teacher network has a weight covered with soft, differentiable masks, while the pruned student network weight is covered with non-differentiable binary masks. The optimization target involves a distillation loss that narrows the gap between the differentiable and non-differentiable network. The proposed method achieves good performance on various datasets.
strength_weakness: strength_weakness: 1. Line 9 in Abstract: SH2->S2H

2. Notation problem: the authors should unify the gradient notations in the pseudocode and the equations.

3. For Table 4: as different methods use different training settings, the number of the training epochs should be indicated.

4. The baselines in Table 4 are a bit out-of-date. The proposed S2HPruner should be compared against latest channel pruning methods, like SCOP [1] and CHEX [2].

[1] github.com/yehuitang/Pruning/tree/master/SCOP_NeurIPS2020

[2] github.com/zejiangh/Filter-GaP
questions: questions: 1. Baselines reflect the actual training capability of a certain codebase. Why do you use 76.15 instead of 76.8 as the ResNet-50 baseline? Please justify.

2. From a high-level perspective, the method is a combination of distillation and pruning. I wonder if the method could outperform the simple combination of pruning and knowledge distillation (at equal training costs)?
limitations: limitations: Limitations are addressed.
paper_summary: paper_summary: In this article, the author proposes using a 0-1 mask (hard network) and a differentiable mask (soft network) with an accuracy gap as a starting point for network distillation, where the distillation function selects Kullback Leibler divergence as the gap measure（S2HPruner）. This method was tested on datasets including CIFAR-100, Tiny ImageNet, and ImageNet for Resnet-50, ViT Swin, and other models.
strength_weakness: strength_weakness: **Q1:**  The abstract section mentions the concept of 'bidirectional', but it's incorrect. My understanding is that S2HPruner only uses sparsity in forward flows and does not involve sparsity in backward flows. Please explain the meaning of this "bidirectional" in detail.

**Q2:** Is there a reason to utilize Kullback Leibler divergence when selecting a gap measure? Is it possible to add corresponding experiments or proof explanations to the article that does not provide an explanation?

**Q3:** How does S2HPruner control network's sparsity? The method section of the article lacked any description. From the description in Figure 2, it appears that the mean is calculated from a differentiable soft mask as a threshold, and if this is the case, it seems that the purpose of controlling the sparsity rate will not be achieved.
questions: questions: Please see the weakness part
limitations: limitations: yes
paper_summary: paper_summary: The paper titled "S2HPruner: Soft-to-Hard Distillation Bridges the Discretization Gap in Pruning" introduces a novel framework designed to address the challenges associated with the discretization gap in neural network pruning techniques. The authors propose a method that bridges the gap between the representation capacities of a continuously relaxed network (soft network) and its pruned counterpart (hard network), aiming to improve the pruning performance without requiring post-pruning fine-tuning.
strength_weakness: strength_weakness: 1. The figures can be further improved in this work. For instance, figure 2 is not good engouth to illustrated the overall pipeline of the work. It is recommanded to explictly show soft net and hard net instead of thier outputs only.
2. The formats of the tables are not consistent.
questions: questions: See weaknesses above
limitations: limitations: It is recommanded to discuss the training time and GPU RAM cost during training and inference. The comparisons between the proposed one and existing SOTAs are necessary.

Comments: title: Response to Rebuttal
comment: Thanks for your approval! We promise to include these results in the revision and cite the corresponding methods.
I am satisfied with the results presented by the author, and I will raise my score. I also hope that the author could include these results in the paper upon acceptance.