Probabilistic Neural Pruning via Sparsity Evolutionary Fokker-Planck-Kolmogorov Equation
Authors: Zhanfeng Mo
Haosen Shi
Sinno Jialin Pan
Venue: ICLR 2025 Spotlight (2025)
Keywords: Optimization for Deep Network
Probabilistic Method
Machine learning
Model compression
Primary area: optimization
TL;DR: We propose a principled probabilistic pruning framework, coined Sparsity Evolutionary FPK Equation that enables us to generate performant masks with desired sparsity via particle simulation of the dynamic of optimal mask distribution.

Abstract: Neural pruning aims to compress and accelerate deep neural networks by identifying the optimal subnetwork within a specified sparsity budget. In this work, we study how to gradually sparsify the unpruned dense model to the target sparsity level with minimal performance drop. Specifically, we analyze the evolution of the population of optimal subnetworks under continuous sparsity increments from a thermodynamic perspective. We first reformulate neural pruning as an expected loss minimization problem over the mask distributions. Then, we establish an effective approximation for the sparsity evolution of the optimal mask distribution, termed the **S**parsity Evolutionary **F**okker-**P**lanck-**K**olmogorov Equation (**SFPK**), which provides closed-form, mathematically tractable guidance on distributional transitions for minimizing the expected loss under an infinitesimal sparsity increment. On top of that, we propose SFPK-pruner, a particle simulation-based probabilistic pruning method, to sample performant masks with desired sparsity from the destination distribution of SFPK. In theory, we establish the convergence guarantee for the proposed SFPK-pruner. Our SFPK-pruner exhibits competitive performance in various pruning scenarios. The code is available on https://github.com/mzf666/SFPK-main.

Published as a conference paper at ICLR 2025
PROBABILISTIC NEURAL PRUNING VIA SPARSITY EVOLUTIONARY PORKER-PLANCCKKRLLMGGOROO EQUATION
Zhanfeng Mo
Haosen Shi
Sinno Jialin Pan
Nanyang Technological University, Singapore zhanfeng001 @ntu. www.tuu.edu..a.. edu. haosen, shi chi.aifffe cuhk edu. hk; sinncpan@cuhk. edu. hk
The Chinese University of Hong Kong
ABSTRACT
Neural pruning aims to compress and accelcrate deep neural networks by identi- fying the optimal subnetwork within specified sparsity budget. In this work, we study how to gradually sparsify the unpruned dense model to the target sparsity level with minimal performance drop. Specifically, analyze the evolution of the population of optimal subnetworks under continuous sparsity increments from thermodynamic perspective. We first reformulate neural pruning an expected loss minimization problem over the mask distributions. Then, we establish an ef- fective approximation for the sparsity evolution of the optimal mask distribution. termed the Sparsity Evolutionary Fokker-Plaane Koomogorov Equation (SFPK). which provides closed-form, mathematically tractable guidance on distributional transitions for minimizing the expected loss under an infinitesimal sparsity in- crement. On top of that, we propose SFPK-pruner, a particle simulation-base probabilistic pruning method, to sample performant masks with desired sparsity from the destination distribution SFPK. In theory, we establish the convergence guarantee for the proposed SFPK- -rrnnrr Our SFPK- pruner exhibits competitive performance in various pruning scenarios. The code is available on GitHub")
INTRODUCTION
Overparameterizeed deep neural networks have shown remarkable success across diverse applications
(He et al. 2016 Howard et al. 2017 Redmon et al., 2016) Radosavovic et al. 2020 Dosovitskiy
et al. 2021 Touvron et al., 2021 Devlin et al. 2018]] Zhang et al. 2015 Brown et al. (202)) How-
ever, their considerable parameter size significantly hinders their efficiency, posing challenges on edge computing scenarios (Wu et al. 2016 Chen & Ran 2019 Yao et al. 2017: Han et al. 2017
Bhattacharya & Lane 2016). To mitigate this issue, a variety of model compression methods (Han
et al. He et al. 2018 Hinton et al.) 2015 Polino
| al. 2018 Courbariaux al.) [013 Chen et al. 2019 have been established to reduce the model size and computation cost with minimal
performance drop. Neural pruning is one of the most mainstream methods due its practicality
and effectiveness (Lee et al., 2019; Tanaka et al.] 2020 Louizos et al.) 2018 Chen et al., 2021 Zhuang al., 2020) Frankle Carbin, 2019; Zhu & Gupta 2018a). The objective of neural prun-
ing is to remove unimportant parameters of a dense model until a specified sparsity constraint is
met while minimizing the model loss to the greatest extent possible. general, neural pruning
challenging, as it requires identifying optimal subnetworks under sparsity constraint, which is high-dimensional zero-one programming problem (Papadimitriou & Steiglitz, 1998).
Can mitigate the difficulty of neural praning if a set of performant subnetworks with slightly lower sparsity is readily accessible? In practice, a performant subnetwork can slightly sparsi- fied with only small performance drop through minor and careful modifications (Hu et al.) 2016 Srinivas & Babul (2015). This implies the potential to derive the optimal subnetworks at the de- sired sparsity level by making minor adjustments to slightly denser optimal subnetworks. Thus, are motivated to study how the population of optimal subnetworks evolves when there is a small increase in sparsity. Following this sparsity evolution of the optimal subnetwork population, the

---

Published as a conference paper at ICLR 2025
unprumed model, which initially consists of the optimal subnetwork population with zero sparsity, can be gradually transformed into the desired optimal subnetworks at the target sparsity level. As a result, we can prune the model by sampling performant subnetworks from this final population.
How does the population of optimal subnetvorks change as sparsity increases progressively? An initial attempt to address this question was made in (Mo et al. 2023 which aims to approximate the trajectory of a carefully constructed optimal subnetwork as sparsity increased from zero to one. However, this approach only considers a single optimal subnetwork each sparsity level, failing to capture the entire population. Furthermore, it assumes that a global optimum could always be found within the vicinity of an arbitrary optimal subnetwork, which is overly restrictive in practice.
Different from (Mo et al., 2023), our approach draws inspiration from thermodynamics, which stud- ies the evolution of particle populations through time-dependent particle density functions. These functions describe the probability distribution of the particle coordinate or velocities under random forces (Kadanoff, 2200). Following this spirit, we study the sparsity evolution of the population of optimal subnetworks from probabilistic perspective. At each sparsity level, we apply the prob- abilistic lifting technique (Wild et al., 2023) to redefine neural pruning as a problem of expected loss minimization over the distribution space of subnetworks. From the thermodynamic perspective, each subnetwork can be analogized to a particle, while a subnetwork distribution at a specific spar- sity level represents particle density pattern at particular time. In this context, the expected loss can be considered as the overall energy of the particle system. The optimal subnetwork distribution that minimizes the expected loss effectively summarizes the quality of the entire optimal subnetwork population by assigning high weight to each optimum. Therefore, we can approximate the sparsity evolution of the population of optimal subnetworks by tracking the sparsity evolution of the opti- mal subnetwork distribution. Instead of solving the optimal subnetwork distribution from scratch at the target sparsity level, this approach breaks down the process into a series of tractable transitions between optimal subnetwork distributions at gradually increasing sparsity levels. As demonstrated later, each transition step can be achieved by carefully refining the quality assignments of the micro- scopic subnetworks over the support of the current sparsity level's optimal distribution.
Contributions. 1) We establish a novel probabilistic pruning framework, termed the Sparsity Evo- Equation (SFPK) (Section 4.2): conceptually similar to the (FPK) equation, which models the time evolution of parti- cle density in thermodynamics, our proposed SFPK theory offers closed-form, mathematically tractable approach to approximate the evolution of the optimal subnetwork distribution as sparsity gradually increases, serving as analog to the FPK equation in probabilistic neural pruning. 2) we develop SFPK-pruner, a principled pruning method based on particle simulation, which utilizes SFPK effectively sample performant subnetworks with desired sparsity (Section 433); addition- ally, we prove the convergence of the SFPK-pruner pruner in Proposition 3 3) we conduct numerical ex- periments showing that SFPK-pruner achieves competitive performance across various models and datasets in both structured and unstructured pruning scenarios (Section 5
lutionary Saaeee--PPancc-ooommooooo
traditional
RELATED WORK
Various pruning frameworks have been proposed. Score- based methods prune parameters based on their importance scores. Different metries (Han et al. 2015| Tanaka et al. 2020 Wang et al. [020 Lubana & Dick 2021 Lee et al. 2019 Jacot et al. 2018 Rachwan et al. (022) are proposed as the score function determine which weights to prune. However, such an estimation is shortsighted for pruning a lot of parameters at once. Regularization-be based methods (Louizos et al. 2018 Chen et al., 2021 Zhuang et al., 2020) prune by converting the original binary constrained optimization prob- lem into continuous unconstrained optimization problem and incorporating soft sparsity penalties. However, these penalties are numerically unstable, making it difficult to converge to desired spar- sity level. Sparse training methods (Zhu & Guptal 201 Sa, Frankle & Carbin 2019, Morcos et al.) 2019 Tai et al. 2022) iteratively prune and tine-tune to approximate optimal sparse subnetworks, yet the computational demands of jointly optimizing parameters and subnetworks limit their scalability to large models and datasets.
Bayesian-basel neural pruning approaches typically incorporate the distribution of model parame- ters while imposing specific sparsity constraints. In (Neklyudov et al., 2017 Louizos et al. 2017), certain priors are utilized to push the posterior distribution of the model parameters to sparse distri-
2

---

Published as a conference paper at ICLR 2025
bution, facilitating subsequent pruning. In (Zhao al. 2019; Molchanov et al. 2017a). Bayesian neural network method called variational dropout is used, where unimportant connections are dis- carded based the dropout results. From a high-level perspective, these methods share similarities with our approach in terms of modeling the behavior of the mask distribution. However, there are two distinctions: 1) focus on series of marginal distributions under different sparsity levels rather than the entire parameter distribution spaces, and 2) we do not follow the posterior estimation paradigm, but instead use stochastic differential equation to model the evolution of the distribution.
PRELIMINARIES
Basic Notations. In this paper, the dense neural network to be pruned is parameterized by E o C RS, with O representing the parameter space of the d prumable parameters. The lp- norm of a vector v is defined as ||vlpp - (E, [v(l))) where v(i] denotes the v's i-th entry. The element-wise product is denoted by O. [n] [I. B, A {m E B ||mll & r denotes the Euclidean ball with radius r > O. We use the terms "probability measure" and "distribution" interchangeably without specification. For any measurable set M R", P(M) defines the collec- tion of all probability measures with a density function (w.rt Lebesgue measure) supported on M. Dirac(:; m) denotes the Dirac delta distribution concentrated on supp(z) denotes the support of the distribution æ. (ni denotes the uniform distribution over M. v(-1-) TR. x Rd + R+ denotes a probability kernel, where v(([mm) E P(RR), and the mapping m + r(M m) is mea- surable for any measurable M C IRd Thus, v(-Im) represents a distribution conditioned on m. v* π _ π (mm - () - d) represents the convolution between a probability kernel v and a distribution æ. law(m) denotes the distribution of the random variable m.
n).
Neural Pruning. In neural pruning, given dense neural network 0" and parameter budget d' (typically d' < d), the objective is identify the optimal subnetwork containing at most d' non-zero parameters while maximizing the model performance: minmm (U,1)* L(m @ 0"), s.t. ||m||o d'.
Here, L(-)= gd + ++ is model energy function to be minimized (such as evaluation loss, predictive error, or generalization bound). m is binary mask, where m[i] equals to 0 indicates that the i-th parameter is pruned, and vice versa. We refer to 1 d' d'/d as the target sparsity of neural pruning. The population of optimal masks refers to the set of masks that achieve the minimum model energy.
Probabilistic Lifting and Convexification. The probabilistic lifting and convexification technique, initially devised to address the notorious nonconvexity in optimization problems (Wild et al. 2023 Section 2). involves two key steps: 1) lifting the original Fuclidean nonconvex optimization problem to the distribution space, and 2) incorporating a convex regularizer, as outlined in (..
min ((x) xe0
Probabillistc lifting
min (1, न A f( ((z))((z)) Convexification " (E)
min (1, + XR|=], (I)
xEP(N)
where ((-) denotes the nonconvex objective function, Ω denotes the nonconvex feasible region, R[- denotes a convex functional, and À is the regularization rate. This approach revises the original optimization into convex distributional optimization problem by substituting Ω with the convex distribution space P(2),, and replacing I(-) with the convex functional (1,-) + R[--- Thanks to the introduced convexity, the global optimum of (T) is proven uniquely exist (Wild al.) 2023). Furthermore, working within the distribution space provides greater convenience for conducting continuous evolution analysis and convergence analysis. Admittedly, these conveniences come with trade-offs: 1) The mnite-dimensional optimization problem is lifted to the aefinite-dimensin tribution space, which is numerically more challenging address. 2) The convex regularizer intro- duces gap proportional to À between the initial objective and the convexified one. To identify the optimum of (1). one needs to resort to generalized variational inference methods (Knoblauch al.) 2022), such optimizing over family of miniteddmmensiona parameterized distributions running infinte-dimennionall gradient flow in the distribution space (Louizos Welling 2017 Wild et al,
dis-
2024 Santana et al. 2021 Glaser et al. 2021 D' D'Angolo& & Fortuin 2021).
Sparsity Evolution of the Optimal Mask Distribution. With the probabilistic lifting and convex- ification technique,
feel free analyze the sparsity evolution of the population of optimal masks in a probabilistic pruning scenario. Let ग. be the optimal mask distribution that minimizes the probabilistic pruning objective at sparsity level f. In intuition, n. weights the quality of each mask
3

---

Published as a conference paper at ICLR 2025
of sparsity t: it assigns large weights to highly performant masks while assigning minimal, even zero, weights to poorly performing masks. Thus, the dynamic of + Ty is essentially characterized by the evolution of the assignment of mask quality under progressive sparsity increments. In prac- tice, a performant mask can be slightly sparsified with a small performance drop through minor and careful alterations. This suggests that, for a small At-sparsity increment, one can derive a from Te through minor adjustments on the quality assignment scheme of Tt. If Te is readily accessible, deriving local transition from Ty T would be easier than solving Te+ar from scratch. Sup- pose the transition from Te to Te+ae can be formulated as T(me e the infinitesimal evolution of Ty can be derived as a distribution valuedd differential equation:
By informally taking At + O,
T++ T. At
A4-0
OcT = [ E [0, 10'1--'/dd dd/d..
(2)
Note that the formulation of (2) bears great similarity with the standard FPK equation (Bogachev
itt al. 2015), which takes the form O.p, A [pr]. Here, Pt is the particle density at time f, and [ a differential operator determined by the microscopic particle dynamic. While the standard FPK equation describes the time evolution of the particle density, 22) describes the distributional transition Te under an dt- dt-sparsity increment, making it the sparsity evolutionary analog of the thermodynamic FPK equation. Therefore, we designate (2) as the Sparsity Evolutionary Fokker- Planck Planck-Kolmooorrre Kolmogorov Equation (SFPK). SFPK provides explicit guidance on transforming To (the initial optimal mask distribution which is highly concentrated on the unpruned model) to T1-d which enables us to sample performant masks within the parameter budget d'. Next, we focus on formalizing the derivation of the SFPK and developing the SFPK-guided probabilistic pruning algorithm.
4 METHODOLOGY
In this section, we first introduce Probabilistic Soft Neural Pruning (Definition D. a convex prob- abilistic relaxation that facilitates our derivation, with provable uniqueness and the existence of its global optimal mask distribution at any sparsity level (Proposition [T. On top of that, we derive SFPK as an approximation of the sparsity evolution of the optimal mask distributions (Proposition 2 and Proposition]] B}. Finally, we propose JFPK-Pruner a particle simulation-bart based pruning method (Algorithm I that enables us sample performant masks from the destination distribution of SFPK.
4.1 PROBABILISTIC SOFT NEURAL PRUNING
There are two research issues for formally establishing SFPK following the derivations in ( Firstly, the minimal increment of the fn-norm based discrete sparsity is 1/d, which hinders us from taking the limit At + O. Secondly, the optimal mask distribution undergoes drastic fluctuations even under an infinitesimal sparsity increase, as there no overlap between the sets of masks at dif- ferent sparsity levels. This leads spiky change in the support of ne, preventing us from analyzing the continuous transition from Te to T-+di-
To address these two issues, first generalize the discrete sparsity to continuous values. We then lax the sparsity constraint to ensure smoother transitions in mask distributions as sparsity increases. This yields the relaxed soft pruning problem:
min [.. A (m() & 0*), MP A (m E Rd s(m) E [1, [1,++p) pl) mc MC
(3)
Here, 0* is the dense model, E J0, 1] is the target sparsity, s(.) : Rd - [0, 1] is a soften sparsity function (for example, 1- - ||// d)) that satisfies s(m) = 1 - | mmllo//d /d, for any m E {0, 13 To ensure that the model loss is evaluated binary using the mask polarizer Pe proposed in (Mo et al. 20233 Appendix A.3) 3 Thereby, an infinitesimal change of the softened sparsity becomes well-defined, while the inherent sparsity of neural pruning is preserved. As & - 0 and p - 0, (3) recovers the original neural pruning problem. making it an amenable proxy of neural pruning. In the sequel, the term "sparsity" refers to the soft sparsity s(-) without further specification. We then augment the relaxed soft pruning problem through probabilistic lifting and convexification.
sparse masks,
polarize the soft mask to be E-nearly
'The formal expression of the distributional transition d.m, = [[m]] will be introduced in Proposition]
'Lett I,& & ((0, a) [1 E, 1))", we set P.(m) é arg wngminmm'ttana [|m mll2-
4

---

Published as a conference paper at ICLR 2025
Definition (Probabilistiic Soft Neural Pruning). Following the previously introduced notations, the Probabilistic Soft Neural Pruning problem at sparsity level is defined as follows:
S Le (m)
/ <(m, m (r(dm))(dmm)
min (Le, .T + [R[m]] xEP(M))
A
+
(4)
Mf
Mr R H R4 is convex smooth kernel.
where À > is the regularization rate and n((:-) R
A practical choice of k. is the radial basis function kernel (Gretton al. 2012). i.e. s(m, m') - exp(-|| m m') /d)
Remark 1. possess a thermodynamics interpretation: each mask m represents the particle coor- dinate, т represents the particle location density, (Ire, T quantifies the external potential that acts on individual particles, and RR[=]] measures the pair-wise interaction energy, such as repulsive energy. From this perspective, neural pruning is equivalent to identifying the equilibrium of the particle sys- tem that minimizes the overall energy of the system. However, since the mask sparsity is restricted to [e, (+,++/ pl. (4) is a constrained thermodynamics system.
As noted in (Arbel et al. 2019 Lemma 25), the convexity of the kernel regularizer x(..)) implies the convexity of -. Unlike the relaxed soft pruning (3). which aims to find the optimal masks of the nonconvex objective Te (-) at sparsity level f, (4p aims to identify the distributional minimizer of
the convex functional (L + XR[:] within the sparsity range [t, + p). As À and p tends to 0, (4) recovers the original polarized soft neural pruning, in the sense that any distribution over the mixture of optimal masks that minimizes ( is optimum of (4) Therefore, (4) is indeed probabilistic extension of the relaxed soft pruning problem while inheriting its theoretical merits in continuous
sparsity analysis. Moreover, (4) benefits from the provable existence and uniqueness of its global minimizer, as demonstrated in Proposition[] The proof is detailed in Appendix D.I
Proposition 1 (Existence and Uniqueness of the Optimal Mask Distribution (informal version of Proposition ). Under some regularity conditions on model loss Le(-). the global minimizer of probabilistic soft neural pruning ( uniquely exists. This global minimizer is denoted as Tr, and referred to as the "optimal mask distribution at sparsity level 1".
4.2 SPARSITY EVOLUTIONARY
CAAKER-PLANCKKKOEE KOLMOGOROV EQUATION (SFPK)
Existing distributional gradient flow methods (Glaser et al. 2021 D' D'Angelo & Fortuin 2021 Wild cet al. (023) are primarily designed for unconstrained optimization problems and are not effective in solving the probabilistic soft neural pruning problem (4) within the feasible sparsity range of [t, + pl. Therefore, an alternative approach is necessary to solve , -'''' in a more tractable manner. As discussed in Section[]]
aim to approximate the evolution of optimal mask distribution t + Ti as the sparsity & increases from 0 to - d' /dd following SFPK introduced in 2 By gradually transforming mo (which can be well approximated by the Dirac distribution concentrated at the unpruned mask), we can achieve the desired 11-d//d-
The derivation of SFPK in 2 hinges on determining the optimal distributional transition operator TH: P (IR)) - P (Rd). In essence, T[me] is expected to push the support of ग. to be di-sparser while ensuring a minimal increase in the expected pruning objective. To obtain T [iTe. we need to estimate Te+ based on πe. To this end, we employ distributional localization technique, which involves two steps: 1) localizing the feasible region to a vicinity of тe, and 2) linearizing the functional objective of (4) at ne. Specifically, we consider candidate distributions that are reachable from Te by a local transition:
(5) Here, re O is the localization radius, and v1-l-) is transition kernel, where v(-1 (m) denotes a transition distribution conditioned on m. The convolution v Te indicates that each candidate distribution can be instantiated by m; + 8, where m, ne is the optimal probabilistic mask and 8 ~ ((-(m,)) ) is local transition conditioned on m,. The constraint in (5) ensures that 8 is a minor displacement with ||ōl|2 & Att The intersection in (5) implies that ò pushes the mask from Mf to Mf+ar. resulting in a Al-sparsity increment. Additionally, based on the functional first-order expansion (Ernzerhof
P (M - {v supp(v(-)m) E Br,s, V m E Mf) nP (ME+ar)
1994). the probabilistic pruning objective CI-] A (Le, + XR[-] can be
linearized at as:
5
C(m) C(me) +
L. me
L(Tee] + (Ie + Apmas -Te),
(6)
8
5

---

Published as a conference paper at ICLR 2025
T
Algorithm 1 SFPK-pruner
1: Input:
target model 0', target bud- get d', localization radius Tr, particle number n., simulation steps K.
Mr
7 At
2: Output: a performant binary mask m.
Made
3: 1- At +- (1-d//d//K
(m)(
m
4: Initialize m² f- 1, ,,....
5: for k = 1 to K do
(m),(m)
6: मे Unif( -- m
m
7: for = to n do
8:
mi+ t mi + T'm) R
9: end for
Illustration of the optimal local transition. Each white dot denotes mask particle, and each blue dot denotes the local optimum of Lg. The black curves denote the density function of т. and Te+ that are highly concentrated on the blue optima.
Figure 1:
<- 1 AL
11: end for
12: mn f topar (Ei m!)
13:
return i
where 35, C is the functional derivative of L at Tu, and (+) _ f(( m)T, (dm) is the k-kernel mean embedding of my- We now introduce the one-step distributional transition problem local- ized approximation of (4).
Definition (Local Distributional Transition). Using the previously introduced notations, the Local Distributional Transition problem associated with is defined as
min Le + Apta v TT, m²), s.t. V E P (MC
We refer the minimizer of 77. denoted by vi. as the optimal local transition kernel. The follow- ing proposition establishes an approximation for v4, The proof is detailed in Appendix D.1
Proposition 2 (Optimal Local Distributional Transition (informal version of Proposition presented in Appendix)). Under some regularity conditions Le(-) and s(-). the optimum of (7) is attained at ve( m) A Dirac(;; (m;; Te At) within error tolerance of order ( Let g (T.
AV, (m), A Vs(m), and re re|[s||2 1, T T'm; Te) is given by
r
SS |
T(m; me)
|
g)
I
g.
(8)
Remark 2. In particular, T(m; Te) At represents the optimal local transition at the microscope. (8) a weighted sum the ascending direction of sparsity and the descending direction of the overall energy g. where the second term is obtained by scaling the projection of g onto the orthogonal space of s. When g aligns with s, the second term becomes zero and T(m; στ) degenerates to s/ s/|s||?; otherwise, it identifies the local displacement that best aligns with g with a A/-spariity increment.
Noticeably, T(m; T.) depends both the microscopic coordinate and the macroscopic dis- tribution n, at sparsity level As illustrated in Figure ! following the optimal local transition, Tt-A: be obtained from Ty by carefully adjusting each mask in support of n; towards the near- est optimum at the sparsity level (1 + At). The readily accessible ne provides explicit guidance for transitioning from n, Tt+Ai by refining the quality assignments of microscopic masks. This process significantly reduces the intractability of solving T++Ar from scratch at a macroscopie level. As 14 * गe provides satisfactory estimation of Tt+Ai, we are now ready to derive the distributional transition [-] in ( and establish the SFPK. The proof is detailed in Appendix D.I
Equation (informal version of Proposition 6). Under some regularity conditions Le(:) and s(-), by taking At - 0, the
Proposition (Sparsity Evolutionary Bokker-Planck, Saaeeeallaaaannoooooooeee
quence } constructed by Tt+At & T. converges weakly to the Sparsity Evolutionary
Kolmoo-oo Equation (SFPK), denoted by d, -V [T(: mg), where V the divergence operator and T(; me) is the optimal local transition defined in Proposition

---

Published as a conference paper at ICLR 2025
Remark 3. Proposition 3 provides explicit guidance on minimizing the overall energy under dt- sparsity increment: at sparsity level t. the optimal local transition induces a vector field ((; my T to minimize the overall energy while arriving Mf + the change in density T (m) should equal the amount of ((: (2) Te-uulx crossing m, which is equivalent the divergence of -T(;; Te) Te at m.
4.3 SFPK-GUIDED PRUNING VIA PARTICLE SIMULATION
As discussed in Section | to conduct neural pruning, one needs to sample performant masks from T1-d' the destination distribution of the SFPK. However, despite SFPK enjoys an exact form, solving & + ne and sampling from "i-d'/d /d remain challenging. Thanks stochastic analysis theory, we show that the SFPK is associated with a McKean- McKean-Vlasov process (Veretennikov 2006 Equation 3.1), denoted by (me))ejo.1) satisfying dm, T(m,, law )m and law (m,) T (see Appendix D.2). Thereby, the task of sampling masks from T1-d' fd is reduced to a stochastic process simulation problem, which can be effectively accomplished by simulating a finite interacting mask particle system.
To achieve this, we propose the SIPK-pruner, which is a particle simulation-based probabilistic pruning algorithm. As outlined in Algorithm [ (a detailed version is presented in Algorithm and Algorithm 3 in Appendix I. the SFPK-pruner aims to simulate the evolution of t + Ty using the evolution of the uniform distribution of n interacting mask particles. At sparsity level t, each mask particle is updated following dm = ['m;; 7t dt, where मं denotes the uniform distribution over (m²) Each gradient term in (8) is evaluated based mini-batch stochastic gradient. As the sparsity increases from O to 1 - d dd the empirical distribution म. turns out to be an effective approximation of TT-d'/d: /d- Finally, we average over the empirical masks and prune the parameters associated with the smallest entries of the averaged mask. We refer the readers to Appendix A.] for a detailed complexity analysis on pruner.
In intuition, the SFPK-pruner can be interpreted as running an interacting ensemble of n mask tra- jectories with increasing sparsity. While each mask is pulled by an attraction force (blue arrows in Figure from the local optimum of the next sparsity level, the interaction energy R[.:) introduces repulsive forces (red arrows Figure D between pairs of trajectories, which encourages the trajec- tories to explore diverse optima. This exploration effect is further enhanced by the gradient noise induced by the mini-batch gradient computation. In practice, our SFPK-pruner is a model-agnostic method since it does not rely specific formulation of the model loss with respect to the model parameters and masks. Additionally, one can implement the mask variable as either filter masks, channel masks, node masks conduct structured pruning with desired granularity.
EXPERIMENTS
We evaluate our SFPK- -rrnerrumer for unstructured and structured pruning in one-shot and gradual prun- ing settings across various deep vision models. The evaluation process of each pruning method involves 3 steps: 1. Prune the well-trained target network 0 to a target sparsity level. 2. Retrain the pruned model for a fixed number of epochs until convergence, 3. Report the averaged prediction accuracy over the last few retraining epochs. The experiment settings are detailed in Appendix A.2
5.1 EXPERIMENTS ON CIFAR-100
We conduct comparison of the one-shot pruning performance of our SFPK -runner against several competitive and popular baseline methods: magnitude pruning (Han et al. 2013). SNIP (Lee al., 2019), SynFlow (Tanaka et al. 2020), GraSP (Wang et al. 2020) and PSO (Mo et al. 2023) on CIFAR-100 (Krizhevsky) 2209). We evaluate the pruning performance of each method at various sparsity levels on 3 representative models: ResNet-20 (He et al.) 2016). VGG16-bn (Simonyan & Zisserman 2015) and WRN-20 (Zagoruyko & Komodakis 2016). For all experiments, we set X 0.2, 10, and K- = 100 for SFPK-pruners.
For unstructured pruning, as shown in Figure 2 our SFPK achieves either the best or compa- rable performance across all the architectures and sparsity levels. Even when ranking the averaged masks of all parameters globally, our SFPK SFPK-pruner demonstrates significant performance improve- ments in highly sparse scenarios. This empirical evidence confirms its ability prevent layer col-
7

---

Published as a conference paper at ICLR 2025
lapse (i.e., eliminating entire layer) in extreme compression scenarios (Tanaka et al. 2020). We further compare the channel pruning performance of the baseline methods in Figure 2 Once again, our SFPK-prune demonstrates competitive performance across all three models and sparsity levels, indicating its capability to generate high-quality structured masks.
Unstructured pruning on Reskiet
Unstructured pruning on VGG16 bn
Unstructured pruning on WRN
70
Eootatoo Accuriit
53
SPPKiours]
25
20
Mag
Gratp
PSD
45
25
50%
93%
25%
27%
50%
90%
92%
95%
97%
98%
30%
93%
97%
Spansity
Spandyy
Sperdty
Stnictured pruning ResNett22
Structured pruniin on VCG16-bn
Structured pruning WAN-20
02.5
70.0
60.0
57.5
85.0
55.0
62.5
SPPKtours]
60.0
Gr
Mag
PSO
SyrFlow
SNIP
Sparsity
Figure 2: Top-1 accuracy (%) of one-shot pruning on CIFAR-100 over 3 random runs.
Table 1: We report the top-1 accuracy (%) of one-shot unstructured pruning without any retraining on ImageNet- K. The top-1 accuracy of the unpruned models is reported in the second row. Boldface indicates the highest top-1 accuracy of each sparsity level.
Model
Metile V1 (72.00)
ResNet 50 (77.01)
Sparsity
Mag
SNIP
SynFlow
PSO
SFPK (ours)
Mag
SNIP
SynFlow
WF
PSO
SFPK (ours)
75%
49.58
4.08
0.06
60.95
48.47
61.57
58.25
5.87
1.08
67.02
67.04
69.69
80%
0.10
56.71
42.97
1.54
58.72
63.38
84%
19,40
0.44
0.09
41.10
31.61
$1.31
24.03
0.68
0.34
46.75
58.98
63.38
87%
5.77
0,29
0.11
25.21
27.30
47.40
9.20
0.36
0.19
32.21
54.25
59.71
90%
0.72
0.18
0,11
12.56
20.53
44.06
3.72
0.39
0.15
17.48
49.08
55.48
5.2
EXPERIMENTS ON IMAGENET-IK
One-shot Pruning. We compare the one-shot unstructured pruning effectiveness of our SFPK- pruner against two strong baselines, magnitude pruning (Han et al. 2015) and WoodFisher (denoted by WF) (Singh & Alistarh, 2020), on the ImageNet-IK benchmark (Deng et al. 20099) using two representative architectures: ResNet-50 (He et al.] 2016) and MobileNet-Vi (Howard et al. 2017). For each baseline method, we directly prune the model each density level without any retraining and then report the resulting top-1 accuracy. For SFPK-pruner, we only apply it once to generate n mask trajectories terminating at 90% sparsity. We then collect intermediate checkpoints of these mask trajectories at various sparsity levels and prune the model accordingly based on the averaged mask at each intermediate sparsity level without any retraining. We ran the -prrner on all the experiments with À 0.2. consistently outperforms other baselines at all sparsity levels. This validates the effectiveness of our proposed SFPK in approximating the sparsity evolution of the optimal mask distribution. Notably, the computation cost of running the SFPK-prumn -rrKeernner with batch size of 256 for n x K = 1500 steps is equivalent to that of only 0.3 epoch of retraining on ImageNet. Thus, the partiele simulation-basee
10, and K 150. As shown in Table [ our SFPK-pruner
8

---

Published as a conference paper at ICLR 2025
SFPK-pruned enables to sample performant masks with modest number of particles, avoiding excessive overhead.
We also evaluate the one-shot structured channel pruning performance of our SFPK- pruner the munsformer-basee DeiT-T architecture (Touvron et al., 2021) against the state-of-the-an pruned transformer, including SCOP (Tang et al., 2020), HVT (Pan al. 2021), UVC (Yu al. 2022b), WDPruning (Yu et al. 2022a), and X-Pruner Yu & Xiang 2023). We apply the SFPK-pruner to prune the model to 50% sparsity with À 0.2,
10, K 1000. Then, retrain the pruned model for 100 epochs following Yu & Xiang, 2023). As shown in Table 2a our SFPK- SFPK-pruner achieves competitive top-1 accuracy compared baseline methods within comparable FLOP bud- get. This showcases the efficacy of our SFPK -rrunerr in structured pruning for transformers.
Table 2: For each experiment, report the top-1 accuracy (%) of the pruned model. In Table 2b and Table 2c we further report the relative performance drop (%) compared to the dense model.
(a) One-shot structured pruning on DeiT-T.
Gradual pruning on ResNet-50.
I
One-shot Structured Pruning
ImageNet-1K
Graduul Unstructured Pruning
DeiT-T
Acc@1
Acc@5
FLOPs (%)
ResNet-50
Dense
Pruned
Rel. Drop
Sparsity
Unpruned
72.2
91.1
100.0
DSR
74.90
71.60
-4.41
80.00
SCOP
68.9
89.0
61.5
Incremental
75.95
74.25
-2.24
73.50
DPF
75.95
75.13
-1.08
79.90
HVT
69.7
89.4
53.8
PSO
69.9
89.4
52.1
GMP + LS
76.69
75.58
-1.45
79.90
WDPruning
70.3
89.8
53.8
VD
76.69
75.28
-1.84
80.00
UVC
70.6
39.1
RIGL + ERK
76.80
75.10
-2.21
80.00
X-Pruner
71.1
90.1
49.2
SNFS + LS
77.00
74,90
-2.73
80.00
SFPK (ours)
71.6
90.3
51.7
STR
77.01
76.19
-1.06
79.55
PSO
77.01
75.84
-1.52
80.00
DNW
77.50
76.20
-1.68
80.00
(b) Gradual pruning on MobileNet-V V1.
77.01
76.76
80.00
ImageNet-IK IK
Gradual Unstructured Pruning
CrAM
77.3
75.80
90.00
SEPK (ours)
77.01
76.18
80.00
MobileNet.V VI
Dense
Pruned
Rel. Drop
Spursity
Mag
77.01
75.15
-2.22
90.00
Incremental
70.60
67.60
.4.25
74.11
STR
72.00
68.35
-5.07
75.28
GMP + LS
76.69
73.91
-3.62
90.00
Mag
72.00
69.90
-2.92
75.28
76.69
73.84
-3.72
90.27
RIGL + ERK
76.80
73.00
-4.95
90.00
WF
72.00
70.09
-2.65
75.28
SNFS LS
77.00
72.90
-5.32
90.00
SEPK (ours)
STR
77.01
74.31
-3.51
90.23
Incrementall
70.60
61.80
-12.46
89.03
PSO
74.63
-3.09
90.00
STR
72.00
62,10
-13.75
89.01
DNW
77.50
74.00
-4.52
90.00
Mag
72.00
63.02
-12.47
89.00
PSO
72.00
63.64
-11.61
90.00
WF
77.01
75.21
-2.34
90.00
WF
72.00
63.87
-11.29
89.00
CIAM
77.3
74.70
-4.66
90.00
SFPK (ours)
72.00
64,10
-10.97
90.00
SFPK (ours)
77.01
75.24
-2.30
90.00
Gradual Pruning. To further demonstrate the effectiveness of our SFPK-pruner, evaluate its gradual pruning performance on the pre-trained ResNet-50 and MobileNet-V models provided in (Kusupati al., 2020). We apply the PrPKerruner to gradually prune the model to 90% sparsity within 10 prune-and-retrain loops. Specifically, the target sparsity and the number of retraining epochs of each pruning shot increase exponentially, while the total retraining epochs
kept 100
to ensure fair comparison with other baseline methods including Mag (Han et al. 2015), Incremen-
tal (Zhu & Gupta 2018b), DPF (Lin et al. 2020), GMP+LS (Gale al.] 2019). VD Molchanov
ct al., 2017b), RIGL +IGL (Evci et al.] 2020). SNFS+LS (Dettmers & Zettlemoyer (020). STR (Kusupati et al. 2020). DNW (Wortsman et al. 2019, WF (Singh Alistarh (020). PSO (Mo) et al. 2023), and CrAM (Peste et al. 2023). We refer the reader to Appendix [C] for additional ex- periments under different training budgets and pretrained checkpoints. We fix
0.2, n = 10,
K = 150 for each pruning shot. In Table 2b and Table 2c our SFPK-pruner demonstrates decent performance compared to various competitive pruning baselines. Notably, the intermediate sparsity undergo fewer retraining epochs (as the total retrain epochs are
checkpoints of our SFPK-prum fixed to 100) than other baseline methods. This validates the effectiveness of our SFPK guiding the gradual sparsification of the model.

---

Published as a conference paper at ICLR 2025
Ablation on the Regularization Rate
Ablation on the SFPK Simulation Scheme
72.5
56.6
72.0
56.4
Top-1 Accuracy (%)
71.0
56.0
70.5
# mask trajectories
10
55.6
63.5
-+.
20
0.0
0.004
0.008
0.01
10.0
50
# SFPKP Simulation Stops
Regularization
Figure 3: Top-1 accuracy (%) for each experiment. The grey horizontal dashed line indicates the
performance of iterative magnitude pruner (Zimmer et al. 2022) under the same settings.
5.3 ABLATION STUDIES
Ablation on the Regularization Rate X. As shown in Definition 1 the convexification technique inevitably introduces small gap proportional À between the original pruning objective and the convexified one. To test the sensitivity of JFPK-pruner" performance to X. we implement the one- shot SFPK-pruner with different values to compress a pretrained ResNet-20 to 95% sparsity on CIFAR-100. Figure 3 (top) shows that our SFPK-pruner is robust against various À values and performs well even with small A. We hypothesize that this is because we use mini-batch stochastic gradients to approximate the gradient terms in (8) implicitly introducing gradient noise that acts as a repulsive random force, serving as surrogate for the convex regularizer.
Ablation the Mask Particle Simulation Scheme. According Algorithm 1 SFPK-prunee relies on numerically simulating the stochastic process dm, = (m, law(m;)) using n interact- ing mask particles. The discretization of the sparsity horizon (controlled by simulation steps K) and the spatial horizon (controlled by n) are crucial to SFPK's performance. We apply a one-shot SFPK-pruner with various (K, n) values to compress pretrained VGG16-bn to 98% sparsity CIFAR- 100. Figure (bottom) shows that larger (K, n) leads to finer simulation resolution and higher performance. In practice, our SFPK-pruner effectively approximates the optimal mask distri- bution's 's sparsity evolution with > 4 and K > 50.
CONCLUSIONS
In this paper, analyze the dynamics of the population of optimal neural pruning masks sparsity increases. We propose the Sparsity Evolutionary Fokker-Planck- Porkeogo Equation (SFPK) as approximation for the evolution of mask distribution. Additionally, pruning method called SFPK-pruner, which uses an interacting mask particle system to sample performant masks desired sparsity levels. The acceleration and scaling of SFPK-pruner for larger models will be addressed in future work.
introduce probabilistic
ACKNOWLEDGEMENT
The research work described in this paper conducted in the JC STEM Lab of Machine Learning and Symbolic Reasoning funded by The Hong Kong Jockey Club Charities Trust. Sinno Pan also thanks the support of the Microsoft Research Asia collaborative research grant.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
I. Ambrosio, N. Gigli, and G. Savare. Gradient Flows: In Metric Spaces and in the Space of Probability Measures. Lectures in Mathematics. ETH Zürich. Birkhäuser Basel, 2006. ISBN 9783764373092. URL htttps://books.googr com www..gg/bookk wtt.//ooosiidds. wNp0sc4gC
Michael Arbel, Anna Korba, Adil SALIM, and Arthur Gretton. Maximum mean discrepancy gradient flow. Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL
neurips ctppa//..p.rrrrr...cpppppee files/ SBpp://lfAAA1ASaaeeeeeeeeeeeaaer
Sourav Bhattacharya and Nicholas D Lane. Sparsification and separation of deep learning layers for constrained resource inference on wearables. Proceedings of the 14th ACM Conference on Embedded Network Sensor Systems CD-ROM, pp. 186-189, 2016.
Billingsley. Convergence of Probability Measures. Wiley Series in Probability and Mathematical goollo e. com. sg/
Statistics. Wiley, 1968. ISBN 9780471072423. URL bookks?1-+990AAAAAAA
V.I. Bogachev, N.V. Krylov, M. Röckner, and S.V. Shaposhnikov. Fukker-Planck-Kolmene Equations. Mathematical Surveys and Monographs. American Mathematical Society,
2015. ISBN 9781470425586. URL https:://oooh www.ge..comcoomooooo..oooommoo. http:/lee....sssoooeoee.....
e
2 (FYCCAAAQAJ
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Jiasi Chen and Xukan Ran. Deep learning with edge computing: A review. Proceedings of the IEEE, 107(8):1655-1674 2019.
Shangyu Chen, Wenya Wang, and Sinno Jialin Pan. Metaquant: Learning quan- tize by learning to penetrate non-differentiable quantization. In Advances in Neu-
ral Information Processing Systems, volume 32. Curran Associates, Inc.. 2019. URL www.sre.ersspppppp httpcc/pppppeeee /iles//pppe//21///12.
Tianyi Chen, Bo Ji, Tianyu DING, Biyi Fang. Guanyi Wang, Zhihui Zhu, Luming Liang, Yixin Shi, Sheng Yi, and Xiao Tu. Only train once: A one-shot neural network training and pruning framework. In Advances in Neural Information Processing Systems, 2021. URL https:// openreview net/ waret//ooummmdpp....
jreCzq
Moonseok Choi, Hyungi Lee, Giung Nam, and Juho Lee. Sparse weight averaging with multiple particles for iterative magnitude pruning. In The Twelfth International Conference on Learning Representations, 2024. URL https: http:///pprreeeeeeeeeeeeeeeeeeee wttp::/wwwweteeeeeeeeeeeee tteet//frrumii--/////e0
/
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: training deep neural networks with binary weights during propagations. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 2, NIPS' 15, pp. 3123-3131, Cambridge, MA, USA, 2015. MIT Press.
Francesco D' Angelo and Vincent Fortuin. Repulsive deep ensembles are bayesian. Advances in Neural Information Processing Systems, 2021. URL URL/https:/ httpthsss///peeeeiiieeeeeee
forum??
Ed=LAKp] PLMbP8
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hier- archical image database. 2009 JEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10.1 (09//VPR.20099550663
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance, 2020. htw.httess//ppppprrrrreeeee/eeer

---

Published as a conference paper at ICLR 2025
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL
http: wwtg/rrrgaiaieeee abs 1810 04805
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: / //openrevi ew.
d=Yi cbF cNTTY
Matthias Ernzerhof. Taylor-series expansion of density functionals. Phys. Rev. A, 50:4593-4607, Dec 1994. doi: 10. 00.100//hyyReev50.4599.. URL https: /ttps::///ink...// aps htttps///inxx.as.ooo///d// apt.//.///...//////1//.10.
I103/PhysRevA. 4593
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learming Research, pp. 2943-2952. PMLR, 13-18 Jul 2020. URL
html
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference Learning Representations, 2019. URL https:// openreview net/ www.:rorummmiamme bbRcF3
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural net- works. ArXiv, 199/199,,,9574, 2019. URL https://api /tttp:::/api.semaatt www.mmattccchooooo org/
CorpusID:
Pierre Glaser, Michael Arbel, and Arthur Gretton. KALE flow: A relaxed KL gradient flow for probabilities with disjoint support. In Advances in Neurul Information Processing Systems, 2021.
URL htt.thtpss//ppeeeeiiiiiie
net /wtt/eouumiie wowoooodddeeeeeoeioooe BBeCV 10a
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola.
A kemel two-sample test. Journal of Machine Learning Research, 13(25):723-773, 2012. URL
http://3mlr wwwpopppppppp..p......tp.....rrt pappeers/vi33. httpa////v3//getooee3emm
html
T. H. Gronwall. Note on the derivatives with respect to a parameter of the solutions of system of differential equations. Annals of Mathematics, 20(4):992-296, 1919. ISSN 0003486X. URL
http:// //ww. jstorr org/ aatttaelleee1122
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, volume 28, Cur- ran Associates, Inc., 2015. URL http:/tss///pccccaaiann///////
neurips. cctpapee/pcc55///////////// ppdf
f hat.aeac..ecc33.a.a.ee2222222033
Song Han, Junlong Kang, Huizi Mao, Yiming Hu, Xin Li, Yubin Li, Dongliang Xie, Hong Luo, Song Yao, Yu Wang, al. Ese: Efficient speech recognition engine with sparse Istm on fpga. In Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmann Gate Arrays, pp. 75 84, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. 2019 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 4335-4344, 2018 URL https://app.
semanticscholas http:/hho..aao.ooooooooroaaaaaa.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CORR, (bb//7744008611, 2017. URL battp://arwe org/ abs/ T704 04861
12

---

Published as a conference paper at ICLR 2025
Hengyuan Hu, Rui Peng, Yu- Yu-Wing Tai, and Chi-Keumi Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03255) 2016.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL
nww..ig...eueppppp cc/
L.P. Kadanoff. Statistical Physics: Statics, Dynamics and Renormalization. Statistical Physies: Statics, Dynamics and Renormalization. World Scientific, 2000. ISBN 9789810237646. URL https::/booke: google com. wwwpaggg/booaeeeeeeo 22dadF5p6yYy
Jeremias Knoblauch, Jack Jewson, and Theodoros Damoulas. An optimization-centric view on bayes' rule: reviewing and generalizing variational inference. J. Mach. Learn. Res., 23(1), jan 2022. ISSN 1532-4435.
Alex Krizhevsky. Learning multiple layers of features from tiny images. pp. 32-33, 2009. httpps://an wwwww cs. toronto........... edu/ wtarrii/eeannngg- featurrrreeeesseeee 200.rrsR--00-9.pp5
2009. URL.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceed-
ings of Machine Learning Research, pp. 5544-5555, PMLR, 13-18 Jul 2020. URL https:
wwwdiinngsmml......mmm .ml.. wwwppprssssii.. prrp http:///wlllkkkupaaataaa0.0 html
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: SINGLE-SHOT NETWORK
PRUNING BASED ON CONNECTION SENSITIVITY. In International Conference on Learn- ing Representations, 2019, URL pttettps:///oenreviieee n forum -id=B1VZ
Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In International Conference on Learning Representations, 2020. URL https: //openreview. htte://ooommm.dSSemmmmee.e.
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational bayesian neural networks. Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, pp. 2218 12277222222222 JMLR.org, 2017.
Christos Louizos, Karen Ullrich, and Max Welling. Bayesian compression for deep learning. Ad- vances in neural information processing systems, 30, 2017.
Christos Louizos, Max Welling, and Diederik P. Kingma. Leaniing sparse neural networks through
10 regularizattin. In International Conference on Learning Representations, 2018. URLhttpss
// //openreview.o wwt. wwwp://oomm..ddl1oeaa0
Ekdeep Singh Lubana and Robert Dick. A gradient flow framework for analyzing network pruning. In International Conference Learning Representations, 2021. URL https:
//openreview.
wettooommmidrrrre 1aamv7QQmm 7QmLUue)
Zhanfeng Mo, Haosen Shi, and Sinno Jialin Pan. Pruning via sparsity-indexed ODE: continuous sparsity viewpoint. In International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings Machine Learning Research, pp. 25018-25036. PMLR, 2023. URL wttphttps//. http:ttts/////rreeeiing wttp:..ppppeee////////22222222 html
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In International conference on machine learning, pp. 2498-2507. PMLR, 201 7a.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2498-2507. PMLR, 06-11 -1 Aug 2017b. URL https:://prosee i ml www.s.pressee.. wttess/vv0/mme chanovi 7a. html
13

---

Published as a conference paper at ICLR 2025
Ari Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: generalizing lottery ticket initializations across datasets and optimiz- In Advances in Neural Information Processing Systems, volume 32. Curran Asso-
ciates, Inc., 2019. URL http:ttss/p:psseeeetetst wew.einnnn.neeeeeeeeeeeeeee neurips htt.gg.....rrrps....papeee/////e pctpppaarr//21////1lll 244.1688772221112266222000000000 Paaper, .dff
Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Structured bayesian pruning via log-normal multiplicative noise. Advances in Neural Information Processing Systems, 30, 2017.
Zizheng Pan, Bohan Zhuang, Jing Liu, Haoyu He, and Jianfei Cai. Scalable vision transformers with hierarchical pooling. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 377-386, October 2021.
Christos H Papadimitriou and Kenneth Steiglitz. Combinatorial optimization: algorithms and com- plexity. Courier Corporation, 1998.
Alexandra Peste, Adrian Vladu, Eldar Kurtie, Christoph H Lampert, and Dan Alistarh. CrAM: compressionawware minimizer. In The Eleventh International Conference Learning Represen- tations, 2023, URL
net/ forum ?id- eT2Bs- 1ea8s-yye yeerr
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantization. In International Conference on Learning Representations, 2018. URL https:
John Rachwan, Daniel Zügner, Bertrand Charpentier, Simon Geisler, Morgane Ayle, and Stephan Günnemann. Winning the lottery ahead of time: Efficient early network pruning. In Pro- ceedings of the 39th International Conference on Machine Learning, volume 162 of Proceed- ings of Machine Learning Research, pp. 18293-18309. PMLR, 17-23 Jul 2022. URL https:
pppoocediiggs. ml 2 http://ar22rraaaaa2222222 html
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 10428-10436, 2020.
Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 779-788. IEEE Computer Society, 2016. doi: 10. N..119//CVR..201.999 URL httph:ttp///omm /ttp:/:///ooooo///////////////// /00 109/CVPR. [2016. 6. 91
Simón Rodríguez Santana, Bryan Zaldivar, and Daniel Hern' andaez- Function-space infer- ence with sparse implicit processes. International Conference on Machine Leurning, 2021. URL htl.https://aaag
deeannicsshoo coscholarr ar pttp//oouuuTT1::22511111
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015.
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. In Proceedings of the 34th International Conference on Neural Informa- tion Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc ISBN 9781713829546.
Suraj Srinivas and R. Venkatesh Babu. Data-free parameter pruning for deep neural networks. CORR, org. absh//bs//11/0
abs/1507.061499 2015. URL
06149 06149
Kai Sheng Tai, Taipeng Tian, and Ser-Nam Lim. Spartan: Differentiable Sparsity via Regularized Transportation. In Advances in Neural Information Processing Systems, 2022.
Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Pro- cessing Systems, 33:6377-6389, 2020.
14

---

Published as a conference paper at ICLR 2025
Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang Xu. Scop: scientific control for reliable neural network pruning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers amp; distillation through attention. In Proceedings of the 38th International Conference Machine Learning. volume 139 of Pro- ceedings of Machine Learning Research, pp. 10347-10357. PMLR, 18-24 Jul 2021. URL ml
htttps://proceedingg wtt.:s.w......r....mmmm v http///v/.v3v/tveeeeeeeea twwvron2a....a..ooooo htmmmmmma..mmmm..m
A. Yu. Veretennikov. On ergodic measures for mckean-vlasov stochastic equations. In Monte Carlo and Quasi-Monte Carlo Methods 2004, pp. 471-486, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg. ISBN 978-3-540-31188-7
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. International Conference on Learning Representations, 2020. URL.
ltttps://openreviit net/
Veit D. Wild, Robert Hu, and Dino Sejdinovic. Generalized variational inference in function spaces: Gaussian measures meet bayesian deep learning. In Proceedings of the 36th International Con- ference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 9781713871088.
Veit David Wild, Sahra Ghalebikesabi, Dino Sejdinovic, and Jeremias Knoblauch. A rigorous link between deep ensembles and (variational) bayesian methods. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
id=eTHawKFT47
Mitchell Wortsman, Ali Farhadi, and Mohammad Rastegari. Discovering neural wirings. In Neu- ral Information Processing Systems, 2019. URL httpp:://app...mmmttieeeeee.eeee www.nteeatttttsooe icscholar.. org/
CorpusID: 1739901:119666
J. Wu, C. Leng, Y. Wang, Q. Hu, and J. Cheng. Quantized convolutional neural networks for mobile devices. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 4820-4828, Los Alamitos, CA, USA, jun 2016. IEEE Computer Society. doi: 10.1109/ 201 2016.5211
Utt.httss://ddett htt.r/eeoootttrrrrr......oooooo. 1100//////...
Shuochao Yao, Yiran Zhao, Aston Zhang, Lu Su, and Tarek Abdelzaher. Deepiot: Compressing deep neural network structures for sensing systems with a compressor critic framework. Proceedings of the 15th ACM conference on embedded network sensor systems, pp. 1-14, 2017.
Fang Yu, Kun Huang, Meng Wang, Yuan Cheng, Wei Chu, and Li Cui. Width amp; depth pruning for vision transformers. Proceedings of the AAAI Conference on Artificial Intelligence, 36(3): 3143-3151, Jun. 2022a. doi: 10.1609/aaai, (233,,2222222 URL Skttp:://o]) .acaaioorr/// index. phtp://At/arriiie////eee///////a
Lu Yu and Wei Xiang- X-pruner: explainable pruning for vision transformers. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24355-24366 2023. URI.
Shixing Yu, Tianlong Chen, Jiayi Shen, Huan Yuan, Jianchao Tan, Sen Yang, Ji Liu, and Zhangyang Wang. Unified visual transformer compression. International Conference on Learning Repre- sentations, 2022b. URL httthtttt:://oeeet
ew net. www.:nte//ooommmmd3/3emmm i UgkCZP
Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al. Mest: Accurate and fast memory eccoomie sparse training framework on the edge. Advances in Neural Information Processing Systems, 34, 2021.
Sergey Zagoruyko and Nikos Komodakis, Wide residual networks, 2016. URL https: //arxiiv. brg/abs 1655. 07146
15

---

Published as a conference paper at ICLR 2025
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, volume 28. Curran Associates. Inc., 2015. URL http::////orreeeednnnss.eeeeeiia c
files/ pddf
Chenglong Zhao, Bingbing Ni, Jian Zhang, Qiwei Zhao, Wenjun Zhang, and Qi Tian. Variational convolutional neural network pruning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2780-2789, 2019.
Michael H. Zhu and Suyog Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression. In International Conference on Learning Reprexentations Workshop, 2018a. URL
Michael H. Zhu and Suyog Gupta. To prune, or not prume: Exploring the efficacy of pruning for model compression, 2018b. 8b. htattsss//peenreriieeentntt//oee
Tao Zhuang, Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng, Kai Shuang, and Xiang Li. Neuron-level structured pruning using polarization regularizer. In Advances in Neural Information Processing Systems, volume 33, pp. 9865 9877. Curran Asso-
ciates, Inc., 2020. URL http/ttps//pppoeediinsssneeet newrreeeeeddnngg...eissssss.s.s http://g....ooooo.o...opp//////a (ttps//dddeeeeeeeeeeeeeeeeeeeeee
Pap.ssaapppppppp. pdf
Max Zimmer, Sebastian Pokutta, and Christoph Spiegel. Back basics: Efficient network com- pression via imp, 2022. URL httppttts////pteeeeee wttp:epwwwweieeeeeeeeeeeeeeeeeee
Liu Ziyin and Zihao Wang. spred: Solving 11 penalty with SGD. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Pro- ceedings of Machine Learning Research, pp. 43407 43422. PMLR, 23-29 Jul 2023. URL
mlr, wttp://gw......prr/////////////a http://222zzziii2222 html
16

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
A.1 IMPLEMENTATION DETAILS OF SFPK-PRUNER
Implementation of SFPK-pruner. A detailed implementation of the SFPK-prunnr (Algorithm D A PyTorch implementation is attached in Algorithm 3
is shown in Algorithm 2
Notably, the mini-batch gradients involved in Algorithm M are crucial for mask particles' simulation. In theory, the SFPK dynamic should be initialized at тo. the optimal mask distribution within My. Since p > Ti is close to, but not identical to, the delta distribution centered on the fully dense mask. In practice, we approximate x by running SFPK with stochastic mini-batch gradient from the delta distribution at the fully dense model. During the SFPK partiele simulation process, the stochasticity of the mini-batch gradients enhances robustness and stability in sampling, while also encouraging the exploration of diverse mask particles.
Complexity analysis. As shown in Algorithm 3 the extra computation is SFPK-prune (). It consists of n x K inner SFPK steps. Each SFPK step includes one bawkwdd pass on a mini-batch (to calculate g) and the computation of sparsity gradient s, transition t crans, and mask update. Let d be the number of parameters of the model and s(m) = 1 - ||m]/// / be the soft sparsity function, can be computed in closed-form in (d). Computing t rans evolves only inner-products with ((d) complexity, and the mask update also has a complexity of (d). The newarddbackward pass dominates the cost of one SFPK step. The cost of one SFPK-prune call equals the cost of "training the model for n x K mini-batches" plus a marginal (n inner- product computation.
In our ImageNet experiments, we set = 150, K = 10, and batch-size = 256. Thus, each () call is as cheap as 1, 500 training steps, which equals 0.3 epochs (1 epoch of ImageNet has 1, 281, 167/256 = 5000 batches). As we perform 10 prune-and-retrain iterations, the total SFPK overhead as inexpensive as merely 0.3 x 10 3 additional training epochs. This marginal when compared to the retraining
SFPK.prune
of the baseline. In Table 12 Table [4]
and Table 16]
we compare the training and inference FLOPs of SFPK with other baselines.
Constraints of SPFK-prumer. Our SFPK works effectively when the target model is "rational," meaning pruning the model usually increases the loss (i.e. harms the performance) rather than decreases it (i.c. improves the performance). For instance, adversarially trained model (c.g., trained to maximize task loss) is considered "irrational", as nullifying some parameters may reduce the task loss. In this case, applying SFPK to this model can still progressively improve its accuracy as sparsity increases, because the SFPK update (8) always pushes the particles towards smaller loss. However, the SFPK trajectory may NOT converge to the true optimal mask distribution dynamic I (++ Tr. which may lead to suboptimal results.
In intuition, this is because the dense mask mask across all sparsity levels, and it fails to guide the distributional transition of the optimal mask. Therefore, the SFPK trajectory may diverge from + n, it gets lost from the beginning In theory. SFPK might fail to approximate f + as the assumptions (AI) and (A2) in Proposition 4 are violated. Hence, the convergence theorem no longer holds. We recommend applying SFPK to sufficiently trained "rational" models (e.g. trained to convergence), where the dense mask is likely to be close to the optimal mask across all sparsity levels.
(1,..., 1) in this scenario is no longer the optimal
O.
A.2 IMPLEMENTATION OF PRUNING EXPERIMENTS
Unstructured and Structured Pruning, The only difference between the unstructured and struc- tured pruning is the granularity of pruning. unstructured pruning, each parameter is prunable. In contrast, structured pruning targets substructures of the neural network, such as all the weights associated with kernel, all the weights associated with a channel, or all the weights associated with an attention head. In this paper, we only consider structured channel pruning, where the weights bonded to a particular channel are governed by a single pruning mask. For pruming methods initially devised for unstructured pruning, we adapt them to the structured pruning scenario by calculating the importance score of a channel based on the sum of the importance scores of all its associated weights.
17

---

Published as a conference paper at ICLR 2025
One-shot and Gradual Pruning. We consider pruning shot that involves three steps: 1. Prune the target model 0 to the target sparsity level 2. Retrain the pruned model for N epochs to regain performance. 3. Evaluate and report the top-1 predictive accuracy of the retrained model. Given target sparsity level i, the one-shot pruning procedure involves single pruning step. In contrast, the gradual pruning procedure consists of multiple pruning steps, with each step compressing the model toward a pre-specified intermediate sparsity level until the target sparsity is achieved. These intermediate sparsity levels represent the sparsity schedule. We use exponential sparsity schedule across all the gradual pruning experiments: for T-shot gradual pruning experiment, the i-th inter- mediate sparsity level g&/K, Thus, the configuration of one-shot or gradual pruning experiment characterized by the pruning method and the retraining scheme, which includes the epoch amount, learning rate schedule, batch size, and optimizer. The configurations of all pruning experiments are claborated in Table TableBaand Table 4
B FULL EXPERIMENT RESULTS
In this section, present the complete experimental results from Section 5 including error bars representing standard deviations, along with results from additional settings. All reported results based on independent trials, following the experimental setup outlined in Appendix A.2 The standard deviations of the gradual pruning results of our SFPK- -prnner are reported in Table
Table 5: For each experiment, we report the top-1 accuracy (%) of the pruned model. In Table 5b and Table 5cc| we further report the relative performance drop (%) compared the dense model.
(a) One-shot structured pruning on DeiT-T.
(c) Gradual pruning on ResNet-50.
ImageNet-IK
Onc-shot Structured Pruning
Image Net-
Gradual Unstructured Pruning
DeiT-T
Acc@1
Ace@5
FLOPs
ResNet-50
Dense
Pruned
Rel. Drop
Sparsity
Unprumed
72.2
91.1
100.0
DSR
74.90
71.60
4.41
80.00
SCOP
68.9
89.0
61.5
Incremental
75.95
74.25
-2.24
73.50
HVT
69.7
89.4
53.8
DPF
75.95
75.13
-1.08
79.90
PSO
69.9
89.4
GMP LS
75.58
-1.45
79.90
WDPruning
70.3
89.8
53.8
VD
76.69
75.28
-1.84
80.00
UVC
70.6
39.1
RIGL ERK
76.80
75.10
-2,21
80.00
X-Pruner
71.1
90.1
49.2
SNFS + LS
77.00
74.90
-2.73
80.00
(ours)
71.6 (0.2)
90.3 (0.1)
76.19
79.55
PSO
77.01
75.84
-1.52
80.00
DNW
77.50
76.20
-1.68
80.00
(b) Gradual pruning on MobileNet-VL.
WF
77.01
76.76
-0.32
80.00
CrAM
75.80
-1,94
ImageNes-1K IK
Gradual Unstructuced Pruning
SFPK (ours)
77.01
76.18 (0.13)
-1.26 (0.14)
80.00
MobileNa-V1
Donse
Pruned
Rel. Drop
Spursity
Mag
77.01
75.15
-2,42
90.00
70.60
-425
GMP II
76.69
73.91
-3.62
90.00
STR
72.00
68.35
-5.07
75.28
VD
76.69
73.84
-3.72
90.27
Mag
72.00
69.90
-2.92
75.28
PSO
72.00
69.16
-3.95
75.28
RIGL ERK
76.80
73.00
90.00
72.00
SNFS IS
77.00
72.90
-5.32
90.00
SFPK (ours)
72:00
69.79 (0.24)
-3.07 (0.29)
75.00
STR
77.01
74.31
-3.51
90.23
Incremental
70.60
61.80
-12.46
89.03
PSO
77.01
74.63
-3.09
90.00
STR
72.00
62.10
-13.75
89.01
DNW
77.50
74.00
-4.52
90.00
Mag
-12.47
WF
77.01
75.21
-2.34
90.00
WF
PSO
72,00
72.00
63.64
63.87
-11.29
-11.61
89.00
90.00
CrAM
77.3
74.70
-4,66
90.00
SFPK
72.00
64.10
-10.97 (0.19)
90.00
SFPK (ours)
77.01
75.24 (0.09)
-2.30 (0.09)
90.00
One-shot pruning on ImageNet-1K. Following the experimental setup in Section 5.2 we perform one-shot unstructured pruning experiments on the ImageNet- (Deng et al. 2009) dataset using ResNet-50 (He et 2016), MobileNet-Vi (Howard et al. (017). and DeiT-T' TTouvron et al. 2021) architectures. We compare our SFPK JFPKerr against several strong baselines, including magnitude pruning (Han et al. 2015). SNIP (Lee et al. 2019), (Tanaka et al. 2020), WoodFisher (Singh & Alistarh 2020), and PSO Mo et al.) 2023). As shown in Table 6 our SFPK-pruner exhibits high performance across various architectures. Our experiment results validate that SFPK effectively approximates the sparsity evolution of the optimal mask distribution, enabling us to sample high- quality masks within acceptable overhead.
18

---

Published as a conference paper at ICLR 2025
Algorithm 2 SFPP--guided Pruning via Particle Simulation (SFPK-prunnr)
1: Input: target budget d", dataset S, batch size b, target model parameterized by f(: 0*). model loss .). soft sparsity function s(-), RBF kernel k(,, +), mask polarizer (.)... localization radius rs. particle number simulation steps K, regularization rate A.
2: Output: performant binary mask m E {0, 1 with | d'.
3: t + 0, Af < (1 d' /d)/K.
4: Initialize m 1, 1..,m.
5: for k = 1 to K do
for
do
7:
+ {mmm))
yx + MiniiBa
9:
g -0 1.,( ((())
o 00)) A و 0 m3)
7(m;) + s/||=||3 ((ri - 1) (1s|lăll@ (s {s))))))* ) (B ( (gTs\/||||
10:
m i f- mi + At
12: end for
13: t + At
14: end for
15: i + top (Ein m))
16:
return m
Algorithm 3 SFPK pruner PyTorch
def in (model, t, dataloadar, +*aggs):
model: masked dense model
8 t: target sparsity
# m: prening steps
$ args: SPPK proning args
for in range (m)
t_tmp i_th_sparsity(t, i) gradual praning schedule
model basel lesseline_pprmme fmodel, t_top, dataloader, **args)
model SPPK_prune (model, t_tmp, dataloader, **args)
model retrain (model)
return model
STPK_prune prune (model, t, dataloaade,, n, r):
model: masked model
t: tazget sparsity
$ n: particle number
$ K: simulation steps
r: local radius
eps 1e-18
masks init_masks (model,
sparsity (model)
for 1 in range (K)
for mask in nasks:
cale_s (mask) cale grad of soft sparsity (1 norm (mask)) explicitly x, y next (dataloader)
reg cale_kappa (mask, masks) 8 cale interaction penalty coss_fn(madel mask, x, y) reg g grass, mask) cale grad w.r.t mask
cale transition
gap (norm(s) norm(g)) * 2 inner__prod
+*
if gap > eps:
Lermi - -8 norm(s) ..
1) gap) 0.5 term2 scale (g inner Lnner_prodde, g) terml)
scale (((r norm(s))
trans terml term2
else:
term? equals if g is parallel to trans norm(s) **
mask trans dt SEPK update return prune (model, (masks))
19

---

Published as a conference paper at ICLR 2025
Table 3: Experiment settings of one-shot pruning CIFAR-100. Here, "Ir" denotes "learning rate", and "Cos" denotes the cosine annealing learning rate schedule, with an initial learning rate of 1e- 4 and 5 warm-up epochs. We follow the implementation of mask polarizer as (Mo et al. 2023 Algorithm 3). Each run takes around 20 to 30 minutes on NVIDIA 40G A100 GPU, with each taking approximately 0.2 seconds.
SFPK update step (line 7 to line I of Algorithm 2
CIFAR- 100
Structured
Unstructured
Model
ResNet-20 70.38
VGG16-bn 75.68
WRN-20
ResNet-20 70.38
VGGI6-bn 75.68
WRN-20
Top-1 accuracy
75.22
75.22
Batch size Epochs
64
64
64
64
64
64
Retrain sctting
100
100
100
100
100
100
Ir
0.01
0.01
0.01
0.01
0.01
0.01
Ir schedule
Cos
Cos
Cos
Cos
Cos
Cos
Optimizer Momentum Weight decay
SGD
SGD
SGD
SGD
SGD
SGD
0.875
0.875
0.875
0.875
0.875
0.875
0
0
0
0
0
Mini Mini-batch size
512
512
$12
512
512
SFPK setting
Simulation steps K
100
100
100
100
100
mask particles n
10
10
10
10
10
Regularization rate
0.2
0.2
0.2
0.2
0.2
0.2
Localization radius Mask polarizer
1.1
1.1
1.01
1.1
1.1
2
One-hot
One-hot
One-hot
One-hot
One-hot
One-hot
Table Experiment settings of pruning experiments on ImageNet- IK. Here, "Ir" denotes "learning rate", and "Cos" denotes the cosine annealing learning rate schedule without any warm-up epoch. We follow the implementation mask polarizer as (Mo et al. 2023 Algorithm 3). The one-shot pruning without retraining experiments shown in [ollow the same SFPK settings as described above. On an NVIDIA 40G A100 GPU, each run on MobileNet- KK takes about 1.5 days, each run on ResNet-50 takes about 2 days, and each
DeÎT-T takes about 4 days. Each SFPK update step (lines 7 to 11 of Algorithm takes approximately 0.4 seconds.
Pruning on ImageNet-IK
Model
ResNet-50 77.01 Unstructured
MobileNetVV 72.00 Unstructured
DeiT-T
Top-1 accuracy
72.20
Granularity
Structured
pruning shots
10
Batch size Epochs
256
256
512
Retrain setting
100
100
100
Ir
1e-2
1e-2
le-4
schedule
Cos
Cos
Cos
Optimizer
SGD
SGD
AdamW
Momentum
0.875
0.875
0.875
Weight decay
2e-5
2e-5
5e-2
Label smoothing
0.1
0.1
0.1
Mini-batch size
256
256
256
SFPK setting
Simulation steps K mask particles n
150
150
1000
10
10
10
Regularization rate X
0.2
0.2
0.2
Localization radius rt
3
8
Mask polarizer P:
One-hot
One-hot
One-hot
20

---

Published as a conference paper at ICLR 2025
ResNet-50 (77.01)
Sparsity
Mag
SNIP
SynFlow
WF
PSO
SFPK (ours)
75%
58.25
5.87
1.08
67.02
67.04
69.69
80%
42.97
1.54
0.54
58.72
63.38
67.42
84%
24.03
0.68
0.34
46.75
58.98
63.38
87%
9.20
0.36
0.19
32.21
54.25
59.71
90%
3.71
0.39
0.15
17.48
49.08
55,48
MobileNet- VI (72.00)
Sparsity
Mag
SNIP
SynFlow
WF
PSO
SFPK (ours)
75%
49.58
4.08
0.06
60.95
48.47
61.57
36.44
0.99
0.10
52.65
44.43
56.71
84%
0.44
0.09
41.10
31.61
51.31
87%
5.77
0.29
0.11
25.21
27.30
47.40
90%
0.72
0.18
0.11
12.56
20.53
44,06
Deit-T (72.20)
Sparsity
Mag
SNIP
SynFlow
PSO
SFPK (ours)
20%
68.14
61.53
1.21
65.91
69.94
37%
39.74
36.50
0.92
59.05
64.22
50%
6.51
10.54
0.99
46.02
55.97
60%
1.47
1.99
0.31
31.82
45.37
75%
0.16
0.25
0.13
12.60
32.70
Table 6: We report the top-1 accuracy (%) of one-shot unstructured pruning without any retrain- ing of ResNet-50 (top), MobileNet-Vi (middle), and DeïT-T (bottom) on ImageNet-IK. Boldface indicates the highest top-1 accuracy of each sparsity level.
Comparisons runtime complexity. In Table 7 we report the actual and theoretical runtime complexities of WF (Singh & Alistarh, (220), PSO (Mo et al., 2023), and our SFPK-pruner for the one-shot pruning experiment on ImageNet-IK 1K with ResNet-50 and MobileNet-Vi at 80% sparsity. Our results show that, in practice, SFPK outperforms WF while maintaining a comparable compu- tational cost. As expected, the computational cost SFPK is approximately 10 times that of PSO, as both methods use 700 discretization steps, while SFPK additionally simulates 10 particles.
We also compare the theoretical complexity of our SFPK-pruner with that of PSO and WF. Let m denote the data batch size, d the model size, K the discretization step for both PSO and SFPK- pruner, and n the number particles in the SFPK-pruner. The theoretical runtime complexity of the SFPK- is O(nK(d + m)), the theoretical runtime complexity of PSO is O(K + m)), and the theoretical runtime complexity of WF is (mdd). As we will show, for large models where d is extremely large, WF becomes significantly expensive due to the need to compute the Hessian matrix. As expected, the computational cost of SFPK is approximately n times that of PSO, as both methods use K discretization steps, while SFPK additionally simulates particles.
95% spursity
ResNet 50 on ImageNet: IK
MobileNet V1 on ImageNet-12 1K
Method
(ours)
PSO
SFPK (ours)
Runtime (s) 9629.23 (64.35)
778.41 (9.06) 7544.41 (53.28) 3023.11 (122.41)
368.41 (9.06)
3066.60 (22.79)
Table 7: Comparison the runtime complexity. We followed the recommended configuration of WF (Singh & Alistarh 2020) and configuration in Table For each experiment, present the av- erage execution time over 3 random runs on an NVIDIA 40G A100 GPU, with standard deviations.
C ADDITIONAL EXPERIMENTS
Visualizing the layer-wise sparsity of SFPK-pruner. Our SFPK -runne achieves distinct layer- wise sparsity ratios compared to ERK and other baselines (e.g., WF (Singh & Alistarh 2020), STR
(Kusupati et al. 2020). SNFS (Dettmers & Zettlemoyer, (0200). effectively avoiding layer collapse. As shown in Figure the layer-wise density (i.e. sparsity) of SFPK ResNet-50 ImageNet
21

---

Published as a conference paper at ICLR 2025
concentrates the earlier layers, without using heuristics like "freezing the first Conv / last FC layer". Unlike ERK, which assigns higher sparsity to deeper layers, SFPK shows higher sparsity in mid-to-deep layers (IDs 30 to 45) with smoother overall pattern. SFPK-pruner's sparsity also differs from WF's in early layers, showing that the sparsity pattern of early layers is crucial for pruning performance.
ImageNet ResNet-50 Layer-wise Density (overall density = 10%)
1,0
Method (Acc@1)
WF (75.21%)
(72. 77290%)
0.8
STR (74.73%)
ERK (74.10%)
(ours] (75.24%)
Density
0.4
0.2
0.0
10
30
50
Layer
Figure 4: Visualizing the layer-wise density patterns.
Ablation study on the rationality of 0", To study how the rationality of 0 affects the effectiveness of SFPK-pruner, we conduct an ablation study on the initialization of SFPK-pruner on WRN-32xX
Zagoruyko & Komodakis 2016) CIFAR- 100 (Krizhevsky) (209).. Specifically, we first pretrain dense WRN-32x4 from scratch for 300 epochs convergence, and save the intermediate model checkpoints 0 at training epoch E {0, 50, 100, 150, 200, 300}. Then, we prune each 0; to 90% sparsity in a one-shot unstructured manner and retrain it for 100 epochs. We report the training accuracy of each setting based on 3 trials.
As shown in Table X the performance of SFPK increases the pretraining epochs increase. Specif- ically, the performance of JFPK-prunee retrains at a decent level even when the model is not pre- trained to convergence. This empirical evidence suggests the robustness of SFPK-prume to the rationality of the model weight.
Pretrained Epach
0
100
150
200
250
300
Dense Acc.
1.02 (0:01)
51.05 (0.12)
59.15 (0.20)
66.52 (0.08)
72.34 (0.12)
76.01 (0.16)
Pruned Ace.
1.03 (0.04)
50.42 (0.14) 14)
54.07 (0.21)
60.84 (0.11)
62.7 (0.11)
65.66 (0.14)
Tuned Acc.
63.27 (0.52)
68.05 (0.33)
72.2 (0.17)
72.42 (0.14)
72.63 (0.09)
73.24 (0.10)
Table 8: Ablation study on the rationality of 0". We report the top-1 accuracy (%) of unstructured one-shot pruning on WRN-32x4 CIFAR-100, For each pruned model, we present the average top-1 accuracy over 3 random runs, with standard deviations parentheses.
Pruning at initialization with SFPK. We compare our SFPK against SNIP (Lee al. 2019). SynFlow (Tanaka al.) 2020). GraSP (Wang et al. 2020), and Mag (Han al. 2015) under the pruning-at-initee at-initialiiztioonn setting. Specifically, we randomly initialize a WRN-20 (Zagoruyko Ko- modakis 2016), prune it to 95% sparsity, and retrain it for 100 epochs to convergence. The average test accuracy among of final epochs is reported based on 3 trials.
As shown in Table g SFPK illustrates either comparable better performance than other base- lines. However, the performance of SFPK can be improved when a well-trained checkpoint is pro- vided. This evidence helps clarify the significance of the 'rational' constraint.
22

---

Published as a conference paper at ICLR 2025
Sparsity 95%
Dense
Mag
SNIP
GraSP
SynFlow
SFPK
PAI Acc.
52.65 (0.21)
54.19 (0.09)
52.38 (0.12)
54.97 (0.17)
55.62 (0.11)
PTP Acc.
75.26
66.47 (0.33) 62.15 (0.19)
53.31 (0.39)
59.23 (0.72)
68.16 (0.11)
We report the top-1 accuracy (%) of unstructured one-shot pruning at initialization (PAI) and post-training pruning (PTP) results with WRN-20 CIFAR-100. For each pruned model, we present the average top-1 accuracy over 3 random runs, with standard deviations in parentheses.
Table 9:
Pruning ResNet-50 to 95% sparsity on ImageNet-IK. Following the setup in Table we prune ResNet-50 (He al. 2016) the 95% sparsity level on ImageNet-IKK (Deng al. 2009) using SFPK with the same hyperparameters. Due to the limited computation resource, the reported ac- curacy is based on one trial, and we will add the results of 2 extra trials into the final version. As
shown in Table [0] our SFPK outperforms various competitive baselines, including WF (Singh & Alistarh 2020). STR (Kusupati et al. 2020), standard global magnitude pruning (Han et 2015).
GMP (Gale et al. 2019), DNW (Wortsman et al. 2019 Molchanov et al. 2017b), and
(Evii et al. 2020
ImageNet- IK
Gradual Unstructured Pruning
ResNet-50
Dense
Pruned
Rel. Drop
Sparsity
GMP
76.69
70.59
-7.95
95.00
VD
76.69
69.41
-9.49
94.92
VD
76.69
71.81
-6.36
94.94
RIGL ERK
76.80
70.00
-8.85
95.00
DNW
77.01
68.30
-11.31
95.00
STR
77.01
70.97
-7.84
94.80
STR
77.01
70.40
-8.58
95.03
Global Magnitude
77.01
71.72
-6.87
95,00
WF
77.01
72.12
-6.30
95.00
SPFK (ours)
77.01
72.23
-6.21
95.00
Table 10:
Unstructured gradual pruming with ResNet-50 on ImageNet- IK. For each experiment, we report the top-1 accuracy (%) of the pruned model and the relative performance drop (%) compared to the dense model.
Diversity of SPFK mask particles. We conduct additional experiments to study the diversity of the simulated particle distribution of SFPK. Specifically, at each intermediate step of SFPK, track the mean and standard deviation of the loss function (the energy) and the average sparsity deviation of each mask particle. The average relative mask deviation of the i-th mask computed by 1/(n - 1)- E, ||m m m/ We visualize the mean and standard deviation (across 10 mask particles) of both the loss value and the relative mask deviation.
As shown in Figure 5 as sparsity increases, the energy of each mask increases without drastic explosion. Meanwhile, the diversity of the masks progressively grows. This suggests that the SFPK mask distribution does not collapse and it generates diverse yet effective mask particles as sparsity decreases, allowing us to sample performant masks the desired sparsity level.
Ablation study on the ensemble effect in SFPK. To identify how the ensemble scheme con- tributes to the performance gain of SFPK, we compare SFPK -prnner with an ensemble-based prun- ing method, termed SWAMP (Choi et al. 2024). We compare non-ensemble SFPK (SFPK w/o ens) with SWAMP on WRN32x4 and ResNet-50 using the same gradual pruning protocol as in Table but with particle number 7. 1. We quote the SWAMP results from the last row of Table 2 and the first row of Table 3 in (Choi et al. 2024). The results are shown in Table [T and [2] We observe that SFPK works well even with only ONE particle, showing that both the SFPK update and particle simulation scheme contribute to the good performance of SFPK.
Table 12 also shows SFPK w/o ens still matches exceeds SWAMP's performance ResNet-50 ImageNet (IN- Valid) with less FLOPS. At 90% sparsity, SFPK achieves +3.4% accuracy increase compared to SWAMP within only x0.05 training epochs. Hence, both the SFPK update (Proposition
23

---

Published as a conference paper at ICLR 2025
Energy evolution
Mask sparsity evolution
Relative mask deviation evolution
2.8
2.6
Beartiv mark deccetioo 190
20
22
Spardit (50
Enerpy
2.0
10
1.6
5
10
0
0
25
50
75
1000
125
150
0
25
500
75
1.00
125
150
0
25
50
75
1.00
125
150
SFPPK Discretizationn Shep
SPPK Disrrellostion Step
SPPK iratinn Step
Figure 5: Visualization of the relative mask deviation among the SFPK mask particles. A larger deviation implies a higher degree of diversity of the SFPK masks.
2 and the particle simulation method (Section 4.2)
contribute non-trivially to the benefits of SFPK. Our SFPK neither alters weights nor uses weight averaging. and enjoys provable convergence (Proposition 3). In contrast, SWAMP is combination of Lı pruning and a weight averaging trick, requiring more training epochs to converge.
SFPK (ours) v.s. SWAMP, WRN-32x4 (78.86) on CIFAR-100
Method
Sparsity
Accuracy
Sparsity
Accuracy
SWAMP
50%
77.29 (0.53)
90%
77.14 (0.33)
SFPK w/o ens
50%
78.04 (0.09)
90%
77.38 (0.14)
SFPK (ours)
50%
78.35 (0.11)
90%
77.61 (0.21)
SWAMP
75%
77.35 (0.39)
95%
76.48 (0.73)
SFPK w/o cns
75%
77.68 (0.17)
95%
76.68 (0.19)
SFPK (ours)
75%
77.90 (0.24)
95%
77.04 (0.26)
Table 11: 'SFPK w/o ens' refers disabling the ensembles in SFPK by setting the particle number n= 1. Conducted on WRN-32x4 on the CIFAR-100 dataset.
Comparison with L -Sreed. We compare our SFPK against the state-of-the-art L, optimizer, L1- Spred (Ziyin & Wang. 2023), showing that our method consistently outperforms L Ly-Sp on both CIFAR- 10/100 and ImageNet. We the same SFPK configuration as in Table 2 On CIFAR- 10/100, we apply SFPK to prune the t timm pretrained ResNet-18 weights to 99% sparsity over 15 pruning shots with total 150 retrain epochs and report the accuracy of all sparsity checkpoints. We use SFPK to prune the same checkpoint as that in (Ziyin & Wang 2023). corchvision ResNet- 50 V2 pretrained weights, to 90% sparsity over 10 pruning shots and 100 retrain epochs. We report the accuracy of the 80% / 90% sparsity checkpoints and quote the best results from the CIFAR / ImageNet forms in the official Github repo of (Ziyin Wang (023). As shown in Figure and Table 13 our SFPK consistently outperforms -Spree across different pruning settings.
Comparison with 500-epoch RigL. For a fair comparison, we reported the results of the 100-epoch RigL+ERK (Evci et al.) (020) in the Section rather than the 500-epoch RigL. (+ERK). To ensure fair comparisons between pruning and sparse-training methods, we conduct controlled experiments, with the same 100-epoch retraining, sparsity constraints, and pretrained weights. That's why we reported the 100-epoch RigL+ERK results, the best result among the 100-epoch RigL variants (Evci ct al.
2020
Figure 2).
Table 14 compares SFPK with the 500-epoch RigL in accuracy and FLOPs. Due to the limited computation resource, we apply SFPK to the torchivis vision ResNet-50 V2 checkpoint com- pensate for the fewer training epochs. The results show that, in practice, where a pretrained model is usually accessible, SFPK outperforms the 500-epoch RigL in both accuracy and efficiency.
Comparison with MEST. Our paper focuses on a fair comparison within a post-training pruning (PTP) setting, e.g. pruning pretrained dense model. We followed the PTP setting in (Singh &
24

---

Published as a conference paper at ICLR 2025
SFPK (ours) v.s. SWAMP, ResNet-50 (77.01) on ImageNet
Method
Sparsity
Accuracy
Epoch
Tra. FLOPs
SWAMP 45.90%
76.56
210
6.72
SFPK w/o ens
50.10%
76.59
10
0.35
SFPK (ours)
50.10%
76.82
0.35
SWAMP 68.80%
76.51
330
10.56
SFPK w/o ens
68.40%
76.38
20
0.55
SFPK (ours)
68.40%
76.59
20
0.55
SWAMP 80.00%
75.69
450
14.4
SFPK w/o ens
80.00%
75.96
40
0.8
SFPK (ours) 80.00%
76.18
40
0.8
SWAMP 86.00%
74.25
570
18.24
SFPK w/o ens
87.41% %
75.19
65
0.99
SFPK (ours) 87.41%
75.65
65
0.99
SWAMP 88.90%
71.81
630
20.16
SFPK w/o ens
90.00%
74.53
100
1.19
SFPK (ours)
90.00%
75.24
100
1.19
Table 12: 'SFPK w/o ens' refers to disabling the ensembles in SFPK by setting the particle number 71 1. Conducted on ResNet-50 on the ImageNet dataset. We reported the training FLOPs in units of 1e18 and inference FLOPs in units Le9
ResNet-18 on CIFAR-10
90
Top1. Accuracy
70
Dense model
SFPK (ours)
L: L--Speed
50
10%s
Compression Ratio
Res-et-18 CIFAR-100
80
Top1 Accuracy
70
Dense model
SFPK (ours)
55
Li-Spr
10a
1065
10a
103s
107
Compression
Figure SFPK (ours) v.s. L1-Spred on ResNet-18 CIFAR-10/100
Alistarh 2020) compare pruning methods via controlled experiments, with the same retraining budgets, sparsity constraints, and pretrained checkpoints. Some sparse-training baselines were not included because the PTP setting differs from the sparse-training setting. PTP focuses on reducing the size of an off-the-shelf pretrained model and inference FLOPs. In contrast, sparse-training meth- ods aim to reduce the pretraining memory footprint and training FLOPs. They start from sparsely and randomly initialized weights and require more training epochs than the retraining epochs in PTP.
We make a direct comparison with MEST Yuan et al. 202I). a state-of-the-ar sparse training method, using the same settings in Table For CIFAR-107100 use unstructured SFPK to
25

---

Published as a conference paper at ICLR 2025
SFPK (ours) v.s. Li sp-ed,, ResNet-50 (V2) ImageNet
Method
Sparsity
Accuracy
Sparsity
Accuracy
I Spred + SFPK+ (ours)
80%
77.52
90%
69.49
80%
77.97
90%
76.56
Table 13: Both methods use the torchvisionn ResNet-50 V2 pretrained checkpoint with 80.34% accuracy.
SFPK (ours) v.s. Rigl, ResNet-50 (77.01) ImageNet
Method
Sparsity
Accuracy
Epoch
Tra. FLOPs
Inf. FLOPs
RigL+
80%
75.1
100
1.34
3.44
RigL+ERK 5x
80%
77.1
500
6.69
3.44
SFPK (ours)
80%
76.18
40
0.80
2.62
SFPK+ (ours)
80%
77.97
40
0.80
2.64
FRR
90%
73.0
100
0.80
1.96
RigL+ERK 5x
90%
76.4
500
3.94
1.96
SFPK (ours)
90%
75.24
100
1.19
1.53
SFPK+ (ours)
90%
76.56
100
1.19
1.53
Table 14: 'SFPK+' refers to applying SFPK the torchvision ResNet-50 pretrained check- point with 80.34% accuracy. We reported the training FL.OPs in units of 1e18 and inference FL.OPs in units le9.
prune pretrained ResNet-32 90%, 95%, and 98% sparsity, then retrain for 150 epochs. For ImageNet, we apply gradual unstructured SFPK pruning to pretrained ResNet-50 to 90% sparsity with a total of 100 retraining epochs and 10 pruning shots. Then, we report the accuracy of the 80% sparsity checkpoint achieved at the 40-th epoch and the final 90% sparsity checkpoint. For fair comparison between our 100-epoch SFPK and the 250-epoch MEST, we use torchvi s sion ResNet-50 V2 pretrained weight to compensate for the fewer training epochs. To ablate the effect of this checkpoint, we also implement 100-epoch MEST that is initialized from the same ResNet-50 V2 checkpoint. We denote it by '100-epoch MEST+', We quote the MEST results from the last row of Table and Table in YYuan et al. 2021).
As shown in Table 15 and Table 16 our SFPK consistently outperforms MEST on CIFAR-10/100 Our 100-cpoch SFPK consistently outperforms 150-epoch MEST on ImageNet. When using the torchvision ResNet-50 V2 pretrained weight, our 100-epoch SFPK again beats both the 250- epoch MEST and 100-epoch MEST+ We also compared their training and inference FLOPs. These results show that it is more beneficial to SFPK especially when well-trained checkpoints accessible.
ResNet-32
CIFAR-10 (Dense Acc.: 94.78)
CIFAR-100 (Dense Acc.: 74.53)
Method
Sparsity
Accuracy
Sparsity
Accuracy
MEST+EM&S
90%
93.27 (0.14)
90%
71.30 (0.31)
SFPK (ours)
90%
94.57 (0.09)
90%
72.67 (0.12)
MEST+EM&S
95%
92.44 (0.13)
95%
70,36 (0.05)
SFPK (ours)
95%
94.05 (0.17)
95%
71.69 (0.15)
MEST+EM&S
98%
90.51 (0.11)
98%
67.16 (0.25)
SFPK (ours)
98%
92.42 (0.20)
98%
67.48 (0.19)
Table 15: SFPK (ours) v.s. MEST on ResNet-32 CIFAR-10/100.
26

---

Published as a conference paper at ICLR 2025
SFPK (ours) v.s. MEST, ResNet50 (77.01) on ImageNet
Method
Sparsity
Accuracy
Epochs
Tra. FLOPs
Inf. FLOPs
MEST x0.5
80%
75.39
75
0.74
1.70
MEST x1
80%
75.75
150
1.27
1.70
MEST x1.7
80%
77.19
250
2.15
1.70
MEST+ x1
80%
75.99
150
1.27
1.70
SFPK (ours)
80%
76.18
40
0.80
2.62
SFPK+ (ours)
80%
77.97
40
0.80
2.64
MEST x0.5
90%
72.58
75
0.39
0.90
MEST x1
90%
75.1
150
0.60
0.90
MEST x1.7
90%
76.13
250
1.11
0.90
SFPK (ours)
90%
75.24
100
1.19
1.53
SFPK+ (ours)
90%
76.56
100
1.19
1.53
Table "SPPK'7'MEST+ refers to applying SFPK/MEST torchvision ResNet-50 pretrained checkpoint with 80.34% accuracy. We reported the training FLOPs in units of 1e18 and inference FLOPs units le9.
D THEORETICAL RESULTS
D.1 PROOF DETAILS
Definition 3 (Weak Convergence). Let be the Lebesgue measure on Id and Ω C Rd be v- measurable set. bounded sequence of distributions {T²) Jnzl C P(2) said to converge weakly [m"]n>
to E (S2), denoted by T7 - if and only if (T" + (TT, Φ) as n + co holds for all bounded and smooth test functions o(-) on Ω.
Proposition (Existence and Uniqueness of the Optimal Mask Distribution (formal version of Proposition 11) Let s(-) Rd + R4 be the soft sparsity function, Le Re + R4 be the model loss junction that a.e. smooth, k((;))) Rd x Rd + R. denotes the RBF function kernel, we consider the following distributional optimization problem
min (Le, T) + XR[T] A f (e(m))((d)) P(MP)
2 / s(m, m (d)r(d)))((())))
(9)
M.
with A > 0 and MP A {m Rd s(m) E it, pl}. For simplicity, we denote the functional objective Li () + XRH. Assume that Lg(,) satisfies the following assumptions:
(Al). Strict sparsity monotonicity: the optimal loss lower sparsity levels is always strictly smaller than at the higher sparsity levels. Specifically, there exists a >> 0, for any sparsity levels t t', such that inf mp Le (e(m) < innfmmm, Le (m) act'' t)2.
(A2). Lower semi-continuity: for all m Rd. any sequence {m')21 that converges to m, it holds that limn inf (m Le(m).
J42
Then, for any sparsity level there exists a n. such that ((m.) inf (L(T) P (M,)), More- over, the global minimizer of C[-] in P (Mf) ) is unique.
Proof of Proposition By definition, the objective functional is lower bounded by 0, which implies O & L inf [[|m] P (Mf)) < co. Thus, there exists sequence (Cn)nn in [ (Mf)), satisfying Cn + C as n + 20 and for each n, 2 x" E P (Mf)) such that C, = [[*"]. Then show the sequence {m")a1 ha2 is tight, that is, for any 0, there exists compact set K C Mf such that for any n E N. we have T (K) > -
Once the tightness of {T" hns1 is established,
apply the Prokhorov's theorem Billingsley
1968 Theorem5511 to extract a subsequence (Λ |&21 of [x" ] and a distribution म, E P (MM))
such that " for - According to (Ambrosio al. 2006 Lemma 5.1.7), (A2) implies
27

---

Published as a conference paper at ICLR 2025
the lower semi-continuity of the linear functional (). Furthermore, the continuity of x((,-) implies that []-] is also lower semi-continuous, which further yields
[[ & lim [|T**] lim Lmz lim =
(10)
8-000
R-+00
n-100
By definition of L* it holds that L & [(m)] & L' showing that the infimum is indeed achievable in P(Mf) at Te, i.e. [[ = L This proves the existence of the optimal mask distribution.
Then prove the uniqueness of n. by contradiction. Suppose L is attained at two different distributions oi and oテ. By (Arbel et al. 2019, Lemma 25), the strict convexity of k(s+) -) implies
the strict convexity of R.].] and C|-- Let B E 10 and m. 8ot + (1 3)m holds that TH E P(Mf") and
C| < 30[=]] + 3) = L
(11)
Thus, this contradicts the definition of L" which implies that L is uniquely attained at m.
To complete the proof, we only need to establish the tightness of the aforementioned sequence [m")aa1 This can be proved via contradiction. Suppose {n" Ins1 is not tight, then there exists
6 > 0 such that for all E N, there exist n& E N, for arbitrary T C Mf with n (Tx) & 6.
This implies
S + $ Le(m)TT* (dm) + AR[="*]
Cn = []
=
MFVR
(12)
+ +
(em)T"*(dm)
(13)
1x
MEVN
> inf n E P(Ix)) T* (Mf (s) in (Le()) MfNTa]
(14)
> inf([[[m] E P(I+)] ciif m Mf
(15)
Let I A M it holds that & Mt : Therefore, the condition (A1) implies that
b., > in : = P(})) + - (L.(m)) m E Ms)
(16)
-inf(C[m] E P(M, })} cinf [e(m) m M
(17)
++
>nf([[T]] a P(M, p- 1 })} + e inf (m): : m Mt } (p-11/1))
(18)
> inf x T PMM +)) + co(p-1/h)3
(19)
By taking the limitation -> do on both sides, it holds that L* > L + eap?, with eap? is a contradiction. Thus, this implies the tightness of (x*) Jn31 and completes the proof.
> which
L
Proposition 5 (Optimal Local Distributional Transition (formal version of Proposition 2). Follow-
ing the notations and conditions introduced in Proposition let aa be the optimal mask distribution
at sparsity level t, let (.) & I k(., m) m)x((dm)) be the k-mean embedding of nu consider the following local distributional transition problem
min Llv) & (The + Ap U - s.t. u E P P(MP+A)
(20)
P (Mf+2) - {u+ supp(v( mm) E BrAc Vm E M2) nP (M{+a))
(21)
where Ft > 0 is the local radius satisfying re rell (s(m)l2 for all m E Mf. We We further assume
that s(-) satisfies
(A3). Local regularity: s(-) provides local norm control. Specifically, for any m, there exists 6 >> O, and C > 0, such that any m' satisfying ||m - ml|2 & e holds that
m' mlla la & Cls(m') sm)).
Then, vi(+Im) A Dirac(:; (:mn At) achieves the optimal value of (0 within an error of order ((A)), that is, Í[v]] L. = O(DP) and v. E P(MP At-C for some constant C. Specifically, let VL (m) - A XV (m) and - (s(m). T(m; n.) is given by
r ?
7(m;m)) A
|
+
- Tg) |s
S8
g
(22)
28

---

Published as a conference paper at ICLR 2025
Proof of Proposition S For simplicity, we denote F() - Le(+) Aprre (-). By standard derivation, it holds that
[l] / ((m)((( [(m)((u (m())
(23)
= ((m) (f (1) (m) 8)dō - -r(mm) dm
(24)
= / F(m) / (00,,nn)),.mmmmm.. 8) <(8 m) dōdm
(25)
// F(m) (Vm(((()))))) 5 - O1
dmd5
(26)
// ((m))m(((00m)) (mm)mmmmmmmmmm ōdmdō +
O(At))
(27)
// (VF(m) 8) () + (OAt))
(28)
To identify a minimizer of Elyj within O(At)) error, one can resort to solving (88) in a point-wise (microscopic) manner. Specifically, we consider the following deterministic optimization problem
min V F 8, s.t. s(m + 6) = =smm)= + At, | < r. 8
(29)
Suppose 8'(m (m) is the minimizer of 29 let v² (-Im) denotes the minimizer of the first term in 20
then it holds that v² (-Imm) = Dirac(: (; o" (m)). This can be proved by definition: for any m M
suppose A A(m) is the support of v* (- m) such that V 8' E A. 1| < r, s(m+ = s(m) At,
then it holds that
/I (VF(m) v" (cl (
(30)
/ (Arcon + ( (VF(m)T)) v² (dō te(dm)
(31)
/ (VF(m) (0m()))))))))))))))))))))))))))) /² (A(m))mm)) +
(32)
/ (VF(m) (VF(m)T((m) (m)) ne (dm)
(33)
(VF(m) T 8) Dirac(dō; 8* (m))xx(dm).
(34)
Notice that solving 5' (m) from 29 is still challenging, since the constraint is nonlinear. Thus, we aim to slightly relax the constraint within an O(Ar?) - error. Specifically, we consider the following convex optimization problem
min (m 6, s.t. (s(m) To = At, ||8|| < At.
(35)
Clearly, (A3) implies that the minimizer of 35 denoted by 80 (m), achieves the optimum of (9
within ((Ar))-orde error. Thus, one can establish an O( ((At")r error approximation for
v" v"(:; m) as Dirac(:; 5'))). The final step to finishing the proof is to derive 8* (m). As (35)
is a convex optimization problem, the optimum is attained at the decision boundary. Moreover, the optimum lies in the linear space spanned by {VF(m) Vs(m)). For simplicity. we denote
g A F(m(m) and A Vs(m), then we parameterized the optimum of (35) ag bs. Finally, the proof is completed by solving the coefficients and from the following equations
s (ag + bs) At,
|lag + bs|l2 = At,
(36)
which yields
2
(Ai
8) a) [g
At,
b
(37)
lg
|
(g' 8)
lg
29

---

Published as a conference paper at ICLR 2025
Therefore, the proof is completed by taking
1/
T(m; TT) 8* (m) = ag + | +
1s
r - (s g)
g
ss
|s |
g.
(38)
Proposition 6 (Derivation of SFPK (formal version of Proposition 3 Following the previously introduced notations and conditions, we further assume that
(A4). Quadratic growth: the growth of expected loss is quadratically bounded w.r. sparsity lev- els. Specifically, there exists 3 > O, for any sparsity levels t < t', such that
inf mEMO
[m(m) < inf (m(m) + 3(' - -t)²
mEM
Then, by laking At - O, the sequence {TRAE) constructed by Tt+ 2 Us Re converges weakly to the Sparsity Evolutionary Pokker-Planck-Kolmoggn Equation (SFPK), denoted by
0, = V (+
(39)
where V [.] denotes the divergence operator and T()m;) is the optimal local transition function defined in Proposition
Proof of Proposition The proof sketch is summarized follows: we first show that the se- quence { converges weakly to the Dustribution-valuee differential equation (#t): that satisfies the SFPK ,, denoting the optimal mask distribution defined in Proposition
(T: we )ee, then we show that 7, Tr for any E [0, d' / with Te
Step 1. To establish the weak convergence of [TAAC) we need to show for any and & A [t/At], holds that
At (((+))A - TAAI) - V [T(: TRAd
(40)
as A + O. By definition, this is equivalent
lim 1 At (m)))))(()) (m))
/ V [T' MxLa) (a)))
(41)
where (b-) an arbitrary bounded smooth test function. Then, the desired weak convergence can be
proved by showing that, for any let & - [1/A],, 7.a: ne as Al - O. According to Proposition
5 for any k k(k-1)Ar 1) = VAaI * ThAi which implies
Af (F(+1)))()) - (m))))))
(42)
1
/ (VEAL FRai(m) (FEAL*FRa)(m) (m))
(43)
A/
/f (v6ar(flm 5)aaar(m - 6) doa.
(44)
A/ //
(mmm O(A))
60(m)
(45)
At /f Vo(() Se (0) O(A)))
(46)
// 7 O(Ar)
(47)
/ Vo(m) ( O(At)
(48)
Vm [m,, R.a M ((m)dm ((At))
(49)
This implics the statement in 41 Then, we only need to show for l A [t/At], T ,. as
At 0. This equivalent show that (T(441) *RAe) 0 as At which implies the
30

---

Published as a conference paper at ICLR 2025
existence and the continuity (w.rt the weak topology) of the limiting process (Te)). Recall that since ()).,, k((.-) and s-) are a.c. -smooth on the support of TRai. the right hand side of (41) is bounded, indicating that (T(+-1)) :a.. d) (AAt) holds for any bounded continuous d(-). J. which further implies (*(*+1)A+ TRai) - 0 as At + 0.
Step 2. Suppose ne is the optimal mask distribution defined in Proposition 4
We aim to prove that To no. it holds that 7; = for any E [0, d' /d]. This is equivalent to show (fT. TT. の) for any bounded continuous Φ(-) and any t E [0, - d'/d]. To this end, we study the evolution of the deviation term K Te, 6) under finite sparsity increment.
First, we show that THA (MM which implies there exist an oracle transition kernel v/j (-I-) and rr > 0 such that Te+ of ne and supp(>? (-mm) C. Br, for any m Mf. Recall that the functional [[. is strictly convex, there exists e > O, such that
c [1 melli2
Def. ÔT: m. -erse) + Clmerad C[me] & LlTeaad C|me]] (A4) BAF. (50)
2
Therefore, by taking r = 28/c, c, we have
sup (T++ Te, #)1 < Te+a Tellzelloll.. & res
which indicates the W-Wasserstein distance (Ambrosio et al. (006) between Te-Ai and My bounded by rx At and hence ar E P(Mf+A)) proving the existence of the oracle transition kernel v Now, we can feel free to represent iT&+ At with v/ TT,, For simplicity, we slightly abuse the notations and denote all the constants as C' and C2 in the remainder of the paper. For any bounded smooth test function Φ holds that
(T++ Met Ø) Ki कe Vi Te, Φ)
(51)
[(v - v/ * ne. の) + (ue- (++- TT). o)
(52)
(T1): localization error
(T2): simulation crror
Then, we bound the terms (TI) and (T2) respectively. For any bounded smooth test function od-) and arbitrary small 17, denote S2, _ (m <(m) /2). By definition,
Ky,+ * (7. - 0)) /ʃ (m(6m (m. - 5) r. (m (mm)ddmm
(53)
// vi v:(6)m)((((m) 7.(m mm)mm)))m( + 8)d5dm + On
(54)
// On
(m) x -
(m) (.) C'n
(55)
// Ong
(7t n. (m))(((m + vd(m)" T8 (1o0 + On
(56)
// aa, (m)) (a0aa
+ 1// Bn
+Cn
(57)
<, Ne, Tc,0) +
(7.(m) ( )mm)
At At
(58)
<| - Te. +
- ( (Tm )mm
At Cn +
< - - +1+
(7. - n (m)))((m)m
max 24
V(log T(: ((::))))
At +n
1n
+ C.
Re, Φ (
(59)
31

---

Published as a conference paper at ICLR 2025
To bound (T1), one need to leverage the localization property of vs. Recall that of is the minimizer
of
min Llv * mel, s.t. V T P(M))-
(60)
This implies Clv TTl Llv, mT] & O. Applying the second-order functional expansion 1/ IT to the left hand side yields that
8
8² L. (vī v.) mz) & Ce
II(ei v) + ATell2, & &CAP..
(61)
8RLLS
On the other hand, since 11 is the minimizer of
min < 5 L ne )1) s.t. uT P(MP)
0 ( 8
L, L,z - v) = tt
(62)
which further implies
5 8TctAL
L, (Vī v1) m. 3 /
8²
(vi -i) x
-2
L
C,st²
(63)
Le + Mrte (vz ve) * *xte) Ccatt
(64)
/) inf inf( Tie + (rx)) 11(1/ - vt) * Te le. G.Pt..
(6.5)
MP
Therefore, the combination of (61 and (65) implies
K(v= vz)) Te, の) < [[ vx) * M.. melle,llo) 1o & C.A..
(66)
For any fixed 1) > O, we combine 59 and (66) and take At + to derive
KTH+i! T(+Ai,
| Te, Te,6)) <C, + C Te, Ø) At + On
(67)
d Te, の) <Cn + Cel Te, ldt.
(68)
Finally, we apply the Gröwnwall inequality (Gronwall 1919 to the function ++ 1( - Tt, Φ) and we prove that
( Te, ه)| Cn 1 L. sCeedi Co ds
(69)
holds for any & E 1 dd/d] and any 1) > 0. By taking !) -> 0, the right-hand side of (69 tends to which completes the proof.
D.2 REALISING SFPK VIA PARTICLE SIMULATION-BASED APPROXIMATION
In this section, we aim to realize the sampling from ग 1 ja by simulating SFPK using an interacting mask particle system. We omit the literature review on some basic concepts, such as infinitesimal generator and the origins of FPK equations, thermodynamic and stochastic analysis, and we refer interested readers to (Bogachev et al. 2015 Wild et al. 2023) for detailed introductions.
Let (m,), be a stochastic process, n. be the particle density function time I, the stochastic analysis theory shows that the time evolution of Te is associated with the adjoint of the infinitesimal generator of the stochastic process. Specifically, suppose n; follows an FPK equation 0, = A Tr, where AA" is a known differential operator. Then, one instantiate π. using stochastic process (m)),, such that law(m;) = Te for all f, and the infinitesimal generator of the process equals to A, which is indeed the adjoint operator of A (that is (A[f). g) = (g))). Finally, the dynamic of (m.), can be derived based on its infinitesimal generator A. In intuition, the infinitesimal generator an operator describing how an arbitrary statistic of the thermodynamic system changes under an infinitesimal time increase.
Following this spirit, we aim to derive the stochastic process associated with SFPK. To this end, only need to identify the adjoin operator of n. > -V (.)... For any bounded smooth
32

---

Published as a conference paper at ICLR 2025
test function (()). want identify A, such that (-V m)) Φ) (m²) Following standard derivation, holds that
(-V- (T(; m)
V (T(m, (m) T(m, me) (
(70)
which implies A[: = (m, Te) V. According the stochastic analysis theory (Wild al. 2023 Equation (54)), the SFPK is associated to the following McKean- Vlasov process:
dm, T(m,, law (aw(m)))di,
(71)
where the drift term is dependent both the microscopic coordinate m, and the macroscopic distribution тe. Fortunately, (Veretenniov) [2066] Section 2) provides a standard approach to sim- ensemble of interacting particles.
ulate solutions to ( using the empirical distribution Formally, the authors show that if we take
n
dmp T(m 7H dt, i= = n, with 7े
Dirac(;m.).
(72)
72
then (Veretennikov 2006
Theorem 2.2) implies that के Te as cc. Therefore, to sample
performant mask from 1-dd /d. we simulate the SFPK dynamic using n interacting mask particles
and update the system according to (72) from 0 to t = 1 - d'/e d'/d. Finally, we can sample desired masks from the empirical distribution IT Ti-d'/d-
33


Reviews:
[{"review_id": "14JJwD7J3g", "reviewer": "Reviewer_3E6s", "summary": "This work formulates the neural network pruning procedure as the evolution of sparse mask distributions in response to increasing levels of sparsity, employing the Fokker-Planck-Kolmogorov equation. Specifically, given a rational dense solution, it provides a theoretically well-founded definition of the distributional transition for the sparse mask, indicating which parameters should be pruned. Experimental results are provided for both one-shot and iterative schemes, as well as for structured and unstructured sparsity, covering convolutional and vision transformer architectures.", "questions": "Q1. In Lines 428-430, it states that _\"the SFPK-pruner with a batch size of $256$ for $n \\times K = 1500$ steps is equivalent to that of only ONE epoch of retraining on ImageNet.\"_ While I understand that running $n = 10$ particles sequentially for $K = 150$ steps (without parallelization) results in a batch size of $256$ over $n \\times K = 1500$ steps, it seems that $256 \\times 1500 = 384000$ samples would correspond to approximately 0.3 epochs of ImageNet (given that the total number of training samples in ImageNet is 1.28 million), rather than one epoch. Could you clarify this point?\n\nQ2. Regarding W4, can SPFK operate with randomly initialized weights? While this clearly disregards the \"rational\" constraint, it is noteworthy that single-shot approaches like SNIP, GraSP, and SynFlow originally claim to find effective sparse masks using only randomly initialized weights (i.e., pruning-at-initialization). I am curious whether SPFK can outperform these methods in identifying a better sparse mask in a pruning-at-initialization manner. Ideally, the relationship would be random mask < SNIP, GraSP, SynFlow < SPFK. Alternatively, demonstrating that SPFK does not outperform SNIP, GraSP, and SynFlow in the pruning-at-initialization setup could also help clarify the significance of the \"rational\" constraint.\n\nQ3. The core of the SFPK pipeline, particularly when compared to Mo et al. (2023), revolves around simulating $n>1$ particles. The paragraph _\"Ablation on the Mask Particle Simulation Scheme\"_ in Section 5.3 illustrates that the proposed SFPK significantly relies on both $n$ and $K$. Notably, the difference between using $n=1$ and $n>1$ is considerable (in Figure 3), highlighting the importance of leveraging multiple particles for the effectiveness of the SFPK methodology. I'm interested in the diversity of the particles in this context. Intuitively, at low sparsity, there are likely many mask candidates that yield effective sparse solutions. However, at high sparsity, could the particles collapse? If that’s the case, it may be advantageous to reduce the number of simulated particles as sparsity increases, potentially enhancing computational efficiency.\n\nQ4. Are all trainable parameters, including convolutional weights, batch normalization scales and biases, and dense weights and biases, subject to pruning? Some studies on pruning convolutional neural networks focus solely on convolutional weights, e.g., the SWAMP paper (Choi et al. 2024) notes, _\"we exclusively perform weight pruning on the convolutional layer, leaving the batch normalization and fully-connected layers unpruned.\"_ This can lead to discrepancies in reported numbers across studies, particularly in the ResNet50 on ImageNet setting, where the final linear layer accounts for about 8% of the total 25,557,032 parameters (i.e., 2,048,000/25,557,032). Could you provide further details on this? It is important to note that allowing the pruning of the final linear weights often leads to significantly better performance at the same level of sparsity (when we compute the sparsity as the number of unpruned parameters divided by the number of all parameters, regardless of such constraints).\n\nSpecifically, the SWAMP results for ImageNet presented in Table 6 indicate 88.9% sparsity, with approximately 2.8 million unpruned parameters, of which 2.0 million are linear weights. When calculating sparsity based solely on prunable parameters  for SWAMP (i.e., convolutional weights), we arrive at approximately 96.9% sparsity. Recalculating SWAMP's sparsity as the number of pruned prunable parameters divided by the total number of prunable parameters yields the following adjustments (see the code block below for details): 80.0% becomes 87.2%, 86.0% becomes 93.7%, and 88.9% becomes 96.9%. While this value may not perfectly align with the sparsity of SFPK, it is noteworthy that SWAMP achieves 75.69% accuracy at 87.2% sparsity, whereas SFPK achieves 75.65% accuracy at 87.4% sparsity, indicating a close alignment in results. Therefore, it is important to assess whether fair comparisons have been adequately conducted for not only SWAMP but also other methods.\n\n```\nResNet50 for ImageNet.\n- 25557032 : the total number of parameters in ResNet50 for ImageNet.\n- 23454912 : the number of convolutional parameters.\n- 2102120 : the number of remaining parameters.\n\nIf there are 5111406 (= 3009286 + 2102120) unpruned parameters,\n- 1 - 5111406 / 25557032 = 80.0% sparsity.\n- 1 - 3009286 / 23454912 = 87.2% sparsity, if we consider only the prunable parameters.\n```\n\nQ99. It's just a very minor detail, but I'm curious—why is it referred to as SFPK instead of SEFPK?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. In neural network pruning, a line of research focuses on iterative methods that start with a dense network with zero sparsity and gradually increase the sparsity level. A well-known example is the Iterative Magnitude Pruning (IMP) algorithm, which, while computationally expensive, is highly effective at producing high-performance sparse networks. This work shares a similar spirit (especially in the prune-and-retrain setup), deriving the sparse mask for each new sparsity level from the previous one, gradually updating the mask as sparsity increases. However, rather than treating the optimal mask at each level as a fixed point estimate, this work consider a distribution over masks, adding a compelling dimension to the method.\n\nS2. As the authors noted, the most closely related work is the ODE framework proposed by Mo et al. (2023), which evolves sparse masks over time by treating the sparsity level as the timestep. This work explicitly acknowledges this connection in the main text and provides a clear discussion in the Introduction that highlights the contributions unique to them. It not only gives proper credit to prior work but also helps readers understand the progression of ideas.\n\nS3. The proposed SFPK methodology is well-motivated, and the flow of Sections 1-4 effectively presents it. Overall, the experiments are well-executed, featuring a comprehensive comparison to various baselines. While there is room for improvement, as noted in the Weaknesses section, the core results are largely well presented.", "weaknesses": "Since I appreciate the underlying concepts of the proposed SPFK methodology, I will primarily concentrate on addressing the experimental concerns here.\n\nW1. The tables do not include error bars, making it unclear how statistically significant the current results are. How many trials were conducted for the values presented? From the caption of Figure 2, is it three trials for all experiments? If space is a concern, the main text could provide only the averaged numbers, while a full table with standard deviations could be included in the appendix.\n\nW2. In the _One-shot Pruning_ experiment in Section 5.2, it is quite intriguing that the SPFK method achieves such performance after one-shot pruning on a given pre-trained dense weight without requiring further training for recovery. However, it is important to note that all three methods—WF, PSO, and SFPK—incur significantly higher costs compared to Mag, SNIP, and SynFlow for obtaining valid results. Including a cost comparison for each method would be beneficial. At the very least, the costs associated with SFPK compared to WF and PSO should be presented (especially since Mag, SNIP, and SynFlow would not even be considered when targeting high sparsity).\n\nW3. In the _One-shot Pruning_ experiment in Section 5.2, Table 1a presents results for the convolutional architecture, while Table 2a displays results for the vision transformer architecture. However, Table 1a only includes results _without_ further training and _unstructured_ sparsity, whereas Table 2a only includes results _with_ further training and _structured_ sparsity. Is there a specific reason for this distinction? From the reader's perspective, it would be beneficial to see comprehensive results for both convolutional and vision transformer architectures for each method—{Mag, SNIP, SynFlow, WF, PSO, SFPK}, particularly for {WF, PSO, SFPK} as they are the main competitors—across all scenarios: with and without further training, and structured and unstructured sparsity.\n\nW4. The SFPK methodology fundamentally assumes \"rational\" pre-trained weights, denoted as $\\theta^\\ast$. I noticed that the paragraph _\"Constraints of SPFK-pruner\"_ in Appendix A.1 discusses this topic, stating, _\"We recommend applying SFPK to sufficiently trained \"rational\" models (e.g., trained to convergence).\"_ However, there is currently no experimental evidence supporting the practical significance of this recommendation. It is particularly important in the prune-and-retrain setup, as the extent of retraining to re-obtain the \"rational\" $\\theta^\\ast$ can consume a substantial portion of the total training budget. I initially expected the Appendix B to address this (after reading lines 448-449), but it still fixes the retraining budget for SFPK at 100 epochs. Currently, there is no discussion on how \"rational\" pre-trained weights need to be for SFPK to function effectively. Including empirical results on this matter would undoubtedly strengthen the paper.\n\nW5. As of 2024, 90% sparsity for ResNet50 on ImageNet is not considered particularly high (especially when all parameters, including the final linear weights, are allowed to be pruned). To convincingly demonstrate the effectiveness of the proposed method, comparative results should be presented in a higher sparsity regime of 95% or more. Additionally, the comparisons primarily involve relatively old baseline methodologies, with the most recent baseline in Table 2c dating back to 2020; it raises the question of whether it is appropriate to claim that _\"our SFPK-pruner demonstrates decent performance compared to SOTA baselines.\"_ For instance, the method called Spartan (Tai et al., NeurIPS 2022) reports an accuracy of 76.17% for 90% sparsity in a 100-epoch setup. Although the pruning community has shifted its focus away from the ResNet50 on the ImageNet setup, and I am not sure whether recent SOTA results are being reported, it remains important to include comparisons with the latest baselines available in this paper.\n\nP.S. As I continue writing, I recognize that providing such extensive experimental results could be burdensome, especially during the rebuttal period. I want to clarify that I do not expect all results to be presented within this limited timeframe (particularly, feel free to skip W3). Please share any results you can within your available time and resources, along with a brief explanation of how you plan to address the mentioned concerns in the future if they are not currently available.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5B4nwAW5e6", "reviewer": "Reviewer_TsAq", "summary": "This paper introduces a method to identify a sparse network by incrementally selecting the mask that results in the smallest performance reduction as sparsity increases. Deriving from  thermodynamics, the authors begin with a population of dense masks and progressively evolve a distribution of sparse masks using the Fokker-Planck-Kolmogorov equation. Their approach offers a way to transition to a higher sparsity mask distribution by increasing the sparsity by a small amount in every step. Additionally, the authors propose a pruning scheme based on this transition, which iteratively refines a population of sparse masks to achieve an effective sparse network.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed algorithm optimizes a population of sparse mask in an evolutionary manner while increasing sparsity to find a performant sparse network. This is a novel which seems to dynamical learn mask structure similar to dynamic sparse training method albeit in an evolutionary manner.\n2. The authors also experimentally validate the proposed pruning method.", "weaknesses": "1. The authors provide a comprehensive experimental evaluation of the proposed method. However, the performance across tasks is similar to dynamic sparse training methods like RiGL and only slightly outperforms it in some cases (as in Table 8). Given this, it would be important to know the computational comparisons of DST methods versus the SFPK pruner. On ImageNet the authors mention that the SFPK pruner adds an overhead equivalent to 3 training epochs, would this mean it is marginally more expensive than DST?.\n2. In order to optimize the mask, the sparsity function chosen by the authors is the polarizer. Is there a reason for choosing this specific operator, how would it compare with other operators such as soft thresholding or even a sigmoid. \n3. Results in Table 1 could be a bit misleading since the pruning at initialization methods like Snip and Grasp are not designed to minimize the loss upon sparsification and hence cannot be compared without any retraining after pruning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JQMY0djwTN", "reviewer": "Reviewer_3JmV", "summary": "Summary\n\n- In this paper, the authors reformulate pruning as an expected loss minimization problem within the mask distribution space and introduce the SFPK to represent distributional transitions. Leveraging the proposed SFPK, they develop the SFPK-pruner, a particle simulation-based probabilistic pruning method.", "questions": "See Weaknesses section. If the responses to my questions are satisfactory, I am open to increasing my rating.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths\n\n- The paper is well-written and easy to follow.\n- Using the probabilistic lifting and convexification technique from [1] to develop a probabilistic soft neural pruning approach is a fresh and intriguing method.\n- The theoretical analysis and approximations supporting the SFPK-guided pruning proposal are reasonable and add an interesting perspective.\n- Overall, I am satisfied with the experimental scale, results, and novelty of the method. Although the requirement to create $ n $ mask particles presents practical challenges for applying this method to large models, such as large language models, I still believe this is a good paper that falls within the acceptance boundary.\n\nReferences\n\n[1] Veit David Wild, Sahra Ghalebikesabi, Dino Sejdinovic, and Jeremias Knoblauch. A rigorous link between deep ensembles and (variational) bayesian methods. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.", "weaknesses": "Weaknesses\n\n- Additional analysis on the computational overhead as $ n $ and $ K $ increase would be beneficial. While these values can be adjusted, a larger $ n $ could lead to a linearly increased memory overhead, and a larger $ K $ could introduce extra computational burden compared to other methods.\n\n- A comparison with similar methods, such as iterative magnitude pruning methods, which also gradually adjust sparsity, would provide useful context.\n\n- It would be helpful to assess the performance of each of the $ n $ particles individually and check if they display functional diversity. Verifying whether the method adequately covers the posterior distribution of the mask or simply converges towards one or a few local minima with good performance would more clearly demonstrate the effectiveness of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hI5Fk19B8R", "reviewer": "Reviewer_Y7sb", "summary": "The authors use probabilistic lifting to reformulate gradual pruning as the evolution of the mask distribution. The authors then assume locality and derive the equations governing the sparsity evolution, which the authors refer to as the SFPK equation. Finally, based on SFPK, they propose a new pruning algorithm, which they demonstrate through extensive experiments its SOTA performance on a range of pruning tasks.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper presents a strong combination of theory and experiments. The paper proposes a theoretically driven algorithm, which achieves the state-of-the-art performance on a range of pruning tasks. The theoretical analyses, especially the relation to the FPK equation, bring new insights for pruning. Overall, the paper has a significant impact to the field.", "weaknesses": "Major points:\n\n1.\tThe authors does not emphasize the critical role of using mini batches in algorithm 1. Mini-batches and stochastic noises are crucial for algorithm 1. If using full batches, then T should be the same for all m_i from the begining, resulting a collapse of the distribution.\n\n2.\tThe performances of SNIP and SynFlow in table 1 seem unreasonably low. I suspect there may be some mistakes in the experiments. Specifically, both methods prune the network at initialization prior to training. I wonder if the authors mistakenly apply those pruning algorithms at the end of training. This could explain the discrepancies in performance, and I recommend the authors review the experimental setup for these pruning algorithms to ensure correctness.\n\n\nMinor points:\n\n1.\tIt may be good to define the distributional transition T when it is first introduced at the end of section 3.\n\n2.\tI don’t find a definition of the mask polarizer $P_\\epsilon$ in the main text. It should be defined.\n\n3.\tThe authors claim their algorithm is robust against different $\\lambda$, but this should be expected as $\\lambda$ is introduced for a technical reason to convexify the optimization problem. From an algorithm perspective, it may not be necessary to have $\\lambda$ at all. It would also be useful to include results in Fig. 3 with $\\lambda=0$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)