Title: How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspective
Venue: NeurIPS 2024
Accepted: yes
Keywords: Network Pruning

Fundamental Limit

Convex Geometry
Abstract: Network pruning is a commonly used measure to alleviate the storage and computational burden of deep neural networks. However, the fundamental limit of network pruning is still lacking. To close the gap, in this work we'll take a first-principles approach, i.e. we'll directly impose the sparsity constraint on the loss function and leverage the framework of statistical dimension in convex geometry, thus enabling us to characterize the sharp phase transition point, which can be regarded as  the fundamental limit of the pruning ratio. Through this  limit, we're able to identify two key factors that determine the pruning ratio limit, namely, weight magnitude and network sharpness. Generally speaking, the flatter the loss landscape or the smaller the weight magnitude, the smaller pruning ratio. Moreover, we provide efficient countermeasures to address the challenges in the computation of the pruning limit, which mainly involves the accurate spectrum estimation of a large-scale and non-positive Hessian matrix. Moreover, through the lens of the pruning ratio threshold,  we can also provide rigorous interpretations on several heuristics in existing pruning algorithms. Extensive experiments are performed which demonstrate that our theoretical pruning ratio threshold coincides very well with the experiments. All codes are available at: https://github.com/QiaozheZhang/Global-One-shot-Pruning

How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspective

Abstract

Network pruning is a commonly used measure to alleviate the storage and computational burden of deep neural networks. However, the fundamental limit of network pruning is still lacking. To close the gap, in this work we'll take a first-principles approach, i.e. we'll directly impose the sparsity constraint on the loss function and leverage the framework of statistical dimension in convex geometry, thus enabling us to characterize the sharp phase transition point, which can be regarded as the fundamental limit of the pruning ratio. Through this limit, we're able to identify two key factors that determine the pruning ratio limit, namely, weight magnitude and network sharpness . Generally speaking, the flatter the loss landscape or the smaller the weight magnitude, the smaller pruning ratio. Moreover, we provide efficient countermeasures to address the challenges in the computation of the pruning limit, which mainly involves the accurate spectrum estimation of a large-scale and non-positive Hessian matrix. Moreover, through the lens of the pruning ratio threshold, we can also provide rigorous interpretations on several heuristics in existing pruning algorithms. Extensive experiments are performed which demonstrate that our theoretical pruning ratio threshold coincides very well with the experiments. All codes are available at: https://github.com/QiaozheZhang/Global-One-shot-Pruning

1 Introduction

Deep neural networks (DNNs) have achieved huge success in the past decade, which relies heavily on the overparametrization, i.e. the number of parameters is normally several order of magnitudes more than the number of data samples. Though being a key enabler for the striking performance of DNN, overparametrization poses huge burden for computation and storage in practice. It is therefore tempting to ask: 1) Can we prune the DNN by a large ratio without performance sacrifice? 2) What's the fundamental limit of network pruning?

For the first question, a key approach is to perform network pruning, which was first introduced by [21]. Network pruning can substantially decrease the number of parameters, thus alleviating the computational and storage burden. The basic idea of network pruning is simple, i.e., to devise metrics to evaluate the significance of parameters and remove the insignificant ones. Various pruning algorithms have been proposed so far: [21; 12; 13; 23; 42; 35; 36; 24; 22; 15] and [13].

In contrast, our understanding on the second question, i.e., the fundamental limit of network pruning, is far less. Some relevant works are: [19] proposed to characterize the degrees of freedom of a DNN by exploiting the framework of Gaussian width. [25] directly applied the above degrees of freedom result to the pruning problem, in the main purpose of unveiling the mechanisms behind the

Lottery Thicket Hypothesis (LTH) [6]. The lower bound of pruning ratio is briefly mentioned in [25], unfortunately, their predicted lower bound does not match the actual value well, in some cases even with big gap. And there is no discussion on the upper bound in that paper.

Despite the above progress, a systematic of the study on the fundamental limit of network pruning is still lacking. To close this gap, we'll take a first principles approach to address this problem and exploit the powerful framework of the high-dimensional convex geometry. Essentially, we impose the sparsity constraint directly on the loss function, thus we can reduce the pruning limit problem to a set intersection problem, then we leverage the powerful approximate kinematics formula [1] in convex geometry, which provides a very sharp phase transition point to easily address the set intersection problem, thus enabling a very tight characterization of the limit of network pruning. Intuitively speaking, the limit of pruning is determined by the dimension of the loss sublevel set (whose definition is in Sec. 2) of the network, the higher the latter, the smaller the former.

The key contributions of this paper can be summarized as follows:

We fully characterize the limit of network pruning, which coincides perfectly with the experiments. Moreover, this limit conveys two valuable messages: 1) The smaller the network sharpness (defined as the trace of the Hessian matrix), the more we can prune the network; 2) The smaller the weight magnitude , the more we can prune the network.

We provide an efficient spectrum estimation algorithm for large-scale Hessian matrices when computing the Gaussian width of a high-dimensional non-convex set.

We present intuitive explanations on many heuristics utilized in existing pruning algorithms through the lens of our pruning ratio limit, which include: (a). Why gradually changing the pruning ratio during iterative pruning is preferred. (b). Why employing l 2 regularization makes significant performance difference in Rare Gems algorithm [28]. (c).Why magnitude pruning might be the optimal pruning strategy.

1.1 Related Work

Pruning Methods: Unstructured pruning involves removing unimportant weights without adhering to some structural constraints. Typical methods in this class include: [13] presented the train-pruneretrain method, which reduces the storage and computation of neural networks by learning only the significant connections. [37; 38] employed the energy consumption of each layer as the metric to determine the pruning order and developed latency tables to identify the layers that should be pruned. [11] proposed dynamic network surgery, which reduced network complexity significantly by pruning connections in real time. [6] proposed pruning by iteratively removing part of the small weights, and based on Frankle's iterative pruning[6], [28] introduced l 2 -norm to constrain the magnitude of unimportant parameters during iterative training. To the best of our knowledge, there is still no systematic study on the fundamental limit of pruning from the theoretical perspective.

Understanding Neural Networks through Convex Geometry: Convex Geometry is a powerful tool for characterizing the performance limit of high-dimensional statistical inference [5] and learning problems. For statistical inference, [5] pioneered to employ the convex geometry to study the recovery threshold of the classical linear inverse problem. For statistical learning, [19] studied the training dimension threshold of the network from a geometric point of view by utilizing the Gordon's Escape theorem[10], which shows that the network can be trained with less degrees of freedom (DoF) than the network size in the affine subspace. The most relevant work to ours is [25], which studied the Lottery Tickets Hypothesis (LTH)[6] by applying the above DoF results in [19] to demonstrate that iteration is needed in LTH and that pruning is impacted by the eigenvalues of the loss landscape. The main difference between [25] and ours are as follows: 1) The lower bound of the pruning ratio is only briefly mentioned in [25], and their predicted lower bound does not match the actual value well, in some cases even with quite big gap (the main reason lies in the spectrum estimation error in their algorithm). 2) The core results in [25] are based on Gordon's Escape theorem [10], which can only provide the lower bound (necessary condition). 3) Rigorous analysis as well as the computational issues regarding the pruning limit are lacking in [25]. In contrast, all the above issues are addressed in this paper.

2 Problem Setup & Key Notions

To explore the fundamental limit of network pruning, we'll take the first principles approach. In specific, we directly impose the sparsity constraint on the original loss function, thus the feasibility of pruning can be reduced to determining whether two sets, i.e. the sublevel set (determined by the Hessian matrix of the loss function) and the k -sparse set intersects. Through this framework, we're able to leverage tools in high-dimensional convex geometry, such as statistical dimension [1], Gaussian width [34] and Approximate Kinematics Formula [1].

Model Setup. Let ˆ y = f ( w , x ) be a deep neural network with weights w ∈ R D and inputs x ∈ R K . For a given training data set { x n , y n } N n =1 and a loss function ℓ , the empirical loss landscape is defined as L ( w ) = 1 N ∑ N n =1 ℓ ( f ( w , x n ) , y n ) .

Pruning Objective. In essence, network pruning can be formulated as the following optimization problem:

where w is the pruned weight and w ∗ is the original one.

Sparse Network. Given a dense network with weights w ∗ , we denote its sparse counterpart as a k -sparse network, whose weight is given by: w k = w ∗ ⊙ m , where ⊙ is element-wise multiplication and ∥ m ∥ 0 = k .

Loss Sublevel Sets. A loss sublevel set of a network is the set of all weights w that achieve the loss up to L ( w ∗ ) + ϵ :

Feasible k -Sparse Pruning. We define the pruning ratio as ρ = k/D and call a sparse weight w k as a feasible k -sparse pruning if it satisfies:

Below are some key notions and results from high dimensional convex geometry, which are of critical importance to our work.

Definition 2.1 (Convex Cone & Conic Hull) A convex cone C ∈ R D is a convex set that satisfy: ∑ i η i x i ∈ C for all η i > 0 and x i ∈ C . The convex conic hull of a set S is defined as:

Definition 2.2 (Gaussian Width [34]) The gaussian width of a subset S ∈ R D is given by:

Gaussian width is useful to characterize the complexity of a convex body. On the other hand, statistical dimension is an important metric to characterize the complexity of convex cones. Intuitively speaking, the bigger the cone, the larger the statistical dimension, as illustrated in Fig. 1(b).

Definition 2.3 (Statistical Dimension [1]) The statistical dimension δ ( C ) of a convex cone C is:

where Π C is the Euclidean metric projector onto C and g ∼ N ( 0 , I D × D ) is a standard normal vector.

To characterize the sufficient and necessary condition of the set intersection, we'll resort to the powerful Approximate Kinematics Formula [1], which basically says that for two convex cones (or generally, sets), if the sum of their statistical dimension exceeds the ambient dimension, these two cones would intersect with probability 1, otherwise they would intersect with probability 0.

Theorem 2.4 (Approximate Kinematics Formula, Theorem 7.1 of [1]) Let C be a convex conic hull of a sublevel set S ( ϵ ) in R D , and draw a random orthogonal basis Q ∈ R D × D . For a k -dimensional subspace S k , it holds that:

samll cone

0

g

0

(b)

0

sublevel set

cone

(a)

subspace

sublevel set

Projection

Unit sphere

sublevel set

(c)

g

3 Bounds of Pruning Ratio

3.1 Lower Bound of Pruning Ratio

In this section, we aim to characterize the lower bound of pruning ratio, i.e. when the pruning ratio falls below some threshold, it's impossible to retain the generalization performance. To establish this impossibility result, we'll leverage the Approximate Kinematics Formula as detailed in Theorem 2.4.

3.1.1 Network Pruning As Set Intersection

To demonstrate that when k is smaller than a given threshold, it is impossible to find a performancepreserving k -sparse network induced by the dense network, we need to prove that the loss sublevel set has no intersection with any k -sparse set resulting from the dense weight, i.e. S ( ϵ ) ∩ { w k } = ∅ .

To that end, it suffices to prove its translated version, namely S w k ( ϵ ) ∩ { 0 } = ∅ , where S w k ( ϵ ) := { w -w k : w ∈ S ( ϵ ) } . To prove the latter, we'll further prove its strengthened version, i.e. the convex conic hull of S w k ( ϵ ) and a random orthogonal rotation of the subspace S ( w k ) , which is comprised of all vectors that share the same zero-pattern as w k (including the point 0 ), has no intersection. Namely, we'll prove that

where Q denotes the Haar-measured orthogonal rotation.

To prove Eq.7, we can easily invoke the necessary condition of the Approximate Kinematics Formula (Theorem 2.4). In order to calculate the involved statistical dimension therein, we choose to calculate the corresponding Gaussian width as the proxy by taking advantage of the following theorem.

Theorem 3.1 (Gaussian Width vs. Statistical Dimension, Proposition 10.2 of [1]) Given a unit sphere S D -1 := { x ∈ R D : ∥ x ∥ = 1 } , let C be a convex cone in R D , then:

To calculate the Gaussian width of C ( S w k ( ϵ )) , we need to project the sublevel set S w k ( ϵ ) onto the surface of the unit sphere centered at origin. which is defined as

and illustrated in Fig. 1(c). It is easy to see that as the distance ∥ x -w k ∥ 2 increases, the projected Gaussian width will decrease, as a result the statistical dimension of the set will also decrease, thus increasing the difficulty of its intersecting with a given subspace.

Theorem 3.2 (Lower Bound of Pruning Ratio) Let C be a convex conic hull of a sublevel set S w k ( ϵ ) in R D . w k doesn't constitute a feasible k -sparse pruning with probability 1, if the following holds:

This theorem tells us that when the dimension k of the sub-network is lower than k L = D -w ( p ( S w k ( ϵ ))) 2 , the subspace will not intersect with S w k ( ϵ ) , i.e., no feasible k -sparse pruning can be

found. Therefore, the lower bound of the pruning ratio of the network can be expressed as:

It's worth mentioning that this lower bound has been provided in [25] by utilizing the Gordon's Escape Theorem[10]. The main difference between our work and [10] lies in that Gordon's Escape Theorem is not strong enough to provide the upper bound (sufficient condition) of the pruning ratio, while the Approximate Kinematic Formula we employ does.

Reformulation of the Sublevel Set. Consider a well-trained deep neural network model with weights w ∗ and a loss function L ( w ) , where w lies in a small neighborhood of w ∗ . By performing a Taylor expansion of L ( w ) at w ∗ , using the fact that the first derivative is equal to 0 and ignoring the higher order terms, the loss sublevel set S ( ϵ ) can be reformulated as:

where ˆ w = w -w ∗ and H denote the Hessian matrix of L ( w ) w.r.t. w . Due to the positivedefiniteness of H , S ( ϵ ) corresponds to an ellipsoid. The related proofs can be found in Appendix D.1.

3.1.2 Gaussain Width of the Ellipsoid

We leverage tools in high-dimensional probability, especially the concentration of measure, which enables us to present a rather precise expression for the Gaussian width of a high-dimensional ellipsoid.

Lemma 3.3 For an ellipsoid S ( ϵ ) defined by : S ( ϵ ) := { w ∈ R D : 1 2 w T Hw ≤ ϵ } , where H ∈ R D × D is a positive definite matrix, its Gaussian width is given by:

where r i = √ 2 ϵ/λ i is the radius of ellipsoidal body and λ i is the i -th eigenvalue of H .

The proof of Lemma 3.3 is in Appendix D.1. The Gaussian width of an ellipsoid has been provided in [19] as in the interval [( √ 2 π ( ∑ D i =1 r 2 i ) 1 / 2 , ( ∑ D i =1 r 2 i ) 1 / 2 ] , in contrast we sharpen the estimation of Gaussian width to a point ( ∑ D i =1 r 2 i ) 1 / 2 . For the settings which involve projection, the squared radius r 2 i should be modified to r 2 i ∥ w ∗ -w k ∥ 2 2 + r 2 i [19]. Therefore, the Gaussian width of projected S ( ϵ ) defined in Eq.(12) equals:

3.1.3 Computable Lower Bound of Pruning Ratio

Combining Eq.(11) and Eq.(14), we obtain the following computable lower bound of the pruning ratio:

Corollary 3.4 Given a well-trained deep neural network with trained weight w ∗ ∈ R D and a loss function L ( w ) , for a k -sparse pruned weight w k , the lower bound of pruning ratio of model is:

where r i = √ 2 ϵ/λ i and λ i is the eigenvalue of the Hessian matrix of L ( w ) w.r.t. w .

3.1.4 Pruning Ratio vs Magnitude & Sharpness

It is evident from Eq.(15) that for a given trained network (whose spectrum of the Hessian matrix is fixed), to minimize the lower bound of the pruning ratio, we just need to minimize ∥ w ∗ -w k ∥ 2 , i.e. the sum of magnitudes of the pruned parameters. Therefore, the commonly-used magnitude-based pruning algorithms get justified. Moreover, it also inspires us to employ the one-shot magnitude pruning algorithm as detailed in Section 4, whose performance proves to be better than other existing algorithms, to the best of our knowledge.

Besides the above-discussed magnitude of the pruned sub-vector, we also identify another important factor that determines the pruning ratio, i.e. the network sharpness , which describes the sharpness of the loss landscape around the minima, as defined by the trace of the Hessian matrix, namely Tr( H ) ([26] and [9], network flatness is the opposite of network sharpness; as sharpness increases, flatness decreases, and vice versa.).

Lemma 3.5 (Pruning Ratio vs. Sharpness) Given a well-trained neural network f ( w ) , where w is the parameters. The lower bound of the pruning ratio and the sharpness obeys:

where H is the hessian matrix of the loss function w.r.t. w .

Lemma 3.5 is obtained by utilizing the Cauchy-Schwarz Inequality, whose proof can be found in Appendix F. It can be seen from Lemma 3.5 that the lower bound of the network pruning ratio is heavily dependent on the sharpness of the network, i.e. flatter networks imply more sparsity. This can be a valuable guideline for both training and pruning the networks. Intuitively, a flatter loss landscape is less sensitive to weight perturbations, indicating greater tolerance to weight removal.

3.2 Upper Bound of Pruning Ratio

In order to establish the upper bound of the pruning ratio, we need to prove that there exists an k -sparse weight vector intersects with the loss sub-level set.

For a given trained weight w ∗ , we split it into two parts, i.e. the unpruned subvector, w 1 = [ w ∗ 1 , w ∗ 2 , . . . , w ∗ k ] and the pruned one w 2 = [ w ∗ k +1 , w ∗ k +2 , . . . , w ∗ D ] . By fixing w 1 , the loss sublevel set can be reformulated as:

In order to prove the existence of a k -sparse weight vector w k , we just need to show that the all-zero vector is in S ( w ′ , ϵ ) . To that end, we'll take advantage of the sufficient condition of the approximate kinematics formula (Theorem 2.4) to show that it suffices to render the statistical dimension of the projected cone of S ( w ′ , ϵ ) being full, i.e. D -k . Thus we can obtain the upper bound of the number of unpruned parameters, i.e. k .

Specifically, by invoking the sufficient part of Theorem 2.4, the upper bound of the pruning ratio by a given pruning strategy is as follows:

Theorem 3.6 (Upper Bound of Pruning Ratio) Given a sublevel set S ( w ′ , ϵ ) in R D -k . To ensure that the all-zero vector 0 ∈ R D -k contained in S ( w ′ , ϵ ) , it suffices that:

The Gaussian width of projected S ( w ′ , ϵ ) can be easily obtained by employing Lemma 3.3, i.e. w ( p ( S ( w ′ , ϵ ))) 2 = ∑ D -k i ˜ r 2 i ∥ w ∗ -w k ∥ 2 2 + ˜ r 2 i , where ˜ r i = √ 2 ϵ/ ˜ λ i , ˜ λ i is the eigenvalue of the hessian matrix of L ([ w 1 , w ′ ]) w.r.t. to w ′ and the fact that ∥ w ∗ -w k ∥ 2 = ∥ w 2 ∥ 2 is used. Correspondingly, the upper bound of the pruning ratio can be expressed as

3.3 Fundamental Limit of Pruning Ratios

As demonstrated above, the pruning ratio can be bounded as follows:

It is easy to notice that the upper bound and lower bound are of nearly identical form. In fact, as we'll elaborate in the following, they are also of quite close value, which implies that we are able to obtain a sharp characterization of the fundamental limit of pruning ratio. Meanwhile, it is worthwhile noting that the pruning limit depends on the magnitude of the final weights, which might be significantly impacted by the weight initialization. Therefore, we still need to explore whether the magnitude of final weights is dependent on the initialization values. In the appendix, we'll demonstrate that once the data, network architecture and training method are fixed, the distribution of trained network weights remains nearly insensitive to the initializations, thus yielding an affirmative answer about the above question.

4 Achievable Scheme & Computational Issues

Thus far we have established the lower bound and upper bound of the pruning ratio by leveraging the Approximate Kinematic Formula in convex geometry [1]. To proceed, we will demonstrate that our obtained bounds are tight in the sense that we can devise an achievable pruning algorithm whose corresponding upper bound is quite close to the lowest possible value of the lower bound. As argued in Corollary 3.4, the magnitude pruning, which removes all the smallest D -k weights, will result in the lowest pruning ratio lower bound. Inspired by this result, we'll focus on the magnitude pruning methods in order to approach the lower bound in the sequel.

For the lower bound part, we need to address several challenges regarding the computation of the Gaussian width of a high-dimensional deformed ellipsoid, which involves tackling the nonpositiveness of the Hessian matrix as well as the spectrum estimation of a large-scale Hessian matrix.

For the upper bound part, we'll focus on a relaxed version of Eq. 1 by introducing the l 1 regularization term, for the sake of computational complexity. Then we'll employ the one-shot magnitude pruning to compress the network.

4.1 Computational Challenges & Countermeasures

To compute the lower bound of the pruning ratio, we need to address the following two challenges:

Gaussian Width of the Deformed Ellipsoid. In practice, it is usually hard for the network to converge to the exact minima, thus leading to a non-positive definite Hessian matrix. In other words, the ideal ellipsoid gets deformed due to the existence of negative eigenvalues. Determining the Gaussian width of the deformed ellipsoid is a challenging task. To address this problem, we resort to convexifying (i.e. taking the convex hull of) the deformed ellipsoid and then calculating the Gaussian width of the latter instead, by proving that the convexifying procedure has no impact on the Gaussian width. (The proof is presented in Appendix D.2).

Improved Spectrum Estimation. Neural networks often exhibit a quite significant number of zero-valued or vanishingly small eigenvalues in their Hessian matrices. It's hard for the spectrum estimation algorithm SLQ (Stochastic Lanczos Quadrature) proposed by [39] to obtain accurate estimation of these small eigenvalues. To address this issue, we propose to enhance the existing large-scale spectrum estimation algorithms by a key modification, i,e, to estimate the number of these exceptionally small eigenvalues by employing the Hessian matrix sampling. See Algorithm 1 for the details of the improved spectrum Estimation algorithm. A comprehensive description of the algorithm and its experimental results are presented in Appendix C.

4.2 Achievable Scheme: l 1 Regularization & One-shot Magnitude Pruning

Inspired by the lower bound as well as upper bound of the pruning ratio, in which the magnitude of pruning parameters plays a key role, it's sensible to focus on the magnitude-based pruning methods.

Algorithm 1 Improved SLQ (Stochastic Lanczos Quadrature) Spectrum Estimation Algorithm

Input: Hermitian matrix A of size n × n , Lanczos iterations m , ESD computation iterations l , gaussian kernel f and variance σ 2 , sampling num s and row l 1 norm threshold ϵ .

Output:

The spectral distribution of matrix A

for i = 2 , ..., l do

1). Get the tridiagonal matrix T of size m × m through Lanczos algorithm[18];

2). Compute m eigenpairs ( λ ( i ) k , v ( i ) k ) from T ;

3). ϕ i σ ( t ) = ∑ m k =1 τ ( i ) k f ( λ ( i ) k ; t, σ ) , where τ ( i ) k = ( v ( i ) k [1]) 2 is the first component of v ( i ) k ; end for

4). Random sample s rows A i of matrix A and calculate every l 1 norm ∥ A i ∥ 1 by take i ∈ [1 , s ] .

5). Compute the number z of ∥ A i ∥ 1 which satisfy ∥ A i ∥ 1 ≤ ϵ .

6). Compute the integration of 1 l ∑ l i =1 ϕ i σ ( t ) , termed as c , ϕ ( t ) = 1 l ∑ l i =1 ϕ i σ ( t ) + cz s -z δ (1 -30 ) . Return: ϕ ( t )

On the other hand, to find exact solutions of our original problem for the best pruning in Eq. 1, it is obviously very hard due to the existence of l 0 norm. To make it feasible, it's natural to perform a convex relaxation of l 0 norm, namely, by employing l 1 regularization instead. Aside from the computational advantage of this relaxation, it is worthy noting that l 1 regularization provides two extra benefits: 1) A large portion of eigenvalues of the trained Hessian matrix are zero-valued or of quite small value, which renders the calculation of the pruning limit more accurately and fast. 2) A large portion of trained weights are of quite small value, thus making the lower bound and upper bound very close. Detailed statistics about the eigenvalues and magnitudes can be found in Figure 6.

Specifically, by utilizing the Lagrange formulation and convex relaxation of l 0 norm, the pruning objective in Eq.1 can be reformulated as:

After training with this relaxed objective, the network weights will be pruned based on magnitudes one time , rather than in an iterative way as in [13; 6; 28]. The performance of the above described pruning scheme (termed as " l 1 regularized one-shot magnitude pruning " and abbreviated as "LOMP") can be found in Table 10 in Appendix. The above stated "zero-dominating" property due to l 1 regularization gets supported in Figure 2(b), where it can be seen that the majority of weights are indeed extremely small.

Unit sphere

(a)

(b)

(c)

400000

0

Frequency

300000

g10)

-5

601)

200000

2

-10

<

R

100000

15

0

-17.5-15.0-12.5-10.0-7.5 -5.0 -2.50.0

20

0

20

40

60

80

100

Param Value (log10)

The above "zero-domination" property turns out to be of critical value for our proposed pruning scheme to nearly achieve the limit (lower bound) of the pruning ratio. Fig. 2(c) illustrates the curve ∥ w 2 ∥ 2 2 , i.e. the l 2 norm of the D -k smallest weights, w.r.t. k/D . The vertical line therein represents ρ L , the lower bound of the pruning ratio predicted in Section 3.1. When k = Dρ L , the curve and the line will intersect as shown in Figure 2(c). Mathematically, the upper bound for the pruning ratio can be approximated as follows:

It can be seen from the above demonstration that the upper bound corresponding to our proposed pruning scheme almost coincides with the minimal lower bound! In other words, we have established the fundamental limit of the pruning ratio. To provide further validation of the above claim, we performed the experiments five times across eight tasks and reported the differences between the upper bound and lower bound, denoted as ∆ , in Table 9.

5 Experiments

In this section, we validate our pruning method as well as the theoretical limit of the pruning ratio by experiments.

Tasks. We evaluate the pruning ratio threshold on: Full-Connect-5(FC5), Full-Connect-12(FC12), AlexNet [17] and VGG16 [27] on CIFAR10 [16], ResNet18 and ResNet50 [14] on CIFAR100 and TinyImageNet [20]. We employ the l 1 regularization during training, and execute a one-shot magnitude-based pruning and assess its performance with various sparsity ratios, in terms of the metrics of accuracy and loss. Detailed descriptions of datasets, networks, hyper-parameters, and eigenspectrum adjustment can be found in Section B of the Appendix. Moreover, the performance comparison between l 1 -regularization-based magnitude pruning and other pruning methods can be found in Table 10 in Appendix.

5.1 Validation of Pruning Lower Bound

After training with the l 1 regularization, we compute the eigenvalues and present the theoretical limit of pruning ratio. By pruning the trained network to various sparsity levels, we depict in Figure 3 both the line of theoretical lower bound and the sparsity-accuracy curve for the above-listed tasks. From the figures we can see that our theoretical result matches the numerical pruning ratio quite well.

100

70

40

10

8

6

4

2

0

0.5

1.0

1.5

Acc & Loss

FC5/CIFAR10

100

70

40

10

8

6

4

2

0

0.5

1.0

FC12/CIFAR10

100

70

40

10

8

6

4

2

0

1

2

AlexNet/CIFAR10

100

70

40

10

8

6

4

2

0

0.5

1.0

1.5

2.0

VGG16/CIFAR10

100

70

40

10

8

6

4

2

0

Pruning Ratio(%)

0.0

0.5

1.0

Acc & Loss

ResNet18/CIFAR100

100

70

40

10

8

6

4

2

0

Pruning Ratio(%)

0.0

0.5

1.0

1.5

ResNet50/CIFAR100

100

70

40

10

8

6

4

2

0

Pruning Ratio(%)

0.0

0.5

1.0

1.5

ResNet18/TinyImageNet

100

70

40

10

8

6

4

2

0

Pruning Ratio(%)

0.0

0.5

1.0

ResNet50/TinyImageNet

Test Loss

Test Accuracy

5.2 Prediction Performance

We present a more detailed comparison between our theoretical limit of pruning ratio, and the actual values by experiments in Table 2, which shows nearly perfect agreement between them. The difference between the theoretical value and the actual value is donated as ∆ .

6 Interpretation of Pruning Heuristics

Equipped with the fundamental limit of network pruning, we're now able to provide rigorous interpretations of several heuristics employed by existing pruning algorithms.

0

50

100

150

200

R (level 7)

0.0

0.5

1.0

Pruning Ratio

TinyImageNet

IMP: ResNet50

0

50

100

150

200

R (level 11)

0.0

0.5

1.0

0

50

100

150

200

R (level 18)

0.0

0.5

1.0

0

50

100

150

200

R (level 20)

0.0

0.5

1.0

0

50

100

150

200

R (without l2)

0.0

0.5

1.0

Pruning Ration

TinyImageNet

Rare Gems: ResNet50

0

50

100

150

200

R (with l2)

0.0

0.5

1.0

0

50

100

150

200

R (without l2)

0.0

0.5

1.0

0

50

100

150

200

R (with l2)

0.0

0.5

1.0

Theory Prediction

Pruning ratio adjustment is needed in IMP. For the IMP (Iterative Magnitude Pruning) algorithm [7], we determine the pruning ratio thresholds for various stages through calculations, as depicted in the top row of Figure 4. It is noteworthy that as the pruning depth gradually increases, the theoretical pruning ratio threshold also increases. Therefore, it is appropriate to prune smaller proportions of weights gradually during iterative pruning, Both [43] and [28] have employed pruning rate adjustment, which gradually prunes smaller proportions of the weights with the iteration of the algorithm.

l 2 -regularization enhances the performance of Rare Gems. For the Rare Gems algorithm [28], it is shown that l 2 regularization makes a significant difference in terms of the final performance, as shown in the bottom row of Figure 4. The main reason behind this phenomenon is: when l 2 -regularization is applied, the pruning ratio tends to be larger than the theoretical limit, however, the absence of l 2 -regularization would result in excessive pruning, which can be regarded as wrong pruning.

7 Conclusion

In this paper we explore the fundamental limit of pruning ratio of deep networks by taking the first principles approach and leveraging the framework of convex geometry. Specifically, we reduce the pruning limit problem to the sets intersection problem, and by taking advantage of the powerful Approximate Kinematic Formula, we are able to sharply characterize the fundamental limit of the network pruning ratio. This fundamental limit conveys a key message as follows: the network pruning limit is mainly determined by the weight magnitude and the network sharpness . These two guidelines can provide intuitive explanations of several heuristics in existing pruning algorithms. Moreover, to address the challenges in computing the involved Gaussian width, we develop an improved spectrum estimation for large-scale and non-positive Hessian matrices. Experiments demonstrate the almost perfect agreement between our theoretical results and the experimental ones.

Limitations. In this paper, the (almost) coincidence of the upper bound and lower bound of the pruning ratio depends on the condition that the removed weights are of quite small value, which is enabled by the l 1 regularization we employed. Therefore, it is important to demonstrate whether the l 1 regularized training is optimal or nearly optimal in the sense of obtaining the smallest sub-network without performance degradation for the original learning problem.

Acknowledgments and Disclosure of Funding

This work was supported in part by the National Key Research and Development Program of China under Grant 2024YFE0103800, in part by the National Natural Science Foundation of China under Grant 62071192. We thank the anonymous reviewers for their valuable and constructive feedback that helped in shaping the final manuscript.

References

Dennis Amelunxen, Martin Lotz, Michael B McCoy, and Joel A Tropp. Living on the edge: Phase transitions in convex programs with random data. Information and Inference: A Journal of the IMA, 3(3):224-294, 2014.

Haim Avron and Sivan Toledo. Randomized algorithms for estimating the trace of an implicit symmetric positive semi-definite matrix. Journal of the ACM (JACM), 58(2):1-34, 2011.

Zhaojun Bai, Gark Fahey, and Gene Golub. Some large-scale matrix computation problems. Journal of Computational and Applied Mathematics, 74(1-2):71-89, 1996.

Riade Benbaki, Wenyu Chen, Xiang Meng, Hussein Hazimeh, Natalia Ponomareva, Zhe Zhao, and Rahul Mazumder. Fast as chita: Neural network pruning with combinatorial optimization. In International Conference on Machine Learning, pages 2031-2049. PMLR, 2023.

Venkat Chandrasekaran, Benjamin Recht, Pablo A Parrilo, and Alan S Willsky. The convex geometry of linear inverse problems. Foundations of Computational mathematics, 12:805-849, 2012.

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018.

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259-3269. PMLR, 2020.

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Pruning neural networks at initialization: Why are we missing the mark? arXiv preprint arXiv:2009.08576, 2020.

Khashayar Gatmiry, Zhiyuan Li, Ching-Yao Chuang, Sashank Reddi, Tengyu Ma, and Stefanie Jegelka. The inductive bias of flatness regularization for deep matrix factorization. arXiv preprint arXiv:2306.13239, 2023.

Yehoram Gordon. On milman's inequality and random subspaces which escape through a mesh in R n . In Geometric Aspects of Functional Analysis: Israel Seminar (GAFA) 1986-87, pages 84-106. Springer, 1988.

Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. Advances in neural information processing systems, 29, 2016.

Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.

Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.

Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4340-4349, 2019.

A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.

Cornelius Lanczos. An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. J. Res. Natl. Bur. Stand. B, 45:255-282, 1950.

Brett W Larsen, Stanislav Fort, Nic Becker, and Surya Ganguli. How many degrees of freedom do we need to train deep networks: a loss landscape perspective. arXiv preprint arXiv:2107.05802, 2021.

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.

Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.

Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710, 2016.

Jian-Hao Luo, Hao Zhang, Hong-Yu Zhou, Chen-Wei Xie, Jianxin Wu, and Weiyao Lin. Thinet: pruning cnn filters for a thinner net. IEEE transactions on pattern analysis and machine intelligence, 41(10):2525-2538, 2018.

Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440, 2016.

Mansheej Paul, Feng Chen, Brett W Larsen, Jonathan Frankle, Surya Ganguli, and Gintare Karolina Dziugaite. Unmasking the lottery ticket hypothesis: What's encoded in a winning ticket's mask? arXiv preprint arXiv:2210.03044, 2022.

Henning Petzka, Michael Kamp, Linara Adilova, Cristian Sminchisescu, and Mario Boley. Relative flatness and generalization. Advances in neural information processing systems, 34:1842018432, 2021.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.

Kartik Sreenivasan, Jy-yong Sohn, Liu Yang, Matthew Grinde, Alliot Nagle, Hongyi Wang, Eric Xing, Kangwook Lee, and Dimitris Papailiopoulos. Rare gems: Finding lottery tickets at initialization. Advances in Neural Information Processing Systems, 35:14529-14540, 2022.

Jingtong Su, Yihang Chen, Tianle Cai, Tianhao Wu, Ruiqi Gao, Liwei Wang, and Jason D Lee. Sanity-checking pruning methods: Random tickets can win the jackpot. Advances in neural information processing systems, 33:20390-20401, 2020.

M. Talagrand. New concentration inequalities for product spaces. Inventionnes Math., 126:505563, 1996.

M. Talagrand. A New Look at Independence. Ann. Probab., 24:1-34, 1996.

Michel Talagrand. Concentration of measure and isoperimetric inequalities in product spaces. Publications Mathématiques de l'Institut des Hautes Etudes Scientifiques, 81:73-205, 1995.

Terence Tao. Topics in random matrix theory, volume 132. American Mathematical Soc., 2012.

Roman Vershynin. Estimation in high dimensions: a geometric perspective. In Sampling Theory, a Renaissance: Compressive Sensing and Other Developments, pages 3-66. Springer, 2015.

Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 409-424, 2018.

Qian Xiang, Xiaodan Wang, Yafei Song, Lei Lei, Rui Li, and Jie Lai. One-dimensional convolutional neural networks for high-resolution range profile recognition via adaptively feature recalibrating and automatically channel pruning. International Journal of Intelligent Systems, 36(1):332-361, 2021.

Tien-Ju Yang, Yu-Hsin Chen, and Vivienne Sze. Designing energy-efficient convolutional neural networks using energy-aware pruning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5687-5695, 2017.

Tien-Ju Yang, Andrew Howard, Bo Chen, Xiao Zhang, Alec Go, Mark Sandler, Vivienne Sze, and Hartwig Adam. Netadapt: Platform-aware neural network adaptation for mobile applications. In Proceedings of the European conference on computer vision (ECCV), pages 285-300, 2018.

Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data), pages 581-590. IEEE, 2020.

Lei You and Hei Victor Cheng. Swap: Sparse entropic wasserstein regression for robust network pruning. In The Twelfth International Conference on Learning Representations, 2024.

Xin Yu, Thiago Serra, Srikumar Ramalingam, and Shandian Zhe. The combinatorial brain surgeon: pruning weights that cancel one another in neural networks. In International Conference on Machine Learning, pages 25668-25683. PMLR, 2022.

Hao Zhou, Jose M Alvarez, and Fatih Porikli. Less is more: Towards compact cnns. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 662-677. Springer, 2016.

Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017.

Liu Ziyin and Zihao Wang. spred: Solving l1 penalty with sgd. In International Conference on Machine Learning, pages 43407-43422. PMLR, 2023.

A Organization of Appendix

The appendix is organized as follows:

Sec. A: an overview of the organization of the appendix.

Sec. B: detail descriptions of the datasets, models, hyper-parameter choices used in our experiments. Additionally, figures illustrating the theoretical pruning threshold are included.

Sec. C: this section delves into the practical calculation of the Gaussian Width. It addresses the challenges associated with the non-positiveness of the Hessian matrix and the spectrum estimation of a large-scale Hessian matrix. Experimental results highlighting the "important eigenvalues" are also showcased.

Sec. D: a comprehensive proof of the Gaussian Width for both the ellipsoid and the deformed ellipsoid is provided.

Sec. E: this section presents the proof of the "sub-sublevel set" utilized in deriving the upper bound of the pruning ratio. Furthermore, a straightforward explanation of the relationship between the general lower bound and the upper bound is offered.

Sec. F: omitted proof of the connection between the sharpness and the lower bound of the pruning ratio is detailed in this section.

Sec. G: performance comparison between the l 1 regularized one-shot magnitude pruning, termed as "LOMP", and several prominent pruning strategies. Results from a hypothetical experiment verifying the importance of magnitude in pruning and comprehensive statistical results from Sec. 4.2 are also included.

Sec. H: limitations of our assumptions and theoretical results.

Sec. I: broader impacts statement of this research.

B Experimental Details

In this section, we describe the datasets, models, hyper-parameter choices and eigenspectrum adjustment used in our experiments. All of our experiments are run using PyTorch 1.12.1 on Nvidia RTX3090s with ubuntu20.04-cuda11.3.1-cudnn8 docker.

B.1 Dataset

CIFAR-10. CIFAR-10 consists of 60,000 color images, with each image belonging to one of ten different classes with size 32 × 32 . The classes include common objects such as airplanes, automobiles, birds, cats, deer, dogs, frogs, horses, ships, and trucks. The CIFAR-10 dataset is divided into two subsets: a training set and a test set. The training set contains 50,000 images, while the test set contains 10,000 images [16]. For data processing, we follow the standard augmentation: normalize channel-wise, randomly horizontally flip, and random cropping.

CIFAR-100. The CIFAR-100 dataset consists of 60,000 color images, with each image belonging to one of 100 different fine-grained classes [16]. These classes are organized into 20 superclasses, each containing 5 fine-grained classes. Similar to CIFAR-10, the CIFAR-100 dataset is split into a training set and a test set. The training set contains 50,000 images, and the test set contains 10,000 images. Each image is of size 32x32 pixels and is labeled with its corresponding fine-grained class. Augmentation includes normalize channel-wise, randomly horizontally flip, and random cropping.

TinyImageNet. TinyImageNet comprises 100,000 images distributed across 200 classes, with each class consisting of 500 images [20]. These images have been resized to 64 × 64 pixels and are in full color. Each class encompasses 500 training images, 50 validation images, and 50 test images. Data augmentation techniques encompass normalization, random rotation, and random flipping. The dataset includes distinct train, validation, and test sets for experimentation.

B.2 Model

In all experiments, pruning skips bias and batchnorm, which have little effect on the sparsity of the network. Non-affine batchnorm is applied in the network, and the initialization of the network is kaiming normal initialization.

Full Connect Network(FC-5, FC-12). We train a five-layer fully connected network (FC-5) and a twelve-layer fully connected network FC-12 on CIFAR-10, the network architecture details can be found in Table 3.

AlexNet [17]. We use the standard AlexNet architecture. In order to use CIFAR-10 to train AlexNet, we upsample each picture of CIFAR-10 to 3 × 224 × 224 . The detailed network architecture parameters are shown in Table 4.

VGG-16 [27]. In the original VGG-16 network, there are 13 convolution layers and 3 FC layers (including the last linear classification layer). We follow the VGG-16 architectures used in [7; 8] to remove the first two FC layers while keeping the last linear classification layer. This finally leads to a 14-layer architecture, but we still call it VGG-16 as it is modified from the original VGG-16 architectural design. Detailed architecture is shown in Table 5. VGG-16 is trained on CIFAR-10.

ResNet-18 and ResNet-50 [14]. We use the standard ResNet architecture for TinyImageNet and tune it for the CIFAR-100 dataset. The detailed network architecture parameters are shown in Table 6. ResNet-18 and ResNet-50 is trained on CIFAR-100 and TinyImageNet.

B.3 Training Hyper-parameters Setup

In this section, we will describe in detail the training hyper-parameters of the Global One-shot Pruning algorithm on multiple datasets and models. The various hyperparameters are detailed in Table 7.

B.4 Sublevel Set Parameters Setup.

Given a dense well-trained neural network with weighted donated as w ∗ , the loss sublevel set is defined as { ˆ w ∈ R D : 1 2 ˆ w T H ˆ w ≤ ϵ } where ˆ w = w -w ∗ , under the assumption that the test data is often unavailable and we also generally assume that the training and test data share the same distribution, thus we use the training data to define the loss sublevel set. We compute the standard deviation of the network's loss across multiple batches on the training data set and denote it by ϵ .

B.5 Theoretical Pruning Ratio

Taking w ∗ as the initial pruning point and calculating the corresponding value of R = ∥ w k -w ∗ ∥ 2 for different pruning ratios k/D . We then plot the corresponding curve of the theoretically predicted pruning ratio and the calculated R in the same graph. The intersection point of these two curves is taken as the lower bound of the theoretically predicted pruning ratio. All results are shown in Figure 5.

0

20

40

60

80

R

0.0

0.5

1.0

Pruning Ratio

0

5

10

15

20

R

0.0

0.5

1.0

0

20

40

60

80

R

0.0

0.5

1.0

0

1

2

3

R

0.0

0.5

1.0

0

2

4

6

R

0.0

0.5

1.0

Pruning Ratio

0

100

200

300

400

R

0.0

0.5

1.0

0

250

500

750

1000

R

0.0

0.5

1.0

0

5

10

R

0.0

0.5

1.0

Theory Prediction

C Practical Calculation of Gaussian Width

In practical experiments, determining the Gaussian width of the ellipsoid defined by the network loss function is a challenging task. There are two primary challenges encountered in this section: 1.) the computation of eigenvalues for high-dimensional matrices poses significant difficulty; 2.) the network fails to converge perfectly to the extremum, leading to a non-positive definite Hessian matrix for the loss function. In this section, we tackle these challenges through the utilization of a fast eigenspectrum estimation algorithm and an algorithm that approximates the Gaussian width of a deformed ellipsoid body. These approaches effectively address the aforementioned problems.

C.1 Improved SLQ (Stochastic Lanczos Quadrature) Spectrum Estimation

0

25

50

75

100

Hessian Matrix Index(%)

0.0

0.2

0.4

0.6

Row L1 Norm

0

25

50

75

100

Hessian Matrix Index(%)

0.00

0.25

0.50

0.75

0

25

50

75

100

Hessian Matrix Index(%)

0

1

2

0

25

50

75

100

Hessian Matrix Index(%)

0

2

4

0

25

50

75

100

Hessian Matrix Index(%)

0

50

100

Row L1 Norm

0

25

50

75

100

Hessian Matrix Index(%)

0

1

2

0

25

50

75

100

Hessian Matrix Index(%)

0.0

0.5

1.0

0

25

50

75

100

Hessian Matrix Index(%)

0

1000

2000

Row L1 Norm

Calculating the eigenvalues of large matrices has long been a challenging problem in numerical analysis. One widely used method for efficiently computing these eigenvalues is the Lanczos algorithm[18], which is presented in Algorithm 2. However, due to the huge amount of parameters of the deep neural network, it is still impractical to use this method to calculate the eigenspectrum of the Hessian matrix of a deep neural network. To tackle this problem, [39] proposed SLQ (Stochastic Lanczos Quadrature) Spectrum Estimation Algorithm, which estimates the overall eigenspectrum distribution based on a small number of eigenvalues obtained by Lanczos algorithm. This method enables the efficient computation of the full eigenvalues of large matrices. Algorithm 2 outlines the step-by-step procedure for the classic Lanczos algorithm, providing a comprehensive guide for its implementation. The algorithm requires the selection of the number of iterations, denoted as m , which determines the size of the resulting triangular matrix T .

In general, the Lanczos algorithm is not capable of accurately computing zero eigenvalues, and this limitation becomes more pronounced when the SLQ algorithm has a small number of iterations. Similarly, vanishingly small eigenvalues are also ignored by Lanczos. However, in a well-trained large-scale deep neural network, the experiment found that the network loss function hessian matrix has a large number of zero eigenvalues and vanishingly small eigenvalues. In the Gaussian width of the ellipsoid, the zero eigenvalues and vanishingly small eigenvalues have the same effect on the width (insensitive to other parameters), and we collectively refer to these eigenvalues as the 'important' eigenvalues. We divide the weight into 100 parts from small to large, calculate the second-order derivative (including partial derivative) of smallest weight in each part, and sum the absolute values of all second-order derivatives of the weight, which corresponds to l 1 -norm of a row in hessian matrix, and the row l 1 -norm is zero or a vanishingly small corresponds to an 'important' eigenvalue, the experimental results can be seen in the Figure 6, from which the number of missing eigenvalues of the SLQ algorithm can be estimated, we then add the same number of 1e-30 as the missing eigenvalues in the Hessian matrix eigenspectrum. All the SLQ algorithm parameters are discribed in Table 8 and the statistical analysis of the l 1 norm of Hessian matrix rows for all experiments is presented in Figure 6. For details of the SLQ algorithm and the improved SLQ algorithm, see Algorithm 3 and Algorithm 1

Algorithm 2 The Lanczos Algorithm

Input: a Hermitian matrix A of size n × n , a number of iterations m Output: a tridiagonal real symmetric matrix T of size m × m initialization: 1. Draw a random vector v 1 of size n × 1 from N (0,1) and normalize it; 2. w ′ 1 = Av 1 ; α 1 = < w ′ 1 , v 1 > ; w 1 = w ′ 1 -α 1 v 1 ; 3. for j = 2 , ..., m do 1). β j = ∥ w j -1 ∥ ; 2). if β j = 0 then stop else v j = w j -1 /β j end if 3). w ′ j = Av j ; 4). α j = < w ′ j , v j > ; 5). w j = w ′ j -α j v j -β j v j -1 ; end for 4. T ( i, i ) = α i , i = 1 , . . . , m ; T ( i, i +1) = T ( i +1 , i ) = β i , i = 1 , . . . , m -1 . Return: T

Algorithm 3 SLQ(Stochastic Lanczos Quadrature) Spectrum Estimation Algorithm

Input: A hermitian matrix A of size n × n , Lanczos iterations m , ESD computation iterations l , gaussian kernel f and variance σ 2 .

Output: The spectral distribution of matrix A

for i = 2 , ..., l do

1). Get the tridiagonal matrix T of size m × m through Lanczos algorithm;

2). Compute m eigenpairs ( λ ( i ) k , v ( i ) k ) from T ;

3).

ϕ

i

σ

end for

4).

ϕ

(

t

) =

1

∑

l Return: ϕ ( t )

∑

m

k

=1

l

i

i

=1

ϕ

τ

σ

(

k

(

t

i

)

)

C.2 Gaussian Width of the Deformed Ellipsoid

After effective training, it is generally assumed that a deep neural network will converge to the global minimum of its loss function. However, in practice, even after meticulous tuning, the network tends to oscillate around the minimum instead of converging to it. This leads to that the Hessian matrix of the loss function would be non-positive definite, and the resulting geometric body defined by this matrix would change from an expected ellipsoid to a hyperboloid, which is unfortunately nonconvex. To quantify the Gaussian width of the ellipsoid corresponding to the perfect minima, we propose to approximate it by convexifying the deformed ellipsoid through replacing the associated negative eigenvalues with its absolute value. This processing turns out to be very effective, as demonstrated by the experimental results.

Lemma C.1 Consider a well-trained neural network with weights w , whose loss function defined by w has a Hessian matrix H . Due to the non-positive definiteness of H , there exist negative eigenvalues. Let the eigenvalue decomposition of H be H = v T Σv , where Σ is a diagonal matrix of eigenvalues. Let D = v T | Σ | v , where | · | means absolute operation. the geometric objects defined by H and D are S ( ϵ ) := { w ∈ R D : 1 2 w T Hw ≤ ϵ } and ˆ S ( ϵ ) := { w ∈ R D : 1 2 w T Dw ≤ ϵ } , then:

(

t

) =

f

(

λ

(

i

k

)

;

t, σ

)

, where

τ

(

k

i

)

= (

v

(

i

k

)

[1])

2

is the first component of

v

(

i

k

)

.

The proof of Lemma C.1 is in Appendix D.2. Lemma C.1 indicates that if the deep neural network converges to a vicinity of the global minimum of the loss function, the Gaussian width of the deformed ellipsoid body can be approximated by taking the convex hull of S ( ϵ ) .

D Gaussian Width of the Sublevel Set

In this section, we provide detailed proofs regarding the Gaussian Width of the sublevel sets of quadratic wells.

D.1 Gaussian Width of the Quadratic Well

Gaussian width is an extremely useful tool to measure the complexity of a convex body. In our proof, we will use the following expression for its definition:

Concentration of measure is a universal phenomenon in high-dimensional probability. Basically, it says that a random variable which depends in a smooth way on many independent random variables (but not too much on any of them) is essentially constant .[30; 31; 32]

Theorem D.1 (Gaussian concentration) Consider a random vector x ∼ N ( 0 , I n × n ) and an LLipschitz function f : R n → R (with respect to the Euclidean metric). Then for t ≥ 0

Therefore, if ϵ is small, f ( x ) can be approximated as f ( x ) ≈ E f ( x ) + √ -2 L 2 ln ϵ .

Lemma D.2 Given a random vector x ∼ N ( 0 , I n × n ) and the inverse of a positive definite Hessian matrix Q = H -1 , where H ∈ R n × n , we have:

Proof.

1.) Concentration of x T Qx

Define f ( x ) = x T Qx , we have:

Eigenvalue Decomposition of Q : Q = UΣU T . w. p. almost 1. Invariance of Gaussian under rotation.

where λ i is the eigenvalue of Q . The lipschitz constant L f of function f ( x ) is :

Let g ( x i ) = 2 λ i x i , whose lipschitz constant is L g = | 2 λ i | . Invoking Theorem D.1, we have:

Therefore, the lipschitz constant of f ( x ) can be approximated by:

Invoking Theorem D.1 again, we establish the concentration of f ( x ) as follows:

2.) Jensen ratio of √ x T Qx :

Therefore, the Jensen ratio of √ f ( x ) can be approximated by:

If Q is a Wishart matrix, i.e. Q = A T A , where A is a random matrix whose elements are independently and identically distributed with unit variance, according to the Marchenko-Pastur law [33], the maximum eigenvalue of Q is approximately 4 n and the trace of Q is approximately n 2 . Therefore, the above Jensen ratio approaches to 1 with decaying rate O ( 1 n ) .

For the inverse of a positive definite Hessian matrix which is of our concern, we take ϵ 1 = ϵ 2 = 10 -4 , numerical simulations show that when the dimension n = 10 5 , the corresponding δ in the above Jensen ratio is on the order of 10 -3 , which is in good agreement with the theoretical value and is arguably negligible. Similar as the case of the above-discussed Wishart matrix, when the dimension n increases, the value of δ will further decrease.

Consequently, we can conclude that E √ f ( x ) ≈ √ E f ( x ) , i.e. E √ x T Qx ≈ √ E x T Qx .

Definition D.3 (Definition of ball) A (closed) ball B ( c, r ) (in R D ) centered at c ∈ R D with radius r is the set

The set B (0 , 1) is called the unit ball . An ellipsoid is just an affine transformation of a ball.

Lemma D.4 (Definition of ellipsoid) . An ellipsoid S centered at the origin is the image L ( B (0 , 1)) of the unit ball under an invertible linear transformation L : R D → R D . An ellipsoid centered at a general point c ∈ R D is just the translate c + S of some ellipsoid S centered at the origin.

Proof.

where Q = LL T is positive definite .

The radius r i along principal axis e i obeys r 2 i = 1 λ i , where λ i is the eigenvalue of Q -1 and e i is eigen vector.

Lemma D.5 (Gaussian width of ellipsoid) . Let S be an ellipsoid in R D defined by the positive definite matrix H ∈ R D × D :

Then w ( S ) 2 or the squared Gaussian width of the ellipsoid satisfies:

where r i = √ 2 ϵ/λ i with λ i is i -th eigenvalue of H .

Concentration of f ( x )

Taylor Expansion

Proof. Let g ∼ N ( 0 , I D × D ) and LL T = 2 ϵ H -1 . Then:

Thus, w ( S ) 2 ≈ 2 ϵ Tr( H -1 ) = ∑ D i r 2 i .

D.2 Gaussian Width of the Deformed Ellipsoid

Generally, it is assumed that the gradient descent algorithm will converge to a minimum point. However, in practice, even with small learning rates, the network may oscillate near the minimum point and not directly converge to it, but rather get very close to it. As a result, the actual Hessian matrix is often not positive definite and its eigenvalues may have negative values.

Lemma D.6 Let the Hessian matrix at the minimum point be denoted by H with eigenvalue λ i , and the Hessian matrix at an oscillation point be denoted by ˆ H with eigenvalue ˆ λ i . The negative eigenvalues of ˆ H have small magnitudes.

Proof. Let the weights at the minimum point be denoted by w 0 and the Hessian matrix at an oscillation point be denoted by ˆ w 0 . Consider a loss function L and a loss landscape defined by L ( w ) , taking Taylor expansion of L ( w ) at w 0 :

Let ˆ w 0 = w 0 + v with v is closed to 0 :

Therefore, the second order derivative of L ( ˆ w 0 ) is:

where L ′′ ( w ) = ˆ H , Let H = ˆ H + H 0 with H 0 is closed to 0 , considering the Weyl inequality:

where ∥ H 0 ∥ 2 is small enough. So if ˆ λ i ( ˆ H ) is less than 0, since ˆ λ i ( ˆ H ) ≥ λ i ( H ) - ∥ H 0 ∥ 2 , its absolute value | ˆ λ i ( ˆ H ) | ≤ ∥ H 0 ∥ 2 -λ i ( H ) ≤ ∥ H 0 ∥ 2 , which means that the negative eigenvalues of the Hessian matrix have small magnitudes.

Lemma D.7 For a sublevel set S ( ϵ ) := { w : w T Hw ≤ ϵ } defined by a matrix H with small magnitude negative eigenvalues. The Gaussian width of S ( ϵ ) can be estimated by obtaining the absolute values of the eigenvalues of the matrix H .

Definition of Ball.

Invariance of Gaussian under rotation.

Proof. Assuming that the eigenvalue decomposition of H is H = v T Σv , where Σ is a diagonal matrix consisting of the eigenvalues of H , let D = v T | Σ | v be a positive definite matrix and M = H -D = v T ( Σ -| Σ | ) v be a negative definite matrix. Consider the definition of S ( ϵ ) :

Therefore, S ( ϵ ) can be expressed as w T Dw ≤ ϵ -w T Mw . Since the magnitudes of the negative eigenvalues of H are very small, we can assume that w T Mw is also small, and thus w T Dw ≤ ϵ -w T Mw can be approximately equal to w T Dw ≤ ϵ . As a result, we can estimate the Gaussian width of S ( ϵ ) by approximating it with the absolute values of the eigenvalues of H .

Corollary D.8 Consider a well-trained neural network with weights w , whose loss function defined by w has a Hessian matrix H . Due to the non-positive definiteness of H , there exist negative eigenvalues. Let the eigenvalue decomposition of H be H = v T Σv , where Σ is a diagonal matrix of eigenvalues. Let D = v T | Σ | v , where | · | means absolute operation. the geometric objects defined by H and D are S ( ϵ ) := { w ∈ R D : 1 2 w T Hw ≤ ϵ } and ˆ S ( ϵ ) := { w ∈ R D : 1 2 w T Dw ≤ ϵ } , then the gaussian width of the two set satisfy:

E Comparison between the Upper and Lower Bound

This section provided the proofs used in the upper bound derivation and roughly analyzed how the lower bound changes when the upper bound varies.

E.1 D -k Dimension Sublevel Set is Still an Ellipsoid

In the derivation of the upper bound for the pruning ratio threshold, we employed a D -k dimensional loss sublevel set:

Perform Taylor expansion to L ([ w 1 , w ′ ]) with respect to w ′ , the sublevel set is represented as:

where H ′ is the Hessian matrix of L ([ w 1 , w ′ ]) with respect to w ′ .

Given that the full sublevel set S ( ϵ ) = { w ∈ R D : 1 2 w T Hw ≤ ϵ } is an ellipsoid body, which implies that H is a positive definite matrix, it is evident that H ′ is the principal submatrix of H . Consequently, H ′ is also a positive definite matrix, which implies that the sublevel set S ( w ′ ) remains an ellipsoid.

E.2 Relationship between the Upper and Lower Bound

Theorem E.1 (Eigenvalue Interlacing Theorem) Suppose A ∈ R n × n is symmetric, Let B ∈ R m × m with m < n be a principal submatrix(obtained by deleting both i -th row and i -th column for some values of i ). Suppose A has eigenvalues λ 1 ≤ · · · ≤ λ n and B has eigenvalues β 1 ≤ · · · ≤ β m . Then

Next, we provide an elucidation on the relationship between the upper bound and lower bound variations:

Lemma E.2 The direct and straightforward relationship between the upper bound and the lower bound can be articulated as follows:

When the eigenvalues change, the upper and lower bounds will change in the same direction;

When the weight magnitude changes, the upper bound will change in the same direction as the upper bound or do not change.

Proof. 1. When the eigenvalues change, the upper and lower bounds will change in the same direction;

By leveraging the Eigenvalue Interlacing Theorem (Theorem E.1), the eigenvalues of the principal submatrix in the upper bound is bounded by the eigenvalues in the lower bound. It's obvious that if the eigenvalues in the lower bound change, the eigenvalues in the upper bound will also change in the same direction, leading to the upper and lower bounds will change in the same direction.

When the weight magnitude changes, the upper bound will change in the same direction as the lower bound or do not change.

It's noted that the number of weights in the lower bound is more than the one of weights in the upper bound. These weights are used to calculate the projection distance. So it's clear that when the weight magnitude changes, the upper bound will change in the same direction as the lower bound or not change.

And if m = n -1 ,

F Sharpness & Lower Bound

In this section, we will provide the relationship between the lower bound of the pruning ratio and the sharpness of the loss landscape w.r.t the weights. We first introduce the definition of sharpness, which is similar to the sharpness definition in [26] and [9]:

Definition F.1 Given a second-order derivable function f ( w ) , where w is the parameters. Considering a hessian matrix H w.r.t. parameters w , the sharpness of f ( w ) w.r.t. parameters is defined as the trace of H , i.e. Tr( H ) .

As defined in definition F.1, a smaller trace indicates a flatter function. Next, we will provide the connection between the sharpness and the lower bound:

Lemma F.2 Given a well-trained neural network f ( w , x ) , where w is the parameters and the x is the input. The lower bound of pruning ratio ρ l and the sharpness Tr( H ) obeys:

where H is the hessian matrix of f ( w ) w.r.t. w .

Proof.

Cauchy-Schwarz Inquality.

It's obvious that if the trace of the hessian matrix becomes smaller, the lower bound will also decrease, indicating a higher sparsity level. Utilizing sharpness as the optimization objective for network pruning is both a rational and efficacious approach.

Corollary F.3 Given a well-trained neural network f ( w , x ) , where w is the parameters and the x is the input. The pruning ratio lower bound and the sharpness obeys:

where H is the hessian matrix of f ( w ) w.r.t. w . An informal version of this corollary can be stated as: sharpness controls the lower bound of the pruning ratio, specifically, a flatter neural network can be pruned more sparsely.

G Full Results

Here we present the full set of experiments performed for the results in the main text.

G.1 The Distance Between the Distribution of Weights

In this section, we provide the total variation(TV) distance between the distribution of trained weights with independent initialization.

G.2 Comparison of Pruning Algorithms

As discussed in Section 4, we adopt l 1 regularization during the training phase, and the hyperparameter for the l 1 regularization is selected empirically. After thorough training, we applied magnitude pruning to reduce the network to the lowest pruning ratio at which the network's performance is maintained, and we didn't apply fine-tuning after pruning.

We validated l 1 regularized one-shot magnitude pruning algorithm(LOMP) against four baselines: dense weight training and three pruning algorithms: (i) Rare Gems(RG) proposed by [28], (ii) Iterative Magnitude Pruning(IMP) donated by [7], (iii) Smart-Ratio (SR) which is the random pruning method proposed by [29]. Table 10 shows the pruning performance of the above algorithms, our pruning algorithm is better performing than other algorithms.

Remark. To demonstrate the effectiveness of l 1 regularization, the dense performance is obtained by training without any regularization. we obtained the dense performance by training without any regularization. According to our theoretical findings, LOMP is the optimal pruning strategy in the pruning-after-training regime, as the l 1 is the closest convex relaxation to the l 0 regularization, and the magnitude pruning can achieve the lowset pruning ratio. Consequently, we compared LOMP with several pruning algorithms that are not part of the pruning-after-training regime. Notably, LOMP outperforms these other pruning algorithms.

Discussion. The l 1 regularization metric is not new, and in practice, it becomes increasingly challenging to empirically select the hyper-parameter for l 1 regularization and train models with l 1 regularization as the model size grows. Nevertheless, some reparametrization techniques to address the l 1 regularization issue, as discussed in [44], may facilitate the use of l 1 regularization for pruning, making l 1 regularization one-shot pruning a very promising approach.

G.3 Comparison of Pruning as Optimization

In this section, we compared our l 1 regularization-based one-shot magnitude pruning with those works treating pruning as optimization [4; 40; 41]. Our results are obtained by searching the hyper-

parameters of l 1 regularization, all the data and model settings are followed from [4]. The comparison results can be found in Table 11. As is shown in Table 11, in extremely high levels of pruning schemes, our proposed method performs better than other methods.

G.4 Small Weights Benefits Pruning

We verify that small sharpness is not equal to high sparsity through hypothetical experiments. Considering that the hessian matrix of network A and network B 1 , B 2 , B 3 , B 4 share eigenvalues { λ 1 , λ 2 , . . . , λ n } , the weight magnitude of network B 1 , B 2 , B 3 , B 4 is 2,3,4,5 times that of network A , we take the eigenvalues and weights from a FC network trained without regularization. In this way, the gap between the curves will be more obvious. For other networks, the trend of the curve gap is consistent, the prediction of the network pruning ratio is shown in the Figure. 7. It is observed from Figure. 7 that as the magnitude of network weights increases, the capacity of the network to tolerate pruning decreases. The pruning ratio threshold is affected not only by loss sharpness but also the magnitude of weights. This finding, on the other hand, provides further evidence of the effectiveness of the l 1 -norm in pruning tasks.

G.5 Statistical Information of Weights in Various Experiments

The same plots as Fig. 2(b) and Fig. 2(c) are provided in Figure 8

0

20

40

60

80

100

120

R

0.0

0.2

0.4

0.6

0.8

1.0

Pruning Ratio

Original

Prediction

×2

×3

×4

×5

12.5

10.0

7.5

5.0

2.5

0.0

0

10000

20000

30000

40000

50000

60000

Frequency

FC5/CIFAR10

0

20

40

60

80

100

12.5

10.0

7.5

5.0

2.5

0.0

2.5

R^2 (log10)

FC5/CIFAR10

15.0

12.5

10.0

7.5

5.0

2.5

0.0

0

25000

50000

75000

100000

125000

150000

Frequency

FC12/CIFAR10

0

20

40

60

80

100

12.5

10.0

7.5

5.0

2.5

0.0

2.5

R^2 (log10)

FC12/CIFAR10

20

15

10

5

0

0.0

0.2

0.4

0.6

0.8

1.0

Frequency

1e6

Alexnet/CIFAR10

0

20

40

60

80

100

20

15

10

5

0

R^2 (log10)

Alexnet/CIFAR10

15

10

5

0

0

50000

100000

150000

200000

250000

300000

Frequency

VGG16/CIFAR10

0

20

40

60

80

100

20

15

10

5

0

R^2 (log10)

VGG16/CIFAR10

15

10

5

0

0

50000

100000

150000

200000

Frequency

ResNet18/CIFAR100

0

20

40

60

80

100

15

10

5

0

R^2 (log10)

ResNet18/CIFAR100

15

10

5

0

0

100000

200000

300000

400000

500000

Frequency

ResNet50/CIFAR100

0

20

40

60

80

100

15

10

5

0

R^2 (log10)

ResNet50/CIFAR100

15

10

5

0

Param Value (log10)

0

50000

100000

150000

200000

Frequency

ResNet18/TinyImageNet

0

20

40

60

80

100

Param Index (%)

15

10

5

0

R^2 (log10)

ResNet18/TinyImageNet

15

10

5

0

Param Value (log10)

0

100000

200000

300000

400000

500000

Frequency

ResNet50/TinyImageNet

0

20

40

60

80

100

Param Index (%)

15

10

5

0

R^2 (log10)

ResNet50/TinyImageNet

H Limitations

In this paper, we consider a well-trained neural network and argue that the weight magnitude and network sharpness with respect to the weights constitute the fundamental limits of a one-shot network pruning task. Although popular methods such as Iterative Magnitude Pruning [7] and the Lottery Ticket Hypothesis [6] involve multiple rounds of one-shot pruning, the fundamental limits of such multi-shot pruning remain unclear. Furthermore, we demonstrate that when the magnitude of the removed weights is small, the upper bound of the pruning ratio nearly coincides with the lower bound. However, not all training strategies result in small removed weights, and in such cases, the upper and lower bounds do not coincide. Nevertheless, weight magnitude and network sharpness with respect to the weights still represent the fundamental limits of a one-shot network pruning task.

I Broader Impacts

Our work aims to advance the theoretical understanding of network pruning, with the anticipation that theoretical insights can guide future designs of network pruning methods. There are no ethically related issues or negative societal consequences in our work.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See corollary 3.4, Lemma 3.5, Theorem 3.6, Section 4.1 and Algorithm 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations of our assumptions and our theoretical results, a limitation section has been included in the appendix, see Section H.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: In the main paper, we introduced all the assumptions and provided intuitive explanations. Comprehensive proofs are included in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Section 4, 5 in the main paper and Section B in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have provided open access to our code, the anonymized URL is included in the abstract.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the details and settings are included in the appendix. See Section B in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provided statistical results of our results under multiple independent runs. See Table 9 and Table 2.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided information of the computer resources in Section B and our code (see the URL in abstract).

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We confirm that our research adheres to the NeurIPS Code of Ethics in every respect and preserves anonymity.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed and we have included an impact statement in the appendix, see Section I.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our research didn't release any data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the assets we used in our paper, See References.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our work introduces no new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors build on the work of Larsen et. al. [1]  to estimate the fundamental limit of pruning i.e the smallest achievable density of a network by pruning. This is done by estimating the statistical dimension of the neural network and leveraging convex geometry.\nSimilar to Larsen et. al. [1] the authors provide a lower bound on the smallest density i.e. the sparsest parameter set that still generalizes well. They also provide an upper bound of this density, showing that such a dense network exists.\nThe authors provide computational methods to estimate these bounds reasonably for a neural network and empirically validate them. \nOverall, the work provides a theoretical characterization of the pruning limit of a network.\n\nSTRENGTHS:\n1. The proposed bounds validate that flatter networks allow more parameters to be pruned as well as the use of magnitude based parameter pruning which has consistently outperformed other pruning criteria.\n2. Algorithms to compute these bounds are also provided and empirically validated.\n3. Experiments across image classifications tasks are provided which align with the computed limits on pruning.\n\nWEAKNESSES:\n1. The analysis of Figure 4 is unclear to me. I cannot follow the interpretation of the provided plots to infer that iterative pruning can be beneficial. The provided plots suggest that removing a large fraction of parameters i.e. having a small pruning ratio, increases the value of R, which is not desirable according to the theory, for a dense network.\nHowever, does this also hold for a partially sparse network which is iteratively pruned like in the case of IMP, how does pruning iteratively mitigate this trend?\n2. The authors have also not confirmed this insight via experiments with L1 regularization if iterative pruning helps over one-shot pruning. I believe the authors need to give a detailed explanation of Section 7.\n3. In addition, can the authors comment on the connection to the pruning fractions shown in Figure 4 to the observations made by Paul et. al. [2] for IMP (Figure 5 in Paul et. al. makes a similar analysis). Is there an optimal pruning fraction for an iterative method like IMP? \n\nThe use of the terms pruning ratio, pruning threshold and sparsity are interchangeably used without concretely defining them, which makes the paper difficult to follow. I would urge the authors to correct this and clarify that the pruning ratio implies the number of nonzero parameters left in the network after pruning (it seems so from the derivation?).\nThe use of sparsity is incorrect in Table 9, sparsity denotes the number of zero parameters in the network.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nThis paper provides a theoretical framework to estimate the pruning limit of a neural network, building on recent work in this direction by Paul et. al. [2] and Larsen et. al. [1] via a convex geometry viewpoint, which can potentially be a useful contribution to the sparsity community.\nI am happy to increase my score further if my concerns are sufficiently addressed.\n\n\n[1] Larsen, Brett W., et al. \"How many degrees of freedom do we need to train deep networks: a loss landscape perspective.\" International Conference on Learning Representations. 2021.\n\n[2] Paul, Mansheej, et al. \"Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?.\" The Eleventh International Conference on Learning Representations. 2022.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper leverages the framework of statistical dimension in convex geometry to characterize the sharp phase transition point, i.e., the fundamental limit of the pruning ratio. Two key factors are found to be important for pruning, weight magnitude and network flatness. The flatter the loss landscape or the smaller the weight magnitude, the smaller pruning ratio. The theoretical analysis also aligns well with the empirical results, demonstrating the effectiveness of the theory proof.\n\nSTRENGTHS:\n1. The powerful approximate kinematics formula in convex geometry is leveraged to provide a very tight characterization of the fundamental limit of the network pruning. \n\n2. It is quite impressive to see the theoretical results perfectly coincide with the experiments. \n\n3. Moreover, the theoretical study can be explored to two common tricks in pruning, gradually pruning and $l_2$ regularization in Rare Gems.\n\n4. Code is provided.\n\nWEAKNESSES:\n1. One thing that sounds strange to me is that one conclusion is that \"The smaller the network flatness (defined as the trace of the Hessian matrix), the more we can prune the network\", which is contrary to the previous common belief, i.e., the flatter the network's landscape, the easier to be pruned. https://arxiv.org/pdf/2205.12694. Could the authors clarify these two counterarguments?\n\n2. While the theoretical proof looks sound, the empirical results in this paper look strange to me.  For instance, it is weird to see that the accuracy maintains the same at a 100% pruning ratio, but drops significantly as it goes below 2%. Does the pruning ratio here stand for a different thing than the common definition, i.e., the ratio of weights that are zero?\n\n3. Can the authors explain why RN50 achieves worse accuracy than RN18 on TinyImageNet? \n\n4. Which pruning algorithm is used for Figure 3? It is surprising to see that we can preserve the original performance at 2% ratio, for RN50 on TinyImageNet.\n\nQUESTIONS:\nPlease see the above weaknesses.\n\nLIMITATIONS:\nProvided.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the theoretical limits of how sparse a network can be pruned without sacrificing performance. The authors formulate pruning as a convex geometry problem. By imposing sparsity constraints on the loss function, they show that the pruning ratio can be bounded by the width of the loss landscape and the magnitude of the weights. They also propose an spectrum estimation algorithm for dealing with large Hessian matrices. The empirical results align well with the theoretical findings.\n\nSTRENGTHS:\n* Understanding the fundamental limits of network pruning is an important problem, and the insights provided by this paper (weight magnitude and network flatness as key factors) could potentially influence future research in the area.\n* The authors have provided detailed proofs and derivations for the theoretical results.\n* The experiments conducted are extensive enough to validate the theoretical findings, and there is a strong alignment between the theoretical and empirical results.\n\nWEAKNESSES:\n* Limited technical novelty: applying convex geometry to attain similar results (e.g. bounds dependent on the Gaussian width) has been explored in prior works, some of which are cited in the related work section. However, the paper lacks discussion of how this work differs from or builds upon existing literature.\n* More intuitions on the theoretical results would be helpful. For example, loss flatness is one of the key factors discussed in the paper, and only its formal definition (trace of the Hessian) is given in the paper. Intuitions like smaller flatness (i.e. flatter loss landscape) means the loss function is less sensitive to perturbations in the weights would be helpful.\n* Inconsistencies in notations: For example, from equation (17) onwards, the loss sublevel set notation takes in the weights as the argument. Is this the same loss sublevel set defined in equation (2), which takes in epsilon as the argument? And how is the epsilon parameter chosen for the loss sublevel set?\n* I would define \"pruning ratio\" earlier in the paper. It was not clear until the contributions section that smaller pruning ratios means more pruning of the network.\n* The authors have introduced \"pruned ratio\", which equals to 1 - pruning ratio, for Table 2. This is the only place in the paper where this term is used. This is confusing. Why did the authors not stick to \"pruning ratio\" throughout the paper?\n* Please consider removing adjectives like \"powerful\" when describing convex geometry and approximate kinematics formula, or \"very tight characterization.\" They are subjective and do not add much to the clarity of the paper.\n* Some typos like \"Lottery Tickets Hypolothsis\" (line 77)\n* Some terms like SLQ (line 82) are not defined until later in the paper.\n\nQUESTIONS:\n* What is novel here in light of prior work showing results connecting Gaussian widths to recovery thresholds [4]?\n* How sensitive are the theoretical bounds to the choice of loss function? Do the results generalize across different loss functions commonly used in deep learning?\n* How computationally expensive is the proposed spectrum estimation algorithm compared to existing methods? Is it practical for very large networks?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tries to answer the question of how sparse can a deep neural network be pruned without increasing the loss function. The authors employ high-dimensional geometry tools such as statistical dimension, Gaussian width, and the Approximate Kinematic Formula to derive the lower bound and upper bound of the ratios of weights that can be pruned. The authors also provide an improved algorithm for estimating the eigenvalues of large Hessian matrices, which is used to estimate the Gaussian width.\n\nSTRENGTHS:\n1. This paper provides an interesting perspective for analyzing the limits of network pruning. \n\n2. The tools from convex geometry are novelly applied to study the problem. The derivations of the lower bound and upper bound are solid.\n\n3. The results provide insights into the problem of pruning neural networks and explain certain behaviors from previous work.\n \n4. The experiments show a certain matching between the theoretical bounds and the actual performance.\n\nWEAKNESSES:\n1. The derived upper bound and lower bound on the pruning ratio depend on the new weights of the pruned networks. This is strange to me, meaning that the limit of the pruning ratio varies when a different weight is used. This also limits the application of the results. \n\n2. The numerical experiments are performed using the L1 regularized loss function. However, this is not considered in the theoretical results. I wonder if the bounds will be different if the loss function is changed.\n\n3. The one-shot magnitude pruning ignores the interaction between the weights, which has been shown to be suboptimal in some recent work. However, the authors claim that it is optimal for the lower and upper bounds. I believe this is because the L1 regularization is used as the loss function, which already takes that into account. This should be clarified.\n\n4. The authors claim that the upper and lower bounds match each other when the proposed L1 regularized loss function is used for training. This might not hold when the original loss is without L1 regularization. This should also be discussed and not overclaimed in general cases.\n\n5. The numerical comparison did not consider many state-of-the-art pruning methods (which are also pruning after training) such as \n[1] R. Benbaki et. al., Fast as CHITA: Neural Network Pruning with Combinatorial Optimization\n[2] L. You et. al., SWAP: Sparse Entropic Wasserstein Regression for Robust Network Pruning\n[3] X. Yu et. al., The combinatorial brain surgeon: Pruning weights that cancel one another in neural networks\n\n6. Relating to the previous comment, the magnitude pruning method used here minimizes the bounds proposed here, which then is used as a support for their usefulness. However, in many cases, the network is not pruned to its limit. \n\n7. The title is overclaimed. While the bounds proposed here are sometimes useful, they are not fundamental limits since there are approximations and relaxation used along the derivations. Moreover, the upper and lower bounds do not match each other in general.\n\nQUESTIONS:\nPlease see the weakness part.\n\nLIMITATIONS:\nYes, the authors did address the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}