Title: On the Sparsity of the Strong Lottery Ticket Hypothesis
Venue: NeurIPS 2024
Accepted: yes
Keywords: strong lottery ticket hypothesis

random subset sum

neural network pruning

random neural network
Abstract: Considerable research efforts have recently been made to show that a random neural network $N$ contains subnetworks capable of accurately approximating any given neural network that is sufficiently smaller than $N$, without any training. 
This line of research, known as the Strong Lottery Ticket Hypothesis (SLTH), was originally motivated by the weaker Lottery Ticket Hypothesis, which states that a sufficiently large random neural network $N$ contains sparse subnetworks that can be trained efficiently to achieve performance comparable to that of training the entire network $N$.
Despite its original motivation, results on the SLTH have so far not provided any guarantee on the size of subnetworks.
Such limitation is due to the nature of the main technical tool leveraged by these results, the Random Subset Sum (RSS) Problem.
Informally, the RSS Problem asks how large a random i.i.d. sample $\Omega$ should be so that we are able to approximate any number in $[-1,1]$, up to an error of $ \epsilon$, as the sum of a suitable subset of $\Omega$. 

We provide the first proof of the SLTH in classical settings, such as dense and equivariant networks, with guarantees on the sparsity of the subnetworks. Central to our results, is the proof of an essentially tight bound on the Random Fixed-Size Subset Sum Problem (RFSS), a variant of the RSS Problem in which we only ask for subsets of a given size, which is of independent interest.

On the Sparsity of the Strong Lottery Ticket Hypothesis

Abstract

Considerable research efforts have recently been made to show that a random neural network N contains subnetworks capable of accurately approximating any given neural network that is sufficiently smaller than N , without any training. This line of research, known as the Strong Lottery Ticket Hypothesis (SLTH), was originally motivated by the weaker Lottery Ticket Hypothesis, which states that a sufficiently large random neural network N contains sparse subnetworks that can be trained efficiently to achieve performance comparable to that of training the entire network N . Despite its original motivation, results on the SLTH have so far not provided any guarantee on the size of subnetworks. Such limitation is due to the nature of the main technical tool leveraged by these results, the Random Subset Sum (RSS) Problem. Informally, the RSS Problem asks how large a random i.i.d. sample Ω should be so that we are able to approximate any number in [ -1 , 1] , up to an error of ε , as the sum of a suitable subset of Ω .

We provide the first proof of the SLTH in classical settings, such as dense and equivariant networks, with guarantees on the sparsity of the subnetworks. Central to our results, is the proof of an essentially tight bound on the Random Fixed-Size Subset Sum Problem (RFSS), a variant of the RSS Problem in which we only ask for subsets of a given size, which is of independent interest.

1 Introduction

The Lottery Ticket Hypothesis (LTH) is a research direction that has attracted considerable attention over the years, stemming from the empirical contrast between the fact that, while large neural networks can be successfully trained to achieve good performance on a given task and successively pruned to a great level of sparsity without compromising their performance, researchers have struggled to train sparse neural networks from scratch. The authors of [12] observed that, using a simple pruning strategy (namely Iterative Magnitude Pruning while rewinding the original weights of the remaining edges to their value at initialization), starting from a sufficiently large random neural networks, it is possible to identify sparse subnetworks that can be trained to achieve the performance achievable by the starting network (see Figure 2 in the appendix for an illustration). The previous statement, namely the LTH, soon gave rise to an even stronger one, corroborated by empirical works [29, 26] which proposed 'training-by-pruning' algorithms (see Section 2 for details), providing evidence that starting from a sufficiently large random neural networks, it is possible to identify sparse subnetworks that exhibit good performance as they are, without changing the original weights (see Figure 3 in the appendix for an illustration). By removing the need to analyze the dynamics of

training, the last statement, namely the Strong Lottery Ticket Hypothesis (SLTH), allowed a fruitful series of rigorous proofs for increasingly more general architectures (see Section 2 for an overview). Such rigorous results can informally be stated as follows:

Theorem 1 (Informal statement of previous SLTH results) . With high probability, a random artificial neural network N Ω with m parameters can be pruned so that the resulting subnetwork N S ε -approximates (i.e., approximates up to an error ε ) any target artificial neural network N t with O ( m/ log 2 (1 /ε )) parameters.

It is important to note that, to this day, we only have proofs on the existence of such subnetworks, also called winning tickets, but it remains an open question how to find them reliably.

All theoretical results on the SLTH however have so far not investigated the interplay between the sparsity of the winning ticket N S and the size of the random neural network N Ω . This is in contrast to the original motivation of the LTH and to the practical application of the aforementioned training-by-pruning algorithms that motivated the SLTH, such as [15, 14]. In fact, to approximate target networks with O ( m/ log 2 (1 /ε )) parameters, essentially all winning tickets N S have Θ( m ) parameters (see Appendix A), thus being roughly of the same size of the original network N Ω . We thus ask the following natural question:

If we want to ε -approximate a family of target artificial neural networks with m t parameters by pruning a fraction α , called sparsity, of the m parameters of a random artificial neural network N Ω , how big should m be?

We are particularly interested in the regime in which the density parameter γ = 1 -α vanishes as the size of the network increases, so that the size of the winning ticket N S is γm = o ( m ) .

The above question has so far remained unanswered as a consequence of the limitation inherited from the core technical tool that has been leveraged so far to prove SLTH results, namely the Random Subset Sum (RSS) Problem [17]. Informally, the RSS asks how large a random i.i.d. sample Ω should be so that we are able to approximate any number in [ -1 , 1] as the sum of a suitable subset of Ω . The applicability of RSS to the SLTH was first recognized by [24] within the proof strategy previously developed in [20].

1.1 Our Contribution

We answer the aforementioned question by introducing and proving a refined variant of the RSS Problem, namely the Random Fixed-Size Subset Sum Problem (RFSS), in which the approximation of the target values should be achieved by only considering subsets of fixed size k from a set of n samples (Theorem 2). We focus on subsets of fixed size k rather than subsets of size up to k for two main reasons. From a theoretical point of view, it is a stronger requirement, and practically speaking, using fixed-size subsets enables us to achieve SLTH results where the layers of the lottery ticket exhibit a uniform structure, potentially offering a computational advantage in their implementation.

In Section 4, we show how the density γ impacts the overparameterization , i.e., the ratio ( m / m t ) between the number of parameters of the original network N Ω and that of the class of target networks N t that can be ε -approximated by pruning N Ω down to a subnetwork N S with γm parameters. In our analysis, we also compare and recover as special cases previous SLTH results such as [24, 20, 8, 3, 9]. For instance, when γm = Θ( m ) , we recover up to a logarithmic factor the result of [24], which states that the overparameterization needed is O (log 2 ( m 2 t / ε 2 ) ) . In the case of Dense Neural Networks, Theorem 3 thus bridges the gap between the two extreme cases of γm = Θ( m t ) and γm = Θ( m ) considered in [20] and [24], respectively. It is worth noting that [24] is often considered an improvement over [20], as it exponentially reduces the overparameterization, albeit at the cost of a trivial sparsity level. Finally, we prove that our bounds on the overparameterization as a function of the subnetwork sparsity are essentially tight.

Organization of the paper. After reviewing the literature on the SLTH in Section 2, we introduce the Random Fixed-Size Subset Sum Problem in Section 3. In Section 4, we explore some applications of the RFSS Problem to the SLTH, and finally draw our conclusions in Section 5. Some limitations of our work, along with its potential impact, are discussed in Section 6.

2 Related Work

The SLTH is named after the LTH, which was introduced by Frankle and Carbin in [12]. At the time of writing, this paper has received over 3,300 citations, attesting to the significance and impact of the research topic. Surveying the LTH is thus besides the scope of this work, and we defer the reader to dedicated surveys such as [16].

The SLTH was empirically motivated by work investigating training-by-pruning algorithms such as [29, 26], namely algorithms that leverage the gradient of the network parameters to learn a good mask of the edges to be retained (i.e., a good subnetwork, called the winning ticket). [29] achieves this by learning a probability associated to each edge, which is then used to sample the edges that should be included in the subnetwork. [26] gets rid of the stochasticity involved in the aforementioned strategies by learning a score associated to each edge; the subnetwork is then determined by including the edges with the highest score. Such strategies are leveraged in [15, 14] in a federated learning setting, in order to improve the communication cost of distributed training by communicating the sampled masks of a fixed shared network, rather than the entire weights. However, these training-by-pruning algorithms are generally not computationally less expensive than classical training, since they also make use of backpropagation to update scores and are applied to a sufficiently large network to find a winning ticket. To reduce the computational cost of finding a good subnetwork, [13] shows, both theoretically and experimentally, that randomly pre-pruning the source network before looking for a winning ticket can be an effective approach. In [23], on top of randomly pruning the source network, some parameters are also frozen. Frozen parameters are forced to be part of the winning ticket and they do not have an associated score, which effectively reduces the search space for the training-by-pruning algorithms.

The first rigorous proof of the SLTH in the case of dense neural networks has been provided by [20], which establishes a framework that was inherited by the subsequent works. [24] crucially shows that the framework in [20] allows the application of the RSS analysis in [17], proving that, with no constraint on the size of the subnetworks, a random network with m -parameters can be pruned to approximate target networks with m/ log(1 /ε ) parameters (we defer the reader to Theorem 3 for details on further constraints on the parameters). An alternative proof of the result in [24] was simultaneously shown in [22]. [8] and [3] successively extended [24] and [22] to convolutional neural networks (CNNs). By leveraging multidimensional generalizations of RSS [7, 2], [6] further extended the SLTH to structured pruning of CNNs and, as a special case, dense networks. Finally, [9] provided a general framework that proves the SLTH for equivariant networks.

As for refinements and generalizations of the above results, [4] shows that, at the cost of a quadratic overhead in the overparameterization w.r.t. [24], the number of layers of the random network N Ω can be reduced to ℓ + 1 , where ℓ is the number of layers of the target networks N z ; furthermore, while previous results only considered networks with ReLU activation, [4] shows how to extend the proof in [24] to a more general class of activations functions. [5] introduces the notion of universal lottery ticket, and show that it is possible to prune a sufficiently overparameterized random network so that the resulting subnetwork (the lottery ticket) can approximate certain class of functions up to an affine transformation of the output of the subnetwork (in this sense being universal). [11] shows how to extend the proof in [24] when neurons have random biases, and adapts the trainingby-pruning algorithm of [26] to find a strong lottery ticket with a desired sparsity level. Motivated by theoretical insights on the existence of sparse strong lottery tickets, [10] develops a framework to plant the latter in large random network and investigates training-by-pruning algorithms, providing evidence that sparse strong lottery tickets typically exists for common machine learning tasks, and the difficulty to find them is of algorithmic nature.

Our proof of the RFSS Problem in Section 3 is based on the second moment method approach first explored by [18], and which has recently been refined to prove multidimensional generalizations of RSS by [7] and [2].

3 Fixed-Size Random Subset Sum

In this section we present our technical contributions on the RFSS, which are the foundation of our proofs regarding the sparsity of the SLTH.

Let us start by introducing some notation. We denote by [ n ] the set { 1 , . . . , n } , for n ∈ N . Given a set Ω = { X 1 , ..., X n } and a set of indices S ⊆ [ n ] we define Σ Ω S = ∑ i ∈ S X i , and we omit Ω when clear from the context. We now define a class of distributions for which our RFSS result holds.

Definition 1 (sum-bounded) . We say that a probability density function f is sum-bounded if there exist positive constants c l and c u such that, for all k ∈ N , given k independent samples X 1 , ..., X k with density f , the density of their sum f Σ [ k ] satisfies

with the lower bound holding for all x ∈ [ -√ k, √ k ] and the upper bound holding for all x ∈ R .

At first, our definition of sum-bounded could look as a weaker version of a classical local limit theorem on the sum of random variables (e.g., see [25, Chapter VII, Theorem 7]). However, that is not the case, since we require a lower bound on the sum for any k , which is needed to prove our main result.

Denote, for all x ∈ [0 , 1] , the binary entropy as

Our main technical result is the following proof of a fixed-size subset variant of the RSS Problem.

Theorem 2. Let 0 < ε < 1 , c hyp ≥ 1 , k, n be integers with 1 ≤ k ≤ n 2 , and let Ω = { X 1 , ..., X n } where the X i 's are i.i.d. random variables with sum-bounded density. There exists a constant c thm such that, if

then for every fixed z ∈ [ -√ k, √ k ] it holds that

Remark. The proof of Theorem 2 is given in Section 3.1, and it actually holds for any 1 ≤ k ≤ λn , for an arbitrary λ ∈ [ 1 / n , 1) . We state the theorem this way for readability and because we are primarily interested in high-sparsity settings (i.e., small size k of the subsets), so considering values of k ≥ n 2 does not add much to our analysis. The same remark also holds for Corollary 1.

The sum-bounded condition of Definition 1 is easily verified for distributions such as the Gaussian distribution. Previous SLTH results rely on a classical resampling argument by [17, Corollary 3.3], which shows how RSS results for Uniform [ -1 , 1] independent random variables naturally extend to independent random variables that contains a uniform distribution, in the sense that they can be expressed as the mixture of distributions one of which is Uniform [ -1 , 1] with constant probability. 1 The next lemma thus proves that the Uniform [ -1 , 1] distribution is sum-bounded 2 . Adetailed proof is provided in Appendix C.

Lemma 1. The Uniform [ -1 , 1] probability density function is sum-bounded, i.e., given a set U n = { U i } i ∈ [ n ] of i.i.d. variables U i with Uniform [ -1 , 1] probability density function, there exist constants c l and c u such that the probability density function f ( x, n ) of the sum Σ U n [ n ] of these variables, for all n ∈ N ,

with the lower bound holding for all x ∈ [ - √ n, √ n ] , and the upper bound holding for all x ∈ R .

Finally, in our proofs on the Sparse SLTH in Section 4, we make use of the following corollary of Theorem 2, which ensures a uniform high probability of hitting any target z ∈ [ -√ k, √ k ] , considering independent random variables that contain a uniform distribution.

Corollary 1. Let 0 < p ≤ 1 and ε ∈ (0 , 1 / 2 ) be constants, k, n with 1 ≤ k ≤ n 2 , and let Ω = { X 1 , ..., X n } be i.i.d. random variables whose density is a mixture of a Uniform ([ -1 , 1]) with probability p , and some other density otherwise. There exists a positive constant c amp that only depends on p such that, if

then

Proof Idea. The corollary follows from three arguments. First, by a standard sampling argument, we can assume that a constant fraction of the sample follows a Uniform [ -1 , 1] distribution. Secondly, by Lemma 1, the uniform probability density function is sum-bounded. We can thus apply Theorem 2, which guarantees a success probability of c thm for approximating a given target. Finally, by a standard probability amplification argument and a union bound applied to Theorem 2, by paying an extra factor log 2 ( k/ε ) in Eq. 1, the constant c thm can be assumed to be 1 -ε , and the existence of a suitable subset S z holds simultaneously for all z ∈ [ -√ k, √ k ] . Details are given in Appendix D.

For k big enough, we can get rid of the squared logarithmic dependency on k in the right hand side of Equation 3, as shown in the following Corollary, whose proof can be found in Appendix E.

Corollary 2. Let 0 < p ≤ 1 and ε ∈ (0 , 1 / 2 ) be constants, k, n be integers with 1 ≤ k ≤ n 2 and k ≥ 2 c amp ( log 2 2 k +2 log 2 k · log 2 1 ε ) . Let Ω = { X 1 , ..., X n } be i.i.d. random variables whose density is a mixture of a Uniform ([ -1 , 1]) with probability p , and some other density otherwise. There exists a positive constant c amp that only depends on p such that, if

then

As customary in conference versions of papers, our proofs adopt the convention of taking ceilings and floors as suitable for non integer fractional terms. This is done in the interest of the reader (and ours), and does not impact the results in any significant way.

3.1 Proof of Theorem 2

Proof of Theorem 2. For simplicity, throughout the proof we will often use c to denote any positive constant. Let S k = { S ⊂ [ n ] | | S | = k } and define, for a fixed z ∈ [ -√ k, √ k ] ,

where Z S = Z S ( z ) = 1 {| Σ S -z | <ε } . Following [18], we exploit the second moment method for RFSS, generalising it to arbitrary k .

We first rewrite Eq. 5 in a more convenient form. Let ˜ S and ˜ S ′ be two independently and uniformly at random chosen subsets of [ n ] of size k , and denote H S ( z ) as the event that Σ S ε -approximates z , namely

it thus suffices to prove that

We have and

Using Eqs. 7 and 8 we can rewrite the r.h.s. of Eq. 5 as follows

Eq. 6 thus becomes

Let I i denote the event {| ˜ S ∩ ˜ S ′ | = i } and I a,b the event ⋃ a ≤ i ≤ b I i . Fix µ ∈ ( λ, 1) . By the law of total probability and independence of I i and H ˜ S , we rewrite the l.h.s. of Eq. 9 as follows:

To conclude the proof, it suffices to show that each addendum in Eqs. 10, 11 and 12 are upperbounded by some constant multiple of ε / √ k , since the lower bound in Definition 1 ensures that

As for the first addendum (Eq. 10), since Pr( H ˜ S ′ | H ˜ S , I k ) = 1 , then

where inequality ( a ) in Eq. 14 is a standard lower bound on ( n k ) holding for all k ≤ n -1 ; in inequality ( b ) in Eq. 14 we used Eq. 1, namely nH 2 ( k n ) ≥ c hyp log 2 k ε ; in inequality ( c ) in Eq. 14 we used that c hyp ≥ 1 .

As for the second addendum (Eq. 11), we next show that

by proving that and

First, observe that I = | ˜ S ∩ ˜ S ′ | follows a Hypergeometric ( n, k, k ) distribution, thus by Chebyshev's inequality

having set c ′ = µ 2 (1 -λ / µ ) 2 > 0 , thus proving Eq. 16. The proof of Eq. 17 is given in Appendix F, concluding the proof of Eq. 15.

As for the third addendum (Eq. 12), in Appendix F we show that

The three bounds on the addenda in Eqs. 10, 11, and 12 (Eqs. 14, 15, and 18, respectively), combined with Eq. 13, conclude the proof.

4 Sparse Strong Lottery Ticket Hypothesis (SSLTH)

We now apply our results on the RFSS problem to the SLTH and obtain guarantees on the sparsity of winning tickets for Dense Neural Networks (DNNs, Theorem 3) and Equivariant NNs (Theorem 4).

The next theorem essentially interpolates between the two extremes of [20][Theorem 2.1] (where γm = Θ( m t ) ) and [24][Theorem 1] (where γm = Θ( m ) ), where we recall that m and m t represent the number of parameters of the overparameterized and the target networks, respectively, and γ is the density of the winning ticket.

We use σ ( · ) to denote the ReLU activation function, i.e., σ ( x ) = x · 1 x ≥ 0 , and ∥ W ∥ to denote the spectral norm of the matrix W . Let F to be a set of target ReLU neural networks f : R d 0 → R d l of depth l such that

For a given f ∈ F , for all i ∈ [ ℓ ] , let ρ i = max { d i -1 / d i , d i / d i -1 } , and ρ = max i ρ i . Then, recalling that c amp is the constant defined in Corollary 1, we have the following result.

Theorem 3 (SSLTH for DNNs) . Let g be a randomly initialized feed-forward 2 ℓ -layer neural network, in which each weight is drawn from a Uniform [ -1 , 1] distribution, of the following form:

Let γ ′ = γ ′ ( ε ) ∈ (0 , 1) , M 2 i ∈ R d i × 2 d i -1 n ∗ i and M 2 i -1 ∈ R 2 d i -1 n ∗ i × d i -1 , with n ∗ i satisfying

With probability at least 1 -ε , for every f ∈ F , where F is defined as in Eq. 19, g can be pruned to obtain a subnetwork of sparsity at least α = 1 -γ that approximates f up to an error ε , having defined γ = ργ ′ .

Proof Idea. The theorem follows from a slight variation of the same approach detailed in [24], in which we use our Corollary 1 instead of [17][Corollary 2.5] when pruning g , allowing us to have control over the size of the pruned network. A detailed proof is provided in Appendix G.

To illustrate a simple example of how Theorem 3 addresses the main question asked in the introduction, consider the case where we want to approximate a target network with m t parameters and ℓ layers, each of width d (so ρ = 1 and γ ′ = γ ), by pruning an overparameterized network to achieve a desired sparsity level of α = 1 -γ . The condition expressed by Equation 20 in Theorem 3

Overparameterization

(

Malach

e(og2)

(322601)0

(2601)

Pensia

mt

E

0(1)

Density y of winning ticket

comes from the use of Corollary 1 when pruning network g , as shown in the proof. If, instead of Corollary 1, we use its simplified variant Corollary 2, it is easy to observe that Equation 20 would become

Using this condition, Theorem 3 then tells us that we need to prune a randomly initialized network with twice as many layers and a number of parameters of the order of d 2 log 2 ℓd 2 ε H ( γ ) .

We will now clarify the connection between Theorem 3 and the earlier results from [20] and [24]. Figure 1 provides a quick visual comparison.

Malach et al.[20]. When all layers have the same width d , [20] showed that any target network with l layers and a total of m t = d 2 l parameters can be ε -approximated by pruning a randomly initialized network with 2 l layers. The overparameterization of this network, relative to the target network, is O ( m 2 t ε 2 log 2 m t ε ) = ˜ O ( m 2 t ε 2 ) . More specifically, the winning ticket found after pruning has a parameter count of the same order as the target network, resulting in a density of γ = ˜ O ( ε 2 m 2 t ) . Notably, this density γ is the inverse of the overparameterization, as the size of the winning ticket matches that of the target network.

Next, we show that Theorem 3 also yields a density that is polynomial in ε m t , when using an overparametrization of Θ ( m 2 t ε 2 ) . Let z = ( m t ε ) , and note that γ ′ = γ in Theorem 3, since all layers have the same width. As n ∗ i in Theorem 3 represents the overparametrization with respect to the target network, let us set n ∗ i = cz 2 , for some constant c . Equation 20 then becomes

We show that the inequality cz 2 ≥ c amp log 2 2 ( cz 3 γ ) γ log 2 ( 1 / γ ) holds for some big enough constant c when setting γ = ε m t = 1 z , which implies that Equation 22 is also satisfied. We get cz ≥ c amp log 2 2 ( cz 2 ) log 2 ( z ) , which is satisfied for a big enough constant c (see Appendix I). Overall, when using an overparametrization Θ ( m 2 t ε 2 ) , we find a winning ticket with density ε m t , as shown in Figure 1.

Pensia et al.[24]. For simplicity, let us still consider target networks where all layers have the same width d , and we apply Theorem 3 using the simplified condition from Equation 21. When

γm = Θ( m ) , i.e. the density γ is a constant as in [24] (see Appendix A), the entropy term H 2 ( γ ′ ) in the right-side of Equation 21 also becomes a constant. In this setting, we indeed recover the result shown in [24][Theorem 1], up to a logarithmic factor, as shown in Figure 1.

Quite similarly to Theorem 3, the next result essentially generalizes [9] up to a factor log 2 1 ε . The theorem is stated with the understanding that for G -equivariant networks, in order to preserve G -equivariance, pruning is best done not with respect to the parameters expressing the network in the canonical basis (i.e. directly on the weights of the network), but with respect to the equivariant parameters , that is those coefficients expressing the linear layers of the network as a linear combination of the elements of the corresponding equivariant basis [9]. For simplicity, due to the technical set-up, we assume all feature spaces being F = ( R d , σ ) , with σ the linear representation of the group G , and the same number n of such feature spaces being stacked in each layer. A G -equivariant linear map from the i th feature space to the i +1 st can be decomposed in a corresponding equivariant basis denoted B i → i +1 = B . Since all feature spaces are the same, we omit the layers' indices. When stacking n feature spaces in the input and output of the i th layer, the full equivariant basis is denoted k n → n , and finally the basis of the G -equivariant maps from F n to F n can be written as the Kronecker product k n → n ⊗B . For any basis B = { b 1 , . . . , b p } , we denote its cardinality p = |B| and define ∥B∥ = max ∥ β ∥ ∞ ∥ ∑ p k =1 β k b k ∥ , with ∥ · ∥ in the r.h.s. being the operator norm inherited from the ℓ p norm.

Theorem 4 (SSLTH for Equivariant Networks) . Let h be a random 2 ℓ -layer G -equivariant network where all equivariant parameters are drawn from a Uniform [ -1 , 1] distribution, every odd layer expressed in the associated equivariant basis k ˜ n → n ⊗ B and every even layer expressed in the associated equivariant basis k n → ˜ n ⊗B . Let γ = γ ( ε ) ∈ (0 , 1) , with ˜ n satisfying

With probability at least 1 -ε , for every ℓ -layer G -equivariant neural network f , with all layers expressed in the associated equivariant basis k n → n ⊗B , h can be pruned to obtain a G -equivariant subnetwork of sparsity at least α = 1 -γ that approximates f up to an error ε .

The proof, which we omit, is analogous to that of Theorem 3, since [9][Theorem 1] exploits the exact same pruning strategy of [24], except for the fact that it is applied not to the original parameters of the equivariant network, but to the network expressed in terms of its equivariant basis (the sparsity α is here also intended with respect to the equivariant parameters count). This allows the construction to apply without losing the property of equivariance in the pruned approximating subnetwork obtained. The crucial step is when Corollary 1 is applied in [9][Lemma 1], instead of [17][Corollary 2.5]. This is done in parallel, multiple times, across non-overlapping coefficients of the equivariant basis. Thanks to the careful preprocessing devised by the authors, this preserves equivariance and at the same time ensures that each application of Corollary 1 is independent of the others.

To conclude the section, we mention that Theorem 4 applies in particular to vanilla CNNs, which are a special case of equivariant neural networks where the group is G = ( Z 2 , +) , recovering previous SLTH results on CNN [8, 3]. Furthermore, we remark that Theorem 3 can be revisited through the improvement upon the 2 ℓ -depth overparameterization devised in [4], i.e., it is possible to provide sparsity guarantees also for overparameterizations requiring depth ℓ +1 only. The analysis is more technical and we omit it, but the ideas are analogous to what shown in [4]. An analogous improvement is suggested as future work in [9].

4.1 Lower bound on the required overparameterization

We now adapt the lower bound of [24] in order to almost match the required overparameterization of our Theorem 3, considering the simple scenario in which we want to approximate the family F of all linear networks with weights forming a matrix having spectral norm less than √ k ; more formally

The formal claim states that, if a network with n parameters can approximate every h W ∈ F with probability at least 1 / 2 (after it is pruned down to k parameters), then the hypothesis of Theorem 2 in Eq. 1 must hold. 3

Theorem 5. Let n, k ∈ N , with 1 ≤ k ≤ λn , having set λ = 1 -1 / 2 π ≈ 0 . 84 . Consider a neural network g with n parameters, and let G k be the set of neural networks that can be formed by pruning g down to k parameters. Let F be as defined in Eq. 23. If it holds that, for some ε < 1 / 16 ,

then it holds that

The theorem follows by adapting the packing argument of [24]. A detailed proof is provided in Appendix H.

5 Conclusions

In this work, we have extended previous results on the Strong Lottery Ticket Hypothesis by quantifying the required overparameterization as a function of the sparsity of the subnetworks. Central to our results is a proof of the Random Fixed-size Subset Sum (RFSS) Problem, a refinement of the seminal Random Subset Sum (RSS) Problem in which the subsets have a required fixed size.

A challenging open problem is to extend our analysis of RFSS to the multidimensional case, in which the random samples and targets are vectors in R d . Previous extension of RSS to the Multidimensional RSS have indeed allowed to prove structured-pruning version of the SLTH [8]. A Multidimensional RFSS result would then allow to quantify, in the structured pruning case, the dependency of the overparameterization w.r.t. the sparsity of the (structured) subnetworks.

Another future direction is to refine our analysis of the RFSS in Theorem 2 in order to improve the probability of success to 1 -ε rather than constant, thus allowing to avoid shaving off the extra factor log 2 (1 /ε ) in our corollaries w.r.t. our lower bound, which is due to the amplification done in Corollary 1 to get to probability 1 -ε .

Finally, an important future direction is to improve training-by-pruning methods such as [29, 26, 11, 10, 23] or to develop new ones, in order to allow to efficiently find strong lottery tickets of a desired sparsity, thus empirically validating our theoretical predictions.

6 Limitations and Impact

Limitations Similar to all the research conducted on the LTH and the SLTH, this work only proves the existence of lottery tickets. To this date, it is not clear if these subnetworks can be found reliably (no formal proof exists) in an efficient manner - however, empirical evidence suggests that efficient algorithms exist (e.g., [29, 26]).

Impact The contribution of this work is primarily theoretical and not confined to a specific domain. Its potential societal impact would, therefore, be closely tied to the particular scenarios to which it is applied. It could be interesting to compare the environmental impact of finding lottery tickets inside overparameterized networks. We also believe that our work has the potential to have a strong environmental impact as sparse NNs have massively reduced inference costs.

Acknowledgments and Disclosure of Funding

This research is supported by the EPSRC grant EP/W005573/1, and by the France 2030 program, managed by the French National Research Agency under grant agreements No. ANR-23-PECL0003 and and ANR-22-PEFT-0002. It was also funded in part by the European Network of Excellence dAIEDGE under Grant Agreement Nr. 101120726, by SmartNet and LearnNet, and by the French government National Research Agency (ANR) through the UCA JEDI (ANR-15-IDEX-01), EUR DS4H (ANR-17-EURE-004), and the 3IA Cˆ ote d'Azur Investments in the Future project with the reference number ANR-19-P3IA-0002.

References

Giampietro Allasia. 'Approximation of the normal distribution functions by means of a spline function'. In: Statistica 41.2 (1981), pp. 325-332.

Sander Borst et al. 'On the Integrality Gap of Binary Integer Programs with Gaussian Data'. In: Mathematical Programming 197.2 (Feb. 2023), pp. 1221-1263. ISSN: 1436-4646. DOI: 10.1007/s10107-022-01828-1 . (Visited on 06/05/2023).

Rebekka Burkholz. 'Convolutional and Residual Networks Provably Contain Lottery Tickets'. In: Proceedings of the 39th International Conference on Machine Learning . Baltimore: PMLR, July 2022, pp. 2414-2433. (Visited on 02/27/2023).

Rebekka Burkholz. 'Most Activation Functions Can Win the Lottery Without Excessive Depth'. In: Thirty-Sixth Conference on Neural Information Processing Systems . Dec. 2022. (Visited on 02/27/2023).

Rebekka Burkholz et al. 'On the Existence of Universal Lottery Tickets'. In: International Conference on Learning Representations . virtual, Apr. 2022. (Visited on 02/27/2023).

Arthur da Cunha, Francesco D'Amore, and Emanuele Natale. 'Polynomially OverParameterized Convolutional Neural Networks Contain Structured Strong Winning Lottery Tickets'. In: Thirty-Seventh Conference on Neural Information Processing Systems . Nov. 2023. (Visited on 04/23/2024).

Arthur Carvalho Walraven da Cunha et al. 'Revisiting the Random Subset Sum Problem'. In: 31st Annual European Symposium on Algorithms, ESA 2023, September 4-6, 2023, Amsterdam, The Netherlands . Ed. by Inge Li Gørtz et al. Vol. 274. LIPIcs. Schloss Dagstuhl Leibniz-Zentrum f¨ ur Informatik, 2023, 37:1-37:11. DOI: 10.4230/LIPICS.ESA.2023.37 . URL: https://doi.org/10.4230/LIPIcs.ESA.2023.37 .

Arthur da Cunha, Emanuele Natale, and Laurent Viennot. 'Proving the Strong Lottery Ticket Hypothesis for Convolutional Neural Networks'. In: ICLR 2022 - 10th International Conference on Learning Representations . Virtual, France, Apr. 2022. (Visited on 08/04/2022).

Damien Ferbach et al. 'A General Framework For Proving The Equivariant Strong Lottery Ticket Hypothesis'. In: The Eleventh International Conference on Learning Representations . Sept. 2022. (Visited on 03/04/2024).

Jonas Fischer and Rebekka Burkholz. 'Plant 'n' Seek: Can You Find the Winning Ticket?' In: International Conference on Learning Representations . Apr. 2022. (Visited on 02/27/2023).

Jonas Fischer, Advait Gadhikar, and Rebekka Burkholz. Lottery Tickets with Nonzero Biases . June 2022. arXiv: 2110.11150 [cs] . (Visited on 05/13/2024).

Jonathan Frankle and Michael Carbin. 'The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks'. In: International Conference on Learning Representations . Sept. 2018. (Visited on 10/20/2023).

Advait Harshal Gadhikar, Sohom Mukherjee, and Rebekka Burkholz. 'Why Random Pruning Is All We Need to Start Sparse'. In: International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA . Ed. by Andreas Krause et al. Vol. 202. Proceedings of Machine Learning Research. PMLR, 2023, pp. 10542-10570. URL: https: //proceedings.mlr.press/v202/gadhikar23a.html .

Berivan Isik et al. 'Adaptive Compression in Federated Learning via Side Information'. In: Proceedings of The 27th International Conference on Artificial Intelligence and Statistics . PMLR, Apr. 2024, pp. 487-495. (Visited on 05/16/2024).

Berivan Isik et al. 'Sparse Random Networks for Communication-Efficient Federated Learning'. In: The Eleventh International Conference on Learning Representations . Sept. 2022. (Visited on 01/18/2024).

Bohan Liu et al. A Survey of Lottery Ticket Hypothesis . Mar. 2024. DOI: 10.48550/arXiv. 2403.04861 . arXiv: 2403.04861 [cs] . (Visited on 05/16/2024).

George S. Lueker. 'Exponantially small bounds on the expected optimum of the partition and subset sum problem'. In: Ramdom Structures and Algorithms 12 (1998), pp. 51-62.

George S. Lueker. 'On the average difference between the solutions to linear and integer knapsack problems'. In: Applied Probability - Computer Science, The Interface . Vol. 1. Birkh¨ auser, 1982.

Florence Jessie MacWilliams and Neil James Alexander Sloane. The Theory of ErrorCorrecting Codes . Vol. 16. North-Holland Mathematical Library. North-Holland Publishing Company, 1977.

Eran Malach et al. 'Proving the Lottery Ticket Hypothesis: Pruning Is All You Need'. In: Proceedings of the 37th International Conference on Machine Learning . ICML'20. JMLR.org, July 2020, pp. 6682-6691. (Visited on 03/26/2023).

James E Marengo, David L Farnsworth, Lucas Stefanic, et al. 'A geometric derivation of the Irwin-Hall distribution'. In: International Journal of Mathematics and Mathematical Sciences 2017 (2017).

Laurent Orseau, Marcus Hutter, and Omar Rivasplata. 'Logarithmic Pruning Is All You Need'. In: Proceedings of the 34th International Conference on Neural Information Processing Systems . NIPS'20. Red Hook, NY, USA: Curran Associates Inc., Dec. 2020, pp. 29252934. ISBN: 978-1-71382-954-6. (Visited on 03/26/2023).

Hikari Otsuka et al. Partial Search in a Frozen Network Is Enough to Find a Strong Lottery Ticket . Feb. 2024. DOI: 10.48550/arXiv.2402.14029 . arXiv: 2402.14029 [cs, stat] . (Visited on 05/15/2024).

Ankit Pensia et al. 'Optimal lottery tickets via SUBSETSUM: logarithmic overparameterization is sufficient'. In: Proceedings of the 34th International Conference on Neural Information Processing Systems . NIPS '20. , Vancouver, BC, Canada, Curran Associates Inc., 2020. ISBN: 9781713829546.

Valentin V. Petrov. Sums of Independent Random Variables . Ergebnisse der Mathematik und ihrer Grenzgebiete. 2. Folge. Springer Berlin, Heidelberg, 1975, 348 pages. DOI: https: //doi.org/10.1007/978-3-642-65809-9 .

Vivek Ramanujan et al. 'What's Hidden in a Randomly Weighted Neural Network?' In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . June 2020, pp. 11890-11899. DOI: 10.1109/CVPR42600.2020.01191 .

N. Shakhaidarova. 'Uniform local and global theorems for densities'. In: Izv. Akad. Nauk UzSSR Ser. Fiz-Mat. Nauk 5 (1966), pp. 90-91.

Irina Shevtsova. Onthe absolute constants in the Berry Esseen type inequalities for identically distributed summands . 2011. arXiv: 1111.6554 [math.PR] .

Hattie Zhou et al. 'Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask'. In: Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019 (NIPS 2019) . 2019, pp. 3592-3602.

A Lower Bound on the Ticket Size in [24]

The claim is a direct consequence of the proof of [24, Theorem 2] (Appendix B). There, in Step 3, it is shown that

where G is the set of subnetworks that can be formed. Let m be the number of parameters of the original network. If we consider subnetworks of size at most γm ( 0 ≤ γ ≤ 1 ), we have 4

which combined with the previous inequality implies

If we have an overparameterized network of size m = O ( d 2 log 2 ( 1 2 ε ) ) , as in [24], we need γm = Θ( m ) for the last inequality to be satisfied (note that log 2 ( e γ ) ≤ 1 , as 0 ≤ γ ≤ 1 ).

B Visualizations

train

train

prune

reinit

prune

C Proof of Uniform [ -1 , 1] being Sum-Bounded

In this section we provide a detailed proof of Lemma 1, which states that the uniform distribution in [ -1 , 1] is sum-bounded, as stated in Definition 1. We remark that, while the proof is written for uniform random variables, it should be possible to extend it to a family of densities which are unimodal, with bounded variance, and bounded third moment.

Proof of Lemma 1. Note first that the distribution of the sum of n i.i.d. variables in [0 , 1] is known as the Irwin-Hall distribution I n . 5 We will use that Var[ I n ] = n 12 , where Var[ X ] denotes the variance of the random variable X .

For n ≥ 2 , f ( x, n ) can be defined as the convolution of f ( x ) = f ( x, 1) and f ( x, n -1) , i.e.,

It is straightforward to show, by induction and an elementary substitution in the integral above, which is relied upon in the inductive step, that f ( x, n ) is symmetric about 0 , that is f ( x, n ) = f ( -x, n ) .

Let us now prove by induction that f ( x, n ) is nondecreasing on the interval [ -n, 0] and nonincreasing over [0 , n ] (for simplicity, since it vanishes outside [ -n, n ] , we can consider directly the negative half and positive half of the real line, respectively, in the argument that follows).

The claims hold trivially for f ( x ) ; also note that

If x ≤ x ′ ≤ -1 . Since x -τ ≤ x ′ -τ ≤ 0 , by inductive hypothesis we have that f ( x -τ, n -1) ≤ f ( x ′ -τ, n -1) over the whole interval τ ∈ [ -1 , 1] . Taking integrals yields f ( x, n ) ≤ f ( x ′ , n ) .

Now, consider the case when x ≤ -1 ≤ x ′ ≤ 0 . If x + 1 ≤ -x ′ -1 , x -τ ≤ x + 1 ≤ -x ′ -1 ≤ -x ′ + τ ≤ -x + τ . By the symmetry about the origin, the inductive hypothesis is f ( x -τ, n -1) = f ( -x + τ, n -1) ≤ f ( -x ′ + τ, n -1) = f ( x ′ -τ, n -1) over the whole interval τ ∈ [ -1 , 1] , since -1 ≤ -x ′ + τ ≤ -x + τ . Taking integrals yields f ( x, n ) ≤ f ( x ′ , n ) . Otherwise, there exists τ 0 such that x -τ 0 = -x ′ -1 , x -τ > -x ′ -1 for all τ ∈ [ -1 , τ 0 ) and x -τ < -x ′ -1 for all τ ∈ ( τ 0 , 1] . By symmetry, using -x = x ′ +1 -τ 0 , f ( x -τ, n -1) = f ( -x + τ, n -1) = f ( x ′ +1+ τ -τ 0 , n -1) . Thus, for all τ ∈ [ -1 , τ 0 ] , via the change of variable σ = -(1 + τ -τ 0 ) in the middle integral below, we obtain that

For all τ ∈ ( τ 0 , 1] , x -τ < -x ′ -1 ≤ -x ′ + τ ≤ -x + τ , by symmetry about the origin we have that f ( x -τ, n -1) ≤ f ( x ′ -τ, n -1) by the inductive hypothesis with the same reasoning of the case x +1 ≤ -x ′ -1 . Taking integrals over the range [ τ 0 , 1] for each term of the inductive hypothesis yields

Eqs. 25 and 26 imply that f ( x, n ) ≤ f ( x ′ n ) .

Trivially, if -1 ≤ x ≤ x ′ ≤ 0 , analogous ideas are put in place as for the previous case, therefore we omit the details. We have thus shown the nondecreasing monotonicity of f ( x, n ) on the negative half of the real line. By the symmetry of f ( x, n ) about the origin, on the positive half of the real line the nondecreasing monotonicity turns into nonincreasing monotonicity, and the proof is complete.

Lower bound (first inequality in Eq. 2). The variance of Σ n [ n ] is n/ 3 since Σ n [ n ] = 2( I n ( n ) -n/ 2) and Var[ I n ( n )] = n/ 12 . We define Z u n = Σ U n [ n ] √ n/ 3 and we note with F n its cumulative distribution function. Z u n has expectation 0 and standard deviation 1. Consider the probability

Now, we use the following form of Berry-Esseen inequality, discussed in [21][p.2]). 6

Theorem 6 (Allasia [1]) . For all n ≥ 1 ,

where Φ( z ) is the cumulative distribution function of the standard normal distribution.

Theorem 6 implies

When n ≥ 18 ,

That is P L ( n ) ≥ C 18 > 0 . When 2 ≤ n < 18 , P L ( n ) = F n (2 √ 3) -F n ( √ 3) = c n > 0 . We thus have

Recall that P L ( n ) = Pr( √ n ≤ Σ U n [ n ] ≤ 2 √ n ) . As the density f ( x, n ) is decreasing on R + , we have

When n = 1 , the density f (1 , 1) = 1 2 . So, by setting c l = min( c ′ l , 1 2 ) , we get that, for all n ≥ 1 , for all 0 ≤ x ≤ √ n :

By a symmetric argument, we also have for all n ≥ 1 , for all - √ n ≤ x ≤ 0 :

Upper bound (second inequality in Eq. 2). Here, we bound the probability distribution function f ( x, n ) of Σ U n [ n ] = √ n / 3 Z n , where we recall that Z u n = Σ U n [ n ] √ n / 3 . Denoting f z the probability distribution function of Z u n , we have

We use the following local limit theorem, discussed in [25][p.214].

Theorem 7 (Sahaidarova [27]) . Let { X n } be a sequence of independent random variables with a common density p ( x ) , such that E [ | X 1 | 3 ] < ∞ , E [ X 1 ] = 0 , E [ X 2 1 ] = 1 and sup p ( x ) ≤ C . Let p n ( x ) be the density of the random variable 1 √ n ∑ n j =1 X j . Then

where ϕ is the probability distribution function of a standard gaussian, A is an absolute constant, and β 3 = E [ | X 1 | 3 ] .

Thus,

Since P L ( n ) ≥ c ′ l then for all n ≥ 2

The theorem can be applied to a uniform continuous distribution with density p u ( x ) = 1 2 √ 3 in the interval [ - √ 3 , √ 3] , which has mean 0 and variance 1 . We thus get, for every x ∈ R ,

In conclusion, setting c u = √ 3 c ′ u , for every x ∈ R it holds that

D Proof of Corollary 1

Proof of Corollary 1. As anticipated, we proceed in three steps.

Step 1: Hoeffding bound. We start by showing, following the idea at the base of [17, Corollary 3.3], that if n ′ is large enough, a standard Hoeffding bound ensures that with high probability a constant fraction of the sample follows a Uniform [ -1 , 1] distribution. Since we assumed that every X i is a mixture of a Uniform [ -1 , 1] distribution with probability p , and another distribution with density g (given by the factors G i ), we can rewrite X i = B i · U i + (1 -B i ) · G i , with U i being the uniform random variable, G i being the random variable with density g , B i being independent Bernoulli random variables with probability p .

̸

Fix α = α ( p ) = p , and assume, for now, that n ′ satisfies Eq. 1, and therefore, since ε < 1 / 2 , choosing c hyp = c hyp ( p ) ≥ ( α -p ) -2 , ensures that, defining ε ′ = ε / 2 ,

and therefore

Thus

that is, with high probability, there is a set of indices I ⊆ [ n ′ ] of size | I | ≥ αn ′ , such that for each i ∈ I it holds B i = 1 , i.e. X i is uniformly distributed.

Step 2: Application of Theorem 2 via rejection-sampling. Lemma 1 ensures that the uniform distribution of the | I | random variables selected in Step 1 is sum-bounded. Conditionally on the event { ∑ n ′ i B i > αn ′ } , we can discard all random variables indexed outside I and apply directly Theorem 2 to αn ′ of the remaining ones, for any fixed k and z ∈ [ -√ k, √ k ] , since αc hyp ≥ 1 by construction. This guarantees a success probability of c ′ thm for approximating the given target z ; thus,

Step 3: Amplification. Finally, by a standard probability amplification argument and a union bound applied to Theorem 2, by paying an extra factor log 2 ( k/ε ) in Eq. 1, the constant c thm can be amplified to 1 -ε , and the existence of a suitable subset S z holds simultaneously for all z ∈ [ -√ k, √ k ] . We now give more details on this amplification.

Recall that ε ′ = ε 2 , and let c amp = c amp ( p ) = 8 c hyp c thm and r = 4 c thm ln k ε . By assumption,

where the last inequality is ensured by ε < 1 / 2 . By Step 2 , we can apply Theorem 2, with ε ′ and n ′ ≥ c hyp log 2 k ε ′ H 2 ( k n ) = n ∗ , allowing us to prove that we can ε ′ -approximate any target z with probability at least c thm. The probability of failing to approximate some given z is then at most 1 -c thm. From the sample Ω of sum-bounded random variables take r subsamples (without replacement) of cardinality n ∗ each, Ω 1 , . . . , Ω r . The probability of failing to approximate some given z with subsetsums from Ω is less than that of failing to approximate it with subsetsums from within every Ω i 's, and the latter probability is at most (1 -c thm ) r ; thus, for every z ∈ [ -√ k, √ k ] ,

By an union bound, we also have that

where the last inequality is ensured by ε < 1 / 2 . This completes the proof.

E Proof of Corollary 2

Proof of Corollary 2. By definition of binary entropy, we have

In particular, since both terms in the previous equation are positive, we get

We now use eq. (28) to derive an upper bound for the quantity c amp H 2 ( k n ) log 2 2 k +2 log 2 k · log 2 1 / ε n , which will be used later:

where from eq. (29) to eq. (30) we used that log 2 n / k ≥ 1 for k ≤ n / 2 , and then the hypothesis k ≥ 2 c amp ( log 2 2 k +2 log 2 k · log 2 1 ε ) directly gives eq. (31). Let us now rewrite eq. (3) in a more convenient form:

Using eq. (31) we get

To satisfy eq. (33), we can then choose n such that

and then apply Corollary 1 to end the proof.

F Details for the proof of Theorem 2

Proof of Eq. 17 Define A = ˜ S ′ \ ˜ S , and observe that

where from Eq. 36 to Eq. 37 and from Eq. 37 to Eq. 38 we used the law of total probability; 7 from Eq. 38 to Eq. 39 we dropped the redundant event H ˜ S in the conditioning, due to conditional independence; finally, from Eq. 39 to Eq. 40 we used Definition 1 which implies that for any i ∈ { µk, ..., k -1 } it holds

Proof of Eq. 18 Let A = ˜ S ′ \ ˜ S . Analogously to the calculations from Eq. 37 to Eq. 39, by the law of total probability we have

where from Eq. 41 to Eq. 42 we used Definition 1, which implies that for any i ∈ { 0 , ..., 9 10 k -1 } it holds

G Proof of Theorem 3

In the proof we will refer to the following results, upon which [24][Theorem 1] relies (the statement below slightly differ as we fix two small typos in their notation and mixing coefficients). With the understanding that by a mixture D of a distribution D 1 and D 2 with probability p it is meant that the pdf (we adopt the convention that this term includes generalised functions, such as Dirac deltas for point masses) of D can be written as a convex combination of the pdf of D 1 and that of D 2 , that is f D = pf D 1 + (1 -p ) f D 2 . For the unfamiliar reader, we note that in the literature this is often stated in short as D = pD 1 +(1 -p ) D 2 .

Lemma 2 ([24][Corollary 1]) . Let X ∼ Uniform [0 , 1] (or X ∼ Uniform [ -1 , 0] ) and Y ∼ Uniform [ -1 , 1] be independent random variables. Let P be the distribution of the XY and δ 0 the Dirac delta at 0 . Let D be the distribution obtained as mixture of δ 0 and P with probability 1 / 2 . Then D is the mixture of a Uniform [ -1 / 2 , 1 / 2 ] and some distribution Q with probability ln(2) / 4 . Corollary 3 ([24][Corollary 2]) . Let X 1 , . . . , X n be iid with distribution D as defined in Lemma 2, where n ≥ C ln( 2 / ε ) for some universal constant C . Then

Proof of Theorem 3. The key idea is exploiting Corollary 1 at each step of the pruning strategy established in [24][Theorem 1], where Corollary 3 is used instead. Without loss of generality, we replace their min { ε, δ } with ε . For the sake of easily following the approach adopted in [24], let us define n ∗ ( x ) as the function

where k = γ ′ n ∗ ( x ) . In the following, we use n ∗ as short for n ∗ (1 /ε ) , and we will only explicitely provide an argument for n ∗ when it is different than 1 /ε . For instance, in the last step of the proof, we will use n ∗ ( 2 ℓd i d i -1 / ε ) , which matches the definition of n ∗ i given in Eq. 20.

Consider [24][Lemma 1]. When approximating a single link (that is, a weight), after the overparameterization (which creates an additional layer of width 2 n ∗ in between the input and the output node) via 4 n ∗ links, instead of pruning via Corollary 3, we prune via Corollary 1 twice in the second layer, that is we ensure that only k = γ ′ n ∗ edges yield the desired approximation, both in the edges corresponding to the positive part of the input weights and in those corresponding to the negative part. Thus we obtain at most 4 k surviving edges, after the preprocessing step and the pruning mask is applied. This yields a sparsity of at least α ′ = 1 -γ ′ . Note that it is because of the preprocessing step that we go from distributions Uniform [ -1 , 1] to distributions D , as defined in Lemma 2, which are shown to be a mixture with Uniform [ -1 , 1] and therefore can be also handled via Corollary 1.

Consider [24][Lemma 2]. When approximating a real-valued multivariate linear function, after the overparameterization (which creates an additional layer of width 2 dn ∗ ( d / ε ) in between the d input nodes and the output node) one simply iterates the ideas of the previous case d times. For each input node, the overparameterization surviving the preprocessing step on the weights of the input layer is 4 n ∗ ( d / ε ) . Pruning the second layer of the overparameterized link for each input via Corollary 1 with k = γ ′ n ∗ ( d / ε ) (again, performing this both on the edges corresponding to the positive part of the input weights and in those corresponding to the negative part), instead of exploiting Corollary 3, yields that at most 4 dk edges survive after the pruning mask is applied. This yields a sparsity of at least α ′ = 1 -γ ′ .

Finally, consider [24][Lemma 3]. When approximating a layer with input dimension d 1 and output dimension d 2 , after the overparameterization (which creates an additional layer of width 2 d 1 n ∗ ( d 1 d 2 / ε ) in between the input nodes and the output nodes) one iterates the ideas of the previous case d 1 times in the input layer through the same preprocessing step, and d 2 times in the output layer, one for each of the d 1 blocks created by the preprocessing (essentially the weights in the input layer are re-used d 2 times). For each input node, the overparameterization surviving the preprocessing step is at most 2( d 2 + 1) n ∗ ( d 1 d 2 / ε ) . Overall, after the preprocessing step, we have at most 2 d 1 ( d 2 + 1) n ∗ ( d 1 d 2 / ε ) parameters. We then use Corollary 1 (with k = γ ′ n ∗ ( d 1 d 2 / ε ) ) to prune the number of parameters between the introduced additional layer and the d 2 outputs down to 2 d 1 d 2 γ ′ n ∗ ( d 1 d 2 / ε ) . As for the edges between the d 1 inputs and the additional layer, only those that reach a neuron in the additional layer, from which there is at least one outgoing edge towards the d 2 outputs, are used; since for each of the d 1 blocks of 2 n ∗ ( d 1 d 2 / ε ) neurons in the additional layer we only kept 2 γ ′ n ∗ ( d 1 d 2 / ε ) outgoing edges to each of the d 2 output neurons, in the worst case (all the nodes involved in the subsetsums are disjoint) we keep 2 d 2 γ ′ n ∗ ( d 1 d 2 / ε ) of them for each of the d 1 neurons. Globally, we are left with a total of at most 2 d 1 d 2 γ ′ n ∗ ( d 1 d 2 / ε ) edges both in the input layer and in the output layer, thus a total of 4 d 1 d 2 γ ′ n ∗ ( d 1 d 2 / ε ) edges survive the pruning. The density of the surviving edges is then less than

where ρ 1 = max { d 1 / d 2 , d 2 / d 1 } and in the last inequality we used that d 1 d 2 / d 1 + d 2 ≤ ρ 1 ( d 1 + d 2 ) since ρ 1 ≥ 1 . This ensures a sparsity α ′ ≥ 1 -ρ 1 γ ′ .

[24][Theorem 1] consists of performing, for every i ∈ [ ℓ ] , the previous step on layer i with input dimension d i -1 and output dimension d i . The overparameterization creates an additional layer of nodes of width 2 d i -1 n ∗ ( 2 ℓd i -1 d i / ε ) in between the d i -1 input nodes and the d i output nodes. Since the construction is stacked ℓ times, this generates 2 ℓ layers for the overparameterized network, which will therefore have a starting number of parameters

Corollary 1 applied to each stacked overparameterized layer instead of Corollary 3 as in the previous step yields that the total number of parameters left after the pruning is

where k i = γ ′ n ∗ ( 2 ℓd i -1 d i / ε ) . Recall that ρ = max i ρ i , where ρ i = max { d i / d i -1 , d i -1 / d i } ≥ 1 . Recall that γ = ργ ′ . We obtain that

Wethen get that the density of the edges surviving the pruning is m t / m ≤ γ , which implies a sparsity of at least α = 1 -γ .

H Proof of Theorem 5

Proof of Theorem 5. Consider the space W k = { W ∈ R d × d : ∥ W ∥ ≤ √ k } , and let P k be a 2 ε -separated set of W k , i.e. a subset P k ⊂ W k such that for all distinct W,W ′ ∈ P k it holds ∥ W -W ′ ∥ > 2 ε . We denote W = W 1 , P = P 1 , and the set of all possible subnetworks of g as G (note that this does not denote G 1 , the set of all subnetworks of size 1 ).

Step 1: Packing argument. In [24][Theorem 2, Step 1], it is shown that any function g ′ can only approximate at most one member of P for bounded input x (say, ∥ x ∥ ≤ 1 ). In particular, this also applies to functions g ′ representing the elements of G k .

Step 2: Relation between |G k | and |P k | . By Step 1 , in [24][Theorem 2, Step 2] it is shown that |P| ≤ 2 |G| , under the assumption of Eq. 24, with G k replaced by G ). Therefore, also by Step 1 , replacing P with P k and G with G k in [24][Theorem 2, Step 2], it holds that |P k | ≤ 2 |G k | . Note that |G k | = ( n k ) , the number of different ways in which we can select k parameters out of n , so we actually get

Step 3: Lower bound on |P k | . Let us now consider a 2 ε -separated set P max k of maximal cardinality. In [24][Theorem 2, Step 3] it is shown that

Here Vol is the Lebesgue measure in R d × d identified with R d 2 .By the exact same argument, replacing W with W k and thus P max with P max k , it holds that

Combining this fact with Eq. 44 applied to P max k implies that

Step 4: Lower bound on n . Consider the standard bound found in [19]

and combine it with with Eq. 45. It follows that

and taking the logarithm of both sides yields the sought lower bound on n :

where from Eq. 46 to Eq. 47 we exploited the definition of λ , which ensures that the first term in the r.h.s. of Eq. 46 is nonnegative; 8 from Eq. 47 to Eq. 48 we used that for all ε < 1 / 16 it holds that

I Details of comparison with Malach et al. [20]

We show that cz ≥ c amp log 2 2 ( cz 2 ) log 2 ( z ) holds for a big enough constant c . Recall that z = m t ε , so we can always assume log 2 ( z ) ≥ 1 . We have

where in ( a ) we used that 2 ab ≤ a + b , and in ( b ) that a + b ≤ 2 ab for a and b greater than 1.

We can then focus on showing that there is a big enough constant c such that cz ≥ 12 c amp log 2 ( c ) log 2 ( z ) . We get c ≥ 12 c amp log 2 ( c ) log 2 2 ( z ) , and we have

2 2 2 2 2 z

We can then focus on c ≥ 12 c amp √ c , which is satisfied for c ≥ 144 c 2 amp .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and the introduction precisely reflect our contribution.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have added a discussion on the limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Assumptions for all results are accurately stated in our statements.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted confrms in every respect with the NeurIPS Code of Ehtics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: A small section discussing the potential impacts of our work has been added to the paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We not releasing any data or models (apart from mathematical models) hence there is no risk of misues.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: Everything is cited properly and no assests are required.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No assets are introduced.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No such research was conducted.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: There are not study participants.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an existence proof for Strong Lottery Tickets(SLTs) by improving over the existing method which used the Subset Sum approximation to construct source networks, which involved using subsets of variable sizes in order to approximate target parameters. Instead the authors show the existence of SLTs using subsets with fixed sizes (RFSS) and leverage this results to show the existence of SLTs.\n\nSTRENGTHS:\n1. The authors propose the Random Fixed Subset Sum approximation instead of the previously used Subset Sum approximation to prove the existence of Strong Lottery Tickets. This results allows them to use a subset of a fixed size to approximate parameters in a target network.\n2. The subsequent existence proof interpolated between the result of Malach et. al. [1] (which uses a single parameter) and Pensia et. al. [2] (which uses a subset of parameters) based on the size of the subset being used. The authors also extend this result to equivariant networks.\n\nWEAKNESSES:\n1. Given the theoretical nature of the work, the overall contribution appears limited as it can only connect the existing results established by [1], [2], [3].\n2. The main insight in this paper is that by using a fixed subset size to approximate parameters in the target, one can exactly estimate the number of parameters required in the strong lottery ticket, which was previously not possible due to the variable size of subsets. However, as Burkholz et. al. [3] have shown via numerical simulations, the number of parameters required to satisfy the theoretical conditions for subset sum proofs can be relatively easily approximated. For example, they use a subset of size 15 for their simulations which is sufficient. Given this, the overall contribution of the paper seems incremental \n\nThe following claims in the paper need to be addressed appropriately.\n1. ‘We provide the first proof of the SLTH in classical settings, such as dense and equivariant networks, with guarantees on the sparsity of the subnetworks.’ While the existence of SLTs has been shown before in various settings, the sparsity of these networks has also been estimated using numerical simulations by Burkholz et. al. [3]. This is primarily because the size of the subset required to effectively approximate a parameter seems to have a natural upper bound (15 as shown in [3]). Hence, while the authors are able to show this exactly, previous theoretical work did a decent approximation. \n2. ‘It is important to note that, to this day, we only have proofs on the existence of such subnetworks, also called winning tickets, but it remains an open question how to find them reliably.’ While the theoretical framework is laid out by the authors to construct the SLTs, the paper lacks numerical constructions of real world examples of neural networks using the proposed framework. Are the networks found by the proposed method sparser than the one's found by [3]? \n\n\n[1] Malach, Eran, et al. \"Proving the lottery ticket hypothesis: Pruning is all you need.\" International Conference on Machine Learning. PMLR, 2020.\n\n[2] Pensia, Ankit, et al. \"Optimal lottery tickets via subset sum: Logarithmic over-parameterization is sufficient.\" Advances in neural information processing systems 33 (2020)\n\n[3] Burkholz, Rebekka. \"Most activation functions can win the lottery without excessive depth.\" Advances in Neural Information Processing Systems 35 (2022)\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nThe work leverages Random Fixed Subset Sum result to adapt a large body of work showing the existence of SLTs. However, in the context of the existing work, this result seems incremental and lacks numerical validation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on the theoretical side of the Strong Lottery Ticket Hypothesis. It proposes and studies a fixed-size version of the canonically used Random Subset Sum (abbreviated RFSS) problem in previous proofs for this hypothesis. Then it applies the result of RFSS to prove that overparameterized networks can be pruned to approximate target networks with guaranteed sparsity. A corresponding lower bound is provided to show the tightness (up to log factor) of the overparameterization needed.\n\nSTRENGTHS:\n- The proposed fixed-size version of RSS is an interesting problem and is potentially of independent interest.\n- Based on the literature reviewed here, this work is the first one that provides a (high probability) size guarantee for the pruned subnetwork.\n- A lower bound on the overparameterization size is also provided, which helps complete the picture and matches the high probability guarantee up to a log factor.\n\nWEAKNESSES:\n- Elaboration of Theorem 3 is helpful in the main text. I do not see how it answers the question proposed in Introduction. Namely, for given target network size $m'$, the internal dimensions $d_\\ell$, and a desired sparsity $\\alpha$, how should we pick the number of parameters (or the number of layers) for our networks? Also, how does it imply the previous results?\n- More discussion should be given to the results on the hypothesis (e.g. elaborations) in Section 4 and 5. While the RFSS result is key, instead of having the whole proof in Section 3, the authors can leave a sketch of the proof idea and where it distinguishes / extends RSS results.\n\nQUESTIONS:\n- On line 225 Theorem 3, is it $r = \\max_i \\min\\{…\\}$ instead of only $\\max_i$ (in which case it’s always at least 1)?\n- What abbreviates to ANN in Theorem 4?\n\nLIMITATIONS:\nAddressed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the strong lottery ticket hypothesis (SLTH), which is roughly a statement that a large random neural network contains a sparse subnetwork whose performance is comparable to the entire network. By analyzing a fixed variant of the random subset sum problem, which is about the size of the sample of i.i.d. random variables to approximate a fixed number, they improved the previous results on the SLTH with the guarantee on the sparsity of the subnetwork.\n\nSTRENGTHS:\n- The idea based on the fixed-size random subset sum problem seems new and works well.\n- The writing is in general very clean, especially the introduction and the related work.\n\nWEAKNESSES:\n- I think the authors should check more about the references for the results in Section 3. (See Questions.)\n- There are many small errors and typos in the manuscript. (See Questions.)\n\nQUESTIONS:\n- The assumption on the sum-boundedness in Definition 1 must hold for a very general density function $f$, since it is even weaker than the statement of the local limit theorem for density.\n(See, e.g., Theorem 7 in Section 7.2 of \"Sums of Independent Random Variables\" by Petrov.)\nThus, I think some parts of Section 3, especially Definition 1 and Lemma 1, can be changed.\n- Below are small errors and typos:\n1) In line 136, there is something missing in \"we $\\Omega$\".\n2) In line 140, $f_{\\Sigma_{[i]}}$ should be $\\Sigma_{[i]}$.\n3) In line 182, \"independent and uniformly at random\" sounds weird.\n4) In Equation (6) and other places, $X_S$ should be $Z_S$.\n5) In line 186, \"Eqs. 5 and 8\" should be \"Eqs. 6 and 8\".\n6) From Equation (19) to Equation (20), why $H_z(\\tilde{S})$ can be dropped?\n7) The equation below 204 seems wrong. I guess $(z-y)$ should be $z$ in the second line.\n8) In line 219, \"Let $F$ to be\" should be \"Let $F$ be\".\n\nLIMITATIONS:\nThe work does not seem to have potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the strong lottery ticket hypothesis from the perspective of pruning sparsity. In particular, the paper noticed that all previous papers about the strong lottery ticket hypothesis fails to characterize the sparsity of the neural network after pruning, and aims at solving this question in the paper. The paper first extends the random subset sum problem to a case where the selected subset has a cardinality constraint. The paper proves a probability bound in terms of the candidate set's size for the random subset sum to achieve a certain accuracy. Then, the paper utilize this bound to the case of the neural networks by utilizing the technique in previous works, and shows the over-parameterization required to achieve a certain accuracy and sparsity after pruning at initialization. The paper also included a lower bound on such over-parameterization requirement.\n\nSTRENGTHS:\n1. The topic considered in the paper is quite interesting and significant. Indeed, one draw-back of the strong lottery ticket hypothesis is that there is no guarantee on the sparsity level after pruning.\n2. A major contribution of the paper is that it extended the random subset sum problem to a case that involves a cardinality constraint on the subset. The proof seems quite technical and nontrivial.\n3. The paper also included a lower-bound on the over-parameterization requirement.\n\nWEAKNESSES:\n1. The statement of Theorem 2 and Corollary 1 seems weird. In particular, it should be noticed that for both Eq.(1) and Eq.(3), $n$ appears in both the left hand side and the right hand side of the inequality. With some simplification, it seems that the dependency on $n$ can be completely eliminated, and the size of $n$ does not contribute to whether the statement holds or not. In this case, it would be greatly necessary to understand how big the constant $c_{hyp}$ is, which is not stated in the theorem. Moreover, if $n$ is really not affecting the approximation error, then a better way to state the theorem would be to change Eq.(1) into a condition on $k$.\n2. The proof of Corollary 1 seems problematic. In particular, in Page 15, the first inequality after line 483, the $\\geq$ should be $\\leq$, since $Pr(\\exists z\\in[-\\sqrt{k},\\sqrt{k}],\\dots) \\geq Pr(\\exists z\\in\\\\{-\\sqrt{k} + i\\epsilon':i\\in[\\frac{2}{\\epsilon'}\\sqrt{k}]\\\\},\\dots)$ (because $\\\\{-\\sqrt{k} + i\\epsilon':i\\in[\\frac{2}{\\epsilon'}\\sqrt{k}]\\\\}\\subseteq [-\\sqrt{k},\\sqrt{k}]$). This breaks the whole proof.\n3. The definition of $\\alpha$ (the sparsity) is ambiguous. From the proof it seems that $\\alpha$ denotes the fraction of the zero elements. In this case, however, the result in Theorem 3 seems trivial. In particular, the size of the neural network scales inversely with the $\\gamma'$ (roughly the fraction of the nonzero elements), because to achieve $\\gamma'$ portion of the nonzero elements, one simply need to initialize $\\frac{1}{\\gamma'}$ independent neural networks and prune completely the rest $\\frac{1}{\\gamma'} - 1$. Moreover, the definition of $r$ is problematic since it seems that by defining $r = \\max\\\\{\\frac{d_i}{d_{i-1}}, \\frac{d_{i-1}}{d_i}\\\\}$, we must have that $r\\geq 1$ (instead of $r\\leq 1$ as claimed in the paper).\n4. This is a minor issue: the upper bound on the over-parameterization does not match the lower bound.\n\nQUESTIONS:\nWhy does the paper focus on the exact sparsity constraint $|S| = k$? Would it be more practical to focus on $|S|\\leq k$?\n\nLIMITATIONS:\nThe paper has a good discussion about its limitation from a practical perspective. However, I believe that the paper needs to add some discussion about the limitation from a theoretical perspective (tightness of the bound, strength of the assumptions etc.)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}