Title: Polynomially Over-Parameterized Convolutional Neural Networks Contain Structured Strong Winning Lottery Tickets
Venue: NeurIPS 2023
Accepted: yes
Keywords: lottery ticket hypothesis

convolutional neural network

network pruning

structured pruning

random subset sum
Abstract: The Strong Lottery Ticket Hypothesis (SLTH) states that randomly-initialised neural networks likely contain subnetworks that perform well without any training. Although unstructured pruning has been extensively studied in this context, its structured counterpart, which can deliver significant computational and memory efficiency gains, has been largely unexplored. One of the main reasons for this gap is the limitations of the underlying mathematical tools used in formal analyses of the SLTH.

In this paper, we overcome these limitations: we leverage recent advances in the multidimensional generalisation of the Random Subset-Sum Problem and obtain a variant that admits the stochastic dependencies that arise when addressing structured pruning in the SLTH. We apply this result to prove, for a wide class of random Convolutional Neural Networks, the existence of structured subnetworks that can approximate any sufficiently smaller network.

This result provides the first sub-exponential bound around the SLTH for structured pruning, opening up new avenues for further research on the hypothesis and contributing to the understanding of the role of over-parameterization in deep learning.

Polynomially Over-Parameterized Convolutional Neural Networks Contain Structured Strong Winning Lottery Tickets

Abstract

The Strong Lottery Ticket Hypothesis (SLTH) states that randomly-initialised neural networks likely contain subnetworks that perform well without any training. Although unstructured pruning has been extensively studied in this context, its structured counterpart, which can deliver significant computational and memory efficiency gains, has been largely unexplored. One of the main reasons for this gap is the limitations of the underlying mathematical tools used in formal analyses of the SLTH. In this paper, we overcome these limitations: we leverage recent advances in the multidimensional generalisation of the Random Subset-Sum Problem and obtain a variant that admits the stochastic dependencies that arise when addressing structured pruning in the SLTH. We apply this result to prove, for a wide class of random Convolutional Neural Networks, the existence of structured subnetworks that can approximate any sufficiently smaller network.

This result provides the first sub-exponential bound around the SLTH for structured pruning, opening up new avenues for further research on the hypothesis and contributing to the understanding of the role of over-parameterization in deep learning.

1 Introduction

Much of the success of deep learning techniques relies on extreme over-parameterization [Zhang et al., 2017, 2021, Novak et al., 2018, Brutzkus et al., 2018, Du et al., 2019, Kaplan et al., 2020]. While such excess of parameters has allowed neural networks to become the state of the art in many tasks, the associated computational cost limits both the progress of those techniques and their deployment in real-world applications. This limitation motivated the development of methods for reducing the number of parameters of neural networks; both in the past [Reed, 1993] and in the present [Blalock et al., 2020, Hoefler et al., 2021].

Although pruning methods have traditionally targeted reducing the size of networks for inference purposes, recent works have indicated that they can also be used to reduce parameter counts during training or even at initialisation without sacrificing model accuracy. In particular, Frankle and Carbin [2019] proposed the Lottery Ticket Hypothesis (LTH) , which conjectures that randomly-initialised

networks contain sparse subnetworks that can be trained and reach the performance of the fullytrained original network. Empirical investigations on the LTH [Zhou et al., 2019, Ramanujan et al., 2020, Wang et al., 2020] pointed towards an even more striking phenomenon: the existence of subnetworks that perform well without any training. This conjecture was named the Strong Lottery Ticket Hypothesis (SLTH) by Pensia et al. [2020].

While the SLTH has been proved for many different classes of neural networks (see Section 2), those works are restricted to unstructured pruning, where the subnetworks are obtained by freely removing individual parameters from the original network. However, this lack of structure can significantly reduce the gains that sparsity can bring, both in terms of memory and computational efficiency. The possibility of removing parameters at arbitrary points of the network implies the need to store the indices of the remaining non-zero parameters, which can become a significant overhead with its own research challenges [Pooch and Nieder, 1973]. Moreover, the theoretical computational gains of unstructured sparsity can also be difficult to realise in standard hardware, which is optimised for dense operations. Most notably, the irregularity of the memory access patterns can lead to both data and instruction cache misses, significantly reducing the performance of the pruned network.

The limitations of parameter-level pruning have motivated extensive research on structured pruning , which constrain the sparsity patterns to reduce the complexity of parameter indexation and, more generally, to make the processing of the pruned network more efficient. A simple example of structured pruning is neuron pruning of fully-connected layers: deletions in the weight matrix are constrained to the level of whole rows/columns. As illustrated by Figure 1, pruning under this constraint produces a smaller network that is still dense, directly reducing the computational costs without any need for extra memory to store indices. Similarly, deleting entire filters in Convolutional Neural Networks (CNNs) [Polyak and Wolf, 2015] or 'heads' in attention-based architectures [Michel et al., 2019] also produces direct reductions in computational costs.

It is important to note that structured pruning is a restriction of unstructured pruning so, theoretically, the former is bound to perform at most as well as the latter. For example, by deleting whole neurons one can remove about 70% of the weights in dense networks without significantly affecting accuracy, while through unstructured pruning one can usually reach 95% sparsity without accuracy loss [Alvarez and Salzmann, 2016, Liu et al., 2019]. In practice, however, the computational advantage of structured pruning can offset this difference: a suitably structured sparse network can be more efficient than an even sparser one that lacks structure. This trade-off between sparsity and actual efficiency has motivated the study of less coarse sparsity patterns since weaker structural constraints such as strided sparsity [Anwar et al., 2017] (Figure 2b) or block sparsity [Siswanto, 2021] (Figure 2c) are already sufficient to deliver the bulk of the computational gains that structured can offer.

Despite its benefits, structured pruning has received little attention in the context of the SLTH. In fact, the work that first proved a version of the SLTH, Malach et al. [2020], has remained the only one to study this scenario, to the best of our knowledge. Moreover, it brings a negative result: the authors prove that removing neurons from a randomly-initialised shallow neural network (a single hidden layer) is equivalent to the random features model 1 (e.g., Rahimi and Recht [2007, 2008]),

Unstructured sparsity

(a) No pattern.

Structured sparsity

(b) Strided pattern.

(c) Block pattern.

which cannot efficiently approximate even a single ReLU neuron [Yehudai and Shamir, 2019]. As most proofs around the SLTH involve pruning a shallow random network to build the desired approximations, those results show that the SLTH can be challenging to tackle when restricted to neuron pruning.

In addition, we believe that a factor hindering progress toward structured versions of the SLTH more generally is a limitation of a result underlying almost all of the theoretical works on the SLTH: a theorem by Lueker on the Random Subset-Sum Problem (RSSP) .

Theorem 1 ([Lueker, 1998, Da Cunha et al., 2023]) . Let X 1 , . . . , X n be independent uniform random variables over [ -1 , 1] , and let ε ∈ (0 , 1 / 3) . There exists a universal constant C > 0 such that, if n ≥ C log(1 /ε ) , then, with probability at least 1 -ε , for all z ∈ [ -1 , 1] there exists S z ⊆ [ n ] for which

In general terms, the theorem states that given a rather small number of random variables, there is a high probability that any target value within an interval of interest can be approximated by a sum of a subset of the random variables. An important remark is that even though Theorem 1 is stated in terms of uniform random variables, it is not hard to extend it to a wide class of distributions. 2

While Theorem 1 closely matches the setup of the SLTH, it only concerns individual random variables and directly applying it to entire random structures, as needed when considering structured pruning, would require an exponential number of random variables. The recent works by Borst et al. [2022], Becchetti et al. [2022] reduced this gap by proving multidimensional versions of Theorem 1. Still, the intricate manipulation of the network parameters in proofs around the SLTH imposes restrictions that are not covered by those results.

Contributions

In this work, we overcome those obstacles and prove that random networks in a wide class of CNNs are likely to contain structured subnetworks that approximate any sufficiently smaller CNN. To the best of our knowledge, our results provide the first sub-exponential bounds around the SLTH for structured pruning of deep neural networks of any kind. More precisely,

Weprove a multidimensional version Theorem 1 that is robust to some dependencies between coordinates, which is crucial for structured pruning (Theorem 5);

We leverage this result and, by combining two types of structured sparsity (block and neuron/filter sparsity), we show that, with high probability and for a wide class of architectures, polynomially over-parameterized random networks can be pruned in a structured manner to approximate any target network (Theorem 2);

Our results cover CNNs, which generalise fully-connected networks as well as many layer types commonly used in modern architectures, such as pooling and normalisation layers;

Additionally, our pruning scheme focuses on filter pruning, which, like neuron pruning, allows for a direct reduction of the size and computational cost relative to the original CNN.

2 Related Work

SLTH Put roughly, research on the SLTH revolves around the following question:

Question. Given an error margin ε > 0 and a target neural network f target , how large must an architecture f random be to ensure that, with high probability on the sampling of its parameters, one can prune f random to obtain a subnetwork that approximates f target up to output error ε ?

Malach et al. [2020] first proved that, for dense networks with ReLU activations, it was sufficient for f random to be twice as deep and polynomially wider than f target . Orseau et al. [2020] showed that the width overhead could be greatly reduced by sampling parameters from a hyperbolic distribution. Pensia et al. [2020] improved the original result for a wide class of weight distribution, requiring only a logarithmic width overhead, which the authors proved to be asymptotically optimal. Da Cunha et al. [2022] generalised those results with optimal bounds to CNNs with non-negative inputs, which Burkholz [2022a] extended to general inputs and to residual architectures. Burkholz [2022a] also reduced the depth overhead to a single extra layer and provided results that include a whole class of activation functions. Burkholz [2022b] obtained similar improvements to dense architectures. Fischer and Burkholz [2021] modified many of the previous arguments to take into consideration networks with non-zero biases. Ferbach et al. [2022] further generalised previous results on CNNs to general equivariant networks. Diffenderfer and Kailkhura [2021] obtained similar SLTH results for binary dense neural networks within polynomial depth and width overhead, which Sreenivasan et al. [2022] improved to polylogarithmic overhead.

Structured pruning Works on structured pruning date back to the early days of the field of neural network sparsification with works such as Mozer and Smolensky [1988] and Mozer and Smolensky [1989]. Since then, a vast literature has been built around the topic, particularly for the pruning of CNNs. For a survey of structured pruning in general, we refer the reader to the associated sections of Hoefler et al. [2021], and to He and Xiao [2023] for a survey on structured pruning of CNNs.

RSSP Pensia et al. [2020] introduced the use of theoretical results on the RSSP in arguments around the SLTH, namely Lueker [1998, Corollary 3.3]. The work by Da Cunha et al. [2023] provides an alternative, simpler proof of this result. Borst et al. [2022] and Becchetti et al. [2022] proved multidimensional versions of the theorem. Theorem 5 diverges from those results in that it supports some dependencies between the entries of random vectors.

3 Preliminaries and contribution

Given n ∈ N , we denote the set { 1 , . . . , n } by [ n ] . The symbol ∗ represents the convolution operation, ⊙ represents the element-wise (Hadamard) product, and ϕ represents the ReLU activation function. The notation ∥·∥ 1 refers to the sum of the absolute values of each entry in a tensor. Similarly, ∥·∥ 2 refers to the square root of the sum of the squares of each entry in a tensor. ∥·∥ max denotes the maximum norm: the maximum among the absolute value of each entry. Sometimes we represent a tensor X ∈ R d 1 ×···× d n by the notation X = ( X i 1 ,...,i n ) i 1 ∈ [ d 1 ] ,...,i n ∈ [ d n ] . We denote the normal probability distribution with mean µ and variance σ 2 by N ( µ, σ 2 ) . We write U ∼ N d 1 ×···× d n to denote that U is a random tensor of size d 1 ×··· × d n with independent and identically distributed (i.i.d.) entries, each following N (0 , 1) . We refer to such random tensors as normal tensors . Finally, we refer to the axis of a 4-D tensor as rows , columns , channels , and kernels (a.k.a. filters), in this order.

For the sake of simplicity, we assume CNNs to be of the form N : [ -1 , 1] D × D × c 0 → R D × D × c ℓ given by

where K i ∈ R d i × d i × c i -1 × c i for i ∈ [ ℓ ] , and the convolutions have no bias and are suitably padded with zeros. Moreover, when the kernels K ( i ) are normal tensors, we say that N is a random CNN .

Our main result is the following.

Theorem 2 (Structured SLTH) . Let D,c 0 , ℓ ∈ N , and ε ∈ R > 0 . For i ∈ [ ℓ ] , let d i , c i , n i ∈ N . Let F be the class of functions from [ -1 , 1] D × D × c 0 to R D × D × c ℓ such that, for each f ∈ F

where, for i ∈ [ ℓ ] , K ( i ) ∈ R d i × d i × c i -1 × c i and ∥ ∥ K ( i ) ∥ ∥ 1 ≤ 1 .

Let also N 0 : [ -1 , 1] D × D × c 0 → R D × D × c ℓ be a 2 ℓ -layered random CNN given by

where for i ∈ [ ℓ ] the kernels L (2 i -1) and L (2 i ) are normal tensors of shape 1 × 1 × c i -1 × 2 n i c i -1 and d i × d i × 2 n i c i -1 × c i , respectively.

Finally, let G be the class of subnetworks that can be obtained by pruning contiguous blocks of parameters and removing entire filters from N 0 .

There exists a universal constant C > 0 , such that if, for i ∈ [ ℓ ] ,

then, with probability at least 1 -ε , we have that, for all f ∈ F ,

The filter removals ensured by Theorem 2 take place at layers 1 , 3 , . . . , 2 ℓ -1 and imply the removal of the corresponding channels in the next layer. The overall modification yields a CNN with kernels ˜ L (1) , . . . , ˜ L (2 ℓ ) such that, for i ∈ [ ℓ ] , the kernels ˜ L (2 i -1) and ˜ L (2 i ) have shape 1 × 1 × c i -1 × 2 c i -1 m i and d i × d i × 2 c i -1 m i × c i , respectively, where m i = √ n i / ( C 1 d i log(1 /ε )) for a universal constant C 1 . Moreover, our proof ensures that the kernels ˜ L (2 i -1) can be required to have a specific type of block sparsity: they can be structured as if pruned by 2 m i -channel-blocked masks, defined as follows.

Definition 3 ( n -channel-blocked mask) . Given a positive integer n , a binary tensor S ∈ { 0 , 1 } d × d × c × cn is called n -channel-blocked if and only if

for all i, j ∈ [ d ] , k ∈ [ c ] , and l ∈ [ cn ] .

We remark that, from a broader perspective, the central aspect of Theorem 2 is that the lower bound on the size of the random CNN depends only on the kernel sizes of the CNNs being approximated.

In subsection 4.2 we discuss the proof of Theorem 2. It requires handling subset-sum problems on multiple random variables at once (random vectors). Furthermore, the inherent parameter-sharing of CNNs creates a specific type of stochastic dependency between coordinates of the random vectors, which we capture with the following definition.

Definition 4 (NSN vector) . A d -dimensional random vector Y follows a normally-scaled normal (NSN) distribution if, for each i ∈ [ d ] , Y i = Z · Z i where Z, Z 1 , . . . , Z d are i.i.d. random variables following a standard normal distribution.

A key technical contribution of ours is a Multidimensional Random Subset Sum (MRSS) result that supports NSN vectors. In subsection 4.1 we discuss the proof of the next theorem, which follows a strategy similar to that of [Borst et al., 2022, Lemmas 1, 15].

Theorem 5 (Normally-scaled MRSS) . Let 0 < ε ≤ 1 / 4 , and let d , k , and n be positive integers such that n ≥ k 2 and k ≥ Cd 3 log d ε for some universal constant C ∈ R > 0 . Furthermore, let X 1 , . . . , X n be d -dimensional i.i.d. NSN random vectors. For any ⃗ z ∈ R d with ∥ ⃗ z ∥ 1 ≤ √ k , there exists with constant probability a subset S ⊆ [ n ] of size k such that ∥ ∥ (∑ i ∈ S X i ) -⃗ z ∥ ∥ max ≤ ε .

While it is possible to naïvely apply Theorem 1 to obtain a version of Theorem 2, doing so leads to an exponential lower bound on the required number of random vectors.

4 Analysis

In this section, after proving our MRSS result (Theorem 5), we discuss how to use it to obtain our main result on structured pruning (Theorem 2). Full proofs are deferred to Appendix B.

4.1 Multidimensional Random Subset Sum for normally-scaled normal vectors

Notation. Given a set S and a positive integer n , we denote by ( S n ) the family of all subsets of S containing exactly n elements of S . Given ε ∈ R > 0 , we define the interval I ε ( z i ) = [ z i -ε, z i + ε ] and the multi-interval I ε ( ⃗ z ) = [ ⃗ z -ε 1 , ⃗ z + ε 1 ] , where 1 = (1 , 1 , . . . , 1) ∈ R d . Moreover, for any event E , we denote its complementary event by E .

In this subsection, we estimate the probability that a set of n random vectors contains a subset that sums up to a value that is ε -close to a given target. The following definition formalises this notion.

Definition 6 (Subset-sum number) . Given (possibly random) vectors X 1 , . . . , X n and a vector ⃗ z , we define the ε -subset-sum number of X 1 , . . . , X n for ⃗ z as

where E ( ⃗ z ) S denotes the event ∥ ∥ (∑ i ∈ S X i ) -⃗ z ∥ ∥ max ≤ ε . We write simply T n,k when X 1 , . . . , X n and ⃗ z are clear from the context.

To prove Theorem 5 we use the second moment method to provide a lower bound on the probability that the subset-sum number T n,k is strictly positive, which implies that at least one subset of the random vectors suitably approximates ⃗ z . Hence, we seek a lower bound on E [ T n,k ] 2 / E [ T 2 n,k ] .

Our first lemma provides a lower bound on the probability that a sum of NSN vectors is ε -close to a target vector, through which one can infer a lower bound on E [ T n,k ] .

Lemma 7 (Sum of NSN vectors) . Let k ∈ N , ε ∈ ( 0 , 1 4 ) , ⃗ z ∈ R d such that ∥ ⃗ z ∥ 1 ≤ √ k and k ≥ 16 . Furthermore, let X 1 , . . . , X k be d -dimensional i.i.d. NSN random vectors with d ≤ k , and let c d = min { 1 d 2 , 1 16 } . It holds that

Overview of the proof. The main technical difficulty lies in the fact that the random vectors X 1 , . . . , X k are NSN vectors. Bounds can be easily derived for the case where the X i are i.i.d. normal random vectors by observing that the sum of normal random variables is also normal.

For i ∈ [ k ] , each entry of X i can be written as Z i · Z i,j where Z i and Z i,j are i.i.d. normal random variables. Conditional on Z 1 , . . . , Z k , the d entries of X = ∑ k i =1 X i are independent and distributed as N (0 , ∑ k i =1 Z 2 i ) . By noticing that Z 2 i is a chi-squared random variable and employing standard concentration inequalities (Lemma 17 in Appendix A) combined with the law of total probability, we can proceed as if the entries of X were normal, up to some correction factors.

Bounding E [ T 2 n,k ] requires handling stochastic dependencies. Thus, we estimate the joint probability that two subsets of k elements of X 1 , . . . , X n sum ε -close to the same target, taking into account that the intersection of the subsets might not be empty. The next lemma provides an upper bound on this joint probability that depends only on the size of the symmetric difference between the two subsets.

Lemma 8 (Sum of NSN vectors) . Let k, j ∈ N 0 with 1 ≤ j ≤ k Furthermore, let X 1 , . . . , X k + j be i.i.d. d -dimensional NSN random vectors with k ≥ Cd 3 log d ε . Let c d = min { 1 d 2 , 1 16 } , A = ∑ j i =1 X i , B = ∑ k i = j +1 X i , and C = ∑ k + j i = k +1 X i . 3 Then, it holds that

Overview of the proof. Let n = k + j . We exploit once more the fact that, for all i ∈ [ n ] , each entry X i can be written as Z i · Z i,j where Z i and Z i,j are i.i.d. normal random variables. Conditional on Z 1 , . . . , Z k + j , the d entries of A , B , and C are independent and distributed as N (0 , ∑ j i =1 Z 2 i ) , N (0 , ∑ k i = j +1 Z 2 i ) , and N (0 , ∑ k + j i = k +1 Z 2 i ) , respectively. Hence, by the concentration inequalities for the sum of chi-squared random variables (Lemma 17 in Appendix A) and by the law of total probability, we can focus on the term

where A i , B i , and C i indicate the i -th entries of A , B , and C , respectively.

Another concentration argument for normal random variables (Lemma 14 in Appendix A), allow us to show that

Thus, we have reduced our argument to the estimation of probabilities of independent normal random variables being close to zero.

The following lemma provides an explicit expression for the variance of the ε -subset-sum number.

Lemma 9 (Second moment of T n,k ) . Let k, n be positive integers. Let S 0 , S 1 , . . . , S k be subsets of [ n ] such that | S 0 ∩ S j | = k -j for j = 0 , 1 , . . . , k . Let S , S ′ be two random variables yielding two subsets of [ n ] of size k drawn independently and uniformly at random. Let X 1 , . . . , X n be d -dimensional i.i.d. NSN random vectors. For any ε > 0 and ⃗ z ∈ R d , the second moment of the ε -subset sum number is

where E ( ⃗ z ) S denotes the event ∥ ∥ (∑ i ∈ S X i ) -⃗ z ∥ ∥ max ≤ ε .

Proof. Let S , S ′ two random variables yielding two elements of ( [ n ] k ) drawn independently and uniformly at random. By the definition of T n,k , we have that

as Pr ( E ( ⃗ z ) S ∩ E ( ⃗ z ) S ′ ) depends only on the size of S ∩ S ′ .

Overview of the proof of Theorem 5

We use the second moment method (Lemma 15 in Appendix A) on the ε -subset-sum number T n,k of X 1 , . . . , X n . Thus, we want to lower bound the right-hand side of

Equivalently, we can provide an upper bound on the inverse, E [ T 2 n,k ] E [ T n,k ] 2 . By Lemma 9,

where S , S ′ , S i and E ( ⃗ z ) S are defined as in the statement of the lemma. Observe also that

By using the two above observations, we have

Lemma 7 provides a lower bound on the term Pr ( E ( ⃗ z ) S 0 ) while Lemma 8 gives an upper bound on the term Pr ( E ( ⃗ z ) S 0 ∩ E ( ⃗ z ) S j ) .

In the full proof, we then show that Pr( | S ∩ S ′ | ≥ k/d ) can be bounded using the Chernoff bound (Lemma 16 in Appendix A) even if we do not deal directly with binomial random variables. This allows us to discard the indices j for which Pr ( E ( ⃗ z ) S 0 ∩ E ( ⃗ z ) S j ) is large, which leads to the result after some technical manipulations.

4.2 Proving SLTH for structured pruning

To prove Theorem 2, we first show how to obtain the same approximation result for a single-layer CNN. Then, we iteratively apply the same argument for all layers of a larger CNN and show that the approximation error stays small.

We define the positive and negative parts of a tensor.

Definition 10. Given a tensor X ∈ R d 1 ×···× d n , the positive and negative parts of X are respectively defined as X + ⃗ i = X ⃗ i · 1 X ⃗ i > 0 and X -⃗ i = -X ⃗ i · 1 X ⃗ i < 0 , where ⃗ i ∈ [ d 1 ] ×···× [ d n ] points at a generic entry of X .

Approximating a single-layer CNN

We first present a preliminary lemma that shows how to prune a single-layer convolution ϕ ( V ∗ X ) in a way that dispenses us from dealing with the ReLU ϕ .

Lemma 11. Let D,d,c, n ∈ N be positive integers, V ∈ R 1 × 1 × c × 2 nc , and X ∈ R D × D × c . Let S 1 ∈ { 0 , 1 } size( V ) be a 2 n -channel-blocked mask. There exists a mask S 2 ∈ { 0 , 1 } size( V ) which only removes filters such that, for each ( i, j, k ) ∈ [ D ] × [ D ] × [2 nc ] , if ˜ S = S 1 GLYPH<18> S 2 , then

Overview of the proof. S 2 ∈ { 0 , 1 } size( V ) is such that ˜ V = V GLYPH<18> ˜ S = ( V GLYPH<18> S 1 ) GLYPH<18> S 2 contains only nonnegative edges going from each input channel t to the output channels 2( t -1) n +1 , . . . , (2 t -1) n , and only non-positive edges going from each input channel t to the output channels (2 t -1) n +1 , . . . , 2 tn . Notice that, after applying the 2 n -channel-blocked mask S 1 , the only possible non-zero entry of the k -th filter is located at its t = ⌈ k 2 n ⌉ -th channel. Hence, for each k ∈ [2 nc ] , we keep filter k if the entry of its t -th channel is non-negative and 2 t -1 = ⌈ k n ⌉ , or it is non-positive and 2 t = ⌈ k n ⌉ .

We approximate a single convolution K ∗ X by pruning a polynomially larger neural network of the form U ∗ ϕ ( V ∗ X ) exploiting only a channel-blocked mask and filter removal: this is achieved using the MRSS result (Theorem 5).

Lemma 12 (Kernel pruning) . Let D,d,c 0 , c 1 , n ∈ N be positive integers, ε ∈ ( 0 , 1 4 ) , M ∈ R > 0 , and C ∈ R > 0 be a universal constant with

Let U ∼ N d × d × 2 nc 0 × c 1 , V ∼ N 1 × 1 × c 0 × 2 nc 0 and S ∈ { 0 , 1 } size( V ) , with S being a 2 n -channelblocked mask. We define N 0 ( X ) = U ∗ ϕ ( V ∗ X ) where X ∈ R D × D × c 0 , and its pruned version N ( S ) 0 ( X ) = U ∗ ϕ (( V GLYPH<18> S ) ∗ X ) . With probability 1 -ε , for all K ∈ R d × d × c 0 × c 1 with ∥ K ∥ 1 ≤ 1 , it is possible to remove filters from N ( S ) 0 to obtain a CNN ˜ N ( S ) 0 for which

Overview of the proof. Exploiting Lemma 11, for each ( r, s, t 1 ) ∈ [ d ] × [ d ] × [ c 1 ] , one can show that

where ˜ S = S 1 GLYPH<18> S 2 , with S 1 being a 2 n -channel-blocked mask and S 2 being a mask that removes filters. Through a Chernoff bound, we show that ˜ V + 1 , 1 ,t 0 , : has at least n/ 3 non-zero entries. Up to reshaping the tensor as a one-dimensional vector, we observe that U : , : ,k, : ˜ V + 1 , 1 ,t 0 ,k is an NSN vector (Lemma 18 in Appendix A). Hence, we can apply a boosted version of the MRSS result (Corollary 19 in Appendix A) and show that, with high probability, for all target filters K with ∥ K ∥ 1 ≤ 1 , we can prune all but roughly √ n/ ( C 1 d log 1 ε ) positive entries of ˜ V + 1 , 1 ,t 0 ,k , with C 1 being a universal constant, such that ∑ k ∈ [ nc 0 ] U : , : ,k, : ˜ V + 1 , 1 ,t 0 , : approximates the channels K : , : ,t 0 , : up to error ε/ (2 d 2 c 0 c 1 ) . The same holds for ∑ k ∈ [ nc 0 ] U : , : ,k, : ˜ V -1 , 1 ,t 0 , : . This pruning can be achieved by applying a third mask S 3 that only removes filters. Through some non-trivial calculations and by applying the Tensor Convolution Inequality (Lemma 20 in Appendix A), one can combine the above results to get the thesis. Notice that the overall pruning can be represented by the mask S 1 GLYPH<18> ( S 2 GLYPH<18> S 3 ) , where S 2 GLYPH<18> S 3 is a mask that only removes filters, and S = S 1 is a 2 n -channel-blocked mask.

Remark 13 . From the proofs of Lemmas 12 and 11, we can see that the overall modification yields a pruned CNN ˆ U ∗ ϕ ( ˆ V ∗ X ) with ˆ V ∈ R 1 × 1 × c 0 × 2 mc 0 and ˆ U ∈ R d × d × 2 mc 0 × c 1 , where m = √ n/ ( C 1 d log 1 ε ) for a universal constant C 1 . Moreover, the kernel ˆ V is obtained through a 3-stage pruning process: First, we apply a 2 n -channel-blocked mask S 1 . Second, we remove filters based on entries' signs through a mask S 2 . Third, we remove filters according to the MRSS result through a mask S 3 . Masks S 2 and S 3 can be combined into a single mask S 2 GLYPH<18> S 3 that only removes filters: overall the pruning process consists of a 2 n -channel-blocked mask and filter removal, which justifies the statement of Theorem 2.

Overview of the proof of Theorem 2

We iteratively apply Lemma 12 to each layer while carefully controlling the approximation error via tools such as the Lipschitz property of ReLU and the Tensor Convolution Inequality (Lemma 20). More precisely, we show that (i) the approximation error does not increase too much at each layer; and (ii) all layer approximations can be combined to approximate the entire target network. Notice that Lemma 12 guarantees that, with high probability, we can approximate all possible target filters K (with ∥ K ∥ 1 ≤ 1 ) from the same 2-layered network. Hence, we get the result for the supremum over all possible choices of target filters.

5 Limitations and future work

In previous works [da Cunha et al., 2022, Burkholz, 2022a] the assumption that the kernel of every second layer has shape 1 × 1 × . . . is only an artifact of the proof since one can readily prune entries of an arbitrarily shaped tensor to enforce the desired shape. In our case, however, the concept of structured pruning can be quite broad, and such reshaping via pruning might not fit some sparsity patterns, depending on the context. The hypothesis on the shape can be a relevant limitation for such use cases. The constructions proposed by Burkholz [2022a,b] appear as a promising direction to overcome this limitation, with the added benefit of reducing the depth overhead.

The convolution operation commonly employed in CNNs can be cumbersome at many points of our analysis. Exploring different concepts of convolution can be an interesting path for future work as it could lead to tidier proofs and more general results. For instance, employing a 3D convolution would spare a factor c in Theorem 2.

Another limitation of our results is the restriction to ReLU as the activation function. Many previous works on the SLTH exploit the fact that ReLU satisfies the identity x = ϕ ( x ) -ϕ ( -x ) . Burkholz [2022a] leveraged that to obtain an SLTH result for CNNs with activation functions f for which f ( x ) -f ( -x ) ≈ x around the origin. Our analysis, on the other hand, does not rely on such property, so adapting the approach of [Burkholz, 2022a] to our setting is not straightforward.

Finally, we remark that the assumption of normally distributed weights might be relaxed. Borst et al. [2022] provided an MRSSP result for independent random variables whose distribution converges 'fast enough' to a Gaussian one. 4 We believe our arguments can serve well as baselines to generalise our results to support random weights distributed as such.

Acknowledgments and Disclosure of Funding

Francesco d'Amore is supported by the MUR Fare2020 Project "PAReCoDi" CUP J43C22000970001.

References

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openreview.net/forum?id=Sy8gdB9xx . 1

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Commun. ACM , 64(3):107-115, 2021. doi: 10.1145/3446776. URL https://doi.org/10.1145/3446776 . 1

Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Sensitivity and generalization in neural networks: an empirical study. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018. URL https://openreview.net/ forum?id=HJC2SzZCW . 1

Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev-Shwartz. SGD learns overparameterized networks that provably generalize on linearly separable data. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018. URL https: //openreview.net/forum?id=rJ33wwxRb . 1

Simon S. Du, Xiyu Zhai, Barnabás Póczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https:// openreview.net/forum?id=S1eK3i09YQ . 1

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR , abs/2001.08361, 2020. URL https://arxiv.org/abs/2001.08361 . 1

Russell Reed. Pruning algorithms-a survey. IEEE Trans. Neural Networks , 4(5):740-747, 1993. doi: 10.1109/72.248452. URL https://doi.org/10.1109/72.248452 . 1

Davis W. Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John V. Guttag. What is the state of neural network pruning? In Inderjit S. Dhillon, Dimitris S. Papailiopoulos, and Vivienne Sze, editors, Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020 . mlsys.org, 2020. URL https://proceedings.mlsys.org/paper_files/ paper/2020/hash/6c44dc73014d66ba49b28d483a8f8b0d-Abstract.html . 1

Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. J. Mach. Learn. Res. , 22:241:1-241:124, 2021. URL http://jmlr.org/papers/v22/21-0366.html . 1, 4

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/forum?id= rJl-b3RcF7 . 1

Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 3592-3602, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ 1113d7a76ffceca1bb350bfe145467c6-Abstract.html . 2

Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's hidden in a randomly weighted neural network? In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020 , pages 11890-11899. Computer Vision Foundation / IEEE, 2020. doi: 10.1109/CVPR42600.2020. 01191. URL https://openaccess.thecvf.com/content_CVPR_2020/html/Ramanujan_ Whats_Hidden_in_a_Randomly_Weighted_Neural_Network_CVPR_2020_paper.html . 2

Yulong Wang, Xiaolu Zhang, Lingxi Xie, Jun Zhou, Hang Su, Bo Zhang, and Xiaolin Hu. Pruning from scratch. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020 , pages 12273-12280. AAAI Press, 2020. URL https: //aaai.org/ojs/index.php/AAAI/article/view/6910 . 2

Ankit Pensia, Shashank Rajput, Alliot Nagle, Harit Vishwakarma, and Dimitris S. Papailiopoulos. Optimal lottery tickets via subset sum: Logarithmic over-parameterization is sufficient. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 1b742ae215adf18b75449c6e272fd92d-Abstract.html . 2, 4

Udo W. Pooch and Al Nieder. A survey of indexing techniques for sparse matrices. ACM Comput. Surv. , 5(2):109-133, 1973. doi: 10.1145/356616.356618. URL https://doi.org/10.1145/ 356616.356618 . 2

Adam Polyak and Lior Wolf. Channel-level acceleration of deep face representations. IEEE Access , 3:2163-2175, 2015. doi: 10.1109/ACCESS.2015.2494536. URL https://doi.org/10.1109/ ACCESS.2015.2494536 . 2

Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 14014-14024, 2019. URL https://proceedings.neurips.cc/paper/ 2019/hash/2c601ad9d2ff9bc8b282670cdd54f69f-Abstract.html . 2

Jose M. Alvarez and Mathieu Salzmann. Learning the number of neurons in deep networks. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain , pages 2262-2270, 2016. URL https://proceedings.neurips.cc/paper/2016/hash/ 6e7d2da6d3953058db75714ac400b584-Abstract.html . 2

Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/ forum?id=rJlnB3C5Ym . 2

Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. ACM J. Emerg. Technol. Comput. Syst. , 13(3):32:1-32:18, 2017. doi: 10.1145/3005348. URL https://doi.org/10.1145/3005348 . 2

Arlene Elizabeth Siswanto. Block sparsity and weight initialization in neural network pruning . PhD thesis, Massachusetts Institute of Technology, 2021. 2

Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In International Conference on Machine Learning , pages 6682-6691. PMLR, 2020. 2, 4

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In John C. Platt, Daphne Koller, Yoram Singer, and Sam T. Roweis, editors, Advances in Neural Information Processing Systems 20, Proceedings of the Twenty-First Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 3-6, 2007 , pages 1177-1184. Curran Associates, Inc., 2007. URL https://proceedings.neurips.cc/paper/2007/hash/ 013a006f03dbc5392effeb8f18fda755-Abstract.html . 2

Ali Rahimi and Benjamin Recht. Uniform approximation of functions with random bases. In 2008 46th annual allerton conference on communication, control, and computing , pages 555-561. IEEE, 2008. 2

Gilad Yehudai and Ohad Shamir. On the power and limitations of random features for understanding neural networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 6594-6604, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ 5481b2f34a74e427a2818014b8e103b0-Abstract.html . 3

George S. Lueker. Exponentially small bounds on the expected optimum of the partition and subset sum problems. Random Structures and Algorithms , 12:51-62, 1998. 3, 4

Arthur Carvalho Walraven Da Cunha, Francesco d'Amore, Frédéric Giroire, Hicham Lesfari, Emanuele Natale, and Laurent Viennot. Revisiting the Random Subset Sum Problem. In Inge Li

Gørtz, Martin Farach-Colton, Simon J. Puglisi, and Grzegorz Herman, editors, 31st Annual European Symposium on Algorithms (ESA 2023) , volume 274 of Leibniz International Proceedings in Informatics (LIPIcs) , pages 37:1-37:11, Dagstuhl, Germany, 2023. Schloss Dagstuhl - LeibnizZentrum für Informatik. ISBN 978-3-95977-295-2. doi: 10.4230/LIPIcs.ESA.2023.37. URL https://drops.dagstuhl.de/opus/volltexte/2023/18690 . 3, 4

Sander Borst, Daniel Dadush, Sophie Huiberts, and Samarth Tiwari. On the integrality gap of binary integer programs with Gaussian data. Mathematical Programming , 2022. doi: 10.1007/s10107022-01828-1. 3, 4, 5, 10

Luca Becchetti, Arthur Carvalho Walraven da Cunha, Andrea Clementi, Francesco d' Amore, Hicham Lesfari, Emanuele Natale, and Luca Trevisan. On the Multidimensional Random Subset Sum Problem. report, Inria & Université Cote d'Azur, CNRS, I3S, Sophia Antipolis, France ; Sapienza Università di Roma, Rome, Italy ; Università Bocconi, Milan, Italy ; Università di Roma Tor Vergata, Rome, Italy, 7 2022. 3, 4

Laurent Orseau, Marcus Hutter, and Omar Rivasplata. Logarithmic Pruning is All You Need. In Advances in Neural Information Processing Systems , volume 33, pages 2925-2934. Curran Associates, Inc., 2020. 4

Arthur da Cunha, Emanuele Natale, and Laurent Viennot. Proving the strong lottery ticket hypothesis for convolutional neural networks. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=Vjki79-619. 4, 10

Rebekka Burkholz. Convolutional and residual networks provably contain lottery tickets. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 2414-2433. PMLR, 2022a. URL https://proceedings.mlr.press/v162/burkholz22a.html . 4, 10

Rebekka Burkholz. Most activation functions can win the lottery without excessive depth. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 18707-18720. Curran Associates, Inc., 2022b. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 76bf7786d311217077bc8bb021946cd9-Paper-Conference.pdf . 4, 10

Jonas Fischer and Rebekka Burkholz. Towards strong pruning for lottery tickets with non-zero biases. CoRR , abs/2110.11150, 2021. URL https://arxiv.org/abs/2110.11150 . 4

Damien Ferbach, Christos Tsirigotis, Gauthier Gidel, and Avishek Joey Bose. A general framework for proving the equivariant strong lottery ticket hypothesis. CoRR , abs/2206.04270, 2022. doi: 10.48550/arXiv.2206.04270. URL https://doi.org/10.48550/arXiv.2206.04270 . 4

James Diffenderfer and Bhavya Kailkhura. Multi-prize lottery ticket hypothesis: Finding accurate binary neural networks by pruning A randomly weighted network. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. URL https://openreview.net/forum?id=U_mat0b9iv . 4

Kartik Sreenivasan, Shashank Rajput, Jy-yong Sohn, and Dimitris S. Papailiopoulos. Finding nearly everything within random binary networks. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, International Conference on Artificial Intelligence and Statistics, AISTATS 2022, 28-30 March 2022, Virtual Event , volume 151 of Proceedings of Machine Learning Research , pages 3531-3541. PMLR, 2022. URL https://proceedings.mlr.press/v151/sreenivasan22a. html . 4

Michael Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. In David S. Touretzky, editor, Advances in Neural Information Processing Systems 1, [NIPS Conference, Denver, Colorado, USA, 1988] , pages 107-115. Morgan Kaufmann, 1988. URL http://papers.nips.cc/paper/119-skeletonization-a-technique-fortrimming-the-fat-from-a-network-via-relevance-assessment . 4

Michael C Mozer and Paul Smolensky. Using relevance to reduce network size automatically. Connection Science , 1(1):3-16, 1989. 4

Yang He and Lingao Xiao. Structured pruning for deep convolutional neural networks: A survey. CoRR , abs/2303.00566, 2023. doi: 10.48550/arXiv.2303.00566. URL https://doi.org/10. 48550/arXiv.2303.00566 . 4

Devdatt P. Dubhashi and Alessandro Panconesi. Concentration of Measure for the Analysis of Randomized Algorithms . Cambridge University Press, 2009. ISBN 978-0-521-88427-3. URL http://www.cambridge.org/gb/knowledge/isbn/item2327542/ . 15

Laurent and P. Massart. Adaptive estimation of a quadratic functional by model selection. The Annals of Statistics , 28(5):1302-1338, 2000. doi: 10.1214/aos/1015957395. 15

Benjamin Doerr. Probabilistic Tools for the Analysis of Randomized Optimization Heuristics. arXiv:1801.06733, 2020. 22

Appendix

A Technical tools

A.1 Concentration inequalities

Lemma 14 (Most-probable normal interval) . Let X follow a zero-mean normal distribution with variance ϕ 2 . For any z, ε ∈ R

Proof. Let φ ( x ) denote the probability density function of X . Then,

If z -ε ≥ ε or z + ε ≤ -ε , the thesis is trivial as φ ( | x | ) decreases in x . W.l.o.g., suppose z is positive and z -ε < ε . Then, -ε < z -ε < ε < z + ε . It follows that

which is non-negative as φ ( x ) ≥ φ ( x +2 ε ) for x ≥ -ε .

Lemma 15 (Second moment method) . If Z is a non-negative random variable then

Lemma 16 (Chernoff-Hoeffding bounds [Dubhashi and Panconesi, 2009]) . Let X 1 , X 2 , . . . , X n be independent random variables such that Pr(0 ≤ X i ≤ 1) = 1 for all i ∈ [ n ] . Let X = ∑ n i =1 X i and E [ X ] = µ . Then, for any δ ∈ (0 , 1) the following holds:

Lemma 17 (Corollary of [Laurent and Massart, 2000, Lemma 1]) . Let X ∼ χ 2 d be a chi-squared random variable with d degrees of freedom. For any t > 0 , it holds that

A.2 Supporting results

Lemma 18 (NSN with positive scalar) . If a d -dimensional random vector Y is such that, for each i ∈ [ d ] , Y i = ˜ Z · ˜ Z i , where ˜ Z 1 , . . . , ˜ Z n are identically distributed random variables following a standard normal distribution, ˜ Z is a half-normal distribution, 5 and ˜ Z, ˜ Z 1 , . . . , ˜ Z n are independent, then Y follows an NSN distribution.

Proof. By Definition 4, Y is NSN if, for each i ∈ [ d ] , Y i = Z · Z i where Z, Z 1 , . . . , Z n are i.i.d. random variables following a standard normal distribution. We know that ˜ Z = | Z | , therefore ˜ Z i = sign ( Z ) sign ( Z i ) | Z i | for each i = 1 , . . . , n , where Z, Z 1 , . . . , Z n are i.i.d. standard normal

̸

implying the thesis.

Corollary 19 (of Theorem 5) . Let d, k , and n be positive integers with n ≥ C 1 dk 2 log ( 1 ε ) and k ≥ C 2 d 3 log d ε for some universal constants C 1 , C 2 ∈ R > 0 . Let X 1 , . . . , X n be d-dimensional i.i.d. NSN random vectors. For any 0 < ε ≤ 1 4 it holds

Proof. Observe that the set [ -1 , 1] d can be 'partitioned' in 1 /ε d many ∞ -norm balls of radius ε . Let s = ⌈ C 1 d log ( 1 ε )⌉ and let us partition the n vectors X 1 , . . . , X n in s disjoint sets G 1 , . . . , G s of at least k 2 vectors each. By Theorem 5, there is a constant c ∈ (0 , 1) such that for each group G i ( i ∈ [ s ] )

It follows that

where Eq. 4 holds by the union bound, Eq. 5 comes from Eq. 3 and the independence of the variables across different G i . By choosing C 1 large enough,

Lemma 20 (Tensor Convolution Inequality) . Given real tensors K and X of respective sizes d × d ′ × c 0 × c 1 and D × D ′ × c 0 , it holds

random variables, as sign ( Z ) sign ( Z i ) is independent of sign ( Z ) and of sign ( Z ) sign ( Z j ) for i = j . Then,

Proof. We have

B Omitted proofs and results

B.1 Multidimensional Random Subset Sum for normally-scaled normal vectors

Proof of Lemma 7. By Definition 4, the j -th entry of each vector X i is ( X i ) j = Z i · Z i,j where each Z i and Z i,j are i.i.d. random variables following a standard normal distribution. Let E ( ↕ ) be the event that k ( 1 -2 √ c d ) ≤ ∑ k i =1 Z 2 i ≤ k ( 1 + 2 √ c d +2 c d ) , and denote X = ∑ k i =1 X i . By the law of total probability, it holds that

As, conditional on Z 1 , . . . , Z k , the d entries of X are independent, it follows that

where the inequality in Eq. 6 holds by applying again the law of total probability.

Conditional on Z 1 , . . . , Z k , we have that ( X ) j ∼ N (0 , ∑ k i =1 Z 2 i ) . Hence,

Notice that the term ∑ i Z 2 i is a sum of chi-square random variables, for which there are known concentration bounds (Lemma 17). By definition of E ( ↕ ) and by applying Lemma 17 to estimate the term Pr ( E ( ↕ ) ) , we get that

As c d k ≥ 1 by hypotheses, 1 -2 e -c d k ≥ 1 / 4 . Then,

where we have used that ∥ ⃗ z ∥ 2 ≤ ∥ ⃗ z ∥ 1 ≤ k in Ineq. 7, that k ≥ 16 , k ≥ d , and ε < 1 / 4 in Ineq. 8, and that in Ineq. 9.

Proof of Lemma 8. Let n = k + j . Since the X i s are NSN random vectors, for each i ∈ [ n ] and j ∈ [ d ] we can write the j -th entry of X i as ( X i ) j = Z i · Z i,j where the variables in { Z i } i ∈ [ n ] and in { Z i,j } i ∈ [ n ] ,j ∈ [ d ] are i.i.d. random variables following a standard normal distribution. By the law of total probability, we have

where the latter equality holds by independence.

Then,

where the latter inequality holds by independence of A i and C i . By Lemma 14, it holds that

where the latter inequality comes from the fact that, conditional on Z 1 , . . . , Z n , we have that A i ∼ N (0 , ∑ j r =1 Z 2 r ) , B i ∼ N (0 , ∑ k r = j +1 Z 2 r ) , and C i ∼ N (0 , ∑ k + j r = k +1 Z 2 r ) for each i ∈ [ d ] . √

We now proceed similarly to the proof of Lemma 7. We denote the event that ( 1 -2 c d ) j ≤ ∑ j i =1 Z 2 i , ∑ k + j i = k +1 Z 2 i by E ( ↓ ) . Then, by Eq. 10 and the law of total probability, we have that

Eq. 11 implies that

By independence of ∑ j r =1 Z 2 r and ∑ k + j r = k +1 Z 2 r and by Lemma 17, we obtain that

Finally, for a large enough constant C , the hypothesis on k implies that k ≥ 2 d c d log πk ( 1 -2 √ c d ) 4 ε 2 . Hence,

Proof of Theorem 5. We use the second moment method (Lemma 15) on the ε -subset-sum number T n,k of X 1 , . . . , X n . Thus, we aim to provide a lower bound on the right-hand side of

Equivalently, we can provide an upper bound on the inverse E [ T 2 n,k ] E [ T n,k ] 2 . By Lemma 9

where S , S ′ , S i and E ( ⃗ z ) S are defined as in the statement of the lemma. Observe also that

By Eq.s 13 and 14, we have

As for the denominator of the second term in Eq. 15, by Lemma 7 we have

As for the numerator of the second term in Eq. 15, Lemma 8 implies that we have

By plugging Eq. 17 and Eq. 16 in Eq. 15, for j ≥ k ( 1 -1 d ) and d > 1 we can upper bound the factor Pr ( E ( ⃗ z ) S 0 ∩E ( ⃗ z ) S j ) Pr ( E ( ⃗ z ) S 0 ) 2 of the summation as follows:

As j ≥ k ( 1 -1 d ) with d > 1 , then

because ( ( 1+2 √ c d +2 c d ) ( 1 -2 √ c d )( 1 -1 d ) ) d is maximised for d = 4 . Let C ′ = 270801 . If d > 1 , by plugging Eq. 18 in Eq. 15, we have that

Notice that, if d = 1 , the same bound holds as Pr ( | S ∩ S ′ | > k d ) = 0 . We now observe that, by the law of total probability

Conditional on S = ˜ S , | S ∩ S ′ | is a hypergeometric random variable with

Since n ≥ k 2 , then k 2 n ≤ 1 . Hence, since Chernoff bounds holds for the hypergeometric distribution [Doerr, 2020, Theorem 1.10.25]

Substituting Eq. 21 in Eq. 20 we get

We can now keep bounding from above E [ T 2 n,k ] E [ T n,k ] 2 by plugging Eq. 22 in Eq. 19:

By Lemma 7, and since 1 + 2 √ c d +2 c d ≤ 2 , we have

By the hypothesis, since k ≥ Cd 3 log d ε for a large enough C , it holds that

where the latter inequality holds as ε ≤ 1 / 4 .

Plugging the inverse of the expression in Eq. 25 in Eq. 12 we obtain the thesis.

B.2 Kernel Pruning

Proof of Lemma 11. S 2 ∈ { 0 , 1 } size( V ) is such that ˜ V = V GLYPH<18> ˜ S = ( V GLYPH<18> S 1 ) GLYPH<18> S 2 contains only nonnegative edges going from each input channel t to the output channels 2( t -1) n +1 , . . . , (2 t -1) n , and only non-positive 6 edges going from each input channel t to the output channels (2 t -1) n +1 , . . . , 2 tn , while all remaining edges are set to zero. Let us define some convenient notations before proceeding

with the proofs. By [ n : m ] we denote the set { n, n +1 , . . . , m } for each pair of integers n ≤ m ∈ N . In formulas, we obtain a tensor ˜ V such that, for each ( t, k ) ∈ [ c ] × [2 nc ] :

To simplify the notation, we define the following indicator functions: for any ( t, k ) ∈ [ c ] × [2 nc ] ,

For each

Observe that only one term survives in the summation in Eq. 28, as there exists only one t ∈ [ c 0 ] such that k ∈ [(2 t -2) n +1 : 2 tn ] , say t ⋆ . Moreover, out of the four additive terms in the expression

at most one is non-zero, due to Definition 10. The ReLU cancels out negative ones, implying that Eq. 28 can be rewritten without the ReLU as a sum of only non-negative terms (out of which, at most one is non-zero) as follows

Finally, by Eq. 26 and Eq. 27, ˜ V + 1 , 1 ,t,k = 0 if k 2 n ̸∈ ( t -1 , t -1 2 ] , and ˜ V -1 , 1 ,t,k = 0 if k 2 n ∈ ( t -1 2 , t ] , which means that in Eq. 29 we can ignore the indicator functions and further simplify the expression as

Proof of Lemma 12. Adopting the same definitions as in Lemma 11 (and Eq. 26), for each ( r, s, t 1 ) ∈ [ d ] × [ d ] × [ c 1 ] we have, by Lemma 11,

We remind the reader that ˜ S = S 1 GLYPH<18> S 2 , with S 1 being a 2 n -channel-blocked mask and S 2 being a mask that removes filters. Define L + i,j,t 0 ,t 1 = ∑ k ∈ [ nc ] U i,j,k,t 1 · ˜ V + 1 , 1 ,t 0 ,k and, similarly, L -i,j,t 0 ,t 1 = ∑ k ∈ [ nc ] U i,j,k,t 1 · ˜ V -1 , 1 ,t 0 ,k . Then,

We now show that, for each t 0 ∈ [ c 0 ] , K : , : ,t 0 , : can be ε -approximated by L + : , : ,t 0 , : by suitably pruning ˜ V + , i.e. by further zeroing entries of ˜ S , and that such pruning corresponds to solving an instance of MRSS according to Theorem 5. The same reasoning applies to K -and L -.

For each t 0 ∈ [ c 0 ] , let

Observe that I ( t 0 ) + consists of the strictly positive entries of ˜ V + 1 , 1 ,t 0 , : . 7 Since the entries of V follow a standard normal distribution, each entry is positive with probability 1 / 2 . By a standard application of Chernoff bounds (Lemma 16 in Appendix A), we then have

provided that the constant C in the bound on n is sufficiently large.

For each k ∈ I ( t 0 ) + , up to reshaping the tensor as a one-dimensional vector, U : , : ,k, : · ˜ V + 1 , 1 ,t 0 ,k is an NSN vector (Definition 4) by Lemma 18 (Appendix B). Thus, for each t 0 ∈ [ c 0 ] and a sufficiently-large value of C , since we have n ≥ Cd 13 c 6 1 log 3 d 2 c 1 c 0 ε , we can apply an amplified version of Theorem 5 (i.e. Corollary 19 in Appendix B with vectors of dimension d 2 c 1 ) to show that, with probability 1 -ε 4 c 0 , for all target filter K such that ∥ K : , : ,t 0 , : ∥ 1 ≤ 1 , there exists a way to zero the entries indexed by I ( t 0 ) + of ˜ S (and thus ˜ V + 1 , 1 ,t 0 , : ), so that the pruned version of L + : , : ,t 0 , : = ∑ k ∈ [ nc 0 ] U : , : ,k, : · ˜ V + 1 , 1 ,t 0 ,k approximates K : , : ,t 0 , : . In particular, there exists another binary mask S + 3 ∈ { 0 , 1 } size ˜ S such that ˆ L + : , : ,t 0 , : = ∑ k ∈ [ nc 0 ] U : , : ,k, : · ˆ V + 1 , 1 ,t 0 ,k approximates K : , : ,t 0 , : , where ˆ V + = ˜ V + GLYPH<18> S + 3 . An analogous argument carries on for a binary mask S -3 and -ˆ L -: , : ,t 0 , : . 8 More formally, let

Then, by Corollary 19,

By the union bound, we have the following:

Since Pr (∣ ∣ ∣ I ( t 0 ) + ∣ ∣ ∣ , ∣ ∣ ∣ I ( t 0 ) -∣ ∣ ∣ > n 3 ) ≥ 1 -ε 2 , then we can remove the conditional event obtaining

To rewrite the latter in terms of the filter K and a mask ˆ S , we notice that pruning L + : , : ,t 0 , : and L -: , : ,t 0 , : separately, with two binary masks, is equivalent to say that there exists a single binary mask S 3 ∈ { 0 , 1 } size ˜ S such that, ˆ L : , : ,t 0 , : can be written as ˆ L : , : ,t 0 , : = ∑ k ∈ [ nc 0 ] U : , : ,k, : · ˆ V 1 , 1 ,t 0 ,k , where ˆ V = ˜ V GLYPH<18> S 3 . Ineq. 31 implies that, with probability 1 -ε , for all target filters K with ∥ K ∥ 1 ≤ 1 such S 3 exists and hence,

Let ˆ S = ˜ S GLYPH<18> S 3 : then ˆ S = S 1 GLYPH<18> S 2 GLYPH<18> S 3 is the combination of a 2 n -channel-blocked mask S = S 1 and a mask S 2 GLYPH<18> S 3 that only removes filters.

Also, whenever such binary mask ˆ S exists, whe have that

where the latter holds by Lemma 11. 9 Then, by the distributive property of the convolution and the triangle inequality,

One can now apply the Tensor Convolution Inequality (Lemma 20) and obtain

Now, observing that the number of entries of the two tensors in the expression above is d 2 c 1 c 0 , and using Ineq. 32, we get that

proving the thesis.

Proof of Theorem 2. To bound the error propagation across layers, we define the layers' outputs

Notice that X ( ℓ ) is the output of the target function, i.e., f ( X ) = X ( ℓ ) .

For brevity's sake, given masks S (2 i -1) ∈ { 0 , 1 } size ( L (2 i -1) ) for each i = 1 , . . . , ℓ , let us denote

We now show that the random network approximates the target one after being pruned only at layers L (2 i -1) for all i = 1 , . . . , ℓ , by masks S (2 i -1) which are the composition of a 2 n 2 i -1 -channelblocked mask and a mask that only removes filters. The key idea is to iteratively applying Lemma 12. Since the ReLU function is 1-Lipschitz, for all i it holds

The key step of the proof is that, for each layer i , since n i ≥ Cd 13 c 6 i log 3 d 2 c i c i -1 ℓ ε for a suitable constant C , we can apply Lemma 12 to get that, with probability at least 1 -ε 2 ℓ , for all i and for all choices of target filters K ( i ) ∈ R d i × d i × c i -1 × c i it holds

Hence, combining Eq. 35 and Eq. 36 we get that, with probability at least 1 -ε 2 ℓ , for all i and all choices of target filters K ( i ) ∈ R d i × d i × c i -1 × c i with ∥ ∥ K ( i ) ∥ ∥ 1 ≤ 1 ,

By a union bound, with probability at least 1 -ε , we get that Eq. 37 holds for all layers i and all choices of target filters K ( i ) ∈ R d i × d i × c i -1 × c i .

Analogously, we can define the pruned layers' outputs

Notice that ˜ X ( ℓ ) is the output of the pruned network, i.e. N ( S (1) ,...,S (2 ℓ ) ) 0 ( X ) = ˜ X ( ℓ ) .

By the same reasoning employed to derive Eq. 36 and Eq. 37 we have that, with probability 1 -ε , for all layers i and all choices of target filters K ( i ) ∈ R d i × d i × c i -1 × c i , the output of all pruned layers satisfies

Moreover, for each 1 ≤ i ≤ ℓ -1 , by the triangle inequality and by Eq. 39,

By the Lipschitz property of ϕ and Lemma 20

By unrolling the recurrence, we get that, with probability 1 -ε , for all choices of target filters K ( i ) ∈ R d i × d i × c i -1 × c i ,

Thus, combining Eq. 39 and Eq. 40, with probability 1 -ε we get that, for each i ∈ [ ℓ ] and for all choices of target filters K ( i ) ∈ R d i × d i × c i -1 × c i ,

We then see that with probability 1 -ε , for 1 ≤ i ≤ ℓ and for all choices of target filters K ( i ) ∈ R d i × d i × c i -1 × c i , by Eq. 33 and Eq. 38, and by the triangle inequality,

Again by the 1-Lipschitz property of the ReLU activation function, and by the distributive property of the convolution operation,

Lemma 20 and the hypothesis ∥ ∥ K ( ℓ ) ∥ ∥ 1 ≤ 1 imply that

Hence, yielding the thesis.

Now, we first apply Eq. 41 and then we unroll the recurrence for all layers (as, with probability 1 -ε , Eq. 41 holds for all layers and for all choices of target filters K ( i ) ∈ R d i × d i × c i -1 × c i ), obtaining

By summing the geometric series and observing that ε < 1 , we conclude that

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe central theme of this paper is the Strong Lottery Ticket Hypothesis (SLTH), which conjectures that « randomly initialised networks contain sparse subnetworks […] that perform well without any training » (lines 26-30). Pruning large networks in order to obtain those « strong lottery tickets » would be a way to limit the computational costs associated to those over-parameterised networks and facilitate their deployment in real-world applications. The SLTH has already been theoretically verified for various classes of networks, but never while restraining the way the pruning must be done in order to obtain those « strong lottery tickets ». The authors aim to theoretically verify that « strong lottery tickets » can be obtained via structural pruning of a certain class of convolutional neural networks. The way they do it is by proving that « networks in a wide class of CNNs are likely to contain structured subnetworks that approximate any sufficiently smaller CNN in the class » (lines 77-79), based on the assumption that their exists a «  sufficiently smaller CNN in the class » that performs well.\n\nSTRENGTHS:\n1. The manipulated concepts, such as the LTH, the SLTH, or the RSSP are well-explained.\n\n2. A clear understanding of the many proofs can be obtained by reading Section 4.\n\nWEAKNESSES:\n_Major_\n\n1. The proof ideas are not that novel. Many cited works from the Related Work section use the same technique in order to obtain similar results, and Theorems 3 and 5 are the only contributions of the paper (lines 81-85; see Question 3).\n\n2. The goal is to demonstrate that winning tickets from the SLTH can be obtained by removing channels from a deep and wide CNN. The underlying assumption here being that there exists a CNN having the same restrictions as discussed in the article AND that is significantly smaller than an over-parameterised one is in fact capable of obtaining good results on given tasks. Considering those limitations (no classification head, that is, no fully-connected layer; no pooling layer; no batch norm layer; ReLU activations only; etc.), such an assumption is not trivial and questions the whole process.\n\n3. The justification for the approach is that for over-parameterised networks, « [their] associated computational cost limits both the progress of [the neural networks in general] and their deployment in real-world applications. » (lines 19-20). Therefore, the justification is on a practical level. But considering the size of the kernels needed for the network to be pruned (with $n_i \\geq Cd^{12}c^6_i ...$, line 139), from the practical point of view, it is virtually impossible to encode such a network even before considering pruning it (which would then be a fastidious process and would imperatively be done in an efficient manner, which is not discussed in the article). \n\n4. The end-goal is to obtain a relatively small network, but taking $n_i \\geq Cd^{12}c^6_i ...$ (line 139), and plugging it into $m_i = \\sqrt{n_i / (C₁ log \\frac{1}{e})}$ (line 150) leads to $m_i \\geq d^6c_i^3...$, which do not seem to produce a network that would be smaller.\n\n\n\n_Typos / Minor_\n\n1. The way the citations are made is inconsistent throughout the paper; their are both appropriate (« The constructions proposed by Burkholz [2022a,b] », line 290) and inappropriate (« both in the past Reed [1993] and in the present [...]. », line 21) inline text citation format.\n\n2. Line 58 : or block sparsity Siswanto [2021] (Figure 1$\\underline{c}$).\n\n3. Some statements are not supported; for example : « Much of the success of deep learning techniques relies on extreme over-parameterisation. » (line 17).\n\nQUESTIONS:\n1. Exactly which lemmas are original and which have been taken from the literature?\n\n2. Is there a reason why you specifically chose having filters (or channels) being removed as structured pruning technique?\n\n3. The third contribution states that « Additionally, our pruning scheme focuses on filter pruning, which, like neuron pruning, allows for a direct reduction of the size of the original CNN » (lines 86-87) (which is not a contribution, but an argument assessing the relevance of the work). It is also stated that « through unstructured pruning [...] one can usually reach 95% sparsity without accuracy loss » (lines 54-55). One could argue that by reaching 95% sparsity, the size of the original network is likely to diminish too. From that perspective, why would channel-pruning still be more desirable?\n\nLIMITATIONS:\nLimitations have been discussed properly, apart from what has been raised in the Weaknesses part of the review.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tries to discuss the Strong Lottery Ticket Hypothesis (SLTH) in the context of structured pruning in convolutional neural networks (CNNs). The SLTH suggests that randomly-initialized neural networks possess subnetworks that can perform well without any training. While unstructured pruning has been extensively studied, the paper explores the less explored area of structured pruning, which can offer significant computational and memory efficiency gains. The authors overcome the limitations of existing mathematical tools by leveraging recent advancements in the multidimensional generalization of the Random Subset-Sum Problem. By introducing a variant that accounts for stochastic dependencies in structured pruning, the paper proves the existence of structured subnetworks in random CNNs that can approximate smaller networks. This theoretical contribution extends the understanding of the role of overparameterization in deep learning and opens new avenues for further research on the SLTH for structured pruning.\n\nSTRENGTHS:\n- The paper addresses an important research gap by focusing on structured pruning in the context of the Strong Lottery Ticket Hypothesis. This expands the understanding of lottery ticket mechanisms beyond unstructured pruning.\n- The authors make use of recent advancements in the multidimensional generalization of the Random Subset-Sum Problem, demonstrating their ability to leverage and adapt existing mathematical tools to overcome limitations in formal analyses of the SLTH.\n\nWEAKNESSES:\nThe SLTH theory is important in both the pruning and neural network architecture search domains. However, I'm not familiar with the specific analysis methods proposed in this paper, so I can't provide professional advice.\n\n- Line 118, \"Preliminaries and contribution\" -> \"Preliminaries\"?\n- In this paper, if the convolution kernel $K$ relies on the normality assumption, as mentioned in Line 131, I suggest the author can cite the Convolution Weight Distribution Assumption discussed in [1].\n- If possible, I recommend that the author use more intuitive language when writing Section 4 and the introduction. This would help researchers in the pruning and neural network architecture search fields to better understand the contributions of this paper.\n\n[1] Huang Z, Shao W, Wang X, et al. Rethinking the pruning criteria for convolutional neural network[J]. Advances in Neural Information Processing Systems (NeurIPS), 2021, 34: 16305-16318.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nThe limitations of the proposed analysis have been discussed in this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates a structured version of strong lottery ticket hypothesis (SLTH) for CNNs, which is important for its computational efficiency against the standard unstructured counterpart. For this purpose, they prove a variant of subset-sum lemma for their situation. Using their subset-sum lemma and splitting lemma for ReLU activations, SLTH for CNNs is proven by combining block-pattern pruning and neuron pruning under the assumption that the network size is polynomially larger than a target network.\n\nSTRENGTHS:\n1. SLTH by structured pruning is an important topic in terms of applicability and computational efficiency of SLTH. The authors are the first to prove the structured version of SLTH for CNNs.\n\n2. They extend the subset-sum lemma, which is a key lemma for SLTH with logarithmic over-parameterization as shown in Pensia et al. 2020, for their purpose.\n\nWEAKNESSES:\nPresentation:\n\n1. Although the overview of their proof is straightforward, the presented proof does not provide any intuition for the evaluation of the required network size.\n\n2. No experiments are provided. Since the structured version of SLTH itself has not yet observed empirically even for MLPs or CNNs, some experiments would be helpful for supporting their claim.\n\n3. They claimed that `To the best of our knowledge, this is the first result around the SLTH for structured pruning of neural networks of any kind` in Introduction, but the previous work by Malach et al. 2020 already provided results on the equivalence between SLTH by neuron pruning and random feature NNs in the 2-layer case.\n\nResults:\n\n4. Even though they use a variant of subset-sum lemma following the approach by Pensia et al. 2020 for logarithmic over-parameterization, the required width in this paper is still polynomially larger than the target one. In this point, I wonder that, If one allows such polynomially large width, the structured version of SLTH might be easily proven by simply extending the previous discussion in Malach et al. or Pensia et al., at least for MLPs. Also I wonder that whether their claim is still valid in practice due to such polynomial over-parameterization. Nevertheless, the first attempt to prove the structured version of SLTH is worthful in itself. I hope the authors would emphasise which technique in their proof is important for either structured pruning or handling CNNs.\n\nQUESTIONS:\n1. Can the proof also work for the easier case of MLPs instead of CNNs? Is the variant of subset-sum lemma in this paper also needed even for MLPs? If so, providing some intuition for MLP case may be helpful to understand the structured SLTH. If not, discussing why your version of subset-sum lemma is needed for CNNs would be helpful for readers.\n\n2. Is the structured version of SLTH with polynomial over-parameterization really practical? I wonder if we can find such a structured subnetwork in randomly initialized CNNs (or MLPs) in a similar way to Ramanujan et al. 2020, where the unstructured version of SLTH is actually observed.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors show that structured Strong Lottery tickets are contained within convolutional neural networks.\nIn order to construct this proof, the authors propose the Multidimensional Random Subset Sum lemma which approximates a target vector by using a set of random vectors which are normally scaled normally distributed.\nThis results builds on the Subset Sum approximation by Lueker, which is commonly used in proving the Strong Lottery Ticket hypothesis for standard architectures.\nThe authors further leverage this lemma to approximate a target function with a structurally pruned CNN which is twice as deep and wider by a factor.\n\nOverall this paper provides an important existence proof for strong lottery tickets in structurally pruned CNNs as a first step in trying to understand structured pruning theoretically.\n\nSTRENGTHS:\n1. The authors for the first time have proven the existence of SLTs in structurally pruned networks giving hope that a computationally efficient and trainable SLT might exist that can be leveraged for efficienct on existing hardware (contrary to current empirical results in literature which are worse than unstructured pruning).\n2. The authors also extend the result of Lueker for random vectors.\n3. The paper is clearly written.\n\nWEAKNESSES:\n1. The bounds provided by the authors still require twice the depth in order to approximate a target function. Burkholz et al have proposed a way to use only L + 1 depth to approximate a target function. Is this still possible with structural pruning in CNNs?\n2. As mentioned by the authors, the proposed construction looks at structured pruning where entire channels are pruned away. Would block sparsity (eg; N:M sparsity) allow for a better Subset Sum bound potentially i.e. are there specific sparsity patterns which can allow sparser constructions. This might guide us towards a identifying layerwise sparsity ratios for training sparse networks.\n\nQUESTIONS:\n1. The construction requires a layer of width $n c_0$ such that after pruning the width is $m c_0$. These factors would effectively determine the overall sparsity a network can achieve. Would this translate to structured pruning results we observe in practice?\n\nLIMITATIONS:\nThe authors provide their construction for ReLU networks. However, Burkholz et al. [1] show existence results for different activation functions.\nDo the proposed existence results also transfer to other activations?\n\n[1] Burkholz, Rebekka. \"Most activation functions can win the lottery without excessive depth.\" NeurIPS 2022", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends analytical work on the strong lottery ticket hypothesis from unstructured to structured pruning, and shows the applicability to Convolutional Neural Networks.\n\nSTRENGTHS:\nI'm not an expert in this domain, but both the contribution looks strong to me.\n\nClarity:\nThe introduction is extensive and appropriately places the current paper into the existing research landscape. I could follow well, even though I'm not directly in this domain. In the analysis section, proofs and other statements are appropriately accompanied by explanatory statements, which makes following them easier.\n\nQuality:\nThe paper is of high quality. It advances a subfield that is actively being researched.\n\nOriginality:\nTo my knowledge, the work is original. It extends various previous work in a non-trivial manner. The authors place the contributions appropriately among previous results and reference related work.\n\nSignificance:\nI really cannot accurately estimate the significance of this work. The overall question around whether structured sparsity in CNNs can induce strong lottery tickets is certainly significant. I'm not sure how much the used structures in this work (for example, every other layer being a 1x1 layer) limit the significance of the results.\n\nWEAKNESSES:\n- I would have liked to see at least a small experimental section, as the discussed topic lends itself well to empirical evaluation.\n- As I said above, just reading the paper gives me very little insight into the meaning and significance of these results, which could be discussed more extensively.\n\nQUESTIONS:\n- In Lemma 8, you deal with variables X_1 through X_{k+j}, but then only condition on Z_1 through Z_k in the text, and Z_1 through Z_n in the statements. It's entirely possible that I misunderstand, but is all of this correct? For example, n is never even introduced as a number.\n- I don't quite understand why the n-channel-blocked-masks are needed. In definition 2 you mention that this defines the structured sparsity of the networks, but doesn't the removal of entire filters by itself already constitute structured sparsity?\n- Related to above: In Lemma 12 you describe the network including the channel-blocked-masks as the \"pruned\" network, but from this we still need to remove filters to obtain the smaller network that appropriately approximates K. So which action exactly constitutes the pruning?\n- In Lemma 11, you state that V * S-tilde \"contains only non-negative edges going from each input channel t [...]\". How can that be? V has arbitrary real numbers, and S-tilde is a binary mask. Maybe I've missed something here, I could see that statement be made when including the ReLU in some way, but I'm a bit lost the way it is now.\n\nLIMITATIONS:\nThe authors have appropriately addressed limitations in Section 5. Questions on societal impact do not apply here.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}