SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Arthur da Cunha
Université Côte d'Azur, Inria, CNRS, I3S
Aarhus University
Aarhus, Denmark
dac@cs.au.dk
Francesco d'Amore
francesco.damore@aalto.fi
Emanuele Natale
emanuele.natale@inria.fr
The Strong Lottery Ticket Hypothesis (SLTH) states that randomly-initialised neural networks likely contain subnetworks that perform well without any training. Although unstructured pruning has been extensively studied in this context, its structured counterpart, which can deliver significant computational and memory efficiency gains, has been largely unexplored. One of the main reasons for this gap is the limitations of the underlying mathematical tools used in formal analyses of the SLTH. In this paper, we overcome these limitations: we leverage recent advances in the multidimensional generalisation of the Random Subset-Sum Problem and obtain a variant that admits the stochastic dependencies that arise when addressing structured pruning in the SLTH. We apply this result to prove, for a wide class of random Convolutional Neural Networks, the existence of structured subnetworks that can approximate any sufficiently smaller network.
This result provides the first sub-exponential bound around the SLTH for structured pruning, opening up new avenues for further research on the hypothesis and contributing to the understanding of the role of over-parameterization in deep learning.
Much of the success of deep learning techniques relies on extreme over-parameterization [Zhang et al., 2017, 2021, Novak et al., 2018, Brutzkus et al., 2018, Du et al., 2019, Kaplan et al., 2020]. While such excess of parameters has allowed neural networks to become the state of the art in many tasks, the associated computational cost limits both the progress of those techniques and their deployment in real-world applications. This limitation motivated the development of methods for reducing the number of parameters of neural networks; both in the past [Reed, 1993] and in the present [Blalock et al., 2020, Hoefler et al., 2021].
Although pruning methods have traditionally targeted reducing the size of networks for inference purposes, recent works have indicated that they can also be used to reduce parameter counts during training or even at initialisation without sacrificing model accuracy. In particular, Frankle and Carbin [2019] proposed the Lottery Ticket Hypothesis (LTH), which conjectures that randomly-initialised
Figure 1: Illustration of neuron pruning. The left side shows the effect of pruning of neurons in the weight-matrix of a fully-connected layer. The rows in white correspond to neurons pruned in the associated layer while the columns in white represent the effect of removing neurons from the previous layers. On the right, we allude to the possibility of collapsing the pruned matrix into a smaller, dense one.
networks contain sparse subnetworks that can be trained and reach the performance of the fully-trained original network. Empirical investigations on the LTH [Zhou et al., 2019, Ramanujan et al., 2020, Wang et al., 2020] pointed towards an even more striking phenomenon: the existence of subnetworks that perform well without any training. This conjecture was named the Strong Lottery Ticket Hypothesis (SLTH) by Pensia et al. [2020].
While the SLTH has been proved for many different classes of neural networks (see Section 2), those works are restricted to unstructured pruning, where the subnetworks are obtained by freely removing individual parameters from the original network. However, this lack of structure can significantly reduce the gains that sparsity can bring, both in terms of memory and computational efficiency. The possibility of removing parameters at arbitrary points of the network implies the need to store the indices of the remaining non-zero parameters, which can become a significant overhead with its own research challenges [Pooch and Nieder, 1973]. Moreover, the theoretical computational gains of unstructured sparsity can also be difficult to realise in standard hardware, which is optimised for dense operations. Most notably, the irregularity of the memory access patterns can lead to both data and instruction cache misses, significantly reducing the performance of the pruned network.
The limitations of parameter-level pruning have motivated extensive research on structured pruning, which constrain the sparsity patterns to reduce the complexity of parameter indexation and, more generally, to make the processing of the pruned network more efficient. A simple example of structured pruning is neuron pruning of fully-connected layers: deletions in the weight matrix are constrained to the level of whole rows/columns. As illustrated by Figure 1, pruning under this constraint produces a smaller network that is still dense, directly reducing the computational costs without any need for extra memory to store indices. Similarly, deleting entire filters in Convolutional Neural Networks (CNNs) [Polyak and Wolf, 2015] or "heads" in attention-based architectures [Michel et al., 2019] also produces direct reductions in computational costs.
It is important to note that structured pruning is a restriction of unstructured pruning so, theoretically, the former is bound to perform at most as well as the latter. For example, by deleting whole neurons one can remove about  $70\%$  of the weights in dense networks without significantly affecting accuracy, while through unstructured pruning one can usually reach  $95\%$  sparsity without accuracy loss [Alvarez and Salzmann, 2016, Liu et al., 2019]. In practice, however, the computational advantage of structured pruning can offset this difference: a suitably structured sparse network can be more efficient than an even sparser one that lacks structure. This trade-off between sparsity and actual efficiency has motivated the study of less coarse sparsity patterns since weaker structural constraints such as strided sparsity [Anwar et al., 2017] (Figure 2b) or block sparsity [Siswanto, 2021] (Figure 2c) are already sufficient to deliver the bulk of the computational gains that structured can offer.
Despite its benefits, structured pruning has received little attention in the context of the SLTH. In fact, the work that first proved a version of the SLTH, Malach et al. [2020], has remained the only one to study this scenario, to the best of our knowledge. Moreover, it brings a negative result: the authors prove that removing neurons from a randomly-initialised shallow neural network (a single hidden layer) is equivalent to the random features model<sup>1</sup> (e.g., Rahimi and Recht [2007, 2008]),
Unstructured sparsity
(a) No pattern.
Figure 2: Examples of different pruning patterns.
Structured sparsity
(b) Strided pattern.
(c) Block pattern.
which cannot efficiently approximate even a single ReLU neuron [Yehudai and Shamir, 2019]. As most proofs around the SLTH involve pruning a shallow random network to build the desired approximations, those results show that the SLTH can be challenging to tackle when restricted to neuron pruning.
In addition, we believe that a factor hindering progress toward structured versions of the SLTH more generally is a limitation of a result underlying almost all of the theoretical works on the SLTH: a theorem by Lueker on the Random Subset-Sum Problem (RSSP).
Theorem 1 ([Lueker, 1998, Da Cunha et al., 2023]). Let  $X_{1},\ldots ,X_{n}$  be independent uniform random variables over  $[-1,1]$ , and let  $\varepsilon \in (0,1 / 3)$ . There exists a universal constant  $C > 0$  such that, if  $n\geq C\log (1 / \varepsilon)$ , then, with probability at least  $1 - \varepsilon$ , for all  $z\in [-1,1]$  there exists  $S_{z}\subseteq [n]$  for which
$$
\left| z - \sum_ {i \in S _ {z}} X _ {i} \right| \leq \varepsilon .
$$
In general terms, the theorem states that given a rather small number of random variables, there is a high probability that any target value within an interval of interest can be approximated by a sum of a subset of the random variables. An important remark is that even though Theorem 1 is stated in terms of uniform random variables, it is not hard to extend it to a wide class of distributions.2
While Theorem 1 closely matches the setup of the SLTH, it only concerns individual random variables and directly applying it to entire random structures, as needed when considering structured pruning, would require an exponential number of random variables. The recent works by Borst et al. [2022], Becchetti et al. [2022] reduced this gap by proving multidimensional versions of Theorem 1. Still, the intricate manipulation of the network parameters in proofs around the SLTH imposes restrictions that are not covered by those results.
In this work, we overcome those obstacles and prove that random networks in a wide class of CNNs are likely to contain structured subnetworks that approximate any sufficiently smaller CNN. To the best of our knowledge, our results provide the first sub-exponential bounds around the SLTH for structured pruning of deep neural networks of any kind. More precisely,
- We prove a multidimensional version Theorem 1 that is robust to some dependencies between coordinates, which is crucial for structured pruning (Theorem 5);
- We leverage this result and, by combining two types of structured sparsity (block and neuron/filter sparsity), we show that, with high probability and for a wide class of architectures, polynomially over-parameterized random networks can be pruned in a structured manner to approximate any target network (Theorem 2);
<sup>1</sup>In the random features model, one can only train the last layer of the network.
${}^{2}$  Distributions whose probability density function  $f$  satisfies  $f\left( x\right)  \geq  b$  for all  $x \in  \left\lbrack  {-a,a}\right\rbrack$  ,for some constants  $a,b > 0$  (see Lueker [1998, Corollary 3.3]).
- Our results cover CNNs, which generalise fully-connected networks as well as many layer types commonly used in modern architectures, such as pooling and normalisation layers;
- Additionally, our pruning scheme focuses on filter pruning, which, like neuron pruning, allows for a direct reduction of the size and computational cost relative to the original CNN.
SLTH Put roughly, research on the SLTH revolves around the following question:
Question. Given an error margin  $\varepsilon > 0$  and a target neural network  $f_{\mathrm{target}}$ , how large must an architecture  $f_{\mathrm{random}}$  be to ensure that, with high probability on the sampling of its parameters, one can prune  $f_{\mathrm{random}}$  to obtain a subnetwork that approximates  $f_{\mathrm{target}}$  up to output error  $\varepsilon$ ?
Malach et al. [2020] first proved that, for dense networks with ReLU activations, it was sufficient for  $f_{\text{random}}$  to be twice as deep and polynomially wider than  $f_{\text{target}}$ . Orseau et al. [2020] showed that the width overhead could be greatly reduced by sampling parameters from a hyperbolic distribution. Pensia et al. [2020] improved the original result for a wide class of weight distribution, requiring only a logarithmic width overhead, which the authors proved to be asymptotically optimal. Da Cunha et al. [2022] generalised those results with optimal bounds to CNNs with non-negative inputs, which Burkholz [2022a] extended to general inputs and to residual architectures. Burkholz [2022a] also reduced the depth overhead to a single extra layer and provided results that include a whole class of activation functions. Burkholz [2022b] obtained similar improvements to dense architectures. Fischer and Burkholz [2021] modified many of the previous arguments to take into consideration networks with non-zero biases. Ferbach et al. [2022] further generalised previous results on CNNs to general equivariant networks. Diffenderfer and Kailkhura [2021] obtained similar SLTH results for binary dense neural networks within polynomial depth and width overhead, which Sreenivasan et al. [2022] improved to polylogarithmic overhead.
Structured pruning Works on structured pruning date back to the early days of the field of neural network sparsification with works such as Mozer and Smolensky [1988] and Mozer and Smolensky [1989]. Since then, a vast literature has been built around the topic, particularly for the pruning of CNNs. For a survey of structured pruning in general, we refer the reader to the associated sections of Hoefler et al. [2021], and to He and Xiao [2023] for a survey on structured pruning of CNNs.
RSSP Pensia et al. [2020] introduced the use of theoretical results on the RSSP in arguments around the SLTH, namely Lueker [1998, Corollary 3.3]. The work by Da Cunha et al. [2023] provides an alternative, simpler proof of this result. Borst et al. [2022] and Becchetti et al. [2022] proved multidimensional versions of the theorem. Theorem 5 diverges from those results in that it supports some dependencies between the entries of random vectors.
3 Preliminaries and contribution
Given  $n \in \mathbb{N}$ , we denote the set  $\{1, \dots, n\}$  by  $[n]$ . The symbol  $*$  represents the convolution operation,  $\odot$  represents the element-wise (Hadamard) product, and  $\phi$  represents the ReLU activation function. The notation  $\| \cdot \|_1$  refers to the sum of the absolute values of each entry in a tensor. Similarly,  $\| \cdot \|_2$  refers to the square root of the sum of the squares of each entry in a tensor.  $\| \cdot \|_{\max}$  denotes the maximum norm: the maximum among the absolute value of each entry. Sometimes we represent a tensor  $X \in \mathbb{R}^{d_1 \times \dots \times d_n}$  by the notation  $X = (X_{i_1, \dots, i_n})_{i_1 \in [d_1], \dots, i_n \in [d_n]}$ . We denote the normal probability distribution with mean  $\mu$  and variance  $\sigma^2$  by  $\mathcal{N}(\mu, \sigma^2)$ . We write  $U \sim \mathcal{N}^{d_1 \times \dots \times d_n}$  to denote that  $U$  is a random tensor of size  $d_1 \times \dots \times d_n$  with independent and identically distributed (i.i.d.) entries, each following  $\mathcal{N}(0, 1)$ . We refer to such random tensors as normal tensors. Finally, we refer to the axis of a 4-D tensor as rows, columns, channels, and kernels (a.k.a. filters), in this order.
For the sake of simplicity, we assume CNNs to be of the form  $N\colon [-1,1]^{D\times D\times c_0}\to \mathbb{R}^{D\times D\times c_\ell}$  given by
$$
N (X) = K ^ {\ell} * \phi \left(K ^ {\ell - 1} * \dots \phi \left(K ^ {1} * X\right)\right),
$$
where  $K^i \in \mathbb{R}^{d_i \times d_i \times c_{i-1} \times c_i}$  for  $i \in [\ell]$ , and the convolutions have no bias and are suitably padded with zeros. Moreover, when the kernels  $K^{(i)}$  are normal tensors, we say that  $N$  is a random CNN.
Theorem 2 (Structured SLTH). Let  $D, c_0, \ell \in \mathbb{N}$ , and  $\varepsilon \in \mathbb{R}_{>0}$ . For  $i \in [\ell]$ , let  $d_i, c_i, n_i \in \mathbb{N}$ . Let  $\mathcal{F}$  be the class of functions from  $[-1,1]^{D \times D \times c_0}$  to  $\mathbb{R}^{D \times D \times c_\ell}$  such that, for each  $f \in \mathcal{F}$
$$
f (X) = K ^ {(\ell)} * \phi \left(K ^ {(\ell - 1)} * \dots \phi \left(K ^ {(1)} * X\right)\right), \tag {1}
$$
where, for  $i\in [\ell ]$ $K^{(i)}\in \mathbb{R}^{d_i\times d_i\times c_{i - 1}\times c_i}$  and  $\left\| K^{(i)}\right\| _1\leq 1$
Let also  $N_0\colon [-1,1]^{D\times D\times c_0}\to \mathbb{R}^{D\times D\times c_\ell}$  be a 2-layered random CNN given by
$$
N _ {0} (X) = L ^ {(2 \ell)} * \phi \left(L ^ {(2 \ell - 1)} * \dots \phi \left(L ^ {(1)} * X\right)\right), \tag {2}
$$
where for  $i \in [\ell]$  the kernels  $L^{(2i-1)}$  and  $L^{(2i)}$  are normal tensors of shape  $1 \times 1 \times c_{i-1} \times 2n_i c_{i-1}$  and  $d_i \times d_i \times 2n_i c_{i-1} \times c_i$ , respectively.
Finally, let  $\mathcal{G}$  be the class of subnetworks that can be obtained by pruning contiguous blocks of parameters and removing entire filters from  $N_0$ .
There exists a universal constant  $C > 0$ , such that if, for  $i \in [\ell]$
$$
n _ {i} \geq C d _ {i} ^ {1 3} c _ {i} ^ {6} \log^ {3} \frac {d _ {i} ^ {2} c _ {i} c _ {i - 1} \ell}{\varepsilon},
$$
then, with probability at least  $1 - \varepsilon$ , we have that, for all  $f \in \mathcal{F}$
$$
\sup_{X\in [-1,1]^{D\times D\times c_{0}}}\min_{g\in \mathcal{G}}\| f(X) - g(X)\|_{\max}\leq \varepsilon .
$$
The filter removals ensured by Theorem 2 take place at layers  $1,3,\ldots ,2\ell -1$  and imply the removal of the corresponding channels in the next layer. The overall modification yields a CNN with kernels  $\tilde{L}^{(1)},\dots ,\tilde{L}^{(2\ell)}$  such that, for  $i\in [\ell ]$ , the kernels  $\tilde{L}^{(2i - 1)}$  and  $\tilde{L}^{(2i)}$  have shape  $1\times 1\times c_{i - 1}\times 2c_{i - 1}m_i$  and  $d_{i}\times d_{i}\times 2c_{i - 1}m_{i}\times c_{i}$ , respectively, where  $m_{i} = \sqrt{n_{i} / (C_{1}d_{i}\log(1 / \varepsilon))}$  for a universal constant  $C_1$ . Moreover, our proof ensures that the kernels  $\tilde{L}^{(2i - 1)}$  can be required to have a specific type of block sparsity: they can be structured as if pruned by  $2m_{i}$ -channel-blocked masks, defined as follows.
Definition 3 ( $n$ -channel-blocked mask). Given a positive integer  $n$ , a binary tensor  $S \in \{0,1\}^{d \times d \times c \times cn}$  is called  $n$ -channel-blocked if and only if
$$
S _ {i, j, k, l} = \left\{\begin{array}{l l}1&\text {i f} \left\lceil \frac {l}{n} \right\rceil = k,\\0&\text {o t h e r w i s e},\end{array}\right.
$$
for all  $i,j\in [d],k\in [c]$  , and  $l\in [cn]$
We remark that, from a broader perspective, the central aspect of Theorem 2 is that the lower bound on the size of the random CNN depends only on the kernel sizes of the CNNs being approximated.
In subsection 4.2 we discuss the proof of Theorem 2. It requires handling subset-sum problems on multiple random variables at once (random vectors). Furthermore, the inherent parameter-sharing of CNNs creates a specific type of stochastic dependency between coordinates of the random vectors, which we capture with the following definition.
Definition 4 (NSN vector). A  $d$ -dimensional random vector  $Y$  follows a normally-scaled normal (NSN) distribution if, for each  $i \in [d]$ ,  $Y_i = Z \cdot Z_i$  where  $Z, Z_1, \ldots, Z_d$  are i.i.d. random variables following a standard normal distribution.
A key technical contribution of ours is a Multidimensional Random Subset Sum (MRSS) result that supports NSN vectors. In subsection 4.1 we discuss the proof of the next theorem, which follows a strategy similar to that of [Borst et al., 2022, Lemmas 1, 15].
Theorem 5 (Normally-scaled MRSS). Let  $0 < \varepsilon \leq 1/4$ , and let  $d, k,$  and  $n$  be positive integers such that  $n \geq k^2$  and  $k \geq Cd^3 \log \frac{d}{\varepsilon}$  for some universal constant  $C \in \mathbb{R}_{>0}$ . Furthermore, let  $X_1, \ldots, X_n$  be  $d$ -dimensional i.i.d. NSN random vectors. For any  $\vec{z} \in \mathbb{R}^d$  with  $\| \vec{z} \|_1 \leq \sqrt{k}$ , there exists with constant probability a subset  $S \subseteq [n]$  of size  $k$  such that  $\left\| \left( \sum_{i \in S} X_i \right) - \vec{z} \right\|_{\max} \leq \varepsilon$ .
While it is possible to naively apply Theorem 1 to obtain a version of Theorem 2, doing so leads to an exponential lower bound on the required number of random vectors.
In this section, after proving our MRSS result (Theorem 5), we discuss how to use it to obtain our main result on structured pruning (Theorem 2). Full proofs are deferred to Appendix B.
4.1 Multidimensional Random Subset Sum for normally-scaled normal vectors
Notation. Given a set  $S$  and a positive integer  $n$ , we denote by  $\binom{S}{n}$  the family of all subsets of  $S$  containing exactly  $n$  elements of  $S$ . Given  $\varepsilon \in \mathbb{R}_{>0}$ , we define the interval  $I_{\varepsilon}(z_i) = [z_i - \varepsilon, z_i + \varepsilon]$  and the multi-interval  $I_{\varepsilon}(\vec{z}) = [\vec{z} - \varepsilon \mathbf{1}, \vec{z} + \varepsilon \mathbf{1}]$ , where  $\mathbf{1} = (1, 1, \ldots, 1) \in \mathbb{R}^d$ . Moreover, for any event  $\mathcal{E}$ , we denote its complementary event by  $\overline{\mathcal{E}}$ .
In this subsection, we estimate the probability that a set of  $n$  random vectors contains a subset that sums up to a value that is  $\varepsilon$ -close to a given target. The following definition formalises this notion.
Definition 6 (Subset-sum number). Given (possibly random) vectors  $X_{1},\ldots ,X_{n}$  and a vector  $\vec{z}$ , we define the  $\varepsilon$ -subset-sum number of  $X_{1},\ldots ,X_{n}$  for  $\vec{z}$  as
$$
T_{X_{1},\ldots ,X_{n}}^{k}(\vec{z}) = \sum_{S\in \binom {[n]}{k}}\mathbf{1}_{\mathcal{E}_{S}^{(z)}},
$$
where  $\mathcal{E}_S^{(\vec{z})}$  denotes the event  $\left\| \left(\sum_{i\in S}X_i\right) - \vec{z}\right\|_{\max}\leq \varepsilon$ . We write simply  $T_{n,k}$  when  $X_{1},\ldots ,X_{n}$  and  $\vec{z}$  are clear from the context.
To prove Theorem 5 we use the second moment method to provide a lower bound on the probability that the subset-sum number  $T_{n,k}$  is strictly positive, which implies that at least one subset of the random vectors suitably approximates  $\vec{z}$ . Hence, we seek a lower bound on  $\mathbb{E}[T_{n,k}]^2 / \mathbb{E}[T_{n,k}^2]$ .
Our first lemma provides a lower bound on the probability that a sum of NSN vectors is  $\varepsilon$ -close to a target vector, through which one can infer a lower bound on  $\mathbb{E}[T_{n,k}]$ .
Lemma 7 (Sum of NSN vectors). Let  $k \in \mathbb{N}$ ,  $\varepsilon \in \left(0, \frac{1}{4}\right)$ ,  $\vec{z} \in \mathbb{R}^d$  such that  $\| \vec{z} \|_1 \leq \sqrt{k}$  and  $k \geq 16$ . Furthermore, let  $X_1, \ldots, X_k$  be  $d$ -dimensional i.i.d. NSN random vectors with  $d \leq k$ , and let  $c_d = \min \left\{\frac{1}{d^2}, \frac{1}{16}\right\}$ . It holds that
$$
\operatorname * {P r} \biggl (\sum_ {i = 1} ^ {k} X _ {i} \in I _ {\varepsilon} (\vec {z}) \biggr) \geq \frac {1}{1 6} \biggl (\frac {2 \varepsilon}{\sqrt {\pi (1 + 2 \sqrt {c _ {d}} + 2 c _ {d}) k}} \biggr) ^ {d}.
$$
Overview of the proof. The main technical difficulty lies in the fact that the random vectors  $X_{1},\ldots ,X_{k}$  are NSN vectors. Bounds can be easily derived for the case where the  $X_{i}$  are i.i.d. normal random vectors by observing that the sum of normal random variables is also normal.
For  $i \in [k]$ , each entry of  $X_{i}$  can be written as  $Z_{i} \cdot Z_{i,j}$  where  $Z_{i}$  and  $Z_{i,j}$  are i.i.d. normal random variables. Conditional on  $Z_{1}, \ldots, Z_{k}$ , the  $d$  entries of  $X = \sum_{i=1}^{k} X_{i}$  are independent and distributed as  $\mathcal{N}(0, \sum_{i=1}^{k} Z_{i}^{2})$ . By noticing that  $Z_{i}^{2}$  is a chi-squared random variable and employing standard concentration inequalities (Lemma 17 in Appendix A) combined with the law of total probability, we can proceed as if the entries of  $X$  were normal, up to some correction factors.
Bounding  $\mathbb{E}[T_{n,k}^2]$  requires handling stochastic dependencies. Thus, we estimate the joint probability that two subsets of  $k$  elements of  $X_1,\ldots ,X_n$  sum  $\varepsilon$ -close to the same target, taking into account that the intersection of the subsets might not be empty. The next lemma provides an upper bound on this joint probability that depends only on the size of the symmetric difference between the two subsets.
Lemma 8 (Sum of NSN vectors). Let  $k, j \in \mathbb{N}_0$  with  $1 \leq j \leq k$ . Furthermore, let  $X_1, \ldots, X_{k+j}$  be i.i.d.  $d$ -dimensional NSN random vectors with  $k \geq C d^3 \log \frac{d}{\varepsilon}$ . Let  $c_d = \min \left\{\frac{1}{d^2}, \frac{1}{16}\right\}$ ,  $A = \sum_{i=1}^{j} X_i$ ,  $B = \sum_{i=j+1}^{k} X_i$ , and  $C = \sum_{i=k+1}^{k+j} X_i$ . Then, it holds that
$$
\underline {{\operatorname {P r} \left(A + B \in I _ {\varepsilon} \left(\vec {z}\right) , B + C \in I _ {\varepsilon} \left(\vec {z}\right)\right)}} \leq 3 \left(\frac {4 \varepsilon^ {2}}{\pi \left(1 - 2 \sqrt {c _ {d}}\right) j}\right) ^ {d}.
$$
3We adopt the convention that  $\sum_{i = 1}^{0}X_{i} = 0$