SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Overview of the proof. Let  $n = k + j$ . We exploit once more the fact that, for all  $i \in [n]$ , each entry  $X_{i}$  can be written as  $Z_{i} \cdot Z_{i,j}$  where  $Z_{i}$  and  $Z_{i,j}$  are i.i.d. normal random variables. Conditional on  $Z_{1}, \ldots, Z_{k+j}$ , the  $d$  entries of  $A$ ,  $B$ , and  $C$  are independent and distributed as  $\mathcal{N}(0, \sum_{i=1}^{j} Z_{i}^{2})$ ,  $\mathcal{N}(0, \sum_{i=j+1}^{k} Z_{i}^{2})$ , and  $\mathcal{N}(0, \sum_{i=k+1}^{k+j} Z_{i}^{2})$ , respectively. Hence, by the concentration inequalities for the sum of chi-squared random variables (Lemma 17 in Appendix A) and by the law of total probability, we can focus on the term
$$
\Pr \left(A _ {i} + B _ {i} \in I _ {\varepsilon} \left(z _ {i}\right), B _ {i} + C _ {i} \in I _ {\varepsilon} \left(z _ {i}\right) \mid Z _ {1}, \dots , Z _ {n}\right),
$$
where  $A_{i}$ ,  $B_{i}$ , and  $C_{i}$  indicate the  $i$ -th entries of  $A$ ,  $B$ , and  $C$ , respectively.
Another concentration argument for normal random variables (Lemma 14 in Appendix A), allow us to show that
$$
\begin{array}{l} \Pr \left(A _ {i} + B _ {i} \in I _ {\varepsilon} \left(z _ {i}\right), B _ {i} + C _ {i} \in I _ {\varepsilon} \left(z _ {i}\right) \mid Z _ {1}, \dots , Z _ {n}\right) \\ = \mathbb {E} _ {B _ {i}} \left[ \Pr \left(A _ {i} \in I _ {\varepsilon} \left(z _ {i} - B _ {i}\right), C _ {i} \in I _ {\varepsilon} \left(z _ {i} - B _ {i}\right) \mid Z _ {1}, \dots , Z _ {n}, B _ {i}\right) \right] \\ = \mathbb {E} _ {B _ {i}} \left[ \Pr \left(A _ {i} \in I _ {\varepsilon} \left(z _ {i} - B _ {i}\right) \mid Z _ {1}, \dots , Z _ {n}, B _ {i}\right) \Pr \left(C _ {i} \in I _ {\varepsilon} \left(z _ {i} - B _ {i}\right) \mid Z _ {1}, \dots , Z _ {n}, B _ {i}\right) \right] \\ \leq \mathbb {E} _ {B _ {i}} \left[ \Pr \left(A _ {i} \in I _ {\varepsilon} (0) \mid Z _ {1}, \dots , Z _ {n}, B _ {i}\right) \Pr \left(C _ {i} \in I _ {\varepsilon} (0) \mid Z _ {1}, \dots , Z _ {n}, B _ {i}\right) \right] \\ = \Pr \left(A _ {i} \in I _ {\varepsilon} (0) \mid Z _ {1}, \dots , Z _ {n}\right) \Pr \left(C _ {i} \in I _ {\varepsilon} (0) \mid Z _ {1}, \dots , Z _ {n}\right). \\ \end{array}
$$
Thus, we have reduced our argument to the estimation of probabilities of independent normal random variables being close to zero.
The following lemma provides an explicit expression for the variance of the  $\varepsilon$ -subset-sum number.
Lemma 9 (Second moment of  $T_{n,k}$ ). Let  $k, n$  be positive integers. Let  $S_0, S_1, \ldots, S_k$  be subsets of  $[n]$  such that  $|S_0 \cap S_j| = k - j$  for  $j = 0, 1, \ldots, k$ . Let  $S, S'$  be two random variables yielding two subsets of  $[n]$  of size  $k$  drawn independently and uniformly at random. Let  $X_1, \ldots, X_n$  be  $d$ -dimensional i.i.d. NSN random vectors. For any  $\varepsilon > 0$  and  $\vec{z} \in \mathbb{R}^d$ , the second moment of the  $\varepsilon$ -subset sum number is
$$
\mathbb {E} \left[ T _ {n, k} ^ {2} \right] = \binom {n} {k} ^ {2} \sum_ {j = 0} ^ {k} \Pr \left(| \mathrm {S} \cap \mathrm {S} ^ {\prime} | = k - j\right) \Pr \left(\mathcal {E} _ {S _ {0}} ^ {(\vec {z})} \cap \mathcal {E} _ {S _ {j}} ^ {(\vec {z})}\right),
$$
where  $\mathcal{E}_S^{(\vec{z})}$  denotes the event  $\left\| \left(\sum_{i\in S}X_i\right) - \vec{z}\right\|_{\max}\leq \varepsilon$
Proof. Let  $S, S'$  two random variables yielding two elements of  $\binom{[n]}{k}$  drawn independently and uniformly at random. By the definition of  $T_{n,k}$ , we have that
$$
\begin{array}{l} \mathbb{E}\left[T_{n,k}^{2}\right] = \mathbb{E}\Bigg[\Bigg(\sum_{S\in \binom {[n]}{k}}\mathbf{1}_{\mathcal{E}_{S}^{(\bar{z})}}\Bigg)\Bigg(\sum_{S^{\prime}\in \binom {[n]}{k}}\mathbf{1}_{\mathcal{E}_{S^{\prime}}^{(\bar{z})}}\Bigg)\Bigg] \\ = \mathbb{E}\Big[\sum_{S,S^{\prime}\in \binom {[n]}{k}}\mathbf{1}_{\mathcal{E}_{S}^{(\bar{z})}}\mathbf{1}_{\mathcal{E}_{S^{\prime}}^{(\bar{z})}}\Big] \\ = \sum_ {S, S ^ {\prime} \in \left( \begin{array}{c} [ n ] \\ k \end{array} \right)} \Pr \left(\mathcal {E} _ {S} ^ {(\vec {z})} \cap \mathcal {E} _ {S ^ {\prime}} ^ {(\vec {z})}\right) \\ = \sum_ {S, S ^ {\prime} \in \left( \begin{array}{c} [ n ] \\ k \end{array} \right)} \Pr \left(\mathcal {E} _ {\mathrm {S}} ^ {(\vec {z})} \cap \mathcal {E} _ {\mathrm {S} ^ {\prime}} ^ {(\vec {z})} \Big | \mathrm {S} = S, \mathrm {S} ^ {\prime} = S ^ {\prime}\right) \Pr \left(\mathrm {S} = S, \mathrm {S} ^ {\prime} = S ^ {\prime}\right) \\ = \binom {n} {k} ^ {2} \sum_ {j = 0} ^ {k} \Pr \left(\mathcal {E} _ {\mathrm {S}} ^ {(\vec {z})} \cap \mathcal {E} _ {\mathrm {S} ^ {\prime}} ^ {(\vec {z})} \Big | | \mathrm {S} \cap \mathrm {S} ^ {\prime} | = k - j\right) \Pr \left(| \mathrm {S} \cap \mathrm {S} ^ {\prime} | = k - j\right), \\ \end{array}
$$
as  $\operatorname*{Pr}\bigl (\mathcal{E}_{\mathrm{S}}^{(\vec{z})}\cap \mathcal{E}_{\mathrm{S}^{\prime}}^{(\vec{z})}\bigr)$  depends only on the size of  $\mathrm{S}\cap \mathrm{S}^{\prime}$
Overview of the proof of Theorem 5
We use the second moment method (Lemma 15 in Appendix A) on the  $\varepsilon$ -subset-sum number  $T_{n,k}$  of  $X_1, \ldots, X_n$ . Thus, we want to lower bound the right-hand side of
$$
\operatorname * {P r} \left(T > 0\right) \geq \frac {\mathbb {E} [ T _ {n , k} ] ^ {2}}{\mathbb {E} [ T _ {n , k} ^ {2} ]}.
$$
Equivalently, we can provide an upper bound on the inverse,  $\frac{\mathbb{E}[T_{n,k}^2]}{\mathbb{E}[T_{n,k}]^2}$ . By Lemma 9,
$$
\mathbb {E} \left[ T _ {n, k} ^ {2} \right] = \binom {n} {k} ^ {2} \sum_ {j = 0} ^ {k} \operatorname * {P r} \left(| \mathrm {S} \cap \mathrm {S} ^ {\prime} | = k - j\right) \operatorname * {P r} \left(\mathcal {E} _ {S _ {0}} ^ {(\vec {z})} \cap \mathcal {E} _ {S _ {j}} ^ {(\vec {z})}\right)
$$
where  $\mathrm{S}, \mathrm{S}', S_i$  and  $\mathcal{E}_S^{(\vec{z})}$  are defined as in the statement of the lemma. Observe also that
$$
\mathbb{E}\left[T_{n,k}\right] = \sum_{S\in \binom {[n]}{k}}\mathbb{E}\left[ \mathbf{1}_{\mathcal{E}_{S}^{(\vec{z})}}\right] = \sum_{S\in \binom {[n]}{k}}\Pr \left(\mathcal{E}_{S}^{(\vec{z})}\right) = \binom {n}{k}\Pr \left(\mathcal{E}_{S_{0}}^{(\vec{z})}\right).
$$
By using the two above observations, we have
$$
\begin{array}{l} \frac {\mathbb {E} [ T _ {n , k} ] ^ {2}}{\mathbb {E} [ T _ {n , k} ^ {2} ]} = \frac {\binom {n} {k} ^ {2}}{\mathbb {E} [ T _ {n , k} ] ^ {2}} \sum_ {j = 0} ^ {k} \operatorname * {P r} \left(| \mathrm {S} \cap \mathrm {S} ^ {\prime} | = k - j\right) \operatorname * {P r} \left(\mathcal {E} _ {S _ {0}} ^ {(\vec {z})} \cap \mathcal {E} _ {S _ {j}} ^ {(\vec {z})}\right) \\ = \sum_ {j = 0} ^ {k} \operatorname * {P r} \left(\left| \mathrm {S} \cap \mathrm {S} ^ {\prime} \right| = k - j\right) \frac {\operatorname * {P r} \left(\mathcal {E} _ {S _ {0}} ^ {\left(\vec {z}\right)} \cap \mathcal {E} _ {S _ {j}} ^ {\left(\vec {z}\right)}\right)}{\operatorname * {P r} \left(\mathcal {E} _ {S _ {0}} ^ {\left(\vec {z}\right)}\right) ^ {2}}. \\ \end{array}
$$
Lemma 7 provides a lower bound on the term  $\operatorname*{Pr}\bigl (\mathcal{E}_{S_0}^{(\vec{z})}\bigr)$  while Lemma 8 gives an upper bound on the term  $\operatorname*{Pr}\bigl (\mathcal{E}_{S_0}^{(\vec{z})}\cap \mathcal{E}_{S_j}^{(\vec{z})}\bigr)$ .
In the full proof, we then show that  $\operatorname*{Pr}\left(|\mathrm{S} \cap \mathrm{S}'| \geq k / d\right)$  can be bounded using the Chernoff bound (Lemma 16 in Appendix A) even if we do not deal directly with binomial random variables. This allows us to discard the indices  $j$  for which  $\operatorname*{Pr}\left(\mathcal{E}_{S_0}^{(z)} \cap \mathcal{E}_{S_j}^{(z)}\right)$  is large, which leads to the result after some technical manipulations.
4.2 Proving SLTH for structured pruning
To prove Theorem 2, we first show how to obtain the same approximation result for a single-layer CNN. Then, we iteratively apply the same argument for all layers of a larger CNN and show that the approximation error stays small.
We define the positive and negative parts of a tensor.
Definition 10. Given a tensor  $X \in \mathbb{R}^{d_1 \times \dots \times d_n}$ , the positive and negative parts of  $X$  are respectively defined as  $X_{\vec{i}}^{+} = X_{\vec{i}} \cdot \mathbf{1}_{X_{\vec{i}} > 0}$  and  $X_{\vec{i}}^{-} = -X_{\vec{i}} \cdot \mathbf{1}_{X_{\vec{i}} < 0}$ , where  $\vec{i} \in [d_1] \times \dots \times [d_n]$  points at a generic entry of  $X$ .
Approximating a single-layer CNN
We first present a preliminary lemma that shows how to prune a single-layer convolution  $\phi (V*X)$  in a way that dispenses us from dealing with the ReLU  $\phi$ .
Lemma 11. Let  $D, d, c, n \in \mathbb{N}$  be positive integers,  $V \in \mathbb{R}^{1 \times 1 \times c \times 2nc}$ , and  $X \in \mathbb{R}^{D \times D \times c}$ . Let  $S_1 \in \{0, 1\}^{\mathrm{size}(V)}$  be a  $2n$ -channel-blocked mask. There exists a mask  $S_2 \in \{0, 1\}^{\mathrm{size}(V)}$  which only removes filters such that, for each  $(i, j, k) \in [D] \times [D] \times [2nc]$ , if  $\tilde{S} = S_1 \odot S_2$ , then
$$
\left(\phi \left(\left(V \odot \tilde {S}\right) * X\right)\right) _ {i, j, k} = \left(\left(V \odot \tilde {S}\right) ^ {+} * X ^ {+} + \left(V \odot \tilde {S}\right) ^ {-} * X ^ {-}\right) _ {i, j, k}.
$$
Overview of the proof.  $S_{2} \in \{0,1\}^{\mathrm{size}(V)}$  is such that  $\tilde{V} = V \odot \tilde{S} = (V \odot S_{1}) \odot S_{2}$  contains only non-negative edges going from each input channel  $t$  to the output channels  $2(t - 1)n + 1, \ldots, (2t - 1)n$ , and only non-positive edges going from each input channel  $t$  to the output channels  $(2t - 1)n + 1, \ldots, 2tn$ . Notice that, after applying the  $2n$ -channel-blocked mask  $S_{1}$ , the only possible non-zero entry of the  $k$ -th filter is located at its  $t = \lceil \frac{k}{2n} \rceil$ -th channel. Hence, for each  $k \in [2nc]$ , we keep filter  $k$  if the entry of its  $t$ -th channel is non-negative and  $2t - 1 = \lceil \frac{k}{n} \rceil$ , or it is non-positive and  $2t = \lceil \frac{k}{n} \rceil$ .
We approximate a single convolution  $K * X$  by pruning a polynomially larger neural network of the form  $U * \phi(V * X)$  exploiting only a channel-blocked mask and filter removal: this is achieved using the MRSS result (Theorem 5).
Lemma 12 (Kernel pruning). Let  $D, d, c_0, c_1, n \in \mathbb{N}$  be positive integers,  $\varepsilon \in \left(0, \frac{1}{4}\right)$ ,  $M \in \mathbb{R}_{>0}$ , and  $C \in \mathbb{R}_{>0}$  be a universal constant with
$$
n \geq C d ^ {1 3} c _ {1} ^ {6} \log^ {3} \frac {d ^ {2} c _ {1} c _ {0}}{\varepsilon}.
$$
Let  $U \sim \mathcal{N}^{d \times d \times 2nc_0 \times c_1}$ ,  $V \sim \mathcal{N}^{1 \times 1 \times c_0 \times 2nc_0}$  and  $S \in \{0,1\}^{\mathrm{size}(V)}$ , with  $S$  being a 2n-channel-blocked mask. We define  $N_0(X) = U * \phi(V * X)$  where  $X \in \mathbb{R}^{D \times D \times c_0}$ , and its pruned version  $N_0^{(S)}(X) = U * \phi((V \odot S) * X)$ . With probability  $1 - \varepsilon$ , for all  $K \in \mathbb{R}^{d \times d \times c_0 \times c_1}$  with  $\| K \|_1 \leq 1$ , it is possible to remove filters from  $N_0^{(S)}$  to obtain a CNN  $\tilde{N}_0^{(S)}$  for which
$$
\sup _ {X: \| X \| _ {\max} \leq M} \left\| K * X - \tilde {N} _ {0} ^ {(S)} (X) \right\| _ {\max} <   \varepsilon M.
$$
Overview of the proof. Exploiting Lemma 11, for each  $(r,s,t_1)\in [d]\times [d]\times [c_1]$ , one can show that
$$
\begin{array}{l} \left(U * \phi \left(\left(V \odot \tilde {S}\right) * X\right)\right) _ {r, s, t _ {1}} = \sum_ {i, j \in [ d ], t _ {0} \in [ c _ {0} ]} \left(\sum_ {k \in [ n c _ {0} ]} U _ {i, j, k, t _ {1}} \cdot \tilde {V} _ {1, 1, t _ {0}, k} ^ {+}\right) \cdot X _ {r - i + 1, s - j + 1, t _ {0}} ^ {+} \\ + \sum_ {i, j \in [ d ], t _ {0} \in [ c _ {0} ]} \left(\sum_ {k \in [ n c _ {0} ]} U _ {i, j, k, t _ {1}} \cdot \tilde {V} _ {1, 1, t _ {0}, k} ^ {-}\right) \cdot X _ {r - i + 1, s - j + 1, t _ {0}} ^ {-}, \\ \end{array}
$$
where  $\tilde{S} = S_{1}\odot S_{2}$ , with  $S_{1}$  being a  $2n$ -channel-blocked mask and  $S_{2}$  being a mask that removes filters. Through a Chernoff bound, we show that  $\tilde{V}_{1,1,t_0,:}^{+}$  has at least  $n / 3$  non-zero entries. Up to reshaping the tensor as a one-dimensional vector, we observe that  $U_{:,k,:}\tilde{V}_{1,1,t_0,k}^{+}$  is an NSN vector (Lemma 18 in Appendix A). Hence, we can apply a boosted version of the MRSS result (Corollary 19 in Appendix A) and show that, with high probability, for all target filters  $K$  with  $\| K\| _1\leq 1$ , we can prune all but roughly  $\sqrt{n / (C_1d\log\frac{1}{\varepsilon})}$  positive entries of  $\tilde{V}_{1,1,t_0,k}^{+}$ , with  $C_1$  being a universal constant, such that  $\sum_{k\in [nc_0]}U_{:,k,:}\tilde{V}_{1,1,t_0,:}^{+}$  approximates the channels  $K_{:,t_0,:}$  up to error  $\varepsilon /(2d^2 c_0c_1)$ . The same holds for  $\sum_{k\in [nc_0]}U_{:,k,:}\tilde{V}_{1,1,t_0,:}^{-}$ . This pruning can be achieved by applying a third mask  $S_{3}$  that only removes filters. Through some non-trivial calculations and by applying the Tensor Convolution Inequality (Lemma 20 in Appendix A), one can combine the above results to get the thesis. Notice that the overall pruning can be represented by the mask  $S_{1}\odot (S_{2}\odot S_{3})$ , where  $S_{2}\odot S_{3}$  is a mask that only removes filters, and  $S = S_{1}$  is a  $2n$ -channel-blocked mask.
Remark 13. From the proofs of Lemmas 12 and 11, we can see that the overall modification yields a pruned CNN  $\hat{U} * \phi(\hat{V} * X)$  with  $\hat{V} \in \mathbb{R}^{1 \times 1 \times c_0 \times 2mc_0}$  and  $\hat{U} \in \mathbb{R}^{d \times d \times 2mc_0 \times c_1}$ , where  $m = \sqrt{n / (C_1d\log\frac{1}{\varepsilon})}$  for a universal constant  $C_1$ . Moreover, the kernel  $\hat{V}$  is obtained through a 3-stage pruning process: First, we apply a  $2n$ -channel-blocked mask  $S_1$ . Second, we remove filters based on entries' signs through a mask  $S_2$ . Third, we remove filters according to the MRSS result through a mask  $S_3$ . Masks  $S_2$  and  $S_3$  can be combined into a single mask  $S_2 \odot S_3$  that only removes filters: overall the pruning process consists of a  $2n$ -channel-blocked mask and filter removal, which justifies the statement of Theorem 2.
Overview of the proof of Theorem 2
We iteratively apply Lemma 12 to each layer while carefully controlling the approximation error via tools such as the Lipschitz property of ReLU and the Tensor Convolution Inequality (Lemma 20). More precisely, we show that (i) the approximation error does not increase too much at each layer; and (ii) all layer approximations can be combined to approximate the entire target network. Notice that Lemma 12 guarantees that, with high probability, we can approximate all possible target filters  $K$  (with  $\| K\| _1\leq 1$ ) from the same 2-layered network. Hence, we get the result for the supremum over all possible choices of target filters.
In previous works [da Cunha et al., 2022, Burkholz, 2022a] the assumption that the kernel of every second layer has shape  $1 \times 1 \times \ldots$  is only an artifact of the proof since one can readily prune entries of an arbitrarily shaped tensor to enforce the desired shape. In our case, however, the concept of structured pruning can be quite broad, and such reshaping via pruning might not fit some sparsity patterns, depending on the context. The hypothesis on the shape can be a relevant limitation for such use cases. The constructions proposed by Burkholz [2022a,b] appear as a promising direction to overcome this limitation, with the added benefit of reducing the depth overhead.
The convolution operation commonly employed in CNNs can be cumbersome at many points of our analysis. Exploring different concepts of convolution can be an interesting path for future work as it could lead to tidier proofs and more general results. For instance, employing a 3D convolution would spare a factor  $c$  in Theorem 2.
Another limitation of our results is the restriction to ReLU as the activation function. Many previous works on the SLTH exploit the fact that ReLU satisfies the identity  $x = \phi(x) - \phi(-x)$ . Burkholz [2022a] leveraged that to obtain an SLTH result for CNNs with activation functions  $f$  for which  $f(x) - f(-x) \approx x$  around the origin. Our analysis, on the other hand, does not rely on such property, so adapting the approach of [Burkholz, 2022a] to our setting is not straightforward.
Finally, we remark that the assumption of normally distributed weights might be relaxed. Borst et al. [2022] provided an MRSSP result for independent random variables whose distribution converges "fast enough" to a Gaussian one. We believe our arguments can serve well as baselines to generalise our results to support random weights distributed as such.
Francesco d'Amore is supported by the MUR Fare2020 Project "PAReCoDi" CUP J43C22000970001.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https://openreview.net/forum?id=Sy8gdB9xx.1
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Commun. ACM, 64(3):107-115, 2021. doi: 10.1145/3446776. URL https://doi.org/10.1145/3446776.1
Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Sensitivity and generalization in neural networks: an empirical study. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openreview.net/forum?id=HJC2SzZCW.1
4The required convergence rate is higher than that ensured by the Berry-Esseen theorem.
Yang He and Lingao Xiao. Structured pruning for deep convolutional neural networks: A survey. CoRR, abs/2303.00566, 2023. doi: 10.48550/arXiv.2303.00566. URL https://doi.org/10.48550/arXiv.2303.00566. 4
Devdatt P. Dubhashi and Alessandro Panconesi. Concentration of Measure for the Analysis of Randomized Algorithms. Cambridge University Press, 2009. ISBN 978-0-521-88427-3. URL http://www.cambridge.org/gb/knowledge/isbn/item2327542/.15
B. Laurent and P. Massart. Adaptive estimation of a quadratic functional by model selection. The Annals of Statistics, 28(5):1302-1338, 2000. doi: 10.1214/aos/1015957395. 15
Benjamin Doerr. Probabilistic Tools for the Analysis of Randomized Optimization Heuristics. arXiv:1801.06733, 2020. 22
A Technical tools
A.1 Concentration inequalities
Lemma 14 (Most-probable normal interval). Let  $X$  follow a zero-mean normal distribution with variance  $\phi^2$ . For any  $z, \varepsilon \in \mathbb{R}$
$$
\operatorname * {P r} \left(X \in [ z - \varepsilon , z + \varepsilon ]\right) \leq \operatorname * {P r} \left(X \in [ - \varepsilon , \varepsilon ]\right).
$$
Proof. Let  $\varphi(x)$  denote the probability density function of  $X$ . Then,
$$
\Pr \left(X \in [ - \varepsilon , \varepsilon ]\right) - \Pr \left(X \in [ z - \varepsilon , z + \varepsilon ]\right) = \int_ {- \varepsilon} ^ {\varepsilon} \varphi (x) d x - \int_ {z - \varepsilon} ^ {z + \varepsilon} \varphi (x) d x.
$$
If  $z - \varepsilon \geq \varepsilon$  or  $z + \varepsilon \leq -\varepsilon$ , the thesis is trivial as  $\varphi(|x|)$  decreases in  $x$ . W.l.o.g., suppose  $z$  is positive and  $z - \varepsilon < \varepsilon$ . Then,  $-\varepsilon < z - \varepsilon < \varepsilon < z + \varepsilon$ . It follows that
$$
\begin{array}{l} \int_ {- \varepsilon} ^ {\varepsilon} \varphi (x) d x - \int_ {z - \varepsilon} ^ {z + \varepsilon} \varphi (x) d x = \int_ {- \varepsilon} ^ {z - \varepsilon} \varphi (x) d x - \int_ {\varepsilon} ^ {z + \varepsilon} \varphi (x) d x \\ = \int_ {- \varepsilon} ^ {z - \varepsilon} \varphi (x) - \varphi (x + 2 \varepsilon) d x \\ \end{array}
$$
which is non-negative as  $\varphi (x)\geq \varphi (x + 2\varepsilon)$  for  $x\geq -\varepsilon$
Lemma 15 (Second moment method). If  $Z$  is a non-negative random variable then
$$
\operatorname * {P r} \left(Z > 0\right) \geq \frac {\mathbb {E} \left[ Z \right] ^ {2}}{\mathbb {E} \left[ Z ^ {2} \right]}.
$$
Lemma 16 (Chernoff-Hoeffding bounds [Dubhashi and Panconesi, 2009]). Let  $X_{1}, X_{2}, \ldots, X_{n}$  be independent random variables such that  $\operatorname*{Pr}(0 \leq X_i \leq 1) = 1$  for all  $i \in [n]$ . Let  $X = \sum_{i=1}^{n} X_{i}$  and  $\mathbb{E}[X] = \mu$ . Then, for any  $\delta \in (0,1)$  the following holds:
1. if  $\mu \leq \mu_{+}$ , then  $\operatorname{Pr}\left(X \geq (1 + \delta)\mu_{+}\right) \leq \exp\left(-\frac{\delta^{2}\mu_{+}}{3}\right)$ ;
2. if  $0 \leq \mu_{-} \leq \mu$ , then  $\operatorname{Pr}\left(X \leq (1 - \delta)\mu_{-}\right) \leq \exp\left(-\frac{\delta^{2}\mu_{+}}{2}\right)$ .
Lemma 17 (Corollary of [Laurent and Massart, 2000, Lemma 1]). Let  $X \sim \chi_d^2$  be a chi-squared random variable with  $d$  degrees of freedom. For any  $t > 0$ , it holds that
1.  $\operatorname*{Pr}\left(X\geq d + 2\sqrt{dt} +2t\right)\leq \exp (-t);$
2.  $\operatorname*{Pr}\left(X\leq d - 2\sqrt{dt}\right)\leq \exp (-t).$
A.2 Supporting results
Lemma 18 (NSN with positive scalar). If a  $d$ -dimensional random vector  $Y$  is such that, for each  $i \in [d]$ ,  $Y_i = \tilde{Z} \cdot \tilde{Z}_i$ , where  $\tilde{Z}_1, \ldots, \tilde{Z}_n$  are identically distributed random variables following a standard normal distribution,  $\tilde{Z}$  is a half-normal distribution, and  $\tilde{Z}, \tilde{Z}_1, \ldots, \tilde{Z}_n$  are independent, then  $Y$  follows an NSN distribution.
Proof. By Definition 4,  $Y$  is NSN if, for each  $i \in [d]$ ,  $Y_{i} = Z \cdot Z_{i}$  where  $Z, Z_{1}, \ldots, Z_{n}$  are i.i.d. random variables following a standard normal distribution. We know that  $\tilde{Z} = |Z|$ , therefore  $\tilde{Z}_{i} = \operatorname{sign}(Z) \operatorname{sign}(Z_{i}) |Z_{i}|$  for each  $i = 1, \ldots, n$ , where  $Z, Z_{1}, \ldots, Z_{n}$  are i.i.d. standard normal
5. I.e.  $\tilde{Z} = |Z|$  where  $Z$  is a standard normal distribution.
random variables, as  $\operatorname{sign}(Z)$  and  $\operatorname{sign}(Z_i)$  is independent of  $\operatorname{sign}(Z)$  and of  $\operatorname{sign}(Z)$  and  $\operatorname{sign}(Z_j)$  for  $i \neq j$ . Then,
$$
\begin{array}{l} Y _ {i} = \tilde {Z} \cdot \tilde {Z} _ {i} \\ = | Z | \cdot \operatorname {s i g n} (Z) \operatorname {s i g n} \left(Z _ {i}\right) | Z _ {i} | \\ = \operatorname {s i g n} (Z) | Z | \cdot \operatorname {s i g n} \left(Z _ {i}\right) | Z _ {i} | \\ = Z \cdot Z _ {i}, \\ \end{array}
$$
implying the thesis.
Corollary 19 (of Theorem 5). Let  $d, k,$  and  $n$  be positive integers with  $n \geq C_1 dk^2 \log \left(\frac{1}{\varepsilon}\right)$  and  $k \geq C_2 d^3 \log \frac{d}{\varepsilon}$  for some universal constants  $C_1, C_2 \in \mathbb{R}_{>0}$ . Let  $X_1, \ldots, X_n$  be  $d$ -dimensional i.i.d. NSN random vectors. For any  $0 < \varepsilon \leq \frac{1}{4}$  it holds
$$
\operatorname * {P r} \left(\forall \vec {z} \in \mathbb {R} ^ {d}: \| \vec {z} \| _ {1} \leq 1, \exists S: | S | = k, \left\| \left(\sum_ {i \in S} X _ {i}\right) - \vec {z} \right\| _ {\max} \leq \varepsilon\right) \geq 1 - \varepsilon .
$$
Proof. Observe that the set  $[-1, 1]^d$  can be "partitioned" in  $1 / \varepsilon^d$  many  $\infty$ -norm balls of radius  $\varepsilon$ . Let  $s = \left[C_1 d \log \left(\frac{1}{\varepsilon}\right)\right]$  and let us partition the  $n$  vectors  $X_1, \ldots, X_n$  in  $s$  disjoint sets  $G_1, \ldots, G_s$  of at least  $k^2$  vectors each. By Theorem 5, there is a constant  $c \in (0, 1)$  such that for each group  $G_i$ $(i \in [s])$
$$
\Pr \left(\exists S \subset G _ {i}: | S | = k, \left\| \left(\sum_ {i \in S} X _ {i}\right) - \bar {z} \right\| _ {\max } \leq \varepsilon\right) \geq c. \tag {3}
$$