SlowGuess/NeurIPS_2023_OCR huggingface dataset:
with the proofs. By  $[n:m]$  we denote the set  $\{n,n + 1,\dots ,m\}$  for each pair of integers  $n\leq m\in \mathbb{N}$ . In formulas, we obtain a tensor  $\tilde{V}$  such that, for each  $(t,k)\in [c]\times [2nc]$ :
$$
\left(V \odot \tilde {S}\right) _ {1, 1, t, k} = \left\{ \begin{array}{l l} V _ {1, 1, t, k} \cdot \mathbf {1} _ {V _ {1, 1, t, k} > 0} & \text {i f} k \in \left[ (2 t - 2) n + 1: (2 t - 1) n \right], \\ V _ {1, 1, t, k} \cdot \mathbf {1} _ {V _ {1, 1, t, k} <   0} & \text {i f} k \in \left[ (2 t - 1) n + 1: 2 t n \right], \\ 0 & \text {o t h e r w i s e .} \end{array} \right. \tag {26}
$$
To simplify the notation, we define the following indicator functions: for any  $(t,k)\in [c]\times [2nc]$
$$
\mathbf {1} _ {\frac {k}{2 n} \in (t - 1, t - \frac {1}{2} ]} = 1 \text {i f f} k \in [ (2 t - 2) n + 1: (2 t - 1) n ], \text {a n d}
$$
$$
\mathbf {1} _ {\frac {k}{2 n} \in \left(t - \frac {1}{2}, t \right]} = 1 \text {i f f} k \in \left[ (2 t - 1) n + 1: 2 t n \right]. \tag {27}
$$
For each  $(i,j,k)\in [D]\times [D]\times [2nc]$ , applying Eq. 26 and using Definition 10, it then holds
$$
\begin{array}{l} \left(\phi \left(\left(V \odot \tilde {S}\right) * X\right)\right) _ {i, j, k} \\ = \phi \left(\sum_ {t = 1} ^ {c _ {0}} \tilde {V} _ {1, 1, t, k} X _ {i, j, t}\right) \\ = \phi \left(\sum_ {t = 1} ^ {c _ {0}} \left(V _ {1, 1, t, k} X _ {i, j, t} \cdot \mathbf {1} _ {V _ {1, 1, t, k} > 0} \mathbf {1} _ {\frac {k}{2 n} \in \left(t - 1, t - \frac {1}{2} \right.} \right] \right. \\ \left. \left. + V _ {1, 1, t, k} X _ {i, j, t} \cdot \mathbf {1} _ {V _ {1, 1, t, k} <   0} \mathbf {1} _ {\frac {k}{2 n} \in \left(t - \frac {1}{2}, t \right]})\right) \right) \\ = \phi \left(\sum_ {t = 1} ^ {c _ {0}} \left(V _ {1, 1, t, k} ^ {+} X _ {i, j, t} \mathbf {1} _ {\frac {k}{2 n} \in \left(t - 1, t - \frac {1}{2} \right]} - V _ {1, 1, t, k} ^ {-} X _ {i, j, t} \mathbf {1} _ {\frac {k}{2 n} \in \left(t - \frac {1}{2}, t \right]}\right)\right) \\ = \phi \Bigg (\sum_ {t = 1} ^ {c _ {0}} \big (V _ {1, 1, t, k} ^ {+} (X _ {i, j, t} ^ {+} - X _ {i, j, t} ^ {-}) \mathbf {1} _ {\frac {k}{2 n} \in \big (t - 1, t - \frac {1}{2} \big ]} \\ \left. + V _ {1, 1, t, k} ^ {-} \left(X _ {i, j, t} ^ {-} - X _ {i, j, t} ^ {+}\right) \mathbf {1} _ {\frac {k}{2 n} \in \left(t - \frac {1}{2}, t \right]}\right). \tag {28} \\ \end{array}
$$
Observe that only one term survives in the summation in Eq. 28, as there exists only one  $t \in [c_0]$  such that  $k \in [(2t - 2)n + 1 : 2tn]$ , say  $t^\star$ . Moreover, out of the four additive terms in the expression
$$
V _ {1, 1, t ^ {\star}, k} ^ {+} (X _ {i, j, t ^ {\star}} ^ {+} - X _ {i, j, t ^ {\star}} ^ {-}) \mathbf {1} _ {\frac {k}{2 n} \in \left(t ^ {\star} - 1, t ^ {\star} - \frac {1}{2} \right]} + V _ {1, 1, t ^ {\star}, k} ^ {-} (X _ {i, j, t ^ {\star}} ^ {-} - X _ {i, j, t ^ {\star}} ^ {+}) \mathbf {1} _ {\frac {k}{2 n} \in \left(t ^ {\star} - \frac {1}{2}, t ^ {\star} \right]},
$$
at most one is non-zero, due to Definition 10. The ReLU cancels out negative ones, implying that Eq. 28 can be rewritten without the ReLU as a sum of only non-negative terms (out of which, at most one is non-zero) as follows
$$
\begin{array}{l} \phi \Bigg(\sum_{t = 1}^{c_{0}}\big(V_{1,1,t,k}^{+}(X_{i,j,t}^{+} - X_{i,j,t}^{-})\mathbf{1}_{\frac{k}{2n}\in \big(t - 1,t - \frac{1}{2}\big]} \\ \left. + V _ {1, 1, t, k} ^ {-} \big (X _ {i, j, t} ^ {-} - X _ {i, j, t} ^ {+} \big) \mathbf {1} _ {\frac {k}{2 n} \in \big (t - \frac {1}{2}, t ] \big)}\right) \\ = \sum_ {t = 1} ^ {c _ {0}} \left(V _ {1, 1, t, k} ^ {+} X _ {i, j, t} ^ {+} \mathbf {1} _ {\frac {k}{2 n} \in \left(t - 1, t - \frac {1}{2} \right]} + V _ {1, 1, t, k} ^ {-} X _ {i, j, t} ^ {-} \mathbf {1} _ {\frac {k}{2 n} \in \left(t - \frac {1}{2}, t \right]}\right). \tag {29} \\ \end{array}
$$
Finally, by Eq. 26 and Eq. 27,  $\tilde{V}_{1,1,t,k}^{+} = 0$  if  $\frac{k}{2n} \notin (t - 1, t - \frac{1}{2}]$ , and  $\tilde{V}_{1,1,t,k}^{-} = 0$  if  $\frac{k}{2n} \in (t - \frac{1}{2}, t]$  which means that in Eq. 29 we can ignore the indicator functions and further simplify the expression as
$$
\begin{array}{l} \sum_ {t = 1} ^ {c _ {0}} \left(V _ {1, 1, t, k} ^ {+} X _ {i, j, t} ^ {+} \mathbf {1} _ {\frac {k}{2 n} \in \left(t - 1, t - \frac {1}{2} \right]} + V _ {1, 1, t, k} ^ {-} X _ {i, j, t} ^ {-} \mathbf {1} _ {\frac {k}{2 n} \in \left(t - \frac {1}{2}, t \right]}\right) \\ = \sum_ {t = 1} ^ {c _ {0}} \left(\tilde {V} _ {1, 1, t, k} ^ {+} X _ {i, j, t} ^ {+} + \tilde {V} _ {1, 1, t, k} ^ {-} X _ {i, j, t} ^ {-}\right) \\ \end{array}
$$
$$
\begin{array}{l} = \left(\sum_ {t = 1} ^ {c _ {0}} \tilde {V} _ {1, 1, t, k} ^ {+} X _ {i, j, t} ^ {+} + \sum_ {t = 1} ^ {c _ {0}} \tilde {V} _ {1, 1, t, k} ^ {-} X _ {i, j, t} ^ {-}\right) \\ = \left(\tilde {V} ^ {+} * X ^ {+} + \tilde {V} ^ {-} * X ^ {-}\right) _ {i, j, k}. \\ \end{array}
$$
Proof of Lemma 12. Adopting the same definitions as in Lemma 11 (and Eq. 26), for each  $(r, s, t_1) \in [d] \times [d] \times [c_1]$  we have, by Lemma 11,
$$
\begin{array}{l} \left(U * \phi \left(\left(V \odot \tilde {S}\right) * X\right)\right) _ {r, s, t _ {1}} \\ = \left(U * \left(\left(\tilde {V} ^ {+} * X ^ {+}\right) + \left(\tilde {V} ^ {-} * X ^ {-}\right)\right)\right) _ {r, s, t _ {1}} \\ = \sum_ {i, j \in [ d ], k \in [ 2 n c _ {0} ]} U _ {i, j, k, t _ {1}} \cdot \left(\left(\tilde {V} ^ {+} * X ^ {+}\right) + \left(\tilde {V} ^ {-} * X ^ {-}\right)\right) _ {r - i + 1, s - j + 1, k} \\ = \sum_ {i, j \in [ d ], k \in [ 2 n c _ {0} ]} U _ {i, j, k, t _ {1}} \cdot \sum_ {t _ {0} \in [ c _ {0} ]} \left(\tilde {V} _ {1, 1, t _ {0}, k} ^ {+} \cdot X _ {r - i + 1, s - j + 1, t _ {0}} ^ {+} \right. \\ \left. + \tilde {V} _ {1, 1, t _ {0}, k} ^ {-} \cdot X _ {r - i + 1, s - j + 1, t _ {0}} ^ {-}\right) \\ = \sum_ {t _ {0} \in [ c _ {0} ]} \sum_ {i, j \in [ d ], k \in [ 2 n c _ {0} ]} \left(U _ {i, j, k, t _ {1}} \cdot \tilde {V} _ {1, 1, t _ {0}, k} ^ {+}\right) \cdot X _ {r - i + 1, s - j + 1, t _ {0}} ^ {+} \\ + \sum_ {t _ {0} \in [ c _ {0} ]} \sum_ {i, j \in [ d ], k \in [ 2 n c _ {0} ]} \left(U _ {i, j, k, t _ {1}} \cdot \tilde {V} _ {1, 1, t _ {0}, k} ^ {-}\right) \cdot X _ {r - i + 1, s - j + 1, t _ {0}} ^ {-} \\ = \sum_ {i, j \in [ d ], t _ {0} \in [ c _ {0} ]} \left(\sum_ {k \in [ 2 n c _ {0} ]} U _ {i, j, k, t _ {1}} \cdot \tilde {V} _ {1, 1, t _ {0}, k} ^ {+}\right) \cdot X _ {r - i + 1, s - j + 1, t _ {0}} ^ {+} \\ + \sum_ {i, j \in [ d ], t _ {0} \in [ c _ {0} ]} \left(\sum_ {k \in [ 2 n c _ {0} ]} U _ {i, j, k, t _ {1}} \cdot \tilde {V} _ {1, 1, t _ {0}, k} ^ {-}\right) \cdot X _ {r - i + 1, s - j + 1, t _ {0}} ^ {-}. \\ \end{array}
$$
We remind the reader that  $\tilde{S} = S_{1}\odot S_{2}$ , with  $S_{1}$  being a  $2n$ -channel-blocked mask and  $S_{2}$  being a mask that removes filters. Define  $L_{i,j,t_0,t_1}^+ = \sum_{k\in [nc]}U_{i,j,k,t_1}\cdot \tilde{V}_{1,1,t_0,k}^+$  and, similarly,  $L_{i,j,t_0,t_1}^- = \sum_{k\in [nc]}U_{i,j,k,t_1}\cdot \tilde{V}_{1,1,t_0,k}^-$ . Then,
$$
\begin{array}{l} \sum_ {i, j \in [ d ], t _ {0} \in [ c _ {0} ]} \left(\sum_ {k \in [ n c _ {0} ]} U _ {i, j, k, t _ {1}} \cdot \tilde {V} _ {1, 1, t _ {0}, k} ^ {+}\right) \cdot X _ {r - i + 1, s - j + 1, t _ {0}} ^ {+} \\ + \sum_ {i, j \in [ d ], t _ {0} \in [ c _ {0} ]} \left(\sum_ {k \in [ n c _ {0} ]} U _ {i, j, k, t _ {1}} \cdot \tilde {V} _ {1, 1, t _ {0}, k} ^ {-}\right) \cdot X _ {r - i + 1, s - j + 1, t _ {0}} ^ {-} \\ = \sum_ {i, j \in [ d ], t _ {0} \in [ c _ {0} ]} L _ {i, j, t _ {0}, t _ {1}} ^ {+} \cdot X _ {r - i + 1, s - j + 1, t _ {0}} ^ {+} + \sum_ {i, j \in [ d ], t _ {0} \in [ c _ {0} ]} L _ {i, j, t _ {0}, t _ {1}} ^ {-} \cdot X _ {r - i + 1, s - j + 1, t _ {0}} ^ {-}. \\ \end{array}
$$
We now show that, for each  $t_0 \in [c_0]$ ,  $K_{\vdots ,\cdot ,t_0,\cdot}$  can be  $\varepsilon$ -approximated by  $L_{\vdots ,\cdot ,t_0,\cdot}^+$  by suitably pruning  $\tilde{V}^{+}$ , i.e. by further zeroing entries of  $\tilde{S}$ , and that such pruning corresponds to solving an instance of MRSS according to Theorem 5. The same reasoning applies to  $K^{-}$  and  $L^{-}$ .
For each  $t_0 \in [c_0]$ , let
$$
I _ {+} ^ {\left(t _ {0}\right)} = \left\{k \in \left\{\left(2 t _ {0} - 2\right) n + 1, \dots , \left(2 t _ {0} - 1\right) n \right\}: \tilde {S} _ {1, 1, t _ {0}, k} = 1 \right\}.
$$
Observe that  $I_{+}^{(t_{0})}$  consists of the strictly positive entries of  $\tilde{V}_{1,1,t_{0},\cdot}^{+}$ . Since the entries of  $V$  follow a standard normal distribution, each entry is positive with probability  $1/2$ . By a standard application of Chernoff bounds (Lemma 16 in Appendix A), we then have
$$
\Pr \left(\left| I _ {+} ^ {(t _ {0})} \right| > \frac {n}{3}\right) \geq 1 - \frac {\varepsilon}{4}, \tag {30}
$$
provided that the constant  $C$  in the bound on  $n$  is sufficiently large.
For each  $k \in I_{+}^{(t_0)}$ , up to reshaping the tensor as a one-dimensional vector,  $U_{:,k,:} \cdot \tilde{V}_{1,1,t_0,k}^{+}$  is an NSN vector (Definition 4) by Lemma 18 (Appendix B). Thus, for each  $t_0 \in [c_0]$  and a sufficiently-large value of  $C$ , since we have  $n \geq Cd^{13}c_1^6\log^3\frac{d^2c_1c_0}{\varepsilon}$ , we can apply an amplified version of Theorem 5 (i.e. Corollary 19 in Appendix B with vectors of dimension  $d^2 c_1$ ) to show that, with probability  $1 - \frac{\varepsilon}{4c_0}$ , for all target filter  $K$  such that  $\| K_{:,t_0,:}\|_1 \leq 1$ , there exists a way to zero the entries indexed by  $I_{+}^{(t_0)}$  of  $\tilde{S}$  (and thus  $\tilde{V}_{1,1,t_0,:}^{+}$ ), so that the pruned version of  $L_{:,t_0,:}^{+} = \sum_{k \in [nc_0]} U_{:,k,:} \cdot \tilde{V}_{1,1,t_0,k}^{+}$  approximates  $K_{:,t_0,:}$ . In particular, there exists another binary mask  $S_3^{+} \in \{0,1\}^{\text{size } \tilde{S}}$  such that  $\hat{L}_{:,t_0,:}^{+} = \sum_{k \in [nc_0]} U_{:,k,:} \cdot \hat{V}_{1,1,t_0,k}^{+}$  approximates  $K_{:,t_0,:}$ , where  $\hat{V}^{+} = \tilde{V}^{+} \odot S_{3}^{+}$ . An analogous argument carries on for a binary mask  $S_3^{-}$  and  $-\hat{L}_{:,t_0,:}^{-}$ . More formally, let
$$
\mathcal {E} _ {t _ {0}, +} ^ {\left(\text {k e r n e l}\right)} = \left\{\forall K: \| K \| _ {1} \leq 1, \exists S _ {3} ^ {+} \in \{0, 1 \} ^ {\text {s i z e} \tilde {S}} \left| \left\| \hat {L} _ {:.;, t _ {0,:}} ^ {+} - K _ {:.;, t _ {0,:}} \right\| _ {\max } \leq \frac {\varepsilon}{2 d ^ {2} c _ {1} c _ {0}} \right. \right\},
$$
$$
\mathcal {E} _ {t _ {0}, -} ^ {\left(\text {k e r n e l}\right)} = \left\{\forall K: \| K \| _ {1} \leq 1, \exists S _ {3} ^ {-} \in \{0, 1 \} ^ {\text {s i z e} \tilde {S}} \left| \left\| \hat {L} _ {:,:; t _ {0,:}} ^ {-} + K _ {:,:; t _ {0,:}} \right\| _ {\max } \leq \frac {\varepsilon}{2 d ^ {2} c _ {1} c _ {0}} \right. \right\}, \text {a n d}
$$
$$
\mathcal {E} ^ {\left(\text {k e r n e l}\right)} = \left(\bigcap_ {t _ {0} \in [ c _ {0} ]} \mathcal {E} _ {t _ {0}, +} ^ {\left(\text {k e r n e l}\right)}\right) \bigcap \left(\bigcap_ {t _ {0} \in [ c _ {0} ]} \mathcal {E} _ {t _ {0}, -} ^ {\left(\text {k e r n e l}\right)}\right).
$$
Then, by Corollary 19,
$$
\Pr \left(\mathcal {E} _ {t _ {0}, +} ^ {\text {(k e r n e l)}} \Big | \Big | I _ {+} ^ {(t _ {0})} \Big | > \frac {n}{3}\right) \geq 1 - \frac {\varepsilon}{4 c _ {0}}, \text {a n d}
$$
$$
\Pr \left(\mathcal {E} _ {t _ {0}, -} ^ {\text {(k e r n e l)}} \Big | \Big | I _ {-} ^ {(t _ {0})} \Big | > \frac {n}{3}\right) \geq 1 - \frac {\varepsilon}{4 c _ {0}}.
$$
By the union bound, we have the following:
$$
\begin{array}{l} \Pr \left(\mathcal {E} ^ {\left(\text {k e r n e l}\right)} \mid \left| I _ {+} ^ {\left(t _ {0}\right)} \right|, \left| I _ {-} ^ {\left(t _ {0}\right)} \right| > \frac {n}{3}\right) \\ = 1 - \Pr \left(\left(\bigcup_ {t _ {0} \in [ c _ {0} ]} \overline {{\mathcal {E}}} _ {t _ {0}, +} ^ {\text {(k e r n e l)}}\right) \bigcup \left(\bigcup_ {t _ {0} \in [ c _ {0} ]} \overline {{\mathcal {E}}} _ {t _ {0}, -} ^ {\text {(k e r n e l)}}\right) \Bigg | \left| I _ {+} ^ {(t _ {0})} \right|, \left| I _ {-} ^ {(t _ {0})} \right| > \frac {n}{3}\right) \\ \geq 1 - \sum_ {t _ {0} \in [ c _ {0} ]} \left[ \Pr \left(\overline {{\mathcal {E}}} _ {t _ {0}, +} ^ {\text {(k e r n e l)}} \mid \left| I _ {+} ^ {(t _ {0})} \right|, \left| I _ {-} ^ {(t _ {0})} \right| > \frac {n}{3}\right) + \Pr \left(\overline {{\mathcal {E}}} _ {t _ {0}, -} ^ {\text {(k e r n e l)}} \mid \left| I _ {+} ^ {(t _ {0})} \right|, \left| I _ {-} ^ {(t _ {0})} \right| > \frac {n}{3}\right) \right] \\ \geq 1 - 2 \sum_ {t _ {0} \in [ c _ {0} ]} \frac {\varepsilon}{4 c _ {0}} \\ \geq 1 - \frac {\varepsilon}{2}. \\ \end{array}
$$
Since  $\operatorname*{Pr}\left(\left|I_{+}^{(t_0)}\right|$ ,  $\left|I_{-}^{(t_0)}\right| > \frac{n}{3}\right) \geq 1 - \frac{\varepsilon}{2}$ , then we can remove the conditional event obtaining
$$
\Pr \left(\mathcal {E} ^ {(\text {k e r n e l})}\right) \geq \Pr \left( \right.\mathcal {E} ^ {(\text {k e r n e l})} \left| \right.\left| I _ {+} ^ {(t _ {0})} \right|, \left| I _ {-} ^ {(t _ {0})} \right| > \frac {n}{3}\left. \right) \Pr \left(\left| I _ {+} ^ {(t _ {0})} \right|, \left| I _ {-} ^ {(t _ {0})} \right| > \frac {n}{3}\right)
$$
7Notice that excluding zero entries implies conditioning on the event that the entry is not zero. However, such an event has zero probability and, thus, does not impact the analysis.
The negative sign in front of  $\hat{L}_{:,t_0,:\}^-$  does not affect the random subset sum result as each entry is independently negative or positive with the same probability.
$$
\begin{array}{l} \geq \left(1 - \frac {\varepsilon}{2}\right) ^ {2} \\ \geq 1 - \varepsilon . \tag {31} \\ \end{array}
$$
To rewrite the latter in terms of the filter  $K$  and a mask  $\hat{S}$ , we notice that pruning  $L_{\vdots ;,t_0,:}^+$  and  $L_{\vdots ;,t_0,:}^-$  separately, with two binary masks, is equivalent to say that there exists a single binary mask  $S_{3} \in \{0,1\}^{\mathrm{size}\tilde{S}}$  such that,  $\hat{L}_{\vdots ;,t_0,:}$  can be written as  $\hat{L}_{\vdots ;,t_0,:} = \sum_{k \in [nc_0]} U_{\vdots ;,k,:} \cdot \hat{V}_{1,1,t_0,k}$ , where  $\hat{V} = \tilde{V} \odot S_3$ . Ineq. 31 implies that, with probability  $1 - \varepsilon$ , for all target filters  $K$  with  $\| K \|_1 \leq 1$  such  $S_3$  exists and hence,
$$
\left\| K - \hat {L} ^ {+} \right\| _ {\max } + \left\| K + \hat {L} ^ {-} \right\| _ {\max } \leq \frac {\varepsilon}{d ^ {2} c _ {1} c _ {0}}. \tag {32}
$$
Let  $\hat{S} = \tilde{S} \odot S_3$ ; then  $\hat{S} = S_1 \odot S_2 \odot S_3$  is the combination of a  $2n$ -channel-blocked mask  $S = S_1$  and a mask  $S_2 \odot S_3$  that only removes filters.
Also, whenever such binary mask  $\hat{S}$  exists, we have that
$$
\begin{array}{l} \sup  _ {X: \| X \| _ {\max } \leq M} \left\| K * X - N _ {0} ^ {(\hat {S})} (X) \right\| _ {\max } \\ = \sup  _ {X: \| X \| _ {\max } \leq M} \left\| K * X - U * \phi \left(\left(V \odot \hat {S}\right) * X\right) \right\| _ {\max } \\ = \sup  _ {X: \| X \| _ {\max } \leq M} \left\| K * X - U * \phi \left(\left(V \odot \tilde {S} \odot S _ {3}\right) * X\right) \right\| _ {\max } \\ = \sup  _ {X: \| X \| _ {\max } \leq M} \left\| K * \left(X ^ {+} - X ^ {-}\right) - U * \left(\left(\hat {V} ^ {+} * X ^ {+}\right) + \left(\hat {V} ^ {-} * X ^ {-}\right)\right) \right\| _ {\max }, \\ \end{array}
$$
where the latter holds by Lemma 11. Then, by the distributive property of the convolution and the triangle inequality,
$$
\begin{array}{l} \sup  _ {X: \| X \| _ {\max } \leq M} \left\| K * \left(X ^ {+} - X ^ {-}\right) - U * \left(\left(\hat {V} ^ {+} * X ^ {+}\right) + \left(\hat {V} ^ {-} * X ^ {-}\right)\right) \right\| _ {\max } \\ = \sup  _ {X: \| X \| _ {\max } \leq M} \left\| K * X ^ {+} - U * \left(\hat {V} ^ {+} * X ^ {+}\right) - K * X ^ {-} - U * \left(\hat {V} ^ {-} * X ^ {-}\right) \right\| _ {\max } \\ \leq \sup  _ {X: \| X \| _ {\max } \leq M} \left\| K * X ^ {+} - U * \left(\hat {V} ^ {+} * X ^ {+}\right) \right\| _ {\max } \\ + \sup  _ {X: \| X \| _ {\max } \leq M} \left\| K * X ^ {-} + U * \left(\hat {V} ^ {-} * X ^ {-}\right) \right\| _ {\max }. \\ \end{array}
$$
One can now apply the Tensor Convolution Inequality (Lemma 20) and obtain
$$
\begin{array}{l} \sup  _ {X: \| X \| _ {\max } \leq M} \left\| K * X ^ {+} - U * \left(\hat {V} ^ {+} * X ^ {+}\right) \right\| _ {\max } \\ + \sup  _ {X: \| X \| _ {\max } \leq M} \left\| K * X ^ {-} + U * \left(\hat {V} ^ {-} * X ^ {-}\right) \right\| _ {\max } \\ \leq \sup  _ {X: \| X \| _ {\max } \leq M} \left\| X ^ {+} \right\| _ {\max } \cdot \left\| K - U * \hat {V} ^ {+} \right\| _ {1} \\ + \sup  _ {X: \| X \| _ {\max } \leq M} \left\| X ^ {-} \right\| _ {\max } \cdot \left\| K + U * \hat {V} ^ {-} \right\| _ {1} \\ = M \cdot \left\| K - U * \hat {V} ^ {+} \right\| _ {1} + M \cdot \left\| K + U * \hat {V} ^ {-} \right\| _ {1}. \\ \end{array}
$$
Now, observing that the number of entries of the two tensors in the expression above is  $d^2 c_1 c_0$ , and using Ineq. 32, we get that
$$
M \cdot \left\| K - U * \hat {V} ^ {+} \right\| _ {1} + M \cdot \left\| K - U * \hat {V} ^ {-} \right\| _ {1}
$$
9The presence of  $S_{3}$  does not influence the proof of Lemma 11.
$$
\begin{array}{l} \leq d ^ {2} c _ {1} c _ {0} \left(\left\| K - U * \hat {V} ^ {+} \right\| _ {\max } + \left\| K - U * \hat {V} ^ {-} \right\| _ {\max }\right) \\ \leq d ^ {2} c _ {1} c _ {0} M \frac {\varepsilon}{d ^ {2} c _ {1} c _ {0}} \\ = \varepsilon M. \\ \end{array}
$$
proving the thesis.
Proof of Theorem 2. To bound the error propagation across layers, we define the layers' outputs
$$
\begin{array}{l} X ^ {(0)} = X, \\ X ^ {(i)} = \phi \left(K ^ {(i)} * X ^ {(i - 1)}\right) \quad \text {f o r} 1 \leq i \leq \ell . \tag {33} \\ \end{array}
$$
Notice that  $X^{(\ell)}$  is the output of the target function, i.e.,  $f(X) = X^{(\ell)}$ .
For brevity's sake, given masks  $S^{(2i - 1)} \in \{0,1\}^{\mathrm{size}(L^{(2i - 1)})}$  for each  $i = 1,\dots ,\ell$ , let us denote
$$
\tilde {L} ^ {(2 i - 1)} = L ^ {(2 i - 1)} \odot S ^ {(2 i - 1)}. \tag {34}
$$
We now show that the random network approximates the target one after being pruned only at layers  $L^{(2i - 1)}$  for all  $i = 1,\dots ,\ell$ , by masks  $S^{(2i - 1)}$  which are the composition of a  $2n_{2i - 1}$ -channel-blocked mask and a mask that only removes filters. The key idea is to iteratively applying Lemma 12. Since the ReLU function is 1-Lipschitz, for all  $i$  it holds
$$
\begin{array}{l} \left\| \phi \left(K ^ {(i)} * X ^ {(i - 1)}\right) - \phi \left(L ^ {(2 i)} * \phi \left(\tilde {L} ^ {(2 i - 1)} * X ^ {(i - 1)}\right)\right) \right\| _ {\max } \\ \leq \left\| K ^ {(i)} * X ^ {(i - 1)} - L ^ {(2 i)} * \phi \left(\tilde {L} ^ {(2 i - 1)} * X ^ {(i - 1)}\right) \right\| _ {\max } \tag {35} \\ \end{array}
$$
The key step of the proof is that, for each layer  $i$ , since  $n_i \geq C d^{13} c_i^6 \log^3 \frac{d^2 c_i c_{i-1} \ell}{\varepsilon}$  for a suitable constant  $C$ , we can apply Lemma 12 to get that, with probability at least  $1 - \frac{\varepsilon}{2 \ell}$ , for all  $i$  and for all choices of target filters  $K^{(i)} \in \mathbb{R}^{d_i \times d_i \times c_{i-1} \times c_i}$  it holds
$$
\left\| K ^ {(i)} * X ^ {(i - 1)} - L ^ {(2 i)} * \phi \left(\tilde {L} ^ {(2 i - 1)} * X ^ {(i - 1)}\right) \right\| _ {\max } <   \frac {\varepsilon}{2 \ell} \cdot \left\| X ^ {(i - 1)} \right\| _ {\max }. \tag {36}
$$
Hence, combining Eq. 35 and Eq. 36 we get that, with probability at least  $1 - \frac{\varepsilon}{2\ell}$ , for all  $i$  and all choices of target filters  $K^{(i)} \in \mathbb{R}^{d_i \times d_i \times c_{i-1} \times c_i}$  with  $\left\| K^{(i)} \right\|_1 \leq 1$ ,
$$
\begin{array}{l} \left\| \phi \left(K ^ {(i)} * X ^ {(i - 1)}\right) - \phi \left(L ^ {(2 i)} * \phi \left(\tilde {L} ^ {(2 i - 1)} * X ^ {(i - 1)}\right)\right) \right\| _ {\max } \\ <   \frac {\varepsilon}{2 \ell} \cdot \left\| X ^ {(i - 1)} \right\| _ {\max } \tag {37} \\ \end{array}
$$
By a union bound, with probability at least  $1 - \varepsilon$ , we get that Eq. 37 holds for all layers  $i$  and all choices of target filters  $K^{(i)} \in \mathbb{R}^{d_i \times d_i \times c_{i-1} \times c_i}$ .
Analogously, we can define the pruned layers' outputs
$$
\tilde {X} ^ {(0)} = X,
$$
$$
\tilde {X} ^ {(i)} = \phi \left(L ^ {(2 i)} * \phi \left(\tilde {L} ^ {(2 i - 1)} * \tilde {X} ^ {(i - 1)}\right)\right) \quad \text {f o r} 1 \leq i \leq \ell . \tag {38}
$$
Notice that  $\tilde{X}^{(\ell)}$  is the output of the pruned network, i.e.  $N_0^{(S^{(1)},\dots,S^{(2\ell)})}(X) = \tilde{X}^{(\ell)}$ .
By the same reasoning employed to derive Eq. 36 and Eq. 37 we have that, with probability  $1 - \varepsilon$ , for all layers  $i$  and all choices of target filters  $K^{(i)} \in \mathbb{R}^{d_i \times d_i \times c_{i-1} \times c_i}$ , the output of all pruned layers satisfies
$$
\begin{array}{l} \left\| \phi \left(K ^ {(i)} * \tilde {X} ^ {(i - 1)}\right) - \phi \left(L ^ {(2 i)} * \phi \left(\tilde {L} ^ {(2 i - 1)} * \tilde {X} ^ {(i - 1)}\right)\right) \right\| _ {\max } \\ <   \frac {\varepsilon}{2 \ell} \cdot \left\| \tilde {X} ^ {(i - 1)} \right\| _ {\max } \tag {39} \\ \end{array}
$$
Moreover, for each  $1 \leq i \leq \ell - 1$ , by the triangle inequality and by Eq. 39,
$$
\begin{array}{l} \left\| \tilde {X} ^ {(i)} \right\| _ {\max } = \left\| \tilde {X} ^ {(i)} - \phi \left(K ^ {(i)} * \tilde {X} ^ {(i - 1)}\right) + \phi \left(K ^ {(i)} * \tilde {X} ^ {(i - 1)}\right) \right\| _ {\max } \\ \leq \left\| \tilde {X} ^ {(i)} - \phi \left(K ^ {(i)} * \tilde {X} ^ {(i - 1)}\right) \right\| _ {\max } + \left\| \phi \left(K ^ {(i)} * \tilde {X} ^ {(i - 1)}\right) \right\| _ {\max } \\ \leq \frac {\varepsilon}{2 \ell} \cdot \left\| \tilde {X} ^ {(i - 1)} \right\| _ {\max } + \left\| \phi \left(K ^ {(i)} * \tilde {X} ^ {(i - i)}\right) \right\| _ {\max }. \\ \end{array}
$$
By the Lipschitz property of  $\phi$  and Lemma 20
$$
\begin{array}{l} \frac {\varepsilon}{2 \ell} \cdot \left\| \tilde {X} ^ {(i - 1)} \right\| _ {\max } + \left\| \phi \left(K ^ {(i)} * \tilde {X} ^ {(i - i)}\right) \right\| _ {\max } \\ \leq \frac {\varepsilon}{2 \ell} \cdot \left\| \tilde {X} ^ {(i - 1)} \right\| _ {\max } + \left\| K ^ {(i)} * \tilde {X} ^ {(i - i)} \right\| _ {\max } \\ \leq \frac {\varepsilon}{2 \ell} \cdot \left\| \tilde {X} ^ {(i - 1)} \right\| _ {\max } + \left\| K ^ {(i)} \right\| _ {1} \left\| \tilde {X} ^ {(i - i)} \right\| _ {\max } \\ = \left\| \tilde {X} ^ {(i - 1)} \right\| _ {\max } \left(1 + \frac {\varepsilon}{2 \ell}\right). \\ \end{array}
$$
By unrolling the recurrence, we get that, with probability  $1 - \varepsilon$ , for all choices of target filters  $K^{(i)} \in \mathbb{R}^{d_i \times d_i \times c_{i-1} \times c_i}$ ,
$$
\left\| \tilde {X} ^ {(i)} \right\| _ {\max } \leq \left\| \tilde {X} ^ {(0)} \right\| _ {\max } \left(1 + \frac {\varepsilon}{2 \ell}\right) ^ {i}. \tag {40}
$$
Thus, combining Eq. 39 and Eq. 40, with probability  $1 - \varepsilon$  we get that, for each  $i\in [\ell ]$  and for all choices of target filters  $K^{(i)}\in \mathbb{R}^{d_i\times d_i\times c_{i - 1}\times c_i}$
$$
\begin{array}{l} \left\| K ^ {(i)} * \tilde {X} ^ {(i - 1)} - L ^ {(2 i)} * \phi (\tilde {L} ^ {(2 i - 1)} * \tilde {X} ^ {(i - 1)}) \right\| _ {\max } \\ <   \frac {\varepsilon}{2 \ell} \cdot \left(1 + \frac {\varepsilon}{2 \ell}\right) ^ {i - 1} \| \tilde {X} ^ {(0)} \| _ {\max } \tag {41} \\ \end{array}
$$
We then see that with probability  $1 - \varepsilon$ , for  $1 \leq i \leq \ell$  and for all choices of target filters  $K^{(i)} \in \mathbb{R}^{d_i \times d_i \times c_{i-1} \times c_i}$ , by Eq. 33 and Eq. 38, and by the triangle inequality,
$$
\begin{array}{l} \left\| X ^ {(\ell)} - \tilde {X} ^ {(\ell)} \right\| _ {\max } \\ = \left\| \phi \left(K ^ {(\ell)} * X ^ {(\ell - 1)}\right) - \phi \left(L ^ {(2 \ell)} * \phi \left(\tilde {L} ^ {(2 \ell - 1)} * \tilde {X} ^ {(\ell - 1)}\right)\right) \right\| _ {\max } \\ \leq \left\| \phi \left(K ^ {(\ell)} * X ^ {(\ell - 1)}\right) - \phi \left(K ^ {(\ell)} * \tilde {X} ^ {(\ell - 1)}\right) \right\| _ {\max } \\ \left. \right. + \left\| \phi \left(K ^ {(\ell)} * \tilde {X} ^ {(\ell - 1)}\right) - \phi \left(L ^ {(2 \ell)} * \phi \left(\tilde {L} ^ {(2 \ell - 1)} * \tilde {X} ^ {(\ell - 1)}\right)\right)\right\| _ {\max } \cdot \\ \end{array}
$$