SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our work demonstrates how a simple modification to the proof in [14] can yield tighter generalization bounds for convolutional networks. Empirically, we have observed that our bound is reasonably tight in basic learning settings (e.g., training 3-layer neural networks on MNIST), which is promising. However, this bound is anticipated to be generally loose for more complex learning settings. One drawback of our bound is its dependence on depth. For instance, our factor  $\rho(w)$  represents a product of  $L$  weight norms, which could increase as the depth  $L$  grows. Additionally, our bound contains a product  $\prod_{l=1}^{L-1} k_l$ , which increases exponentially with depth. As demonstrated in section 3, this indicates that the bound is likely reasonable only when  $L$  depends logarithmically on the input's dimension. Consequently, our bound may not be suitable for very deep convolutional networks. Finally, building upon the analysis in [14], our bounds are specific to a certain class of ReLU neural networks and do not readily extend to architectures with different activation functions.
This paper delves into the theoretical aspects of deep learning, specifically examining the reasons behind the strong generalization capabilities of sparse neural networks. Such insights may enrich our understanding of the success of deep learning and specific architectures, such as convolutional networks [30] and MLP-mixers [34]. However, since this is a theoretical study, we do not anticipate any direct negative societal impacts stemming from our research.
8 Additional Experimental Details
Compute. Each of the runs was done using a single GPU for at most 20 hours on a computing cluster with several available GPU types (e.g., GeForce RTX 2080, GeForce RTX 2080 Ti, Quadro RTX 6000, Tesla V-100, GeForce RTX A6000, A100, and GeForce GTX 1080 Ti).
Calculating our bound. Before discussing the calculation of the bound, let us first define how we model a convolutional neural network within the framework outlined in section 2.
We focus on ReLU convolutional networks which comprise a fully-connected layer on top of a series of convolutional layers. The  $l$ th convolutional layer possesses a kernel size of  $k_{l}$ . Each neuron in this  $l$ th layer accepts  $k_{l}$  neurons from its predecessor layer as input (therefore its degree is  $\deg(G)_l = k_l$ ), each with dimension  $c_{l-1}$ , and outputs a single output node of dimension  $c_l$ . The top linear layer  $L$  following the series of convolutional layers receives a set of  $k_L$  nodes as input (this represents the image resultant from applying multiple convolutional layers), where each node is of the output channel dimension of the previous layer. The output of this layer is of dimension  $k$ .
The bound derived in Theorem 3.2 of the main text can be written as follows:
$$
\begin{array}{l} \frac {2 \sqrt {2} (\rho (w) + 1)}{m} \cdot \left(1 + \sqrt {2 (\log (2) L + \sum_ {l = 1} ^ {L - 1} \log (\deg (G) _ {l}) + \log (C))}\right) \\ \cdot \sqrt {\max  _ {j _ {0} , \dots , j _ {L}} \prod_ {l = 1} ^ {L - 1} | \operatorname * {p r e d} (l , j _ {l}) | \cdot \sum_ {i = 1} ^ {m} \| z _ {j _ {0}} ^ {0} (x _ {i}) \| _ {2} ^ {2}} + 3 \sqrt {\frac {\log (2 (\rho (w) + 2) ^ {2} / \delta)}{2 m}}, \\ \end{array}
$$
To compute this bound, we set  $\delta = 0.001$  (the bound holds uniformly with probability  $1 - \delta$ ) and  $\gamma = 1$ . The norm  $\rho(w)$  is calculated as the product of the norms of each one of the network's layers  $\|w^l\|_F$ . The norm of each of these layers is simply the Frobenius norm of the kernel tensor of the layer. For the last fully-connected layer we calculate the norm of the full weight matrix. For the  $l$ th convolutional layer,  $k_l = |\operatorname{pred}(l,j_l)| = \deg(G)_l$  is the kernel's size. For instance, if the layer is a  $3\times 3$  or  $2\times 2$  convolutional layer, the size is 9 or 4, respectively. It is worth noting that the bound is independent of the size of the image  $k_L$  that the fully-connected layer receives as input.
Finally, to calculate  $\max_{j_1,\ldots ,j_L}\sum_{i = 1}^m\| z_{j_0}^0 (x_i)\| _2^2$  with  $j_{l - 1}\in \operatorname {pred}(l,j_l)$ . It is important to note that this term is bounded by  $\max_{j_0}\sum_{i = 1}^{m}\| z_{j_0}^0 (x_i)\| _2^2$ , where the maximum is taken over all input patches. To calculate this quantity, we sum the squared norms of the  $j_0$ th pixel across all samples and return the maximum value across all possible  $j_0$  choices.
9 Proofs
Lemma 2.2. Let  $P$  be a distribution over  $\mathbb{R}^{c_0d_0} \times [C]$  and  $\mathcal{F} \subset \{f': \mathcal{X} \to \mathbb{R}^C\}$ . Let  $S = \{(x_i, y_i)\}_{i=1}^m$  be a dataset of i.i.d. samples selected from  $P$  and  $X = \{x_i\}_{i=1}^m$ . Then, with probability at least  $1 - \delta$  over the selection of  $S$ , for any  $f_w \in \mathcal{F}$ , we have
Proof. By Lemma 3.1 in [17], with probability at least  $1 - \delta$  over the selection of  $S$ , for any  $f_w \in \mathcal{F}$  we have
$$
\operatorname {e r r} _ {P} \left(f _ {w}\right) - \operatorname {e r r} _ {S} ^ {\gamma} \left(f _ {w}\right) \leq 2 \mathcal {R} _ {X} \left(\mathcal {F} _ {| \gamma}\right) + 3 \sqrt {\frac {\log (2 / \delta)}{2 m}},
$$
where  $\mathcal{F}_{|\gamma} = \{(x,y)\mapsto \ell_{\gamma}(-\mathcal{M}(f_w(x),y))\mid f_w\in \mathcal{F}\}$ $\ell_{\gamma}(u,y)\coloneqq \min (1,\max (0,1 + \frac{\mathcal{M}(u,y)}{\gamma}))$  is the ramp loss function and  $\mathcal{M}(u,j) = u_j - \max_{i\neq j}u_i$  (these quantities are standard [53, 54, 55]). Next, by Lemma A.3 in [17],  $\ell_{\gamma}(\mathcal{M}(\cdot ,r))$  is a  $(2 / \gamma)$ -Lipschitz function. Hence, by Corollary 1 in [56],
$$
\mathcal {R} _ {X} (\mathcal {F} _ {| \gamma}) \leq \frac {2 \sqrt {2}}{\gamma m} \cdot \mathbb {E} _ {\xi_ {i r} \sim U [ \pm 1 ]} \left[ \sup _ {f _ {w}} \sum_ {i, r} \xi_ {i r} f _ {w} (x _ {i}) _ {r} \right] = \frac {2 \sqrt {2}}{\gamma} \cdot \mathcal {R} _ {X} (\mathcal {F})
$$
Proof. Let  $W \in \mathbb{R}^{h \times p}$  be a matrix and let  $w_{1}, \ldots, w_{h}$  be the rows of the matrix  $W$ . Define a function  $Q_{j}(v) \coloneqq \left( \sum_{i=1}^{m} \xi_{i} \cdot \sigma \left( \frac{v^{\top}}{\|v\|_{2}} f_{j}(x_{i}) \right) \right)^{2}$  taking a vector  $v \in \mathbb{R}^{p}$ . We notice that
$$
\begin{array}{l} \sum_ {j = 1} ^ {q} \left\| \sum_ {i = 1} ^ {m} \xi_ {i} \cdot \sigma (W _ {j} f _ {j} (x _ {i})) \right\| _ {2} ^ {2} = \sum_ {j = 1} ^ {q} \sum_ {r = 1} ^ {h} \| w _ {j r} \| _ {2} ^ {2} \left(\sum_ {i = 1} ^ {m} \xi_ {i} \cdot \sigma (\frac {w _ {j r} ^ {\top}}{\| w _ {j r} \| _ {2}} f _ {j} (x _ {i}))\right) ^ {2} \\ = \sum_ {j = 1} ^ {q} \sum_ {r = 1} ^ {h} \| w _ {j r} \| _ {2} ^ {2} \cdot Q _ {j} (w _ {j r}). \\ \end{array}
$$
For any  $w_{j1},\ldots ,w_{jh}$  , we have
$$
\sum_ {r = 1} ^ {h} \| w _ {j r} \| _ {2} ^ {2} \cdot Q _ {j} \left(w _ {j r}\right) \leq R \cdot \max  _ {r} Q _ {j} \left(w _ {j r}\right), \tag {4}
$$
which is obtained for  $\hat{w}_{j1},\dots,\hat{w}_{jh}$ , where  $\hat{w}_{ji} = 0$  for all  $i\neq r^{*}$  and  $\hat{w}_{jr^*}$  of norm  $R$  for some  $r^*\in [h]$ . Together with the fact that  $g$  is a monotonically increasing function, we obtain
$$
\begin{array}{l} \mathbb{E}_{\xi}\sup_{\substack{f\in \mathcal{F}\\ W_{j}\colon \| W_{j}\|_{F}\leq R}}g\left(\sqrt{\sum_{j = 1}^{q}\left\|\sum_{i = 1}^{m}\xi_{i}\cdot\sigma(W_{j}f_{j}(x_{i}))\right\|_{2}^{2}}\right) \\ \leq \mathbb{E}_{\xi}\sup_{\substack{f\in \mathcal{F}\\ w_{1}\ldots ,w_{q}:\| w_{j}\|_{2} = R}}g\left(\sqrt{\sum_{j = 1}^{q}\big|\sum_{i = 1}^{m}\xi_{i}\cdot\sigma(w_{j}^{\top}f_{j}(x_{i}))\big|^{2}}\right) \\ \leq \mathbb{E}_{\xi}\sup_{\substack{j\in [q], f\in \mathcal{F}\\ w_{1}\ldots ,w_{q}:\| w_{j}\|_{2} = R}}g\left(\sqrt{q\cdot\big|\sum_{i = 1}^{m}\xi_{i}\cdot\sigma(w_{j}^{\top}f_{j}(x_{i}))\big|^{2}}\right) \\ = \mathbb{E}_{\xi}\sup_{\substack{j\in [q],  f\in \mathcal{F}\\ w:  \| w\|_{2} = R}}g\left(\sqrt{q}\cdot \big|\sum_{i = 1}^{m}\xi_{i}\cdot \sigma (w^{\top}f_{j}(x_{i}))\big|\right). \\ \end{array}
$$
Since  $g(|z|)\leq g(z) + g(-z)$
$$
\begin{array}{l} \mathbb{E}_{\substack{\xi \\ j\in [q], f\in \mathcal{F}\\ w: \| w\|_{2} = R}}\sup_{\substack{f\in \mathcal{F}\\ }}g\left(\sqrt{q}\cdot \big|\sum_{i = 1}^{m}\xi_{i}\cdot \sigma \big(w^{\top}f_{j}(x_{i})\big)\big|\right) \\ \leq \mathbb{E}_{\substack{j\in [q],  f\in \mathcal{F}\\ w:  \| w\|_{2} = R}}\sup_{f\in \mathcal{F}}g\left(\sqrt{q}\cdot \sum_{i = 1}^{m}\xi_{i}\cdot \sigma (w^{\top}f_{j}(x_{i}))\right) \\ + \mathbb{E}_{\substack{j\in [q], f\in \mathcal{F}\\ w: \| w\|_{2} = R}}g\left(-\sqrt{q}\cdot \sum_{i = 1}^{m}\xi_{i}\cdot \sigma (w^{\top}f_{j}(x_{i}))\right) \\ = 2\mathbb{E}_{\substack{j\in [q], f\in \mathcal{F}\\ w: \| w\|_{2} = R}}\sup_{\substack{f\in \mathcal{F}\\ }}g\left(\sqrt{q}\cdot \sum_{i = 1}^{m}\xi_{i}\cdot \sigma \big(w^{\top}f_{j}(x_{i})\big)\right), \\ \end{array}
$$
where the last equality follows from the symmetry in the distribution of the  $\xi_{i}$  random variables. By Equation 4.20 in [51] and Cauchy-Schwartz, we have the following:
$$
\begin{array}{l} \mathbb{E}_{\xi}\sup_{\substack{j\in [q],  f\in \mathcal{F}\\ w:  \| w\|_{2} = R}}g\left(\sqrt{q}\cdot \sum_{i = 1}^{m}\xi_{i}\cdot \sigma (w^{\top}f_{j}(x_{i}))\right) \\ \leq \mathbb{E}_{\xi}\sup_{\substack{j\in [q],  f\in \mathcal{F}\\ w:  \| w\|_{2} = R}}g\left(\sqrt{q}\cdot \sum_{i = 1}^{m}\xi_{i}\cdot w^{\top}f_{j}(x_{i})\right) \\ \leq \mathbb{E}_{\substack{j\in [q],\\ w:  \| w\|_{2} = R}}\sup_{f\in \mathcal{F}}g\left(\sqrt{q}\cdot \| w\|_{2}\left\| \sum_{i = 1}^{m}\xi_{i}\cdot f_{j}(x_{i})\right\|_{2}\right) \\ \leq \mathbb {E} _ {\xi} \sup  _ {j \in [ q ], f \in \mathcal {F}} g \left(\sqrt {q} R \left\| \sum_ {i = 1} ^ {m} \xi_ {i} \cdot f _ {j} (x _ {i}) \right\| _ {2}\right). \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {R} _ {X} \left(\mathcal {F} _ {G, \rho}\right) \leq \frac {\rho}{m} \cdot \left(1 + \sqrt {2 (\log (2) L + \sum_ {l = 1} ^ {L - 1} \log (\deg (G) _ {l}) + \log (C))}\right) \\ \cdot \sqrt {\max  _ {j _ {0} , \dots , j _ {L}} \prod_ {l = 1} ^ {L - 1} | \operatorname {p r e d} (l , j _ {l}) | \cdot \sum_ {i = 1} ^ {m} \| z _ {j _ {0}} ^ {0} (x _ {i}) \| _ {2} ^ {2}}, \\ \end{array}
$$
Proof. We denote by  $f_{w}$  an arbitrary member of  $\mathcal{F}_{G,\rho}$  and  $w_{jl}^{l}$  the weights of the  $j_{l}$ th neuron of the  $l$ th layer. Due to the homogeneity of the ReLU function, each function  $f_{w} \in \mathcal{F}_{G,\rho}$  can be rewritten as  $f_{\hat{w}}$ , where  $\hat{w}^{L} \coloneqq \rho \frac{w^{L}}{\|w^{L}\|_{F}}$  and for all  $l < L$  and  $j_{l} \in [d_{l}]$ ,  $\hat{w}_{j_{l}}^{l} \coloneqq \frac{w_{j_{l}}^{l}}{\max_{j \in [d_{l}]}\|w_{j}^{l}\|_{F}}$ . In particular, we have  $\mathcal{F}_{G,\rho} \subset \hat{\mathcal{F}}_{G,\rho} \coloneqq \{f_{w} \mid \| w^{L}\|_{F} \leq \rho$  and  $\forall i < L, j_{l} \in [d_{l}] : \| w_{j_{l}}^{l}\|_{F} \leq 1\}$  since the ReLU function is homogeneous.
We first consider that
$$
\begin{array}{l} m \mathcal {R} := m \mathcal {R} _ {X} (\hat {\mathcal {F}} _ {G, \rho}) = \mathbb {E} _ {\xi} \left[ \sup  _ {\hat {w}} \sum_ {i = 1} ^ {m} \sum_ {r = 1} ^ {C} \xi_ {i r} f _ {\hat {w}} (x _ {i}) _ {r} \right] \\ = \mathbb {E} _ {\xi} \left[ \sup  _ {\hat {w}} \sum_ {i = 1} ^ {m} \sum_ {r = 1} ^ {C} \xi_ {i r} \sum_ {j _ {L - 1}} \langle \hat {w} _ {r j _ {L - 1}} ^ {L}, z _ {j _ {L - 1}} ^ {L - 1} (x _ {i}) \rangle \right] \\ = \mathbb {E} _ {\xi} \left[ \sup  _ {\hat {w}} \sum_ {j _ {L - 1}} \sum_ {r = 1} ^ {C} \left(\hat {w} _ {r j _ {L - 1}} ^ {L}\right) ^ {\top} \sum_ {i = 1} ^ {m} \xi_ {i r} z _ {j _ {L - 1}} ^ {L - 1} \left(x _ {i}\right) \right] \\ \leq \rho \cdot \mathbb {E} _ {\xi} \left[ \sup  _ {\hat {w}} \max  _ {r, j _ {L - 1}} \left\| \sum_ {i = 1} ^ {m} \xi_ {i r} z _ {j _ {L - 1}} ^ {L - 1} (x _ {i}) \right\| _ {2} \right] \\ \end{array}
$$
where the last inequality follows from moving the norm of  $\hat{w}^L$  to the vector  $\hat{w}_{rj_{L-1}}^L$  for maximizing the term  $\sum_{j_{L-1}}\sum_{r=1}^{C}(\hat{w}_{rj_{L-1}}^L)^\top\sum_{i=1}^{m}\xi_{ir}z_{j_{L-1}}^{L-1}(x_i)$  and applying the Cauchy-Schwartz inequality. Next, we apply Jensen's inequality,
$$
\begin{array}{l} \mathbb {E} _ {\xi} \left[ \sup  _ {\hat {w}} \max  _ {r, j _ {L - 1}} \left\| \sum_ {i = 1} ^ {m} \xi_ {i r} z _ {j _ {L - 1}} ^ {L - 1} (x _ {i}) \right\| _ {2} \right] \leq \frac {1}{\lambda} \cdot \log \mathbb {E} _ {\xi} \sup  _ {\hat {w}, r, j _ {L - 1}} \exp \left(\lambda \left\| \sum_ {i = 1} ^ {m} \xi_ {i r} z _ {j _ {L - 1}} ^ {L - 1} (x _ {i}) \right\| _ {2}\right) \\ \leq \frac {1}{\lambda} \cdot \log \mathbb {E} _ {\hat {w}, r, j _ {L - 1}} \exp \left(\lambda \sqrt {\left\| \sum_ {i = 1} ^ {m} \xi_ {i r} z _ {j _ {L - 1}} ^ {L - 1} (x _ {i}) \right\| _ {2} ^ {2}}\right), \\ \end{array}
$$
where the supremum is taken over the weights  $\hat{w}_{j_l}^l$  ( $l \in [L]$ ,  $j_l \in [d_l]$ ) that are described above. Next, we use Lemma 3.4,
$$
\begin{array}{l} m \mathcal {R} \leq \frac {\rho}{\lambda} \cdot \log \left(\mathbb {E} _ {\xi} \sup  _ {\hat {w}, r, j _ {L - 1}} \exp \left(\lambda \cdot \sqrt {\left\| \sum_ {i = 1} ^ {m} \xi_ {i r} \cdot z _ {j _ {L - 1}} ^ {L - 1} (x _ {i}) \right\| _ {2} ^ {2}}\right)\right) \\ = \frac {\rho}{\lambda} \cdot \log \left(\mathbb {E} _ {\xi} \sup  _ {\hat {w}, r, j _ {L - 1}} \exp \left(\lambda \cdot \sqrt {\left\| \sum_ {i = 1} ^ {m} \xi_ {i r} \cdot \sigma \left(\hat {w} _ {j _ {L - 1}} ^ {L - 1} v _ {j _ {L - 1}} ^ {L - 2} (x _ {i})\right) \right\| _ {2} ^ {2}}\right)\right) \\ \leq \frac {\rho}{\lambda} \cdot \log \left(2 \mathbb {E} _ {\xi} \sup  _ {\hat {w}, r, j _ {L - 1}} \exp \left(\lambda \cdot \sqrt {\left\| \sum_ {i = 1} ^ {m} \xi_ {i r} \cdot v _ {j _ {L - 1}} ^ {L - 2} (x _ {i}) \right\| _ {2} ^ {2}}\right)\right) \\ = \frac {\rho}{\lambda} \cdot \log \left(2 \mathbb {E} _ {\xi} \sup  _ {\hat {w}, r, j _ {L - 1}} \exp \left(\lambda \cdot \sqrt {\sum_ {j _ {L - 2} \in \operatorname* {p r e d} (L - 1 , j _ {L - 1})} ^ {m} \left\| \sum_ {i = 1} ^ {m} \xi_ {i r} \cdot \sigma \left(\hat {w} _ {j _ {L - 2}} ^ {L - 2} v _ {j _ {L - 2}} ^ {L - 3} (x _ {i})\right) \right\| _ {2} ^ {2}}\right)\right) \\ \leq \frac {\rho}{\lambda} \cdot \log \left(4 \mathbb {E} _ {\xi} \sup  _ {\hat {w}, r, j _ {L - 1}, j _ {L - 2}} \exp \left(\lambda \cdot \sqrt {\left| \operatorname* {p r e d} (L - 1 , j _ {L - 1}) \right| \left\| \sum_ {i = 1} ^ {m} \xi_ {i r} \cdot v _ {j _ {L - 2}} ^ {L - 3} \left(x _ {i}\right) \right\| _ {2} ^ {2}}\right)\right), \\ \end{array}
$$
where the supremum is taken over the parameters of  $f_{\hat{w}}$  and  $j_1 \in \operatorname{pred}(L, j_0)$ . By applying this process recursively  $L$  times, we obtain the following inequality,
$$
m \mathcal {R} \leq \frac {\rho}{\lambda} \cdot \log \left(2 ^ {L} \mathbb {E} _ {\xi} \max  _ {r, j _ {0}, \dots , j _ {L}} \exp \left(\lambda \cdot \sqrt {\prod_ {l = 1} ^ {L - 1} | \operatorname {p r e d} (l , j _ {l}) |} \cdot \left\| \sum_ {i = 1} ^ {m} \xi_ {i r} \cdot z _ {j _ {0}} ^ {0} (x _ {i}) \right\| _ {2}\right)\right), \tag {5}
$$
where the maximum is taken over  $j_0, \ldots, j_L$ , such that,  $j_{l-1} \in \operatorname{pred}(l, j_l)$  and  $r \in [C]$ . We notice that
$$
\begin{array}{l} \mathbb {E} _ {\xi} \max  _ {r, j _ {0}, \dots , j _ {L}} \exp \left(\lambda \cdot \sqrt {\prod_ {l = 1} ^ {L - 1} | \operatorname {p r e d} (l , j _ {l}) |} \cdot \left\| \sum_ {i = 1} ^ {m} \xi_ {i r} \cdot z _ {j _ {0}} ^ {0} \left(x _ {i}\right) \right\| _ {2}\right) \\ \leq \sum_ {r, j _ {0}, \dots , j _ {L}} \mathbb {E} _ {\xi} \exp \left(\lambda \cdot \sqrt {\prod_ {l = 1} ^ {L - 1} | \operatorname {p r e d} (l , j _ {l}) |} \cdot \left\| \sum_ {i = 1} ^ {m} \xi_ {i r} \cdot z _ {j _ {0}} ^ {0} \left(x _ {i}\right) \right\| _ {2}\right) \tag {6} \\ \leq C \prod_ {l = 1} ^ {L - 1} \deg (G) _ {l} \cdot \max  _ {r, j _ {0}, \dots , j _ {L}} \mathbb {E} _ {\xi} \exp \left(\lambda \cdot \sqrt {\prod_ {l = 1} ^ {L - 1} | \operatorname* {p r e d} (l , j _ {l}) |} \cdot \left\| \sum_ {i = 1} ^ {m} \xi_ {i r} \cdot z _ {j _ {0}} ^ {0} (x _ {i}) \right\| _ {2}\right). \\ \end{array}
$$
Following the proof of Theorem 1 in [14], by applying Jensen's inequality and Theorem 6.2 in [57] we obtain that for any  $\alpha >0$
$$
\mathbb {E} _ {\xi} \exp \left(\alpha \left\| \sum_ {i = 1} ^ {m} \xi_ {i r} \cdot z _ {j _ {0}} ^ {0} (x _ {i}) \right\| _ {2}\right) \leq \exp \left(\frac {\alpha^ {2} \sum_ {i = 1} ^ {m} \| z _ {j _ {0}} ^ {0} (x _ {i}) \| _ {2} ^ {2}}{2} + \alpha \sqrt {\sum_ {i = 1} ^ {m} \| z _ {j _ {0}} ^ {0} (x _ {i}) \| _ {2} ^ {2}}\right). \tag {7}
$$
Hence, by combining equations 5-7 with  $\alpha = \lambda \cdot \sqrt{\prod_{l=1}^{L-1} |\operatorname{pred}(l,j_l)|}$ , we obtain that
$$
\begin{array}{l} m \mathcal {R} \leq \frac {\rho}{\lambda} \cdot \log \left(2 ^ {L} C \prod_ {l = 1} ^ {L - 1} \deg (G) _ {l} \cdot \max  _ {r, j _ {0}, \dots , j _ {L}} \mathbb {E} _ {\xi} \exp \left(\lambda \cdot \sqrt {\prod_ {l = 1} ^ {L - 1} | \operatorname* {p r e d} (l , j _ {l}) |} \cdot \left\| \sum_ {i = 1} ^ {m} \xi_ {i r} \cdot z _ {j _ {0}} ^ {0} (x _ {i}) \right\| _ {2}\right)\right) \\ = \frac {\rho}{\lambda} \cdot \max  _ {j _ {0}, \dots , j _ {L}} \log \left(2 ^ {L} C \prod_ {l = 1} ^ {L - 1} \deg (G) _ {l} \cdot \mathbb {E} _ {\xi} \exp \left(\lambda \cdot \sqrt {\prod_ {l = 1} ^ {L - 1} | \operatorname* {p r e d} (l , j _ {l}) |} \cdot \left\| \sum_ {i = 1} ^ {m} \xi_ {i r} \cdot z _ {j _ {0}} ^ {0} (x _ {i}) \right\| _ {2}\right)\right) \\ \leq \frac {\rho \cdot (\log (2) L + \sum_ {l = 1} ^ {L - 1} \log (\deg (G) _ {l}) + \log (C))}{\lambda} \\ + \frac {\lambda \rho \max  _ {r , j _ {0} , \dots , j _ {L}} \prod_ {l = 1} ^ {L - 1} | \operatorname {p r e d} (l , j _ {l}) | \cdot \sum_ {i = 1} ^ {m} \| z _ {j _ {0}} ^ {0} (x _ {i}) \| _ {2} ^ {2}}{2} \\ + \rho \sqrt {\max  _ {j _ {0} , \cdots , j _ {L}} \prod_ {l = 1} ^ {L - 1} | \operatorname {p r e d} (l , j _ {l}) | \cdot \sum_ {i = 1} ^ {m} \| z _ {j _ {0}} ^ {0} (x _ {i}) \| _ {2} ^ {2}} \\ \end{array}
$$
The choice  $\lambda = \sqrt{\frac{2(\log(2)L + \sum_{l = 1}^{L - 1}\log(\deg(G)_l) + \log(C))}{\max_{j_0,\ldots ,j_L}\prod_{l = 1}^{L - 1}|\operatorname {pred}(l,j_l)|\cdot\sum_{i = 1}^m\|z_{j_0}^0(x_i)\|_2^2}}$  gives the desired inequality.
$$
\begin{array}{l} \operatorname {e r r} _ {P} \left(f _ {w}\right) - \operatorname {e r r} _ {S} ^ {\gamma} \left(f _ {w}\right) \leq \frac {2 \sqrt {2} (\rho (w) + 1)}{\gamma m} \cdot \left(1 + \sqrt {2 (\log (2) L + \sum_ {l = 1} ^ {L - 1} \log (\deg (G) _ {l}) + \log (C))}\right) \\ \cdot \sqrt {\max  _ {j _ {0} , \dots , j _ {L}} \prod_ {l = 1} ^ {L - 1} | \operatorname * {p r e d} (l , j _ {l}) | \cdot \sum_ {i = 1} ^ {m} \| z _ {j _ {0}} ^ {0} (x _ {i}) \| _ {2} ^ {2}} + 3 \sqrt {\frac {\log (2 (\rho (w) + 2) ^ {2} / \delta)}{2 m}}, \\ \end{array}
$$
where the maximum is taken over  $j_0, \ldots, j_L$ , such that,  $j_{l-1} \in \operatorname{pred}(l, j_l)$  for all  $l \in [L]$ .
Proof. Let  $t \in \mathbb{N} \cup \{0\}$  and  $\mathcal{G}_t = \mathcal{F}_{G,t}$ . By Lemma 2.2, with probability at least  $1 - \frac{\delta}{t(t + 1)}$ , for any function  $f_w \in \mathcal{G}_t$ , we have
$$
\operatorname {e r r} _ {P} \left(f _ {w}\right) - \operatorname {e r r} _ {S} ^ {\gamma} \left(f _ {w}\right) \leq \frac {2 \sqrt {2}}{\gamma} \cdot \mathcal {R} _ {X} \left(\mathcal {G} _ {t}\right) + 3 \sqrt {\frac {\log (2 / \delta)}{2 m}}. \tag {8}
$$
By Proposition 3.1 in the main text, we have
$$
\begin{array}{l} \mathcal {R} _ {X} (\mathcal {G} _ {t}) \leq \frac {t}{m} \cdot \left(1 + \sqrt {2 (\log (2) L + \sum_ {l = 1} ^ {L - 1} \log (\deg (G) _ {l}) + \log (C))}\right) \\ \cdot \sqrt {\max  _ {j _ {0} , \dots , j _ {L}} \prod_ {l = 1} ^ {L - 1} | \operatorname {p r e d} (l , j _ {l}) | \cdot \sum_ {i = 1} ^ {m} \| z _ {j _ {0}} ^ {0} (x _ {i}) \| _ {2} ^ {2}}, \\ \end{array}
$$
because of the union bound over all  $t \in \mathbb{N}$ , equation 1 holds uniformly for all  $t \in \mathbb{N}$  and  $f_w \in \mathcal{G}_t$  with probability at least  $1 - \delta$ . For each  $f_w$  with norm  $\rho(w)$  we then apply the bound with  $t = \lceil \rho(w) \rceil$  since  $f_w \in \mathcal{G}_t$ , we have
$$
\begin{array}{l} \operatorname {e r r} _ {P} \left(f _ {w}\right) - \operatorname {e r r} _ {S} ^ {\gamma} \left(f _ {w}\right) \leq \frac {2 \sqrt {2} \cdot t}{\gamma m} \cdot \left(1 + \sqrt {2 (\log (2) L + \sum_ {l = 1} ^ {L - 1} \log (\deg (G) _ {l}) + \log (C))}\right) \\ \cdot \sqrt {\max  _ {j _ {0} , \cdots , j _ {L}} \prod_ {l = 1} ^ {L - 1} | \operatorname {p r e d} (l , j _ {l}) | \cdot \sum_ {i = 1} ^ {m} \| z _ {j _ {0}} ^ {0} (x _ {i}) \| _ {2} ^ {2}} + 3 \sqrt {\frac {\log (2 (t + 1) ^ {2} / \delta)}{2 m}} \\ \leq \frac {2 \sqrt {2} (\rho (w) + 1)}{\gamma m} \left(1 + \sqrt {2 (\log (2) L + \sum_ {l = 1} ^ {L - 1} \log (\deg (G) _ {l}) + \log (C))}\right) \\ \cdot \sqrt {\max  _ {j _ {0} , \dots , j _ {L}} \prod_ {l = 1} ^ {L - 1} | \operatorname {p r e d} (l , j _ {l}) | \sum_ {i = 1} ^ {m} \| z _ {j _ {0}} ^ {0} \left(x _ {i}\right) \| _ {2} ^ {2}} + 3 \sqrt {\frac {\log \left(2 \left(\rho (w) + 2\right) ^ {2} / \delta\right)}{2 m}}, \\ \end{array}
$$
which proves the desired bound.