SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. We bound  $U_{1}$  in Theorem 3 for an infinite-width NN. For an infinite-width NN, the NTK keeps unchanged during training:
$$
\hat {\Theta} \left(\boldsymbol {w} _ {t}; \boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right)\rightarrow \Theta \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right) \cdot \mathbf {I} _ {k}.
$$
Then for our loss path kernel, for any  $z,z^{\prime}\in \mathcal{Z}$  and any  $\mathsf{K}(\cdot ,\cdot ;\mathcal{S}^{\prime})\in \mathcal{K}_{T},$
$$
\begin{array}{l} K (z, z ^ {\prime}; \mathcal {S} ^ {\prime}) = \int_ {0} ^ {T} \bar {K} (\boldsymbol {w} _ {t}; \boldsymbol {z}, \boldsymbol {z} ^ {\prime}) d t \\ = \int_ {0} ^ {T} \nabla_ {f} \ell \left(\boldsymbol {w} _ {t}, \boldsymbol {z}\right) ^ {\top} \hat {\Theta} \left(\boldsymbol {w} _ {t}; \boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right) \nabla_ {f} \ell \left(\boldsymbol {w} _ {t}, \boldsymbol {z} ^ {\prime}\right) d t \tag {12} \\ = \int_ {0} ^ {T} \nabla_ {f} \ell (\boldsymbol {w} _ {t}, \boldsymbol {z}) ^ {\top} \Theta (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) \cdot \mathbf {I} _ {k} \nabla_ {f} \ell (\boldsymbol {w} _ {t}, \boldsymbol {z} ^ {\prime}) d t \\ = \Theta (\pmb {x}, \pmb {x} ^ {\prime}) \cdot \int_ {0} ^ {T} \nabla_ {f} \ell (\pmb {w} _ {t}, \pmb {z}) ^ {\top} \nabla_ {f} \ell (\pmb {w} _ {t}, \pmb {z} ^ {\prime}) d t. \\ \end{array}
$$
Consider a  $\rho$ -Lipschitz loss function, i.e.  $\| \nabla_f \ell(\boldsymbol{w}_t, \boldsymbol{z}) \| \leq \rho$ , e.g.  $\rho = 1$  for hinge loss and logistic loss,  $\rho = \sqrt{2}$  for cross-entropy loss with one-hot labels. Then we have
$$
- \rho^ {2} \leq \nabla_ {f} \ell (\boldsymbol {w} _ {t}, \boldsymbol {z}) ^ {\top} \nabla_ {f} \ell (\boldsymbol {w} _ {t}, \boldsymbol {z} ^ {\prime}) \leq \rho^ {2}.
$$
$$
- \rho^ {2} T \leq \int_ {0} ^ {T} \nabla_ {f} \ell (\boldsymbol {w} _ {t}, \boldsymbol {z}) ^ {\top} \nabla_ {f} \ell (\boldsymbol {w} _ {t}, \boldsymbol {z} ^ {\prime}) d t \leq \rho^ {2} T.
$$
Then by this inequality and Eq. (12),
$$
- \rho^ {2} T | \Theta (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) | \leq K (\boldsymbol {z}, \boldsymbol {z} ^ {\prime}; S ^ {\prime}) \leq \rho^ {2} T | \Theta (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) |.
$$
Since  $\Theta (\pmb {x}_i,\pmb {x}_i)\geq 0$  for  $i\in [n]$ $\mathsf{K}(\pmb {z}_i,\pmb {z}_i;\mathcal{S}')\leq \rho^2 T\cdot \Theta (\pmb {x}_i,\pmb {x}_i)$  and  $\mathrm{Tr}(\mathsf{K}(\mathbf{Z},\mathbf{Z});\mathcal{S}^{\prime})\leq \mathrm{Tr}(\rho^{2}T\cdot$ $\Theta (\mathbf{X},\mathbf{X}))$
$$
\begin{array}{l} \Delta (\boldsymbol {z} _ {i}, \boldsymbol {z} _ {j}) = \frac {1}{2} \left[ \sup _ {\mathsf {K} (\cdot , \cdot ; \mathcal {S} ^ {\prime}) \in \mathcal {K} _ {T}} \mathsf {K} (\boldsymbol {z} _ {i}, \boldsymbol {z} _ {j}; \mathcal {S} ^ {\prime}) - \inf _ {\mathsf {K} (\cdot , \cdot ; \mathcal {S} ^ {\prime}) \in \mathcal {K} _ {T}} \mathsf {K} (\boldsymbol {z} _ {i}, \boldsymbol {z} _ {j}; \mathcal {S} ^ {\prime}) \right] \\ \leq \frac {1}{2} \left[ \rho^ {2} T | \Theta (\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}) | - (- \rho^ {2} T | \Theta (\boldsymbol {x}, \boldsymbol {x} _ {i}) |) \right] \\ = \rho^ {2} T \left| \Theta (\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}) \right| \\ \end{array}
$$
Take these terms into  $U_{1}$ ,
$$
\begin{array}{l} \hat {\mathcal {R}} _ {\mathcal {S}} \left(\mathcal {G} _ {T}\right) \leq \frac {B}{n} \sqrt {\sup  _ {\mathsf {K} (\cdot , \cdot ; \mathcal {S} ^ {\prime}) \in \mathcal {K} _ {T}} \operatorname {T r} \left(\mathsf {K} \left(\mathbf {Z} , \mathbf {Z}; \mathcal {S} ^ {\prime}\right)\right) + \sum_ {i \neq j} \Delta \left(z _ {i} , z _ {j}\right)} \\ \leq \frac {B}{n} \sqrt {\operatorname {T r} \left(\rho^ {2} T \cdot \Theta (\mathbf {X} , \mathbf {X})\right) + \sum_ {i \neq j} \rho^ {2} T \left| \Theta \left(\boldsymbol {x} _ {i} , \boldsymbol {x} _ {j}\right) \right|} \tag {13} \\ = \frac {\rho B \sqrt {T}}{n} \sqrt {\sum_ {i , j} | \Theta (\boldsymbol {x} _ {i} , \boldsymbol {x} _ {j}) |}. \\ \end{array}
$$
E.2 Generalization bound for stable algorithms
Let  $S'$  and  $S_1'$  be two datasets that only differ in one data point. We make the following stability assumption for gradient decent.
Assumption 1 (uniform stability of GD.). Assume  $\| A_t(S') - A_t(S'_1) \| = \frac{ct}{n}$  for some constant  $c > 0$ . Assume  $\ell(\boldsymbol{w}, \boldsymbol{z})$  is  $L_{\ell}$ -Lipschitz and  $\beta_{\ell}$ -smooth for any  $\boldsymbol{z} \in \mathcal{Z}$ . Then  $\| \nabla_{\boldsymbol{w}} \ell(A_t(S'), \boldsymbol{z}) - \nabla_{\boldsymbol{w}} \ell(A_t(S'_1), \boldsymbol{z}) \| \leq \beta_{\ell} \| A_t(S') - A_t(S'_1) \| = c \beta_{\ell} \frac{t}{n}$ .
This kind of stability results of GD and SGD are proved in [8, 25]. Under these stability and smoothness assumptions, we can bound the deviation of  $g(\pmb{z}_i)$  from its expectation with high probability and further bound the complexity based on Theorem 3.
Theorem 8. For any  $\delta \in (0,1)$ , let  $\mathcal{G}_T^\delta \subset \mathcal{G}_T$  be a  $1 - \delta$  subset of  $\mathcal{G}_T$ , i.e.  $\left|\mathcal{G}_T^\delta\right| = (1 - \delta) |\mathcal{G}_T|$ . Under Assumption 1, we have at least one of such  $\mathcal{G}_T^\delta$ ,
$$
\hat {\mathcal {R}} _ {\mathcal {S}} (\mathcal {G} _ {T} ^ {\delta}) \leq \left(2 L _ {\ell} ^ {2} T + c L _ {\ell} \beta_ {\ell} T ^ {2}\right) \sqrt {\frac {2 \log (\frac {2 n}{\delta})}{n}}.
$$
This bound will naturally translate into a generalization bound by equipping with Theorem 1. This bound has a convergence rate of  $\tilde{O}(1/\sqrt{n})$ . It shows that the complexity of NN trained by GD has a polynomial dependence on  $L_{\ell}$ ,  $\beta_{\ell}$ , and training time  $T$ .
Proof. Let  $\mathcal{G}_T^{\delta, \mathcal{S}} = \left\{g(\mathbf{Z}) = (g(\mathbf{z}_1), \ldots, g(\mathbf{z}_n)): g \in \mathcal{G}_T^\delta \right\}$ . Then similar bound as  $U_2$  in Theorem 3 holds for  $\mathcal{G}_T^\delta$ ,
$$
\hat {\mathcal {R}} _ {\mathcal {S}} \left(\mathcal {G} _ {T} ^ {\delta}\right) \leq \inf  _ {\epsilon > 0} \left(\frac {\epsilon}{n} + \sqrt {\frac {2 \ln \mathcal {N} \left(\mathcal {G} _ {T} ^ {\delta , \mathcal {S}} , \epsilon , \| | | _ {1}\right)}{n}}\right) \tag {14}
$$
We consider the covering of  $\mathcal{G}_T^\delta$  and upper bound the right hand side. Without loss of generality, suppose  $S'$  and  $S_1'$  differ in the first sample. That is  $S' = \{z_1', \ldots, z_n'\}$  and  $S_1' = \{\hat{z}_1', \ldots, z_n'\}$ . For any fixed  $i \in [n]$  and  $g_{\mathsf{K}_{S_1'}}$ ,  $g_{\mathsf{K}_{S'}} \in \mathcal{G}_T$ ,
$$
\begin{array}{l} g _ {\mathsf {K} _ {S _ {1} ^ {\prime}}} \left(\boldsymbol {z} _ {i}\right) - g _ {\mathsf {K} _ {\mathcal {S} ^ {\prime}}} \left(\boldsymbol {z} _ {i}\right) \\ = \frac {1}{n} \mathsf {K} _ {\mathcal {S} ^ {\prime}} (\boldsymbol {z} _ {i}, \boldsymbol {z} _ {1} ^ {\prime}) - \frac {1}{n} \mathsf {K} _ {S _ {1} ^ {\prime}} (\boldsymbol {z} _ {i}, \hat {\boldsymbol {z}} _ {1} ^ {\prime}) + \sum_ {j = 2} ^ {n} \frac {1}{n} \mathsf {K} _ {\mathcal {S} ^ {\prime}} (\boldsymbol {z} _ {i}, \boldsymbol {z} _ {j} ^ {\prime}) - \sum_ {j = 2} ^ {n} \frac {1}{n} \mathsf {K} _ {S _ {1} ^ {\prime}} (\boldsymbol {z} _ {i}, \boldsymbol {z} _ {j} ^ {\prime}) \\ \end{array}
$$
The first two terms are
$$
\begin{array}{l} \frac {1}{n} \mathsf {K} _ {\mathcal {S} ^ {\prime}} (\boldsymbol {z} _ {i}, \boldsymbol {z} _ {1} ^ {\prime}) - \frac {1}{n} \mathsf {K} _ {S _ {1} ^ {\prime}} (\boldsymbol {z} _ {i}, \hat {\boldsymbol {z}} _ {1} ^ {\prime}) \\ = \frac {1}{n} \int_ {0} ^ {T} \left\langle \nabla_ {\boldsymbol {w}} \ell (A _ {t} (\mathcal {S} ^ {\prime}), \boldsymbol {z} _ {i}), \nabla_ {\boldsymbol {w}} \ell (A _ {t} (\mathcal {S} ^ {\prime}), \boldsymbol {z} _ {1} ^ {\prime}) \right\rangle d t - \frac {1}{n} \int_ {0} ^ {T} \left\langle \nabla_ {\boldsymbol {w}} \ell (A _ {t} (S _ {1} ^ {\prime}), \boldsymbol {z} _ {i}), \nabla_ {\boldsymbol {w}} \ell (A _ {t} (S _ {1} ^ {\prime}), \hat {\boldsymbol {z}} _ {1} ^ {\prime}) \right\rangle d t \\ = \frac {1}{n} \int_ {0} ^ {T} \left\langle \nabla_ {\boldsymbol {w}} \ell (A _ {t} (\mathcal {S} ^ {\prime}), \boldsymbol {z} _ {i}), \nabla_ {\boldsymbol {w}} \ell (A _ {t} (\mathcal {S} ^ {\prime}), \boldsymbol {z} _ {1} ^ {\prime}) \right\rangle - \left\langle \nabla_ {\boldsymbol {w}} \ell (A _ {t} (S _ {1} ^ {\prime}), \boldsymbol {z} _ {i}), \nabla_ {\boldsymbol {w}} \ell (A _ {t} (S _ {1} ^ {\prime}), \hat {\boldsymbol {z}} _ {1} ^ {\prime}) \right\rangle d t \\ \leq \frac {1}{n} \int_ {0} ^ {T} \| \nabla_ {\boldsymbol {w}} \ell (A _ {t} (\mathcal {S} ^ {\prime}), \boldsymbol {z} _ {i}) \| \| \nabla_ {\boldsymbol {w}} \ell (A _ {t} (\mathcal {S} ^ {\prime}), \boldsymbol {z} _ {1} ^ {\prime}) \| + \| \nabla_ {\boldsymbol {w}} \ell (A _ {t} (S _ {1} ^ {\prime}), \boldsymbol {z} _ {i}) \| \| \nabla_ {\boldsymbol {w}} \ell (A _ {t} (S _ {1} ^ {\prime}), \hat {\boldsymbol {z}} _ {1} ^ {\prime}) \| d t \\ \leq \frac {1}{n} \int_ {0} ^ {T} L _ {\ell} ^ {2} + L _ {\ell} ^ {2} d t \\ = \frac {1}{n} 2 L _ {\ell} ^ {2} T \\ \end{array}
$$
The last two terms are
In total,
$$
\begin{array}{l} g _ {\mathsf {K} _ {S ^ {\prime}} ^ {\prime}} \left(\boldsymbol {z} _ {i}\right) - g _ {\mathsf {K} _ {S ^ {\prime}}} \left(\boldsymbol {z} _ {i}\right) \\ \leq \frac {1}{n} 2 L _ {\ell} ^ {2} T + \left(1 - \frac {1}{n}\right) c L _ {\ell} \beta_ {\ell} \frac {T ^ {2}}{n} \\ = \frac {1}{n} \left(2 L _ {\ell} ^ {2} T + \left(1 - \frac {1}{n}\right) c L _ {\ell} \beta_ {\ell} T ^ {2}\right) \\ \leq \frac {1}{n} \left(2 L _ {\ell} ^ {2} T + c L _ {\ell} \beta_ {\ell} T ^ {2}\right) \\ \end{array}
$$
Similarly  $g_{\mathsf{K}_{S^{\prime}}}(z_i) - g_{\mathsf{K}_{S_1^{\prime}}}(z_i)\leq \frac{1}{n}\left(2L_\ell^2 T + cL_\ell \beta_\ell T^2\right)$ . Thus  $\left|g_{\mathsf{K}_{S^{\prime}}}(z_i) - g_{\mathsf{K}_{S_1^{\prime}}}(z_i)\right|\leq \frac{1}{n}\left(2L_\ell^2 T + cL_\ell \beta_\ell T^2\right)$ . Then by McDiarmid's inequality, for any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$
$$
\left| g _ {\mathsf {K} _ {S ^ {\prime}}} \left(\boldsymbol {z} _ {i}\right) - \underset {S ^ {\prime}} {\mathbb {E}} \left[ g _ {\mathsf {K} _ {S ^ {\prime}}} \left(\boldsymbol {z} _ {i}\right) \right] \right| \leq \left(2 L _ {\ell} ^ {2} T + c L _ {\ell} \beta_ {\ell} T ^ {2}\right) \sqrt {\frac {\log \left(\frac {2}{\delta}\right)}{2 n}}
$$
Then, by a union bound, with probability at least  $1 - \delta$ , for all  $i \in [n]$
$$
\left| g _ {\mathsf {K} _ {\mathcal {S} ^ {\prime}}} \left(\boldsymbol {z} _ {i}\right) - \underset {\mathcal {S} ^ {\prime}} {\mathbb {E}} \left[ g _ {\mathsf {K} _ {\mathcal {S} ^ {\prime}}} \left(\boldsymbol {z} _ {i}\right) \right] \right| \leq \left(2 L _ {\ell} ^ {2} T + c L _ {\ell} \beta_ {\ell} T ^ {2}\right) \sqrt {\frac {\log \left(\frac {2 n}{\delta}\right)}{2 n}}
$$
This means that with probability at least  $1 - \delta$ , for all  $i \in [n]$
$$
g _ {\mathsf {K} _ {S ^ {\prime}}} (\boldsymbol {z} _ {i}) \in \left[ \underset {\mathcal {S} ^ {\prime}} {\mathbb {E}} \left[ g _ {\mathsf {K} _ {S ^ {\prime}}} (\boldsymbol {z} _ {i}) \right] - \left(2 L _ {\ell} ^ {2} T + c L _ {\ell} \beta_ {\ell} T ^ {2}\right) \sqrt {\frac {\log \left(\frac {2 n}{\delta}\right)}{2 n}}, \underset {\mathcal {S} ^ {\prime}} {\mathbb {E}} \left[ g _ {\mathsf {K} _ {S ^ {\prime}}} (\boldsymbol {z} _ {i}) \right] + \left(2 L _ {\ell} ^ {2} T + c L _ {\ell} \beta_ {\ell} T ^ {2}\right) \sqrt {\frac {\log \left(\frac {2 n}{\delta}\right)}{2 n}} \right]
$$
Use the  $g_{\mathsf{K}_{\mathcal{S}^{\prime}}}$  in this range to construct the  $\mathcal{G}_T^\delta$ . Then for any  $g_1(\mathbf{Z}), g_2(\mathbf{Z}) \in \mathcal{G}_T^{\delta, \mathcal{S}}$
$$
\begin{array}{l} g _ {1} (\mathbf {Z}) - g _ {2} (\mathbf {Z}) = \sum_ {i = 1} ^ {n} \left| g _ {1} \left(\boldsymbol {z} _ {i}\right) - g _ {2} \left(\boldsymbol {z} _ {i}\right) \right| \\ \leq \sum_ {i = 1} ^ {n} 2 \left(2 L _ {\ell} ^ {2} T + c L _ {\ell} \beta_ {\ell} T ^ {2}\right) \sqrt {\frac {\log \left(\frac {2 n}{\delta}\right)}{2 n}} \\ = \left(2 L _ {\ell} ^ {2} T + c L _ {\ell} \beta_ {\ell} T ^ {2}\right) \sqrt {2 n \log (\frac {2 n}{\delta})}. \\ \end{array}
$$
Take  $\epsilon$  as this value, then  $\mathcal{N}(\mathcal{G}_T^{\delta ,\mathcal{S}},\epsilon ,\| \big\| _1) = 1$  . Take this into Eq. (14),
$$
\begin{array}{l} \hat {\mathcal {R}} _ {\mathcal {S}} \left(\mathcal {G} _ {T} ^ {\delta}\right) \leq \inf  _ {\epsilon > 0} \left(\frac {\epsilon}{n} + \sqrt {\frac {2 \ln \mathcal {N} \left(\underline {{\mathcal {G}}} _ {T} ^ {\delta , \mathcal {S}} , \epsilon , \| \| _ {1}\right)}{n}}\right) \\ \leq \left(2 L _ {\ell} ^ {2} T + c L _ {\ell} \beta_ {\ell} T ^ {2}\right) \sqrt {\frac {2 \log \left(\frac {2 n}{\delta}\right)}{n}}. \\ \end{array}
$$
E.3 Norm-constrained neural network
For simplicity, we consider the one-dimensional output in this subsection, i.e.  $k = 1$ , and assume the loss function is  $\rho$ -lipschitz for the model output,  $\| \ell (\hat{\pmb{y}},\pmb {y}) - \ell (\hat{\pmb{y}}^{\prime},\pmb {y})\| \leq \rho (\hat{\pmb{y}} -\hat{\pmb{y}}^{\prime})$  for every  $\hat{\pmb{y}},\hat{\pmb{y}}^{\prime}\in \mathbb{R}$ . Consider one-layer NNs:
$$
\mathcal {F} = \left\{f (\boldsymbol {w}, \boldsymbol {x}) = \boldsymbol {w} ^ {T} \boldsymbol {x}: \boldsymbol {w} \in \mathbb {R} ^ {d}, \boldsymbol {x} \in \mathbb {R} ^ {d} \right\}.
$$
Proposition 2. For the function class of one-layer NN defined above,
$$
\sup  _ {\mathsf {K} _ {\mathcal {S} ^ {\prime}} \in \mathcal {K} _ {T}} \operatorname {T r} (\mathsf {K} _ {\mathcal {S} ^ {\prime}} (\mathbf {Z}, \mathbf {Z})) \leq \rho^ {2} T \sum_ {i = 1} ^ {n} \| \boldsymbol {x} _ {i} \| ^ {2}.
$$
For this one-layer NN, we do not need a norm constraint. For  $L$ -layer NNs with norm constraints:
$$
\mathcal {F} = \left\{f (\boldsymbol {w}, \boldsymbol {x}) = W ^ {L} \sigma \left(W ^ {L - 1} \dots \sigma \left(W ^ {1} \boldsymbol {x}\right)\right): \| W _ {t} ^ {h} \| \leq B _ {i}, t \in [ 0, T ] \right\}
$$
where  $W^{h} \in \mathbb{R}^{d_{h} \times d_{h-1}}$  for  $h \in [L]$  with  $d_{L} = 1, d_{0} = d$ .  $\sigma$  is the element-wise activation function and is 1-lipschitz with  $\sigma(0) = 0$ . With these norm constraints of the parameters during the training, we can further bound the trace term in  $U_{1}$  as follows.
Theorem 9. For the function class of  $L$ -layer NN defined above,
$$
\sup  _ {\mathsf {K} _ {\mathcal {S} ^ {\prime}} \in \mathcal {K} _ {T}} \operatorname {T r} (\mathsf {K} _ {\mathcal {S} ^ {\prime}} (\mathbf {Z}, \mathbf {Z})) \leq \rho^ {2} T \sum_ {i = 1} ^ {n} \| \boldsymbol {x} _ {i} \| ^ {2} \prod_ {j = 1} ^ {L} B _ {j} ^ {2} \sum_ {h = 1} ^ {L} \frac {1}{B _ {h} ^ {2}}.
$$
This bound shows that this quantity has a linear relation with  $T$ . With a finer constraint of  $\| W_t^i\|$  during the training, for example,  $\| W_t^i\| \leq B_{i,t'}, t \in [t',t' + 1]$ , we can get tighter bound. Although similar to previous norm-based bounds that have a polynomial dependence with the norms of the parameters, our bound has a clear dependence on training time  $T$ , which is not achievable from previous approaches. But note these bounds can be very loose since they are worse-case bounds.
E.3.1 Proof of Proposition 2
$$
\begin{array}{l} \mathrm {K} _ {\mathcal {S} ^ {\prime}} (\boldsymbol {z}, \boldsymbol {z} ^ {\prime}) = \int_ {0} ^ {T} \left\langle \ell^ {\prime} \left(f _ {t} (\boldsymbol {x}), \boldsymbol {y}\right) \nabla_ {\boldsymbol {w}} f _ {t} (\boldsymbol {x}), \ell^ {\prime} \left(f _ {t} \left(\boldsymbol {x} ^ {\prime}\right), \boldsymbol {y} ^ {\prime}\right) \nabla_ {\boldsymbol {w}} f _ {t} \left(\boldsymbol {x} ^ {\prime}\right) \right\rangle d t \\ = \int_ {0} ^ {T} \left\langle \ell^ {\prime} \left(f _ {t} (\boldsymbol {x}), \boldsymbol {y}\right) \boldsymbol {x}, \ell^ {\prime} \left(f _ {t} \left(\boldsymbol {x} ^ {\prime}\right), \boldsymbol {y} ^ {\prime}\right) \boldsymbol {x} ^ {\prime} \right\rangle d t \\ = \int_ {0} ^ {T} \ell^ {\prime} \left(f _ {t} (\boldsymbol {x}), \boldsymbol {y}\right) \ell^ {\prime} \left(f _ {t} \left(\boldsymbol {x} ^ {\prime}\right), \boldsymbol {y} ^ {\prime}\right) \langle \boldsymbol {x}, \boldsymbol {x} ^ {\prime} \rangle d t \\ \leq \rho^ {2} | \langle \boldsymbol {x}, \boldsymbol {x} ^ {\prime} \rangle | T. \\ \end{array}
$$
$$
\operatorname {T r} \left(\mathrm {K} _ {\mathcal {S} ^ {\prime}} (\mathbf {Z}, \mathbf {Z})\right) = \sum_ {i = 1} ^ {n} \mathrm {K} _ {T} \left(\boldsymbol {z} _ {i}, \boldsymbol {z} _ {i}\right) \leq \sum_ {i = 1} ^ {n} \rho^ {2} \left\| \boldsymbol {x} _ {i} \right\| ^ {2} T = \rho^ {2} T \sum_ {i = 1} ^ {n} \left\| \boldsymbol {x} _ {i} \right\| ^ {2}.
$$
Since this holds for any  $\mathsf{K}_{\mathcal{S}'} \in \mathcal{K}_T$
$$
\sup  _ {\mathsf {K} _ {\mathcal {S} ^ {\prime}} \in \mathcal {K} _ {T}} \operatorname {T r} \left(\mathsf {K} _ {\mathcal {S} ^ {\prime}} (\mathbf {Z}, \mathbf {Z})\right) \leq \rho^ {2} T \sum_ {i = 1} ^ {n} \| \boldsymbol {x} _ {i} \| ^ {2}.
$$
E.3.2 Proof of Theorem 9
Proof. Denote
$$
f ^ {h} (\boldsymbol {x}) = W ^ {h} g ^ {h - 1} (\boldsymbol {x}) \in \mathbb {R} ^ {d _ {h}}, \quad g ^ {h - 1} (\boldsymbol {x}) = \sigma (f ^ {h - 1} (\boldsymbol {x})) \in \mathbb {R} ^ {d _ {h - 1}}, \quad h \in [ L ]
$$
$$
\frac {\partial f (\boldsymbol {w} , \boldsymbol {x})}{\partial W ^ {h}} = b ^ {h} (\boldsymbol {x}) \left(g ^ {h - 1} (\boldsymbol {x})\right) ^ {T} \in \mathbb {R} ^ {d _ {h} \times d _ {h - 1}}, \quad h \in [ L ]
$$
$$
b ^ {h} (\boldsymbol {x}) = \left\{ \begin{array}{l l} 1 \in \mathbb {R}, & h = L, \\ D ^ {h} (\boldsymbol {x}) \left(W ^ {h + 1}\right) ^ {T} b ^ {h + 1} (\boldsymbol {x}) \in \mathbb {R} ^ {d _ {h}} & h \in [ L - 1 ], \end{array} \right.
$$
$$
D ^ {h} (\boldsymbol {x}) = \operatorname {d i a g} \left(\dot {\sigma} \left(f ^ {h} (\boldsymbol {x})\right)\right) \in \mathbb {R} ^ {d _ {h} \times d _ {h}}, \quad h \in [ L - 1 ].
$$
Then for any  $h\in [L]$  , we can compute
$$
\begin{array}{l} \left\langle \frac {\partial f (\boldsymbol {w} , \boldsymbol {x})}{\partial W ^ {h}}, \frac {\partial f (\boldsymbol {w} , \boldsymbol {x} ^ {\prime})}{\partial W ^ {h}} \right\rangle \\ = \left\langle b ^ {h} (\boldsymbol {x}) \left(g ^ {h - 1} (\boldsymbol {x})\right) ^ {T}, b ^ {h} \left(\boldsymbol {x} ^ {\prime}\right) \left(g ^ {h - 1} \left(\boldsymbol {x} ^ {\prime}\right)\right) ^ {T} \right\rangle \quad \text {(i n n e r p r o d u c t o f m a t r i c e s)} \\ = \operatorname {T r} \left(g ^ {h - 1} (\boldsymbol {x}) b ^ {h} (\boldsymbol {x}) ^ {T} b ^ {h} \left(\boldsymbol {x} ^ {\prime}\right) \left(g ^ {h - 1} \left(\boldsymbol {x} ^ {\prime}\right)\right) ^ {T}\right) \\ = \operatorname {T r} \left(\left(g ^ {h - 1} \left(\boldsymbol {x} ^ {\prime}\right)\right) ^ {T} g ^ {h - 1} (\boldsymbol {x}) b ^ {h} (\boldsymbol {x}) ^ {T} b ^ {h} \left(\boldsymbol {x} ^ {\prime}\right)\right) \\ = \left\langle g ^ {h - 1} (\boldsymbol {x}), g ^ {h - 1} \left(\boldsymbol {x} ^ {\prime}\right) \right\rangle \cdot \left\langle b ^ {h} (\boldsymbol {x}), b ^ {h} \left(\boldsymbol {x} ^ {\prime}\right) \right\rangle , \\ \end{array}
$$
where we have for the first term,
$$
\begin{array}{l} \left\langle g ^ {h - 1} (\boldsymbol {x}), g ^ {h - 1} \left(\boldsymbol {x} ^ {\prime}\right) \right\rangle \leq \left\| g ^ {h - 1} (\boldsymbol {x}) \right\| \left\| g ^ {h - 1} \left(\boldsymbol {x} ^ {\prime}\right) \right\| \\ \leq \prod_ {j = 1} ^ {h - 1} \left| \left| W ^ {j} \right| \right| ^ {2} \| \boldsymbol {x} \| \| \boldsymbol {x} ^ {\prime} \|. \\ \end{array}
$$
The second term can be bounded as
$$
\begin{array}{l} \left\langle b ^ {h} (\boldsymbol {x}), b ^ {h} \left(\boldsymbol {x} ^ {\prime}\right) \right\rangle \leq \left\| b ^ {h} (\boldsymbol {x}) \right\| \left\| b ^ {h} \left(\boldsymbol {x} ^ {\prime}\right) \right\| \\ = \left\| D ^ {h} (\boldsymbol {x}) \left(W ^ {h + 1}\right) ^ {T} b ^ {h + 1} (\boldsymbol {x}) \right\| \left\| D ^ {h} \left(\boldsymbol {x} ^ {\prime}\right) \left(W ^ {h + 1}\right) ^ {T} b ^ {h + 1} \left(\boldsymbol {x} ^ {\prime}\right) \right\| \\ \leq \left\| D ^ {h} (\boldsymbol {x}) \right\| \left\| W ^ {h + 1} \right\| \left\| b ^ {h + 1} (\boldsymbol {x}) \right\| \left\| D ^ {h} \left(\boldsymbol {x} ^ {\prime}\right) \right\| \left\| W ^ {h + 1} \right\| \left\| b ^ {h + 1} \left(\boldsymbol {x} ^ {\prime}\right) \right\| \\ \leq \left\| W ^ {h + 1} \right\| ^ {2} \left\| b ^ {h + 1} (\boldsymbol {x}) \right\| \left\| b ^ {h + 1} \left(\boldsymbol {x} ^ {\prime}\right) \right\| \\ \leq \prod_ {j = h + 1} ^ {L} \left\| W ^ {j} \right\| ^ {2}. \\ \end{array}
$$
Thus in total,
$$
\begin{array}{l} \left\langle \frac {\partial f (\boldsymbol {w} , \boldsymbol {x})}{\partial W ^ {h}}, \frac {\partial f (\boldsymbol {w} , \boldsymbol {x} ^ {\prime})}{\partial W ^ {h}} \right\rangle = \left\langle g ^ {h - 1} (\boldsymbol {x}), g ^ {h - 1} \left(\boldsymbol {x} ^ {\prime}\right) \right\rangle \cdot \left\langle b ^ {h} (\boldsymbol {x}), b ^ {h} \left(\boldsymbol {x} ^ {\prime}\right) \right\rangle \\ \leq \prod_ {j = 1} ^ {h - 1} \left\| W ^ {j} \right\| ^ {2} \| \boldsymbol {x} \| \| \boldsymbol {x} ^ {\prime} \| \cdot \prod_ {j = h + 1} ^ {L} \left\| W ^ {j} \right\| ^ {2} \\ = \| \boldsymbol {x} \| \| \boldsymbol {x} ^ {\prime} \| \frac {\prod_ {j = 1} ^ {L} \| W ^ {j} \| ^ {2}}{\| W ^ {h} \| ^ {2}} \\ \end{array}
$$
Since the tangent kernel  $\langle \nabla_{\boldsymbol{w}}f(\boldsymbol{w},\boldsymbol{x}),\nabla_{\boldsymbol{w}}f(\boldsymbol{w},\boldsymbol{x}^{\prime})\rangle = \sum_{h = 1}^{L}\left\langle \frac{\partial f(\boldsymbol{w},\boldsymbol{x})}{\partial W^{h}},\frac{\partial f(\boldsymbol{w},\boldsymbol{x}^{\prime})}{\partial W^{h}}\right\rangle$  we obtain an upper bound for the tangent kernel,
$$
\langle \nabla_ {\boldsymbol {w}} f (\boldsymbol {w}, \boldsymbol {x}), \nabla_ {\boldsymbol {w}} f (\boldsymbol {w}, \boldsymbol {x} ^ {\prime}) \rangle \leq \| \boldsymbol {x} \| \| \boldsymbol {x} ^ {\prime} \| \prod_ {j = 1} ^ {L} \| W ^ {j} \| ^ {2} \sum_ {h = 1} ^ {L} \frac {1}{\| W ^ {h} \| ^ {2}}.
$$
$$
\begin{array}{l} \mathsf {K} _ {\mathcal {S} ^ {\prime}} (\boldsymbol {z}, \boldsymbol {z} ^ {\prime}) = \int_ {0} ^ {T} l ^ {\prime} (f (\boldsymbol {w} _ {t}, \boldsymbol {x}), \boldsymbol {y}) l ^ {\prime} (f (\boldsymbol {w} _ {t}, \boldsymbol {x} ^ {\prime}), \boldsymbol {y} ^ {\prime}) \left\langle \nabla_ {\boldsymbol {w}} f (\boldsymbol {w} _ {t}, \boldsymbol {x}), \nabla_ {\boldsymbol {w}} f (\boldsymbol {w} _ {t}, \boldsymbol {x} ^ {\prime}) \right\rangle d t \\ \leq \int_ {0} ^ {T} \rho^ {2} \| \boldsymbol {x} \| \| \boldsymbol {x} ^ {\prime} \| \prod_ {j = 1} ^ {L} \left\| W ^ {j} \right\| ^ {2} \sum_ {h = 1} ^ {L} \frac {1}{\| W ^ {h} \| ^ {2}} d t \\ \leq \int_ {0} ^ {T} \rho^ {2} \| \boldsymbol {x} \| \| \boldsymbol {x} ^ {\prime} \| \prod_ {j = 1} ^ {L} B _ {j} ^ {2} \sum_ {h = 1} ^ {L} \frac {1}{B _ {h} ^ {2}} d t \\ = \rho^ {2} T \| \boldsymbol {x} \| \| \boldsymbol {x} ^ {\prime} \| \prod_ {j = 1} ^ {L} B _ {j} ^ {2} \sum_ {h = 1} ^ {L} \frac {1}{B _ {h} ^ {2}}. \\ \end{array}
$$
$$
\operatorname {T r} \left(\mathrm {K} _ {\mathcal {S} ^ {\prime}} (\mathbf {Z}, \mathbf {Z})\right) \leq \rho^ {2} T \sum_ {i = 1} ^ {n} \| \boldsymbol {x} _ {i} \| ^ {2} \prod_ {j = 1} ^ {L} B _ {j} ^ {2} \sum_ {h = 1} ^ {L} \frac {1}{B _ {h} ^ {2}}.
$$
Since this holds for any  $\mathsf{K}_{\mathcal{S}'} \in \mathcal{K}_T$ ,