SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Arora, S., Ge, R., Neyshabur, B., and Zhang, Y. Stronger generalization bounds for deep nets via a compression approach. arXiv preprint arXiv:1802.05296, 2018.
[2] Arora, S., Du, S., Hu, W., Li, Z., and Wang, R. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pp. 322-332. PMLR, 2019.
[3] Arora, S., Du, S. S., Hu, W., Li, Z., Salakhutdinov, R. R., and Wang, R. On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems, 32, 2019.
[4] Awasthi, P., Frank, N., and Mohri, M. On the rademacher complexity of linear hypothesis sets. arXiv preprint arXiv:2007.11045, 2020.
[5] Bartlett, P. L. and Mendelson, S. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482, 2002.
[6] Bartlett, P. L., Foster, D. J., and Telgarsky, M. J. Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
[7] Bartlett, P. L., Harvey, N., Liaw, C., and Mehrabian, A. Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks. The Journal of Machine Learning Research, 20(1):2285-2301, 2019.
[8] Bassily, R., Feldman, V., Guzmán, C., and Talwar, K. Stability of stochastic gradient descent on nonsmooth convex losses. Advances in Neural Information Processing Systems, 33:4381-4391, 2020.
[9] Belkin, M., Hsu, D., Ma, S., and Mandal, S. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849-15854, 2019.
[10] Boucheron, S., Lugosi, G., and Massart, P. Concentration inequalities: A nonasymptotic theory of independence. Oxford university press, 2013.
[11] Cao, Y. and Gu, Q. Generalization bounds of stochastic gradient descent for wide and deep neural networks. Advances in neural information processing systems, 32, 2019.
[12] Chen, R. T. Q. torchdiffeq, 2018. URL https://github.com/rtqichen/torchdiffreq.
[13] Chen, W., Gong, X., and Wang, Z. Neural architecture search on imagenet in fourgpu hours: A theoretically inspired perspective. arXiv preprint arXiv:2102.11535, 2021.
[14] Chen, Y., Huang, W., Nguyen, L., and Weng, T.-W. On the equivalence between neural network and support vector machine. Advances in Neural Information Processing Systems, 34, 2021.
[15] Chizat, L. and Bach, F. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory, pp. 1305-1338. PMLR, 2020.
[16] Cristianini, N., Shawe-Taylor, J., et al. An introduction to support vector machines and other kernel-based learning methods. Cambridge university press, 2000.
[17] Domingos, P. Every model learned by gradient descent is approximately a kernel machine. arXiv preprint arXiv:2012.00152, 2020.
[57] Weng, T.-W., Zhang, H., Chen, P.-Y., Yi, J., Su, D., Gao, Y., Hsieh, C.-J., and Daniel, L. Evaluating the robustness of neural networks: An extreme value theory approach. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=BkUH1MZ0b.
[58] Xu, A. and Raginsky, M. Information-theoretic analysis of generalization capability of learning algorithms. Advances in Neural Information Processing Systems, 30, 2017.
[59] Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
A Additional Experiments
(a) Logistic loss for training samples
(b) Difference of logistic loss
(c) Logistic loss and bound
Figure A.4: Experiment (I). (a) shows the dynamics of logistic loss for 5 randomly selected training samples for NN trained by gradient flow (NN GF), NN trained by gradient descent (NN GD), and the equivalent general kernel machine (KM) in Theorem 2. The dynamics of NN GF and KM overlap and verify the equivalence in Theorem 2. The dynamics of NN GF and NN GD are consistently close throughout the training process. (b) shows the differences between NN GF and KM are 0, which verifies our equivalence in Theorem 2. The differences between NN GD and KM are also small. (c) shows NN GF's training loss, test loss, test error, and population loss bounds we estimated. Bound with  $U_{1} / U_{2}$  is the bound for  $L_{\mu}(w)$  by applying  $U_{1} / U_{2}$  in Corollary 1. (d) shows  $U_{1}$  and  $U_{2}$  in Theorem 3 first increase then converge after sufficient training time. The numerical experiments demonstrate that our complexity bound is tight and can capture the generalization gap well. As the number of  $S'$  increases, our bound converges to the true supremum value.
(d) Rademacher complexity bound
Experiments are implemented with PyTorch [46] on 24G A5000 and 32G V100 GPUs.
A.1 Computation Cost of Experiments
In Experiment (I), estimating the bound with  $20~S^{\prime}$  (solving 20 gradient flow ODE) costs 500s and training NN costs 0.29s. The GPU memory required by estimating the bound is 2406MB and training NN requires 1044MB.
GPU hours	CIFAR-10	CIFAR-100
RS + Gene(w, S) (Ours)	0.036	0.037
Training one NN architecture to convergence	1.83	2.56
Table 3: Averaged computational cost (GPU hours) for one architecture in Experiment (III).
For Experiment (III), we report the averaged computational cost (GPU hours) of our approach for one architecture and the computational cost of training one NN architecture to convergence in
(c) Logistic loss and bound
(d) Rademacher complexity bound
Figure A.5: Experiment (I) on CIFAR-10. Generalization bound for three-layer NN (3072-100-100-1) trained on binary CIFAR-10 (cat and dog). The experiment demonstrates that our complexity bound is tight and can capture the generalization gap well.
Table 3. Note our approach calculates  $\operatorname{Gene}\left( {\mathbf{w},\mathcal{S}}\right)$  only after training for 1 or 2 epochs,which saves computational cost a lot.
A.2 Comparison with Existing Generalization Bounds
To make a stronger comparison with existing work, we have conducted two additional experiments below to show that our bound is much tighter than prior work on both finite-width NNs and infinite-width NNs.
(1) For the finite-width NN, we compare with previous uniform convergence bounds (VC dimension [7] and norm-based bounds [6]) as the NTK-based bounds [2, 11] are limited to infinite or ultra-wide NNs. Note our bounds are also uniform convergence bounds. We follow the same setting in our Experiment I, and calculate the bound for two-layer NN with 100 hidden nodes (finite-width) at training time  $\mathrm{T} = 1000$  and sample size  $\mathrm{n} = 1000$ . Here  $L$  is the number of layers,  $p$  is the number of parameters,  $W^i$  is the weight matrix for the  $i$ -th layer, and  $m$  is the largest width of NN including the input dimension. Constants in big  $O$  are ignored in the calculation. The results in Table 4 show that our bound is much tighter than previous VC dimensions and norm-based bounds.
Finally, we remark that existing works have observed that VC dimension and norm-based bounds are mostly vacuous (see e.g., Figure 5 of [44] and Figure 4 in [1]) while our bound is non-vacuous as shown above and in Figure 2 and Figure A.5.
(2) For the infinite-width NN, we compare with the NTK-based bound in Cao & Gu [11] in a similar setting as Experiment I – two-layer NN and binary MNIST (1 vs. 7) with  $n = 1000$ . Note that their bound only holds for an ultra-wide NN. We compute the infinite-width NTK using Neural Tangents [45] and calculate their bound accordingly. For our bound, we train a two-layer NN with 1000 hidden nodes (to approximate an ultra-wide NN) and compute our bound at  $T = 1000$  (almost convergence). Constants in big  $O$  are ignored in the calculation. The results in Table 5 show that our bound is much tighter than previous NTK-based bounds.
Table 4: Comparison with existing uniform convergence bounds (VC dimension [7] and norm-based bounds [6]). "Finite sample estimate": Bound estimated from  $100\mathcal{S}'$ . "Extreme value estimate": Bound estimated from extreme value theory. See more detail in Appendix A.3.
Method	Bound	Value	T tighter over [6]
VC dimension [7]	O(Lplog(p)/√n)	55957.3	-
Norm-based bound [6]	O(1/√n ∏i=1L ||Wi||2 ∑i=1L ||W^T||2/2,1 ||W^i||2/3)3/2 ln(m))	140.7	-
Ours
Finite sample estimate	RgS(GT)	0.03	4690×
Extreme value estimate	RgS(GT)	0.06	2345×
Table 5: Comparison with NTK-based bounds [11].
Method	Bound	Value	Tighter over [11]
Cao & Gu [11]	\(\tilde{O}(L \cdot \sqrt{\frac{Y^{\top}(\Theta)^{-1}Y}{n}})\)	1.44	-
Ours
Finite sample estimate	\(\hat{\mathcal{R}}_{\mathcal{S}}^{gf}(\mathcal{G}_T)\)	0.026	55.38×
Extreme value estimate	\(\hat{\mathcal{R}}_{\mathcal{S}}^{gf}(\mathcal{G}_T)\)	0.08	18.00×
A.3 Estimate Supremum with Extreme Value Theory
In the all above experiments, we estimate the supremum in the Rademacher bound in Theorem 3 with the maximum over a finite set. To get the true supremum value of the bound, we apply the extreme value theory to estimate it, similar to estimating the local Lipschitz of NN in [57]. We show the finite maximum is close to the true supremum value, validating using a finite maximum as an estimate for our bound. We first state the following result from extreme value theory and then explain how we can apply it to estimate our bounds.
Theorem 6 (Fisher-Tippett-Gnedenko theorem). Let  $X_{1}, X_{2}, \dots, X_{m}$  be a sequence of independent and identically-distributed random variables with cumulative distribution function  $F$ . Suppose that there exist two sequences of real numbers  $a_{m} > 0$  and  $b_{m} \in \mathbb{R}$  such that the following limits converge to a non-degenerate distribution function:
$$
\lim  _ {m \rightarrow \infty} \mathbb {P} (\frac {\max  \left\{X _ {1} , X _ {2} , \cdots , X _ {m} \right\} - a _ {m}}{b _ {m}} \leq x) = G (x).
$$
Then the limit distribution  $G$  belongs to either the Gumbel class (Type I), the Fréchet class (Type II), or the Reverse Weibull class (Type III):
$$
G u m b e l c l a s s (T y p e I): \quad G (x) = \exp \left\{- \exp \left(- \frac {x - a _ {w}}{b _ {w}}\right) \right\}, x \in \mathbb {R},
$$
$$
\text {F r é c h e t c l a s s (T y p e I I) :} \quad G (x) = \left\{ \begin{array}{l l} 0, & x <   a _ {w}, \\ \exp \left\{- \left(\frac {x - a _ {w}}{b _ {w}}\right) ^ {- c _ {w}} \right\}, & x \geq a _ {w}, \end{array} \right.
$$
$$
\text {R e v e r s e W e i b u l l c l a s s (T y p e I I I) :} \quad G (x) = \left\{ \begin{array}{l l} \exp \left\{- \left(\frac {a _ {w} - x}{b _ {w}}\right) ^ {- c _ {w}} \right\}, & x <   a _ {w}, \\ 1, & x \geq a _ {w}, \end{array} \right.
$$
where  $a_w \in \mathbb{R}, b_w > 0$  and  $c_w > 0$  are the location, scale, and shape parameters, respectively.
Our bounds are finite as long as  $\ell(\boldsymbol{w}, \boldsymbol{z})$  is  $L_{\ell}$ -Lipschitz (will explain in detail later). Thus we are particularly interested in the reverse Weibull class, as its CDF has a finite right end-point (denoted as  $a_{w}$ ). The right end-point reveals the upper limit of the distribution, known as the extreme value. In our case, the extreme value is exactly the supremum we want to estimate.
To compute  $U_{2}$ , we compute an upper bound of it by setting  $\epsilon$  as the largest  $\ell_{1}$  distance between any two  $g(\mathbf{Z}) \in \mathcal{G}_T^S$  and  $\mathcal{N}(\mathcal{G}_T^S, \epsilon, \| \cdot \|_1) = 1$  because in this case any  $g(\mathbf{Z}) \in \mathcal{G}_T^S$  will satisfy as an  $\epsilon$  cover.
$$
U _ {2} = \inf  _ {\epsilon > 0} \left(\frac {\epsilon}{n} + \sqrt {\frac {2 \ln \mathcal {N} (\mathcal {G} _ {T} ^ {S} , \epsilon , \| | | _ {1})}{n}}\right) \leq \sup  _ {g _ {1}, g _ {2} \in \mathcal {G} _ {T}} \frac {1}{n} \| g _ {1} (\mathbf {Z}) - g _ {2} (\mathbf {Z}) \| _ {1}
$$
Denote the right hand side as  $U_2^*$ . Since each  $\mathsf{K}(z_i,z_j;S')\leq L_\ell T^2,U_2^*\leq 2L_\ell T^2$
Note  $\mathbf{Z}$  is fixed. Then for all  $g_{1}, g_{2} \in \mathcal{G}_{T}$ ,  $\|g_{1}(\mathbf{Z}) - g_{2}(\mathbf{Z})\|_{1}$  are i.i.d. random variables, because  $g \in \mathcal{G}_{T}$  are trained from i.i.d.  $\mathcal{S}'$ . Consider a finite set  $\mathcal{G}_{N_{g}} \subset \mathcal{G}_{T}$  with size  $N_{g} = \left|\mathcal{G}_{N_{g}}\right|$ . As  $N_{g} \to \infty$ ,
$$
\hat {U} _ {2} \triangleq \max _ {g _ {1}, g _ {2} \in \mathcal {G} _ {N _ {g}}} \| g _ {1} (\mathbf {Z}) - g _ {2} (\mathbf {Z}) \| _ {1} \stackrel {{P}} {\to} U _ {2} ^ {*}, N _ {g} \to \infty .
$$
Then we can apply extreme value theory to estimate  $U_{2}^{*}$ . We generate  $N_{b}$  batch of  $\mathcal{G}_{N_g}$ , compute their  $\hat{U}_2$  and store them in a set. Then with these  $\hat{U}_2$ 's, we perform a maximum likelihood estimation of reverse Weibull distribution parameters, and the location estimate  $a_w$  is used as an estimate of the  $U_{2}^{*}$ . To validate that reverse Weibull distribution is a good fit for the empirical distribution of the  $\hat{U}_2$ 's, we conduct Kolmogorov-Smirnov goodness-of-fit test (a.k.a. K-S test) to calculate the K-S test statistics D and corresponding p-values. The null hypothesis is that  $\hat{U}_2$ 's follow a reverse Weibull distribution.
We follow the same setting as Experiment I and want to estimate  $U_{2}^{*}$  at  $T = 1000$ . Figure A.6 shows a result of estimating  $U_{2}^{*}$  with  $N_{b} = 50$  and  $N_{g} = 1000$ . The estimated  $U_{2}^{*} = 0.06$  is quite close to the finite maximum, validating using a finite maximum as an estimate for our bound.
Figure A.6: Estimating  $U_{2}^{*}$  with  $N_{b} = 50$  finite maximum over sets of size  $N_{g} = 1000$ . The red line is the fitted probability distribution function (PDF) of the reverse Weibull distribution. The small D-statistics of K-S test (ks) and large p-values (pval) show the hypothesized reverse Weibull distribution fits the empirical distribution of  $\hat{U}_{2}$  well. The estimated  $U_{2}^{*} = 0.06$  is quite close to the finite maximum, validating using a finite maximum as an estimate for our bound.
For  $U_{1}$
$$
U _ {1} = \frac {B}{n} \sqrt {\sup  _ {\mathsf {K} (\cdot , \cdot ; \mathcal {S} ^ {\prime}) \in \mathcal {K} _ {T}} \operatorname {T r} (\mathsf {K} (\mathbf {Z} , \mathbf {Z} ; \mathcal {S} ^ {\prime})) + \sum_ {i \neq j} \Delta (\boldsymbol {z} _ {i} , \boldsymbol {z} _ {j})}
$$
$\operatorname{Tr}(\mathsf{K}(\mathbf{Z}, \mathbf{Z}; S'))$  and  $\mathsf{K}(\pmb{z}_i, \pmb{z}_j; S')$  for  $i, j \in [n]$  are random variables that only depend on  $S'$ . For different  $S', S'' \in \operatorname{supp}(\mu^{\otimes n})$ ,  $\operatorname{Tr}(\mathsf{K}(\mathbf{Z}, \mathbf{Z}; S'))$  and  $\operatorname{Tr}(\mathsf{K}(\mathbf{Z}, \mathbf{Z}; S''))$  are i.i.d. random variables. Similarly for  $\mathsf{K}(\pmb{z}_i, \pmb{z}_j; S')$ ,  $i, j \in [n]$ . We assume the finite maximum of each random variable follows a reverse Weibull distribution and estimate their supremum then sum them together to get
(a) Estimating  $\sup_{\mathbf{K}(\cdot ,\cdot ;\mathcal{S}')\in \mathcal{K}_T}\mathrm{Tr}(\mathsf{K}(\mathbf{Z},\mathbf{Z};\mathcal{S}'))$
(b) Estimating  $\sup_{\mathsf{K}(\cdot ,\cdot ;\mathcal{S}')\in \mathcal{K}_T}\mathsf{K}(z_1,z_2;\mathcal{S}')$
Figure A.7: Estimating the components in  $U_{1}$  with  $N_{b} = 50$  finite maximum over sets of size  $N_{g} = 1000$ . The estimated extreme values are close to the finite maximum, validating using a finite maximum as an estimate for our bound.
$U_{1}$ . Figure A.7 shows estimating  $\sup_{\mathsf{K}(\cdot ,\cdot ;\mathcal{S}^{\prime})\in \mathcal{K}_{T}}\mathrm{Tr}(\mathsf{K}(\mathbf{Z},\mathbf{Z};\mathcal{S}^{\prime}))$  and  $\sup_{\mathsf{K}(\cdot ,\cdot ;\mathcal{S}^{\prime})\in \mathcal{K}_{T}}\mathsf{K}(z_{1},z_{2};\mathcal{S}^{\prime})$ . The estimated extreme values are close to the finite maximum, validating using a finite maximum as an estimate for our bound. Due to the computational cost of estimating  $n^2$  extreme values, we leave estimating  $U_{1}$  as future work.
B Integrability of Loss Tangent Kernel
By the assumption that the loss is continuously differentiable, the loss gradient  $\nabla_w\ell (\pmb {w}(t),\pmb {z})$  is continuous w.r.t.  $\pmb {w}(t)$ . Together with  $\pmb {w}(t)$  is continuous w.r.t.  $t$ ,  $\nabla_w\ell (\pmb {w}(t),\pmb {z})$  is continuous w.r.t.  $t$ . After the inner product, the LTK  $\bar{K} (\pmb {w}(t);z,z^{\prime})$  is still continuous w.r.t.  $t$ . By the continuity of LTK on the compact set  $[0,T]$ , LTK is bounded and Riemann integrable on  $[0,T]$ . Therefore, the integral in LPK is well-defined.
For the full-batch gradient flow (1) we considered in this paper,  $\boldsymbol{w}(t)$  is differentiable therefore continuous w.r.t.  $t$  by the gradient flow equation. Hence, the integral in LPK is well-defined.
For stochastic gradient flow, the continuity of the path  $\boldsymbol{w}(t)$  can be argued as follows. For each time interval  $[t, t + 1]$ , we assume that the same batch of data is used. Within this time interval,  $\boldsymbol{w}(t)$  is continuous w.r.t.  $t$ . For the next time interval  $[t + 1, t + 2]$ , even if a different data batch is used, the gradient flow ODE initializes the  $\boldsymbol{w}(t + 1)$  with the solution from the end point of the previous interval,  $[t, t + 1]$ . This ensures the continuity of  $\boldsymbol{w}(t)$  across distinct time intervals. In short, the continuity of  $\boldsymbol{w}(t)$  within each time interval, combined with the initialization of the ODE, will ensure the continuity of  $\boldsymbol{w}(t)$  in the entire time interval  $[0, T]$ .
C Complete Proofs for Gradient Flow
C.1 Proof of Theorem 2
Theorem 2. Suppose  $\pmb{w}(T) = \pmb{w}_T$  is a solution of (1) at time  $T$  with initialization  $\pmb{w}(0) = \pmb{w}_0$ . Then for any  $\pmb{z} \in \mathcal{Z}$ ,
Proof. Consider the gradient flow:
$$
\frac {d \boldsymbol {w} _ {t}}{d t} = - \nabla_ {\boldsymbol {w}} L _ {S} (\boldsymbol {w} _ {t}) = - \frac {1}{n} \sum_ {i = 1} ^ {n} \nabla_ {\boldsymbol {w}} \ell (\boldsymbol {w} _ {t}, \boldsymbol {z} _ {i}). \tag {6}
$$
For any differentiable loss function  $\ell (\pmb {w},\pmb {z})$  , by chain rule,
$$
\frac {d \ell (\boldsymbol {w} _ {t} , \boldsymbol {z})}{d t} = \left\langle \nabla_ {\boldsymbol {w}} \ell (\boldsymbol {w} _ {t}, \boldsymbol {z}), \frac {d \boldsymbol {w} _ {t}}{d t} \right\rangle . \tag {7}
$$
Plug in the gradient flow expression of  $\frac{d\boldsymbol{w}_t}{dt}$  in (6) into (7). We have
$$
\begin{array}{l} \frac {d \ell (\boldsymbol {w} _ {t} , \boldsymbol {z})}{d t} = \left\langle \nabla_ {\boldsymbol {w}} \ell (\boldsymbol {w} _ {t}, \boldsymbol {z}), - \nabla_ {\boldsymbol {w}} L _ {S} (\boldsymbol {w} _ {t}) \right\rangle \\ = \left\langle \nabla_ {\boldsymbol {w}} \ell (\boldsymbol {w} _ {t}, \boldsymbol {z}), - \frac {1}{n} \sum_ {i = 1} ^ {n} \nabla_ {\boldsymbol {w}} \ell (\boldsymbol {w} _ {t}, \boldsymbol {z} _ {i}) \right\rangle \quad \text {(b y} \\ = - \frac {1}{n} \sum_ {i = 1} ^ {n} \left\langle \nabla_ {\boldsymbol {w}} \ell \left(\boldsymbol {w} _ {t}, \boldsymbol {z}\right), \nabla_ {\boldsymbol {w}} \ell \left(\boldsymbol {w} _ {t}, \boldsymbol {z} _ {i}\right) \right\rangle \quad (\text {b y}) \\ = - \frac {1}{n} \sum_ {i = 1} ^ {n} \bar {\mathrm {K}} \left(\boldsymbol {w} _ {t}; \boldsymbol {z}, \boldsymbol {z} _ {i}\right). \quad \text {(b y D e f i n i t i o n 3 o f t h e L T K)} \\ \end{array}
$$
Integrate both sides from 0 to  $T$  over the path  $w(t)$  taken by the parameters during the gradient flow,
$$
\begin{array}{l} \ell (\boldsymbol {w} _ {T}, \boldsymbol {z}) - \ell (\boldsymbol {w} _ {0}, \boldsymbol {z}) = \int_ {0} ^ {T} - \frac {1}{n} \sum_ {i = 1} ^ {n} \bar {\mathrm {K}} (\boldsymbol {w} _ {t}; \boldsymbol {z}, \boldsymbol {z} _ {i}) d t \\ = - \frac {1}{n} \sum_ {i = 1} ^ {n} \int_ {0} ^ {T} \bar {\mathsf {K}} (\boldsymbol {w} _ {t}; \boldsymbol {z}, \boldsymbol {z} _ {i}) d t \quad \text {(B y l i n e a r i t y o f i n t e g r a t i o n)} \\ = \sum_ {i = 1} ^ {n} - \frac {1}{n} K _ {T} \left(\boldsymbol {z}, \boldsymbol {z} _ {i}; \mathcal {S}\right). \quad \text {(B y D e f i n i t i o n 4 o f L P K)} \\ \end{array}
$$
Thus we have
$$
\ell (\boldsymbol {w} _ {T}, \boldsymbol {z}) = \sum_ {i = 1} ^ {n} - \frac {1}{n} K _ {T} (\boldsymbol {z}, \boldsymbol {z} _ {i}; \mathcal {S}) + \ell (\boldsymbol {w} _ {0}, \boldsymbol {z}).
$$
Below, we prove that  $\mathsf{K}_T(z,z';\mathcal{S}): \mathcal{Z} \times \mathcal{Z} \to \mathbb{R}$  is a valid kernel. We can show  $\mathsf{K}_T(z,z';\mathcal{S})$  is a valid kernel by proving  $\mathsf{K}_T(z,z';\mathcal{S})$  is continuous and the kernel matrix  $\mathsf{K}_T(\mathbf{Z},\mathbf{Z};\mathcal{S}) \in \mathbb{R}^{n \times n}$  is positive semi-definite (PSD) for any finite subset of  $\mathcal{Z}$ ,  $z_1, \dots, z_n \in \mathcal{Z}$  [53]. By Definition 4 of the loss path kernel,
$$
\begin{array}{l} \mathsf {K} _ {T} (\boldsymbol {z}, \boldsymbol {z} ^ {\prime}; \mathcal {S}) = \int_ {0} ^ {T} \bar {\mathsf {K}} (\boldsymbol {w} _ {t}; \boldsymbol {z}, \boldsymbol {z} ^ {\prime}) d t \\ = \int_ {0} ^ {T} \left\langle \nabla_ {\boldsymbol {w}} \ell \left(\boldsymbol {w} _ {t}, \boldsymbol {z}\right), \nabla_ {\boldsymbol {w}} \ell \left(\boldsymbol {w} _ {t}, \boldsymbol {z} ^ {\prime}\right) \right\rangle d t. \\ \end{array}
$$
Since  $\ell(\boldsymbol{w}_t, \boldsymbol{z})$  is continuously differentiable for  $\boldsymbol{w}_t$  and  $\boldsymbol{z}$ ,  $\nabla_{\boldsymbol{w}} \ell(\boldsymbol{w}_t, \boldsymbol{z})$  is continuous for  $\boldsymbol{z}$ . After the inner product and integration, it is still a continuous function. Thus  $\mathsf{K}_T(\boldsymbol{z}, \boldsymbol{z}'; \mathcal{S})$  is a continuous function for  $\boldsymbol{z}$ .
Denote  $\Phi_t(z) = \nabla_{\boldsymbol{w}}\ell (\boldsymbol{w}_t,z)$ , then by Definition 4 of the loss path kernel,
$$
\begin{array}{l} \mathsf {K} _ {T} (z, z ^ {\prime}; \mathcal {S}) = \int_ {0} ^ {T} \bar {\mathsf {K}} (\boldsymbol {w} _ {t}; z, z ^ {\prime}) d t \\ = \int_ {0} ^ {T} \left\langle \nabla_ {\boldsymbol {w}} \ell (\boldsymbol {w} _ {t}, \boldsymbol {z}), \nabla_ {\boldsymbol {w}} \ell (\boldsymbol {w} _ {t}, \boldsymbol {z} ^ {\prime}) \right\rangle d t \\ = \int_ {0} ^ {T} \left\langle \Phi_ {t} (\boldsymbol {z}), \Phi_ {t} (\boldsymbol {z} ^ {\prime}) \right\rangle d t. \\ \end{array}
$$
For  $\forall \pmb{u} \in \mathbb{R}^n$  and  $\forall \{z_i\}_{i=1}^n \subseteq \mathcal{Z}$ ,
$$
\begin{array}{l} \boldsymbol {u} ^ {T} \mathrm {K} _ {T} (\mathbf {Z}, \mathbf {Z}; \mathcal {S}) \boldsymbol {u} = \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} u _ {i} u _ {j} \mathrm {K} _ {T} (\boldsymbol {z} _ {i}, \boldsymbol {z} _ {j}; \mathcal {S}) \\ = \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} u _ {i} u _ {j} \int_ {0} ^ {T} \left\langle \Phi_ {t} \left(\boldsymbol {z} _ {i}\right), \Phi_ {t} \left(\boldsymbol {z} _ {j}\right) \right\rangle d t \\ = \int_ {0} ^ {T} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} u _ {i} u _ {j} \left\langle \Phi_ {t} \left(\boldsymbol {z} _ {i}\right), \Phi_ {t} \left(\boldsymbol {z} _ {j}\right) \right\rangle d t \\ = \int_ {0} ^ {T} \left\langle \sum_ {i = 1} ^ {n} u _ {i} \Phi_ {t} \left(\boldsymbol {z} _ {i}\right), \sum_ {j = 1} ^ {n} u _ {j} \Phi_ {t} \left(\boldsymbol {z} _ {j}\right) \right\rangle d t \\ = \int_ {0} ^ {T} \left\| \sum_ {i} ^ {n} u _ {i} \Phi_ {t} (\boldsymbol {z} _ {i}) \right\| ^ {2} d t \\ \geq 0. \\ \end{array}
$$
Thus the matrix  $\mathsf{K}_T(\mathbf{Z},\mathbf{Z};\mathcal{S})$  is PSD and  $\mathsf{K}_T(z,z';\mathcal{S})$  is therefore a valid kernel.
$$
U _ {1} = \frac {B}{n} \sqrt {\sup  _ {\mathsf {K} (\cdot , \cdot ; \mathcal {S} ^ {\prime}) \in \mathcal {K} _ {T}} \operatorname {T r} \left(\mathsf {K} (\mathbf {Z} , \mathbf {Z} ; \mathcal {S} ^ {\prime})\right) + \sum_ {i \neq j} \Delta (z _ {i} , z _ {j})},
$$
$$
U _ {2} = \inf _ {\epsilon > 0} \left(\frac {\epsilon}{n} + \sqrt {\frac {2 \ln \mathcal {N} (\mathcal {G} _ {T} ^ {S} , \epsilon , \| | | _ {1})}{n}}\right),
$$
C.2.1 Proof of  $U_{1}$
Proof. Recall the definition of  $\mathcal{G}_T$ ,