SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yilan Chen
UCSD CSE
yilan@ucsd.edu
Wei Huang
RIEKN AIP
wei.huang.vr@riken.jp
Charlotte Loh
MIT EECS
cloh@mit.edu
akash.srivastava@ibm.com
Lam M. Nguyen
LamNguyen.MLTD@ibm.com
Tsui-Wei Weng*
UCSD HDSI
lweng@ucsd.edu
Deep neural networks have been increasingly used in real-world applications, making it critical to ensure their ability to adapt to new, unseen data. In this paper, we study the generalization capability of neural networks trained with (stochastic) gradient flow. We establish a new connection between the loss dynamics of gradient flow and general kernel machines by proposing a new kernel, called loss path kernel. This kernel measures the similarity between two data points by evaluating the agreement between loss gradients along the path determined by the gradient flow. Based on this connection, we derive a new generalization upper bound that applies to general neural network architectures. This new bound is tight and strongly correlated with the true generalization error. We apply our results to guide the design of neural architecture search (NAS) and demonstrate favorable performance compared with state-of-the-art NAS algorithms through numerical experiments.
Deep learning models have been increasingly used in applications with significant societal impact. Therefore, it is crucial to ensure that these models perform well not only on the training data but also on the new and unseen data. Classical learning theory attributes the generalization ability of machine learning (ML) models to the small complexity of the hypothesis class [56]. However, modern ML models, such as deep neural networks (NNs), can have billions of parameters yet still exhibit strong generalization abilities [26, 9]. This is because various elements of the learning algorithms, including optimization methods, training data, and neural architectures, can all influence the inductive bias, which in turn shapes the generalization abilities of neural networks [29, 40]. While the overall hypothesis class may be large, the "effective domain" of this class, which ultimately determines the model's generalization abilities, is often much smaller [54, 24, 50, 15]. Hence, it is vital to develop algorithmic generalization bounds to capture this effective domain of the hypothesis class.
There has been significant work investigating the generalization of neural networks in their infinite-width regime through kernel methods [28, 3, 32, 2, 11]. They showed that an infinite-width NN
*Correspondence to: Yilan Chen and Tsui-Wei Weng.
Table 1: Comparison with existing NTK-based generalization bounds.  $\Theta = \Theta (\mathbf{X},\mathbf{X})$  is the NTK on training samples and  $\mathbf{H}^{\infty}$  is the NTK of the first layer.  $L$  represents the number of NN layers. We highlight some unique properties of our results in blue color. "During training" means our bound can be calculated at any time during training while existing NTK-based bounds only hold for the NNs at convergence. "Multi-outputs" means our bound holds for NNs with multiple outputs.
Arora et al. [2]	Cao & Gu [11]	Ours
Bound	√2Y^T(H∞)^{-1}Y/n	ˆ(L·√Y^T(Θ)^{-1}Y/n)	Theorem 3, Theorem 5
Model	Ultra-wide two-layer FCNN	Ultra-wide FCNN	General continuously differentiable NN
Data	i.i.d. data with ||x|| = 1	i.i.d. data with ||x|| = 1	i.i.d. data
Loss	Square loss	Logistic loss	Continuously differentiable & bounded loss
During training	No	No	Yes
Multi-outputs	No	No	Yes
Training algorithm	GD	SGD	(Stochastic) gradient flow
trained by gradient flow and squared loss is equivalent to a kernel regression with neural tangent kernel (NTK) [28, 3]. Moreover, Arora et al. [2], Cao & Gu [11] further characterized the generalization behaviors of such ultra-wide NNs by deriving data-dependent and NTK-based generalization bounds. However, they only considered ultra-wide fully connected NNs with a square (or logistic) loss function. In practice, NNs are usually not ultra-wide and have more complex architectures such as CNNs. Hence, it is crucial to establish generalization bounds that hold in a more general setting.
In this paper, we analyze the generalization capability of NNs trained using (stochastic) gradient flow across a wide range of NN architectures. Our key technical contribution is to establish a new connection between the loss dynamics of (stochastic) gradient flow and general kernel machines with a special kernel that we named the loss path kernel. This new kernel calculates the similarity between two data points by integrating the inner product of the loss gradient evaluated at these points, along the path determined by the gradient flow. Based on this connection, we develop a novel generalization bound by analyzing the complexity of kernel machines induced by various training sets. Our generalization bound is tight and can be applied to a broad class of NN architectures, not restricted to ultra-wide NNs (see Table 1 for a comparison with existing results derived from NTK theory). Numerical experiments demonstrate that our bound maintains a strong correlation with the true generalization error of NNs trained with gradient descent (GD) (see Figure 1 and 2 in Sec. 6 & 7). Given this observation, we use our generalization bound to guide the design of neural architecture search (NAS) and demonstrate through numerical experiments that our approach can achieve a favorable performance compared with state-of-the-art training-free and minimum-training NAS algorithms [37, 13, 39]. In summary, our contributions are:
- In Sec. 4.1 and 5, we show for the first time that the loss of NNs trained by (stochastic) gradient flow is equivalent to a general kernel machine. This result enables us to investigate the generalization capability of NNs from the perspective of kernel theory.
- In Sec. 4.2 and 5, we derive tight generalization bounds for NNs based on the aforementioned equivalence. Our result is very general as it holds for any continuously differentiable NN architectures including finite-width and infinite-width NNs. Experiments demonstrate that our bounds are tight ( $4690 \times$  tighter than existing norm-based bounds and  $55 \times$  tighter than existing NTK-based bounds as shown in Appendix A.2) and highly correlated with the true generalization error.
- In Sec. 6 and 7, we apply our theory to study special cases including infinite-width NNs, stable algorithms, and norm-constrained NNs. We apply our bound to guide the design of NAS. Numerical experiments demonstrate that our approach achieves a favorable performance compared with state-of-the-art NAS algorithms on NAS-Bench-201 benchmark [19].
Generalization theory in deep learning. Generalization is a crucial aspect of deep learning theory and various techniques have been proposed to study it. For example, Bartlett et al. [7] derived tight bounds for the VC dimension of NNs with ReLU activation functions. There is also a line of work that measures the capacity of NNs based on different norms, margins [5, 41, 6, 44], and sharpness-based measures [42, 43, 1] to explain the generalization behaviors of NNs. Additionally, there are theories studying the generalization of NNs from PAC-Bayes [35, 22] and information-theoretical approach
[48, 58]. For example, Dziugaite & Roy [22] numerically evaluated and optimized the PAC-Bayes bound of stochastic NN and obtained a non-vacuous generalization bound. In contrast, we study the generalization of NNs by building a new connection between NNs and kernel machines. We refer readers to Valle-Pérez & Louis [55] for a more comprehensive review of the generalization theory of NNs.
Neural tangent kernel (NTK). NTK was first introduced in Jacot et al. [28], where the authors demonstrated that a fully-trained, infinite-width NN follows kernel gradient descent in the function space with respect to the NTK. Under gradient flow and squared loss, the fully-trained infinite-width NN is equivalent to kernel regression with the NTK [28, 3]. Chen et al. [14] further established the equivalence between infinite-width NNs and regularized kernel machines. Arora et al. [2] studied the generalization capacity of ultra-wide, two-layer NNs trained by GD and square loss, while Cao & Gu [11] examined the generalization of deep, ultra-wide NNs trained by stochastic gradient descent (SGD) and logistic loss. Both studies derived generalization bounds of converged NNs based on NTK. Besides, Huang et al. [27] studied the convergence and generalization of PAC-Bayesian learning for deep, ultra-wide NNs. Later, Domingos [17] showed that every model trained by gradient flow is a "kernel machine" with the weights and bias as functions of input data, which however can be much more complex than a typical kernel machine and our general kernel machine in Definition 2. Chen et al. [14] showed that every NN trained by gradient flow is a general kernel machine but their kernel is valid only in very limited cases – when the loss gradient of output is a constant. Otherwise, the kernel is not symmetric and not valid. In this paper, we consider an equivalence between the loss of NNs and general kernel machines, which resolves the previous asymmetric problem of the kernel function and also makes the generalization analysis of multi-outputs easier.
Neural Architecture Search (NAS). NAS aims to automate the discovery of top-performance neural networks to reduce human efforts. However, most existing NAS algorithms require heavy training of a supernet or intensive architecture evaluations, suffering from heavy resource consumption [47, 34, 18, 33]. Thus, it is crucial to develop training-free or minimum-training NAS algorithms to reduce the computational cost and select the best architecture at the same time [37, 13, 39]. Since our generalization bound has a strong correlation with the true generalization error, we apply it to design a new minimum-training NAS algorithm. We demonstrate in Table 2 that with a simple random search algorithm, our approach can achieve a favorable performance compared with state-of-the-art training-free and minimum-training NAS algorithms.
3 Kernel Machine and Loss Path Kernel
In this section, we define notation, provide a brief overview of kernel methods, and introduce the main concept of interest—the loss path kernel.
Consider a supervised learning problem where the task is to predict an output variable in  $\mathcal{V} \subseteq \mathbb{R}^k$  using a vector of input variables in  $\mathcal{X} \subseteq \mathbb{R}^d$ . Let  $\mathcal{Z} \triangleq \mathcal{X} \times \mathcal{Y}$ . We denote the training set by  $S \triangleq \{\pmb{z}_i\}_{i=1}^n$  with  $\pmb{z}_i \triangleq (\pmb{x}_i, \pmb{y}_i) \in \mathcal{Z}$ . We assume each point is drawn i.i.d. from an underlying distribution  $\mu$ . Let  $\mathbf{X} = [\pmb{x}_1, \dots, \pmb{x}_n]^T \in \mathbb{R}^{n \times d}$ ,  $\mathbf{Y} = [\pmb{y}_1, \dots, \pmb{y}_n]^T \in \mathbb{R}^{n \times k}$ , and  $\mathbf{Z} = [\mathbf{X}, \mathbf{Y}] \in \mathbb{R}^{n \times (d + k)}$ .
We express a neural network in a general form  $f(\boldsymbol{w}, \boldsymbol{x}) : \mathbb{R}^p \times \mathbb{R}^d \to \mathbb{R}^k$  where  $\boldsymbol{w} \in \mathbb{R}^p$  represents its parameters and  $\boldsymbol{x} \in \mathbb{R}^d$  is an input variable. The goal of a learning algorithm is to find a set of parameters that minimizes a population risk  $L_{\mu}(\boldsymbol{w}) = \mathbb{E}_{z \sim \mu}[\ell(\boldsymbol{w}, z)]$  where  $\ell(\boldsymbol{w}, z) \triangleq \ell(f(\boldsymbol{w}, x), y)$  is a loss function. Throughout this paper, we assume that  $\ell(\boldsymbol{w}, z) \in [0,1]$  and is continuously differentiable. In practice, the underlying distribution  $\mu$  is unknown so the learning algorithm minimizes an empirical risk on the training set  $S$  instead:  $L_S(\boldsymbol{w}) = \frac{1}{n} \sum_{i=1}^{n} \ell(\boldsymbol{w}, z_i)$ . The generalization gap is defined as  $L_{\mu}(\boldsymbol{w}) - L_S(\boldsymbol{w})$ . The loss gradient with respect to the parameters  $\boldsymbol{w}$  is  $\nabla_{\boldsymbol{w}} \ell(\boldsymbol{w}, z) = \nabla_{\boldsymbol{w}} f(\boldsymbol{w}, x)^{\top} \nabla_f \ell(f(\boldsymbol{w}, x), y) \in \mathbb{R}^{p \times 1}$ . Our analysis only requires that  $\ell(\boldsymbol{w}, z)$  is continuously differentiable w.r.t.  $\boldsymbol{w}$  and  $z$  so  $f$  can be either a fully connected neural network or a convolutional network or a residual network.
3.2 Kernel Method
Kernel methods [16, 51, 53] search for linear relations in high-dimensional feature space by using kernel functions. Rather than computing the coordinates in the feature space explicitly, they only need to calculate the inner product between data pairs, making it computationally easier.
A kernel is a function  $K: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$ , such that for all  $\pmb{x}, \pmb{x}' \in \mathcal{X}$ ,  $K(\pmb{x}, \pmb{x}') = \langle \Phi(\pmb{x}), \Phi(\pmb{x}') \rangle$  where  $\Phi: \mathcal{X} \to \mathcal{F}$  is a mapping from  $\mathcal{X}$  to an (inner product) feature space  $\mathcal{F}$ .
Proposition 1 (Shawe-Taylor et al. [53]). A function  $K: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$ , which is either continuous or has a finite domain, is a kernel function if and only if it is a symmetric function and, for any finite subset of  $\mathcal{X}$ ,  $\pmb{x}_1, \dots, \pmb{x}_n \in \mathcal{X}$ , the matrix  $K(\mathbf{X}, \mathbf{X})$  is positive semi-definite, where  $K(\mathbf{X}, \mathbf{X})$  is a  $n \times n$  matrix whose  $(i,j)$ -th entry is  $K(\pmb{x}_i, \pmb{x}_j)$ .
Definition 1 (Kernel machine). Let  $\mathcal{H}$  be the reproducing kernel Hilbert space (RKHS) corresponding to a kernel  $K(\pmb{x},\pmb{x}^{\prime}) = \langle \Phi (\pmb{x}),\Phi (\pmb{x}^{\prime})\rangle$ . A kernel machine  $g:\mathcal{X}\to \mathbb{R}$  is a linear function in  $\mathcal{H}$  such that its weight vector  $\beta$  can be expressed as a linear combination of the training points, i.e.  $g(\pmb {x}) = \langle \beta ,\Phi (\pmb {x})\rangle +b = \sum_{i = 1}^{n}a_{i}K(\pmb{x}_{i},\pmb {x}) + b$ , where  $\beta = \sum_{i = 1}^{n}a_{i}\Phi (\pmb{x}_{i})$  and  $b$  is a constant. The RKHS norm of  $g$  is  $\| g\|_{\mathcal{H}} = \| \sum_{i = 1}^{n}a_{i}\Phi (\pmb{x}_{i})\| = \sqrt{\sum_{i,j}a_{i}a_{j}K(\pmb{x}_{i},\pmb{x}_{j})}$ .
Next, we introduce general kernel machine, which generalizes the concept of kernel machine.
Definition 2 (General kernel machine). A general kernel machine  $g: \mathcal{X} \to \mathbb{R}$  with a kernel  $K(\pmb{x},\pmb{x}^{\prime})$  is  $g(\pmb{x}) = \sum_{i=1}^{n} a_i K(\pmb{x}_i, \pmb{x}) + h(\pmb{x})$ , where  $h: \mathcal{X} \to \mathbb{R}$  is a function of  $\pmb{x}$ . When  $h(\pmb{x})$  is a constant,  $g(\pmb{x})$  reduces to a kernel machine in Definition 1.
3.3 Neural Tangent Kernel and Loss Path Kernel
Neural tangent kernel (NTK) has been introduced by Jacot et al. [28] to establish an equivalence between infinite-width NNs and kernel regression. After then, there is a growing line of work applying NTK theory to study properties of over-parameterized NNs, such as optimization convergence [21, 20] and generalization capability [2, 11]. The neural tangent kernel [28] associated with a NN  $f(\boldsymbol{w}, \boldsymbol{x})$  at  $\boldsymbol{w}$  is defined as  $\hat{\Theta}(\boldsymbol{w}; \boldsymbol{x}, \boldsymbol{x}') = \nabla_{\boldsymbol{w}} f(\boldsymbol{w}, \boldsymbol{x}) \nabla_{\boldsymbol{w}} f(\boldsymbol{w}, \boldsymbol{x}')^\top \in \mathbb{R}^{k \times k}$ . Under certain conditions, such as infinite width limit and NTK parameterization, the NTK converges to a deterministic limit kernel  $\Theta(\boldsymbol{x}, \boldsymbol{x}') \cdot \mathbf{I}_k$  that remains constant during training:  $\hat{\Theta}(\boldsymbol{w}; \boldsymbol{x}, \boldsymbol{x}') \to \Theta(\boldsymbol{x}, \boldsymbol{x}') \cdot \mathbf{I}_k$ , where  $\Theta(\boldsymbol{x}, \boldsymbol{x}') : \mathbb{R}^d \times \mathbb{R}^d \to \mathbb{R}$  is a scalar kernel and  $\mathbf{I}_k$  is a  $k \times k$  identity matrix. Next, we introduce the main concepts of interest in this paper: the loss tangent kernel and the loss path kernel. They are central to characterizing the generalization behaviors of NNs trained by (stochastic) gradient flow.
Definition 3 (Loss Tangent Kernel (LTK)  $\overline{\mathsf{K}}$ ). The loss tangent kernel associated with the loss function  $\ell(\boldsymbol{w}, \boldsymbol{z})$  is defined as  $\overline{\mathsf{K}}(\boldsymbol{w}; \boldsymbol{z}, \boldsymbol{z}') = \langle \nabla_{\boldsymbol{w}} \ell(\boldsymbol{w}, \boldsymbol{z}), \nabla_{\boldsymbol{w}} \ell(\boldsymbol{w}, \boldsymbol{z}') \rangle \in \mathbb{R}$ .
The LTK  $\bar{\mathsf{K}}$  has a natural connection with the NTK  $\hat{\Theta}$  by applying the chain rule:
$$
\bar {K} (\boldsymbol {w}; \boldsymbol {z}, \boldsymbol {z} ^ {\prime}) = \nabla_ {f} \ell (\boldsymbol {w}, \boldsymbol {z}) ^ {\top} \hat {\Theta} (\boldsymbol {w}; \boldsymbol {x}, \boldsymbol {x} ^ {\prime}) \nabla_ {f} \ell (\boldsymbol {w}, \boldsymbol {z} ^ {\prime}).
$$
Next, we introduce the loss path kernel, which integrates the LTK along a given path of the parameters. Later, we will characterize this path via the gradient flow dynamics.
Definition 4 (Loss Path Kernel (LPK)  $\mathsf{K}_T$ ). Suppose the weights follow a continuous path  $\boldsymbol{w}(t): [0, T] \to \mathbb{R}^p$  in their domain with a starting point  $\boldsymbol{w}(0) = \boldsymbol{w}_0$ , where  $T$  is a predetermined constant. This path is determined by the training set  $\mathcal{S}$  and the training time  $T$ . We define the loss path kernel associated with the loss function  $\ell(\boldsymbol{w}, \boldsymbol{z})$  along the path as  $\mathsf{K}_T(\boldsymbol{z}, \boldsymbol{z}'; \mathcal{S}) \triangleq \int_0^T \bar{\mathsf{K}}(\boldsymbol{w}(t); \boldsymbol{z}, \boldsymbol{z}') \mathrm{d}t$ .
In Appendix B, we show LTK is Riemann integrable so the integral in the above definition is well-defined. Intuitively, the LTK  $\bar{K}(\boldsymbol{w}; \boldsymbol{z}, \boldsymbol{z}')$  measures the similarity between data points  $\boldsymbol{z}$  and  $\boldsymbol{z}'$  by comparing their loss gradients when evaluated using a fixed neural network parameter  $\boldsymbol{w}$ . The LPK  $\mathsf{K}_T(\boldsymbol{z}, \boldsymbol{z}'; \mathcal{S})$  measures the overall similarity during the entire training time.
3.4 Rademacher Complexity
Rademacher complexity [52] measures the complexity of a hypothesis class. It takes into account the data distribution and is a central concept in statistical learning theory. Next, we recall its definition and a generalization upper bound via Rademacher complexity.
Definition 5 (Empirical Rademacher complexity  $\hat{\mathcal{R}}_{\mathcal{S}}(\mathcal{G})$ ). Let  $\mathcal{F} = \left\{f:\mathcal{X}\to \mathbb{R}^k\right\}$  be a hypothesis class. We denote  $\mathcal{G}$  as the set of loss functions associated with each function in  $\mathcal{F}$ , defined as  $\mathcal{G} = \{g:(\pmb {x},\pmb {y})\rightarrow \ell (f(\pmb {x}),\pmb {y}),f\in \mathcal{F}\}$ . The empirical Rademacher complexity of  $\mathcal{G}$  with respect to a sample set  $\mathcal{S}$  is defined as:  $\hat{\mathcal{R}}_{\mathcal{S}}(\mathcal{G}) = \frac{1}{n}\mathbb{E}_{\pmb{\sigma}}\left[\sup_{g\in \mathcal{G}}\sum_{i = 1}^{n}\sigma_{i}g(\pmb{z}_{i})\right]$ , where  $\pmb {\sigma} = (\sigma_1,\dots ,\sigma_n)$  and  $\sigma_{i}$  are independent uniform random variables taking values in  $\{+1, - 1\}$ .
Theorem 1 (Theorem 3.3 in Mohri et al. [38]). Let  $\mathcal{G}$  be a family of functions mapping from  $\mathcal{Z}$  to  $[0,1]$ . Then for any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$  over the draw of an i.i.d. sample set  $\mathcal{S} = \{z_1,\dots ,z_n\}$ , the following holds for all  $g\in \mathcal{G}$ :  $\mathbb{E}_z[g(z)] - \frac{1}{n}\sum_{i = 1}^n g(z_i)\leq 2\hat{\mathcal{R}}_{\mathcal{S}}(\mathcal{G}) + 3\sqrt{\frac{\log(2 / \delta)}{2n}}$ .
4 Gradient Flow
In this section, we establish a new connection between the loss dynamics of gradient flow and a general kernel machine equipped with the LPK. Using this result, we introduce a new generalization bound by analyzing the complexity of the collection of kernel machines induced by all possible training sets. Our analysis applies to a wide range of neural network architectures, as long as they are continuously differentiable. Our numerical experiments validate the tightness of our bound and its strong correlation with the true generalization error.
4.1 Loss Dynamics of Gradient Flow and Its Equivalence with General Kernel Machine
Consider the gradient flow dynamics (gradient descent with infinitesimal step size):
$$
\frac {\mathrm {d} \boldsymbol {w} (t)}{\mathrm {d} t} = - \nabla_ {\boldsymbol {w}} L _ {S} (\boldsymbol {w} (t)) = - \frac {1}{n} \sum_ {i = 1} ^ {n} \nabla_ {\boldsymbol {w}} \ell (\boldsymbol {w} (t), \boldsymbol {z} _ {i}). \tag {1}
$$
The above ODE is well-defined for a wide variety of conditions, e.g. local Lipschitz-continuity of the gradient or semi-convexity of the loss function [49, 23]. Next, we establish its connection with the general kernel machine (KM) in the following theorem.
Theorem 2 (Equivalence with general KM.). Suppose  $\boldsymbol{w}(T) = \boldsymbol{w}_T$  is a solution of (1) at time  $T$  with initialization  $\boldsymbol{w}(0) = \boldsymbol{w}_0$ . Then for any  $z \in \mathcal{Z}$ ,
$$
\ell (\boldsymbol {w} _ {T}, \boldsymbol {z}) = \sum_ {i = 1} ^ {n} - \frac {1}{n} K _ {T} (\boldsymbol {z}, \boldsymbol {z} _ {i}; \mathcal {S}) + \ell (\boldsymbol {w} _ {0}, \boldsymbol {z}),
$$
where  $\mathsf{K}_T$  is defined in Definition 4.
The above theorem demonstrates that the loss of the NN at a certain fixed time is a general kernel machine. Herein,  $\mathsf{K}_T$  is the LPK and we prove in Appendix C.1 that it is a valid kernel. Unlike previous NTK works that establish the equivalence between infinite-width NNs and kernel machines, our equivalence is much more general and holds for any NN that is continuously differentiable. Based on this equivalence, we characterize the generalization of NNs from the perspective of kernels. Note that  $\mathsf{K}_T$  is a function of  $S$  and this property enables us to establish a data-dependent generalization bound shortly.
4.2 Generalization Bounds
We introduce the main result in this section: a generalization bound for NNs whose weights follow gradient flow in (1) at time  $T$ . We derive this bound by analyzing the Rademacher complexity of the function class of kernel machines induced by different training sets with constrained RKHS norms. Recall that each training set yields a distinct LPK. We define the collection of all such LPKs by
$$
\mathcal {K} _ {T} \triangleq \left\{\mathrm {K} _ {T} (\cdot , \cdot ; \mathcal {S} ^ {\prime}): \mathcal {S} ^ {\prime} \in \operatorname {s u p p} \left(\mu^ {\otimes n}\right), \frac {1}{n ^ {2}} \sum_ {i, j} \mathrm {K} _ {T} \left(\boldsymbol {z} _ {i} ^ {\prime}, \boldsymbol {z} _ {j} ^ {\prime}; \mathcal {S} ^ {\prime}\right) \leq B ^ {2} \right\}, \tag {2}
$$
where  $B > 0$  is some constant,  $\mathcal{S}' = \{z_1',\dots,z_n'\}$ ,  $\mu^{\otimes n}$  is the joint distribution of  $n$  i.i.d. samples drawn from  $\mu$ , and  $\mathrm{supp}(\mu^{\otimes n})$  is the support set of  $\mu^{\otimes n}$ . Recall that  $\mathcal{S} = \{z_1,\dots,z_n\}$  is the training
set. Note the set in (2) includes the case of  $\mathcal{S}' = \mathcal{S}$  if  $\frac{1}{n^2} \sum_{i,j} \mathsf{K}_T(z_i, z_j; \mathcal{S}) \leq B^2$ . Then we introduce a class of general kernel machines, corresponding to all different kernels in  $\mathcal{K}_T$ .
$$
\mathcal {G} _ {T} \triangleq \left\{g (\boldsymbol {z}) = \sum_ {i = 1} ^ {n} - \frac {1}{n} \mathsf {K} (\boldsymbol {z}, \boldsymbol {z} _ {i} ^ {\prime}; \mathcal {S} ^ {\prime}) + \ell (\boldsymbol {w} _ {0}, \boldsymbol {z}): \mathsf {K} (\cdot , \cdot ; \mathcal {S} ^ {\prime}) \in \mathcal {K} _ {T} \right\}.
$$
Note that  $g(z) \in \mathcal{G}_T$  corresponds to  $\ell(\boldsymbol{w}_T, \boldsymbol{z})$  trained from one possible dataset  $S' \in \mathrm{supp}(\mu^{\otimes n})$ . Next, we compute the Rademacher complexity of  $\mathcal{G}_T$  and use it to obtain a generalization bound.