SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yijun Dong*
yd1319@nyu.edu
Kevin Miller*
Oden Institute for Computational
Engineering & Science
ksmiller@utexas.edu
Qi Lei
& Center of Data Science
q1518@nyu.edu
rward@math.utexas.edu
Despite the empirical success and practical significance of (relational) knowledge distillation that matches (the relations of) features between teacher and student models, the corresponding theoretical interpretations remain limited for various knowledge distillation paradigms. In this work, we take an initial step toward a theoretical understanding of relational knowledge distillation (RKD), with a focus on semi-supervised classification problems. We start by casting RKD as spectral clustering on a population-induced graph unveiled by a teacher model. Via a notion of clustering error that quantifies the discrepancy between the predicted and ground truth clusterings, we illustrate that RKD over the population provably leads to low clustering error. Moreover, we provide a sample complexity bound for RKD with limited unlabeled samples. For semi-supervised learning, we further demonstrate the label efficiency of RKD through a general framework of cluster-aware semi-supervised learning that assumes low clustering errors. Finally, by unifying data augmentation consistency regularization into this cluster-aware framework, we show that despite the common effect of learning accurate clusterings, RKD facilitates a "global" perspective through spectral clustering, whereas consistency regularization focuses on a "local" perspective via expansion.
The immense volume of training data is a vital driving force behind the unparalleled power of modern deep neural networks. Nevertheless, data collection can be quite resource-intensive; in particular, data labeling may be prohibitively costly. With an aim to lessen the workload associated with data labeling, semi-supervised learning capitalizes on more easily accessible unlabeled data via diverse approaches of pseudo-labeling. For instance, data augmentation consistency regularization [Sohn et al., 2020, Berthelot et al., 2019] generates pseudo-labels from the current model predictions on carefully designed data augmentations. Alternatively, knowledge distillation [Hinton et al., 2015] can be leveraged to gauge representations of unlabeled samples via pretrained teacher models.
Figure 1: The complementary perspectives of RKD and DAC regarding the data. RKD learns the pairwise relations among data (e.g., edge  $w_{\mathbf{xx}'}$  that characterizes the similarity between  $\mathbf{x}$  and  $\mathbf{x}'$ ) over the population "globally" via spectral clustering, as illustrated in Section 4. Alternatively, DAC discovers the "local" clustering structure through the expansion of neighborhoods based on overlapping augmentation sets (e.g.,  $\mathcal{A}(\mathbf{x}) \cap \mathcal{A}(\mathbf{x}'') \neq \emptyset$ ), as elaborated in Section 6.
Meanwhile, as the scale of training state-of-the-art models explodes, computational costs and limited data are becoming significant obstacles to learning from scratch. In this context, (data-free) knowledge distillation [Liu et al., 2021, Wang and Yoon, 2021] (where pretrained models are taken as teacher oracles without access to their associated training data) is taking an increasingly crucial role in learning efficiently from powerful existing models.
Despite the substantial progresses in knowledge distillation (KD) in practice, its theoretical understanding remains limited, possibly due to the abundant diversity of KD strategies. Most existing theoretical analyses of KD [Ji and Zhu, 2020, Allen-Zhu and Li, 2020, Harutyunyan et al., 2023, Hsu et al., 2021, Wang et al., 2022] focus on feature matching between the teacher and student models [Hinton et al., 2015], while a much broader spectrum of KD algorithms [Liu et al., 2019, Park et al., 2019, Chen et al., 2021, Qian et al., 2022] demonstrates appealing performance in practice. Recently, relational knowledge distillation (RKD) [Park et al., 2019, Liu et al., 2019] has achieved remarkable empirical successes by matching the inter-feature relationships (instead of features themselves) between the teacher and student models. This drives us to focus on RKD and motivates our analysis with the following question:
What perspective of the data does relational knowledge distillation learn, and how efficiently?
(RKD learns spectral clustering.) In light of the connection between RKD and instance relationship graphs (IRG) [Liu et al., 2019], intuitively, RKD learns the ground truth geometry through a graph induced by the teacher model. We formalize this intuition from a spectral clustering perspective (Section 4). Specifically, for a multi-class classification problem, we introduce a notion of clustering error (Definition 3.1) that measures the difference between the predicted and ground truth class partitions. Through low clustering error guarantees, we illustrate that RKD over the population learns the ground truth partition (presumably unveiled by the teacher model) via spectral clustering (Section 4.1). Moreover, RKD on sufficiently many unlabeled samples learns a similar partition with low clustering error (Section 4.2). By appealing to standard generalization analysis [Wainwright, 2019, Bartlett and Mendelson, 2003, Wei and Ma, 2019] for unbiased estimates of the parameterized RKD loss, we show that the unlabeled sample complexity of RKD is dominated by a polynomial term in the model complexity (Theorem 4.2, Theorem 4.3).
As one of the most influential and classical unsupervised learning strategies, clustering remains an essential aspect of modern learning algorithms with weak/no supervision. For example, contrastive learning [Caron et al., 2020, Grill et al., 2020, Chen and He, 2021] has proven to learn good representations without any labels by encouraging (spectral) clustering [HaoChen et al., 2021, Lee et al., 2021, Parulekar et al., 2023]. However, the effects of clustering in the related schemes of semi-supervised learning (SSL) are less well investigated. Therefore, we pose a follow-up question:
How does cluster-aware semi-supervised learning improve label efficiency and generalization?
<sup>2</sup>An IRG is a graph with samples as vertices and sample relationships, characterized by their mutual distances in a feature space, as edges. RKD encourages the student model to learn such IRGs from the teacher model.
(Cluster-aware SSL is label-efficient.) We formulate a cluster-aware SSL framework (Section 5) where assuming a low clustering error (Definition 3.1) endowed by learning from unlabeled data, the label complexity scales linearly in the number of clusters, up to a logarithmic factor (Theorem 5.1).
Since clustering is a rather generic notion that can be facilitated from various perspectives, to better understand spectral clustering as learned by RKD, we compare the clustering mechanism of RKD with that of another common example of cluster-aware SSL—data augmentation consistency (DAC) regularization, while exploring the following question:
What are the similarities and discrepancies between different cluster-aware strategies?
(RKD and DAC learn clustering from complementary perspectives.) We unify the existing analysis [Yang et al., 2023] tailored for DAC regularization [Sohn et al., 2020] into the cluster-aware SSL framework (Section 6). In particular, while facilitating clustering similarly to RKD, DAC reduces the clustering error through an expansion-based mechanism characterized by "local" neighborhoods induced by data augmentations (Theorem 6.2). This serves as a complementary perspective to RKD which approximates spectral clustering on a graph Laplacian that reflects the underlying "global" structure of the population (Figure 1).
Relational knowledge distillation. Since the seminal work of Hinton et al. [2015], knowledge distillation has become a foundational method for time and memory-efficient deep learning. Recent years have seen different KD strategies. Some works learn to directly match the output (response) [Hinton et al., 2015, Chen et al., 2017, Ba and Caruana, 2014] or intermediate layers (features) [Romero et al., 2014, Zagoruyko and Komodakis, 2016a, Kim et al., 2018] of the teacher network. More recently, Park et al. [2019] introduced the idea of relational knowledge distillation (RKD), and concurrent work [Liu et al., 2019] introduced the "instance relationship graph" (IRG). They essentially presented similar ideas to learn the inter-feature relationship between samples instead of matching models' responses or features individually. A comprehensive survey of relational (relation-based) KD can be found in [Gou et al., 2021].
Despite the empirical success stories, the theoretical understanding of KD remained limited. Prior theoretical work is mostly constrained to linear classifier/nets [Phuong and Lampert, 2019, Ji and Zhu, 2020] or neural tangent kernel (NTK) analysis [Harutyunyan et al., 2023]. Hsu et al. [2021] leverages knowledge distillation to improve the sample complexity analysis of large neural networks with potentially vacuous generalization bounds.
Data augmentation consistency regularization. Data augmentation serves as an implicit or explicit regularization to improve sample efficiency and avoid overfitting. Initially, people applied simple transformations to labeled samples and add them to the training set [Krizhevsky et al., 2017, Simard et al., 2002, Simonyan and Zisserman, 2014, He et al., 2016, Cubuk et al., 2018]. Traditional transformations preserve image semantics, including (random) perturbations, distortions, scales, crops, rotations, and horizontal flips. Subsequent augmentations may alter the labels jointly with the input samples, e.g. Mixup [Zhang et al., 2017], Cutout [DeVries and Taylor, 2017], and Cutmix [Yun et al., 2019]. Theoretical works have demonstrated different functionalities of data augmentation such as incorporating invariances [Mei et al., 2021], amplifying strong features [Shen et al., 2022], or reducing variance during the learning procedure [Chen et al., 2020].
Recent practices explicitly add consistency regularization to enforce prediction or representation similarities [Laine and Aila, 2016, Bachman et al., 2014, Sohn et al., 2020]. Not only does this way incorporates unlabeled samples, but Yang et al. [2023] also theoretically demonstrated other benefits: explicit consistency regularization reduces sample complexity more significantly and handles misspecification better than simply adding augmented data into the training set.
Graph-based learning. Although we leverage graphs to model the underlying clustering structure of the population, the RKD algorithm that we study applies to generic data distributions and does not involve any graph constructions over the training data. The latter is crucial for modern learning settings as graph construction can be prohibitively expensive on common large-scale datasets.
Nevertheless, assuming that explicit graph construction is affordable/available, there exists a rich history of nonparametric graph-based semi-supervised classification models for transductive learn-
ing [Zhu et al., 2003, Bertozzi and Merkurjev, 2019, Zhou et al., 2003, Belkin et al., 2004, Belkin and Niyogi, 2004]. These methods have seen success in achieving high accuracy results in the low-label rate regime [Calder et al., 2020, Bertozzi and Merkurjev, 2019]. Furthermore, assuming inherited graph structures of data, recent progress in graph neural networks (GNN) [Zhou et al., 2020] and graph convolutional networks (GCN) [Welling and Kipf, 2016] has provided a connection between these classical graph-based methods and powerful deep learning models for a variety of problem settings [Zhou et al., 2020].
Notations. For any event  $e$ , let  $\mathbb{1}\{e\} = 1$  if  $e$  is true, and  $\mathbb{1}\{e\} = 0$  otherwise. For any positive integers  $n, K \in \mathbb{N}$ , we denote  $[n] = \{1, \ldots, n\}$  and  $\Delta_K \triangleq \left\{(p_1, \ldots, p_K) \in [0,1]^K \mid \sum_{k=1}^K p_k = 1\right\}$ . For a finite set  $\mathcal{X}$ ,  $|\mathcal{X}|$  denotes the size of  $\mathcal{X}$ , and Unif  $(\mathcal{X})$  denotes the uniform distribution over  $\mathcal{X}$  (i.e.,  $\operatorname{Unif}(\mathbf{x}) = 1 / |\mathcal{X}|$  for all  $\mathbf{x} \in \mathcal{X}$ ). For a distribution  $P$  over  $\mathcal{X}$  and any  $n \in \mathbb{N}$ ,  $P^n$  represents the joint distribution over  $\mathcal{X}^n$  such that  $\{\mathbf{x}_i\}_{i \in [n]} \sim P^n$  is a set of  $n$  i.i.d. samples. Given any matrix  $\mathbf{M} \in \mathbb{R}^{n \times k}$  ( $n \geq k$  without loss of generality), let  $\sigma(\mathbf{M}) = \{\sigma_i(\mathbf{M}) | i \in [k]\}$  be the singular values of  $\mathbf{M}$  with  $\sigma_1(\mathbf{M}) \geq \dots \geq \sigma_k(\mathbf{M})$ . While for any symmetric  $\mathbf{M} \in \mathbb{R}^{n \times n}$ , let  $\lambda(\mathbf{M}) = \{\lambda_i(\mathbf{M}) | i \in [n]\}$  be the eigenvalues of  $\mathbf{M}$  such that  $\lambda_1(\mathbf{M}) \leq \dots \leq \lambda_n(\mathbf{M})$ . Given any labeling function  $y: \mathcal{X} \to [K]$ , let  $\overrightarrow{y}: \mathcal{X} \to \{0,1\}^K$  be its one-hot encoding.
3.1 Spectral Clustering on Population-induced Graph
We consider a  $K$ -class classification problem over an unknown data distribution  $P: \mathcal{X} \times [K] \to [0,1]$  (while overloading  $P$  for the probability measure over  $\mathcal{X}$  without ambiguity). Let  $y_*: \mathcal{X} \to [K]$  be the ground truth classification that introduced a natural partition  $\{\mathcal{X}_k\}_{k \in [K]}$  where  $\mathcal{X}_k \triangleq \{\mathbf{x} \in \mathcal{X} \mid y_*(\mathbf{x}) = k\}$  such that  $\bigcup_{k=1}^{K} \mathcal{X}_k = \mathcal{X}$  and  $\mathcal{X}_k \cap \mathcal{X}_{k'} = \emptyset$  for all  $k \neq k'$ . Meanwhile, we specify a function class  $\mathcal{F} \subset \{f: \mathcal{X} \to \mathbb{R}^K\}$  to learn from where each  $f \in \mathcal{F}$  is associated with a prediction function  $y_f(\mathbf{x}) \triangleq \operatorname{argmax}_{k \in [K]} f(\mathbf{x})_k$ .
We characterize the geometry of the data population $^3$ $\mathcal{X}$  with an undirected weighted graph  $G_{\mathcal{X}} = (\mathcal{X}, \mathbf{W}(\mathcal{X}))$  such that: (i) For every vertex pair  $\mathbf{x}, \mathbf{x}' \in \mathcal{X}$ , the edge weight  $w_{\mathbf{xx}'} \geq 0$  characterizes the similarity between  $(\mathbf{x}, \mathbf{x}')$ . For example,  $w_{\mathbf{xx}'}$  between a pair of samples from the same class is larger than that between a pair from different classes (ii)  $P(\mathbf{x}) = w_{\mathbf{x}} \triangleq \sum_{\mathbf{x}' \in \mathcal{X}} w_{\mathbf{xx}'}$  such that  $\sum_{\mathbf{x} \in \mathcal{X}} \sum_{\mathbf{x}' \in \mathcal{X}} w_{\mathbf{xx}'} = 1$ , intuitively implying that a more representative instance  $\mathbf{x}$  (corresponding to a larger  $w_{\mathbf{x}}$ ) has a higher probability of being sampled. We remark that such population-induced graph  $G_{\mathcal{X}}$  is reminiscent of the population augmentation graph for studying contrastive learning[4] [HaoChen et al., 2021].
Let  $\mathbf{W}(\mathcal{X})$  with  $\mathbf{W}_{\mathbf{x}\mathbf{x}^{\prime}}(\mathcal{X}) = w_{\mathbf{x}\mathbf{x}^{\prime}}$  be the weighted adjacency matrix of  $G_{\mathcal{X}}$ ; let  $\mathbf{D}(\mathcal{X})$  be the diagonal matrix of the weighted degrees  $\{w_{\mathbf{x}}\}_{\mathbf{x}\in \mathcal{X}}$ . The (normalized) graph Laplacian takes the form  $\mathbf{L}(\mathcal{X}) = \mathbf{I} - \overline{\mathbf{W}} (\mathcal{X})$  where  $\overline{\mathbf{W}} (\mathcal{X})\triangleq \mathbf{D}(\mathcal{X})^{-1 / 2}\mathbf{W}(\mathcal{X})\mathbf{D}(\mathcal{X})^{-1 / 2}$  is the normalized adjacency matrix. Then, the spectral clustering [von Luxburg, 2007] on  $G_{\mathcal{X}}$  can be expressed as a (rank- $K$ ) Nyström approximation of  $\overline{\mathbf{W}} (\mathcal{X})$  in terms of the weighted outputs  $\mathbf{D}(\mathcal{X})^{\frac{1}{2}}f(\mathcal{X})$ :
$$
\mathcal {F} _ {\mathbf {L} (\mathcal {X})} \triangleq \underset {f \in \mathcal {F}} {\operatorname {a r g m i n}} \left\{R (f) \triangleq \left\| \overline {{\mathbf {W}}} (\mathcal {X}) - \mathbf {D} (\mathcal {X}) ^ {\frac {1}{2}} f (\mathcal {X}) f (\mathcal {X}) ^ {\top} \mathbf {D} (\mathcal {X}) ^ {\frac {1}{2}} \right\| _ {F} ^ {2} \right\}. \tag {1}
$$
To quantify the alignment between the ground truth class partition and the clustering predicted by  $f \in \mathcal{F}$ , we introduce the notion of clustering error.
For demonstration purposes, we assume an arbitrarily large, but finite, population  $|\mathcal{X}| < \infty$  (as in HaoChen et al. [2021], e.g.,  $\mathcal{X} = \mathbb{F}^d$  where  $\mathbb{F}$  is the set of all floating-point numbers with finite precision). Meanwhile, our analyses can be generalized to any compact set  $\mathcal{X}$  via functional analysis tools (e.g., replacing the adjacency matrix  $\mathbf{W}(\mathcal{X})$  with an adjacency operator).
4The key difference between the population augmentation graph [HaoChen et al., 2021] and the population-induced graph described here is that for analyzing RKD, we consider a graph over the original, instead of the augmented, population. It's also worth highlighting that we introduce such graph structure only for the sake of analysis, while our data distribution is generic. Therefore, we focus on learning from a general function class  $\mathcal{F}$  without constraining to graph neural networks.
Definition 3.1 (Clustering error). Given any  $f \in \mathcal{F}$ , we define the majority labeling
$$
\widetilde {y} _ {f} (\mathbf {x}) \triangleq \underset {k \in [ K ]} {\operatorname {a r g m a x}} \mathbb {P} _ {\mathbf {x} ^ {\prime} \sim P (\mathbf {x})} \left[ y _ {*} \left(\mathbf {x} ^ {\prime}\right) = k \mid y _ {f} \left(\mathbf {x} ^ {\prime}\right) = y _ {f} (\mathbf {x}) \right], \tag {2}
$$
along with the minority subsets associated with  $f$ :  $M(f) \triangleq \{ \mathbf{x} \in \mathcal{X} \mid \widetilde{y}_f(\mathbf{x}) \neq y_*(\mathbf{x}) \}$  such that  $P(M(f))$  quantifies the clustering error of  $f$ . For any  $\mathcal{F}' \subseteq \mathcal{F}$ , let  $\mu(\mathcal{F}') \triangleq \sup_{f \in \mathcal{F}'} P(M(f))$ .
Intuitively,  $M(f)$  characterizes the difference between  $K$ -partition of  $\mathcal{X}$  by the ground truth  $y_*$  and by the prediction function  $y_f$  associated with  $f$ ; while  $\mu(\mathcal{F}')$  quantifies the worse-case clustering error of all functions  $f \in \mathcal{F}'$ . In Section 4.1, we will demonstrate that spectral clustering on  $G_{\mathcal{X}}$  (Equation (1)) leads to provably low clustering error  $\mu(\mathcal{F}_{\mathbf{L}(\mathcal{X})})$ .
3.2 Relational Knowledge Distillation
For RKD, we assume access to a proper teacher model  $\psi : \mathcal{X} \to \mathcal{W}$  (for a latent feature space  $\mathcal{W}$ ) that induces a graph-revealing kernel:  $k_{\psi}(\mathbf{x}, \mathbf{x}') = \frac{w_{\mathbf{x}\mathbf{x}'}}{w_{\mathbf{x}}w_{\mathbf{x}'}}$ . Then, the spectral clustering on  $G_{\mathcal{X}}$  in Equation (1) can be interpreted as the population RKD loss:
$$
R \left(f\right) = \mathbb {E} _ {\mathbf {x}, \mathbf {x} ^ {\prime} \sim P (\mathbf {x}) ^ {2}} \left[ \left(k _ {\psi} \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right) - f (\mathbf {x}) ^ {\top} f \left(\mathbf {x} ^ {\prime}\right)\right) ^ {2} \right].
$$
While with only limited unlabeled samples  $\mathbf{X}^u = \{\mathbf{x}_j^u\mid j\in [N]\}\sim P(\mathbf{x})^N$  in practice, we consider the analogous empirical RKD loss:
$$
\mathcal {F} _ {\mathbf {L} \left(\mathbf {X} ^ {u}\right)} \triangleq \underset {f \in \mathcal {F}} {\operatorname {a r g m i n}} \left\{\widehat {R} _ {\mathbf {X} ^ {u}} (f) \triangleq \frac {2}{N} \sum_ {i = 1} ^ {N / 2} \left(f \left(\mathbf {x} _ {2 i - 1} ^ {u}\right) ^ {\top} f \left(\mathbf {x} _ {2 i} ^ {u}\right) - k _ {\psi} \left(\mathbf {x} _ {2 i - 1} ^ {u}, \mathbf {x} _ {2 i} ^ {u}\right)\right) ^ {2} \right\}, \tag {3}
$$
where  $\widehat{R}_{\mathbf{X}^u}(f)$  serves as an unbiased estimate for  $R(f)$  (Proposition D.1).
Further, for semi-supervised setting with a small set of labeled samples  $(\mathbf{X},\mathbf{y}) = \{(\mathbf{x}_i,y_i)\}_{i\in [n]}\sim$ $P(\mathbf{x},y)^n$  (usually  $n\ll N$ ) independent of  $\mathbf{X}^u$ , let  $\ell :\mathbb{R}^{K}\times [K]\to \{0,1\}$  be the zero-one loss:  $\ell (f(\mathbf{x}),y) = \mathbb{1}\{y_f(\mathbf{x})\neq y\}$ . We denote  $\mathcal{E}(f)\triangleq \mathbb{E}_{(\mathbf{x},y)\sim P}[\ell (f(\mathbf{x}),y)]$  and  $\widehat{\mathcal{E}}(f)\triangleq \frac{1}{n}\sum_{i=1}^{n}\ell(f(\mathbf{x}_i),y_i)$  as the population and empirical losses, respectively, and consider a proper learning setting with the ground truth  $f_*\triangleq \mathrm{argmin}_{f\in \mathcal{F}}\mathcal{E}(f)$ . Then, SSL with RKD aims to find  $\min_{f\in \mathcal{F}}\widehat{\mathcal{E}}(f) + \widehat{R}_{\mathbf{X}^u}(f)$ . Alternatively, for an overparameterized setting with  $\left(\mathrm{argmin}_{f\in \mathcal{F}}\widehat{\mathcal{E}}(f)\right)\cap$ $\mathcal{F}_{\mathbf{L}(\mathbf{X}^u)}\neq \emptyset$ , SSL with RKD can be formulated as:  $\min_{f\in \mathcal{F}_{\mathbf{L}(\mathbf{X}^u)}}\widehat{\mathcal{E}}(f)$ .
4 Relational Knowledge Distillation as Spectral Clustering
In this section, we show that minimizing either the population RKD loss (Section 4.1) or the empirical RKD loss (Section 4.2) leads to low clustering errors  $\mu\left(\mathcal{F}_{\mathbf{L}(\mathcal{X})}\right)$  and  $\mu\left(\mathcal{F}_{\mathbf{L}(\mathbf{X}^{u})}\right)$ , respectively.
4.1 Relational Knowledge Distillation over Population
Starting with minimization of the population RKD loss  $f \in \mathcal{F}_{\mathbf{L}(\mathcal{X})} = \operatorname*{argmin}_{f \in \mathcal{F}} R(f)$  (Equation (1)), let  $\lambda(\mathbf{L}(\mathcal{X})) = (\lambda_1, \dots, \lambda_{|\mathcal{X}|})$  be the eigenvalues of the graph Laplacian in the ascending order,  $0 = \lambda_1 \leq \dots \leq \lambda_{|\mathcal{X}|}$  with an arbitrary breaking of ties.
A key assumption of RKD is that the teacher model  $\psi$  (with the corresponding graph-revealing kernel  $k_{\psi}$ ) is well aligned with the underlying ground truth partition, formalized as below:
Assumption 4.1 (Appropriate teacher models). For  $G_{\mathcal{X}}$  unveiled through the teacher model  $k_{\psi}(\cdot, \cdot)$ , we assume  $\lambda_{K+1} > 0$  (i.e.,  $G_{\mathcal{X}}$  has at most  $K$  disconnected components); while the ground truth classes  $\{\mathcal{X}_k\}_{k \in [K]}$  are well-separated by  $G_{\mathcal{X}}$  such that  $\alpha \triangleq \frac{\sum_{k \in [K]} \sum_{\mathbf{x} \in \mathcal{X}_k} \sum_{\mathbf{x}' \notin \mathcal{X}_k} w_{\mathbf{x}\mathbf{x}'}}{2\sum_{\mathbf{x} \in \mathcal{X}} \sum_{\mathbf{x}' \in \mathcal{X}} w_{\mathbf{x}\mathbf{x}'}} \ll 1$  (i.e., the fraction of weights of inter-class edges is sufficiently small).
Overparameterization is a ubiquitous scenario when learning with neural networks:  $f \in \mathcal{F}$  is parameterized by  $\theta \in \Theta$  with  $\dim(\Theta) > N + n$  such that the minimization problem has infinitely many solutions.
In particular,  $\alpha \in [0,1]$  reflects the extent to which the ground truth classes  $\{\mathcal{X}_k\}_{k\in [K]}$  are separated from each other, as quantified by the fraction of edge weights between nodes of disparate classes. In the ideal case,  $\alpha = 0$  when the ground truth classes are perfectly separated such that every  $\mathcal{X}_k$  is a connected component of  $G_{\mathcal{X}}$ , while  $\lambda_{K + 1} > \lambda_K = 0$ .
Meanwhile, to reduce the generic function class  $\mathcal{F}$  down to a class of reasonable prediction functions, we introduce the following regularity conditions on the boundedness and margin: