SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Assumption 4.2 ( $\beta$ -skeleton boundedness and  $\gamma$ -margin). For any  $f \in \mathcal{F}$  with  $P(M(f) \cap \mathcal{X}_k) \leq P(\mathcal{X}_k) / 2$  for all  $k \in [K]$ , we assume there exists a skeleton subset  $\mathbf{S} = [\mathbf{s}_1; \ldots; \mathbf{s}_K] \in \mathcal{X}^K$  with  $\mathbf{s}_k = \operatorname{argmax}_{\mathbf{x} \in \mathcal{X} \setminus M(f)} f(\mathbf{x})_k$  such that  $y_f(\mathbf{s}_k) = k$  for every  $k \in [K]$  and  $\operatorname{rank}(f(\mathbf{S})) = K$ .
(i) We say that  $f$  is  $\beta$ -skeleton bounded if  $\sigma_1(f(\mathbf{S})) \leq \beta$  for some reasonably small  $\beta$ .
(ii) Let the  $k$ -th margin of  $f$  be  $\gamma_k \triangleq f(\mathbf{s}_k)_k - \max_{\mathbf{x} \in M(f): y_f(\mathbf{x}) \neq k} f(\mathbf{x})_k$ . We say that  $f$  has a  $\gamma$ -margin if  $\min_{k \in [K]} \gamma_k > \gamma$  for some sufficiently large  $\gamma > 0$ .
Intuitively,  $\mathbf{S}$  can be viewed as a set of  $K$  samples where  $f$  makes the "most confident" prediction in each class. Assumption 4.2 ensures the boundedness of predictions made on such a skeleton subset  $\| f(\mathbf{S})\|_2 \leq \beta$ , as well as a margin  $\gamma$  by which the "most confident" prediction of  $f$  in each class  $\mathbf{s}_k$  can be separated from the minority samples predicted to lie in other classes  $\{\mathbf{x} \in M(f)|y_f(\mathbf{x}) \neq k\}$ . As a toy example, when  $y_f: \mathcal{X} \to [K]$  is surjective, and the columns in  $f(\mathcal{X}) \in \mathbb{R}^{|\mathcal{X}| \times K}$  consist of identity vectors of the predicted clusters  $f(\mathbf{x})_k = \mathbb{1}\{y_f(\mathbf{x}) = k\}$ , Assumption 4.2 is satisfied with  $\beta = 1$  and  $\gamma = 1$ . Meanwhile, the following Remark 4.1 highlights that although Assumption 4.2 cannot be guaranteed with spectral clustering alone, it is generally satisfied in the semi-supervised learning setting with additional supervision/regularization (cf. Example C.1).
Remark 4.1 (Limitation of spectral clustering alone). Notice that for a generic function class  $\mathcal{F}$ , spectral clustering alone is not sufficient to guarantee Assumption 4.2, especially the existence of a skeleton subset  $\mathbf{S}$  or a large enough margin  $\gamma$ . As counter-exemplified in Example C.1, Equation (1) can suffer from large clustering error  $\mu\left(\mathcal{F}_{\mathbf{L}(\mathcal{X})}\right)$  when applied alone and failing to satisfy Assumption 4.2.
To learn predictions with accurate clustering, RKD requires either (i) additional supervision/regularization in end-to-end settings like semi-supervised learning or (ii) further fine-tuning in unsupervised representation learning settings like contrastive learning [HaoChen et al., 2021]. For end-to-end learning in practice, RKD is generally coupled with standard KD (i.e., feature matching) and weak supervision [Park et al., 2019, Liu et al., 2021, Ma et al., 2022], both of which help the learned function  $f$  satisfy Assumption 4.2 with a reasonable margin  $\gamma$ .
Throughout this work, we assume  $\mathcal{F}$  is sufficiently regularized (with weak supervision in Section 5 and consistency regularization in Section 6) such that Assumption 4.2 always holds. Under Assumption 4.1 and Assumption 4.2, the clustering error of RKD over the population  $\mu\left(\mathcal{F}_{\mathbf{L}(\mathcal{X})}\right)$  is guaranteed to be small given a good teacher model  $\psi$  that leads to a negligible  $\alpha$ :
Theorem 4.1 (RKD over population, proof in Appendix C.1). Under Assumption 4.1 and Assumption 4.2 for every  $f \in \mathcal{F}_{\mathbf{L}(\mathcal{X})}$ , the clustering error with the population (Equation (1)) satisfies
$$
\mu \left(\mathcal {F} _ {\mathbf {L} (\mathcal {X})}\right) \leq 2 \max  \left(\frac {\beta^ {2}}{\gamma^ {2}}, 1\right) \cdot \frac {\alpha}{\lambda_ {K + 1}}.
$$
Theorem 4.1 suggests that the clustering error over the population is negligible under mild regularity assumptions (i.e., Assumption 4.2) when (i) the ground truth classes are well-separated by  $G_{\mathcal{X}}$  revealed through the teacher model  $k_{\psi}(\cdot, \cdot)$  (i.e.,  $\alpha \ll 1$  in Assumption 4.1) and (ii) the  $(K + 1)$ th eigenvalue  $\lambda_{K + 1}$  of the graph Laplacian  $\mathbf{L}(\mathcal{X})$  is not too small. As we review in Appendix C.3, the existing result Lemma C.4 [Louis and Makarychev, 2014] unveils the connection between  $\lambda_{K + 1}$  and the sparsest  $K$ -partition (Definition C.1) of  $G_{\mathcal{X}}$ . Intuitively, a reasonably large  $\lambda_{K + 1}$  implies that the partition of the  $K$  ground truth classes is the "only" partition of  $G_{\mathcal{X}}$  into  $K$  components by removing a sparse set of edges (Corollary C.5).
Recall the majority labeling (Equation (2)) and the associated minority subset  $M(f)$  which intuitively describes the difference between clustering of the  $y_{*}$  and that of  $y_{f}$ . We denote  $f(\mathbf{x}) = [f(\mathbf{x})_1, \dots, f(\mathbf{x})_K]$ .
4.2 Relational Knowledge Distillation on Unlabeled Samples
We now turn our attention to a more practical scenario with limited unlabeled samples and bound the clustering error  $\mu\left(\mathcal{F}_{\mathbf{L}(\mathbf{X}^u)}\right)$  granted by minimizing the empirical RKD loss (Equation (3)).
To cope with  $\mathcal{F} \ni f: \mathcal{X} \to \mathbb{R}^K$ , we recall the notion of Rademacher complexity for vector-valued functions from Maurer [2016]. Let  $\operatorname{Rad}(\rho)$  be the Rademacher distribution (i.e., with uniform probability  $\frac{1}{2}$  over  $\{-1, 1\}$ ) and  $\rho \sim \operatorname{Rad}^{N \times K}$  be a random matrix with i.i.d. Rademacher entries. Given any  $N \in \mathbb{N}$ , with  $f(\mathbf{x})_k$  denoting the  $k$ -th entry of  $f(\mathbf{x}) \in \mathbb{R}^K$ , we define
$$
\mathfrak {R} _ {N} \left(\mathcal {F}\right) \triangleq \mathbb {E} _ {\substack {\mathbf {X} ^ {u} \sim P (\mathbf {x}) ^ {N} \\ \boldsymbol {\rho} \sim \operatorname {R a d} ^ {N \times K}}} \left[ \sup  _ {f \in \mathcal {F}} \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {k = 1} ^ {K} \rho_ {i k} \cdot f \left(\mathbf {x} _ {i} ^ {u}\right) _ {k} \right] \tag{4}
$$
as the Rademacher complexity of the vector-valued function class  $\mathcal{F} \ni f: \mathcal{X} \to \mathbb{R}^K$ .
Since the empirical RKD loss is an unbiased estimate of its population correspondence (Proposition D.1), we can leverage the standard generalization analysis in terms of Rademacher complexity to provide an unlabeled sample complexity for RKD.
Theorem 4.2 (Unlabeled sample complexity of RKD, proof in Appendix D.2). Assume there exist  $B_{f}, B_{k_{\psi}} > 0$  such that  $\| f(\mathbf{x})\|_2^2 \leq B_f$  and  $k_{\psi}(\mathbf{x},\mathbf{x}') \leq B_{k_{\psi}}$  for all  $\mathbf{x},\mathbf{x}' \in \mathcal{X}$ ,  $f \in \mathcal{F}$ . Given any  $f_{|\mathbf{X}^u} \in \mathcal{F}_{\mathbf{L}(\mathbf{X}^u)}$ ,  $f_{|\mathcal{X}} \in \mathcal{F}_{\mathbf{L}(\mathcal{X})}$ ,  $\delta \in (0,1)$ , with probability at least  $1 - \delta / 2$  over  $\mathbf{X}^u \sim P(\mathbf{x})^N$ ,
$$
R \left(f _ {| \mathbf {X} ^ {u}}\right) - R \left(f _ {| \mathcal {X}}\right) \leq 1 6 \sqrt {2 B _ {f}} \left(B _ {f} + B _ {k _ {\psi}}\right) \Re_ {N / 2} (\mathcal {F}) + 2 \left(B _ {k _ {\psi}} + B _ {f}\right) ^ {2} \sqrt {\frac {\log (4 / \delta)}{N}}.
$$
Theorem 4.2 implies that the unlabeled sample complexity of RKD is dominated by the Rademacher complexity of  $\mathcal{F}$ . In Appendix D.3, we further concretize Theorem 4.2 by instantiating  $\Re_{N/2}(\mathcal{F})$  via existing Rademacher complexity bounds for neural networks [Golowich et al., 2018].
With  $\Delta \rightarrow 0$  as  $N$  increasing (Theorem 4.2), the clustering error of minimizing the empirical RKD loss can be upper bounded as follows.
Theorem 4.3 (RKD on unlabeled samples, proof in Appendix D.4). Under Assumption 4.1 and Assumption 4.2 for every  $f \in \mathcal{F}_{\mathbf{L}(\mathbf{X}^u)}$ , given any  $\delta \in (0,1)$ , if there exists  $\Delta < (1 - \lambda_K)^2$  such that  $R(f_{|\mathbf{X}^u}) \leq R(f_{|\mathcal{X}}) + \Delta$  for all  $f_{|\mathbf{X}^u} \in \mathcal{F}_{\mathbf{L}(\mathbf{X}^u)}$  and  $f_{|\mathcal{X}} \in \mathcal{F}_{\mathbf{L}(\mathcal{X})}$  with probability at least  $1 - \delta / 2$  over  $\mathbf{X}^u \sim P(\mathbf{x})^N$ , then error of clustering with the empirical graph (Equation (3)) satisfies the follows: for any  $K_0 \in [K]$  such that  $\lambda_{K_0} < \lambda_{K+1}$  and  $C_{K_0} \triangleq \frac{(1 - \lambda_{K_0})^2 - (1 - \lambda_K)^2}{\Delta} = O(1)$ ,
$$
\mu \left(\mathcal {F} _ {\mathbf {L} (\mathbf {X} ^ {u})}\right) \leq 2 \max  \left(\frac {\beta^ {2}}{\gamma^ {2}}, 1\right) \cdot \left(\frac {\alpha}{\lambda_ {K + 1}} + \frac {(1 + (K - K _ {0}) C _ {K _ {0}}) \Delta}{(1 - \lambda_ {K _ {0}}) ^ {2} - (1 - \lambda_ {K + 1}) ^ {2}}\right).
$$
Theorem 4.3 ensures that given sufficient unlabeled samples (proportional to the Rademacher complexity of  $\mathcal{F}$ ) such that  $\Delta \to 0$ , the clustering error  $\mu\left(\mathcal{F}_{\mathbf{L}(\mathbf{X}^u)}\right)$  from empirical RKD is nearly as low as  $\mu\left(\mathcal{F}_{\mathbf{L}(\mathcal{X})}\right)$  from population RKD, up to an additional error term that scales linearly in  $\Delta$ .
5 Label Efficiency of Cluster-aware Semi-supervised Learning
In this section, we demonstrate the label efficiency of learning from a function class with low clustering error (Definition 3.1), like the ones endowed by RKD discussed in Section 4.
Specifically, given any cluster-aware function subclass  $\mathcal{F}' \subseteq \mathcal{F}$  with low clustering error (e.g.,  $\mathcal{F}_{\mathbf{L}(\mathcal{X})}$  and  $\mathcal{F}_{\mathbf{L}(\mathbf{X}^u)}$ ), for a set of  $n$  i.i.d. labeled samples  $(\mathbf{X}, \mathbf{y}) \sim P(\mathbf{x}, y)^n$ , we have the following generalization guarantee:
Theorem 5.1 (Label complexity, proof in Appendix B.1). Given any cluster-aware  $\mathcal{F}' \subseteq \mathcal{F}$  with  $\mu(\mathcal{F}') \ll 1$ , assuming that  $(\mathbf{X}, \mathbf{y})$  contains at least one sample per class, for any  $\delta \in (0,1)$ , with probability at least  $1 - \delta/2$  over  $(\mathbf{X}, \mathbf{y}) \sim P(\mathbf{x}, y)^n$ ,  $\widehat{f} \in \operatorname*{argmin}_{f \in \mathcal{F}'} \widehat{\mathcal{E}}(f)$  satisfies
$$
\mathcal {E} (\widehat {f}) - \mathcal {E} (f _ {*}) \leq 4 \sqrt {\frac {2 K \log (2 n)}{n} + 2 \mu (\mathcal {F} ^ {\prime})} + \sqrt {\frac {2 \log (4 / \delta)}{n}}, \tag {5}
$$
e.g., conditioned on  $\mathbf{X}^u$ ,  $\widehat{f}_{|\mathbf{X}^u} \in \operatorname*{argmin}_{f \in \mathcal{F}_{\mathbf{L}(\mathbf{X}^u)}} \widehat{\mathcal{E}}(f)$  satisfies Equation (5) with  $\mu(\mathcal{F}_{\mathbf{L}(\mathbf{X}^u)})$ .
Theorem 5.1 implies that with low clustering error (e.g., endowed by unsupervised methods like RKD in Theorem 4.1/Theorem 4.3, DAC in Theorem 6.2), the labeled sample complexity is as low as  $\widetilde{O}(K)$ , linear in the number of clusters and asymptotically optimal up to a logarithmic factor.
Remark 5.1 (Class imbalance, elaborated in Appendix B.2). In Theorem 5.1, while the label complexity scales as  $n = \widetilde{O}(K)$  with i.i.d. sampling, we meanwhile assume that  $(\mathbf{X}, \mathbf{y})$  contains at least one labeled sample per class. Intuitively, when the  $K$  classes are balanced (i.e.,  $|\mathcal{X}_k| = |\mathcal{X}| / K$  for all  $k \in [K]$ ), an analogy to the coupon collector problem implies that  $n = O(K \log(K))$  i.i.d. labeled samples are sufficient in expectation. Nevertheless, with class imbalance, the label complexity can be much worse (e.g., when  $|\mathcal{X}_K| \ll |\mathcal{X}_k|$  for all  $k \in [K-1]$ , collecting one label  $K$  takes  $1 / P(\mathcal{X}_K) \gg K$  labeled samples). In Appendix B.2, we show that such label inefficiency can be circumvented by leveraging a cluster-aware prediction  $f \in \mathcal{F}'$  and drawing  $O(\log(K))$  labeled samples uniformly from each of the  $K$  predicted clusters, instead of i.i.d. from the entire population.
Remark 5.2 (Coreset selection). While Theorem 5.1 yields an asymptotically optimal label complexity guarantee with i.i.d. sampling, it is of significant practical value to be more judicious in selecting labeled samples, especially for the low-label-rate regime (e.g.,  $n = O(K)$ ). Therefore, in the experiments (Appendix A), we further investigate—alongside i.i.d. sampling<sup>8</sup)—a popular coreset selection method [Bilmes, 2022, Krause and Golovin, 2014], where the coreset  $(\mathbf{X}, \mathbf{y})$  is statistically more representative of the data distribution  $P$ .
In particular, coreset selection can be recast as a facility location problem [Krause and Golovin, 2014] characterized by the teacher model:  $\max_{\mathbf{X} \subset \mathcal{X}} \sum_{\mathbf{x}' \in \mathcal{X}} \max_{\mathbf{x} \in \mathbf{X}} k_{\psi}(\mathbf{x}, \mathbf{x}')$ , whose optimizers can be approximated heuristically via the stochastic greedy [Mirzasoleiman et al., 2015] submodular optimization algorithm [Schreiber et al., 2020]. Intuitively, the facility location objective encourages the coreset  $\mathbf{X}$  to be representative of the entire population  $\mathcal{X}$  in a pairwise similarity sense, and the coreset approximation identified by the stochastic greedy method is nearly optimal (in the facility location objective) up to a multiplicative constant [Mirzasoleiman et al., 2015].
6 Data Augmentation Consistency Regularization as Clustering
In light of the broadness of the notion of clustering, investigating the discrepancy between different types of cluster-awareness is crucial for understanding the effect of spectral clustering brought by RKD beyond the low clustering error demonstrated in Section 4. In this section, we leverage the existing theoretical tools from Wei and Ma [2019], Cai et al. [2021] to unify DAC regularization into the cluster-aware semi-supervised learning framework in Theorem 5.1. Specifically, we demonstrate the effect of DAC as a "local" expansion-based clustering [Yang et al., 2023], which works in complement with the "global" spectral clustering facilitated by RKD.
Start by recalling the formal notion of expansion-based data augmentation (Definition 6.1) and data augmentation consistency (DAC) error (Definition 6.2) from Wei et al. [2021]:
Definition 6.1 (Expansion-based data augmentation ([Wei et al., 2021] Definition 3.1)). For any  $\mathbf{x} \in \mathcal{X}$ , we consider a set of class-invariant data augmentations  $\mathcal{A}(\mathbf{x}) \subset \mathcal{X}$  such that  $\{\mathbf{x}\} \subsetneq \mathcal{A}(\mathbf{x}) \subseteq \mathcal{X}_{y_{*}(x)}$ . We say  $\mathbf{x}, \mathbf{x}' \in \mathcal{X}$  lie in neighborhoods of each other if their augmentation sets have a non-empty intersection:  $\mathrm{NB}(\mathbf{x}) \triangleq \{\mathbf{x}' \in \mathcal{X} \mid \mathcal{A}(\mathbf{x}) \cap \mathcal{A}(\mathbf{x}') \neq \emptyset\}$  for  $\mathbf{x} \in \mathcal{X}$ ; and  $\mathrm{NB}(S) \triangleq \bigcup_{\mathbf{x} \in S} \mathrm{NB}(\mathbf{x})$  for  $S \subseteq \mathcal{X}$ . Then, we quantify strength of such data augmentations via the  $c$ -expansion property ( $c > 1$ ): for any  $S \subseteq \mathcal{X}$  such that  $P(S \cap \mathcal{X}_k) \leq P(\mathcal{X}_k) / 2 \forall k \in [K]$ ,
$$
P \left(\operatorname {N B} (S) \cap \mathcal {X} _ {k}\right) > \min  \left\{c \cdot P \left(S \cap \mathcal {X} _ {k}\right), P \left(\mathcal {X} _ {k}\right) \right\} \forall k \in [ K ].
$$
Definition 6.2 (DAC error [Wei et al., 2021] (3.3)). For any  $f \in \mathcal{F}$  and  $\mathcal{F}' \subseteq \mathcal{F}$ , let
$$
\nu \left(f\right) \triangleq \mathbb {P} _ {\mathbf {x} \sim P (\mathbf {x})} \left[ \exists \mathbf {x} ^ {\prime} \in \mathcal {A} (\mathbf {x}) \text {s . t .} y _ {f} (\mathbf {x}) \neq y _ {f} (\mathbf {x} ^ {\prime}) \right], \quad \nu \left(\mathcal {F} ^ {\prime}\right) \triangleq \sup  _ {f \in \mathcal {F} ^ {\prime}} \nu (f).
$$
7Notice that assuming at least one labeled sample per class (i.e.,  $n\geq K$  ) is necessary for any generalization guarantees. Otherwise, depending on the missing classes, the excess risk in Equation (5) can be arbitrarily bad.
For the low-label-rate experiments, to ensure consistent performance, we follow the common practice [Calder et al., 2020] and slightly deviate from the assumption by sampling i.i.d. from each ground truth class instead.
We adopt the theoretical framework in Wei et al. [2021] for generic neural networks with smooth activation function  $\phi$  and consider
$$
\mathcal {F} = \left\{f (\mathbf {x}) = \mathbf {A} _ {p} \phi \left(\dots \mathbf {A} _ {2} \phi \left(\mathbf {A} _ {1} \mathbf {x}\right) \dots\right) \Bigg |   \mathbf {A} _ {\iota} \in \mathbb {R} ^ {d _ {\iota} \times d _ {\iota - 1}} \forall \iota \in [ p ], d = \max  _ {\iota = 0, \dots , p} d _ {\iota} \right\}.
$$
With such function class  $\mathcal{F}$  of  $p$ -layer neural networks with maximum width  $d$  and weights  $\{\mathbf{A}_{\iota}\}_{\iota = 1}^{p}$ , we recall the notion of all-layer margin from Wei et al. [2021] Appendix C.2. By decomposing  $f = f_{2p - 1} \circ \dots \circ f_1$  such that  $f_{2\iota - 1}(\mathbf{z}) = \mathbf{A}_{\iota}\mathbf{z}$  for all  $\iota \in [p]$  and  $f_{2\iota}(\mathbf{z}) = \phi (\mathbf{z})$  for all  $\iota \in [p - 1]$ , we consider a perturbation  $\delta = (\delta_{1},\dots ,\delta_{2p - 1})$  (where  $\delta_{2\iota -1},\delta_{2\iota}\in \mathbb{R}^{d_{\iota}}$ ) to each layer of  $f$ :
$$
\begin{array}{l} f _ {1} (\mathbf {x}, \boldsymbol {\delta}) = f _ {1} (\mathbf {x}, \boldsymbol {\delta} _ {1}) = f _ {1} (\mathbf {x}) + \boldsymbol {\delta} _ {1} \| \mathbf {x} \| _ {2}, \\ f _ {\iota} (\mathbf {x}, \boldsymbol {\delta}) = f _ {\iota} (\mathbf {x}, \boldsymbol {\delta} _ {1}, \dots , \boldsymbol {\delta} _ {\iota}) = f _ {\iota} \left(f _ {\iota - 1} (\mathbf {x}, \boldsymbol {\delta})\right) + \boldsymbol {\delta} _ {\iota} \| f _ {\iota - 1} (\mathbf {x}, \boldsymbol {\delta}) \| _ {2} \\ \end{array}
$$
such that  $f(\mathbf{x}, \boldsymbol{\delta}) = f_{2p-1}(\mathbf{x}, \boldsymbol{\delta})$ . Then, the all-layer margin  $m: \mathcal{F} \times \mathcal{X} \times [K] \to \mathbb{R}_{\geq 0}$  is defined as the minimum norm of  $\boldsymbol{\delta}$  that is sufficient to perturb the classification of  $f(\mathbf{x})$ :
$$
m (f, \mathbf {x}, y) \triangleq \min  _ {\boldsymbol {\delta}} \sqrt {\sum_ {\iota = 1} ^ {2 p - 1} \left\| \boldsymbol {\delta} _ {\iota} \right\| _ {2} ^ {2}} \quad s. t. \quad \underset {k \in [ K ]} {\operatorname {a r g m a x}} f (\mathbf {x}, \boldsymbol {\delta}) _ {k} \neq y. \tag {6}
$$
Moreover, Wei et al. [2021] introduced the robust margin for the expansion-based data augmentation  $\mathcal{A}$  (Definition 6.1), which intuitively quantifies the worst-case preservation of the label-relevant information in the augmentations of  $\mathbf{x}$ , measured with respect to  $f$ :
$$
m _ {\mathcal {A}} (f, \mathbf {x}) \triangleq \min  _ {\mathbf {x} ^ {\prime} \in \mathcal {A} (\mathbf {x})} m (f, \mathbf {x} ^ {\prime}, y _ {f} (\mathbf {x})). \tag {7}
$$
We say the expansion-based data augmentation  $\mathcal{A}$  is margin-robust with respect to the ground truth  $f_{*}$  if  $\min_{\mathbf{x} \in \mathcal{X}} m_{\mathcal{A}}(f_{*}, \mathbf{x}) > 0$  is reasonably large.
Then, we cast DAC regularization<sup>9</sup> as reducing the function class  $\mathcal{F}$  via constraints on the robust margins at the  $N$  unlabeled samples  $\mathbf{X}^u$ : for some  $0 < \tau < \min_{\mathbf{x} \in \mathcal{X}} m_{\mathcal{A}}(f_*, \mathbf{x})$ ,
$$
\mathcal {F} _ {\mathbf {X} ^ {u}} ^ {\tau} \triangleq \left\{f \in \mathcal {F} \mid m _ {\mathcal {A}} \left(f, \mathbf {x} _ {i} ^ {u}\right) \geq \tau \forall i \in [ N ] \right\}. \tag {8}
$$
Leverage the existing unlabeled sample complexity bound for DAC error from Wei et al. [2021]:
Proposition 6.1 (Unlabeled sample complexity of DAC ([Wei et al., 2021] Theorem 3.7)). Given any  $\delta \in (0,1)$ , with probability at least  $1 - \delta /2$  over  $\mathbf{X}^u$ ,
$$
\nu \left(\mathcal {F} _ {\mathbf {X} ^ {u}} ^ {\tau}\right) \leq \widetilde {O} \left(\frac {\sum_ {\iota = 1} ^ {p} \sqrt {d} \| \mathbf {A} _ {\iota} \| _ {F}}{\tau \sqrt {N}}\right) + O \left(\sqrt {\frac {\log (1 / \delta) + p \log (N)}{N}}\right),
$$
where  $\widetilde{O} (\cdot)$  suppresses polylogarithmic factors in  $N$  and  $d$ .
The clustering error of Equation (8) is as low as its DAC error, up to a constant factor that depends on the augmentation strength, characterized by the  $c$ -expansion (Definition 6.1):
Theorem 6.2 (Expansion-based clustering, proof in Appendix E.3). For Equation (8) with  $\mathcal{A}$  admitting  $c$ -expansion (Definition 6.1), the clustering error is upper bounded by the DAC error in Proposition 6.1:
$$
\mu \left(\mathcal {F} _ {\mathbf {X} ^ {u}} ^ {\tau}\right) \leq \max  \left\{\frac {2}{c - 1}, 2 \right\} \cdot \nu \left(\mathcal {F} _ {\mathbf {X} ^ {u}} ^ {\tau}\right).
$$
In Theorem 6.2, (i)  $c$  characterizes the augmentation strength (i.e., the perturbation on label-irrelevant information), whereas (ii)  $\tau$  quantifies the margin robustness (i.e., the preservation of label-relevant information) for the expansion-based data augmentation. Ideally, we would like both  $c$  and  $\tau$  to be reasonably large, while there exist trade-offs between the augmentation strength and margin robustness (e.g., overly strong augmentations inevitably perturb label-relevant information).
In Appendix E.2, we further bridge the conceptual notion of DAC regularization in Equation (8) with common DAC regularization algorithms in practice like FixMatch [Sohn et al., 2020].
Remark 6.1 (Clustering with DAC v.s. RKD). As Figure 1 demonstrated, in contrast to RKD that unveils the spectral clustering of a population-induced graph from a "global" perspective, DAC alternatively learns a "local" clustering structure through the expansion of neighborhoods NB ( $\cdot$ ) characterized by sets of data augmentations  $\mathcal{A}(\cdot)$  (Definition 6.1).
Therefore, DAC and RKD provide complementary perspectives for each other. On one hand, the "local" clustering structure revealed by DAC effectively ensures a reasonable margin in Assumption 4.2 (Remark 4.1). On the other hand, when the augmentation strength  $c$  is insufficient (i.e.,  $\mathcal{A}(\cdot)$  is not expansive enough) for DAC to achieve a low clustering error  $\mu\left(\mathcal{F}_{\mathbf{X}^u}^\tau\right)$  (Theorem 6.2), the supplementary "global" perspective of RKD connects the non-overlapping neighborhoods in the same classes and brings better classification accuracies, as we empirically verified in Appendix A.
7 Discussions, Limitations, and Future Directions
This work provides a theoretical analysis of relational knowledge distillation (RKD) in the semi-supervised classification setting from a clustering perspective. Through a cluster-aware semi-supervised learning (SSL) framework characterized by a notion of low clustering error, we demonstrate that RKD achieves a nearly optimal label complexity (linear in the number of clusters) by learning the underlying geometry of the population as revealed by the teacher model via spectral clustering. With a unified view of data augmentation consistency (DAC) regularization in the cluster-aware SSL framework, we further illustrate the complementary "global" and "local" perspectives of clustering learned by RKD and DAC, respectively.
As an appealing future direction, domain adaptation in knowledge distillation is a common scenario in practice where the training data of the teacher and student models come from different distributions. Although distributional shifts can implicitly reflect the alignment between the ground truth and the teacher model (Assumption 4.1) in our analysis, an explicit and specialized study on RKD in the domain adaptation setting may lead to better theoretical guarantees and deeper insights.