Title: Cluster-aware Semi-supervised Learning: Relational Knowledge Distillation Provably Learns Clustering
Venue: NeurIPS 2023
Accepted: yes
Keywords: Relational knowledge distillation

Semi-supervised learning

Spectral clustering

Sample complexity
Abstract: Despite the empirical success and practical significance of (relational) knowledge distillation that matches (the relations of) features between teacher and student models, the corresponding theoretical interpretations remain limited for various knowledge distillation paradigms. In this work, we take an initial step toward a theoretical understanding of relational knowledge distillation (RKD), with a focus on semi-supervised classification problems. We start by casting RKD as spectral clustering on a population-induced graph unveiled by a teacher model. Via a notion of clustering error that quantifies the discrepancy between the predicted and ground truth clusterings, we illustrate that RKD over the population provably leads to low clustering error. Moreover, we provide a sample complexity bound for RKD with limited unlabeled samples. For semi-supervised learning, we further demonstrate the label efficiency of RKD through a general framework of cluster-aware semi-supervised learning that assumes low clustering errors. Finally, by unifying data augmentation consistency regularization into this cluster-aware framework, we show that despite the common effect of learning accurate clusterings, RKD facilitates a "global" perspective through spectral clustering, whereas consistency regularization focuses on a "local" perspective via expansion.

Cluster-aware Semi-supervised Learning: Relational Knowledge Distillation Provably Learns Clustering

Abstract

Despite the empirical success and practical significance of (relational) knowledge distillation that matches (the relations of) features between teacher and student models, the corresponding theoretical interpretations remain limited for various knowledge distillation paradigms. In this work, we take an initial step toward a theoretical understanding of relational knowledge distillation (RKD), with a focus on semi-supervised classification problems. We start by casting RKD as spectral clustering on a population-induced graph unveiled by a teacher model. Via a notion of clustering error that quantifies the discrepancy between the predicted and ground truth clusterings, we illustrate that RKD over the population provably leads to low clustering error. Moreover, we provide a sample complexity bound for RKD with limited unlabeled samples. For semi-supervised learning, we further demonstrate the label efficiency of RKD through a general framework of cluster-aware semi-supervised learning that assumes low clustering errors. Finally, by unifying data augmentation consistency regularization into this cluster-aware framework, we show that despite the common effect of learning accurate clusterings, RKD facilitates a 'global' perspective through spectral clustering, whereas consistency regularization focuses on a 'local' perspective via expansion.

1 Introduction

The immense volume of training data is a vital driving force behind the unparalleled power of modern deep neural networks. Nevertheless, data collection can be quite resource-intensive; in particular, data labeling may be prohibitively costly. With an aim to lessen the workload associated with data labeling, semi-supervised learning capitalizes on more easily accessible unlabeled data via diverse approaches of pseudo-labeling. For instance, data augmentation consistency regularization [Sohn et al., 2020, Berthelot et al., 2019] generates pseudo-labels from the current model predictions on carefully designed data augmentations. Alternatively, knowledge distillation [Hinton et al., 2015] can be leveraged to gauge representations of unlabeled samples via pretrained teacher models.

Kevin Miller ∗

Rachel Ward

A(x)

X

X

XA

A(x")

̸

Meanwhile, as the scale of training state-of-the-art models explodes, computational costs and limited data are becoming significant obstacles to learning from scratch. In this context, (data-free) knowledge distillation [Liu et al., 2021, Wang and Yoon, 2021] (where pretrained models are taken as teacher oracles without access to their associated training data) is taking an increasingly crucial role in learning efficiently from powerful existing models.

Despite the substantial progresses in knowledge distillation (KD) in practice, its theoretical understanding remains limited, possibly due to the abundant diversity of KD strategies. Most existing theoretical analyses of KD [Ji and Zhu, 2020, Allen-Zhu and Li, 2020, Harutyunyan et al., 2023, Hsu et al., 2021, Wang et al., 2022] focus on feature matching between the teacher and student models [Hinton et al., 2015], while a much broader spectrum of KD algorithms [Liu et al., 2019, Park et al., 2019, Chen et al., 2021, Qian et al., 2022] demonstrates appealing performance in practice. Recently, relational knowledge distillation (RKD) [Park et al., 2019, Liu et al., 2019] has achieved remarkable empirical successes by matching the inter-feature relationships (instead of features themselves) between the teacher and student models. This drives us to focus on RKD and motivates our analysis with the following question:

What perspective of the data does relational knowledge distillation learn, and how efficiently?

(RKD learns spectral clustering.) In light of the connection between RKD and instance relationship graphs (IRG) [Liu et al., 2019] 2 , intuitively, RKD learns the ground truth geometry through a graph induced by the teacher model. We formalize this intuition from a spectral clustering perspective (Section 4). Specifically, for a multi-class classification problem, we introduce a notion of clustering error (Definition 3.1) that measures the difference between the predicted and ground truth class partitions. Through low clustering error guarantees, we illustrate that RKD over the population learns the ground truth partition (presumably unveiled by the teacher model) via spectral clustering (Section 4.1). Moreover, RKD on sufficiently many unlabeled samples learns a similar partition with low clustering error (Section 4.2). By appealing to standard generalization analysis [Wainwright, 2019, Bartlett and Mendelson, 2003, Wei and Ma, 2019] for unbiased estimates of the parameterized RKD loss, we show that the unlabeled sample complexity of RKD is dominated by a polynomial term in the model complexity (Theorem 4.2, Theorem 4.3).

As one of the most influential and classical unsupervised learning strategies, clustering remains an essential aspect of modern learning algorithms with weak/no supervision. For example, contrastive learning [Caron et al., 2020, Grill et al., 2020, Chen and He, 2021] has proven to learn good representations without any labels by encouraging (spectral) clustering [HaoChen et al., 2021, Lee et al., 2021, Parulekar et al., 2023]. However, the effects of clustering in the related schemes of semisupervised learning (SSL) are less well investigated. Therefore, we pose a follow-up question:

How does cluster-aware semi-supervised learning improve label efficiency and generalization?

(Cluster-aware SSL is label-efficient.) We formulate a cluster-aware SSL framework (Section 5) where assuming a low clustering error (Definition 3.1) endowed by learning from unlabeled data, the label complexity scales linearly in the number of clusters, up to a logarithmic factor (Theorem 5.1).

Since clustering is a rather generic notion that can be facilitated from various perspectives, to better understand spectral clustering as learned by RKD, we compare the clustering mechanism of RKD with that of another common example of cluster-aware SSL-data augmentation consistency (DAC) regularization, while exploring the following question:

What are the similarities and discrepancies between different cluster-aware strategies?

(RKD and DAC learn clustering from complementary perspectives.) We unify the existing analysis [Yang et al., 2023] tailored for DAC regularization [Sohn et al., 2020] into the cluster-aware SSL framework (Section 6). In particular, while facilitating clustering similarly to RKD, DAC reduces the clustering error through an expansion-based mechanism characterized by 'local' neighborhoods induced by data augmentations (Theorem 6.2). This serves as a complementary perspective to RKD which approximates spectral clustering on a graph Laplacian that reflects the underlying 'global' structure of the population (Figure 1).

2 Related Works

Relational knowledge distillation. Since the seminal work of Hinton et al. [2015], knowledge distillation has become a foundational method for time and memory-efficient deep learning. Recent years have seen different KD strategies. Some works learn to directly match the output (response) [Hinton et al., 2015, Chen et al., 2017, Ba and Caruana, 2014] or intermediate layers (features) [Romero et al., 2014, Zagoruyko and Komodakis, 2016a, Kim et al., 2018] of the teacher network. More recently, Park et al. [2019] introduced the idea of relational knowledge distillation (RKD), and concurrent work [Liu et al., 2019] introduced the 'instance relationship graph' (IRG). They essentially presented similar ideas to learn the inter-feature relationship between samples instead of matching models' responses or features individually. A comprehensive survey of relational (relation-based) KD can be found in [Gou et al., 2021].

Despite the empirical success stories, the theoretical understanding of KD remained limited. Prior theoretical work is mostly constrained to linear classifier/nets [Phuong and Lampert, 2019, Ji and Zhu, 2020] or neural tangent kernel (NTK) analysis [Harutyunyan et al., 2023]. Hsu et al. [2021] leverages knowledge distillation to improve the sample complexity analysis of large neural networks with potentially vacuous generalization bounds.

Data augmentation consistency regularization. Data augmentation serves as an implicit or explicit regularization to improve sample efficiency and avoid overfitting. Initially, people applied simple transformations to labeled samples and add them to the training set [Krizhevsky et al., 2017, Simard et al., 2002, Simonyan and Zisserman, 2014, He et al., 2016, Cubuk et al., 2018]. Traditional transformations preserve image semantics, including (random) perturbations, distortions, scales, crops, rotations, and horizontal flips. Subsequent augmentations may alter the labels jointly with the input samples, e.g. Mixup [Zhang et al., 2017], Cutout [DeVries and Taylor, 2017], and Cutmix [Yun et al., 2019]. Theoretical works have demonstrated different functionalities of data augmentation such as incorporating invariances [Mei et al., 2021], amplifying strong features [Shen et al., 2022], or reducing variance during the learning procedure [Chen et al., 2020].

Recent practices explicitly add consistency regularization to enforce prediction or representation similarities [Laine and Aila, 2016, Bachman et al., 2014, Sohn et al., 2020]. Not only does this way incorporates unlabeled samples, but Yang et al. [2023] also theoretically demonstrated other benefits: explicit consistency regularization reduces sample complexity more significantly and handles misspecification better than simply adding augmented data into the training set.

Graph-based learning. Although we leverage graphs to model the underlying clustering structure of the population, the RKD algorithm that we study applies to generic data distributions and does not involve any graph constructions over the training data. The latter is crucial for modern learning settings as graph construction can be prohibitively expensive on common large-scale datasets.

Nevertheless, assuming that explicit graph construction is affordable/available, there exists a rich history of nonparametric graph-based semi-supervised classification models for transductive learn-

ing [Zhu et al., 2003, Bertozzi and Merkurjev, 2019, Zhou et al., 2003, Belkin et al., 2004, Belkin and Niyogi, 2004]. These methods have seen success in achieving high accuracy results in the low-label rate regime [Calder et al., 2020, Bertozzi and Merkurjev, 2019]. Furthermore, assuming inherited graph structures of data, recent progress in graph neural networks (GNN) [Zhou et al., 2020] and graph convolutional networks (GCN) [Welling and Kipf, 2016] has provided a connection between these classical graph-based methods and powerful deep learning models for a variety of problem settings [Zhou et al., 2020].

3 Problem Setup

Notations. For any event e , let ✶ { e } = 1 if e is true, and ✶ { e } = 0 otherwise. For any positive integers n, K ∈ N , we denote [ n ] = { 1 , . . . , n } and ∆ K ≜ { ( p 1 , . . . , p K ) ∈ [0 , 1] K ∣ ∣ ∣ ∑ K k =1 p k = 1 } . For a finite set X , |X| denotes the size of X , and Unif ( X ) denotes the uniform distribution over X ( i.e. , Unif( x ) = 1 / |X| for all x ∈ X ). For a distribution P over X and any n ∈ N , P n represents the joint distribution over X n such that { x i } i ∈ [ n ] ∼ P n is a set of n i.i.d. samples. Given any matrix M ∈ R n × k ( n ≥ k without loss of generality), let σ ( M ) = { σ i ( M ) | i ∈ [ k ] } be the singular values of M with σ 1 ( M ) ≥ · · · ≥ σ k ( M ) . While for any symmetric M ∈ R n × n , let λ ( M ) = { λ i ( M ) | i ∈ [ n ] } be the eigenvalues of M such that λ 1 ( M ) ≤ · · · ≤ λ n ( M ) . Given any labeling function y : X → [ K ] , let - → y : X → { 0 , 1 } K be its one-hot encoding.

3.1 Spectral Clustering on Population-induced Graph

We consider a K -class classification problem over an unknown data distribution P : X × [ K ] → [0 , 1] (while overloading P for the probability measure over X without ambiguity). Let y ∗ : X → [ K ] be the ground truth classification that introduced a natural partition {X k } k ∈ [ K ] where X k ≜ { x ∈ X | y ∗ ( x ) = k } such that ⋃ K k =1 X k = X and X k ∩ X k ′ = ∅ for all k = k ′ . Meanwhile, we specify a function class F ⊂ { f : X → R K } to learn from where each f ∈ F is associated with a prediction function y f ( x ) ≜ argmax k ∈ [ K ] f ( x ) k .

̸

We characterize the geometry of the data population 3 X with an undirected weighted graph G X = ( X , W ( X )) such that: (i) For every vertex pair x , x ′ ∈ X , the edge weight w xx ′ ≥ 0 characterizes the similarity between ( x , x ′ ) . For example, w xx ′ between a pair of samples from the same class is larger than that between a pair from different classes (ii) P ( x ) = w x ≜ ∑ x ′ ∈X w xx ′ such that ∑ x ∈X ∑ x ′ ∈X w xx ′ = 1 , intuitively implying that a more representative instance x (corresponding to a larger w x ) has a higher probability of being sampled. We remark that such population-induced graph G X is reminiscent of the population augmentation graph for studying contrastive learning 4 [HaoChen et al., 2021].

Let W ( X ) with W xx ′ ( X ) = w xx ′ be the weighted adjacency matrix of G X ; let D ( X ) be the diagonal matrix of the weighted degrees { w x } x ∈X . The (normalized) graph Laplacian takes the form L ( X ) = I -W ( X ) where W ( X ) ≜ D ( X ) -1 / 2 W ( X ) D ( X ) -1 / 2 is the normalized adjacency matrix. Then, the spectral clustering [von Luxburg, 2007] on G X can be expressed as a (rankK ) Nyström approximation of W ( X ) in terms of the weighted outputs D ( X ) 1 2 f ( X ) :

To quantify the alignment between the ground truth class partition and the clustering predicted by f ∈ F , we introduce the notion of clustering error.

Definition 3.1 (Clustering error) . Given any f ∈ F , we define the majority labeling

̸

along with the minority subsets associated with f : M ( f ) ≜ { x ∈ X | ˜ y f ( x ) = y ∗ ( x ) } such that P ( M ( f )) quantifies the clustering error of f . For any F ′ ⊆ F , let µ ( F ′ ) ≜ sup f ∈F ′ P ( M ( f )) .

Intuitively, M ( f ) characterizes the difference between K -partition of X by the ground truth y ∗ and by the prediction function y f associated with f ; while µ ( F ′ ) quantifies the worse-case clustering error of all functions f ∈ F ′ . In Section 4.1, we will demonstrate that spectral clustering on G X (Equation (1)) leads to provably low clustering error µ ( F L ( X ) ) .

3.2 Relational Knowledge Distillation

For RKD, we assume access to a proper teacher model ψ : X → W (for a latent feature space W ) that induces a graph-revealing kernel: k ψ ( x , x ′ ) = w xx ′ w x w x ′ . Then, the spectral clustering on G X in Equation (1) can be interpreted as the population RKD loss :

While with only limited unlabeled samples X u = { x u j ∣ ∣ j ∈ [ N ] } ∼ P ( x ) N in practice, we consider the analogous empirical RKD loss :

where ̂ R X u ( f ) serves as an unbiased estimate for R ( f ) (Proposition D.1).

̸

Further, for semi-supervised setting with a small set of labeled samples ( X , y ) = { ( x i , y i ) } i ∈ [ n ] ∼ P ( x , y ) n (usually n ≪ N ) independent of X u , let ℓ : R K × [ K ] → { 0 , 1 } be the zero-one loss: ℓ ( f ( x ) , y ) = ✶ { y f ( x ) = y } . We denote E ( f ) ≜ E ( x ,y ) ∼ P [ ℓ ( f ( x ) , y )] and ̂ E ( f ) ≜ 1 n ∑ n i =1 ℓ ( f ( x i ) , y i ) as the population and empirical losses, respectively, and consider a proper learning setting with the ground truth f ∗ ≜ argmin f ∈F E ( f ) . Then, SSL with RKD aims to find min f ∈F ̂ E ( f )+ ̂ R X u ( f ) . Alternatively, for an overparameterized setting 5 with ( argmin f ∈F ̂ E ( f ) ) ∩ F L ( X u ) = ∅ , SSL with RKD can be formulated as: min f ∈F L ( X u ) ̂ E ( f ) .

̸

4 Relational Knowledge Distillation as Spectral Clustering

In this section, we show that minimizing either the population RKD loss (Section 4.1) or the empirical RKDloss (Section 4.2) leads to low clustering errors µ ( F L ( X ) ) and µ ( F L ( X u ) ) , respectively.

4.1 Relational Knowledge Distillation over Population

Starting with minimization of the population RKD loss f ∈ F L ( X ) = argmin f ∈F R ( f ) (Equation (1)), let λ ( L ( X )) = ( λ 1 , . . . , λ |X| ) be the eigenvalues of the graph Laplacian in the ascending order, 0 = λ 1 ≤ · · · ≤ λ |X| with an arbitrary breaking of ties.

A key assumption of RKD is that the teacher model ψ (with the corresponding graph-revealing kernel k ψ ) is well aligned with the underlying ground truth partition, formalized as below:

Assumption 4.1 (Appropriate teacher models) . For G X unveiled through the teacher model k ψ ( · , · ) , we assume λ K +1 > 0 ( i.e. , G X has at most K disconnected components); while the ground truth classes {X k } k ∈ [ K ] are well-separated by G X such that α ≜ ∑ k ∈ [ K ] ∑ x ∈X k ∑ x ′ / ∈X k w xx ′ 2 ∑ x ∈X ∑ x ′ ∈X w xx ′ ≪ 1 ( i.e. , the fraction of weights of inter-class edges is sufficiently small).

In particular, α ∈ [0 , 1] reflects the extent to which the ground truth classes {X k } k ∈ [ K ] are separated from each other, as quantified by the fraction of edge weights between nodes of disparate classes. In the ideal case, α = 0 when the ground truth classes are perfectly separated such that every X k is a connected component of G X , while λ K +1 > λ K = 0 .

Meanwhile, to reduce the generic function class F down to a class of reasonable prediction functions, we introduce the following regularity conditions on the boundedness and margin:

Assumption 4.2 ( β -skeleton boundedness and γ -margin) . For any f ∈ F with P ( M ( f ) ∩ X k ) ≤ P ( X k ) / 2 for all k ∈ [ K ] , we assume there exists a skeleton subset S = [ s 1 ; . . . ; s K ] ∈ X K with 6 s k = argmax x ∈X\ M ( f ) f ( x ) k such that y f ( s k ) = k for every k ∈ [ K ] and rank ( f ( S )) = K .

(i) We say that f is β -skeleton bounded if σ 1 ( f ( S )) ≤ β for some reasonably small β .

̸

(ii) Let the k -th margin of f be γ k ≜ f ( s k ) k -max x ∈ M ( f ): y f ( x ) = k f ( x ) k . We say that f has a γ -margin if min k ∈ [ K ] γ k > γ for some sufficiently large γ > 0 .

̸

Intuitively, S can be viewed as a set of K samples where f makes the 'most confident' prediction in each class. Assumption 4.2 ensures the boundedness of predictions made on such a skeleton subset ∥ f ( S ) ∥ 2 ≤ β , as well as a margin γ by which the 'most confident' prediction of f in each class s k can be separated from the minority samples predicted to lie in other classes { x ∈ M ( f ) | y f ( x ) = k } . As a toy example, when y f : X → [ K ] is surjective, and the columns in f ( X ) ∈ R |X|× K consist of identity vectors of the predicted clusters f ( x ) k = ✶ { y f ( x ) = k } , Assumption 4.2 is satisfied with β = 1 and γ = 1 . Meanwhile, the following Remark 4.1 highlights that although Assumption 4.2 cannot be guaranteed with spectral clustering alone, it is generally satisfied in the semi-supervised learning setting with additional supervision/regularization ( cf. Example C.1).

Remark 4.1 (Limitation of spectral clustering alone) . Notice that for a generic function class F , spectral clustering alone is not sufficient to guarantee Assumption 4.2, especially the existence of a skeleton subset S or a large enough margin γ . As counter-exemplified in Example C.1, Equation (1) can suffer from large clustering error µ ( F L ( X ) ) when applied alone and failing to satisfy Assumption 4.2.

To learn predictions with accurate clustering, RKD requires either (i) additional supervision/regularization in end-to-end settings like semi-supervised learning or (ii) further fine-tuning in unsupervised representation learning settings like contrastive learning [HaoChen et al., 2021]. For end-to-end learning in practice, RKD is generally coupled with standard KD ( i.e. , feature matching) and weak supervision [Park et al., 2019, Liu et al., 2021, Ma et al., 2022], both of which help the learned function f satisfy Assumption 4.2 with a reasonable margin γ .

Throughout this work, we assume F is sufficiently regularized (with weak supervision in Section 5 and consistency regularization in Section 6) such that Assumption 4.2 always holds. Under Assumption 4.1 and Assumption 4.2, the clustering error of RKD over the population µ ( F L ( X ) ) is guaranteed to be small given a good teacher model ψ that leads to a negligible α :

Theorem 4.1 (RKD over population, proof in Appendix C.1) . Under Assumption 4.1 and Assumption 4.2 for every f ∈ F L ( X ) , the clustering error with the population (Equation (1) ) satisfies

Theorem 4.1 suggests that the clustering error over the population is negligible under mild regularity assumptions ( i.e. , Assumption 4.2) when (i) the ground truth classes are well-separated by G X revealed through the teacher model k ψ ( · , · ) ( i.e. , α ≪ 1 in Assumption 4.1) and (ii) the ( K +1) th eigenvalue λ K +1 of the graph Laplacian L ( X ) is not too small. As we review in Appendix C.3, the existing result Lemma C.4 [Louis and Makarychev, 2014] unveils the connection between λ K +1 and the sparsest K -partition (Definition C.1) of G X . Intuitively, a reasonably large λ K +1 implies that the partition of the K ground truth classes is the 'only' partition of G X into K components by removing a sparse set of edges (Corollary C.5).

4.2 Relational Knowledge Distillation on Unlabeled Samples

We now turn our attention to a more practical scenario with limited unlabeled samples and bound the clustering error µ ( F L ( X u ) ) granted by minimizing the empirical RKD loss (Equation (3)).

To cope with F ∋ f : X → R K , we recall the notion of Rademacher complexity for vector-valued functions from Maurer [2016]. Let Rad( ρ ) be the Rademacher distribution ( i.e. , with uniform probability 1 2 over {-1 , 1 } ) and ρ ∼ Rad N × K be a random matrix with i.i.d. Rademacher entries. Given any N ∈ N , with f ( x ) k denoting the k -th entry of f ( x ) ∈ R K , we define

as the Rademacher complexity of the vector-valued function class F ∋ f : X → R K .

Since the empirical RKD loss is an unbiased estimate of its population correspondence (Proposition D.1), we can leverage the standard generalization analysis in terms of Rademacher complexity to provide an unlabeled sample complexity for RKD.

Theorem 4.2 (Unlabeled sample complexity of RKD, proof in Appendix D.2) . Assume there exist B f , B k ψ > 0 such that ∥ f ( x ) ∥ 2 2 ≤ B f and k ψ ( x , x ′ ) ≤ B k ψ for all x , x ′ ∈ X , f ∈ F . Given any f | X u ∈ F L ( X u ) , f |X ∈ F L ( X ) , δ ∈ (0 , 1) , with probability at least 1 -δ/ 2 over X u ∼ P ( x ) N ,

Theorem 4.2 implies that the unlabeled sample complexity of RKD is dominated by the Rademacher complexity of F . In Appendix D.3, we further concretize Theorem 4.2 by instantiating R N/ 2 ( F ) via existing Rademacher complexity bounds for neural networks [Golowich et al., 2018].

With ∆ → 0 as N increasing (Theorem 4.2), the clustering error of minimizing the empirical RKD loss can be upper bounded as follows.

Theorem 4.3 (RKD on unlabeled samples, proof in Appendix D.4) . Under Assumption 4.1 and Assumption 4.2 for every f ∈ F L ( X u ) , given any δ ∈ (0 , 1) , if there exists ∆ < (1 -λ K ) 2 such that R ( f | X u ) ≤ R ( f |X ) +∆ for all f | X u ∈ F L ( X u ) and f |X ∈ F L ( X ) with probability at least 1 -δ/ 2 over X u ∼ P ( x ) N , then error of clustering with the empirical graph (Equation (3) ) satisfies the follows: for any K 0 ∈ [ K ] such that λ K 0 < λ K +1 and C K 0 ≜ ( 1 -λ K 0 ) 2 -(1 -λ K ) 2 ∆ = O (1) ,

Theorem 4.3 ensures that given sufficient unlabeled samples (proportional to the Rademacher complexity of F ) such that ∆ → 0 , the clustering error µ ( F L ( X u ) ) from empirical RKD is nearly as low as µ ( F L ( X ) ) from population RKD, up to an additional error term that scales linearly in ∆ .

5 Label Efficiency of Cluster-aware Semi-supervised Learning

In this section, we demonstrate the label efficiency of learning from a function class with low clustering error (Definition 3.1), like the ones endowed by RKD discussed in Section 4.

Specifically, given any cluster-aware function subclass F ′ ⊆ F with low clustering error ( e.g. , F L ( X ) and F L ( X u ) ), for a set of n i.i.d. labeled samples ( X , y ) ∼ P ( x , y ) n , we have the following generalization guarantee:

Theorem 5.1 (Label complexity, proof in Appendix B.1) . Given any cluster-aware F ′ ⊆ F with µ ( F ′ ) ≪ 1 , assuming that ( X , y ) contains at least one sample per class, for any δ ∈ (0 , 1) , with probability at least 1 -δ/ 2 over ( X , y ) ∼ P ( x , y ) n , ̂ f ∈ argmin f ∈F ′ ̂ E ( f ) satisfies

e.g. , conditioned on X u , ̂ f | X u ∈ argmin f ∈F L ( X u ) ̂ E ( f ) satisfies Equation (5) with µ ( F L ( X u ) ) .

Theorem 5.1 implies that with low clustering error ( e.g. , endowed by unsupervised methods like RKD in Theorem 4.1/Theorem 4.3, DAC in Theorem 6.2), the labeled sample complexity is as low as ˜ O ( K ) , linear in the number of clusters and asymptotically optimal 7 up to a logarithmic factor.

Remark 5.1 (Class imbalance, elaborated in Appendix B.2) . In Theorem 5.1, while the label complexity scales as n = ˜ O ( K ) with i.i.d. sampling, we meanwhile assume that ( X , y ) contains at least one labeled sample per class. Intuitively, when the K classes are balanced ( i.e. , |X k | = |X| /K for all k ∈ [ K ] ), an analogy to the coupon collector problem implies that n = O ( K log( K )) i.i.d. labeled samples are sufficient in expectation. Nevertheless, with class imbalance , the label complexity can be much worse ( e.g. , when |X K | ≪ |X k | for all k ∈ [ K -1] , collecting one label K takes 1 /P ( X K ) ≫ K labeled samples). In Appendix B.2, we show that such label inefficiency can be circumvented by leveraging a cluster-aware prediction f ∈ F ′ and drawing O (log( K )) labeled samples uniformly from each of the K predicted clusters, instead of i.i.d. from the entire population.

Remark 5.2 (Coreset selection) . While Theorem 5.1 yields an asymptotically optimal label complexity guarantee with i.i.d. sampling, it is of significant practical value to be more judicious in selecting labeled samples, especially for the low-label-rate regime ( e.g. , n = O ( K ) ). Therefore, in the experiments (Appendix A), we further investigate-alongside i.i.d. sampling 8 -a popular coreset selection method [Bilmes, 2022, Krause and Golovin, 2014], where the coreset ( X , y ) is statistically more representative of the data distribution P .

In particular, coreset selection can be recast as a facility location problem [Krause and Golovin, 2014] characterized by the teacher model: max X ⊂X ∑ x ′ ∈X max x ∈ X k ψ ( x , x ′ ) , whose optimizers can be approximated heuristically via the stochastic greedy [Mirzasoleiman et al., 2015] submodular optimization algorithm [Schreiber et al., 2020]. Intuitively, the facility location objective encourages the coreset X to be representative of the entire population X in a pairwise similarity sense, and the coreset approximation identified by the stochastic greedy method is nearly optimal (in the facility location objective) up to a multiplicative constant [Mirzasoleiman et al., 2015].

6 Data Augmentation Consistency Regularization as Clustering

In light of the broadness of the notion of clustering, investigating the discrepancy between different types of cluster-awareness is crucial for understanding the effect of spectral clustering brought by RKD beyond the low clustering error demonstrated in Section 4. In this section, we leverage the existing theoretical tools from Wei and Ma [2019], Cai et al. [2021] to unify DAC regularization into the cluster-aware semi-supervised learning framework in Theorem 5.1. Specifically, we demonstrate the effect of DAC as a 'local' expansion-based clustering [Yang et al., 2023], which works in complement with the 'global' spectral clustering facilitated by RKD.

Start by recalling the formal notion of expansion-based data augmentation (Definition 6.1) and data augmentation consistency (DAC) error (Definition 6.2) from Wei et al. [2021]:

̸

Definition 6.1 (Expansion-based data augmentation ([Wei et al., 2021] Definition 3.1)) . For any x ∈ X , we consider a set of class-invariant data augmentations A ( x ) ⊂ X such that { x } ⊊ A ( x ) ⊆ X y ∗ ( x ) . We say x , x ′ ∈ X lie in neighborhoods of each other if their augmentation sets have a non-empty intersection: NB( x ) ≜ { x ′ ∈ X | A ( x ) ∩ A ( x ′ ) = ∅} for x ∈ X ; and NB( S ) ≜ ⋃ x ∈ S NB( x ) for S ⊆ X . Then, we quantify strength of such data augmentations via the c -expansion property ( c > 1 ): for any S ⊆ X such that P ( S ∩ X k ) ≤ P ( X k ) / 2 ∀ k ∈ [ K ] ,

Definition 6.2 (DAC error [Wei et al., 2021] (3.3)) . For any f ∈ F and F ′ ⊆ F , let

̸

We adopt the theoretical framework in Wei et al. [2021] for generic neural networks with smooth activation function ϕ and consider

With such function class F of p -layer neural networks with maximum width d and weights { A ι } p ι =1 , we recall the notion of all-layer margin from Wei et al. [2021] Appendix C.2. By decomposing f = f 2 p -1 ◦ · · · ◦ f 1 such that f 2 ι -1 ( z ) = A ι z for all ι ∈ [ p ] and f 2 ι ( z ) = ϕ ( z ) for all ι ∈ [ p -1] , we consider a perturbation δ = ( δ 1 , · · · , δ 2 p -1 ) (where δ 2 ι -1 , δ 2 ι ∈ R d ι ) to each layer of f :

such that f ( x , δ ) = f 2 p -1 ( x , δ ) . Then, the all-layer margin m : F × X × [ K ] → R ≥ 0 is defined as the minimum norm of δ that is sufficient to perturb the classification of f ( x ) :

̸

Moreover, Wei et al. [2021] introduced the robust margin for the expansion-based data augmentation A (Definition 6.1), which intuitively quantifies the worst-case preservation of the label-relevant information in the augmentations of x , measured with respect to f :

We say the expansion-based data augmentation A is margin-robust with respect to the ground truth f ∗ if min x ∈X m A ( f ∗ , x ) > 0 is reasonably large.

Then, we cast DAC regularization 9 as reducing the function class F via constraints on the robust margins at the N unlabeled samples X u : for some 0 < τ < min x ∈X m A ( f ∗ , x ) ,

Leverage the existing unlabeled sample complexity bound for DAC error from Wei et al. [2021]:

Proposition 6.1 (Unlabeled sample complexity of DAC ([Wei et al., 2021] Theorem 3.7)) . Given any δ ∈ (0 , 1) , with probability at least 1 -δ/ 2 over X u ,

where ˜ O ( · ) suppresses polylogarithmic factors in N and d .

The clustering error of Equation (8) is as low as its DAC error, up to a constant factor that depends on the augmentation strength, characterized by the c -expansion (Definition 6.1):

Theorem 6.2 (Expansion-based clustering, proof in Appendix E.3) . For Equation (8) with A admitting c -expansion (Definition 6.1), the clustering error is upper bounded by the DAC error in Proposition 6.1:

In Theorem 6.2, (i) c characterizes the augmentation strength ( i.e. , the perturbation on label-irrelevant information), whereas (ii) τ quantifies the margin robustness ( i.e. , the preservation of label-relevant information) for the expansion-based data augmentation. Ideally, we would like both c and τ to be reasonably large, while there exist trade-offs between the augmentation strength and margin robustness ( e.g. , overly strong augmentations inevitably perturb label-relevant information).

Remark 6.1 (Clustering with DAC v.s. RKD) . As Figure 1 demonstrated, in contrast to RKD that unveils the spectral clustering of a population-induced graph from a 'global' perspective, DAC alternatively learns a 'local' clustering structure through the expansion of neighborhoods NB( · ) characterized by sets of data augmentations A ( · ) (Definition 6.1).

Therefore, DAC and RKD provide complementary perspectives for each other. On one hand, the 'local' clustering structure revealed by DAC effectively ensures a reasonable margin in Assumption 4.2 (Remark 4.1). On the other hand, when the augmentation strength c is insufficient ( i.e. , A ( · ) is not expansive enough) for DAC to achieve a low clustering error µ ( F τ X u ) (Theorem 6.2), the supplementary 'global' perspective of RKD connects the non-overlapping neighborhoods in the same classes and brings better classification accuracies, as we empirically verified in Appendix A.

7 Discussions, Limitations, and Future Directions

This work provides a theoretical analysis of relational knowledge distillation (RKD) in the semisupervised classification setting from a clustering perspective. Through a cluster-aware semisupervised learning (SSL) framework characterized by a notion of low clustering error, we demonstrate that RKD achieves a nearly optimal label complexity (linear in the number of clusters) by learning the underlying geometry of the population as revealed by the teacher model via spectral clustering. With a unified view of data augmentation consistency (DAC) regularization in the clusteraware SSL framework, we further illustrate the complementary 'global' and 'local' perspectives of clustering learned by RKD and DAC, respectively.

As an appealing future direction, domain adaptation in knowledge distillation is a common scenario in practice where the training data of the teacher and student models come from different distributions. Although distributional shifts can implicitly reflect the alignment between the ground truth and the teacher model (Assumption 4.1) in our analysis, an explicit and specialized study on RKD in the domain adaptation setting may lead to better theoretical guarantees and deeper insights.

Another avenue for future exploration involves RKD on different (hidden) layers of the neural network. In this work, we consider RKD on the output layer following the standard practice [Park et al., 2019], whereas RKD on additional hidden layers has been shown to further improve the performance [Liu et al., 2019]. While the analysis in Section 4 remains valid for hidden-layer features of low dimensions ( ≈ K ), Assumption 4.1 tends to fail ( i.e. , α can be large) for high-dimensional features, which may require separate and careful consideration.

References

Allen-Zhu and Y. Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816 , 2020.

Ba and R. Caruana. Do deep nets really need to be deep? Advances in neural information processing systems , 27, 2014.

Bachman, O. Alsharif, and D. Precup. Learning with pseudo-ensembles. Advances in neural information processing systems , 27, 2014.

L. Bartlett and S. Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. J. Mach. Learn. Res. , 3(null):463-482, Mar. 2003. ISSN 1532-4435.

Belkin and P. Niyogi. Semi-supervised learning on Riemannian manifolds. Machine learning , 56: 209-239, 2004.

Belkin, I. Matveeva, and P. Niyogi. Regularization and semi-supervised learning on large graphs. In Learning Theory: 17th Annual Conference on Learning Theory, COLT 2004, Banff, Canada, July 1-4, 2004. Proceedings 17 , pages 624-638. Springer, 2004.

Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural information processing systems , 32, 2019.

L. Bertozzi and E. Merkurjev. Graph-based optimization approaches for machine learning, uncertainty quantification and networks. In Handbook of Numerical Analysis , volume 20, pages 503-531. Elsevier, 2019.

Bilmes. Submodularity In Machine Learning and Artificial Intelligence. Arxiv , abs/2202.00132, Jan 2022.

Cai, R. Gao, J. Lee, and Q. Lei. A theory of label propagation for subpopulation shift. In International Conference on Machine Learning , pages 1170-1182. PMLR, 2021.

Calder, B. Cook, M. Thorpe, and D. Slepˇ cev. Poisson learning: Graph-based semi-supervised learning at very low label rates. In Proceedings of the 37th International Conference on Machine Learning , pages 1306-1316. Proceedings of Machine Learning Research, Nov. 2020.

Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. 2020.

Chen, P. Li, Z. Yan, B. Wang, and L. Zhang. Deep metric learning with graph consistency. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 982-990, 2021.

Chen, W. Choi, X. Yu, T. Han, and M. Chandraker. Learning efficient object detection models with knowledge distillation. Advances in neural information processing systems , 30, 2017.

Chen, E. Dobriban, and J. H. Lee. A group-theoretic framework for data augmentation. The Journal of Machine Learning Research , 21(1):9885-9955, 2020.

Chen and K. He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15750-15758, 2021.

D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501 , 2018.

D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pages 702-703, 2020.

Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

DeVries and G. W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 , 2017.

Eckart and G. Young. The approximation of one matrix by another of lower rank. Psychometrika , 1(3):211-218, Sep 1936. ISSN 1860-0980.

Golowich, A. Rakhlin, and O. Shamir. Size-independent sample complexity of neural networks. In Conference On Learning Theory , pages 297-299. PMLR, 2018.

H. Golub and C. F. Van Loan. Matrix computations . JHU press, Baltimore, MD, USA, 2013.

Gou, B. Yu, S. J. Maybank, and D. Tao. Knowledge distillation: A survey. International Journal of Computer Vision , 129:1789-1819, 2021.

J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems , 33:21271-21284, 2020.

Z. HaoChen, C. Wei, A. Gaidon, and T. Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems , 34:5000-5011, 2021.

Harutyunyan, A. S. Rawat, A. K. Menon, S. Kim, and S. Kumar. Supervision complexity and its role in knowledge distillation. arXiv preprint arXiv:2301.12245 , 2023.

He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015.

Hsu, Z. Ji, M. Telgarsky, and L. Wang. Generalization bounds via distillation. arXiv preprint arXiv:2104.05641 , 2021.

Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4700-4708, 2017.

Ji and Z. Zhu. Knowledge distillation in wide neural networks: Risk bound, data efficiency and imperfect teacher. Advances in Neural Information Processing Systems , 33:20823-20833, 2020.

Kim, S. Park, and N. Kwak. Paraphrasing complex network: Network compression via factor transfer. Advances in neural information processing systems , 31, 2018.

Krause and D. Golovin. Submodular function maximization. In Tractability: Practical Approaches to Hard Problems . Cambridge University Press, February 2014.

Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.

Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60(6):84-90, 2017.

Laine and T. Aila. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242 , 2016.

Ledoux and M. Talagrand. Probability in Banach Spaces: isoperimetry and processes . Springer Science & Business Media, 2013.

D. Lee, Q. Lei, N. Saunshi, and J. Zhuo. Predicting what you already know helps: Provable self-supervised learning. Advances in Neural Information Processing Systems , 34:309-323, 2021.

Liu, J. Cao, B. Li, C. Yuan, W. Hu, Y. Li, and Y. Duan. Knowledge distillation via instance relationship graph. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7096-7104, 2019.

Liu, W. Zhang, J. Wang, and J. Wang. Data-free knowledge transfer: A survey. arXiv preprint arXiv:2112.15278 , 2021.

Louis and K. Makarychev. Approximation algorithm for sparsest k-partitioning. In Proceedings of the twenty-fifth annual ACM-SIAM symposium on Discrete algorithms , pages 1244-1255. SIAM, 2014.

Ma, Y. Chen, and Z. Akata. Distilling knowledge from self-supervised teacher by embedding graph alignment. arXiv preprint arXiv:2211.13264 , 2022.

Maurer. A vector-contraction inequality for rademacher complexities. In International Conference on Algorithmic Learning Theory , pages 3-17. Springer, 2016.

Mei, T. Misiakiewicz, and A. Montanari. Learning with invariances in random features and kernel models. In Conference on Learning Theory , pages 3351-3418. PMLR, 2021.

Mirzasoleiman. Big Data Summarization Using Submodular Functions . PhD thesis, ETH Zurich, 2017.

Mirzasoleiman, A. Badanidiyuru, A. Karbasi, J. Vondrák, and A. Krause. Lazier than lazy greedy. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 29, 2015.

Park, D. Kim, Y. Lu, and M. Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3967-3976, 2019.

Parulekar, L. Collins, K. Shanmugam, A. Mokhtari, and S. Shakkottai. Infonce loss provably learns cluster-preserving representations. arXiv preprint arXiv:2302.07920 , 2023.

Phan. huyvnphan/pytorch_cifar10, jan 2021.

Phuong and C. Lampert. Towards understanding knowledge distillation. In International Conference on Machine Learning , pages 5142-5151. PMLR, 2019.

Qian, H. Li, and J. Hu. Improved knowledge distillation via full kernel matrix transfer. In Proceedings of the 2022 SIAM International Conference on Data Mining (SDM) , pages 612-620. SIAM, 2022.

Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550 , 2014.

Schreiber, J. Bilmes, and W. S. Noble. apricot: Submodular selection for data summarization in python. Journal of Machine Learning Research , 21(161):1-6, 2020.

Shen, S. Bubeck, and S. Gunasekar. Data augmentation as feature manipulation. In International Conference on Machine Learning , pages 19773-19808. PMLR, 2022.

Y. Simard, Y. A. LeCun, J. S. Denker, and B. Victorri. Transformation invariance in pattern recognition-tangent distance and tangent propagation. In Neural networks: tricks of the trade , pages 239-274. Springer, 2002.

Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems , 33:596-608, 2020.

von Luxburg. A tutorial on spectral clustering. Statistics and Computing , 17(4):395-416, Dec. 2007. ISSN 1573-1375.

J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019.

Wang, S. Lohit, M. J. Jones, and Y. Fu. What makes a" good" data augmentation in knowledge distillation-a statistical perspective. In Advances in Neural Information Processing Systems , 2022.

Wang and K.-J. Yoon. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

Wei and T. Ma. Data-dependent sample complexity of deep neural networks via lipschitz augmentation. Advances in Neural Information Processing Systems , 32, 2019.

Wei, K. Shen, Y. Chen, and T. Ma. Theoretical analysis of self-training with deep networks on unlabeled data. In International Conference on Learning Representations , 2021.

Welling and T. N. Kipf. Semi-supervised classification with graph convolutional networks. In J. International Conference on Learning Representations (ICLR 2017) , 2016.

Yang, Y. Dong, R. Ward, I. S. Dhillon, S. Sanghavi, and Q. Lei. Sample efficiency of data augmentation consistency regularization. In International Conference on Artificial Intelligence and Statistics , pages 3825-3853. PMLR, 2023.

Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6023-6032, 2019.

Zagoruyko and N. Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928 , 2016a.

Zagoruyko and N. Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146 , 2016b.

Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017.

Zhou, O. Bousquet, T. Lal, J. Weston, and B. Schölkopf. Learning with local and global consistency. In S. Thrun, L. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing Systems , volume 16. MIT Press, 2003.

Zhou, G. Cui, S. Hu, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun. Graph neural networks: A review of methods and applications. AI Open , 1:57-81, 2020. ISSN 2666-6510.

Zhu, Z. Ghahramani, and J. Lafferty. Semi-supervised learning using Gaussian fields and harmonic functions. In Proceedings of the 20th International Conference on International Conference on Machine Learning , pages 912-919, Washington, DC, USA, Aug. 2003. AAAI Press. ISBN 978-1-57735-189-4.

A Experiments

In this section, we present experimental results on CIFAR-10/100 [Krizhevsky and Hinton, 2009] to demonstrate the efficacy of combining DAC and RKD ( i.e. , the 'local' and 'global' perspectives of clustering) for semi-supervised learning in the low-label-rate regime. The experiment code can be found at https://github.com/dyjdongyijun/Semi_Supervised_Knowledge_Distillation.

We consider the (data-free) knowledge distillation setting, wherein the student model cannot access the training data of the teacher model. That is, the student models only have access to limited training samples, with very few labels, drawn from a potentially different data distribution than which the teacher model is pretrained on. More precisely, given a labeled sample set ( X , y ) of size n and an unlabeled sample set X u of size N ( n ≪ N ) 10 , we train a student model (WideResNet [Zagoruyko and Komodakis, 2016b]) in the low-label-rate regime with as few as 4 labeled samples per class ( i.e. , n = 4 K , or n = 40 for CIFAR-10 and n = 400 for CIFAR-100), along with N = 25000 , 40000 , and 50000 unlabeled samples ( i.e. , 50% , 80% , 100% of the entire CIFAR-10/100 training set).

Then, combining DAC and RKD, we solve the following objective in the experiments:

where ̂ L CE ( X , y ) ( f ) = 1 n ∑ n i =1 ℓ CE ( y i | f ( x i )) denotes the supervised cross-entropy loss with ℓ CE ( y | f ( x )) = -log ( softmax ( f ( x )) y ) . The DAC and RKD losses are as described in Equation (9) and Equation (3), respectively. For the regularization hyper-parameters, we have λ DAC = 1 and λ RKD = 0 for DAC (only), while λ RKD = 0 . 001 for DAC + RKD.

DAC via FixMatch. We take FixMatch [Sohn et al., 2020]-a state-of-the-art algorithm in the low-label-rate regime-as the semi-supervised learning baseline. By combining RKD with FixMatch, we show that RKD brings additional boosts to the classification accuracy, especially with unsuitable data augmentations and/or limited unlabeled data.

Following the formulation and implementation in Sohn et al. [2020], FixMatch involves a pair of weak and strong data augmentations ( x w , x s ) for each unlabeled sample x u . For a current model f , the DAC loss given by FixMatch can be expressed as

where T = 1 is a fixed temperature hyper-parameter. τ DAC is the threshold below which the pseudo-label gauged on x w i is considered 'with low confidence' and therefore discarded. We set τ DAC = 0 . 95 for CIFAR-10 and τ DAC = 0 . 8 for CIFAR-100.

In FixMatch, the weak augmentations include random flips and crops, whereas the strong augmentations additionally involve RandAugment [Cubuk et al., 2020]. To investigate the scenarios with unsuitable data augmentations and/or limited unlabeled data, we conduct experiments with various (i) augmentation strength ( i.e. , magnitude of transformations) m = 2 , 6 , 10 in RandAugment [Cubuk et al., 2020] and (ii) unlabeled sample size N = 25000 , 40000 , 50000 ( i.e. , 50% , 80% , 100% of the entire CIFAR-10/100 training set).

RKD. We warm up with the in-distribution case in the CIFAR-10 experiments (Table 1) where the teacher and student models are (pre)trained on the same dataset ( i.e. , CIFAR-10). Concretely, the teacher model ψ : X → W is a Densenet 161 [Huang et al., 2017, Phan, 2021] pretrained on the entirety of the CIFAR-10 training set (via supervised ERM).

Taking a step further, we consider an out-of-distribution scenario in the CIFAR-100 experiments (Table 2) where the teacher model is pretrained on ImageNet [Deng et al., 2009] via an unsupervised contrastive learning method-SwAV [Caron et al., 2020]. Intuitively, we choose teacher models

pretrained via SwAV as it encourages both data clustering and consistency of cluster assignment under data augmentations without involving supervision, which tends to provide representations that generalize better in a cluster-aware semi-supervised learning setting.

Given the teacher models, the output (response) of each sample is taken as the corresponding teacher feature such that W = R K . In light of the empirical success of RKD based on simple kernel functions [Park et al., 2019, Liu et al., 2019], we focus on the (shifted) cosine kernel, k ψ ( x , x ′ ) =

With data augmentation ( e.g. , when coupling with FixMatch), we assume that weak augmentations ( i.e. , random flips and crops) bring little perturbations on responses of the teacher model ( i.e. , the teacher features), intuitively as the predictions made by a good teacher model ought to be invariance under weak augmentations. Thereby, we can considerably reduce memory burden and accelerate the RKD process by storing the teacher features ψ ( x ) of the original data x ∈ X offline, instead of inferring every augmented sample x ′ ∈ A ( x ) with the large teacher model ψ ( x ′ ) online.

Labeled data selection. As alluded to in Remark 5.2, the success of low-label-rate (i.e., n ≪ N ) semi-supervised classification naturally depends on the quality of the (extremely) limited amount of labeled data to represent the underlying population. We perform experiments in which the selection of labeled data is non-adaptive (Uniform) or adaptive (StochasticGreedy). The non-adaptive setting, Uniform, selects labeled data uniformly in each ground truth class 11 . This contrasts with the strictly i.i.d. sampling assumption used in our theoretical results but constitutes a reasonable alternative that has been used in other low-label rate semi-supervised classification works ( e.g. , [Calder et al., 2020]).

We also utilize a coreset selection strategy (StochasticGreedy) for identifying 'representative' inputs to use as labeled data in an adaptive manner. This labeled set is selected via a stochastic greedy [Mirzasoleiman et al., 2015] optimization problem to (approximately) minimize a facility location objective 12 . An in-depth discussion of coreset selection methods is outside the scope of the current work, but we refer the interested reader to the enlightening references [Bilmes, 2022, Krause and Golovin, 2014, Mirzasoleiman, 2017] regarding submodular optimization methods in machine learning.

Results. From Table 1 and Table 2, we observe that the incorporation of RKD with FixMatch ( i.e. , adding the 'global' perspective of RKD to the 'local' perspective of DAC) generally brings additional improvements to the classification accuracy, especially when the data augmentation strength is inappropriate and/or the number of unlabeled data is limited.

Training details. Throughout the experiments, we used weight decay 0 . 0005 . We train the student model via stochastic gradient descent (SGD) with Nesterov momentum 0 . 9 for 2 17 iterations (batches) with a batch size 64 ∗ 8 = 2 9 (consisting of 64 labeled samples and 64 ∗ 7 unlabeled samples). The initial learning rate is 0 . 03 , decaying with a cosine scheduler. The test accuracies are evaluated 2 7 times in total evenly throughout the 2 17 iterations ( i.e. , one test evaluation after every 2 10 iterations) on an EMA model with a decay rate 0 . 999 . The average and standard deviation of the best test accuracy ( i.e. , early stopping with the maximum patience 128 ) are reported for each experiment over 3 arbitrary random seeds. Both CIFAR-10/100 experiments are conducted on one NVIDIA A40 GPU.

B Proofs and Discussions for Section 5

B.1 Proof of Theorem 5.1

Lemma B.1 (Rademacher complexity with cluster-aware functions (generalization of Yang et al. [2023] Theorem 8)) . For a fixed F L ( X u ) , let

be the loss function class. Then, its Rademacher complexity can be upper-bounded by

Proof of Lemma B.1. Given a set of i.i.d. samples ( X , y ) ∼ P ( x , y ) n , let

denote the number of distinct patterns over ( X , y ) in ℓ ◦ F L ( X u ) . Then, by Massart's finite lemma, the empirical Rademacher complexity with respect to ( X , y ) is upper bounded by

By the concavity of √ log ( · ) , we know that,

Since P ( M ( f )) ≤ µ ( F L ( X u ) ) ≤ 1 2 for all f ∈ F L ( X u ) , by union bound, we have

where ι counts the number of minority samples in the n samples. Plugging in this last line into Equation (11) yields Equation (10).

Proof of Theorem 5.1. Since ℓ ◦ F L ( X u ) is 1 -bounded with the zero-one loss ℓ , Lemma F.1 implies that with probability at least 1 -δ/ 2 over ( X , y ) ,

Then, incorporating the upper bound of R n ( ℓ ◦ F L ( X u ) ) from Lemma B.1 completes the proof.

B.2 Cluster-wise Labeling with Cluster-aware Predictions

Recall the notions of majority labeling ˜ y f ( · ) (Equation (2)) and minority subset M ( f ) (Definition 3.1). We first specify the cluster-aware predictions that are useful for labeling with the following nondegeneracy assumption.

Assumption B.1 ( ( m 0 , c 0 ) -non-degenerate predictions) . For any prediction function f ∈ F , we say f is ( m 0 , c 0 ) -non-degenerate if the following properties are satisfied:

It is worth pointing out that although the majority labeling ˜ y f ( · ) depends on the ground truth and therefore remains unknown without label acquisition, under Assumption B.1, the associated partition { X f k } k ∈ [ K ] over X depends only on f and is known without labeling. This is because y f ( · ) and ˜ y f ( · ) induce the same partition over X when both y f ( · ) and ˜ y f ( · ) are non-degenerate ( i.e. , being surjective functions onto [ K ] , a necessary condition of Assumption B.1 (i) and (ii)). We also notice that Assumption B.1 (iii) can be generally satisfied with a reasonably large c when learning via Equation (3) with a sufficiently small µ ( F L ( X u ) ) ( cf. Theorem 4.1 or Theorem 4.3). For example, when c 0 · µ ( F L ( X u ) ) ≤ min k ∈ K P ( X f k ) , Assumption B.1 (iii) is automatically satisfied.

With the cluster-aware predictions in Assumption B.1, we show that O (log( K )) i.i.d. labeled samples per predicted cluster are sufficient to ensure at least one labeled sample per class. That is, the cluster-wise labeling with ( m 0 , c 0 ) -non-degenerate predictions guarantees a label complexity of O ( K log( K )) even with class imbalance.

Proposition B.2 (Cluster-wise labeling) . Given any ( m 0 , c 0 ) -non-degenerate prediction function f ∈ F (Assumption B.1) with sufficiently large m 0 ≫ log c 0 (2 K ) and c 0 ≥ 2 , for each k ∈ [ K ] , acquire labels of m ( m ≤ m 0 ) unlabeled samples { x u ki } i ∈ [ m ] ∼ Unif ( X f k ) m drawn i.i.d. from X f k . Then, for any δ ∈ ( 2 K c m 0 0 , 1 ) , when m ≥ log c 0 ( 2 K δ ) , { y ∗ ( x u ki ) } k ∈ [ K ] ,i ∈ [ m ] = [ K ] have at least one ground truth label per class with probability at least 1 -δ .

Proof of Proposition B.2. We first observe that

where

̸

Since Assumption B.1 implies P ( M ( f ) ∩ X f k )/ P ( X f k ) ≤ 1 c 0 for all k ∈ [ K ] , we have

where the second and the third inequalities follow from the facts that 1 -q ≥ exp( -2 q ) for all 0 ≤ q ≤ 1 2 and exp( -q ) ≥ 1 -q for all 0 ≤ q ≤ 1 , respectively. It is straightforward to observe that when m ≥ log c 0 ( 2 K δ ) , we have 2 K c m 0 ≤ δ . With min k ∈ [ K ] ∣ ∣ ∣ X f k ∣ ∣ ∣ = m 0 , taking up to m ≤ m 0 unlabeled samples for each k ∈ [ K ] , we can achieve δ as small as δ min = 2 K c m 0 0 .

C Proofs and Discussions for Section 4.1

In Appendix C.1, we proceed with the proof of low clustering error guarantee (Theorem 4.1) for minimizing the population RKD loss (Equation (1)). In Appendix C.2, we extend the discussion in Remark 4.1 on the limitation of spectral clustering alone in the end-to-end setting via an illustrative toy counter-example. In Appendix C.3, we clarify the connection between our results and corresponding spectral graph theoretic notions of sparsest k -partitions.

C.1 Proof of Theorem 4.1

Given any labeling function y : X → [ K ] , let - → y : X → { 0 , 1 } K be the one-hot encoded labeling function such that for y ( X ) ∈ [ K ] |X| , - → y ( X ) ∈ { 0 , 1 } |X|× K . Further, given f ∈ F , we denote U X ≜ D ( X ) 1 / 2 - → ˜ y f ( X ) , F X ≜ D ( X ) 1 / 2 f ( X ) , and Y X ≜ D ( X ) 1 / 2 - → y ∗ ( X ) such that U X , F X , Y X ∈ R |X|× K .

̸

Since L ( X ) = I -W ( X ) , the classical argument with Gershgorin circle theorem [Golub and Van Loan, 2013, Theorem 7.2.1] implies that 0 ≼ L ( X ) ≼ 2 I , whereas k ψ being positive semidefinite further implies that W ( X ) ≽ 0 and therefore 0 ≼ L ( X ) ≼ I (see the proof of Claim C.3). Consider the spectral decomposition of L ( X ) = I -W ( X ) ,

where { v i ∈ R |X| } i ∈ [ |X| ] are the orthonormal eigenvectors associated with 0 = λ 1 ≤ · · · ≤ λ |X| ≤ 1 , respectively, that form a basis of R |X| .

Lemma C.1. Given any f ∈ F that satisfies Assumption 4.2, we have

̸

Proof of Lemma C.1. Without ambiguity, we overload the notation of each sample x ∈ X as the corresponding index in [ |X| ] such that, given any sample subset S = { s i } i ∈ [ | S | ] ⊂ X , following the MATLAB notation for matrix indexing, Π S ≜ I |X| (: , S ) ∈ R |X|×| S | where I |X| denotes the |X| × |X| identity matrix. With respect to S , we also denote F S ≜ F X ( S, :) = Π ⊤ S F X and Y S ≜ Y X ( S, :) = Π ⊤ S Y X .

We first show that for any S = [ s 1 ; . . . ; s K ] ∈ X K that satisfies rank ( F S ) = K ,

To see this, we start by replacing Z on the right-hand side of Equation (14) with the corresponding least square solution such that

where F † X = ( F ⊤ X F X ) -1 F ⊤ X . Then, we can decompose the left-hand side of Equation (14) via the orthogonal projection F X F † X :

where P F ≜ F X F † X = F X ( F ⊤ X F X ) -1 F ⊤ X and P S ≜ F X ( Π ⊤ S F X ) -1 Π ⊤ S . By observing that

P S P F = F X ( Π ⊤ S F X ) -1 Π ⊤ S F X ( F ⊤ X F X ) -1 F ⊤ X = P F = P F P F , we can upper bound ∥ ( P F -P S ) Y X ∥ 2 F such that

which leads to Equation (14).

Now, we consider the sample subset S ∈ X K described in Assumption 4.2. For given f ∈ F , partitioning X into the majority and minority subsets, X\ M ( f ) and M ( f ) , respectively yields

we have

Meanwhile, in the case when y ∗ ( x ) = y ∗ ( s k ) for all k ∈ [ K ] , we have - → y ∗ ( S ) - → y ∗ ( x ) = 0 , and therefore

Since s k / ∈ M ( f ) , we have - → y ∗ ( S ) = - → ˜ y f ( S ) whose rows lie in the canonical bases of { e k ∈ R K } k ∈ [ K ] ; whereas for x ∈ M ( f ) , we have y ∗ ( x ) = ˜ y f ( x ) . Therefore, in the case when y ∗ ( x ) = y ∗ ( s k ) for some k ∈ [ K ] ,

̸

̸

where σ K ( f ( S ) -⊤ ) = 1 /β , and with ˜ y f ( s k ) = y ∗ ( s k ) = y ∗ ( x ) = ˜ y f ( x ) implying y f ( s k ) = y f ( x ) ,

̸

∥ ∥ - → y ∗ ( x ) - - → y ∗ ( S ) ⊤ f ( S ) -⊤ f ( x ) ∥ ∥ 2 2 ≥ ∥ ∥ - → y ∗ ( x ) ⊤ ( - → y ∗ ( x ) - - → y ∗ ( S ) ⊤ f ( S ) -⊤ f ( x ) )∥ ∥ 2 2 = 1 . Overall, we've shown that

which leads to

̸

Finally, combining Equation (14) and Equation (15) completes the proof.

LemmaC.2. With y k ∈ [0 , 1] |X| denoting the k -th column of Y X ≜ D ( X ) 1 / 2 - → y ∗ ( X ) ∈ [0 , 1] |X|× K ,

Proof of Lemma C.2. It is sufficient to observe that, by construction, for every k ∈ [ K ] ,

̸

̸

̸

Claim C.3. For all f ∈ F L ( X ) minimizing Equation (1) , Range ( D ( X ) 1 2 f ( X ) ) = span { v 1 , . . . , v K } .

Proof of Claim C.3. Since 0 ≼ L ( X ) ≼ 2 I , we have -I ≼ W ( X ) ≼ I . Meanwhile, let k ψ ( X , X ) ∈ R |X|×|X| be the population kernel matrix such that k ψ ( X , X ) xx ′ = k ψ ( x , x ′ ) . Then by defintion, k ψ ( X , X ) = D ( X ) -1 W ( X ) D ( X ) -1 , and therefore

That is, the eigenvalues of L ( X ) and those of W ( X ) satisfies

while the eigenvectors are shared such that, with the spectral decomposition of L ( X ) in Equation (13),

Therefore, the K eigenvalues of W ( X ) of maximal modulus correspond exactly to the K eigenvalues of L ( X ) that are closest to 0 , the eigenvalues whose eigenspaces are relevant for spectral clustering.

Recalling F X ≜ D ( X ) 1 2 f ( X ) , we can rewrite Equation (1) as

It follows directly from the Eckart-Young-Mirsky theorem [Eckart and Young, 1936] that Range( F X ) is the subspace spanned by the eigenvectors V K = [ v 1 , . . . , v K ] ∈ R |X|× K associated with the maximum K eigenvalues of W ( X ) :

Proof of Theorem 4.1. Recall F X ≜ D ( X ) 1 / 2 f ( X ) , Y X ≜ D ( X ) 1 / 2 - → y ∗ ( X ) , and let y k ∈ [0 , 1] |X| be the k -th column of Y X . Leveraging Lemma C.1, we have for any f ∈ F ,

Recall that the orthonormal eigenvectors { v i ∈ R |X| } i ∈ [ |X| ] associated with eigenvalues 0 = λ 1 ≤ · · · ≤ λ |X| of L ( X ) form a basis of R |X| . For every k ∈ [ K ] , ( y k -F X z k ) ∈ R |X| can be decomposed as the linear combination of { v i } i ∈ [ |X| ] such that

Meanwhile, Claim C.3 implies that, for f ∈ F L ( X ) , with V K = [ v 1 , . . . , v K ] ∈ R |X|× K , we have Range( F X ) = Range ( V K ) . That is, v ⊤ i F X z k = 0 for all i > K , and by taking

Overall, for every k ∈ [ K ] , there exists z k ∈ R K such that

and Lemma C.2 implies that

which completes the proof.

C.2 Limitation of Spectral Clustering Alone

To ground the discussion in Remark 4.1, here, we raise a toy counter-example where spectral clustering alone fails to satisfy Assumption 4.2 and suffers from a large clustering error. Intuitively, such a pitfall of spectral clustering is caused by a suboptimal choice of basis for the predictions f ( X ) in Equation (1) due to the inherited rotation invariance of the minimizers of R ( f ) .

Example C.1 (Pitfall of spectral clustering alone) . We consider a binary classification problem with two balanced ground truth classes {X 1 , X 2 } perfectly partitioned in G X as disconnected components. Then, when the prediction y f ( x ) ≜ argmax k ∈ [ K ] f ( x ) k is made with a uniformly random break of ties, Equation (1) alone suffers from µ ( F L ( X ) ) ≥ 1 4 in expectation, due to the violation of Assumption 4.2.

Nevertheless, with as few as one labeled sample per class { ( x i , i ) | i = 1 , 2 } , weak supervision via the cross-entropy loss

facilitates satisfaction of Assumption 4.2 with a provably large margin γ . For instance, assuming ∥ f ( x ) ∥ 2 ≤ ∥ f ( S ) ∥ 2 ≤ β and w x = 1 / |X| for all x ∈ X (simplified for illustration purposes), for any f ∈ F L ( X ) under weak supervision such that

Assumption 4.2 is satisfied with

In particular, with the minimum feasible cross-entropy loss ̂ L CE ( f ) = log ( 1 + e - √ 2 β ) , we have γ ≥ √ 2 β .

We remark that with DAC regularizations like FixMatch [Sohn et al., 2020] and MixMatch [Berthelot et al., 2019], model predictions are matched against pseudo-labels that are gauged based on data augmentations, usually via the cross-entropy loss. Therefore, DAC has analogous effects on the margins of the learned prediction functions f as the additional supervision described above.

Moreover, when coupling Equation (1) with the conventional KD ( i.e. , feature matching), the learned prediction functions f inherit the boundedness and margin of the teacher model, which are presumably satisfactory.

Rationale for Example C.1. Recall that λ 1 ≤ · · · ≤ λ |X| are the eigenvalues of the graph Laplacian L ( X ) = I -W ( X ) , and let { v 1 , · · · , v |X| } be the associated normalized eigenvectors. For a binary classification with {X 1 , X 2 } perfectly separated in G X , we can write λ 1 = λ 2 = 0 , while v 1 , v 2 are the scaled identity vectors of X 1 , X 2 , respectively, such that v i ( x ) = 1 √ |X i | ✶ { x ∈ X i } for i = 1 , 2 .

By denoting V 2 = [ v 1 , v 2 ] ∈ R |X|× 2 , the Eckart-Young-Mirsky theorem [Eckart and Young, 1936] implies that, for any orthogonal matrix Q = [ q 1 ; q 2 ] ∈ R 2 × 2 ,

is a minimizer of R ( f ) in Equation (1). That is, f ( x ) = 1 √ w x |X i | q i for x ∈ X i , i = 1 , 2 .

Spectral clustering alone. However, by taking q 1 = 1 √ 2 [1; 1] and q 2 = 1 √ 2 [ -1; 1] , with a uniformly random break of ties, the corresponding f ∈ F L ( X ) will misclassify half of x ∈ X 1 as X 2 in expectation. Then, since P ( X 1 ) = P ( X 2 ) = 1 2 , by Definition 3.1, we have M ( f ) = { x ∈ X 1 | y f ( x ) = 2 } such that µ ( F L ( X ) ) ≥ P ( M ( f )) = 1 4 in expectation.

Spectral clustering in Example C.1 fails to yield low clustering error due to the potential violation of Assumption 4.2. For example, consider any prediction function f ∈ F L ( X ) described in Example C.1.

(i) When argmin x ∈X\ M ( f ) w x ⊆ X 1 , recalling s k = argmax x ∈X\ M ( f ) f ( x ) k and f ( x ) = 1 √ w x |X i | q i for x ∈ X i such that, since |X 1 | = |X 2 | ,

there does not exist a skeleton subset S = [ s 1 ; s 2 ] as described in Assumption 4.2.

̸

(ii) Otherwise, suppose one can find a skeleton subset S = [ s 1 ; s 2 ] as described in Assumption 4.2. Without loss of generality, by taking (argmin x ∈X w x ) ∩ M ( f ) = ∅ in the problem setup, there exists x ∈ M ( f ) ⊂ X 1 (with y f ( x ) = 2 ) such that w x ≤ w s 1 . Therefore,

̸

implies a trivial margin γ ≤ 0 , which violates γ > 0 in Assumption 4.2.

With additional weak supervision. We first observe that for both i ∈ [2] ,

which implies that

where -log ( e ̂ L CE ( f ) -1 ) > 0 since ̂ L CE ( f ) < log ( 1 + e -β ) < log(2) .

Since we assume ∥ f ( x ) ∥ 2 ≤ β for all x ∈ X , f ( x i ) 2 1 + f ( x i ) 2 2 ≤ β 2 for both i = 1 , 2 , we therefore have

This similarly holds for f ( x 2 ) 2 -f ( x 1 ) 2 .

Recall now that f ( x ) = 1 √ w x |X i | q i for x ∈ X i , i = 1 , 2 and x i ∈ X i for i = 1 , 2 . Assuming w x = 1 / |X| for all x ∈ X , we have

Therefore, γ 1 ≥ f ( x 1 ) 1 -f ( x 2 ) 1 and γ 2 ≥ f ( x 2 ) 2 -f ( x 1 ) 2 , which together imply that

Now examine the two extremes of log ( 1 + e - √ 2 β ) ≤ ̂ L CE ( f ) < log ( 1 + e -β ) . When the cross-entropy loss achieves its minimum feasible value ̂ L CE ( f ) = log ( 1 + e - √ 2 β ) , we have that γ ≥ √ 2 β , whereas the margin can be nearly trivial γ ≥ ϵ CE with ϵ CE → 0 when the cross-entropy loss is on the larger side ̂ L CE ( f ) → log ( 1 + e -β ) .

C.3 Connection with Sparsest k -partition

We now identify the connection between the eigenvalue λ K +1 and spectral graph theoretic notions of clusteredness of the underlying graph. We recall the notion of sparsest k -partitions [Louis and Makarychev, 2014] based on the Dirichlet conductance of subgraphs.

̸

Definition C.1 (Sparsest k -partition, HaoChen et al. [2021] Definition 3.4, Louis and Makarychev [2014] Problem 1.1) . Given a weighted undirected graph G = ( X , w ) , for any k ∈ [ |X| ] , let S = { { S i } i ∈ [ k ] ∣ ∣ ∪ i ∈ [ k ] S i = X , S i ∩ S j = ∅ , S i = ∅ ∀ i = j, i, j ∈ [ k ] } be the set of all k -partitions of G . The sparsest k -partition of G is defined as

̸

where ϕ G ( S ) ≜ (∑ x ∈ S ∑ x ′ / ∈ S w xx ′ ) / (∑ x ∈ S w x ) is the Dirichlet conductance of S ⊆ X .

It is worth mentioning that ϕ k is a non-decreasing function in k [HaoChen et al., 2021]. In particular, ϕ k = 0 when the graph has at least k disconnected components; whereas λ K +1 > 0 (Assumption 4.1) implies that ϕ K +1 > 0 .

Meanwhile, Louis and Makarychev [2014] unveils the following connection between the sparsest k -partition of the graph and the k ′ th smallest eigenvalue of its graph Laplacian:

Lemma C.4 (Louis and Makarychev [2014] Proposition 1.2) . Given any weighted undirected graph G = ( X , w ) , let 0 = λ 1 ≤ · · · ≤ λ |X| be the eigenvalues of the normalized graph Laplacian in the ascending order. For any k, k ′ ∈ [ |X| ] , k < k ′ , there exists a partition { S i } i ∈ [ k ] ∈ S of X such that, for all i ∈ [ k ] ,

Leveraging the existing result Lemma C.4 [Louis and Makarychev, 2014] from spectral graph theory, we have the following corollary.

Corollary C.5 (Clustering in terms of sparsest k -partitions) . Under Assumption 4.1 and Assumption 4.2 for every f ∈ F L ( X ) , error of clustering with the population (Equation (1) ) satisfies that, if α > 0 and ϕ k > 0 for all k ∈ [ K ] , then

We also notice that when α = 0 , Theorem 4.1 automatically implies that µ ( F L ( X ) ) = 0 . Intuitively, Corollary C.5 implies that except for the partition of the K ground truth classes {X} k ∈ [ K ] ( i.e. , when α > 0 ), G X cannot be partitioned into other K components by removing a sparse set of edges from G X .

Proof of Corollary C.5. Lemma C.4 implies that for any k ∈ [ K ] , there exists a partition { S i } i ∈ [ k ] ∈ S of X such that

and therefore

which completes the proof when recalling Theorem 4.1.

D Proofs and Discussions for Section 4.2

Here, we show that ̂ R X u ( f ) shares the same minimizer as the population Laplacian regularizer R ( f ) in expectation (Proposition D.1) and provide generalization bounds for ̂ R X u ( f ) (Theorem 4.2) accordingly.

D.1 Unbiased Estimations of Population Laplacian

Proposition D.1. ̂ R X u ( f ) (Equation (3) ) serves as an unbiased estimate for R ( f ) (Equation (1) ):

Proof of Proposition D.1. We first observe that R ( f ) can be expanded as following:

Then, with k ψ ( x , x ′ ) = w xx ′ w x w x ′ , ̂ R X u ( f ) in Equation (3), w x = P ( x ) , and w x ′ = P ( x ′ ) , we have

D.2 Proof of Theorem 4.2

Proof of Theorem 4.2. By defining

we have R ( f ) = E x , x ′ ∼ P ( x ) 2 [ ̂ r x , x ′ ( f )] and ̂ R X u ( f ) = 2 N ∑ N/ 2 i =1 ̂ r x u 2 i -1 , x u 2 i ( f ) as

where {( x u 2 i -1 , x u 2 i )} N/ 2 i =1 are i.i.d. given X u ∼ P ( x ) N . Consider the function class

Since 0 ≤ k ψ ( x , x ′ ) ≤ B k ψ , and by Cauchy-Schwarz inequality, f ( x ) ⊤ f ( x ′ ) ≤ ∥ f ( x ) ∥ 2 ∥ f ( x ′ ) ∥ 2 ≤ B f , we have 0 ≤ ̂ r x , x ′ ( f ) ≤ ( B k ψ + B f ) 2 for all x , x ′ ∈ X , which means that ̂ r · , · ◦ F is ( B k ψ + B f ) 2 -bounded. Leveraging Lemma F.1 gives that, with probability at least 1 -δ/ 2 over X u ,

Now, it remains to show that R N/ 2 ( ̂ r · , · ◦ F ) ≤ 4 √ 2 B f ( B f + B k ψ ) R N/ 2 ( F ) . For this, we recall Equation (4) and observe that

where with Talagrand's lemma [Ledoux and Talagrand, 2013, Theorem 4.12] for compositions with scalar Lipschitz functions,

Since

By the vector-contraction inequality for Rademacher complexities [Maurer, 2016, Corollary 4], since ∥ f ( x ) ∥ 2 2 is ( 2 √ B f ) -Lipschitz in f ( x ) , with f ( x ) k denoting the k -th entry of f ( x ) ∈ R K ,

Overall, we have R N/ 2 ( ̂ r · , · ◦ F ) ≤ 4 √ 2 B f ( B f + B k ψ ) R N/ 2 ( F ) which completes the proof.

D.3 Unlabeled Sample Complexity with Deep Neural Networks

Here, we ground Theorem 4.2 by exemplifying R N/ 2 ( F ) leveraging the existing generalization bound [Golowich et al., 2018] for deep neural networks.

Claim D.2 (Rademacher complexity of deep neural networks [Golowich et al., 2018]) . Recall the notion of Rademacher complexity for a vector-valued function class F ∋ f : X → R K from Equation (4) . Adopting the existing results in Golowich et al. [2018] and following the notations in Section 6, we consider F as a class of deep neural networks:

with weight matrices { A ι | ι ∈ [ p ] } and a 1 -Lipschitz, positive-homogeneous activation function ϕ ( · ) ( e.g. , ReLU) applied entry-wisely. Let B F = ∏ p ι =1 B F ,ι and ∥ x ∥ 2 ≤ B X for all x ∈ X . Then, by Golowich et al. [2018] Theorem 1, we have

D.4 Proof of Theorem 4.3

Proof of Theorem 4.3. Let U X ≜ D ( X ) 1 / 2 f | X u ( X ) , F X ≜ D ( X ) 1 / 2 f |X ( X ) , and y k ∈ [0 , 1] |X| be the k -th column of Y X ≜ D ( X ) 1 / 2 - → y ∗ ( X ) . We denote P U ≜ U X U † X as the orthogonal projection onto Range( U X ) and P ⊥ U ≜ I -P U as the orthogonal complement of P U .

Recalling from the proof of Theorem 4.1, Lemma C.1 implies that for all f | X u ∈ F L ( X u ) ,

where the second equality comes from taking z k = U † X y k for every k ∈ [ K ] .

Consider the spectral decomposition of L ( X ) in Equation (13). Since the orthonormal eigenvectors { v i ∈ R |X| } i ∈ [ |X| ] associated with eigenvalues 0 = λ 1 ≤ · · · ≤ λ |X| ≤ 1 form a basis of R |X| , by denoting V K ≜ [ v 1 , . . . , v K ] ∈ R |X|× K and V ⊥ K ≜ [ v K +1 , . . . , v |X| ] ∈ R |X|× K , for every k ∈ [ K ] ,

It is shown in the proof of Theorem 4.1 that

, whereas by observing that ∥ y k ∥ 2 2 = ∑ x ∈X w x · ✶ { y ∗ ( x ) = k } = P ( X k ) , ∑ k ∈ [ K ] ∥ ∥ V ⊤ K P ⊥ U y k ∥ ∥ 2 2 ≤ ∑ k ∈ [ K ] ∥ y k ∥ 2 2 ∥ ∥ P ⊥ U V K ∥ ∥ 2 F = ∥ ∥ P ⊥ U V K ∥ ∥ 2 F ∑ k ∈ [ K ] P ( X k ) = ∥ ∥ P ⊥ U V K ∥ ∥ 2 F . Overall, we have

Upper bounding ∥ ∥ P ⊥ U V K ∥ ∥ 2 F . We first recall from Equation (13) that W ( X ) = ∑ |X| i =1 (1 -λ i ) v i v ⊤ i . For f |X ∈ F L ( X ) , Eckart-Young-Mirsky theorem [Eckart and Young, 1936] implies that F X F ⊤ X = ∑ K i =1 (1 -λ i ) v i v ⊤ i , and

Meanwhile, the assumption ∆ < (1 -λ K ) 2 implies that rank ( P ⊥ U ) = |X| K , and thus

Otherwise by contradiction, suppose rank ( P U ) = rank ( U X ) < K , then Eckart-Young-Mirsky theorem [Eckart and Young, 1936] implies that

which contradicts R ( f | X u ) ≤ R ( f |X ) +∆ .

Furthermore, since ∥ v i ∥ 2 = 1 and P ⊥ U is an orthogonal projection, for all i ∈ [ |X| ] ,

For every i ∈ [ |X| ] , we denote ξ i ≜ ∥ ∥ P ⊥ U v i ∥ ∥ 2 2 . Then, upper bounding ∥ ∥ P ⊥ U V K ∥ ∥ 2 F = ∑ K i =1 ξ i can be recast as a linear programming problem:

whose dual can be expressed as

By taking

the dual optimum can be upper bounded by

With both the primal Equation (20) and the dual Equation (21) being feasible and bounded, the strong duality then implies that

which, when plugging back into Equation (16), completes the proof.

E Proofs and Discussions for Section 6

E.1 Expansion-based Data Augmentation

For the detailed analysis, we further recall the following notion of constant expansion and its relation with the concept of c -expansion in Definition 6.1 from Wei et al. [2021].

Definition E.1 ( ( q, ξ ) -constant expansion [Wei et al., 2021]) . We say the expansion-based data augmentation (Definition 6.1) satisfies ( q, ξ ) -constant expansion if for any S ⊆ X such that P ( S ) ≥ q and P ( S ∩ X k ) ≤ P ( X k ) / 2 for all k ∈ [ K ] ,

Lemma E.1 ( c -expansion implies constant expansion [Wei et al., 2021]) . The c -expansion property (Definition 6.1) implies ( ξ c -1 , ξ ) -constant expansion (Definition E.1) for any ξ > 0 .

Proof of Lemma E.1. Consider a subset S ⊆ X such that P ( S ) ≥ q and P ( S ∩ X k ) ≤ P ( X k ) / 2 . Let S k ≜ S ∩ X k for every k ∈ [ K ] .

(ii) If c ≥ 2 , since c -1 ≥ 1 and P ( X k ) -P ( S k ) ≥ P ( S k ) , we have

Overall, we have

E.2 Data Augmentation Consistency Regularization

To bridge Equation (8) with common DAC regularization algorithms in practice, in Example E.1, we instantiate FixMatch [Sohn et al., 2020] - a state-of-the-art semi-supervised learning algorithm that leverages DAC regularization by encouraging similar predictions of proper weak and strong data augmentations x w , x s ∈ A ( x ) of the same sample x .

Example E.1 (FixMatch [Sohn et al., 2020]) . FixMatch minimizes the loss between the outputs of the strong augmentation f ( x s ) and the pseudo-label of the weak augmentation y f ( x w ) when the confidence of such pseudo-labeling is high ( i.e. , max k ∈ [ K ] f ( x w ) k is larger than a threshold).

To connect FixMatch with Equation (8) , we consider a fixed (unknown) margin-robust neighborhood F ′ ⊇ F τ X u ∋ f ∗ where the strong augmentation characterizes the worst-case (minimum) robust margin x s ∈ ⋂ f ∈F ′ argmin x ′ ∈A ( x ) m ( f, x ′ , y f ( x )) ; while the weak augmentation preserves the classification y f ( x w ) = y f ( x ) .

Then, for any f ∈ F ′ during learning, minimizing the cross-entropy loss between softmax ( f ( x s )) and the one-hot label - → y f ( x w ) is equivalent to enforcing a large enough margin m ( f, x s , y f ( x )) ≥ τ , whereas m ( f, x ′ , y f ( x )) ≥ m ( f, x s , y f ( x )) ≥ τ for all x ′ ∈ A ( x ) by construction implies m A ( f, x ) ≥ τ as required by the constraints in Equation (8) .

E.3 Proof of Theorem 6.2

Proof of Theorem 6.2. We first recall from Lemma E.1 that c -expansion implies ( 2 ν ( f ) c -1 , 2 ν ( f ) ) -constant expansion (Definition E.1) for any f ∈ F . Therefore, it is sufficient to show that with ( q, 2 ν ( f )) -constant expansion ( i.e. , q = 2 ν ( f ) c -1 ),

For any f ∈ F τ X u , we notice that Equation (22) is automatically satisfied when P ( M ( f )) < q . Otherwise, when P ( M ( f )) ≥ q , then along with P ( S ∩ X k ) ≤ P ( X k ) / 2 for all k ∈ [ K ] , ( q, 2 ν ( f )) -constant expansion implies

Meanwhile, we observe that by union bound,

̸

because for any x ∈ NB( M ( f )) \ M ( f ) , there exists x ′ ∈ M ( f ) such that one can find some x ′′ ∈ A ( x ) ∩ A ( x ′ ) ; and therefore exactly one of the follows must hold

̸

̸

̸

Leveraging Equation (23) and Equation (24), we have

F Technical Lemmas

Here, we briefly review the standard Rademacher-complexity-based generalization analysis based on McDiarmid's inequality and a classical symmetrization argument Wainwright [2019], Bartlett and Mendelson [2003].

Lemma F.1 (Generalization guarantee with Rademacher complexity) . Given a distribution over a set Z 13 , we consider a set of i.i.d. samples { z i } i ∈ [ n ] ∼ Z n with Z = [ z 1 , . . . , z n ] and a B -bounded function class H = { h : Z → R | | h ( z ) -h ( z ′ ) | ≤ B ∀ z , z ′ ∈ Z} . By denoting

̸

then for h ∗ ∈ argmin h ∈H H ( h ) and ̂ h ∈ argmin h ∈H ̂ H Z ( h ) , we have that with probability at least 1 -δ/ 2 over Z ,

where R n ( H ) is the Rademacher complexity of H ,

̸

Proof of Lemma F.1. Let g ( z 1 , . . . , z n ) ≜ g ( Z ) ≜ sup h ∈H H ( h ) -̂ H Z ( h ) . Then h ∈ H being B -bounded implies that for any z ∼ Z and i ∈ [ n ] ,

Therefore, by McDiarmid's inequality [Bartlett and Mendelson, 2003],

where exp ( -2 nt 2 B 2 ) ≤ δ 4 when t ≥ √ B 2 log(4 /δ ) 2 n .

Meanwhile, by a classical symmetrization argument (e.g., proof of Wainwright [2019] Theorem 4.10), we can bound the expectation E [ g ( Z )] : for an independent sample set Z ′ ∼ Z n that is also independent of Z ,

where in the last line we have used the law of iterated conditional expectation. Then, with Rademacher random variables ρ = ( ρ 1 , . . . , ρ n ) ∼ Rad n , we have

Therefore, with probability at least 1 -δ/ 4 over Z , we have

while the same tail bound holds for sup h ∈H -H ( h ) + ̂ H Z ( h ) via an analogous argument.

By recalling the definition of ̂ h and h ∗ , we can decompose the excess risk H ( ̂ h ) -H ( h ∗ ) as

where ̂ H Z ( ̂ h ) -̂ H Z ( h ∗ ) ≤ 0 by the basic inequality. With both ̂ h, h ∗ ∈ H , we have

and therefore

Overall, with probability at least 1 -δ/ 2 over Z ,

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies relational knowledge distillation (RKD) on semi-supervised learning setting. This is different from the previous theoretical results on knowledge distillation, which mostly focus on feature matching (not relational). They introduce a new notion called low clustering error, which quantifies the difference betweeen the predicted and ground truth clusterings, and show that RKD can provably produce low clustering error. They also have a few other results, including the label efficiency theorm on a cluster-aware semi-supervised learning framework with low clustering errors, and unifying data augmentation consistency regularization with RKD, showing that consistency reuglarization focuses on the local perspective, while RKD improves over the global perspective.\n\nSTRENGTHS:\nOriginality: Since we already have seen a few papers on connecting spectral clustering with contrastive learning or other learning methods, the spectral clustering part of this paper is not particularly surprising. However, I do like the relational part, which focuses on the relationships between the features, rather than the feature matching itself. I think this is a very interesting and important direction to pursue. Moreover, the authors also investigate the benefit of clustering awareness in the semi-supervised learning scenario, which is novel. \n\nQuality: I think this is a well written paper with rigorous definitions, theorems and proofs. \n\nClarity: Since this is a theory paper, I will not say it is straightforward to read. However, I think the authors did a very good job in the introduction to demonstrate the main contributions of this paper, and present the technical theorems and assumptions in a structured way. Therefore I would say this paper is well written. \n\nSignificance: I think this paper is indeed interesting, because it provides an interesting connection between relational knowledge distillation learning and spectral clustering, and also provide theoretical characterization on various aspects.\n\nWEAKNESSES:\nIt is a bit difficult for me to understand the intuition of Assumption 4.2, and the toy example after assumption 4.2 is too special, which does not explain much about the complicated definitions. It would be nice to further elaborate on the intuitions, and perhaps give a non-trivial example of this assumption, and why this assumption is a mild assumption. \n\nPersonally I like Theorem 4.1 better, and I do not feel it is really necessary to consider the limited unlabeled samples setting (because usually we may assume the unlabeled samples are cheap). Therefore, it might improve the presentation of this paper, if the author can add more motivations on Sec 4.2. (if there is no such motivation, it's also fine, I think it is a minor spot.)\n\nThe current connections between different results are weak. Specifically, I cannot easily understand how the last DAC result connects with the previous two. I might be wrong, but I initially felt that the third result says the combination of DAC and RKD can improve the performance. However, the statement of Theorem 6.2 seems only contains DAC, not RKD. Therefore, it seems to me that the first and third results are in parallel: both of them are providing upper bounds of clustering error. Is it true? \n-- if this is true, I think the presentation of the introduction should be updated, to avoid potential misconceptions.\n\nQUESTIONS:\nI do not have other questions.\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper points out that RKD over a population-induced graph given by a teacher model leads to low clustering error (Spectral Clustering), which provably benefits to semi-supervised learning and other downstream tasks. Consequently, it provides a theoretical analysis of relational knowledge distillation (RKD) in the semi-supervised classification setting from a clustering perspective.\n\nSTRENGTHS:\nThis paper mainly focuses on expounding the close relationship between RKD and Spectral Clustering from a theoretic perspective, which definitely contributes to the increase of attention on RKD. According to this seminal work, RKD can be regarded as a principle choice to extract knowledge from a pre-trained large model.\n\nWEAKNESSES:\nThe weakness of this paper is obvious: The experimental part is too weak. There is no empirical study in the main text, and the experimental results in the appendix are limited. In addition, in some cases, RKD even deteriorates the Top-1 performance. The authors provided no explanation on these cases. Besides, CIFAR-10 and CIFAR-100 are not quite challenging datasets, and they are too similar. It is suggested conducting experiments on the large-scale datasets to better demonstrate the effectiveness of the proposed method.\n\nQUESTIONS:\nMy principle concern is the inadequate empirical study. As mentioned above, there are mainly two issues should be addressed. First, in the current results (Table 1 and Table 2), why the performance of the proposed approach cannot consistently outperform the baseline? Second, what is the performance of the proposed method on other large-scale datasets? If these issues can be resolved, I will recommend this work.\n\nLIMITATIONS:\nI agree with the limitations provided in the last section. Besides, please see the Weaknesses and Questions parts for my concern. I hope the review is helpful for this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a comprehensive theoretical proof of the relationship between RKD and clustering from the perspective of spectral clustering. The theoretical proof presented in the paper is detailed and reasonable. Firstly, RKD is regarded as a spectral clustering problem of the teacher model, and then the introduced clustering error is used to prove that RKD leads to low clustering error. Finally, the paper presents the sampling complexity bound of RKD in the case of limited unlabeled samples.\n\nSTRENGTHS:\n1.The theoretical proof presented in the paper is detailed and reasonable. \n2. Label efficiency of cluster-aware SSL is demonstrated as well.\n\nWEAKNESSES:\n1. It appears that the author attempts to convey numerous viewpoints in this paper, resulting in a somewhat confused logical structure. As a result, the paper is somewhat difficult to read and understand, possibly due to my unfamiliarity with the writing style of purely theoretical papers.\n2. While the author's arguments are generally reasonable, there seems to be a lack of strong correspondence between the presented problem and the method of reasoning. It would be beneficial to understand the author's motivations and thought process behind exploring RKD at a theoretical level, particularly in relation to demonstrating its label efficiency in the SSL setting.\n\nQUESTIONS:\nAs I am less familiar with purely theoretical work, I find that in Section 7, the author continues to discuss the theoretical analysis of RKD in different settings. While providing relevant theoretical proofs is important based on previous empirical work, it would be beneficial to consider adding a discussion section that explores potential new directions for practical applications.\n\nLIMITATIONS:\n1. The research presented in the paper holds significant theoretical value. It would be beneficial to enhance the overall clarity and organization of the paper's structure. This will improve the readability and friendliness of the paper for fellow researchers in the field.\n2. In the field of machine learning, it is highly recommended to conduct necessary experiments to strengthen the scientific rigor of the study.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}