SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Another avenue for future exploration involves RKD on different (hidden) layers of the neural network. In this work, we consider RKD on the output layer following the standard practice [Park et al., 2019], whereas RKD on additional hidden layers has been shown to further improve the performance [Liu et al., 2019]. While the analysis in Section 4 remains valid for hidden-layer features of low dimensions  $(\approx K)$ , Assumption 4.1 tends to fail (i.e.,  $\alpha$  can be large) for high-dimensional features, which may require separate and careful consideration.
YD was supported in part by the Office of Naval Research N00014-18-1-2354, NSF DMS-1952735, DOE ASCR DE-SC0022251, and UT Austin Graduate School Summer Fellowship. KM was supported by the Peter J. O'Donnell Jr. Postdoctoral Fellowship at the Oden Institute at the University of Texas, Austin. QL acknowledges the support of NYU Research Catalyst Prize and Whitehead Fellowship for Junior Faculty in Biomedical and Biological Sciences. RW was supported in part by AFOSR MURI FA9550-19-1-0005, NSF DMS-1952735, NSF IFML grant 2019844, NSF DMS-N2109155, and NSF 2217033. The authors wish to thank IFML for generously providing computational resources, as well as the anonymous reviewers for their helpful comments and suggestions.
Z. Allen-Zhu and Y. Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816, 2020.
J. Ba and R. Caruana. Do deep nets really need to be deep? Advances in neural information processing systems, 27, 2014.
P. Bachman, O. Alsharif, and D. Precup. Learning with pseudo-ensembles. Advances in neural information processing systems, 27, 2014.
P. L. Bartlett and S. Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. J. Mach. Learn. Res., 3(null):463-482, Mar. 2003. ISSN 1532-4435.
M. Belkin and P. Niyogi. Semi-supervised learning on Riemannian manifolds. Machine learning, 56: 209-239, 2004.
S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032, 2019.
S. Zagoruyko and N. Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928, 2016a.
S. Zagoruyko and N. Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016b.
H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.
D. Zhou, O. Bousquet, T. Lal, J. Weston, and B. Schölkopf. Learning with local and global consistency. In S. Thrun, L. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing Systems, volume 16. MIT Press, 2003.
J. Zhou, G. Cui, S. Hu, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun. Graph neural networks: A review of methods and applications. AI Open, 1:57-81, 2020. ISSN 2666-6510.
X. Zhu, Z. Ghahramani, and J. Lafferty. Semi-supervised learning using Gaussian fields and harmonic functions. In Proceedings of the 20th International Conference on International Conference on Machine Learning, pages 912-919, Washington, DC, USA, Aug. 2003. AAAI Press. ISBN 978-1-57735-189-4.
A Experiments
In this section, we present experimental results on CIFAR-10/100 [Krizhevsky and Hinton, 2009] to demonstrate the efficacy of combining DAC and RKD (i.e., the "local" and "global" perspectives of clustering) for semi-supervised learning in the low-label-rate regime. The experiment code can be found at https://github.com/dyjdongyijun/Semi_Supervised_Knowledge_Distillation.
We consider the (data-free) knowledge distillation setting, wherein the student model cannot access the training data of the teacher model. That is, the student models only have access to limited training samples, with very few labels, drawn from a potentially different data distribution than which the teacher model is pretrained on. More precisely, given a labeled sample set  $(\mathbf{X},\mathbf{y})$  of size  $n$  and an unlabeled sample set  $\mathbf{X}^u$  of size  $N$  ( $n\ll N$ )<sup>10</sup>, we train a student model (WideResNet [Zagoruyko and Komodakis, 2016b]) in the low-label-rate regime with as few as 4 labeled samples per class (i.e.,  $n = 4K$ , or  $n = 40$  for CIFAR-10 and  $n = 400$  for CIFAR-100), along with  $N = 25000$ , 40000, and 50000 unlabeled samples (i.e.,  $50\%$ ,  $80\%$ ,  $100\%$  of the entire CIFAR-10/100 training set).
Then, combining DAC and RKD, we solve the following objective in the experiments:
$$
\min  _ {f \in \mathcal {F}} \underbrace {\widehat {\mathcal {L}} _ {(\mathbf {X} , \mathbf {y})} ^ {\mathrm {C E}} (f)} _ {\text {S u p e r v i s e d c r o s s - e n t r o p y}} + \underbrace {\lambda_ {\text {D A C}} \cdot \widehat {\mathcal {L}} _ {\mathbf {X} ^ {u}} ^ {\mathrm {D A C}} (f)} _ {\text {U n s u p e r v i s e d D A C l o s s}} + \underbrace {\lambda_ {\text {R K D}} \cdot \widehat {R} _ {\mathbf {X} ^ {u}} (f)} _ {\text {e m p i r i c a l R K D l o s s}},
$$
where  $\widehat{\mathcal{L}}_{(\mathbf{X},\mathbf{y})}^{\mathrm{CE}}(f) = \frac{1}{n}\sum_{i=1}^{n}\ell_{\mathrm{CE}}(y_i|f(\mathbf{x}_i))$  denotes the supervised cross-entropy loss with  $\ell_{\mathrm{CE}}(y|f(\mathbf{x})) = -\log\left(\operatorname{softmax}(f(\mathbf{x}))_y\right)$ . The DAC and RKD losses are as described in Equation (9) and Equation (3), respectively. For the regularization hyper-parameters, we have  $\lambda_{\mathrm{DAC}} = 1$  and  $\lambda_{\mathrm{RKD}} = 0$  for DAC (only), while  $\lambda_{\mathrm{RKD}} = 0.001$  for DAC + RKD.
DAC via FixMatch. We take FixMatch [Sohn et al., 2020]-a state-of-the-art algorithm in the low-label-rate regime—as the semi-supervised learning baseline. By combining RKD with FixMatch, we show that RKD brings additional boosts to the classification accuracy, especially with unsuitable data augmentations and/or limited unlabeled data.
Following the formulation and implementation in Sohn et al. [2020], FixMatch involves a pair of weak and strong data augmentations  $(\mathbf{x}^w, \mathbf{x}^s)$  for each unlabeled sample  $\mathbf{x}^u$ . For a current model  $f$ , the DAC loss given by FixMatch can be expressed as
$$
\widehat {\mathcal {L}} _ {\mathbf {X} ^ {u}} ^ {\mathrm {D A C}} (f) \triangleq
$$
$$
\left. \operatorname {m e a n} \left(\left\{\ell_ {\mathrm {C E}} \left(y _ {f} \left(\mathbf {x} _ {i} ^ {w}\right) \mid f \left(\mathbf {x} _ {i} ^ {s}\right)\right) \right| \max  \left(\operatorname {s o f t m a x} \left(\frac {f \left(\mathbf {x} _ {i} ^ {w}\right)}{T}\right)\right) \geq \tau_ {\mathrm {D A C}}, i \in [ N ] \right\}\right), \tag {9}
$$
where  $T = 1$  is a fixed temperature hyper-parameter.  $\tau_{\mathrm{DAC}}$  is the threshold below which the pseudo-label gauged on  $\mathbf{x}_i^w$  is considered "with low confidence" and therefore discarded. We set  $\tau_{\mathrm{DAC}} = 0.95$  for CIFAR-10 and  $\tau_{\mathrm{DAC}} = 0.8$  for CIFAR-100.
In FixMatch, the weak augmentations include random flips and crops, whereas the strong augmentations additionally involve RandAugment [Cubuk et al., 2020]. To investigate the scenarios with unsuitable data augmentations and/or limited unlabeled data, we conduct experiments with various (i) augmentation strength (i.e., magnitude of transformations)  $m = 2,6,10$  in RandAugment [Cubuk et al., 2020] and (ii) unlabeled sample size  $N = 25000,40000,50000$  (i.e.,  $50\% ,80\% ,100\%$  of the entire CIFAR-10/100 training set).
RKD. We warm up with the in-distribution case in the CIFAR-10 experiments (Table 1) where the teacher and student models are (pre)trained on the same dataset (i.e., CIFAR-10). Concretely, the teacher model  $\psi : \mathcal{X} \to \mathcal{W}$  is a Densenet 161 [Huang et al., 2017, Phan, 2021] pretrained on the entirety of the CIFAR-10 training set (via supervised ERM).
Taking a step further, we consider an out-of-distribution scenario in the CIFAR-100 experiments (Table 2) where the teacher model is pretrained on ImageNet [Deng et al., 2009] via an unsupervised contrastive learning method-SwAV [Caron et al., 2020]. Intuitively, we choose teacher models
It is worth mentioning that, in practice, we relax the assumption of the independence between  $(\mathbf{X},\mathbf{y})$  and  $\mathbf{X}^u$  and recycle  $\mathbf{X}$  in  $\mathbf{X}^u$ .
pretrained via SwAV as it encourages both data clustering and consistency of cluster assignment under data augmentations without involving supervision, which tends to provide representations that generalize better in a cluster-aware semi-supervised learning setting.
Given the teacher models, the output (response) of each sample is taken as the corresponding teacher feature such that  $\mathcal{W} = \mathbb{R}^K$ . In light of the empirical success of RKD based on simple kernel functions [Park et al., 2019, Liu et al., 2019], we focus on the (shifted) cosine kernel,  $k_{\psi}(\mathbf{x},\mathbf{x}^{\prime}) = 1 + \frac{\psi(\mathbf{x})^{\top}\psi(\mathbf{x}^{\prime})}{\|\psi(\mathbf{x})\|_2\|\psi(\mathbf{x}^{\prime})\|_2}$ , in the experiments.
With data augmentation (e.g., when coupling with FixMatch), we assume that weak augmentations (i.e., random flips and crops) bring little perturbations on responses of the teacher model (i.e., the teacher features), intuitively as the predictions made by a good teacher model ought to be invariance under weak augmentations. Thereby, we can considerably reduce memory burden and accelerate the RKD process by storing the teacher features  $\psi(\mathbf{x})$  of the original data  $\mathbf{x} \in \mathcal{X}$  offline, instead of inferring every augmented sample  $\mathbf{x}' \in \mathcal{A}(\mathbf{x})$  with the large teacher model  $\psi(\mathbf{x}')$  online.
Labeled data selection. As alluded to in Remark 5.2, the success of low-label-rate (i.e.,  $n \ll N$ ) semi-supervised classification naturally depends on the quality of the (extremely) limited amount of labeled data to represent the underlying population. We perform experiments in which the selection of labeled data is non-adaptive (Uniform) or adaptive (StochasticGreedy). The non-adaptive setting, Uniform, selects labeled data uniformly in each ground truth class<sup>11</sup>. This contrasts with the strictly i.i.d. sampling assumption used in our theoretical results but constitutes a reasonable alternative that has been used in other low-label rate semi-supervised classification works (e.g., [Calder et al., 2020]).
We also utilize a coreset selection strategy (StochasticGreedy) for identifying "representative" inputs to use as labeled data in an adaptive manner. This labeled set is selected via a stochastic greedy [Mirzasoleiman et al., 2015] optimization problem to (approximately) minimize a facility location objective<sup>12</sup>. An in-depth discussion of coreset selection methods is outside the scope of the current work, but we refer the interested reader to the enlightening references [Bilmes, 2022, Krause and Golovin, 2014, Mirzasoleiman, 2017] regarding submodular optimization methods in machine learning.
Table 1: Top-1 accuracy on CIFAR-10.
Label selection	Sample sizes (n, N)	SSL algorithm	Augmentation Strength
High	Medium	Weak
Uniform	(40, 50000)	FixMatch	89.47 ± 3.83	87.89 ± 1.43	81.36 ± 1.41
FixMatch + RKD	89.88 ± 0.41	89.10 ± 1.14	84.61 ± 1.23
(40, 40000)	FixMatch	87.60 ± 1.21	85.30 ± 1.35	83.74 ± 3.07
FixMatch + RKD	90.19 ± 0.19	88.65 ± 1.81	86.68 ± 1.06
(40, 25000)	FixMatch	81.73 ± 1.27	81.43 ± 2.98	76.84 ± 1.98
FixMatch + RKD	87.06 ± 2.04	87.24 ± 0.39	82.75 ± 2.35
StochasticGreedy	(40, 50000)	FixMatch	85.48 ± 1.67	89.39 ± 0.82	78.98 ± 0.52
FixMatch + RKD	91.19 ± 0.26	90.00 ± 0.64	87.11 ± 4.33
(40, 40000)	FixMatch	84.51 ± 1.34	84.88 ± 6.68	70.70 ± 5.32
FixMatch + RKD	85.68 ± 1.19	88.31 ± 1.36	85.12 ± 3.49
(40, 25000)	FixMatch	79.74 ± 8.02	73.97 ± 1.98	73.86 ± 3.29
FixMatch + RKD	83.72 ± 4.73	82.09 ± 3.27	80.65 ± 3.67
Results. From Table 1 and Table 2, we observe that the incorporation of RKD with FixMatch (i.e., adding the "global" perspective of RKD to the "local" perspective of DAC) generally brings additional improvements to the classification accuracy, especially when the data augmentation strength is inappropriate and/or the number of unlabeled data is limited.
<sup>11</sup>For fair comparison and consistent performance among different configurations in the experiments, we use a separate random number generator with a fixed random seed to ensure that the same set of labeled samples is selected for all experiments.
<sup>12</sup>We use the Apricot [Schreiber et al., 2020] submodular optimization codebase in Python.
Table 2: Top-1 accuracy on CIFAR-100.
Label selection	Sample sizes (n, N)	SSL algorithm	Augmentation Strength
High	Medium	Weak
Uniform	(400, 50000)	FixMatch	46.85 ± 1.68	45.67 ± 0.51	45.84 ± 0.35
FixMatch + RKD	49.33 ± 0.42	47.97 ± 0.37	47.65 ± 0.17
(400, 40000)	FixMatch	44.11 ± 0.51	42.98 ± 0.56	42.74 ± 0.42
FixMatch + RKD	46.24 ± 0.64	45.78 ± 0.94	45.13 ± 1.08
(400, 25000)	FixMatch	36.60 ± 0.03	35.86 ± 0.51	33.59 ± 1.54
FixMatch + RKD	37.87 ± 0.98	36.69 ± 0.41	35.69 ± 1.51
StochasticGreedy	(400, 50000)	FixMatch	47.45 ± 0.30	48.64 ± 0.35	46.79 ± 1.05
FixMatch + RKD	50.41 ± 1.41	50.67 ± 1.62	49.09 ± 1.31
(400, 40000)	FixMatch	45.93 ± 0.21	44.78 ± 1.51	44.62 ± 0.72
FixMatch + RKD	47.62 ± 0.67	47.28 ± 0.56	46.07 ± 0.86
(400, 25000)	FixMatch	39.40 ± 0.31	39.01 ± 0.68	38.22 ± 0.42
FixMatch + RKD	40.88 ± 0.61	40.73 ± 0.46	39.20 ± 0.73
Training details. Throughout the experiments, we used weight decay 0.0005. We train the student model via stochastic gradient descent (SGD) with Nesterov momentum 0.9 for  $2^{17}$  iterations (batches) with a batch size  $64*8 = 2^9$  (consisting of 64 labeled samples and  $64*7$  unlabeled samples). The initial learning rate is 0.03, decaying with a cosine scheduler. The test accuracies are evaluated  $2^7$  times in total evenly throughout the  $2^{17}$  iterations (i.e., one test evaluation after every  $2^{10}$  iterations) on an EMA model with a decay rate 0.999. The average and standard deviation of the best test accuracy (i.e., early stopping with the maximum patience 128) are reported for each experiment over 3 arbitrary random seeds. Both CIFAR-10/100 experiments are conducted on one NVIDIA A40 GPU.
B Proofs and Discussions for Section 5
Lemma B.1 (Rademacher complexity with cluster-aware functions (generalization of Yang et al. [2023] Theorem 8)). For a fixed  $\mathcal{F}_{\mathbf{L}(\mathbf{X}^u)}$ , let
$$
\ell \circ \mathcal {F} _ {\mathbf {L} \left(\mathbf {X} ^ {u}\right)} = \left\{\ell (f (\cdot), \cdot): \mathcal {X} \times \mathcal {Y} \rightarrow \{0, 1 \} \mid f \in \mathcal {F} _ {\mathbf {L} \left(\mathbf {X} ^ {u}\right)} \right\}
$$
be the loss function class. Then, its Rademacher complexity can be upper-bounded by
$$
\Re_ {n} \left(\ell \circ \mathcal {F} _ {\mathbf {L} \left(\mathbf {X} ^ {u}\right)}\right) \leq \sqrt {\frac {2 K \log (2 n)}{n} + 2 \mu \left(\mathcal {F} _ {\mathbf {L} \left(\mathbf {X} ^ {u}\right)}\right)}. \tag {10}
$$
Proof of Lemma B.1. Given a set of i.i.d. samples  $(\mathbf{X},\mathbf{y})\sim P(\mathbf{x},\mathbf{y})^n$ , let
$$
\left. \right. \widehat {\mathfrak {S}} _ {(\mathbf {X}, \mathbf {y})} \left(\ell \circ \mathcal {F} _ {\mathbf {L} (\mathbf {X} ^ {u})}\right) \triangleq \left|\left\{\left(\ell \left(f (\mathbf {x} _ {1}), y _ {1}\right), \dots , \ell \left(f (\mathbf {x} _ {n}), y _ {n}\right)\right) \mid f \in \mathcal {F} _ {\mathbf {L} (\mathbf {X} ^ {u})} \right\}\right|.
$$
denote the number of distinct patterns over  $(\mathbf{X},\mathbf{y})$  in  $\ell \circ \mathcal{F}_{\mathbf{L}(\mathbf{X}^u)}$ . Then, by Massart's finite lemma, the empirical Rademacher complexity with respect to  $(\mathbf{X},\mathbf{y})$  is upper bounded by
$$
\widehat {\mathfrak {R}} _ {(\mathbf {X}, \mathbf {y})} \left(\ell \circ \mathcal {F} _ {\mathbf {L} (\mathbf {X} ^ {u})}\right) \leq \sqrt {\frac {2 \log \left(\widehat {\mathfrak {S}} _ {(\mathbf {X} , \mathbf {y})} \left(\ell \circ \mathcal {F} _ {\mathbf {L} (\mathbf {X} ^ {u})}\right)\right)}{n}}.
$$
By the concavity of  $\sqrt{\log(\cdot)}$ , we know that,
$$
\begin{array}{l} \mathfrak {R} _ {n} \left(\ell \circ \mathcal {F} _ {\mathbf {L} (\mathbf {X} ^ {u})}\right) = \mathbb {E} _ {(\mathbf {X}, \mathbf {y})} \left[ \widehat {\mathfrak {R}} _ {(\mathbf {X}, \mathbf {y})} \left(\ell \circ \mathcal {F} _ {\mathbf {L} (\mathbf {X} ^ {u})}\right) \right] \\ \leq \mathbb {E} _ {(\mathbf {X}, \mathbf {y})} \left[ \sqrt {\frac {2 \log \left(\widehat {\mathfrak {S}} _ {(\mathbf {X} , \mathbf {y})} \left(\ell \circ \mathcal {F} _ {\mathbf {L} \left(\mathbf {X} ^ {u}\right)}\right)\right)}{n}} \right] \tag {11} \\ \leq \sqrt {\frac {2 \log \left(\mathbb {E} _ {(\mathbf {X} , \mathbf {y})} \left[ \widehat {\mathfrak {S}} _ {(\mathbf {X} , \mathbf {y})} \left(\ell \circ \mathcal {F} _ {\mathbf {L} (\mathbf {X} ^ {u})}\right) \right]\right)}{n}}. \\ \end{array}
$$
Since  $P(M(f)) \leq \mu\left(\mathcal{F}_{\mathbf{L}(\mathbf{X}^u)}\right) \leq \frac{1}{2}$  for all  $f \in \mathcal{F}_{\mathbf{L}(\mathbf{X}^u)}$ , by union bound, we have
$$
\begin{array}{l} \mathbb {E} _ {(\mathbf {X}, \mathbf {y})} \left[ \widehat {\mathfrak {S}} _ {(\mathbf {X}, \mathbf {y})} \left(\ell \circ \mathcal {F} _ {\mathbf {L} (\mathbf {X} ^ {u})}\right) \right] \\ \leq \sum_{\iota = 0}^{n}\binom {n}{\iota}\mu \left(\mathcal{F}_{\mathbf{L}(\mathbf{X}^{u})}\right)^{\iota}\left(1 - \mu \left(\mathcal{F}_{\mathbf{L}(\mathbf{X}^{u})}\right)\right)^{n - \iota}\cdot \left( \begin{array}{c}n - \iota +1\\ \min \left\{K,n - \iota \right\} -1 \end{array} \right)2^{K + \iota} \\ \leq (2 n) ^ {K} \sum_ {\iota = 0} ^ {n} \binom {n} {\iota} \left(2 \mu \left(\mathcal {F} _ {\mathbf {L} \left(\mathbf {X} ^ {u}\right)}\right)\right) ^ {\iota} \left(1 - \mu \left(\mathcal {F} _ {\mathbf {L} \left(\mathbf {X} ^ {u}\right)}\right)\right) ^ {n - \iota} \tag {12} \\ = (2 n) ^ {K} \left(1 - \mu \left(\mathcal {F} _ {\mathbf {L} (\mathbf {X} ^ {u})}\right) + 2 \mu \left(\mathcal {F} _ {\mathbf {L} (\mathbf {X} ^ {u})}\right)\right) ^ {n} \\ \leq (2 n) ^ {K} e ^ {n \mu \left(\mathcal {F} _ {\mathbf {L} (\mathbf {X} ^ {u})}\right)}, \\ \end{array}
$$
where  $\iota$  counts the number of minority samples in the  $n$  samples. Plugging in this last line into Equation (11) yields Equation (10).
Proof of Theorem 5.1. Since  $\ell \circ \mathcal{F}_{\mathbf{L}(\mathbf{X}^u)}$  is 1-bounded with the zero-one loss  $\ell$ , Lemma F.1 implies that with probability at least  $1 - \delta /2$  over  $(\mathbf{X},\mathbf{y})$ ,
$$
\mathcal {E} \left(\widehat {f} _ {| \mathbf {X} ^ {u}}\right) - \mathcal {E} \left(f _ {*}\right) \leq 4 \Re_ {n} \left(\ell \circ \mathcal {F} _ {\mathbf {L} (\mathbf {X} ^ {u})}\right) + \sqrt {\frac {2 \log \left(4 / \delta\right)}{n}}.
$$
Then, incorporating the upper bound of  $\Re_n\left(\ell \circ \mathcal{F}_{\mathbf{L}(\mathbf{X}^u)}\right)$  from Lemma B.1 completes the proof.
B.2 Cluster-wise Labeling with Cluster-aware Predictions
Recall the notions of majority labeling  $\widetilde{y}_f(\cdot)$  (Equation (2)) and minority subset  $M(f)$  (Definition 3.1). We first specify the cluster-aware predictions that are useful for labeling with the following nondegeneracy assumption.
Assumption B.1  $((m_0, c_0)$ -non-degenerate predictions). For any prediction function  $f \in \mathcal{F}$ , we say  $f$  is  $(m_0, c_0)$ -non-degenerate if the following properties are satisfied:
(i)  $y_{f}:\mathcal{X}\rightarrow [K]$  is surjective (i.e.,  $\{y_f(\mathbf{x})\mid \mathbf{x}\in \mathcal{X}\} = [K])$
(ii)  $\left|\mathcal{X}_k^f\right| \geq m_0$  for all  $k \in [K]$ , where we define  $\mathcal{X}_k^f \triangleq \{\mathbf{x} \in \mathcal{X} \mid \widetilde{y}_f(\mathbf{x}) = k\}$ .
(iii) There exists some  $c_0 \geq 2$  such that  $c_0 \cdot P\left(M(f) \cap \mathcal{X}_k^f\right) \leq P\left(\mathcal{X}_k^f\right)$  for all  $k \in [K]$ .
It is worth pointing out that although the majority labeling  $\widetilde{y}_f(\cdot)$  depends on the ground truth and therefore remains unknown without label acquisition, under Assumption B.1, the associated partition  $\left\{\mathcal{X}_k^f\right\}_{k\in [K]}$  over  $\mathcal{X}$  depends only on  $f$  and is known without labeling. This is because  $y_{f}(\cdot)$  and  $\widetilde{y}_f(\cdot)$  induce the same partition over  $\mathcal{X}$  when both  $y_{f}(\cdot)$  and  $\widetilde{y}_f(\cdot)$  are non-degenerate (i.e., being surjective functions onto  $[K]$ , a necessary condition of Assumption B.1 (i) and (ii)). We also notice that Assumption B.1 (iii) can be generally satisfied with a reasonably large  $c$  when learning via Equation (3) with a sufficiently small  $\mu\left(\mathcal{F}_{\mathbf{L}(\mathbf{X}^u)}\right)$  (cf. Theorem 4.1 or Theorem 4.3). For example, when  $c_0\cdot \mu \left(\mathcal{F}_{\mathbf{L}(\mathbf{X}^u)}\right)\leq \min_{k\in K}P\left(\mathcal{X}_k^f\right)$ , Assumption B.1 (iii) is automatically satisfied.
With the cluster-aware predictions in Assumption B.1, we show that  $O(\log(K))$  i.i.d. labeled samples per predicted cluster are sufficient to ensure at least one labeled sample per class. That is, the cluster-wise labeling with  $(m_0, c_0)$ -non-degenerate predictions guarantees a label complexity of  $O(K \log(K))$  even with class imbalance.
Proposition B.2 (Cluster-wise labeling). Given any  $(m_0, c_0)$ -non-degenerate prediction function  $f \in \mathcal{F}$  (Assumption B.1) with sufficiently large  $m_0 \gg \log_{c_0}(2K)$  and  $c_0 \geq 2$ , for each  $k \in [K]$ , acquire labels of  $m$  ( $m \leq m_0$ ) unlabeled samples  $\{\mathbf{x}_{ki}^u\}_{i \in [m]}\sim \mathrm{Unif}\left(\mathcal{X}_k^f\right)^m$  drawn i.i.d. from  $\mathcal{X}_k^f$ . Then, for any  $\delta \in \left(\frac{2K}{c_0^{m_0}}, 1\right)$ , when  $m \geq \log_{c_0}\left(\frac{2K}{\delta}\right)$ ,  $\{y_*(\mathbf{x}_{ki}^u)\}_{k \in [K], i \in [m]} = [K]$  have at least one ground truth label per class with probability at least  $1 - \delta$ .
Proof of Proposition B.2. We first observe that
$$
\begin{array}{l} \mathbb {P} \left[ \left\{y _ {*} \left(\mathbf {x} _ {k i} ^ {u}\right) \right\} _ {k \in [ K ], i \in [ m ]} = [ K ] \right] = \mathbb {P} [ \forall k ^ {\prime} \in [ K ], \exists k \in [ K ], i \in [ m ] \text {s . t .} y _ {*} \left(\mathbf {x} _ {k i} ^ {u}\right) = k ^ {\prime} ] \\ \geq \mathbb {P} [ \forall k \in [ K ], \exists i \in [ m ] \text {s . t .} y _ {*} (\mathbf {x} _ {k i} ^ {u}) = k ] \\ = \prod_ {k = 1} ^ {K} \mathbb {P} _ {\left\{\mathbf {x} _ {k i} ^ {u} \right\} _ {i \in [ m ]} \sim \operatorname {U n i f} \left(\mathcal {X} _ {k} ^ {f}\right) ^ {m}} \left[ \exists i \in [ m ] \text {s . t .} y _ {*} \left(\mathbf {x} _ {k i} ^ {u}\right) = k \right], \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {P} [ \exists i \in [ m ] \text {s . t .} y _ {*} (\mathbf {x} _ {k i} ^ {u}) = k ] = 1 - \mathbb {P} [ \forall i \in [ m ], y _ {*} (\mathbf {x} _ {k i} ^ {u}) \neq k ] \\ = 1 - \left(\mathbb {P} _ {\mathbf {x} \sim \operatorname {U n i f} \left(\mathcal {X} _ {k} ^ {f}\right)} \left[ y _ {*} (\mathbf {x}) \neq \widetilde {y} _ {f} (\mathbf {x}) \right]\right) ^ {m} \\ = 1 - \left(\mathbb {P} _ {\mathbf {x} \sim \operatorname {U n i f} \left(\mathcal {X} _ {k} ^ {f}\right)} [ \mathbf {x} \in M (f) ]\right) ^ {m} \\ = 1 - \left(\frac {P \left(M (f) \cap \mathcal {X} _ {k} ^ {f}\right)}{P \left(\mathcal {X} _ {k} ^ {f}\right)}\right) ^ {m}. \\ \end{array}
$$
Since Assumption B.1 implies  $P\left(M(f)\cap \mathcal{X}_k^f\right)\bigg{/}P\left(\mathcal{X}_k^f\right)\leq \frac{1}{c_0}$  for all  $k\in [K]$ , we have
$$
\mathbb {P} \left[ \{y _ {*} (\mathbf {x} _ {k i} ^ {u}) \} _ {k \in [ K ], i \in [ m ]} = [ K ] \right] \geq \left(1 - \left(\frac {1}{c _ {0}}\right) ^ {m}\right) ^ {K} \geq \exp \left(- \frac {2 K}{c _ {0} ^ {m}}\right) \geq 1 - \frac {2 K}{c _ {0} ^ {m}},
$$