Title: Enhancing Semi-Supervised Learning via Representative and Diverse Sample Selection
Venue: NeurIPS 2024
Accepted: yes
Keywords: Semi-supervised learning

Sample selection

Low-budget learning
Abstract: Semi-Supervised Learning (SSL) has become a preferred paradigm in many deep learning tasks, which reduces the need for human labor. Previous studies primarily focus on effectively utilising the labelled and unlabeled data to improve performance. However, we observe that how to select samples for labelling also significantly impacts performance, particularly under extremely low-budget settings. The sample selection task in SSL has been under-explored for a long time. To fill in this gap, we propose a Representative and Diverse Sample Selection approach (RDSS). By adopting a modified Frank-Wolfe algorithm to minimise a novel criterion $\alpha$-Maximum Mean Discrepancy ($\alpha$-MMD), RDSS samples a representative and diverse subset for annotation from the unlabeled data. We demonstrate that minimizing $\alpha$-MMD enhances the generalization ability of low-budget learning. Experimental results show that RDSS consistently improves the performance of several popular SSL frameworks and outperforms the state-of-the-art sample selection approaches used in Active Learning (AL) and Semi-Supervised Active Learning (SSAL), even with constrained annotation budgets. Our code is available at [RDSS](https://github.com/YanhuiAILab/RDSS).

Enhancing Semi-Supervised Learning via Representative and Diverse Sample Selection

Abstract

Semi-Supervised Learning (SSL) has become a preferred paradigm in many deep learning tasks, which reduces the need for human labor. Previous studies primarily focus on effectively utilising the labelled and unlabeled data to improve performance. However, we observe that how to select samples for labelling also significantly impacts performance, particularly under extremely low-budget settings. The sample selection task in SSL has been under-explored for a long time. To fill in this gap, we propose a Representative and Diverse Sample Selection approach (RDSS). By adopting a modified Frank-Wolfe algorithm to minimise a novel criterion α -Maximum Mean Discrepancy ( α -MMD), RDSS samples a representative and diverse subset for annotation from the unlabeled data. We demonstrate that minimizing α -MMD enhances the generalization ability of low-budget learning. Experimental results show that RDSS consistently improves the performance of several popular SSL frameworks and outperforms the state-of-the-art sample selection approaches used in Active Learning (AL) and Semi-Supervised Active Learning (SSAL), even with constrained annotation budgets. Our code is available at RDSS.

1 Introduction

Semi-Supervised Learning (SSL) is a popular paradigm which reduces reliance on large amounts of labeled data in many deep learning tasks [40, 59]. Previous SSL research mainly focuses on effectively utilising labelled and unlabeled data. Specifically, labelled data directly supervise model learning, while unlabeled data help learn a desirable model that makes consistent and unambiguous predictions [53]. Besides, we also find that how to select samples for annotation will greatly affect model performance, particularly under extremely low-budget settings (see Section 7.2).

The prevailing sample selection methods in SSL have many shortcomings. For example, random sampling may introduce imbalanced class distributions and inadequate coverage of the overall data distribution, resulting in poor performance. Stratified sampling randomly selects samples within each class, which is impractical in real-world scenarios where the label for each sample is unknown. Existing researchers also employ representativeness and diversity strategies to select appropriate samples for annotation. Representativeness [13] ensures that the selected subset distributes similarly with the entire dataset, and diversity [54] is designed to select informative samples by pushing away

a) Representativeness only

…

b) Diversity only

…

c)

𝛼

-MMD (ours)

…

them in feature space. And focusing on only one aspect presents significant limitations (Figure 1a and b). To address these issues, Xie et al. [57] and Wang et al. [50] employ a combination of the two strategies for sample selection. These methods set a fixed ratio for representativeness and diversity, restricting the ultimate performance through our empirical evidence (see Section 7.4). Fundamentally, they lack a theoretical basis to substantiate their effectiveness.

We observe that Active Learning (AL) primarily focuses on selecting the right samples for annotation, and numerous studies transfer the sample selection methods of AL into SSL, giving rise to SemiSupervised Active Learning (SSAL) [51]. However, most of these approaches exhibit several limitations: (1) They require randomly selected samples to begin with, which expends a portion of the labelling budget, making it difficult to work effectively with a very limited budget ( e.g. , 1% or even lower) [6]; (2) They involve human annotators in iterative cycles of labelling and training, leading to substantial labelling overhead [57]; (3) They are coupled with the model training so that samples for annotation need to be re-selected every time a model is trained [50]. In summary, selecting the appropriate samples for annotation is challenging in SSL.

To address these challenges, we propose a Representative and Diverse Sample Selection approach (RDSS) that requests annotations only once and operates independently of the downstream tasks. Specifically, inspired by the concept of Maximum Mean Discrepancy (MMD) [14], we design a novel criterion named α -MMD. It aims to strike a balance between representativeness and diversity via a trade-off parameter α (Figure 1c), for which we find an optimal interval adapt to different budgets. By using a modified Frank-Wolfe algorithm called Generalized Kernel Herding without Replacement (GKHR), we can get an efficient approximate solution to this minimization problem.

We prove that under certain Reproducing Kernel Hilbert Space (RKHS) assumptions, α -MMD effectively bounds the difference between training with a constrained versus an unlimited labelling budget. This implies that our proposed method could significantly enhance the generalization ability of learning with limited labels. We also give a theoretical assessment of GKHR with some supplementary numerical experiments, showing that GKHR performs well in learning with limited labels.

Furthermore, we evaluate our proposed RDSS across several popular SSL frameworks on the datasets CIFAR-10/100 [19], SVHN [30], STL-10 [9] and ImageNet [10]. Extensive experiments show that RDSS outperforms other sample selection methods widely used in SSL, AL or SSAL, especially with a constrained annotation budget. Besides, ablation experimental results demonstrate that RDSS outperforms methods using a fixed ratio.

The main contributions of this article are as follows:

We propose RDSS, which selects representative and diverse samples for annotation to enhance SSL by minimizing a novel criterion α -MMD. Under low-budget settings, we develop a fast and efficient algorithm, GKHR, for optimization.

We prove that our method benefits the generalizability of the trained model under certain assumptions and rigorously establish an optimal interval for the trade-off parameter α adapt to the different budgets.

We compare RDSS with sample selection strategies widely used in SSL, AL or SSAL, the results of which demonstrate superior sample efficiency compared to these strategies. In addition, we conduct ablation experiments to verify our method's superiority over the fixed-ratio approach.

2 Related Work

Semi-Supervised Learning Semi-Supervised Learning (SSL) effectively utilizes sparse labeled data and abundant unlabeled data for model training. Consistency Regularization [34, 20, 45], PseudoLabeling [21, 56] and their hybrid strategies [40, 63, 35] are commonly used in SSL. Consistency Regularization ensures the model's output stays stable even when there's noise or small changes in the input, usually from the data augmentation [55]. Pseudo-labelling integrates high-confidence data pseudo-labels directly into training, adhering to entropy minimization [23]. Moreover, an integrative approach that combines the aforementioned strategies can also achieve substantial results [53, 59]. Even though these approaches have been proven effective, they usually assume that labelled samples are randomly selected from each class ( i.e., stratified sampling), which is not practical in real-world scenarios where the label for each sample is unknown.

Active Learning Active learning (AL) aims to optimize the learning process by selecting the appropriate samples for labelling, reducing reliance on large labelled datasets. There are two different criteria for sample selection: uncertainty and representativeness. Uncertainty sampling selects samples about which the current model is most uncertain. Earlier studies utilized posterior probability [22, 49], entropy [18, 26], and classification margin [47] to estimate uncertainty. Recent research regards uncertainty as training loss [17, 60], influence on model performance [11, 24] or the prediction discrepancies between multiple classifiers [8]. However, uncertainty sampling methods may exhibit performance disparities across different models, leading researchers to focus on representativeness sampling, which aims to align the distribution of selected subset with that of the entire dataset [36, 39, 27]. Most AL approaches are difficult to perform well under extremely low-label settings. This may be because they usually require randomly selected samples to begin with and involve human annotators in iterative cycles of labelling and training, leading to substantial labelling overhead.

Model-Free Subsampling Subsampling is a statistical approach which selects a subset with size m as a surrogate for the full dataset with size n ≫ m . While model-based subsampling methods depend heavily on the model assumptions [1, 61], improper choice of the model could lead to bad performance of estimation and prediction. In that case, model-free subsampling is preferred in data-driven modelling tasks, as it does not depend on the model assumptions. There are mainly two kinds of popular model-free subsampling methods. The one is induced by minimizing statistical discrepancies, which forces the distribution of subset to be similar to that of full data, in other words, selects representative subsamples, such as Wasserstein distance [13], energy distance [28], uniform design [65], maximum mean discrepancy [7] and generalized empirical F -discrepancy [66]. The other tends to select a diverse subset containing as many informative samples as possible [54]. The above-mentioned methodologies either exclusively focus on representativeness or diversity, which are difficult to effectively apply to SSL.

3 Problem Setup

Let X be the unlabeled data space, Y be the label space, X n = { x i } i ∈ [ n ] ⊂ X be the full unlabeled dataset containing pairwise different data, and I m = { i 1 , i 2 , · · · , i m } ⊂ [ n ]( m<n ) be an index set contained in [ n ] , our goal is to find an index set I ∗ m = { i ∗ 1 , i ∗ 2 , · · · , i ∗ m } ⊂ [ n ]( m<n ) such that the selected set of samples X I ∗ m = { x i ∗ 1 , x i ∗ 2 , · · · , x i ∗ m } is the most informative. After that, we can get access to the true labels of selected samples and use the set of labelled data S = { ( x i , y i ) } i ∈I ∗ m and the rest of the unlabeled data to train a deep learning model.

Following the methodology of previous works, we use representativeness and diversity as criteria for evaluating the informativeness of selected samples. Representativeness ensures the selected samples distribute similarly to the full unlabeled dataset. Diversity is proposed to prevent an excessive concentration of selected samples in high-density areas of the full unlabeled dataset. Furthermore, the cluster assumption in SSL suggests that the data tend to form discrete clusters, in which boundary points are likely to be located in the low-density area. Therefore, under this assumption, selected samples with diversity contain more boundary points than the non-diversified ones, which is desired in training classifiers.

As a result, our goal can be formulated by solving the following problem:

where Rep ( X I m , X n ) and Div ( X I m , X n ) quantify the representativeness and diversity of selected samples respectively and λ is a hyperparameter to balance the trade-off representativeness and diversity.

Besides, we propose another two fundamental settings which are beneficial to the implementation of the framework: (1) Low-budget learning. The budget for many of the real-world tasks which require sample selection procedures is relatively low compared to the size of unlabeled data. Therefore, we set m/n ≤ 0 . 2 in default in the following context, including the analysis of the sampling algorithm and the experiments; (2) Sampling without Replacement. Compared with the setting of sampling with replacement, sampling without replacement offers several benefits which better match our tasks, including bias and variance reduction, precision increase and representativeness enhancement [25, 46].

4 Representative and Diversity Sample Selection

The Representative and Diverse Sample Selection (RDSS) framework consists of two steps: (1) Quantification. We quantify the representativeness and diversity of selected samples by a novel concept called α -MMD (6), where λ is replaced by α as the trade-off hyperparameter; (2) Optimization. We optimize α -MMD by GKHR algorithm to obtain the optimally selected samples X I ∗ m .

4.1 Quantification of Diversity and Representativeness

In classical statistics and machine learning problems, the inner product of data points x , y ∈ X , defined by ⟨ x , y ⟩ , is employed to as a similarity measure between x , y . However, the application of linear functions can be very restrictive in real-world problems. In contrast, kernel methods use kernel functions k ( x , y ) , including Gaussian kernels (RBF), Laplacian kernels and polynomial kernels, as non-linear similarity measures between x , y , which are actually inner products of the projections of k ( x , y ) in some high-dimensional feature space [29].

Let k ( · , · ) be a kernel function on X × X , and we employ k ( · , · ) to measure the similarity between any two points and the average similarity, denoted by

to measure the similarity between the selected samples. Obviously, S ( X I m ) can evaluate the diversity of X I m since larger similarity implies smaller diversity.

As a statistical discrepancy which measures the distance between distributions, the maximum mean discrepancy (MMD) is introduced here to quantify the representativeness of X I m to X n . Proposed by Gretton et al. [14], MMD is formally defined below:

Definition 4.1 ( Maximum Mean Discrepancy ) . Let P, Q be two Borel probability measures on X . Suppose f is sampled from the unit ball in a reproducing kernel Hilbert space (RKHS) H associated with its reproducing kernel k ( · , · ) , i.e., ∥ f ∥ H ≤ 1 , then the MMD between P and Q is defined by

where X,X ′ ∼ P and Y, Y ′ ∼ Q are independent copies.

We can next derive the empirical version for MMD that is able to measure the representativeness of X I m = { x i } i ∈I m relative to X n = { x i } n i =1 by replacing P, Q with the empirical distribution constructed by X I m , X n in (3):

Optimization objective. Set Rep ( · , · ) = -MMD 2 k ( · , · ) and Div ( · ) = -S k ( · ) in (1), where k is a proper kernel function, our optimization objective becomes

Set λ = 1 -α αm , since ∑ n i =1 ∑ n j =1 k ( x i , x j ) is a constant, the objective function in (5) can be rewritten by

which defines a new concept called α -MMD, denoted by MMD k,α ( X I m , X n ) . This new concept distinguishes our method from those existing methods, which is essential for developing the sampling algorithms and theoretical analysis. Note that α -MMD degenerates to classical MMD when α = 1 and degenerates to average similarity when α = 0 . As α decreases, λ increases, thereby encouraging the diversity for sample selection.

Remark 1. In the following context, all the kernels are assumed to be characteristic and positive definite if not specified. The following illustrates the advantages of the two properties.

Characteristics kernels. The MMD is generally a pseudo-metric on the space of all Borel probability distributions, implying that the MMD between two different distributions can be zero. Nevertheless, MMDbecomes a proper metric when k is a characteristic kernel, i.e., P → ∫ X k ( · , x ) dP for any Borel probability distribution P on X [29]. Therefore, MMD induced by characteristic kernels can be more appropriate for measuring representativeness.

Positive definite kernels. Aronszajn [2] showed that for every positive definite kernel k ( · , · ) , i.e., its Gram matrix is always positive definite and symmetric, it uniquely determines an RKHS H and vice versa. This property is not only important for evaluating the property of MMD [43] but also required in optimizing MMD [32] by Frank-Wolfe algorithm.

4.2 Sampling Algorithm

In the previous research [36, 27, 50, 38, 58], sample selection is usually modelled by a nonconvex combinatorial optimization problem. In contrast, following the idea of [4], we regard min I m ∈ [ n ] MMD 2 k,α ( X I m , X n ) as a convex optimization problem by exploiting the convexity of α -MMD, and then solve it by a fast iterative minimization procedure derived from Frank-Wolfe algorithm (see Appendix A for derivation details):

where f I p ( x i ) = ∑ j ∈I p k ( x i , x j ) -αp ∑ n l =1 k ( x i , x l ) . As an extension of kernel herding [7], its corresponding algorithm (see Algorithm 2) is called Generalized Kernel Herding (GKH). Note that f I p ( x i ) is iteratively updated in Algorithm 2, which can save a lot of running time. However, GKH can select repeated samples that contradict the setting of sampling without replacement. To address this issue, we propose a modified iterating formula based on (7):

Algorithm 1 Generalized Kernel Herding without Replacement

Require: Data set X n = { x 1 , · · · , x n } ⊂ X ; the number of selected samples m < n ; a positive definite, characteristic and radial kernel k ( · , · ) on X × X ; trade-off parameter α ≤ 1 .

Ensure: Selected samples X I ∗ m = { x i ∗ 1 , · · · , x i ∗ m } .

1: For each x i ∈ X n calculate µ ( x i ) := ∑ n j =1 k ( x j , x i ) /n .

2: Set β 1 = 1 , S 0 = 0 , I = ∅ .

3: for p ∈ { 1 , · · · , m } do

4: i ∗ p ∈ arg min i ∈ [ n ] \I ∗ p S p -1 ( x i ) -αµ ( x i )

5: For all i ∈ [ n ] \I ∗ p , update S p ( x i ) = (1 -β p ) S p -1 ( x i ) + β p k ( x i ∗ p , x i )

6: I ∗ p +1 ←I ∗ p ∪ { i ∗ p } , p ← p +1 , set β p = 1 /p .

7: end for

which admits no repetitiveness in the selected samples. Its corresponding algorithm (see Algorithm 1) is thereby named as Generalized Kernel Herding without Replacement (GKHR), employed as the sampling algorithm for RDSS.

Computational complexity. Despite the time cost for calculating kernel functions, the computational complexity of GKHR is O ( mn ) , since in each iteration, the steps in lines 4 and 5 of Algorithm 2 respectively require O ( n ) computations. Note that GKH has the same order of computational complexity as GKHR.

5 Theoretical Analysis

5.1 Generalization Bounds

Recall the core-set approach in [36], i.e., for any h ∈ H ,

where T is the full labeled dataset and S ⊂ T is the core set, R ( h ) is the expected risk of h , ̂ R T ( h ) , ̂ R S ( h ) are empirical risk of h on T, S . The first term ̂ R S ( h ) is unknown before we label the selected samples, and the second term | R ( h ) -̂ R T ( h ) | can be upper bounded by the so-called generalization bounds [3, 64] which do not depend on the choice of core set. Therefore, to control the upper bound of R ( h ) , we only need to analyse the upper bound of the third term | ̂ R T ( h ) -̂ R S ( h ) | called core-set loss, which requires several mild assumptions. Shalit, et al. [37] derived a MMDtype upper bound for | ̂ R T ( h ) -̂ R S ( h ) | to estimate individual treatment effect, while our bound is generalized to a wider range of tasks.

Let H 1 = { h | h : X → Y} be a hypothesis set in which we are going to select a predictor and suppose that the labelled data T = { ( x i , y i ) } n i =1 are i.i.d. sampled from a random vector ( X,Y ) defined on X × Y . We firstly assume that H 1 is an RKHS, which is mild in machine learning theory [3, 5].

Assumption 5.1. H 1 is an RKHS associated with bounded positive definite kernel k 1 where the norm of any h ∈ H 1 is bounded by K h .

We further make RKHS assumptions on the functional space of E ( Y | X ) and Var( Y | X ) that are fundamental in the field of conditional distribution embedding [41, 43].

Assumption 5.2. There is an RKHS H 2 associated with bounded positive definite kernel k 2 such that E ( Y | X ) ∈ H 2 and the norm of any E ( Y | X ) is bounded by K m .

Assumption 5.3. There is an RKHS H 3 associated with bounded positive definite kernel k 3 such that Var( Y | X ) ∈ H 3 and the norm of any Var( Y | X ) is bounded by K s .

We next give a α -MMD-type upper bound for the core-set loss by the following theorem:

Theorem 5.4. Take k = k 2 1 + k 1 k 2 + k 3 , then under assumptions 1-3, for any selected samples S ⊂ T , there exists a positive constant K c such that the following inequality holds:

where 0 ≤ α ≤ 1 , 0 ≤ max x ∈X k ( x , x ) = K and X S , X T are projections of S, T on X .

Therefore, minimizing α -MMD can optimize the generalization bound for R ( h ) and benefit the generalizability of the trained model (predictor).

5.2 Finite-Sample-Error-Bound for GKHR

The concept of convergence does not apply to analyzing GKHR. With n fixed, GKHR iterates for at most n times and then returns X I ∗ n = X n . Consequently, we analyze the performance of GKHR by its finite-sample-error bound. Previous to that, we make an assumption on the mean of f I ∗ p over the full unlabeled dataset.

Assumption 5.5. For any I ∗ p returned by GKHR, 1 ≤ p ≤ m -1 , there exists p + 1 elements { x j l } p +1 l =1 in X n such that

When m is not relatively small, this assumption is rather unrealistic. Nevertheless, under our lowbudget setting, especially when m ≪ n , the assumption becomes an extension of the principle that "the minimum is never larger than the mean", which still probably makes sense. We can then show that the decaying rate for optimization error of GKHR can be upper bounded by O (log m/m ) :

Theorem 5.6. Let X I ∗ m be the samples selected by GKHR, under assumption 4, it holds that

where B = 2 K , 0 ≤ max x ∈X k ( x , x ) = K , C 2 α = (1 -α ) 2 K where K is defined in Lemma B.6.

6 Choice of Kernel and Hyperparameter Tuning

In this section, we make some suggestions for choosing the kernel and tuning the hyperparameter α .

Choice of kernel. Recall Remark 1 in Section 4.1, we only consider characteristic and positive definite kernels in RDSS. Since the Gaussian kernels are the most commonly used kernels in the field of machine learning and statistics [3, 15], we introduce Gaussian kernel as our choice, which is defined by k ( x , y ) = exp( -∥ x -y ∥ 2 2 /σ 2 ) . The bandwidth parameter σ is set to be the median distance between samples in the aggregate dataset [15], i.e., σ = Median( {∥ x -y ∥ 2 | x , y ∈ X n } ) , since the median is robust and also compromises between extreme cases.

Tuning trade-off hyperparameter α . According to Theorem 5.6 and Lemma B.3, by straightforward deduction we have

to upper bound the MMD between the selected samples and the full dataset under a low-budget setting. We can just set α ∈ [1 -1 √ m , 1) so that the upper bound of the MMD would not be larger than the one of α -MMD in the perspective of the order of magnitude.

7 Experiments

In this section, we first explain the implementation details of our method RDSS in Section 7.1. Next, we compare RDSS with other sampling methods by integrating them into two state-of-the-art (SOTA) SSL approaches (FlexMatch [63] and Freematch [53]) on five datasets (CIFAR-10/100, SVHN, STL-10 and ImageNet-1k) in Section 7.2. The details of the datasets, the visualization results and the computational complexity of different sampling methods are shown in Appendix D.2, D.3, and D.4, respectively. We also compare against various AL/SSAL approaches in Section 7.3. Lastly, we make quantitative analyses of the trade-off parameter α in Section 7.4.

7.1 Implementation Details of Our Method

First, we leverage the pre-trained image feature extraction capabilities of CLIP [33], a vision transformer architecture, to extract features. Subsequently, the [CLS] token features produced by the model's final output are employed for sample selection. During the sample selection phase, the Gaussian kernel function is chosen as the kernel method to compute the similarity of samples in an infinite-dimensional feature space. The value of σ for the Gaussian kernel function is set as explained in Section 6. To ensure diversity in the sampled data, we introduce a penalty factor given by α = 1 -1 √ m , where m denotes the number of selected samples. Concretely, we set m = { 40 , 250 , 4000 } for CIFAR-10, m = { 400 , 2500 , 10000 } for CIFAR-100, m = { 250 , 1000 } for SVHN, m = { 40 , 250 } for STL-10 and m = { 100000 } for ImageNet. Next, the selected samples are used for two SSL approaches, which are trained and evaluated on the datasets using the codebase Unified SSL Benchmark (USB) [52]. The optimizer for all experiments is standard stochastic gradient descent (SGD) with a momentum of 0 . 9 [44]. The initial learning rate is 0 . 03 with a learning rate decay of 0 . 0005 . We use ResNet-50 [16] for the ImageNet experiment and Wide ResNet-28-2 [62] for other datasets. Finally, we evaluate the performance with the Top-1 classification accuracy metric on the test set. Experiments are run on 8*NVIDIA Tesla A100 (40 GB) and 2*Intel 6248R 24-Core Processor. We average our results over five independent runs.

7.2 Comparison with Other Sampling Methods

Main results We apply RDSS on Flexmatch and Freematch to compare with the following three baselines and two SOTA methods in SSL under different annotation budget settings. The baselines conclude Stratified , Random and k -Means , while the two SOTA methods are USL [50] and ActiveFT [57]. The results are shown on Table 1 from which we have several observations: (1) Our proposed RDSS achieves the highest accuracy, outperforming other sampling methods, which underscores the effectiveness of our approach; (2) USL attains suboptimal results under most budget settings yet exhibits a significant gap compared to RDSS, particularly under severely constrained ones. For instance, FreeMatch achieves a 4 . 95% rise on the STL-10 with a budget of 40 ; (3) In most experiments, RDSS either approaches or surpasses the performance of stratified sampling, especially on SVHN and STL-10. However, the stratified sampling method is practically infeasible given that the category labels of the data are not known a priori.

Results on ImageNet We also compare the second-best method USL with RDSS on ImageNet. Following the settings of FreeMatch [53], we select 100k samples for annotation. FreeMatch, using RDSS and USL as sampling methods, achieves 58 . 24% and 56 . 86% accuracy, respectively, demonstrating a substantial enhancement in the performance of our method over the USL approach.

Table 1: Comparison with other sampling methods. Due to stratified sampling limitations, the results are marked in grey. Top and second-best performances are bolded and underlined, respectively, excluding stratified sampling. Metrics represent mean accuracy and standard deviation over five independent runs.

7.3 Comparison with AL/SSAL Approaches

First, we compare RDSS against various traditional AL approaches on CIFAR-10/100. AL approaches conclude CoreSet [36], VAAL [39], LearnLoss [60] and MCDAL [8]. For a fair comparison, we

exclusively use samples selected by RDSS for supervised learning compared to other AL approaches, considering that AL relies solely on labelled samples for supervised learning. The implementation details are shown in Appendix D.5. The experimental results are presented in Table 2, from which we observe that RDSS achieves the highest accuracy under almost all budget settings when relying solely on labelled data for supervised learning, with notable improvements on CIFAR-100.

Second, we compare RDSS with sampling methods used in SSAL when applied to the same SSL framework ( i.e., FlexMatch or FreeMatch) on CIFAR-10. The sampling methods conclude CoreSetSSL [36], MMA [42], CBSSAL [12], and TOD-Semi [17]. In detail, we tune recent SSAL approaches with their public implementations and run experiments under an extremely low-budget setting, i.e., 40 samples in a 20-random-and-20-selected setting. Table 3 illustrates that the performance of most SSAL approaches falls below that of random sampling methods under extremely low-budget settings. This inefficiency stems from the dependency of sample selection on model performance within the SSAL framework, which struggles when the model is weak. Our model-free method, in contrast, selects samples before training, avoiding these pitfalls.

Third, we directly compare RDSS with the above AL/SSAL approaches when applied to SSL, which may better reflect the paradigm differences. The experimental results and analysis are in the Appendix D.6.

7.4 Trade-off Parameter α

We analyze the effect of different α with Freematch on CIFAR-10/100. The results are presented in Table 4, from which we have several observations: (1) Our proposed RDSS achieves the highest accuracy under all budget conditions, surpassing those that employ a fixed value; (2) The α that achieve the best or the second best performance are within the interval we set, which is in line with our theoretical derivation in Section 6; (3) The experimental outcomes exhibit varying degrees of reduction compared to our approach when the representativeness or diversity term is removed.

8 Conclusion

In this work, we propose a model-free sampling method, RDSS, to select a subset from unlabeled data for annotation in SSL. The primary innovation of our approach lies in the introduction of α -MMD, designed to evaluate the representativeness and diversity of selected samples. Under a low-budget setting, we develop a fast and efficient algorithm GKHR for this problem using the Frank-Wolfe algorithm. Both theoretical analyses and empirical experiments demonstrate the effectiveness of RDSS. In future research, we would like to apply our methodology to scenarios where labelling is cost-prohibitive, such as in the medical domain.

References

M. Ai, J. Yu, H. Zhang, and H. Wang. Optimal subsampling algorithms for big data regressions. Statistica Sinica , 31(2):749-772, 2021.

N. Aronszajn. Theory of reproducing kernels. Transactions of the American mathematical society , 68(3): 337-404, 1950.

F. Bach. Learning theory from first principles. Draft of a book, version of Sept , 6:2021, 2021.

F. Bach, S. Lacoste-Julien, and G. Obozinski. On the equivalence between herding and conditional gradient algorithms. arXiv preprint arXiv:1203.4523 , 2012.

A. Bietti and J. Mairal. Group invariance, stability to deformations, and complexity of deep convolutional representations. The Journal of Machine Learning Research , 20(1):876-924, 2019.

Y.-C. Chan, M. Li, and S. Oymak. On the marginal benefit of active learning: Does self-supervision eat its cake? In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 3455-3459. IEEE, 2021.

Y. Chen, M. Welling, and A. Smola. Super-samples from kernel herding. arXiv preprint arXiv:1203.3472 , 2012.

J. W. Cho, D.-J. Kim, Y. Jung, and I. S. Kweon. Mcdal: Maximum classifier discrepancy for active learning. IEEE transactions on neural networks and learning systems , 2022.

A. Coates, A. Ng, and H. Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 215-223. JMLR Workshop and Conference Proceedings, 2011.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

A. Freytag, E. Rodner, and J. Denzler. Selecting influential examples: Active learning with expected model output changes. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part IV 13 , pages 562-577. Springer, 2014.

M. Gao, Z. Zhang, G. Yu, S. Ö. Arık, L. S. Davis, and T. Pfister. Consistency-based semi-supervised active learning: Towards minimizing labeling cost. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X 16 , pages 510-526. Springer, 2020.

S. Graf and H. Luschgy. Foundations of quantization for probability distributions . Springer, 2007.

A. Gretton, K. Borgwardt, M. Rasch, B. Schölkopf, and A. Smola. A kernel method for the two-sampleproblem. Advances in neural information processing systems , 19, 2006.

A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. The Journal of Machine Learning Research , 13(1):723-773, 2012.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

S. Huang, T. Wang, H. Xiong, J. Huan, and D. Dou. Semi-supervised active learning with temporal output discrepancy. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3447-3456, 2021.

A. J. Joshi, F. Porikli, and N. Papanikolopoulos. Multi-class active learning for image classification. In 2009 ieee conference on computer vision and pattern recognition , pages 2372-2379. IEEE, 2009.

A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.

S. Laine and T. Aila. Temporal ensembling for semi-supervised learning. In International Conference on Learning Representations , 2016.

D.-H. Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML , volume 3, page 896. Atlanta, 2013.

D. D. Lewis and J. Catlett. Heterogeneous uncertainty sampling for supervised learning. In Machine learning proceedings 1994 , pages 148-156. Elsevier, 1994.

M. Li, R. Wu, H. Liu, J. Yu, X. Yang, B. Han, and T. Liu. Instant: Semi-supervised learning with instance-dependent thresholds. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Z. Liu, H. Ding, H. Zhong, W. Li, J. Dai, and C. He. Influence selection for active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9274-9283, 2021.

S. L. Lohr. Sampling: design and analysis . Chapman and Hall/CRC, 2021.

W. Luo, A. Schwing, and R. Urtasun. Latent structured active learning. Advances in Neural Information Processing Systems , 26, 2013.

R. Mahmood, S. Fidler, and M. T. Law. Low budget active learning via wasserstein distance: An integer programming approach. arXiv preprint arXiv:2106.02968 , 2021.

S. Mak and V. R. Joseph. Support points. The Annals of Statistics , 46(6A):2562-2592, 2018.

K. Muandet, K. Fukumizu, B. Sriperumbudur, B. Schölkopf, et al. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends® in Machine Learning , 10(1-2):1-141, 2017.

Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. 2011.

V. I. Paulsen and M. Raghupathi. An introduction to the theory of reproducing kernel Hilbert spaces , volume 152. Cambridge university press, 2016.

L. Pronzato. Performance analysis of greedy algorithms for minimising a maximum mean discrepancy. arXiv preprint arXiv:2101.07564 , 2021.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

M. Sajjadi, M. Javanmardi, and T. Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. Advances in neural information processing systems , 29, 2016.

H. Schmutz, O. Humbert, and P.-A. Mattei. Don't fear the unlabelled: safe semi-supervised learning via debiasing. In The Eleventh International Conference on Learning Representations , 2022.

O. Sener and S. Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations , 2018.

U. Shalit, F. D. Johansson, and D. Sontag. Estimating individual treatment effect: generalization bounds and algorithms. In International conference on machine learning , pages 3076-3085. PMLR, 2017.

Q. Shao, K. Zhang, B. Du, Z. Li, Y. Wu, Q. Chen, J. Wu, and J. Chen. Comprehensive subset selection for ct volume compression to improve pulmonary disease screening efficiency. In Artificial Intelligence and Data Science for Healthcare: Bridging Data-Centric AI and People-Centric Healthcare , 2024.

S. Sinha, S. Ebrahimi, and T. Darrell. Variational adversarial active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5972-5981, 2019.

K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems , 33:596-608, 2020.

L. Song, J. Huang, A. Smola, and K. Fukumizu. Hilbert space embeddings of conditional distributions with applications to dynamical systems. In Proceedings of the 26th Annual International Conference on Machine Learning , pages 961-968, 2009.

S. Song, D. Berthelot, and A. Rostamizadeh. Combining mixmatch and active learning for better accuracy with fewer labels. arXiv preprint arXiv:1912.00594 , 2019.

B. K. Sriperumbudur, K. Fukumizu, A. Gretton, B. Schölkopf, and G. R. Lanckriet. On the empirical estimation of integral probability metrics. 2012.

I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning , pages 1139-1147. PMLR, 2013.

A. Tarvainen and H. Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems , 30, 2017.

S. K. Thompson. Sampling , volume 755. John Wiley & Sons, 2012.

S. Tong and D. Koller. Support vector machine active learning with applications to text classification. Journal of machine learning research , 2(Nov):45-66, 2001.

M. J. Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press, 2019.

K. Wang, D. Zhang, Y. Li, R. Zhang, and L. Lin. Cost-effective active learning for deep image classification. IEEE Transactions on Circuits and Systems for Video Technology , 27(12):2591-2600, 2016.

X. Wang, L. Lian, and S. X. Yu. Unsupervised selective labeling for more effective semi-supervised learning. In European Conference on Computer Vision , pages 427-445. Springer, 2022.

X. Wang, Z. Wu, L. Lian, and S. X. Yu. Debiased learning from naturally imbalanced pseudo-labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1464714657, 2022.

Y. Wang, H. Chen, Y. Fan, W. Sun, R. Tao, W. Hou, R. Wang, L. Yang, Z. Zhou, L.-Z. Guo, et al. Usb: A unified semi-supervised learning benchmark for classification. Advances in Neural Information Processing Systems , 35:3938-3961, 2022.

Y. Wang, H. Chen, Q. Heng, W. Hou, Y. Fan, Z. Wu, J. Wang, M. Savvides, T. Shinozaki, B. Raj, et al. Freematch: Self-adaptive thresholding for semi-supervised learning. arXiv preprint arXiv:2205.07246 , 2022.

X. Wu, Y. Huo, H. Ren, and C. Zou. Optimal subsampling via predictive inference. Journal of the American Statistical Association , (just-accepted):1-29, 2023.

Q. Xie, Z. Dai, E. Hovy, T. Luong, and Q. Le. Unsupervised data augmentation for consistency training. Advances in neural information processing systems , 33:6256-6268, 2020.

Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10687-10698, 2020.

Y. Xie, H. Lu, J. Yan, X. Yang, M. Tomizuka, and W. Zhan. Active finetuning: Exploiting annotation budget in the pretraining-finetuning paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23715-23724, 2023.

Y. Xu, D. Zhang, S. Zhang, S. Wu, Z. Feng, and G. Chen. Predictive and near-optimal sampling for view materialization in video databases. Proceedings of the ACM on Management of Data , 2(1):1-27, 2024.

L. Yang, Z. Zhao, L. Qi, Y. Qiao, Y. Shi, and H. Zhao. Shrinking class space for enhanced certainty in semi-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16187-16196, 2023.

D. Yoo and I. S. Kweon. Learning loss for active learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 93-102, 2019.

J. Yu, H. Wang, M. Ai, and H. Zhang. Optimal distributed subsampling for maximum quasi-likelihood estimators with massive data. Journal of the American Statistical Association , 117(537):265-276, 2022.

S. Zagoruyko and N. Komodakis. Wide residual networks. In Procedings of the British Machine Vision Conference 2016 . British Machine Vision Association, 2016.

B. Zhang, Y. Wang, W. Hou, H. Wu, J. Wang, M. Okumura, and T. Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems , 34:18408-18419, 2021.

H. Zhang and S. X. Chen. Concentration inequalities for statistical inference. Communications in Mathematical Research , 37(1):1-85, 2021.

J. Zhang, C. Meng, J. Yu, M. Zhang, W. Zhong, and P. Ma. An optimal transport approach for selecting a representative subsample with application in efficient kernel density estimation. Journal of Computational and Graphical Statistics , 32(1):329-339, 2023.

M. Zhang, Y. Zhou, Z. Zhou, and A. Zhang. Model-free subsampling method based on uniform designs. IEEE Transactions on Knowledge and Data Engineering , 2023.

A Algorithms

A.1 Derivation of Generalized Kernel Herding (GKH)

Proof. The proof technique is borrowed from [32]. Let us firstly define a weighted modification of α -MMD. For any w ∈ R n such that w ⊤ 1 = 1 , the weighted α -MMD is defined by

where K = [ k ( x i , x j )] 1 ≤ i,j ≤ n , K = 1 ⊤ K1 /n 2 , p = ( e ⊤ 1 K1 /n, · · · , e ⊤ n K1 /n ) , { e i } n i =1 is the set of standard basis of R n . It is obvious that for any I p ⊂ [ n ] ,

where ( w p ) i = 1 /p if i ∈ I p , and ( w p ) i = 0 if not. Therefore, weighted α -MMD is indeed a generalization of α -MMD. Let

we obtain the quadratic form expression of weighted α -MMD by MMD 2 k,α, X n ( w ) = w ⊤ K ∗ w , where K ∗ is strictly positive definite if the unlabeled data are pairwise different, w = w n and k is a characteristic kernel according to [32].

̸

̸

Recall our low-budget setting (so w = w n holds) and assumption for kernel, K ∗ is indeed a strictly positive definite matrix. Thus MMD 2 k,α, X n is a convex functional w.r.t. w , leading to the fact that min w ⊤ 1 =1 MMD 2 k,α, X n ( w ) can be solved by Frank-Wolfe algorithm. Then for 1 ≤ p < n ,

Let e i ∗ p = s p , under uniform step size in Frank-Wolfe algorithm, we have

as the update formula of Frank-Wolfe algorithm, which is equivalent to

then we can immediately derive the iterating formula in (7).

A.2 Pseudo Codes

Algorithm 2 Generalized Kernel Herding

Require: Data set X n = { x 1 , · · · , x n } ⊂ X ; the number of selected samples m < n ; a positive definite, characteristic and radial kernel k ( · , · ) on X × X ; trade-off parameter α ≤ 1 .

Ensure: selected samples X I ∗ m = { x i ∗ 1 , · · · , x i ∗ m } .

1: For each x i ∈ X n calculate µ ( x i ) := ∑ n j =1 k ( x j , x i ) /n .

2: Set β 1 = 1 , S 0 = 0 , I = ∅ .

3: for p ∈ { 1 , · · · , m } do

4: i ∗ p ∈ arg min i ∈ [ n ] S p -1 ( x i ) -αµ ( x i )

5: For all i ∈ [ n ] , update S p ( x i ) = (1 -β p ) S p -1 ( x i ) + β p k ( x i ∗ p , x i )

6: I ∗ p +1 ←I ∗ p ∪ { i ∗ p } , p ← p +1 , set β p = 1 /p .

7: end for

B Technical Lemmas

Lemma B.1 (Lemma 2 [32]) . Let ( t k ) k and ( α k ) k be two real positive sequences and A be a strictly positive real. If t k satisfies

with α k = 1 /k for all k , then t k < A (2 + log k ) / ( k +1) for all k > 1 .

Lemma B.2. The selected samples X I ∗ m generated by GKH (Algorithm 2) satisfies

where B = 2 K , 0 ≤ max x ∈X k ( x , x ) ≤ K , M 2 α is defined by

Proof. Following the notations in Appendix A, let p α = α p , we could straightly follow the proof for finite-sample-size error bound of kernel herding with predefined step sizes given by [32] to derive Lemma B.2, without any other technique. The detailed proof is omitted.

Lemma B.3. Let H be an RKHS over X associated with positive definite kernel k , and 0 ≤ max x ∈X k ( x , x ) ≤ K . Let X m = { x i } i m =1 , Y n = { y j } j m =1 , x i , y j ∈ X . Then for any α ≤ 1 ,

Proof.

Lemma B.4 (Proposition 12.31 [48]) . Suppose that H 1 and H 2 are reproducing kernel Hilbert spaces of real-valued functions with domains X 1 and X 2 , and equipped with kernels k 1 and k 2 , respectively. Then the tensor product space H = H 1 ⊗H 2 is an RKHS of real-valued functions with domain X 1 ×X 2 , and with kernel function

Lemma B.5 (Theorem 5.7 [31]) . Let f ∈ H 1 and g ∈ H 2 , where H 1 , H 2 be two RKHS containing real-valued functions on X , which is associated with positive definite kernel k 1 , k 2 and canonical feature map ϕ 1 , ϕ 2 , then for any x ∈ X ,

where

and ϕ 1 + ϕ 2 is the canonical feature map of H 1 + H 2 . Furthermore,

Lemma B.6. For any unlabeled dataset X n ⊂ X and any subset X I m ,

Lemma B.6 is directly derived from the definition of α -MMD.

C Proof of Theorems

Proof for Theorem 5.4. The proof borrows the technique introduced in [37] for decomposing the expected risk of hypotheses.

Firstly, let us denote that H 4 = H 1 ⊗H 1 + H 1 ⊗H 2 + H 3 , with kernel k 4 = k 2 1 + k 1 k 2 + k 3 and canonical feature map ϕ 4 = ϕ 1 ⊗ ϕ 1 + ϕ 1 ⊗ ϕ 2 + ϕ 3 .

Under the assumptions in Theorem 5.4, according to Theorem 4 in [41], we have for any x ∈ X ,

where ϕ 1 , ϕ 2 , ϕ 3 are canonical feature maps in H 1 , H 2 , H 3 . Denote that m = E [ Y | X ] and s = Var( Y | X ) . Now by definition,

where

where the fourth equality holds by Lemma B.4 and the last equality holds by Lemma B.5, then f ∈ H 4 , and

where the second inequality holds by Lemma B.5. Therefore, let β = 1 / ( K 2 h +2 K h K m + K s ) we have ∥ βf ∥ H 4 = β ∥ f ∥ H 4 ≤ 1 . Then

where P T denotes the empirical distribution constructed by X T , so does P S . Recall Lemma B.3, we have Theorem 5.4.

Proof for Theorem 5.6. Following the notations in Appendix A, we further define

Let p α = α p , we have ( p α -K ̂ w ) ∝ 1 . Define

where ̂ g ( w ) = ( w -̂ w ) ⊤ K ( w -̂ w ) . The related details for proving the equality are omitted, since they are completely given by the proof of alternative expression of MMD in Pronzato [32]. By the convexity of ̂ g ( · ) , for j = arg min i ∈ [ n ] \I ∗ p f I ∗ p ( x i ) ,

where the second inequality holds with the assumption in Theorem 5.6

therefore, we have for B = 2 K ,

where w p +1 = p w p / ( p +1)+ e j / ( p +1) , and obviously B upper bounds ( e j -w p ) ⊤ K ( e j -w p ) . Since α ≤ 1 , it holds from Lemma B.6 that

therefore by Lemma B.1, we have

D Additional Experimental Details and Results

D.1 Supplementary Numerical Experiments on GKHR

Consider the fact that GKH is a convergent algorithm (Lemma B.2) and the finite-sample-size error bound (10) holds without any assumption on the data, we conduct some numerical experiments to empirically compare GKHR with GKH on datasets generated by four different distributions on R 2 . Firstly, we define four distributions on R 2 :

Gaussian mixture model 1 which consists of four Gaussian distributions G 1 , G 2 , G 3 , G 4 with mixture weights [0 . 95 , 0 . 01 , 0 . 02 , 0 . 02] ,

Gaussian mixture model 2 which consists of four Gaussian distributions G 1 , G 2 , G 3 , G 4 with mixture weights [0 . 3 , 0 . 2 , 0 . 15 , 0 . 35] ,

Uniform distribution 1 which consists of a uniform distribution defined in a circle with radius 0 . 5 , and a uniform distribution defined in a annulus with inner radius 4 and outer radius 6 ,

Uniform distribution 2 defined on [ -10 , 10] 2 .

where

(a) n=1000

(b) n=3000

(c) n=10000

(d) n=30000

0.01

0.02

0.01

0.01

D

D

0.02

0.01

0.03

0.02

100

m

160

80

100

m

160

180

Uniform Distribution 1

Uniform Distribution 2

0.02

0.02

0.01

10'0

D

D

0.01

L0°0

-0.02

60

08

100120

140160

180200

0.03

60

80

100120

180200

m

0.01

0.01

0.005

0.005

D

D

0.005

0.005

0.01

L0'0

100

200

300

400

500

009

-0.015

100

200

300

400

500

600

m

0.01

Uniform Distribution 1

0.015

Uniform Distribution 2

0.005

0.01

D

D

0.005

0.005

0.01

0.005

0.015

100

200

300

400

500

600

0.01

100

200

300

400

500

m

m

GaussianMixture Model1

10

SL00

0.01

0.005

D

0.005

200

400

600

800

10001200

14001600

18002000

0.01

200

400

600

800

10001200

1400 160018002000

Uniform Distribution 1

Uniform Distribution 2

10°°

D

100

1000

1200

200

400

600

008

1000

1200

1400

1600

m

m

0.025

GaussianMixture Model 1

0.02

0.015

D0.01

D

0.005

0

0.005

1000

2000

3000

m

4000

5000

2000

3000

m

4000

5000

Uniform Distribution 1

Uniform Distribution 2

10°d

D

1000

2000

3000

4000

5000

6000

1000

2000

3000

4000

0009

6000

m

m

To consistently evaluate the performance gap between GKHR and GKH at the same order of magnitude, we propose the following criterion

where D 1 = MMD 2 k,α ( X (1) I ∗ m , X n ) , D 2 = MMD 2 k,α ( X (2) I ∗ m , X n ) , X (1) I m is the selected samples from

GKHR and X (2) I m is the selected samples from GKH. Positive value of D implies that GKH outperforms GKHR, and negative values of D implies that GKHR outperforms GKH. Large absolute value of D shows large performance gap.

The experiments are conducted as follows. We generate 1000,3000,10000,30000 random samples from the four distributions separately, then use GKHR and GKH for sample selection under the low-budget setting, i.e., m/n ≤ 0 . 2 . The α is set by m/n . We report the results over ten independent runs in Figure 2, which shows that although the performance gap tends to grow as m grows, when m is relatively small, the performance of GKHR is similar to that of GKH. Therefore, under the low-budget setting, GKHR and GKH have similar performance on minimizing α -MMD over various type of distributions, which convinces us that GKHR could work well in the sample selection task.

D.2 Datasets

For experiments, we choose five common datasets: CIFAR-10/100, SVHN, STL-10 and ImageNet. CIFAR-10 and CIFAR-100 contain 60,000 images with 10 and 100 categories, respectively, among which 50,000 images are for training, and 10,000 images are for testing; SVHN contains 73,257 images for training and 26,032 images for testing; STL-10 contains 5,000 images for training, 8,000 images for testing and 100,000 unlabeled images as extra training data. ImageNet spans 1,000 object classes and contains 1,281,167 training and 100,000 test images. The training sets of the above datasets are considered as the unlabeled dataset for sample selection.

D.3 Visualization of Selected Samples

To offer a more intuitive comparison between various sampling methods, we visualized samples chosen by stratified, random, k -Means, USL, ActiveFT and RDSS (ours). We generate 5000 samples from a Gaussian mixture model defined on R 2 with 10 components and uniform mixture weights. One hundred samples are selected from the entire dataset using different sampling methods. The visualisation results in Figure 3 indicate that our selected samples distribute more similarly with the entire dataset than other counterparts.

D.4 Computational Complexity and Running Time

We compute the time complexity of various sampling methods and recorded the time required to select 400 samples on the CIFAR-100 dataset for each method. The results are presented in Table 5, where m represents the annotation budget, n denotes the total number of samples, and T indicates the number of iterations. The sampling time was obtained by averaging the duration of three independent runs of the sampling code on an idle server without any workload. As illustrated by the results, the sampling efficiency of our method surpasses that of all other methods except for random and stratified sampling. This discrepancy is likely because the execution time of other algorithms is affected by the number of iterations T .

(a) Stratified

(b) Random

(f) RDSS (ours)

(d) USL

(c) K-Means

D.5 Implementation Details of Supervised Learning Experiments

We use ResNet-18 [16] as the classification model for all AL approaches and our method. Specifically, Wetrain the models for 300 epochs using SGD optimizer (initial learning rate= 0 . 1 , weight decay= 5 e -4 , momentum= 0 . 9 ) with batch size 128 . Finally, we evaluate the performance with the Top-1 classification accuracy metric on the test set.

D.6 Direct Comparison with AL/SSAL

The comparative results with AL/SSAL approaches are shown in Figure 4 and Figure 5, respectively. The specific values corresponding to the comparative results in the above two figures are shown in Table 6. And the above results are from [8], [12] and [17].

According to the results, we have several observations: (1) AL approaches often necessitate significantly larger labelling budgets, exceeding RDSS by 125 or more on CIFAR-10. This is primarily because AL paradigms are solely dependent on labelled samples not only for classification but also for feature learning. (2) SSAL and our methods leverage unlabeled samples, surpassing traditional AL approaches. However, this may not directly reflect the advantages of RDSS, as such performance enhancements could be inherently attributed to the SSL paradigm itself. Nonetheless, these experimental outcomes offer insightful implications: SSL may represent a more promising paradigm under scenarios with limited annotation budgets.

0

2000

4000

6000

8000

10000

Budget

80.0

82.5

85.0

87.5

90.0

92.5

95.0

CoreSet

VAAL

LearnLoss

MCDAL

CoreSetSSL

CBSSAL

TOD-Semi

FlexMatch+RDSS (Ours)

FreeMatch+RDSS (Ours)

0

2000

4000

6000

8000

10000

Budget

35

40

45

50

55

60

65

70

75

CoreSet

VAAL

LearnLoss

MCDAL

CoreSetSSL

CBSSAL

TOD-Semi

FlexMatch+RDSS (Ours)

FreeMatch+RDSS (Ours)

E Limitation

The choice of α depends on the number of full unlabeled data points, independent of the information on the shape of data distribution. This may lead to a loss of effectiveness of RDSS on those datasets with complicated distribution structures. However, it outperforms fixed-ratio approaches on the datasets under different budget settings.

F Potential Societal Impact

Positive societal impact. Our method ensures the representativeness and diversity of the selected samples and significantly improves the performance of SSL methods, especially under low-budget settings. This reduces the cost and time of data annotation and is particularly beneficial for resourceconstrained research and development environments, such as medical image analysis.

Negative societal impact. When selecting representative data for analysis and annotation, the processing of sensitive data may be involved, increasing the risk of data leakage, especially in sensitive fields such as medical care and finance. It is worth noting that most algorithms applied in these sensitive areas are subject to this risk.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We summarize our contributions in the last paragraph of Section 1. The RDSS framework, including the quantification method and sample algorithm, is illustrated in Section 4. Theoretical analysis on RDSS is presented in Section 5. Section 6 suggests the choice for kernel and tuning parameters. The comparison results with other methods are shown in Section 7.2 and Section 7.3. And we analyze the effect of different α in Section 7.4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our work in Appendix E.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: For the generalization bound in Section 5.1, the required assumptions are Assumption 5.1, 5.2 and 5.3, the proof is given by the Proof of Theorem 5.4 in Appendix C. For the finite-sample-error bound in Section 5.2, the required assumption is Assumption 5.5, the proof is given by the Proof of Theorem 5.6 in Appendix C. Other technical lemmas and their proofs or references are presented in Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We disclose the implementation details of the main experiments for reproduction in Section 7.1 and Appendix D.5. We submit the code of our proposed method as supplemental material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The data used for experiments are all publicly available datasets, as referenced in the penultimate paragraph of Section 1. And we submit the code as supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify the implementation details of main experiments in Section 7.1 and Appendix D.5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Each result of main experiments shows mean accuracy and standard deviation over five independent runs in Section 7.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We illustrate the compute resources utilized by the experimental implementation in Section 7.1 and calculate the time of execution in Appendix D.4

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the NeurIPS Code of Ethics and ensure that the research conducted in the paper conforms with the NeurIPS Code of Ethics in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the potential societal impacts of our work in Appendix F.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the original paper that produced the assets used in the paper and ensure that the use of these assets complies with the relevant licenses.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We submit our code and the corresponding documentation as supplementary materials.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper suggests a new sampling method for the labeled set of semi-supervised learning. This sampling method, termed RDSS, selects a set of examples that is both representative of the data, and diverse. The paper shows that using such a sampling function improves both freematch and flexmatch, and compares it against other sampling methods, and methods from AL and SSAL.\n\nSTRENGTHS:\nThe idea of the paper is good, and is well supported by theory. The experimental setup does convince me that the suggested method is better than random sampling when picking the labeled set of SSL. However, a better comparison to previous works is required, see the Weaknesses section.\n\nClarity: The paper is clearly written, the idea is well presented and intuitive, and the paper is easy to read and follow.\n\nWEAKNESSES:\nSome of the claims made by paper already appeared in previous art. Specifically, [1] showed that \"traditional\" AL methods do not pick bad labeled sets for SSL when compared to random sampling. [2] showed that when the labeled set is particularly small, instead of traditional AL techniques, one should focus on labeling examples that are more typical and diverse, showing that such a method can drastically improve both AL and sampling techniques for SSL. [3] presented sampling strategy, showing that picking examples that are representative and diverse examples for the labeled set of  SSL improves it by a big margin in low-budget scenarios.\n\nThe proposed manuscript does not reference or compare to any of these works. This affects both the novelty, significance and quality of the proposed method: the novelty is somewhat more limited, as many of the ideas overlap with existing works. The significance of this work is impacted, as while the problem at hand is important, it is unclear if the presented ideas pose significant advancement over the existing methods, and the quality is diminished, as a lot of comparisons are missing in the experimental setup.\n\nSpecifically, any low-budget strategy could be potentially applied to SSL as well, so those methods should be compared against as well. See for example [4], [5].\n\nAdditionally, the vice-versa argument should also hold -- if AL methods can be applied in this case, this method can be used as a method for picking labeled examples for active learning purposes and should be tested as such, as the literature in AL is much broader than the literature of picking the labeled set in SSL, which can provide a much wider context for the given work.\n\nIn addition, the framing of the paper is a bit unclear to me. I think the paper could benefit from explaining use cases in which one has the option to pick in advance the labeled set for SSL, which is not already covered by AL use cases.\n\n-------\n\n[1] Mittal, Sudhanshu, et al. Parting with illusions about deep active learning. (2019).\n\n[2] Hacohen, Guy et al. Active learning on a budget: Opposite strategies suit high and low budgets. (2022).\n\n[3] Yehuda, Ofer, et al. \"Active learning through a covering lens.\" (2022).\n\n[4] Mahmood, Rafid, et al. \"Low budget active learning via wasserstein distance: An integer programming approach.\" (2021).\n\n[5] Wen, Ziting, et al. \"NTKCPL: Active Learning on Top of Self-Supervised Model by Estimating True Coverage.\" (2023).\n\nQUESTIONS:\nCan you please elaborate on how the proposed method differs from the idea suggested in [2]?\n\nHow is the problem of selecting the labeled set of SSL different from the problem setting of active learning?\n\nLIMITATIONS:\nnot relevant", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Representative and Diverse Sample Selection approach (RDSS) that utilizes a modified Frank-Wolfe algorithm to minimize a novel α-Maximum Mean Discrepancy (α-MMD) criterion, aiming to select a representative and diverse subset from unlabeled data for annotation. Experimental results demonstrate that RDSS consistently improves the performance of several popular semi-supervised learning frameworks and outperforms state-of-the-art sample selection methods used in Active Learning (AL) and Semi-Supervised Active Learning (SSAL), even under constrained annotation budgets.\n\nSTRENGTHS:\n1.This paper is in Well-written, logically organized, and smoothly expressed.\n2. The presented results demonstrate the effectiveness of the proposed approach.\n\nWEAKNESSES:\n1. The author conducted tests on two baseline methods(FlexMatch [58] and Freematch [50]), but neither of them represents the current state-of-the-art.\n2. Some details of the experiments are unclear, such as in Table 3.\n\nQUESTIONS:\n1. The definition and usage of variable X in the article are inconsistent. \n2. Is Y in the section starting from line 141 representing the point as X? If so, I suggest using notations like Xi, Xj for clarity.\n3. In Chapter 6, the determination of the kernel and parameters seems arbitrary. Could you provide some proofs, theories, or experiments to justify them?\n4. The SOTA models selected by the author in the experiments are somewhat outdated. It is recommended to include some more updated methods.\n5. In the experiment section, the author mentions the limitations of stratified sampling. What do these limitations refer to? Why was this method excluded? From the results, it seems that stratified sampling outperforms the proposed method in several settings.\n6. In Table 2, what sampling method did the other comparison methods adopt?\n7. What dataset does Table 3 represent? What is the data distribution?\n8. The author should objectively evaluate their method, including its limitations.\n\nLIMITATIONS:\nas Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper proposes a new sample selection method, RDSS, for the SSL task. RDSS considers both the representativeness and diversity of the selected sample and achieves state-of-the-art performance. This is achieved by the proposed α-MMD criterion and an efficient optimization algorithm GKHR.

STRENGTHS:
1. RDSS considers both representativeness and diversity of samples, which is a convincing strategy, and the experimental results also demonstrate the effectiveness of this motivation.
2. Sufficient theoretical analysis and experimental comparisons are conducted to demonstrate the effectiveness of the proposed method.

WEAKNESSES:
I would like to see images of the actual selected samples and visualizations of the feature distribution to demonstrate that RDSS indeed balances the representativeness and diversity.

QUESTIONS:
Please refer to the weakness.

LIMITATIONS:
Please refer to the weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nChoice of the labeled set in the semi supervised learning is critical for the final performance of the model. This problem can also be looked as AL with SSL, or single shot AL with SSL (in other words similar to experimental design). This works provides a way to select the seed set which is representative, as well as diverse. The problem is reduced to minimizing MMD and similarity score of the selected examples. The paper finally proposes a greedy algorithm, and compare the proposed method against various subset selection baselines, and AL.\n\nSTRENGTHS:\nI like the motivation of the problem and a neat theoretical derivation of the objective, and the provided theoretical analysis. Paper was also easy to follow and experiments are compelling.\n\nWEAKNESSES:\n- From a purely combinatorial point of view, I think that the final objective is supermodular in nature. Given the vast literature on submodular/supermodular functions, is it not possible to get an algorithm purely from that standpoint? If so, how different would it be from the proposed one? \n\n- Can one derive things such as leverage scores to detect the outlier-ness of a given point (or any other score)? If so, then couldn't one use something such as diversity - outlier score (or add a score that models likelihood) , with diversity such as Facility location function, and optimize the final objective using greedy? \n\n- In experiments I believe one of the strong baselines such as facility location function is missing. Facility Location has a rich history and have been used in several instances in Active Learning ([1, 2, 3, 4]). I believe authors can add a small discussion on FL and add that baseline. Furthermore, other diversity based approaches have also been considered in the past [5]\n\n- Now a days a lot of focus is also for doing finetuning of existing CLIP models [3]. I'd appreciate one experiment on fine-tuning the CLIP models using the proposed method. \n\n\nReferences\n- [1] Submodularity in machine learning and artificial intelligence\n- [2] An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models\n- [3] LabelBench: A Comprehensive Framework for Benchmarking Adaptive Label-Efficient Learning\n- [4] Deep Submodular Peripteral networks\n- [5] GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning\n\nQUESTIONS:\nRefer to the weaknesses.\n\nLIMITATIONS:\nRefer to the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}