RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learning

1 Introduction . Deep learning algorithms have had great success over the past few years , often achieving human or superhuman performance in various tasks like computer vision [ 10 ] , speech recognition [ 18 ] , natural language processing [ 5 ] , and video games [ 45 ] . One of the significant factors attributing to the recent success of deep learning is the availability of large amounts of labeled data [ 55 ] . However , creating large labeled datasets is often time-consuming and expensive in terms of costs . Moreover , some domains like medical imaging require a domain expert for labeling , making it nearly impossible to create a large labeled set . In order to reduce the dependency on the availability of labeled data , semi-supervised learning ( SSL ) algorithms [ 7 ] were proposed to train models using large amounts of unlabeled data along with the available labeled data . Recent works [ 42 , 56 , 4 , 53 ] show that semi-supervised learning algorithms can achieve similar performance to standard supervised learning using significantly fewer labeled data instances . 1coResets for EfficienT and Robust semI-supErVised lEarning 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , the current SOTA SSL algorithms are compute-intensive with large training times . For example , from our personal experience , training a WideResNet model [ 60 ] on a CIFAR10 [ 27 ] dataset with 4000 labels using the SOTA FixMatch algorithm [ 53 ] for 500000 iterations takes around four days on a single RTX2080Ti GPU . This also implies increased energy consumption and an associated carbon footprint [ 54 ] . Furthermore , it is common to tune these SSL algorithms over a large set of hyper-parameters , which means that the training needs to be done hundreds and sometimes thousands of times . For example , [ 44 ] performed hyperparameter tuning by running 1000 trails of Gaussian Process-based Blackbox optimization [ 16 ] for each SSL algorithm ( which runs for 500000 iterations ) . This process implies significantly higher experimental turnaround times , energy consumption , and CO2 emissions . Furthermore , this is not something that can be done at most universities and smaller companies . The first problem we try to address in this work is : Can we efficiently train a semi-supervised learning model on coresets of unlabeled data to achieve faster convergence and reduction in training time ? Despite demonstrating encouraging results on standard and clean datasets , current SSL algorithms perform poorly when OOD data or class imbalance is present in the unlabeled set [ 44 , 8 ] . This performance degradation can be attributed to the fact that the current SSL algorithms assume that both the labeled set and unlabeled set are sampled from the same distribution . A visualization of OOD data and class imbalance in the unlabeled set is shown in Figure 1 . Several recent works [ 59 , 8 , 17 ] were proposed to mitigate the effect of OOD in unlabeled data , in turn improving the performance of SSL algorithms . However , the current SOTA robust SSL method [ 17 ] is 3X slower than the standard SSL algorithms , further increasing the training times , energy costs , and CO2 emissions . The second problem we try to address in this work is : In the case where OOD data or class imbalance exists in the unlabeled set , can we robustly train an SSL model on coresets of unlabeled data to achieve similar performance to existing robust SSL methods while being significantly faster ? To this end , we propose RETRIEVE , a coreset selection framework that enables faster convergence and robust training of SSL algorithms . RETRIEVE selects coreset of the unlabeled data resulting in minimum labeled set loss when trained upon in a semi-supervised manner . Intuitively , RETRIEVE tries to achieve faster convergence by selecting data instances from the unlabeled set whose gradients are aligned with the labeled set gradients . Furthermore , RETRIEVE also achieves distribution matching by selecting a coreset from the unlabeled set with similar gradients to the labeled set . 1.1 Our Contributions . The contributions are our work can be summarized as follows : • RETRIEVE Framework : We propose a coreset selection algorithm RETRIEVE for efficient and robust semi-supervised learning . RETRIEVE poses the coreset selection as a discrete-continuous bi-level optimization problem and solves it efficiently using an online approximation of single-step gradient updates . Essentially , RETRIEVE selects a coreset of the unlabeled set , which , when trained using the combination of the labeled set and the specific unlabeled data coreset , minimizes the model loss on the labeled dataset . We also discuss several implementation tricks to speed up the coreset selection step significantly ( c.f. , Section 3.3 , Section 3.4 ) • RETRIEVE in Traditional SSL : We empirically demonstrate the effectiveness of RETRIEVE in conjunction with several SOTA SSL algorithms like VAT , Mean-Teacher , and FixMatch . The speedups obtained by RETRIEVE are shown in Figure 2a . Specifically , we see that RETRIEVE consistently achieves close to 3× speedup with accuracy degradation of around 0.7 % . RETRIEVE also achieves more than 4.2× speedup with a slightly higher accuracy degradation . Furthermore , when RETRIEVE is trained for more iterations , RETRIEVE can match the performance of VAT while having a 2× speedup ( see VAT Extended bar plot in Figure 2a ) . RETRIEVE also consistently outperforms simple baselines like early stopping and random sampling . • RETRIEVE in Robust SSL : We further demonstrate the utility of RETRIEVE for robust SSL in the presence of OOD data and imbalance in the unlabeled set . We observe that with the VAT SSL algorithm , RETRIEVE outperforms SOTA robust SSL method DS3L [ 17 ] ( with VAT ) while being around 5× faster . RETRIEVE also significantly outperforms just VAT and random sampling . 1.2 Related Work . Semi-supervised learning : Several papers have been proposed for semi-supervised learning over the past few years . Due to space constraints , we do not talk about generative [ 50 , 48 , 26 , 11 , 19 , 30 , 3 ] and graph-based [ 62 , 33 ] methods for SSL in this work . We instead focus on the main components of the existing SOTA SSL algorithms , viz. , a ) consistency regularization and b ) entropy minimization . The consistency regularization component forces the model to have consistent prediction given an unlabeled data point and its perturbed ( or augmented ) version . The Entropy-minimization component forces the model instances to have low-entropy predictions on unlabeled data instances to ensure that the classes are well separated . One can achieve entropy minimization by directly adding the entropy loss component on the unlabeled class prediction or using methods like Pseudo-Labeling to enforce it implicitly . Mean-Teacher [ 56 ] approach uses a consistency regularization component that forces the predictions of the exponential moving average of the model to be the same as the model prediction of the augmented unlabeled images . VAT [ 42 ] instead computes the perturbation of the unlabeled data point that changes the prediction distribution the most and enforces the model to have the same prediction on both unlabeled data instance and unlabeled data instance with computed perturbation as a form of consistency regularization . MixMatch [ 4 ] uses K standard image augmentations for consistency regularization and enforces entropy minimization by using a sharpening function on the average predicted distribution of K augmentations of unlabeled data instances . FixMatch [ 53 ] induces consistency regularization by forcing the model to have the same prediction on a weakly augmented and strongly augmented image instance . Furthermore , FixMatch [ 53 ] also employs confidence thresholding to mask unlabeled data instances on which the model ’ s prediction confidence is below a threshold from being used in consistency loss . Robust Semi-supervised learning : Several methods have been proposed to make the existing semi-supervised learning algorithms robust to label noise in labeled data and robust to OOD data in the unlabeled set . A popular approach [ 49 , 52 ] to deal with label noises and class imbalance in a supervised learning setting is by reweighing each data instance and jointly learning these weights along with the model parameters . Safe-SSL ( DS3L ) [ 17 ] is a recently proposed SOTA method for robust SSL learning . DS3L is similar to the reweighting in the supervised case and adopts a reweighting approach to deal with OOD data in the unlabeled set . Safe-SSL uses a neural network to predict the weight parameters of unlabeled instances that result in maximum labeled set performance , making it a bi-level optimization problem . In this regard , both RETRIEVE and Safe-SSL approach solves a bi-level optimization problem , except that RETRIEVE solves a discrete optimization problem at the outer level , thereby enabling significant speedup compared to SSL algorithms and an even more considerable speedup compared to safe-SSL ( which itself is 3× slower than SSL algorithms ) . In contrast to safe-SSL and other robust SSL approaches , RETRIEVE achieves both efficiency and robustness . Other approaches for robust SSL include UASD [ 9 ] uses an Uncertainty aware self-distillation with OOD filtering to achieve robust performance and a distributionally robust model to deal with OOD [ 8 ] . Coreset and subset selection methods : Coresets [ 13 ] are small and informative weighted data subsets that approximate original data . Several works [ 57 , 39 , 24 , 23 ] have studied coresets for efficient training of deep learning models in the supervised learning scenarios . CRAIG [ 39 ] selects representative coresets of the training data that closely estimates the full training gradient . Another approach , GLISTER [ 24 ] posed the coreset selection as optimizing the validation set loss for efficient learning focused on generalization . Another approach , GRAD-MATCH [ 23 ] select subsets that approximately match the full training loss or validation loss gradient using orthogonal matching pursuit . Similarly , coreset selection methods [ 57 , 51 , 2 , 24 ] were also used for active learning scenario , where a subset of data instances from the unlabeled set is selected to be labeled . Finally , several recent works have used submodular functions for finding diverse and representative subsets for data subset selection [ 34 , 21 , 57 , 58 ] . 2 Preliminaries . Notation : DenoteD = { xi , yi } ni=1 to be the labeled set with n labeled data points , and U = { xj } mj=1 to be the unlabeled set with m data points . Let θ be the classifier model parameters , ls be the labeled set loss function ( such as cross-entropy loss ) and lu be the unlabeled set loss , e.g . consistencyregularization loss , entropy loss , etc .. Denote LS ( D , θ ) = ∑ i∈D ls ( θ , xi , yi ) and LU ( U , θ , m ) =∑ j∈U milu ( xj , θ ) where m ∈ { 0 , 1 } m is the binary mask vector for unlabeled set . For notational convenience , we denote lsi ( θ ) = ls ( xi , yi , θ ) and denote luj ( θ ) = lu ( xj , θ ) . Semi-supervised loss : Following the above notations , the loss function for many existing SSL algorithms can be written as LS ( D , θ ) + λLU ( U , θ , m ) , where λ is the regularization coefficient for the unlabeled set loss . For Mean Teacher [ 56 ] , VAT [ 42 ] , MixMatch [ 4 ] , the mask vector m is made up entirely of ones , whereas for FixMatch [ 53 ] , m is confidence-thresholded binary vector , indicating whether to include an unlabeled data instance or not . Usually , LS is a cross-entropy loss for classification experiments and squared loss for regression experiments . A detailed formulation of the loss function LU used in different SSL algorithms is given in Appendix C Robust Semi-supervised loss : However , for robust semi-supervised loss , the mask vector m is replaced with a weight vector w ∈ Rm denoting the contribution of data instances in the unlabeled set . The weight vector w is unknown and needs to be learned . The weighted SSL loss is : LS ( D , θ ) + λLU ( U , θ , w ) , where λ is the regularization coefficient for the unlabeled set loss . The state-of-the-art robust SSL method , Safe-SSL [ 17 ] poses the learning problem as : outer−level︷ ︸︸ ︷ w∗ = argmin w LS ( D , argmin θ ( LS ( D , θ ) + λLU ( U , θ , w ) ) ︸ ︷︷ ︸ inner−level ) ( 1 ) In order to solve the problem at the inner level efficiently , Safe-SSL [ 17 ] method uses a single-gradient step approximation to estimate the inner problem solution . The weight vector learning problem after the one-step approximation is : w∗ = argmin w LS ( D , θ − α∇θLS ( D , θ ) − αλ∇θLU ( U , θ , w ) Safe-SSL also uses a single-step gradient approximation to solve the outer level problem as well . As discussed before , the optimization problem of the Safe-SSL [ 17 ] algorithm involves continuous optimization at both inner and outer levels , whereas for RETRIEVE , the outer level involves a discrete optimization problem which makes it significantly faster than Safe-SSL .

Summary: This paper proposes RETRIEVE to address the computational cost issue in previous semi-supervised learning (SSL) algorithms. The key idea is to train SSL on a subset of unlabeled data (coreset) instead of the entire unlabeled data. To do so, it must solve the problem of selecting the coreset efficiently and effectively. RETRIEVE tackles this problem by solving a two-level optimization problem such that the selected coreset minimizes the labeled set loss. This paper evaluates RETRIEVE on two settings---traditional SSL and robust SSL---and demonstrates its improvement in terms of speed, energy consumption, and CO2 emissions compared to prior work.