ABC: Auxiliary Balanced Classifier for Class-imbalanced Semi-supervised Learning

1 Introduction . Recently , numerous deep neural network ( DNN ) -based semi-supervised learning ( SSL ) algorithms have been proposed to improve the performance of DNNs by utilizing unlabeled data when only a small amount of labeled data is available . These algorithms have shown effective performance in various tasks . However , most existing SSL algorithms assume class-balanced datasets , whereas the class distributions of many real-world datasets are imbalanced . It is well known that classifiers trained on class-imbalanced data tend to be biased toward the majority classes . This issue can be more problematic for SSL algorithms that use predicted labels of unlabeled data for their training , because the labels predicted by the algorithm trained on class-imbalanced data become even more severely imbalanced [ 18 ] . For example , Figure 1 ( b ) presents biased predictions of ReMixMatch [ 3 ] , a recent SSL algorithm , trained on CIFAR-10-LT , which is a class-imbalanced dataset with the amount of Class 0 being 100 times more than that of Class 9 , as depicted in Figure 1 ( a ) . Although there are various class-imbalanced learning techniques , they are usually designed for labeled data , and thus can not be simply combined with SSL algorithms under class-imbalanced SSL ( CISSL ) scenarios . Recently , a few CISSL algorithms have been proposed , but the CISSL problem is still underexplored . We propose a new CISSL algorithm that can effectively use unlabeled data , while mitigating class imbalance by using an existing DNN-based SSL algorithm [ 3 , 29 ] as the backbone and introducing an auxiliary balanced classifier ( ABC ) of a single layer . The ABC is attached to a representation 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . layer immediately preceding the classification layer of the backbone , based on the argument that a classification algorithm ( i.e. , backbone ) can learn high-quality representations even if its classifier is biased toward the majority classes [ 17 ] . The ABC is trained to be balanced across all classes by using a mask that rebalances the class distribution , similar to re-sampling in previous SSL studies [ 2 , 7 , 13 , 16 ] . Specifically , the mask stochastically regenerates a class-balanced subset of a minibatch on which the ABC is trained . The ABC is trained simultaneously with the backbone , so that the ABC can use high-quality representations learned from all data points in the minibatch using the backbone . In this way , the ABC can overcome the limitations of the previous resampling techniques , overfitting on minority-class data or loss of information on majority-class data [ 6 , 27 ] . Moreover , to place decision boundaries in low-density regions by utilizing unlabeled data , we use consistency regularization , a recent SSL technique , which enforces the classification outputs of two augmented or perturbed versions of the same unlabeled example to remain unchanged . In particular , we encourage the ABC to be balanced across classes when using consistency regularization by selecting unlabeled examples with the same probability for each class using a mask . Figure 1 ( c ) illustrates that compared to the results of ReMixMatch in Figure 1 ( b ) , the class distribution of the predicted labels becomes more balanced using the proposed algorithm trained on the same dataset . Our experimental results under various scenarios demonstrate that the proposed algorithm achieves state-of-the-art performance . Through qualitative analysis and an ablation study , we further investigate the contribution of each component of the proposed algorithm . The code for the proposed algorithm is available at https : //github.com/LeeHyuck/ABC . 2 Related Work . Semi-supervised learning ( SSL ) Recently , several SSL techniques that utilize unlabeled data have been proposed . Entropy minimization [ 12 ] encourages the classifier outputs to have low entropy for unlabeled data , as in pseudo-labels [ 22 ] . Mixup regularization [ 4 , 32 ] makes the decision boundaries farther away from the data clusters by encouraging the prediction for an interpolation of two inputs to be the same as the interpolation of the prediction for each input . Consistency regularization [ 26 , 24 , 30 ] encourages a classifier to produce similar predictions for perturbed versions of the same unlabeled input . To create perturbed unlabeled inputs , various data augmentation techniques have been used . For example , FixMatch [ 29 ] and ReMixMatch [ 3 ] used strong augmentation methods such as Cutout [ 10 ] and RandomAugment [ 8 ] . FixMatch and ReMixMatch are used as the backbone of the proposed algorithm ; they are described in Section 3.2 . Class-imbalanced learning ( CIL ) As a popular approach for CIL , re-sampling techniques [ 16 , 7 , 2 , 13 ] balance the number of training samples for each class in the training set . As another popular approach , re-weighting techniques [ 23 , 14 , 33 ] re-weight the loss for each class by a factor inversely proportional to the number of data points belonging to that class . Although these approaches are simple , they have some drawbacks . For example , oversampling from minority classes can cause overfitting , whereas undersampling from majority classes can cause information loss [ 6 ] . In the case of re-weighting , gradients can be calculated to be abnormally large when the class imbalance is severe , resulting in unstable training [ 6 , 1 ] . Many attempts have been made to alleviate these problems , such as effective re-weighting [ 9 ] and meta-learning-based re-weighting [ 28 , 15 ] . New forms of losses have also been proposed [ 6 , 27 ] . In [ 36 , 19 ] , knowledge is transferred from the data of majority classes to the data of minority classes . These CIL algorithms were designed for labeled data and require label information ; thus , they are not applicable to unlabeled data . In [ 17 ] , it was found that biased classification is mainly due to the classification layer and that a classification algorithm can learn meaningful representations even from a class-imbalanced training set . Based on this finding , we design the ABC to use high-quality representations learned from class-imbalanced data utilizing FixMatch [ 29 ] and ReMixMatch [ 3 ] . Class-imbalanced semi-supervised learning ( CISSL ) There have been few studies on CISSL . In [ 35 ] , it was found that more accurate decision boundaries can be obtained in class-imbalanced settings through self-supervised learning and semi-supervised learning . DARP [ 18 ] refines biased pseudolabels by solving a convex optimization problem . CReST [ 34 ] , a recent self-training technique , mitigates class imbalance by using pseudo-labeled unlabeled data points classified as minority classes with a higher probability than those classified as majority classes . 3 Methodology . 3.1 Problem setting . Suppose that we have a labeled dataset X = { ( xn , yn ) : n ∈ ( 1 , ... , N ) } , where xn ∈ Rd is the nth labeled data point and yn ∈ { 1 , . . . , L } is the corresponding label . We also have an unlabeled dataset U = { ( um ) : m ∈ ( 1 , . . . , M ) } , where um ∈ Rd is the mth unlabeled data point . We express the ratio of the amount of labeled data as β = NM+N . Generally , β < 0.5 , because label acquisition is costly and laborious . We denote the number of labeled data points of class l as Nl , i.e. , ∑L l=1Nl = N , and assume that the L classes are sorted according to cardinality in descending order , i.e. , N1 ≥ N2 ≥ · · · ≥ NL . We denote the ratio of the class imbalance as γ = N1NL . Under class-imbalanced scenarios , γ 1 . Following previous CIL studies , we define the half of the classes containing a large amount of data as the majority classes , and the other half of the classes , containing a small amount of data , as the minority classes . Following [ 34 ] , we assume that X and U share the same class distribution , i.e. , the labeled and unlabeled datasets are class-imbalanced to the same extent . From X and U , we generate minibatches MBX = { ( xb , yb ) : b ∈ ( 1 , ... , B ) } ⊂ X and MBU = { ( ub ) : b ∈ ( 1 , . . . , B ) } ⊂ U for each iteration of training , where B is the minibatch size . Using these minibatches for training , we aim to learn a model f : Rd → { 1 , ... L } that performs effectively on a class-balanced test set . 3.2 Backbone SSL algorithm . We attach the ABC to the backbone ’ s representation layer , so that it can utilize the high-quality representations learned by the backbone . We use FixMatch [ 29 ] or ReMixMatch [ 3 ] as the backbone , as these two have achieved state-of-the-art SSL performance . FixMatch uses the classification loss calculated from the weakly augmented labeled data point α ( xb ) generated by flipping and cropping the image , and the consistency regularization loss calculated from the weakly augmented unlabeled data point α ( ub ) and strongly augmented unlabeled data point A ( ub ) generated by Cutout [ 10 ] and RandomAugment [ 8 ] . ReMixMatch predicts the class label of the weakly augmented unlabeled data point α ( ub ) using distribution alignment and sharpening , and assigns the predicted label to the strongly augmented unlabeled data point A ( ub ) . These strongly augmented unlabeled data point A ( ub ) and strongly augmented labeled data point A ( xb ) are used to conduct mixup regularization . ReMixMatch also conducts consistency regularization in a manner similar to FixMatch and selfsupervised learning using the rotation of the image [ 11 , 39 ] . FixMatch and ReMixMatch have greatly improved the SSL performance by learning high-quality representations using strong data augmentation . However , these algorithms can be significantly biased toward the majority classes in class-imbalanced settings . Using FixMatch and ReMixMatch as the backbone of the proposed algorithm , we ensure that the ABC enjoys high-quality representations learned by the backbone , while replacing the backbone ’ s biased classifier . To train the ABC , we reuse the weakly augmented data and strongly augmented data used by the backbone to decrease the computational cost . Although we use FixMatch and ReMixMatch as the backbone in this study , the ABC can also be combined with other DNN-based SSL algorithms , as long as they use weakly augmented data and strongly augmented data . 3.3 ABC for class-imbalanced Semi-supervised learning . To train the ABC to be balanced , we first generate 0/1 mask M ( xb ) for each labeled data point xb using a Bernoulli distribution B ( · ) with the parameter set to be inversely proportional to the number of data points of each class . This setting makes B ( · ) generate mask 1 with high probability for the data points in the minority classes , but with low probability for those in the majority classes . Then , the classification loss is multiplied by the generated mask , so that the ABC can be trained with a balanced classification loss . Multiplying the classification loss by the 0/1 mask can be interpreted as oversampling of the data points in the minority classes , whereas it can be interpreted as undersampling of those in the majority classes . In representation learning , oversampling and undersampling techniques have shown overfitting and information loss problems , respectively . In contrast , the ABC can overcome these problems because it uses the representations learned by the backbone , which is trained on all data points in the minibatch . The use of the 0/1 mask to construct the balanced loss , instead of directly creating a balanced subset , allows the backbone and the ABC to be trained from the same minibatches . Therefore , the representations of minibatches calculated for training the backbone can be used again for training the ABC . Consequently , the proposed algorithm only requires a slightly increased time cost compared to training the backbone alone . This is confirmed in Section 4.3 . The overall procedure of balanced training with 0/1 mask for the ABC attached to a representation layer of the backbone is presented in Figure 2 . The classification loss for the ABC , Lcls , with 0/1 mask M ( · ) is expressed as Lcls = 1 B B∑ b=1 M ( xb ) H ( ps ( y|α ( xb ) ) , pb ) , ( 1 ) M ( xb ) = B ( NL Nyb ) , ( 2 ) where H is the standard cross-entropy loss , α ( xb ) is an augmented labeled data point , ps ( y|α ( xb ) ) is the predicted class distribution using the ABC for α ( xb ) , and pb is the one-hot label for xb .

Summary: The authors introduce a semi-supervised learning algorithm (SSL) for class-imbalanced scenarios.  In particular, an Auxiliar Balanced Classifier coupled with a 0/1 mask is used to favor the cost regularization. The methodology is interesting, and the experiments demonstrate that the proposed method outperforms state-of-the-art techniques on image classification tasks, even for large datasets under challenging imbalance ratios.