ABC: Auxiliary Balanced Classifier for Class-Imbalanced Semi-Supervised Learning
Year: 2021

Abstract:
Existing semi-supervised learning (SSL) algorithms typically assume classbalanced datasets, although the class distributions of many real-world datasets are imbalanced. In general, classifiers trained on a class-imbalanced dataset are biased toward the majority classes. This issue becomes more problematic for SSL algorithms because they utilize the biased prediction of unlabeled data for training. However, traditional class-imbalanced learning techniques, which are designed for labeled data, cannot be readily combined with SSL algorithms. We propose a scalable class-imbalanced SSL algorithm that can effectively use unlabeled data, while mitigating class imbalance by introducing an auxiliary balanced classifier (ABC) of a single layer, which is attached to a representation layer of an existing SSL algorithm. The ABC is trained with a class-balanced loss of a minibatch, while using high-quality representations learned from all data points in the minibatch using the backbone SSL algorithm to avoid overfitting and information loss. Moreover, we use consistency regularization, a recent SSL technique for utilizing unlabeled data in a modified way, to train the ABC to be balanced among the classes by selecting unlabeled data with the same probability for each class. The proposed algorithm achieves state-of-the-art performance in various class-imbalanced SSL experiments using four benchmark datasets.

heading: 1 Introduction
text: Recently, numerous deep neural network (DNN)-based semi-supervised learning (SSL) algorithms have been proposed to improve the performance of DNNs by utilizing unlabeled data when only a small amount of labeled data is available. These algorithms have shown effective performance in various tasks. However, most existing SSL algorithms assume class-balanced datasets, whereas the class distributions of many real-world datasets are imbalanced. It is well known that classifiers trained on class-imbalanced data tend to be biased toward the majority classes. This issue can be more problematic for SSL algorithms that use predicted labels of unlabeled data for their training, because the labels predicted by the algorithm trained on class-imbalanced data become even more severely imbalanced [18]. For example, Figure 1 (b) presents biased predictions of ReMixMatch [3], a recent SSL algorithm, trained on CIFAR-10-LT, which is a class-imbalanced dataset with the amount of Class 0 being 100 times more than that of Class 9, as depicted in Figure 1 (a). Although there are various class-imbalanced learning techniques, they are usually designed for labeled data, and thus cannot be simply combined with SSL algorithms under class-imbalanced SSL (CISSL) scenarios. Recently, a few CISSL algorithms have been proposed, but the CISSL problem is still underexplored.
We propose a new CISSL algorithm that can effectively use unlabeled data, while mitigating class imbalance by using an existing DNN-based SSL algorithm [3, 29] as the backbone and introducing an auxiliary balanced classifier (ABC) of a single layer. The ABC is attached to a representation
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
layer immediately preceding the classification layer of the backbone, based on the argument that a classification algorithm (i.e., backbone) can learn high-quality representations even if its classifier is biased toward the majority classes [17]. The ABC is trained to be balanced across all classes by using a mask that rebalances the class distribution, similar to re-sampling in previous SSL studies [2, 7, 13, 16]. Specifically, the mask stochastically regenerates a class-balanced subset of a minibatch on which the ABC is trained. The ABC is trained simultaneously with the backbone, so that the ABC can use high-quality representations learned from all data points in the minibatch using the backbone. In this way, the ABC can overcome the limitations of the previous resampling techniques, overfitting on minority-class data or loss of information on majority-class data [6, 27].
Moreover, to place decision boundaries in low-density regions by utilizing unlabeled data, we use consistency regularization, a recent SSL technique, which enforces the classification outputs of two augmented or perturbed versions of the same unlabeled example to remain unchanged. In particular, we encourage the ABC to be balanced across classes when using consistency regularization by selecting unlabeled examples with the same probability for each class using a mask. Figure 1 (c) illustrates that compared to the results of ReMixMatch in Figure 1 (b), the class distribution of the predicted labels becomes more balanced using the proposed algorithm trained on the same dataset. Our experimental results under various scenarios demonstrate that the proposed algorithm achieves state-of-the-art performance. Through qualitative analysis and an ablation study, we further investigate the contribution of each component of the proposed algorithm. The code for the proposed algorithm is available at https://github.com/LeeHyuck/ABC.

heading: 2 Related Work
text: Semi-supervised learning (SSL) Recently, several SSL techniques that utilize unlabeled data have been proposed. Entropy minimization [12] encourages the classifier outputs to have low entropy for unlabeled data, as in pseudo-labels [22]. Mixup regularization [4, 32] makes the decision boundaries farther away from the data clusters by encouraging the prediction for an interpolation of two inputs to be the same as the interpolation of the prediction for each input. Consistency regularization [26, 24, 30] encourages a classifier to produce similar predictions for perturbed versions of the same unlabeled input. To create perturbed unlabeled inputs, various data augmentation techniques have been used. For example, FixMatch [29] and ReMixMatch [3] used strong augmentation methods such as Cutout [10] and RandomAugment [8]. FixMatch and ReMixMatch are used as the backbone of the proposed algorithm; they are described in Section 3.2.
Class-imbalanced learning (CIL) As a popular approach for CIL, re-sampling techniques [16, 7, 2, 13] balance the number of training samples for each class in the training set. As another popular approach, re-weighting techniques [23, 14, 33] re-weight the loss for each class by a factor inversely proportional to the number of data points belonging to that class. Although these approaches are simple, they have some drawbacks. For example, oversampling from minority classes can cause overfitting, whereas undersampling from majority classes can cause information loss [6]. In the case of re-weighting, gradients can be calculated to be abnormally large when the class imbalance is severe, resulting in unstable training [6, 1]. Many attempts have been made to alleviate these problems, such as effective re-weighting [9] and meta-learning-based re-weighting [28, 15]. New forms of losses have also been proposed [6, 27]. In [36, 19], knowledge is transferred from the data of
majority classes to the data of minority classes. These CIL algorithms were designed for labeled data and require label information; thus, they are not applicable to unlabeled data. In [17], it was found that biased classification is mainly due to the classification layer and that a classification algorithm can learn meaningful representations even from a class-imbalanced training set. Based on this finding, we design the ABC to use high-quality representations learned from class-imbalanced data utilizing FixMatch [29] and ReMixMatch [3].
Class-imbalanced semi-supervised learning (CISSL) There have been few studies on CISSL. In [35], it was found that more accurate decision boundaries can be obtained in class-imbalanced settings through self-supervised learning and semi-supervised learning. DARP [18] refines biased pseudolabels by solving a convex optimization problem. CReST [34], a recent self-training technique, mitigates class imbalance by using pseudo-labeled unlabeled data points classified as minority classes with a higher probability than those classified as majority classes.

heading: 3 Methodology

heading: 3.1 Problem setting
text: Suppose that we have a labeled dataset X = {(xn, yn) : n ∈ (1, ..., N)}, where xn ∈ Rd is the nth labeled data point and yn ∈ {1, . . . , L} is the corresponding label. We also have an unlabeled dataset U = {(um) : m ∈ (1, . . . ,M)}, where um ∈ Rd is the mth unlabeled data point. We express the ratio of the amount of labeled data as β = NM+N . Generally, β < 0.5, because label acquisition is costly and laborious. We denote the number of labeled data points of class l as Nl, i.e., ∑L l=1Nl = N , and assume that the L classes are sorted according to cardinality in descending order, i.e., N1 ≥ N2 ≥ · · · ≥ NL. We denote the ratio of the class imbalance as γ = N1NL . Under class-imbalanced scenarios, γ 1. Following previous CIL studies, we define the half of the classes containing a large amount of data as the majority classes, and the other half of the classes, containing a small amount of data, as the minority classes. Following [34], we assume that X and U share the same class distribution, i.e., the labeled and unlabeled datasets are class-imbalanced to the same extent. From X and U , we generate minibatches MBX = {(xb, yb) : b ∈ (1, ..., B)} ⊂ X and MBU = {(ub) : b ∈ (1, . . . , B)} ⊂ U for each iteration of training, where B is the minibatch size. Using these minibatches for training, we aim to learn a model f : Rd → {1, ...L} that performs effectively on a class-balanced test set.

heading: 3.2 Backbone SSL algorithm
text: We attach the ABC to the backbone’s representation layer, so that it can utilize the high-quality representations learned by the backbone. We use FixMatch [29] or ReMixMatch [3] as the backbone, as these two have achieved state-of-the-art SSL performance. FixMatch uses the classification loss calculated from the weakly augmented labeled data point α (xb) generated by flipping and cropping the image, and the consistency regularization loss calculated from the weakly augmented unlabeled data point α (ub) and strongly augmented unlabeled data point A (ub) generated by Cutout [10] and RandomAugment [8]. ReMixMatch predicts the class label of the weakly augmented unlabeled data point α (ub) using distribution alignment and sharpening, and assigns the predicted label to the strongly augmented unlabeled data point A (ub). These strongly augmented unlabeled data point A (ub) and strongly augmented labeled data point A (xb) are used to conduct mixup regularization. ReMixMatch also conducts consistency regularization in a manner similar to FixMatch and selfsupervised learning using the rotation of the image [11, 39]. FixMatch and ReMixMatch have greatly improved the SSL performance by learning high-quality representations using strong data augmentation. However, these algorithms can be significantly biased toward the majority classes in class-imbalanced settings.
Using FixMatch and ReMixMatch as the backbone of the proposed algorithm, we ensure that the ABC enjoys high-quality representations learned by the backbone, while replacing the backbone’s biased classifier. To train the ABC, we reuse the weakly augmented data and strongly augmented data used by the backbone to decrease the computational cost. Although we use FixMatch and ReMixMatch as the backbone in this study, the ABC can also be combined with other DNN-based SSL algorithms, as long as they use weakly augmented data and strongly augmented data.

heading: 3.3 ABC for class-imbalanced Semi-supervised learning
text: To train the ABC to be balanced, we first generate 0/1 mask M (xb) for each labeled data point xb using a Bernoulli distribution B (·) with the parameter set to be inversely proportional to the number of data points of each class. This setting makes B (·) generate mask 1 with high probability for the data points in the minority classes, but with low probability for those in the majority classes. Then, the classification loss is multiplied by the generated mask, so that the ABC can be trained with a balanced classification loss. Multiplying the classification loss by the 0/1 mask can be interpreted as oversampling of the data points in the minority classes, whereas it can be interpreted as undersampling of those in the majority classes. In representation learning, oversampling and undersampling techniques have shown overfitting and information loss problems, respectively. In contrast, the ABC can overcome these problems because it uses the representations learned by the backbone, which is trained on all data points in the minibatch. The use of the 0/1 mask to construct the balanced loss, instead of directly creating a balanced subset, allows the backbone and the ABC to be trained from the same minibatches. Therefore, the representations of minibatches calculated for training the backbone can be used again for training the ABC. Consequently, the proposed algorithm only requires a slightly increased time cost compared to training the backbone alone. This is confirmed in Section 4.3. The overall procedure of balanced training with 0/1 mask for the ABC attached to a representation layer of the backbone is presented in Figure 2. The classification loss for the ABC, Lcls, with 0/1 mask M (·) is expressed as
Lcls = 1
B B∑ b=1 M (xb)H (ps (y|α (xb)) , pb) , (1)
M (xb) = B ( NL Nyb ) , (2)
where H is the standard cross-entropy loss, α (xb) is an augmented labeled data point, ps (y|α (xb)) is the predicted class distribution using the ABC for α (xb), and pb is the one-hot label for xb.

heading: 3.4 Consistency regularization for ABC
text: To increase the margin between the decision boundary and the data points using unlabeled data, we conduct consistency regularization for the ABC, similar to the way in FixMatch. Specifically, we first obtain the predicted class distribution ps (y|α (ub)) for a weakly augmented unlabeled data point α (ub) using the ABC and use it as a soft pseudo-label qb. Then, for two strongly augmented unlabeled data points A1 (ub) and A2 (ub), we train the ABC to produce their predicted class distributions, ps (y|A1 (ub)) and ps (y|A2 (ub)), to be close to qb. In class-imbalanced settings, because most unlabeled data points belong to majority classes, most weakly augmented unlabeled data points can be predicted as the majority classes. Then, consistency regularization would be conducted with a higher frequency for the majority classes, which can cause a classifier to be biased toward the majority classes. To prevent this issue, we conduct consistency regularization in a modified manner that is suitable for class-imbalance problems. Specifically,
whereas FixMatch minimizes entropy by converting the predicted class distribution for a weakly augmented data point into a one-hot pseudo-label, we directly use the predicted class distribution as a soft pseudo-label. We do not pursue entropy minimization for the ABC because it can accelerate biased classification toward certain classes. Moreover, we once again generate 0/1 mask M (·) for each unlabeled data point ub based on a soft pseudo label qb, and multiply the consistency regularization loss for ub by the generated mask, so that the ABC can be trained with a class-balanced consistency regularization loss. Note that existing resampling techniques are not applicable to unlabeled data, because they require a label for each data point. In contrast, we make it possible to resample unlabeled data by using the soft pseudo-label and the 0/1 mask. The consistency regularization loss, Lcon, with 0/1 mask M (·) is expressed as
Lcon = 1
B B∑ b=1 2∑ k=1 M (ub) I (max (qb) ≥ τ)H (ps (y|Ak (ub)) , qb) , (3)
M (ub) = B ( NL Nq̂b ) , (4)
where I is the indicator function, max (qb) is the highest predicted assignment probability for any class, representing the confidence of prediction, and τ is the confidence threshold. To avoid the unwanted effects of inaccurate soft pseudo-labels during consistency regularization, we only use the weakly augmented unlabeled data points whose confidence is higher than the threshold τ , similar to that in FixMatch. To take full advantage of few unlabeled data points with prediction confidence values that are higher than the confidence threshold τ in the early stage of training, we gradually decrease the parameter of the Bernoulli distribution B (·) for ub from 1 to NL/Nq̂b , where q̂b is the one-hot pseudo-label obtained from qb. Following previous studies [3, 24, 29, 4], we do not backpropagate gradients for pseudo-label prediction. The overall procedure for consistency regularization for the ABC is shown in Appendix A.

heading: 3.5 End-to-end training
text: Unlike a recent CIL trend to finetune a classifier in a balanced manner after representation learning is completed (i.e., decoupled learning of representations and a classifier) [17, 27], we obtain a balanced classifier by training the proposed algorithm end-to-end. We train the proposed algorithm with the sum of losses from Sections 3.3 and 3.4, and the loss for the backbone, Lback. The total loss function Ltotal is expressed as
Ltotal = Lcls + Lcon + Lback. (5)
Whereas we use the sum of the losses for the backbone and ABC for training the proposed algorithm, we predict the class labels of new data points using only the ABC. In our experiments in Sections 4.4 and 4.5, we show that the proposed algorithm trained end-to-end produces better performance than competing algorithms with decoupled learning of representations and a classifier, and we analyze possible reasons. We present the pseudo code of the proposed algorithm in Appendix B.

heading: 4 Experiments

heading: 4.1 Experimental setup
text: We created class-imbalanced versions of CIFAR-10, CIFAR-100 [21], and SVHN [25] datasets to conduct experiments under various ratios of class imbalance γ and various ratios of the amount of labeled data β. For class-imbalance types, we first consider long-tailed (LT) imbalance in which the number of data points exponentially decreases from the largest to the smallest class, i.e., Nk = N1 × γ− k−1 L−1 , where γ = N1NL . We also consider step imbalance [5] in which the whole majority classes have the same amount of data and the whole minority classes also have the same amount of data. Two types of class imbalance for the considered datasets are illustrated in Appendix C. For the main setting, we set γ = 100, N1 = 1000, and β = 20% for CIFAR-10 and SVHN, and γ = 20, N1 = 200 and β = 40% for CIFAR-100. Similar to [18], we set γ of CIFAR-100 to be relatively small because CIFAR-100 has only 500 training data points for each class. To evaluate the
performance of the proposed algorithm on large-scale datasets, we also conducted experiments on 7.5M data points of 256 by 256 images from the LSUN dataset [37].
We compared the performance of the proposed algorithm with that of various baseline algorithms. Specifically, we considered the following baseline algorithms:
• Deep CNN (vanilla algorithm): This is trained on only labeled data with the cross-entropy loss.
• BALMS [27] (CIL algorithm): This state-of-the-art CIL algorithm does not use unlabeled data.
• VAT [24], ReMixMatch [3], and FixMatch [29] (SSL algorithms): These are state-of-the-art SSL algorithms, but do not consider class imbalance.
• FixMatch+CReST+PDA and ReMixMatch+CReST+PDA (CISSL algorithms): CReST+PDA [34] mitigates class imbalance by using unlabeled data points classified as the minority classes with a higher probability than those classified as the majority classes.
• ReMixMatch+DARP and FixMatch+DARP (CISSL algorithms): These algorithms use DARP [18] to refine the pseudo labels obtained from ReMixMatch or FixMatch.
• ReMixMatch+DARP+cRT and FixMatch+DARP+cRT (CISSL algorithms): Compared to ReMixMatch+DARP and FixMatch+DARP, these algorithms finetune the classifier using cRT [17].
For the structure of the deep CNN used in the proposed and baseline algorithms, we used Wide ResNet-28-2 [38]. We trained the proposed algorithm for 250, 000 iterations with a batch size of 64. The confidence threshold τ was set to 0.95 based on experiments with various values of τ in Appendix D. We used the Adam optimizer [20] with a learning rate of 0.002, and used Cutout [10] and RandomAugment [8] for strong data augmentation, following [18]. Similar to [3, 4], we evaluated the performance of the proposed algorithm using an exponential moving average of the parameters over iterations with a decay rate of 0.999, instead of scheduling the learning rate. In Tables 1-5, we used the overall accuracy and the accuracy only for minority classes as performance measures. We repeated the experiments five times under the main setting, and three times under the step imbalance and other settings of β and γ. We report the average and standard deviation of the performance measures over repeated experiments. For the vanilla algorithm, FixMatch+DARP+cRT, and ReMixMatch+DARP+cRT, which suffered from overfitting, we measured performance every 500 iterations and recorded the best performance. Further details of the experimental setup are described in Appendix E.

heading: 4.2 Experimental results
text: The performance of the competing algorithms under the main setting are summarized in Table 1. We can observe that the proposed algorithm achieved the highest overall performance, with improved performance for minority classes. Interestingly, VAT, an SSL algorithm, showed similar performance to the vanilla algorithm, and worse performance than BALMS, a CIL algorithm. Similarly, FixMatch and ReMixMatch, which do not consider class imbalance, showed poor performance for minority classes. Although BALMS mitigated class imbalance, it produced poor overall performance, as it did not use unlabeled data for training. This demonstrates the importance of using unlabeled data for training, even in the class-imbalanced setting. FixMatch+CReST+PDA and ReMixMatch+CReST+PDA mitigated class imbalance by using unlabeled data points classified as the minority classes with a higher probability, but produced lower performance than the proposed algorithm. This may be because even if all unlabeled data points classified as minority classes are additionally used for training, their amount is still less than that of the data in majority classes, while the proposed algorithm uses class-balanced minibatches by generating the 0/1 mask. Fixmatch+DARP and ReMixMatch+DARP slightly mitigated class imbalance by refining biased pseudo-labels, but resulted in lower performance than the proposed algorithm. This may be because even perfect pseudo labels cannot change the underlying class-imbalanced distribution of the training data. By additionally using a rebalancing technique cRT, FixMatch(ReMixMatch)+DARP+cRT performed better than FixMatch(ReMixMatch)+DARP. However, FixMatch(ReMixMatch)+DARP+cRT still performed worse than FixMatch(ReMixMatch)+ABC, although it also uses high-quality representations learned by FixMatch(ReMixMatch) and techniques for mitigating class imbalance. The superior performance of FixMatch(ReMixMatch)+ABC over FixMatch(ReMixMatch)+DARP+cRT is probably because FixMatch(ReMixMatch)+ABC was trained end-to-end, and the ABC was also trained using unlabeled data. We discuss this in more detail in Sections 4.4 and 4.5. Overall, the algorithms combined with
ReMixMatch performed better than the algorithms combined with FixMatch. In addition to the overall accuracy and minority-class-accuracy, we also compared the performance of the competing algorithms in terms of the geometric mean (G-mean) of class-wise accuracy under the main setting in Appendix F.
FixMatch [29] 72.3±0.33 / 53.8±0.63 88.0±0.30 / 79.4±0.54 51.0±0.20 / 32.8±0.41 w/ CReST+PDA [34] 76.6±0.46 / 61.4±0.85 89.1±0.69 / 81.7±1.18 51.6±0.29 / 36.4±0.46
w/ DARP [18] 73.7±0.98 / 57.0±2.12 88.6±0.19 / 80.5±0.54 51.4±0.37 / 33.9±0.77 w/ DARP+cRT [18] 78.1±0.89 / 66.6±1.55 89.9±0.44 / 83.5±0.61 54.7±0.46 / 41.2±0.42
w/ ABC 81.1±0.82 / 72.0±1.77 92.0±0.38 / 87.9±0.73 56.3±0.19 / 43.4±0.42 ReMixMatch [3] 73.7±0.39 / 55.9±0.87 89.8±0.42 / 82.8±0.68 54.0±0.29 / 37.1±0.37 w/ CReST+PDA [34] 75.7±0.34 / 59.6±0.76 90.9±0.20 / 85.2±0.39 54.6±0.48 / 38.1±0.69 w/ DARP [18] 74.4±0.41 / 56.9±0.67 90.2±0.22 / 83.5±0.40 54.5±0.33 / 37.7±0.58
w/ DARP+cRT [18] 78.5±0.61 / 66.4±1.68 92.1±0.48 / 87.6±0.75 55.1±0.45 / 43.6±0.58 w/ ABC 82.4±0.45 / 75.7±1.18 93.9±0.16 / 92.5±0.4 57.6±0.26 / 46.7±0.50
To evaluate the performance of the proposed algorithm in various settings, we conducted experiments using ReMixMatch, FixMatch, and the CISSL algorithms considered in Table 1, while changing the ratio of class imbalance γ and the ratio of the amount of labeled data β. The results for CIFAR-10 are presented in Table 2, and the results for SVHN and CIFAR-100 are presented in Appendix G. In Table 2, we can observe that the proposed algorithm achieved the highest overall accuracy with greatly improved performance for minority classes for all settings. Because FixMatch+DARP+cRT and ReMixMatch+DARP+cRT do not use unlabeled data for classifier tuning, the difference in performance between FixMatch(ReMixMatch)+DARP+cRT and the proposed algorithm increased as the ratio of the amount of labeled data β decreased and as the ratio of class imbalance γ increased. In addition, the difference in performance between FixMatch(ReMixMatch)+CReST+PDA and the proposed algorithm tended to increase as the ratio of class imbalance γ increased, because the difference between the number of labeled data points belonging to the majority classes and the number of unlabeled data points classified as the minority classes increases with γ.
FixMatch [29] 70.0±0.59 / 48.9±1.04 74.9±0.63 / 58.2±1.28 81.2±0.07 / 70.7±0.36 68.5±0.60 / 45.8±1.15 w/ CReST+PDA [34] 73.9±0.40 / 58.9±1.14 77.6±0.73 / 64.0±1.39 83.3±0.10 / 75.7±0.39 70.0±0.82 / 49.4±1.52 w/ DARP+cRT [18] 74.6±0.98 / 59.2±2.12 79.0±0.25 / 67.7±0.95 83.6±0.42 / 77.1±1.19 73.2±0.85 / 57.1±1.13
w/ ABC 77.2±1.60 / 65.7±2.85 81.5±0.29 / 72.9±0.96 85.2±0.51 / 80.2±0.64 77.1±0.46 / 64.4±0.92 ReMixMatch [3] 71.5±0.51 / 52.2±1.08 75.8±0.10 / 59.4±0.17 81.5±0.17 / 70.7±0.32 69.9±0.23 / 48.4±0.60 w/ CReST+PDA [34] 73.8±0.32 / 56.6±0.43 78.6±0.73 / 64.8±1.49 83.9±0.26 / 75.4±0.52 71.3±0.77 / 50.8±1.59 w/ DARP+cRT [18] 75.9±1.20 / 62.1±3.10 81.0±0.16 / 70.7±0.72 84.5±0.80 / 77.8±1.67 73.9±0.59 / 57.4±1.45
w/ ABC 79.8±0.36 / 70.8±0.92 84.3±1.03 / 80.6±0.97 87.5±0.31 / 84.6±1.19 80.6±0.66 / 72.1±1.51
We also conducted experiments under a step-imbalance setting, where the class imbalance was more noticeable. This setting assumes a more severely imbalanced class distribution than the LT imbalance settings, because half of the classes have very scarce data. The experimental results for CIFAR-10 are presented in Table 3, and the results for SVHN and CIFAR-100 are presented in Appendix H. In Table 3, we can see that the proposed algorithm achieved the best performance, and the performance margin is greater than that of the LT imbalance settings. ReMixMatch+CReST+PDA showed relatively low performance compared to the other algorithms.
To evaluate the performance of the proposed algorithm on a large-scale dataset, we also conducted experiments on the LSUN dataset [37], which is naturally a long-tailed dataset. Among the algorithms considered in Tables 2 and 3, those combined with CReST were excluded for comparison, because CReST requires loading of the whole unlabeled data in the repeated process of updating pseudolabels, which is not possible for the large-scale LSUN dataset. Instead, we additionally considered FixMatch+cRT and ReMixMatch+cRT for comparison. The experimental results are presented in Table 4. The proposed algorithm showed better performance than the other baseline algorithms. DARP resulted in degradation of the performance, possibly because the scale of the LSUN dataset is very large. Specifically, DARP solves a convex optimization with all unlabeled data points to refine the pseudo labels. As the scale of the unlabeled dataset increases, this optimization problem becomes more difficult to solve and, consequently, the pseudo-labels could be refined inaccurately. Unlike the results for other datasets, the algorithms combined with FixMatch performed better than the algorithms combined with ReMixMatch.

heading: 4.3 Complexity of the proposed algorithm
text: The proposed algorithm requires additional parameters for the ABC, but the number of the additional parameters is negligible compared to the number of parameters of the backbone. For example, the ABC additionally required only 0.09% and 0.87% of the number of backbone parameters for CIFAR-10 with 10 classes and CIFAR-100 with 100 classes, respectively. Moreover, because the ABC shares the representation layer of the backbone, it does not significantly increase the memory usage and training time. Furthermore, we could train the proposed algorithm on the large-scale LSUN dataset without a significant increase in computation cost, because the entire training procedure could be carried out using minibatches of data. In contrast, the algorithms combined with DARP required convex optimization for all pseudo-labels, which significantly increased the computation cost as the number of classes or the amount of data increased. Similarly, it required significant time to train the algorithms combined with CReST, because CReST requires iterative re-training with a labeled set expanded by adding unlabeled data points with pseudo-labels. We present the floating point operations per second (FLOPS) for each algorithm using Nvidia Tesla-V100 in Appendix I.

heading: 4.4 Qualitative analysis of high-quality representations and balanced classification
text: The ABC can use high-quality representations learned by the backbone when performing balanced classification. To verify this, in Figure 3, we present t-distributed stochastic neighbor embedding (t-SNE) [31] of the representations of the CIFAR-10 test set learned by the ABC (without SSL backbone), FixMatch+ABC, and ReMixMatch+ABC on CIFAR-10-LT under the main setting. Different colors indicate different classes. As expected, “ABC (without SSL backbone)" failed to learn class-separable representations because sufficient data were not used for training while using the 0/1 mask. In contrast, by training the backbone (FixMatch or ReMixMatch) together with the ABC, the proposed algorithm could use the entire data and learn high-quality representations. In this example, ReMixMatch produced more separable representations than FixMatch, which shows that the choice of the backbone affects the performance of the proposed algorithm, as expected.
The proposed algorithm can also mitigate class imbalance by using the ABC. To verify this, we compare the confusion matrices of the predictions on the test set of CIFAR-10 using ReMixMatch, ReMixMatch+DARP+cRT, and ReMiMatch+ABC trained on CIFAR-10 under the main setting in Figure 4. In the confusion matrices, the value in the ith row and the jth column represents the ratio of the amount of data belonging to the ith class to the amount of data predicted as the jth class. Each cell has a darker red color when the ratio is larger. We can see that ReMixMatch often misclassified data points in the minority classes (e.g., classes 8 and 9 into classes 0 and 1). This may be because ReMixMatch does not consider class imbalance, and thus biased pseudo-labels were used for training. ReMixMatch+DARP+cRT produced a more balanced class-distribution compared to ReMixMatch by additionally using DARP+cRT. However, a significant number of data points in the minority classes were still misclassified as majority classes. In contrast, ReMixMatch+ABC classified the test data points in the minority classes with higher accuracy, and produced a significantly more balanced class distribution than ReMixMatch+DARP+cRT, as shown in Figure 4 (c). As both ReMixMatch+DARP+cRT and ReMixMatch+ABC use ReMixMatch to learn representations, the performance gap between these two algorithms results from the different characteristics of the ABC versus DARP+cRT as follows. First, DARP+cRT does not use unlabeled data for training its classifier after representations learning is completed, whereas the ABC uses unlabeled data with unbiased pseudo-labels for its training. Second, whereas DARP+cRT decouples the learning of representations and training of a classifier, the ABC is trained end-to-end interactively with representations learned by the backbone. We also present the confusion matrices of the predictions on the test set of CIFAR-10 using FixMatch, FixMatch+DARP+cRT, and FixMatch+ABC as well as the confusion matrices of the pseudo-labels on the same dataset using ReMixMatch, ReMixMatch+DARP+cRT, ReMixMatch+ABC, FixMatch, FixMatch+DARP+cRT, and FixMatch+ABC in Appendix J. Moreover, we compare the ABC and the classifier of DARP+cRT in more detail using the validation loss plots in Appendix K.

heading: 4.5 Ablation study
text: We conducted an ablation study on CIFAR-10-LT in the main setting to investigate the effect of each element of the proposed algorithm. The results for ReMixMatch+ABC are presented in Table 5, where each row indicates the proposed algorithm with the described conditions in that row. The results are summarized as follows. 1) If we did not gradually decrease the parameter of the Bernoulli distribution B (·) when conducting consistency regularization, then an overbalance problem occurred
because of unlabeled data misclassified as minority classes. 2) Without consistency regularization for the ABC, the decision boundary did not clearly separate each class. 3) Without using the 0/1 mask for Lcls and Lcon, the ABC was trained to be biased toward the majority classes. 4) Without confidence threshold τ for consistency regularization, training became unstable and, consequently, the ABC was trained to be biased toward certain classes. 5) Similarly, if hard pseudo-labels, instead of soft pseudo-labels, were used for consistency regularization, then the ABC was biased toward certain classes. 6) If the ABC was solely used without the backbone, the performance decreased because the ABC could not use high-quality representations learned by the backbone. 7) When we used a re-weighting technique [13] instead of a mask for the ABC, training became unstable because of abnormally large gradients calculated for training on the data of the minority classes. 8) The decoupled training of the backbone and ABC resulted in decreased classification performance, as was also analyzed in Section 4.4. Similarly, we present the results of the ablation study for FixMatch+ABC in Appendix L.

heading: 5 Conclusion
text: We introduced the ABC, which is attached to a state-of-the-art SSL algorithm, for CISSL. The ABC can utilize high-quality representations learned by the backbone, while being trained to make classbalanced predictions. The ABC also utilizes unlabeled data by conducting consistency regularization in a modified way for class-imbalance problems. The experimental results obtained under various settings demonstrate that the proposed algorithm outperforms the baseline algorithms. We also conducted a qualitative analysis and an ablation study to verify the contribution of each element of the proposed algorithm. The proposed algorithm assumes that the labeled and unlabeled data are class-imbalanced to the same extent. In the future, we plan to release this assumption by adopting a module for estimating class distribution. Deep learning algorithms can be applied to many societal problems. However, if the training data are imbalanced, the algorithms could be trained to make socially biased decisions in favor of the majority groups. The proposed algorithm can contribute to solving these issues. However, there is also a potential risk that the proposed algorithm could be used as a tool to identify minorities and discriminate against them. It should be ensured that the proposed method cannot be used for any purpose that may have negative social impacts.

heading: Acknowledgments
text: This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2018R1C1B6004511, 2020R1A4A10187747).

Review summaries:
The authors introduce a semi-supervised learning algorithm (SSL) for class-imbalanced scenarios.  In particular, an Auxiliar Balanced Classifier coupled with a 0/1 mask is used to favor the cost regularization. The methodology is interesting, and the experiments demonstrate that the proposed method outperforms state-of-the-art techniques on image classification tasks, even for large datasets under challenging imbalance ratios.

This paper deals with the class-imbalanced semi-supervised learning problem. The paper utilizes existing semi-supervised learning algorithms to acquire good representation, and additionally introduces an auxiliary balanced classifier called ABC, which is trained on a class-balanced subset of a minibatch to mitigate the biased results caused by class-imbalance. The ABC is trained using not only the balanced subset but also unlabeled data with consistency loss. The experimental results show that the proposed method outperforms existing methods by large margin.