ABC: Auxiliary Balanced Classifier for Class-imbalanced Semi-supervised Learning
Authors: Hyuck Lee
Seungjae Shin
Heeyoung Kim
Venue: NeurIPS 2021 Poster (2021)
Keywords: auxiliary balanced classifier
class-imbalanced semi-supervised learning

Abstract: Existing semi-supervised learning (SSL) algorithms typically assume class-balanced datasets, although the class distributions of many real world datasets are imbalanced. In general, classifiers trained on a class-imbalanced dataset are biased toward the majority classes. This issue becomes more problematic for SSL algorithms because they utilize the biased prediction of unlabeled data for training. However, traditional class-imbalanced learning techniques, which are designed for labeled data, cannot be readily combined with SSL algorithms. We propose a scalable class-imbalanced SSL algorithm that can effectively use unlabeled data, while mitigating class imbalance by introducing an auxiliary balanced classifier (ABC) of a single layer, which is attached to a representation layer of an existing SSL algorithm. The ABC is trained with a class-balanced loss of a minibatch, while using high-quality representations learned from all data points in the minibatch using the backbone SSL algorithm to avoid overfitting and information loss. Moreover, we use consistency regularization, a recent SSL technique for utilizing unlabeled data in a modified way, to train the ABC to be balanced among the classes by selecting unlabeled data with the same probability for each class. The proposed algorithm achieves state-of-the-art performance in various class-imbalanced SSL experiments using four benchmark datasets.

ABC: Auxiliary Balanced Classifier for Class-Imbalanced Semi-Superiie Learning
Hyuck Lee Seungjae Shin Heeyoung Kim
Department of Industrial and Systems Engineering, KAIST
{dlgur0921, tmdw00910, heeyoungkim) Skaist..aaaaaaaaaaaaaaaaaaaa kr
Abstract
Existing semi-supervised learning (SSL) algorithms typically assume class- balanced datasets, although the class distributions of many real-world datasets are imbalanced. In general, classifiers trained on a caassiimbllnceed dataset are biased toward the majority classes. This issue becomes more problematic for SSL algorithms because they utilize the biased prediction of unlabeled data for training. However, traditional maass-imbalanced learning techniques, which designed for labeled data, cannot be readily combined with SSL algorithms. We propose a scal- able blass-imbalanced SSL. algorithm that effectively unlabeled data, while mitigating class imbalance by introducing an auxiliary balanced classifier (ABC) of single layer, which is attached gorithm. The ABC is trained with a class-balanced loss of minibatch, while using high-quality representations learned from all data points in the minibatch using the backbone SSL algorithm to avoid overfitting and information loss. Moreover, we use consistency regularization, recent SSL technique for utilizing unlabeled data in a modified way. to train the ABC to be balanced among the classes by selecting unlabeled data with the same probability for each class. The proposed algorithm achieves state-of-the-art performance in various class-imbalanced SSL. experiments using four benchmark datasets.
representation layer of an existing SSL al-
Introduction
Recently, numerous deep neural network (DNN)- based sumissupee learning (SSL) algorithms have been proposed to improve the performance of DNNs by utilizing unlabeled data when only small amount of labeled data is available. These algorithms have shown effective performance in various tasks. However, most existing SSL algorithms assume class-balanced datasets, whereas the class distributions of many real-world datasets are imbalanced. It is well known that classifiers trained on class-imbalanced data tend to be biased toward the majority classes. This issue can be more problematic for SSL algorithms that use predicted labels of unlabeled data for their training, because the labels predicted by the algorithm trained class-imbale ianslimmalanced data become severely imbalanced [8]. For example, Figure T (b) presents biased predictions of ReMixMatch 3. recent SSL algorithm, trained on CIFAR-10- 10-LT, which is class-imbalanced dataset with the amount of Class 0 being 100 times more than that of Class 9, as depicted in Figure T (a). Although there various class-imbalanced learning techniques, they are usually designed for labeled data, and thus cannot be simply combined with SSL algorithms under cuss-imballanced SSL (CISSL) scenarios. Recently, few CISSL algorithms have been proposed, but the CISSL problem still underexplored.
more
We propose CISSL algorithm that can effectively use unlabeled data, while mitigating class imbalance by using an existing DNN-based SSL algorithm [3] 29 as the backbone and introducing an auxiliary balanced classifier (ABC) of single layer. The ABC is attached to representation
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

1003
1935
1000
Untebeled Diste
Prediction
Prediction
of semster
4002
Labeled Date
1000
Test Set
1000
Test Set
1200
11200
1000
1002
690
600
1002
000
Class
Index
(a) caass-imbllanced training set
(b) ReMixMatch
(c) Proposed algorithm
Predictions on a class-balanced test set using ReMixMatch (b) and the proposed algorithm (c) trained on a laas-iibalaaceed training set (a).
Figure 1:
layer immediately preceding the classification layer of the backbone, based on the argument that classification algorithm (i.e., backbone) can learn high-quality representations even if its classifier biased toward the majority classes [17]. The ABC is trained to be balanced across all classes by using a mask that rebalances the class distribution, similar to re-sampling in previous SSL studies 7 13 16.. Specifically, the mask stochastically regenerates a class-balanced subset of minibatch on which the ABC is trained. The ABC is trained simultancously with the backbone, so that the ABC can use high-quality representations learned from all data points in the minibatch using the backbone. In this way, the ABC can overcome the limitations of the previous resampling techniques, overfitting on minority-clase data or loss of information on majority-clan data 6 27
Moreover, to place decision boundaries in low-density regions by utilizzig unlabeled data, we use consistency regularization, recent SSL technique, which enforces the classification outputs of two augmented or perturbed versions of the same unlabeled example to remain unchanged. In particular, encourage the ABC to be balanced across classes when using consistency regularization by selecting unlabeled examples with the same probability for each class using a mask. Figure [ (c) illustrates that compared the results of ReMixMatch in Figure (b). the class distribution of the predicted labels becomes more balanced using the proposed algorithm trained on the same dataset. Our experimental results under various scenarios demonstrate that the proposed algorithm achieves state-of-the ahrtthe-rt performance. Through qualitative analysis and an ablation study, we further investigate the contribution of each component of the proposed algorithm. The code for the proposed algorithm is available at https///giuub.coo//eeeetuuk////
2 Related Work
Semi-supervised learning (SSL) Recently, several SSL techniques that utilize unlabeled data have been proposed. Entropy minimization [12] encourages the classifier outputs to have low entropy for unlabeled data, as in pseudo-labels (22). Mixup regularization (4 32] makes the decision boundaries farther away from the data clusters by encouraging the prediction for an interpolation of two inputs to be the same as the interpolation of the prediction for each input. Consistency regularization 26 [4 [3) encourages a classifier to produce similar predictions for perturbed versions of the same unlabeled input. To create perturbed unlabeled inputs, various data augmentation techniques have been used. For example, FixMatch [29 and ReMixMatch
3 used strong augmentation methods Augment g FixMatch and ReMix Match are used as the backbone
such as Cutout [10] and
of the proposed algorithm; they are described in Section 3.2
-las--imallanced learning (CIL) As a popular approach for CIL, re-sampling techniques [16 7
2 balance the number of training samples for each class in the training set. As another popular approach, re-weighting techniques (23 [4 33 re-weight the loss for each class by factor inversely proportional to the number of data points belonging to that class. Although these approaches are simple, they have some drawbacks. For example, oversampling from minority classes can cause overfitting, whereas undersampling from majority classes can cause information loss [6]. In the case of re-weight,ng, gradients can calculated to be abnormally large when the class imbalance is severe, resulting in unstable training 66. [. Many attempts have been made to alleviate these problems, such as effective re-weighting 9 and meta-learning-
re-weighting (28 15). New
forms of losses have also been proposed 6 27 In 19), knowledge is transferred from the data of

---

majority classes to the data of minority classes. These CIL algorithms were designed for labeled data and require label information; thus, they are not applicable to unlabeled data. In [17] it was found that biased classification is mainly due to the classification layer and that classification algorithm can learn meaningful representations even from a class-imbalanced training set. Based on this finding.
we design the ABC to use high-quality representations learned from class-imbalanced data utilizing FixMatch [29] and Match [3].
Class-imbalanced semi-supervised learning (CISSL) There have been few studies on CISSL. In
[35. it was found that more accurate decision boundaries can be obtained in class-imbalanced settings through self-supervised learning and semi cemi-super learning. DARP (18] refines biased pseudo- labels by solving a convex optimization problem. CReST [34]. a recent self-traiming technique, mitigates class imbalance by using pseudo-labeles unlabeled data points classified as minority classes with higher probability than those classified as majority classes.
3 Methodology
3.1 Problem setting
Suppose that we have a labeled dataset X' [(5n: (n) : n (1,.., N)), where dn E TRd is the nth labeled data point and y/ E {L,..., - is the corresponding label. We also have an unlabeled dataset U = {(um) m E (1,.... M)). where tmm E IRd is the mth unlabeled data point. We express the ratio of the amount of labeled data as 3 = M+N. Generally, B < 0.5, because label acquisition is costly and laborious. We denote the number of labeled data points of class as N₁, i.e., Σ/= N₁ = N, and assume that the L classes are sorted according to cardinality in descending order, i..., N1 2 N2 2 ... 2 NN. We denote the ratio of the class imbalance as y Nr class-imbalanced scenarios, 7 >> 1. Following previous CIL studies, we define the half of the classes containing a large amount data as the majority classes, and the other half of the classes, containing small amount of data, as the minority classes. Following [34), we assume that X' and U share the same class distribution, ie.. the labeled and unlabeled datasets are class- imbalanced to the same extent. From X and U, we generate minibatches MBx { yb): b (1,..., B)} c X and MBN = {w,)) E (I...., B)} C U for each iteration of training. where B is the minibatch size. Using these minibatches for training, we aim to learn a model f Rd - { that performs effectively class-balanced test set.
N
Ni. Under
3.2 Backbone SSL algorithm
We attach the ABC to the backbone's representation layer, so that it can utilize the high-quality representations learned by the backbone. We use FixMatch [29] or ReMixMatch (3) as the backbone. as these two have achieved statc-of-the-at SSL performance. FixMatch uses the classification loss calculated from the weakly augmented labeled data point (2x) generated by flipping and cropping the image, and the consistency regularization loss calculated from the weakly augmented unlabeled data point a (ux) and strongly augmented unlabeled data point A (us) generated by Cutout 10) and Random Augment [8]. ReMixMatch predicts the class label of the weakly augmented unlabeled data point a (ux) using distribution alignment and sharpening, and assigns the predicted label the strongly augmented unlabeled data point A (us). These strongly augmented unlabeled data point A (ub) and strongly augmented labeled data point A (xx) are used to conduct mixup regularization. ReMixMatch also conducts consistency regularization in manner similar to FixMatch and self- supervised learning using the rotation of the image [ 39]. FixMatch and ReMixMatch have greatly improved the SSL performance by learning high-quality representations using strong data augmentation. However, these algorithms can be significantly biased toward the majority classes in class-imbalanced settings.
Using FixMatch and ReMixMatch as the backbone of the proposed algorithm, we ensure that the ABC enjoys high-quality representations learned by the backbone, while replacing the backbone's s biased classifier. To train the ABC, we reuse the weakly augmented data and strongly augmented data used by the backbone to decrease the computational cost. Although we use FixMatch and Metch as the backbone in this study, the ABC can also be combined with other DNN-based SSL algorithms,, long they use weakly augmented data and strongly augmented data.
3

---

3.3 ABC for class-imbalanced Semi-supervised learning
To train the ABC to be balanced, we first generate 0/1 mask M (ax) for each labeled data point ry using a Bernoulli distribution B (.) with the parameter set to be inversely proportional to the number of data points of each class. This setting makes B generate mask with high probability for the data points in the minority classes, but with low probability for those in the majority classes. Then, the classification loss is multiplied by the generated mask, so that the ABC can be trained with a balanced classification loss. Multiplying the classification loss by the 0/1 mask can be interpreted as oversampling of the data points in the minority classes, whereas can be interpreted as undersampling of those in the majority classes. In representation learning, oversampling and undersampling techniques have shown overfitting and information loss problems, respectively. In contrast, the ABC can overcome these problems because it uses the representations learned by the backbone, which is trained on all data points in the minibatch. The use of the 0/1 mask to construct the balanced loss, instead of directly creating balanced subset, allows the backbone and the ABC to be trained from the same minibatches. Therefore, the representations of minibatches calculated for training the backbone can be used again for training the ABC. Consequently, the proposed algorithm only requires a slightly increased time cost compared to training the backbone alone. This confirmed in Section 4.3 The overall procedure balanced training with 0/1 mask for the ABC attached to a representation layer of the backbone is presented in Figure The classification loss for the ABC, Less with 0/1 mask M (-) expressed
B
Lats =
M (Tb) H (ps (yla (m%)) PP))
(1)
B
b=1
N1 Ng
M (xx) = B
(2)
where H is the standard cross-entropy loss, a (ar)) is an augmented labeled data point, p. (ula (2%)) the predicted class distribution using the ABC for a (2r), and ps is the one-hot label for zp.
Lets
Forword Propagation of Labeled data
In
Gradients from Algarithm
Backbone SSL Algorithm
Augmentedd
Gradients from ABC
Labeled data
Deep Representation
Bad&bon Classifier
Figure 2: Overall procedure for balanced training of the ABC with a 0/1 mask
3.4 Consistency regularization for ABC
To increase the margin between the decision boundary and the data points using unlabeled data, we conduct consistency regularization for the ABC, similar to the way in FixMatch. Specifically, we first obtain the predicted class distribution p. (y (us)) for weakly augmented unlabeled data point (us) using the ABC and use as a soft pseudo-label qo- Then, for two strongly augmented unlabeled data points A₁ (un) and Ag (up). we train the ABC to produce their predicted class distributions, Ps (y (u))) and ps (ylAz (u,)), to be close to 96-
In class-imbalamme imbalanced settings. because most unlabeled data points belong to majority classes, most weakly augmented unlabeled data points can be predicted as the majority classes. Then, consistency regularization would be conducted with higher frequency for the majority classes, which can cause a classifier to be biased toward the majority classes. To prevent this issue, we conduct consistency problems. Specifically,
regularization in modified manner that is suitable for

---

whereas FixMatch minimizes entropy by converting the predicted class distribution for weakly augmented data point into one-hot pseudo-label, we directly use the predicted class distribution as soft pseudo-label. We do not pursue entropy minimization for the ABC because it can accelerate biased classification toward certain classes. Moreover, we once again generate 0/1 mask M (.) for each unlabeled data point 4s based a soft pseudo label qb. and multiply the consistency regularization loss for up by the generated mask, so that the ABC can be trained with class-balanced consistency regularization loss. Note that existing resampling techniques are not applicable to unlabeled data, because they require a label for each data point. contrast, we make it possible to resample unlabeled data by using the soft pseudo-label and the 0/1 mask. The consistency regularization loss, Leon, with 0/1 mask M (.) is expressed as
1 B b=1k=1
Loon
M (ub) I (max (go) 2 r) H (Ps (ylAA (u6)) .%6,,
(3)
N NgO
M (us) B
(4)
where is the indicator function, max (qs) is the highest predicted assignment probability for any class, representing the confidence of prediction, and 7 is the confidence threshold. To avoid the unwanted effects of inaccurate soft pseudo-labels during consistency regularization, we only use the weakly augmented unlabeled data points whose confidence is higher than the threshold т. similar to that in FixMatch. To take full advantage of few unlabeled data points with prediction confidence values that are higher than the confidence threshold T in the early stage of training, we gradually decrease the parameter of the Bernoulli distribution B (+) for uh from 1 N / /Ng, where Tb is the one-hot pseudo-label obtained from qu- Following previous studies 3 24 29 4
we do
not backpropagate gradients for pseudo-label prediction. The overall procedure for consistency regularization for the ABC is shown in Appendix A.
3.5 End-to-en training
Unlike recent CIL trend to finetune a classifier in a balanced manner after representation learning is completed (i.e., decoupled learning of representations and classifier) 27]. we obtain balanced classifier by training the proposed algorithm end-to-end. We train the proposed algorithm with the sum of losses from Sections 3.3 and 3.4 and the loss for the backbone, Lback- The total loss function
Liotal is expressed as
Ltotal = Lote + Lion + Lbbck-
(5)
Whereas we use the sum of the losses for the backbone and ABC for training the proposed algorithm. we predict the class labels of new data points using only the ABC. In our experiments in Sections 4.4 and 4.5) we show that the proposed algorithm trained end-to-end produces better performance than competing algorithms with decoupled learning of representations and classifier, and we analyze possible reasons. We present the pseudo code of the proposed algorithm in Appendix B.
Experiments
4.1 Experimental setup
We created class-imbalanced versions of CIFAR-10, 10, CIFAR-100 (11]. and SVHN 25 datasets to conduct experiments under various ratios of class imbalance 7 and various ratios of the amount of labeled data which the number of data points exponentially decreases from the largest to the smallest class, i.e.,
3. For class-imbalance types, we first consider toiget (LT) imbalance in
N₂ N1 x - +i; where majority classes have the same amount of data and the whole minority classes also have the same amount of data. Two types of class imbalance for the considered datasets are illustrated in Appendix C. For the main setting, we set y = 100, Ni = 1000, and 3 20% for CIFAR-10 and SVHN, and
N. We also consider step imbalance S in which the whole
Ni.
7 = 20, N1 200 and = 40% for CIFAR-100. Similar to 18. we set 7 of CIFAR- to be relatively small because CIFAR-100 has only 500 training data points for each class. To evaluate the
5

---

performance of the proposed algorithm on large-scale datasets, we also conducted experiments on 7.5M data points of 256 by 256 images from the LSUN dataset [37].
We compared the performance of the proposed algorithm with that of various baseline algorithms. Specifically, we considered the following baseline algorithms:
Deep CNN (vanilla algorithm): This is trained on only labeled data with the cross-entropy loss.
BALMS 27
(CIL algorithm): This satte-of-the-ar CIL algorithm does not use unlabeled data.
VAT 24
ReMixxatch [3]. and FixMatch [29 (SSL algorithms): These are state-of-the-ann SSL
algorithms, but do not consider class imbalance.
Fi.Match=CR&SSTPDAA and RcMixMalch+CRRsSTPPDD (CISSL algorithms); CReST+PDA 34 mitigates class imbalance by using unlabeled data points classified as the minority classes with higher probability than those classified as the majority classes.
ReMixMatch+DARR and FixMatch+DARP (CISSL algorithms): These algorithms use DARP to refine the pseudo labels obtained from ReMixMatch or FixMatch.
18
ReMix.aatchDDRRPoRRT and FixMatch+DARP+oR (CISSL algorithms): Compared to ReMix- Match+DARP and FixMatch+DARR, these algorithms finetune the classifier using cRT
For the structure of the deep CNN used in the proposed and baseline algorithms, used Wide ResNet-28-2 [38. We trained the proposed algorithm for 250, 000 iterations with a batch size of 64. The confidence threshold r was set to 0.95 based on experiments with various values of in Appendix D. We used the Adam optimizer (20 with a learning rate of 0.002, and used Cutout [10] and RandomAugment [8] for strong data augmentation, following [8]. Similar to B 4 evaluated the performance of the proposed algorithm using an exponential moving average of the parameters over iterations with decay rate of 0.999, instead of scheduling the learning rate. In Tables
5 we used the overall accuracy and the accuracy only for minority classes as performance measures. We repeated the experiments five times under the main setting, and three times under the step imbalance and other settings of [3 and 7, We report the average and standard deviation of the performance measures over repeated experiments. For the vanilla algorithm, FixMatch+DARP4<RR and ReMixMatch+DARRP4RR which suffered from overfitting, we measured performance every 500 iterations and recorded the best performance. Further details of the experimental setup are described in Appendix E.
4.2 Experimental results
The performance of the competing algorithms under the main setting are summarized in Table We can observe that the proposed algorithm achieved the highest overall performance, with improved performance for minority classes. Interestingly, VAT, an SSL algorithm, showed similar performance to the vanilla algorithm, and worse performance than BALMS, CIL algorithm. Similarly, FixMatch and ReMixMatch, which do not consider class imbalance, showed poor performance for minority classes. Although BALMS mitigated class imbalance, it produced poor overall performance, it did not use unlabeled data for training. This demonstrates the importance of using unlabeled data for train- and
ing, even in the class-imbal imbalancee setting. FixMatch+CResTTHH mitigated class imbalance by using unlabeled data points classified as the minority classes with higher probability, but produced lower performance than the proposed algorithm. This may be because even if all unlabeled data points classified as minority classes are additionally used for training, their amount is still less than that of the data in majority classes, while the proposed algorithm uses class-balanced minibatches by generating the 0/1 mask. Fixmatch+DARP and ReMix- Match+DARP slightly mitigated class imbalance by refining biased pseudo-labels, but resulted in lower performance than the proposed algorithm. This may be because even perfect pseudo labels cannot change the underlying haaasimmalaaneed distribution of the training data. By additionally using rebalancing technique cRT, (RMMaach(ReMixMattee DARRP+RRT performed better than FicMatchdReMMixMMachhhhoo. worse than FixMarch(ReMMixMMachhhh However, (ieMaachh +DARPPoF still performed although it also uses high-quality representations learned by FixMMacchRReMixMMtch and techniques for mitigating class imbalance. The superior performance over
of FixMMachh(eeiixMMacchaaAAA FirMarchReeMixsMccchhaaa was trained end-to-end, and the ABC was also trained using unlabeled data. We discuss this in more detail in Sections 4.4 and 4.5 Overall, the algorithms combined with
is probably because

---

ReMixxatch performed better than the algorithms combined with FixMatch. In addition to the overall accuracy and minority- class-accuracy, we also compared the performance of the competing algorithms in terms of the geometric mean (G-mean) of class-wise accuracy under the main setting in Appendix F.
Table I: Overall accuacy/iinoorty-lla--ccuuu under the main setting
CIFAR-I0-LT
SVHN-LT
CIFAR-100-LI
Algorithm
100, , = 20%
100, 8 = 20%
7 = 20, 8 = 40%
Vanilla
55.321 1.88
77.020.07/ 23.3
10.121.15/ 25.2:0.0
VAT [24]
55.320.8 28.221.55
81.320.07 08.210.88
40.4go.3// 24.820.
BALMS 27
70.720.58/ 69.821.08
87.620.5a/
85.020.07
50.2 54/ 42.9g1.00
FixMatch [29
72.320.3l 53. 829.63
88. 28.02o.a0/
79.4
51.0go m000om////0 32.822..1
w/ CReST+PDA 34
76.6.0.o / 01.44...5
$9.140.09
81.711.18
51.620 20/ 36,4
w/ DARP 18
73.7 98 57. 022.12
88, 19/ 80.5+0.54
51/450a7/ 33.9
w/ DARP+cRT
18
78.1 sa/ 66.621.55
89.920.3a/ 83.510.61
54.720.00 10/ 41.220042
w/ ABC
81.1 /2/ 22.0
92.0 38/ 87.9
56.320.19/ 43.4
ReMixMatch
73.7 55.9
89.8 89.8 82.8+0.66
54.020.2a/ 37.1
w/ CReST+PDA 34
75.7 aa/ 59. 623.76
20.9 85, 29
54.620.4z /// 38.1=0.00
w/ DARP [8
74.4+0.43 41/ 56.9 20.07
30.2+o.22/ / 83, 83.5+0.40
54.5+0.a/
37, 7
w/ DARP+cRT [8
78.5+0.61/ 66, 4+1.68
4 87, 6+0.70
55.1+0.45/
43. 6..60...8
w/ ABC
82.4 75.7=1.18
#3.9+ 92.5
57.6 46.7=0.50 =0.50
To evaluate the performance of the proposed algorithm in various settings, we conducted experiments using ReMixMatch, FixMatch, and the CISSL algorithms considered in Table | while changing the ratio of class imbalance y and the ratio of the amount of labeled data B. The results for CIFAR-10 are presented in Table [ and the results for SVHN and CIFAR- -00 are presented in Appendix G. In Ta we can observe that the proposed algorithm achieved the highest overall accuracy with greatly improved performance for minority classes for all settings. Because FixMatchDDARP+ct and ReMixMaich-DARR-oRR do not use unlabeled data for classifier tuning, the difference in performance between the ratio of the amount of labeled data B decreased and as the ratio of class imbalance y increased. In addition, the difference in performance between Fix MatcchhReMiiMMMcch--RRreTT the proposed algorithm tended to increase as the ratio of class imbalance 7 increased, because the difference between the number of labeled data points belonging to the majority classes and the number of unlabeled data points classified as the minority classes increases with 7.
and the proposed algorithm increased as
and
Table 2: Overall accuracy/minority-olas accuracy for CIFAR-10 under various settings
CIFAR-10-LT
Algorithm
7= 100, 3 == 10%
100, 3 = 30%
7 = 50, B = 20%
? 7= 150, 8 = 20%
FixMatch [2 w/ CReST+PDA
70.020.30/ 18.9
74.910.ss/ 58.221.08
81.220.07/ 70.720.36
68.5 sa/ 45.821.15
34
73.910.40/ 58.9
77.610-75/
64.021139
83.320 10/ 75.720.39
70.0 s2/ 49.421.22
w/ DARP+cRT
18
74.620 59.2a2.12
29/ 67.720.90
43/ 77.1
73.2 so/ 10.157.111n 121.13
w/ ABC
77.2 65.7
21.5 72.9
s1/
80.2
77.1 es/ 64.4:0.92
ReMixMatch CReST+PDA
71.5+0.51 1..2
75.8+0.10/
59.4+0.17
81.5+0.17/ 70.3+0.32
69.9+0.2s/ 23/ 48 4+0.c0
73.8+0.32/ 0..44
78.6#0.75/
64.84+1.4
23
75.4+0.52
71.3+0.m1/ 50. 50.8+1.80
w/ DARP+<RT 18
75.9 1
81 11/0 70.7
84.5
77.8 77.8+1.07
73.9 so/ 57 37.4+1.45
w/ ABC
79.8 =0.8
84.3 80.6
87.5 84,621.19
80.6 =2.1
We also conducted experiments under a step-imbalance setting, where the class imbalance was more noticeable. This setting assumes a more severely imbalanced class distribution than the LT imbalance settings, because half of the classes have very scarce data. The experimental results for CIFAR-10 are presented Table and the results for SVHN and CIFAR-100 are presented in Appendix H. Table 3 we can see that the proposed algorithm achieved the best performance, and the performance margin is greater than that of the LT imbalance settings. ReMiixMach-CCReST--P.. showed relatively low performance compared to the other algorithms.
7

---

Table 3: Overall accuracy/minnrrty-llass accuracy on CIFAR-10 under step imbalance setting
CIFAR-10-Siep. 7 100,
20%
Algorithm
w/-
w/ CR 33
w/ DARP 18
w/ ABC
FixMatch (2
5...0zo.s s4/ 11.8x1
71.1 48. 2:226
69.8 15.1 12:70
o 5..0
ReMixMatch 60.8 /0/ 25. 11.25
64.620.97/ 33.5 23.55
72.3=1.77/ 50.6
1.74 65.7
To evaluate the performance of the proposed algorithm on large-scale dataset, we also conducted experiments on the LSUN dataset (37,, which naturally a long-tailed dataset. Among the algorithms considered in Tables 2 and 3] those combined with CReST were excluded for comparison, because CReST requires loading of the whole unlabeled data in the repeated process of updating pseudo- labels, which not possible for the large-scale LSUN dataset. Instead, we additionally considered Fat and ReMixMatch+<RRT for comparison. The experimental results are presented in Table 4 The proposed algorithm showed better performance than the other baseline algorithms. DARP resulted in degradation of the performance, possibly because the scale of the LSUN dataset very large. Specifically, DARP solves a convex optimization with all unlabeled data points to refine the pseudo labels. As the scale of the unlabeled dataset increases, this optimization problem becomes more difficult solve and, consequently, the pseudo-labels could be refined inaccurately. Unlike the results for other datasets, the algorithms combined with FixMatch performed better than the algorithms combined with ReMixMatch.
Table 4: Overall accuracy/minority class accuracy for the large-scale LSUN dataset
LSUN,
7 100, B 20%
Algorithm
w/- +
w/ cRT 17
w/ DARP [8]
w/ DARP+<RT [8
w/ ABC
FixMatch 29
73.1 / 35.3
77.5
51.8/188 65.6/ / 44.1
75.8 69.5
78.9 75.5
ReMixMatch
69.4 49.1
75.4/ 69.5
72.1/ 67.5
76.9 69.5
4.3 Complexity of the proposed algorithm
The proposed algorithm requires additional parameters for the ABC, but the number of the additional parameters is negligible compared to the number of parameters of the backbone. For example, the ABC additionally required only 0.09% and 0.87% of the number of backbone parameters for CIFAR-10 with 10 classes and CIFAR-100 with 100 classes, respectively. Moreover, because the ABC shares the representation layer of the backbone, it does not significantly increase the memory usage and training time. Furthermore, we could train the proposed algorithm on the large-scale LSUN dataset without a significant increase in computation cost, because the entire training procedure could be carried out using minibatches of data. In contrast, the algorithms combined with DARP required convex optimization for all pseudo-labels, which significantly increased the computation cost as the number of classes or the amount of data increased. Similarly, it required significant time to train the algorithms combined with CReST, because CReST requires iterative re-training with labeled set expanded by adding unlabeled data points with pseudo-labels. We present the floating point operations per second (FLOPS) for each algorithm using Nvidia Tesla-V100 in Appendix I.
4.4 Qualitative analysis of high-quality representations and balanced classification
The ABC can use high-quality representations learned by the backbone when performing balanced classification. To verify this, in Figure we present t-distributed stochastic neighbor embedding (t-SNE) BII of the representations of the CIFAR-10 test set learned by the ABC (without SSL backbone), ABXM and ReMixMatchABCC on CIFAR-10-LT under the main setting. Different colors indicate different classes. As expected, "ABC (without SSL backbone)" failed to leam class-separable representations because sufficient data were not used for training while using the 0/1 mask. In contrast, by training the backbone (FixMatch or ReMix together with the ABC, the proposed algorithm could use the entire data and learn high-quality representations. In this example, Match produced more separable representations than FixMatch, which shows that the choice of the backbone affects the performance of the proposed algorithm, as expected.
8

---

(a)ABC (without SSL backbone)
(c) RcMixMatch+ABBC
Figure 3: t-SNE of the proposed algorithm and the ABC (without SSL backbone)
The proposed algorithm also mitigate class imbalance by using the ABC. To verify this, compare the confusion matrices of the predictions on the test set of CIFAR-10 using ReMixMatch, ReMixMalch+DARP+oRTI, and RcMiMatch+ABBC trained on CIFAR-10 under the main setting in Figure 4 In the confusion matrices, the value in the vth row and the jth column represents the ratio of the amount of data belonging to the ith class to the amount of data predicted as the jth class. Each cell has a darker red color when the ratio is larger. We can see that ReMixMatch often misclassified data points in the minority classes (e.g.. classes 8 and into classes 0 and 1). This may be because ReMix Match does not consider class imbalance, and thus biased pseudo-labels were produced a more balanced flass-distribution compared
used for training. Mathh to ReMixMatch by additionally using DARP+cRT. However, a significant number of data points in the minority classes were still misclassified as majority classes. In contrast, ReMixMatch++ABC classified the test data points in the minority classes with higher accuracy, and produced significantly more balanced class distribution than RcMixMatch+DARP444 DBMiixMatch+DARRPP-
shown in Figure X (c). As both and ReMix use ReMixMatch to learn representations, the performance gap between these two algorithms results from the different characteristics of the ABC versus DARP+cRT as follows. First, DARP+cRT does not use unlabeled data for training its classifier after representations learning is completed, whereas the ABC uses unlabeled data with unbiased pseudo-labels for its training. Second, whereas DARP+cRT decouples the learning of representations and training of classifier, the ABC is trained end-to-end interactively with representations learned by the backhone. We also present the confusion matrices of the predictions on the test set of CIFAR-10 using FixMatch, FixMatch+DARPtoR and ABC as well as the confusion matrices ReMix-
of the pseudo-labels the same dataset using ReMch,attch, ReMixMatch-DARP-o5 Match+ABCC FixMatch, TrxMatch+DARP-eee and AaCc in Appendix J. Moreover, we compare the ABC and the classifier of DARP+cRT in more detail using the validation loss plots in Appendix K.
Proettruund
Proksteduued
Ma)MMMixattch
(c) ABC
Figure 4: Confusion matrices of the predictions on the test set of CIFAR-10
4.5
Ablation study
We conducted an ablation study on CIFAR-10-LT in the main setting to investigate the effect of each element of the proposed algorithm. The results for ReMix
are presented in Table 5. where each row indicates the proposed algorithm with the described conditions in that row. The results are summarized as follows. 1) If we did not gradually decrease the parameter of the Bernoulli distribution B (-) when conducting consistency regularization, then overbalance problem occurred

---

because of unlabeled data misclassified as minority classes. 2) Without consisteney regularization for the ABC, the decision boundary did not clearly separate each class. 3) Without using the 0/1 mask for Lets and Trcon, the ABC was trained to be biased toward the majority classes. 4) Without confidence threshold J for consistency regularization, training became unstable and, consequently, the ABC was trained to be biased toward certain classes. 5) Similarly, if hard pseudo-labels, instead of soft pseudo-labels, were used for consistency regularization, then the ABC was biased toward certain classes. 6) If the ABC was solely used without the backbone, the performance decreased because the ABC could not use high-quality representations learned by the backbone. 7) When used a re-weighting technique [3] instead of mask for the ABC, training became unstable because of abnormally large gradients calculated for training on the data of the minority classes. 8) The decoupled training of the backbone and ABC resulted in decreased classification performance, as was also analyzed in Section 4.4 Similarly, we present the results of the ablation study for AiX in Appendix I..
Table 5: Ablation study for ReMixMatch++ABC on CIFAR-IO-LT, 100,
20%
Ablation study
Overall
Minority
RcMixxMtcch+ABC (proposed ulgorithm)
82.4
75.7
Without gradually decreasing the purameter of B (-) for consistency regularization
81.8
74.6
Without consistency regularization for the ABC
79.4
66.9
Without using the 0/1 mask for the consistency regularization loss Luon
79.0
69.2
Without using the 0/1 mask for the classification loss Lois
74.4
57.8
Without using the confidence threshold for consistency regularization
74.3
75.4
Using hard pscudo labels for consistency regularization Without training backbone (ABC without SSL backbone)
70.2
75.1
68.7
56.2
Training the ABC with a weightttn technique Decoupled training of the backbone and ABC
81.2
74.1
79.5
72.3
5 Conclusion
We introduced the ABC, which is attached to state-of-the-art SSL algorithm, for CISSL. The ABC can utilize high-quality representations learned by the backbone, while being trained to make class- balanced predictions. The ABC also utilizes unlabeled data by conducting consistency regularization in a modified way for class-imbalance problems. The experimental results obtained under various settings demonstrate that the proposed algorithm outperforms the baseline algorithms. We also conducted a qualitative analysis and an ablation study to verify the contribution of each element of the proposed algorithm. The proposed algorithm assumes that the labeled and unlabeled data are class-imbalanced to the same extent. In the future, we plan to release this assumption by adopting a module for estimating class distribution. Deep learning algorithms can be applied many societal problems. However, if the training data are imbalanced, the algorithms could be trained to make socially biased decisions favor of the majority groups. The proposed algorithm can contribute to solving these issues. However, there is also potential risk that the proposed algorithm could be used as tool to identify minorities and discriminate against them. It should be ensured that the proposed method cannot be used for any purpose that may have negative social impacts.
Acknowledgments
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2018R ICB660455,, 2020R 101877777
References
[1] An, J., Ying, L., and Zhu, Y. (2021). Why resampling outperforms reweighting for correcting sampling bias with stochastic gradients. In International Conference on Learning Representations.
[2] Barandela, R., Rangel, F.,, Sánchez, J. S., and Ferri, J. (2003). Restricted decontamination for the imbalanced training sample problem. In Iberoamerican congress on pattern recognition, pages 424-431 Springer.
10

---

[3] Berthelot, D., Carlini, N., Cubuk, E. D., Kurakin, A., Sohn, K., Zhang, H.,, and Raffel, C. (2019a). Remixmatch: Semi- -uperviiee learning with distribution matching and augmentation anchoring. In International Conference Learning Representations.
[4] Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., and Raffel, C. A. (2019b). Mixmatch: A holistic approach to semi-supervised learning. In Wallach, H., Larochelle, H., Beygelzimer, A., FAlcéé-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
[5] Buda, M., Maki, A., and Mazurowski, M. A. (2018). A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249-259)
[6] Cao, K., Wei, C., Gaidon, A., Arechiga, N., and Ma, T. (2019). Learning imbalanced datasets with nabel-ditrrbuttion-aaar margin loss. In Wallach, H., Larochelle, H.. Beygelzimer, A., d' d'Alché-Buc, F. Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
[7] Chawla, N. V., Bowyer, K. W., Hall, L. O., and Kegelmeyer, W. P. (2002). Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357.
[8] Cubuk, D., Zoph, B., Shlens, J., and Le, Q. V. (2020). Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702-703.
[9] Cui, Y., Jia, M., Lin, T.-Y., Song, Y., and Belongie, S. (2019). Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9268-9277
[10] DeVries, T. and Taylor, G. W. (2017). Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 100.::0070
[11] Gidaris, S., Singh, P., and Komodakis, N. (2018). Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations.
[12] Grandvalet, Y. and Bengio, Y. (2005). Semi-supervised learning by entropy minimization. In Saul, L., Weiss, Y., and Bottou, L.,, editors, Advances in Neural Information Processing Systems, volume 17. MIT Press.
[13] He, H. and Garcia, E. A. (2009). Learning from imbalanced data. IEEE Transactions on knowledge and data engineering. 21: 1284-1284
[14] Huang, C., Li, Y., Loy, C. C., and Tang, X. (2016). Learning deep representation for imbalanced classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5375-5384.
[15] Jamal, M. A., Brown, M., Yang, M.-H., Wang. I., and Gong, B. (2020). Rethinking class- balanced methods for long-tailed visual recognition from domain adaptation perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7610-7619.
[16] JAPKOWICZ. N. (2000). The class imbalance problem: Significance and strategies. In Proc. 2000 International Conference on Artificial Intelligence, volume 1, pages 111-117.
[17] Kang, B., Xie, S., Rohrbach, M., Yan, Z., Gordo, A., Feng, J., and Kalantidis, Y. (2020). Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations.
[18] Kim, J., Hur, Y., Park, S., Yang, E., Hwang, S. J., and Shin, J. (2020a). Distribution aligning refinery of labudo-label for imbalanced semi-supervised learning. In Larochelle. H.. Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 14567-14579 Curran Associates, Inc
[19] Kim, J., Jeong. J., and Shin, J. (2020b). M2m: Imbalanced classification via major-to-minor translation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recog- nition, pages 13896-13905.

---

[20] Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. In ICLR, 2015.
[21] Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. Technical report, Department of Computer Science, University of Toronto.
[22] Lee, D.-H. et al. (2013). Pseudo-label: The simple and efficient -umi-supe learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3.
[23] Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. (2013). Distributed represen- tations of words and phrases and their compositionality. In Burges, C. J. C.,, Bottou, I.., Welling, M., Ghahramani, Z.,, and Weinherger, K. Q., editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.
[24] Miyato, T., Maeda, S.-i., Koyama, M., and Ishii, S. (2018). Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 197991997--1993.
[25] Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng. A. Y. (2011). Reading digits in natural images with unsupervised feature learning. In NIPS Workshop. 2011.
[26] Park, S., Park, J., Shin, S.-J., and Moon, I.C.. (2018). Adversarial dropout for supervised and semi-supervised learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
[27] Ren, J., Yu, C., sheng. Ma, X., Zhao, H., Yi, S., and Li, (2020). Balanced meta-sofimax for long-tailed visual recognition. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 4175 4186-4 Curran Associates, Inc.
[28] Ren, M., Zeng, W., Yang, B., and Urtasun, R. (2018). Learning to reweight examples for robust deep learning. In International Conference on Machine Learning, pages 4334 4343. PMLR.
[29] Sohn, K., Berthelot, D., Carlini, N., Zhang. 7., Zhang. H., Raffel, C. A., Cubuk, E. D., Kurakin, A., and Li, C.-L. (2020). Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances Neural Information Processing Systems, 33.
[30] Tarvainen, A. and Valpola, H. (2017). Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
[31] Van der Maaten, L. and Hinton, G. (2008). Visualizing data using t-sne. Journal of machine learning research, 9(11).
[32] Verma, V., Lamb, A., Kannala, J., Bengio, Y., and Lopez-Paz, D. (2019). Interpolation consis- tency training for semi-supervised learning. In Proceedings of the Twenty-Eight International Joint Conference on Artificial Intelligence, IJCAI-19, pages 3635-3641. International Joint Con- ferences Artificial Intelligence Organization.
[33] Wang, Y.-X., Ramanan, D., and Hebert, M. (2017). Learning to model the tail. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
[34] Wei, C., Sohn, K., Mellina, C., Yuille, A., and Yang. F. (2021). Crest: class-rebalancing self- training framework for imbalanced semi-supervised learning. arXiv preprint arXiv:2102.09559,
[35] Yang, Y. and Xu, Z. (2020). Rethinking the value of labels for improving iasaliimbllneeed learning. Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 19290-19301. Curran Associates, Inc.
[36] Yin, X., Yu, X., Sohn, K., Liu, X., and Chandraker, M. (2018). Feature transfer learning for deep face recognition with under-represented data. arXiv e-prints, pages arXiv-1803.
12

---

[37] Yu, F., Zhang, Y., Song, S.. Seff. A., and Xiao. J. (2015). Lsun: Construction of a large-scate scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03366 03365.
[38] Zagoruyko. S. and Komodakis, N. (2016). Wide residual networks. arXiv: 060..0005110.... 14.6
arXiv preprint
[39] Zhai, X., Oliver, A., Kolesnikov. A., and Beyer, L. (2019). S41: supe-supe cemi-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1476-1485.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We included the code and instruction in the supplemental material. We will also upload the code at githup with the copyright after the review process.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1 and Appendix E.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Section 4.2
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We summarize the resources and
FL.PS in Section 4.2 and Appendix
4. If you using existing assets (e.g., code, data, models) curating/releusing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cited the creators of CIFAR-10, SVHN, CIFAR-100, LSUN in Section 4.1
(b) Did you mention the license of the assets? [N/A] In their homepage, the creators of the datasets request to cite their work rather than mentioning the license.
(c) Did you include any new assets either the supplemental material or as a URL? [Yes] We included the code for training the proposed algorithm as the supplemental material. We will also upload it at githup with the copyright after review process.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We used the benchmark datasets cited in Section 4.1
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] We used the benchmark datasets cited in Section 4.1
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "y43o59Ahcif", "reviewer": "Reviewer_zh24", "summary": "This work takes class-imbalanced data distribution into account for semi-supervised learning, which is prevailing in realistic scenarios. The problem setting is reasonable and is under-explored.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed algorithm, ABC, attaches an auxiliary balanced classifier to the representation layer of an existing SSL algorithm. The auxiliary classifier uses representations learned from all data points and is trained on a class-balanced subset so that it can avoid the bias towards the majority classes, thus improve generalization performance.\n\nI have some concerns about this work:\na) This novelty is limited. As the authors point out, the superiority of ABC probably lies in that it is trained using unlabeled data, which is not adopted by previous methods. The performance improvement of ABC seems not surprising.\nb) The failure of class-imbalanced learning (CIL) techniques (i.e., BALMS, LDAM) is obvious because they do not use unlabeled data for training. But what if we combine the existing CIL methods with SSL modules to make use of unlabeled data? I think it is a straightforward solution, and I wonder what is the superiority of ABC compared to such solutions.\nc) The conclusion is too wordy. Some redundant explanations can be deleted to make it more concise.\n\n>>\n\nThanks for the author response. It addresses my concerns on the superior performance for the proposal. I increase my score.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "0.8"}, "extra_text": {}}, {"review_id": "r5BP-8gK_F6", "reviewer": "Reviewer_Jr6h", "summary": "The paper proposes a new algorithm for semi-supervised learning in the class imbalanced setting by adding to the neural architecture an additional classifier head which is trained on the balanced datasets (through resampling).\n", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality.\nThe idea of using random resampling inversely proportionally to class cardinality is rather standard. The architectures with an additional classification head as well as consistency regularization are also known, but their combination is quite interesting.\n\nQuality.\nThe experimental results show that the proposed method improves the classification performance on three real datasets. The ablation study is performed.\n\nClarity.\nThe paper is in places hard to follow. The authors should add the pseudocode summarizing the proposed approach.  Some details that seem to have a non-neglectable impact on the results, like \"gradually decreasing the parameter of B(·)for consistency regularization\" are not strongly motivated and can be easily overlooked. The discussion of results is sometimes laconic.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors measure the classification performance only with accuracy on a balanced datasets and minority accuracy. It would be interesting to see other metrics appropriate for imbalanced data such as G-mean as well.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "i-DevfEH6ro", "reviewer": "Reviewer_dcif", "summary": "This paper deals with the class-imbalanced semi-supervised learning problem. The paper utilizes existing semi-supervised learning algorithms to acquire good representation, and additionally introduces an auxiliary balanced classifier called ABC, which is trained on a class-balanced subset of a minibatch to mitigate the biased results caused by class-imbalance. The ABC is trained using not only the balanced subset but also unlabeled data with consistency loss. The experimental results show that the proposed method outperforms existing methods by large margin.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper deals with a practically very important problem of class-imbalanced semi-supervised learning. I found the proposed method was reasonable and technically sound. Although the novelty of the method may not be outstanding given the prior works of [17], [3] and [4], the paper presents a good extension of these prior works to the practically important problem. The experiment is basically thorough and convincing. In addition, the paper is clearly and well written, thus easy to follow. \n\nOverall, I think this is a good paper and has merit to the community, but I have some concerns and questions as follows.\n\n* Is there any specific reason of using mask based on Bernoulli distribution to create the balanced subset? I think it is more straightforward to sample the same amount of data in each class to create balanced subset. For example, how about sampling data from uniform distribution using Gumbel softmax?\n* According to Table 4, incorporating DARP seems harmful. Do authors have any thoughts on the reason of this phenomenon? Moreover, given this phenomenon, I think it is more reasonable to compare the proposed method with “w/ cRT” variant rather than “w/ DARP+cRT” variant in Table 1-3.\n* L316-319: I am skeptical about the contribution of end-to-end training to the accuracy improvement. Looking at Table 5, the techniques related to consistency regularization, or using unlabeled data to train ABC, has dominant effect. For example, the accuracy drops to the same level as an existing method if the consistency regularization is removed; the variant of “Without using the 0/1 mask for the consistency regularization loss $L_{con}$” achieves overall accuracy of 79.0, which is close to 78.5 achieved by an existing CISSL method, namely ReMixMatch w/ DARP + cRT. \n\nMinor comments and questions:\n* L137-138: Is $\\frac{N_{y_b}}{N_L}$ a mistake for $\\frac{N_L}{N_{y_b}}$? The same doubt on L163-164, too.\n* L163-164: How is $N_{\\hat{q_b}}$ calculated? Is it calculated mini-batch wise?\n* L252: “CRT” -> “cRT”\n* L293: “ABC only (without backbone)”. Does this mean a single fully connected layer is attached directly on image input (I believe not)? The same question to L335-337, too. Please clarify the exact architecture in this condition.\n* Table 5: Does each row mean the proposed algorithm + the condition in each row or does it mean all the conditions described earlier are also applied? For example, does “Without consistency regularization for the ABC” mean “Without gradually decreasing the parameter of B () for consistency regularization AND consistency regularization for the ABC”?\n* [17] is published at ICLR 2020, not 2019\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I believe the authors addressed these aspects in section 5.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours"}, "extra_text": {}}, {"review_id": "TVlJLV__6N4", "reviewer": "Reviewer_2qyx", "summary": "The authors introduce a semi-supervised learning algorithm (SSL) for class-imbalanced scenarios.  In particular, an Auxiliar Balanced Classifier coupled with a 0/1 mask is used to favor the cost regularization. The methodology is interesting, and the experiments demonstrate that the proposed method outperforms state-of-the-art techniques on image classification tasks, even for large datasets under challenging imbalance ratios.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Though the paper does not present an outstanding theoretical advance, the methodology seems to benefit real-world tasks for semi-supervised and imbalanced datasets. Overall, the manuscript is easy to follow, holding careful experimentation and method comparison.\n\nPros. \n\n-A consistency regularization approach is coupled within a deep learning framework for image classification under unlabelled data.\n\n-In general, deep learning models treat class-imbalanced-datasets problems only from labeled data; this algorithm uses unlabelled data during training.\n\n-The experiments include several scenarios (easy and complex), demonstrating that the author's proposal outperforms the included literature.\n\n-An ablation analysis is presented, describing in detail the role of each of the algorithm stages.\n\nCons. and comments\n\n-The mathematical foundation could be enhanced. In particular, the threshold hyperparameter tau seems to be crucial for the mask. \nHowever, neither experimental, not theoretical insights are provided.\n\n-Please number all the equations in the manuscript.\n\n-Please take care with the counters notations in lines 89-92. The authors mix i, n, and m indexes.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The proposed method is helpful for several artificial intelligence applications, in particular, scenarios including semi-supervised and imbalanced datasets. Then, societal impact, though not directly referenced, is clear concerning the presented approach.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for 

This paper has four favorable reviews, describing beneficial methodology for semi-supervised learning for imbalanced data, and good experimental results for real data sets. In contrast, a reviewer pointed out that the combination of the existing class-imbalance learning (CIL) and semi-supervised learning is straightforward, and therefore the novelty is limited. The authors argue that it is not straightforward to utilize unlabeled data in CIL and that there are no such studies in the past. However, related research is not entirely absent; for example, 
T. Iwata, A. Fujino, N. Ueda, "Semi-supervised Learning for Maximizing the Partial AUC," proc. of AAAI 2020 has already been proposed. 
The authors should compare the proposed algorithm not only with CIL and naive combined methods, but also with other existing related works. 
Decision: Accept (Poster)