Imbalanced Semi-supervised Learning with Bias Adaptive Classifier
Authors: Renzhen Wang
Xixi Jia
Quanziang Wang
Yichen Wu
Deyu Meng
Venue: ICLR 2023 poster (2023)
Keywords: semi-supervised learning
weakly-supervised learning

Abstract: Pseudo-labeling has proven to be a promising semi-supervised learning (SSL) paradigm. Existing pseudo-labeling methods commonly assume that the class distributions of training data are balanced. However, such an assumption is far from realistic scenarios and thus severely limits the performance of current pseudo-labeling methods under the context of class-imbalance. To alleviate this problem, we design a bias adaptive classifier that targets the imbalanced SSL setups. The core idea is to automatically assimilate the training bias caused by class imbalance via the bias adaptive classifier, which is composed of a novel bias attractor and the original linear classifier. The bias attractor is designed as a light-weight residual network and learned through a bi-level learning framework, which enables the bias adaptive classifier to fit imbalanced training data, while the linear classifier can provide unbiased label prediction for each class. We conduct extensive experiments under various imbalanced semi-supervised setups, and the results demonstrate that our method can be applied to different pseudo-labeling models and is superior to current state-of-the-art methods.

Published as a conference paper at ICLR 2023
IMBALANCED SEMI-SUPERVISED LEARNING WITH BIAS ADAPTIVE CLASSIFIER
Renzhen Wang's Xixi Jia?, Quanziang Wang', Yichen Wu², Deyu Meng ai'an Jiaotong University. "Xidian University, "City University of Hong Kong 4 Maacau University of Science and Technology, *Peng Cheng Laboratory {rzwang, dymeng @ma xjtu. edu.cn
ABSTRACT
Pseudo-labeling has proven to be a promising semi-supervised learning (SSL) paradigm. Existing pseudo-labeling methods commonly assume that the class dis- tributions of training data are balanced. However, such an assumption is far from realistic scenarios and thus severely limits the performance of current pseudo- labeling methods under the context class-imbalance. To alleviate this problem, we design a bias adaptive classifier that targets the imbalanced SSL setups. The core idea is to automatically assimilate the training bias caused by class imbalance via the bias adaptive classifier, which is composed of a novel bias attractor and the original linear classifier. The bias attractor is designed as a light-weight residual network and optimized through bi-level learning framework. Such learning strategy enables the bias adaptive classifier to fit imbalanced training data, while the linear classifier can provide unbiased label prediction for each class. We con- duct extensive experiments under various imbalanced semi-supervins setups, and the results demonstrate that our method can be applied to different pseudo-labeling models and is superior to current saate-ofthhe-aeee the-art methods.
INTRODUCTION
Semi-supervised learning (SSL) (Chapelle al. 2009) has proven to be promising for exploiting un- labeled data to reduce the demand for labeled data. Among existing SSL methods, pseudo-labeling (Lee et al. 2013), using the model's is class prediction as labels train against, has attracted increasing attention in recent years. Despite the great success, pseudo-labeling methods are commonly based on a basic assumption that the distribution of labeled and/or unlabeled data are Mlass-balanced Such an assumption is too rigid to be satisfied for many practical applications, as realistic phenomena al- ways follows skewed distributions. Recent works (Hyun et al.. 2020 Kim et al.) (0200)) have found that class- iass-imbalance significantly degrades the performance of pseudo-labeling methods. The main reason is that pseudo-labeling usually involves pscudo-label prediction for unlabeled data, and an initial model trained on imbalanced data easily mislabels the minority class samples the majority ones. This implies that the subsequent training with such biased pseudo-labels will aggravate the imbalance of training data and further bias the model training.
To address the affreementione issues, recent literature attempts to introduce pseudo-label re- balancing strategies into existing pseudo-labeling methods. Such re-balancing strategy requires the class distribution of unlabeled data as prior knowledge (Wei et al.] 2021 Lee et al. 2021) or needs to estimate the class distribution of the unlabeled data during training (Kim et al. 2020a Lai
et al.) 2022). However, most of the data in imbalanced SSL are unlabeled and the pseudo-labels estimated by SSL algorithms are unreliable, which makes these methods sub-optimal in practice, especially when there are great class distribution mismatch between labeled and unlabeled data.
In this paper, we investigate pseudo-labeling SSL methods in the context of flass-ibbllace,, in which class distributions of labeled and unlabeled data may differ greatly. In such general scenario, the current ofate-of-thh-arr FixMhtch (Sohn et al. 2020) may suffer from performance degradation. To illustrate this, we design an experiment where the entire training data (labeled data + unlabeled
"Correpooniing author.

---

Published as a conference paper at ICLR 2023
022
samples
Predictdd class dis-
0.000
0.00
6.09
6.0
Class Indrex
Class
Clesk index
(a) Class distribution
(h) Per-class recall
(c) Predicted class distribution
Figure 1: Experiments on CIFAR-IO-LT. (a) Labeled set is class-ibbalaneed with imbalance ratio
y 100, while the whole training data remains balanced. Analysis (b) per-class recall and (c) predicted class distribution for Upper bound (trained with the whole training data with ground truth labels), Lower bound (trained with the labeled data only), FixMatch and Ours the balanced test set. Note that predicted class distribution is averaged by the predicted scores for all samples.
data) are balanced yet the labeled data are imbalanced, as shown in [a). Then, we can obtain an upper bound model by training the classification network on the whole training data (the underlying true labels of unlabeled data are given during training) and a lower bound model (trained with the labeled data only). As shown Fig. (b))), the performance of FixMatch much worse than the upper bound model, and degrades significantly from the majority classes to the tail classes, which instantiates the existence of imbalance bias. Mcanwhile, for the last two tail classes, the performance of FixMatch is even worse than the lower bound model, which indicates the existence of pseudo- label bias brought by inaccurate pseudo-labels and it further deteriorates the model training.
To address this problem, propose learning adapt classifier (L2AC) framework to protect the linear classifier of deep classification network from the training bias. Specifically, we propose a bias adaptive classifier which equips the linear classifier with bias attractor (parameterized by resid- ual transformation). The linear classifier aims to provide an unbiased label prediction and the bias attractor attempts to assimilate the training bias arising from class imbalance. To this end, we learn the L2AC with a bi-level learning framework: the lower-level optimization problem updates the modified network with bias adaptive classifier over both labeled and unlabeled data for better repre- sentation learning: the upper-level problem tunes the bias attractor over an online class-balanced set (re-sampled from the labeled training data) for making the linear classifier predict unbiased labels. As a result, the bias adaptive classifier can not only fit the biased training data but also make the linear classifier generalize well towards each class (i.e., tend to equal preference to each class). In Fig. [c). we show that the linear classifier learned by our L2AC can well approximate the predicted class distribution of the upper bound model, indicating that I 1.2C obtains an unbiased classifier.
In summary, contributions are mainly three-fold: (1) We propose to learn a bias adaptive classi- fier to assimilate online training bias arising from class imbalance and pseudo-labels. The proposed L2AC framework is aggostic and can be applied to various pseudo-labeling SSL methods; (2) We develop a bi-level learning paradigm to optimize the parameters involved in our method. This allows the online training bias to be decoupled from the linear classifier such that the resulting network can generalize well towards each class. (3) We conduct extensive experiments on various imbalanced SSL setups, and the results demonstrate the superiority of the proposed method. The htwp:///www.....eeee////aa/iiiia
source code is made publicly available at
2 RELATED WORK
iaalane learning attempts to learn the models that generalize well to each classes from
imbalanced data. Recent studies can be divided into three categories: Re-sampling (He & Garcia
2009 Chawla et al. 2002 Buda et al. 20188 Byrd & Lipton 2019) that samples the data to rearrange the class distribution of training data; Re Re-weighting (Khan et al. [017]] Cui et al.) 2019 Cao et al.)
2019 Lin et al. 2017 Ren et al. 2018 Shu et al. 2019 Tan et al. 2020 Jamal et al.
2020 that
assigns weights for each class or even each sample to balance the training data; Transfer fer learning (Wan et al.) 2017 Liu et al. 2019 Yin et al. 2019 Kim et al., 2020b Chu et al., 2020 Liu et al.,
2020 Wang al. 2020) that transfers knowledge from head classes to tail classes. Besides, most
recent works tend to decouple the learning of representation and classifier (Kang et al., 2020 Zhou
et al. 2020: Tang et al. 2020 Zhang al. 2021b). However, it is difficult directly extend these
2

---

Published as a conference paper at ICLR 2023
techniques to imbalanced SSL, the distribution of unlabeled data is unknown and may be greatly different from that of labeled data.
Semi-supervised learning targets to learn from both labeled and unlabeled data, which includes two main lines of researches, namely aseudo-labeling and consistency regularization. Pscudo-labeling (Lee al. 2013] Xie et al.] 22200a by [Shnn al., 2020 Zhang al., 2021a) is evolved from tropy minimization (Grandvalet & Bengio 2004) and commonly trains the model using labeled data together with unlabeled data whose labels are generated by the model itself. Consistency regular-
ization (Sajjadi et al. 2016,] Tarvainen & Valpola 2017 Berthelot et al. 2 MMyato et al. 2018 Berthelot al., 2019g) aims to impose classification invariance loss on unlabeled data upon pertur- bations. Despite their success, most of these methods are based on the assumption that the labeled and unlabeled data follow uniform label distribution. When used for class-imbalance, these methods suffer from significant performance degradation due to the imbalance bias and pseudo-label bias.
Imbalanced semi-supervised learning has been drawing extensive attention recently. Yang & Xu
(2020) pointed out that SSL can benefit dlass-imbalanced. learning. Hyun et al. (2020) proposed
suppressed consistency loss to suppress the loss on minority classes. Kim et al. (2020a) introduced
convex optimization method to refine raw pseudo-labels. Similarly, Lai et al. (2022) estimated the
mitigating vector to refine the pseudo-labell, and Oh et al. (2022) proposed to blend the pseudo-
labels from the linear classifier with those from
siiillarttybbssee classifier. Guo & Li (2022)
found a fixed threhold for pseudo-labeled sample selection biased towards head classes and in turn proposed optimize an adaptive threhold for each class. Assumed that labeled and unlabeled data share the same distribution, Wei et al. (2021) proposed re-sampling method to iteratively refine the model, and Lee al. (002)) proposed an auxiliary classifier combined with re-sampling technique to mitigate class imbalance. Most recently, Wang et al.] (2022) proposed to combine counterfactual reasoning and adaptive margins to remove the bias from the pseudo-labess. Different from these methods, this paper aims to learn an explicit bias attractor that could protect the linear classifier from the training bias and make it generalize well towards each class.
METHODOLOGY
3.1 PROBLEM SETUP AND BASELINES
Imbalanced SSL involves labeled dataset D: {Xx,, Yn) (n) and an unlabeled dataset D., {xm } m== where x, is a training example and yn E {0, 1) is its corresponding label. We denote the number of training examples of class l within Dj and D.,, as Nr and Ms, respectively. In class- imbalanced scenario, the class distribution of the training data is skewed, namely, the imbalance ratio >> 1 always holds. Note that the class distribution of Du, i... {M Jk= is usually unknown in practice. Given Dj and D.., our goal is to learn a classification model that is able to correctly predict the labels of test data. We denote a deep classification model V f foxt with the feature extractor faxt and the linear classifier fols, where @ and & the
n := max; N >> 1 or Tu := max; Ms
min, N₁.
ming M,
1
parameters of f and foh respectively, and is function composition operator.
With pseudo-labeling techniques, current state-of-the-arn SSL methods (Xie et al., 2020b Sohn ct al. 2020; Zhang et al., 202 generate pseudo-labels for unlabeled data to augment the training datasel. For unlabeled sample Xm. its pseudo-label 8m. can be a 'hard' one-hot label (Lee et al. 2013 Sohn et al. 2020 Zhang al. 202 or sharpened 'soft' label (Xie et al., 2020a Wang al. 2021). The model is then trained on both labeled and pseudo-labeled samples. Such learning scheme is typically formulated as an optimization problem with objective function L = Li + Lus where X, is tlyper-paramete for balancing labeled data loss L. and pseudo-labeled data loss Lu-- To be more specific, C; P.. Ex,eD, H (VV(x))
where Dr denotes a batch of labeled data sampled from D,, H is cross-entropy loss; Lu 2. ExmeD. (mmax(pm) 2 T) ( (X(m) Fm)), D..
where Pm = softmax( V (x,,)) represents the output probability, and 7 is a predefined threshold for masking out inaccurate ageudo-labelle data. For simplicity, we reformulate L as
1
Σ H(V(x).
Σ Xx)()x)))))..
+ [
D
x.eD
X.ED.
where X; = A. 1 (max(p;) 2 r). The pseudo-labeling framework has achieved remarkable success in standard SSL scenarios. However, under clas--imbalanced setting, the model is easily biased
3

---

Published as a conference paper at ICLR 2023
Learning Adapt Classifier
Bias Adaptive Classifier
F F
Biu Do
fo
f
Afw
(a))
Forward Eq. (3)
Backward by Eq. (4)
8
Forward by Eq. (5)
ycle
Backward-ond cawpaardd-o-bbwwwrrre by Eq. (6)
The schematic illustration of the bi-level learning framework of our method, which includes four main steps: Forward process to compute the lower level loss in Eq. 33: ii) Backward process to update the classification network parameters d(w), where w are variables used for parameterizing ø: (iii) Forward process to compute the upper-level loss in Eq. (5); iv)
Figure 2: Learning to adapt classifier for imbalanced SSL. Left:
update Right: The proposed bias adaptive classifier, which consists of the original linear classifier fals and a bias attractor Af.-
during training, such that the generated pseudo-labels labels can be even more biased and severely degrades the performance of minority classes. Moreover, due to the confirmation bias issue (Tarvainen &
Valpola 2017 Arazo et al. 2020), the model itself hard to rectify such training bias.
3.2
LEARNING TO ADAPT CLASSIFIER
Our goal enhance the existing pseudo-labeling SSL methods by making full use of both labeled and unlabeled data, while protecting the linear classifier from the training bias (imbalance bias and pseudo-label bias). To this end, we design to learn a bias adaptive classifier that equips the linear classifier with a bias attractor in order to assimilate complicated training bias.
The proposed bias adaptive classifier: As shown in Fig. 2 The bias adaptive classifier (denoted as F) consists of two modules: the linear classifier fele and nonlinear network A&w. (dubbed bias attractor). The bias attractor is implemented by imposing a residual transformation on the output of the linear classifier, i.e., plugging A fw after fel: and then bridging their outputs with shorteut connection. Mathematically, the bias adaptive classifier I can be formulated as
( = (I Afw) o (2)
(2)
where z = fo (x) IRd, denotes identity mapping, and fu with parameters ws is multi-layer perceptron (MLP) with one hidden layer in this paper. We design such a bias adaptive classifier for the following two considerations. On the one hand, the bias attractor fo adopts nonlinear network which can assimilate complicated training bias in theory due the universal approximation properties Since the whole bias adaptive classifier (i.e., classifier with the bias attractor) is required to fit the imbalanced training data, we hope the bias attractor could indeed help the linear classifier to learn the unbiased class conditional distribution (i.c., let the classifier less effected by the biases.). By contrast, in the original classification network, single linear classifier required fit biased training data such that it is easily misled by class imbalance and biased pseudo-labels. On the other hand, the residual connection conveniently makes the bias attractor plug-in module, i.e., assimilate the training bias during training and be removed in the test stage, and it also has been proven successful easing the training deep networks (He et al., 2016 Long et al. 2016).
Learning bias adaptive classifier: With the proposed bias adaptive classifier FL.s- the modified classification network can be formulated Eे = Fu.e fext, To make full use of the whole training data (D, U D..) for better representation learning,
minimize the following loss function:
L
D
Σ H(((x)),))+
Duul
Σ x.(((x))...).
(3)
x,E
In theory, an ML.P approximate almost any continuous function Homik al. 1989
4

---

Published as a conference paper at ICLR 2023
This involves optimization problem with respect to three parts of parameters {8, o, w) which can be jointly optimized via the stochastic gradient decent (SGD) in an end-to-end manner. However, such a training strategy poses critical challenge: there is no prior knowledge on fg to predict unbiased label prediction and on Afw to assimilate the training bias. In other words, we cannot guarantee the training bias to be exactly decoupled from the linear classifier fo To address this problem, we take w as hyper-parametent associated with & and design a bi-level learning algorithm to jointly optimize the network parameters { and
w.
We illustrate the process in Fig. and Algorithm In each training iteration t, we update the network parameters {0, ) by gradient descent (0'-1, o (w)) = (0', ""' -"o'")-Voo.dh V.
(4)
where is the learning rate. Note that herein assume that w is only directly related to the linear classifier fs via o (w)) which implies that the subsequent optimization of w will not affect the
feature extractor foxt, We then tune the hyper-paramette w to make the linear classifier g++I(w generalize well towards each class, and we thus minimize the following loss function of the network V/ = fo (w) 1 over class-balanced set (dynamically sampled from the labeled training set);
Chal
H(S (w) Sg (w)
(x:),
B
X
where B C Dr is a batch of class-balanced labeled samples, which can be implemented by class- aware sampling (Shen al. 2016). This loss function reflects the effect of the hyper-parameter w on making the linear classifier generalize well towards each class, we thus optimize w by
wh+1 w - 1/ Chol
(6)
where is the learning rate on w. Note that in Eq. (6) we need to compute second-order gra- dient Chal with respect to w, which can be easily implemented through popular deep learning frameworks such as Pytorch (Paszke et al., 2019) in practice.
In summary, the proposed bi-level learning framework ensures that 1) the linear classifier fal can fit unbiased class-conditional distribution by minimizing the empirical risk over balanced data via Eq. (5) and 2) the bias attractor Afw can handle the implicit training bias by training the bias adaptive classifier Fu.s over the imbalanced training data by Eq. (3). As such, our proposed Afw protect fuls from the training bias. Additionally, our L2AC is more efficient than most existing bi-level learning methods (Finn et al. 2017 Ren et al. 2018 Shu et al.) 2019). To illustrate this, we herein give brief complexity analysis of our algorithm. Since our L2AC introduces bi-level optimization problem, it requires one extra forward passes in Eq. (5) and backward pass in Eq. (6) compared to regular single-level optimization problem. However, in the backward pass, the second-order gradient of w in Eq. (6) only requires to unroll the gradient graph of the linear classifier sgle As a result, the backward-on backward automatic differentiation in Eq. (6 demands a lightweight of overhead, i.c.,
approximately # (arams(/s))) x training time of one full backward pass.
#Porams(V) W)
3.3 THEORETICAL ANALYSIS
Note that the update of the parameter w aims to minimize the problem Eq. ( we herein give brief debiasing analysis of our proposed L2AC by showing how the value of bias attractor (-(-) change with the update w. We use notation Bw Ia to denote the gradient operation of ʃ at wt and superscript T to denote the vector/matrix transpose. We have the following proposition.
of
Proposition 3.1 Let pl denote the predicted probability x;,
x,, then Eq.
66 can be rewritten as
1 Σ
dAʃi
wlt w² + Tj(
G,
8w 1
12
where G; = A(p.-y)) 06 1ot -
(5++)), which represents the similarity between the gradient of the sample x; and the average gradient of the whole balanced set B.
Sch" (9.6)
de
This shows that the update of the parameter affects the value of the bias attractor, i.e the value of A is adjusted according to the interaction G, between x; and B. If G. > then Afr is increased, otherwise A fi. is decreased, indicating I.2AC adaptively assimilate the training bias.
5

---

Published as a conference paper at ICLR 2023
Table I: Comparison results on CIFAR-10 under two typical imbalanced SSL settings, i.c. Y
T = %a and # 7u (11 = 100). The performance (bACC/ / GM) is reported in the form of tstd across three random runs.
CIFAR- (0 %v)
CIFAR-10
%r)
Methods
100
150
7uu
(uniform)
7u 100 (reversed)
Vanilla
68.8 / 11.00...11
65.6_o.43
14.0
58.810.13 /5.00.1/51.0001111
58.810 13 51.010.11
w/ Re Ra-salnging
55.8_0.47 / 43. 10.30
52.2_0.03 / / 21.49
55.8_..47 / 10.30
55.820.47 45. 10.30
w/ LDAM--RRW
62.8-o.17 / 58, 2:060
57.9-0.20 /30.440.30
62.8=a.17
58, 940..0
62.8+a.17/ 58. 38.940.00
w/ cRT
63.2_a.45 / 10.49
59.3_o.10 554.610.72
63.21...5 / 19.910.40
63.210.45 45 13.9910.
MixMatch
24.8_ / 49. 2..00
62.5-n.31 /42.5+1.6
41.32a.76 / 12.0+1.34
47.920.09 / /2.550..5
w/ DARP
67.9-0.14 / 61, 60.1
65.8-n.52 55..5+2.0
86.7=0.50 / 20.82
T2.9+0.24 77.000.33
w/ SaR
66.8_o.92 / 59.9 11.32
68.41a.20 12.01
65.5 o1 10.92
w/ DASO
69.8-1.10 / 69.3
66.5_1.99 25.4
75.5 74.64.0.7
65.7
62.041.22
w/ ABC
75.7=0.76 47477+0.47
68.5=n..0
56.44+..0
72.1=0.3/ 41..2+4..4
62.9+0
59.9+0.60
w/ L/A (ours)
76.6
/ 75.77 1..08
72.1 _o.62 / 10.33..
87.2 16.71.0.0
74.0 22 12.911.01
FixMaich
71.5=0.72 / 166666 66.8
68.4 / 59.99
68.9
422.8
65. 65.5 / 226 26..000.44
w/ DARP
75.5-0.05 / 773.0
70.4=0.25 64.9940.17
85.4 / 85.0+0.65
74.9 51 72.3+1.18
w/ CReST+
77.5_a.15/ / 10.11
72.1_0.74 68.911.29
N/A
N/A
w/ SaR
77.6=a.42 #0.9
71.5=n.23 /66.9
85.9 / 85.3
78.3 76. 16.12
w/ DASO
78.3_ / 10.5
74.6_0.74 771.710.52
87.9 / 87.710.44
79.5 91 78.990.
w/ ABC.
80.2
78.991..9
74.7
12.2
81.3 / 10.2
70.3 67.910.70
w/ L2CC (ours)
82.1 0..7
81.550..44
77.6=0.53
/aeaaaa/777.888 75.8
89.5 89.2
82.2
81..711.3
4 EXPERIMENTS
We evaluate our approach on four benchmark datasets: CIFAR- 10, CIFAR- 100 (Krizhvsky et al., 2009), STL-10 (Coates et al. 2011) and SUN397 (Xiao et al. 2010), which are broadly used in imbalanced learning and SSL tasks. We adopt balanced accuracy (bACC) (Huang et al. 2016 Wang et al. 2017 and geometric mean scores (GM) (Kubat et al. 1997 Branco et al. 2016 as the evaluation metrics. We evaluate our L2AC under two different settings: 1) both labeled and unlabeled data follow the same class distribution, i.e., y == 7t = Tui 2) labeled and unlabeled data have different class distributions, i.e., 7u * Tu. where Tu is commonly unknown.
4.1 RESULTS ON CIFAR-10
Dataset. We follow the same experiment protocols as Kim et al. (2020a). In detail, a labeled set and an unlabeled set are randomly sampled from the original training data, keeping the number of images for each class to be the same. Then both the two sets are tailored to be imbalanced by randomly discarding training images according the predefined imbalance ratios 7 and Tu- We denote the number of the most majority class within labeled and unlabeled data as N2 and M,. respectively, and we then have N₂- = N1 -2,² and M2 M1 72% where E₁ k=}. We initially set N1 = 1500 and M, 3000 following Kim al. (2020a), and further ablate the proposed L22CC under various labeled ratios in Section 4.4 The test set remains unchanged and class-balancee
Setups. The experimental setups are consistent with Kim al. (2020a). Concretely, we em- ploy Wide ResNer-28-2 (Oliver et al. 2018) as our backbone network and adopt Adam optimizer (Kingma & Ba 2015) for 500 training epochs, each of which has 500 itcrations. To evaluate the model. we use its exponential moving average (EMA) version, and report the average test accuracy of the last 20 epochs following Berthelot et al. (011b)). See Appendix D.I for more details.
Results under 7/2 Yu- We evaluate the proposed L2AC based on two widely-used SSL meth- ods: MixMatch (Berthelot et al.] (019b) and FixMatch (Sohn et al. (2020), and compare it with the following methods: 1) The Vanilla model merely trained with labeled data; 2) Recent re-balancing methods that are trained with labeled data by considering class imbalance, including: Re-sampling
(Jakkowic7 2000). I.DAM-DRW (Cao et al., 2019) and cRT (Kang et al. (022)): 3) Recent imbal- anced SSL methods, including: DARP (Kim ct al. 2020a), CReST+ (Wei et al. 2021). ABC (Lee ct al. 202T), SaR (Lai et al., 2022) and DASO (Oh al. 2022) Please refer to Appendix C for more details about these methods. The main results are shown in Table I It can be observed that L2AC significantly improves MixMatch and FixMatch at least 9% absolute gain bACC and at least 14%
bACC and GM are defined as the arithmetic and geometric mean over class-wise sensitivity, respectively.

---

Published as a conference paper at ICLR 2023
Table 2: Comparison results on CIFAR-100 and STL-10 under two different imbalance ratios. The performance (bACC / GM) is reported in the form of mean:istd across three random runs.
CIFAR- (11 %a
STI.-10 (%v =N/A)
Methods
"/ = 10
2 20
11
1t 20
FixMatch
55.110.00 10:53
49.51..38 / /34.21101
69.610.60
662.611.1
65.5_o.05 / 26 166.010.d0 10.44
w/ DARP
56.3 25
40.72
50.2+a.18 / 36 200000
72.9+0.21 69.540.18
74.9=0.51 / /7233+1.1
w/ ABC
58.2.0.08
20.25
53.1
42. =0.2
78.2
77.300.30
72.7=0.08
770.6 20.6
w/ DASO
58.3 10.59/ 50.3
51
41..00000
53.0g0.27
33.55 1..45
78.2:0.63
77. 17.55
75.420.81
/74.441000
w/ T/2. (ours)
57.8
52. =0.31
52.640.13
43.0+0.45
79.9
79. +0.49
77.0=a.65 775.8=00.0
Table 3: Comparison results on large-scale SUN397. The performance (bACC / GM) is reported in the form of meantstd across three random runs.
Methods
bACC/GM 38.3
Methods
BACC/GM
Vanilla
22..9
DARP Kim al.
2020a
45.5 / 37 137.540.04
cRT (Kang al. 2020
39.3 / /33.7
ABC
Lee et al. 2021
27.0 / /3..2
FixMateh SShnn al. 2020
44.9 /35.720.66
(our))
28.8 19 40.6
on GM for all settings. This implies that our L2AC benefits the two baselines by learning an unbi- ased linear classifier. Moreover, our L2AC consistently surpasses all the comparison methods over both evaluation metrics. Take the extremely imbalanced case of = 150 for example, compared with the second best comparison method, our L2AC achieves up 3.6% bACC gain and 4.9% GM gain upon MixMatch, and around 3.0% bACC gain and 3.6% GM gain upon FixMatch.
Results under 7 * Yu- The class distribution of labeled and unlabeled data can be arguably different in practice. We herein simulate two typical scenarios following Oh et al. (2022 i.e., the unlabeled set follows an uniform class distribution (7u, = 1) and reversed long-tailed class distribution against the labeled data (7 = 100 (reversed)). Note that CReST+ (Wei et al. 2021) fails this case as they require the class distribution of unlabeled data as prior knowledge for training. As shown in Table L2AC can consistently improve both MixMatch and FixMatch by large margin. An interesting observation is that the baselines MixMach and FixMatch under 7/- = perform much worse than that under 7v- 100, even if more unlabeled data are added for % 1. This is mainly because the models under imbalanced SSL setting have a strong bias to generate incorrect labels for the tail classes, which will impair the entire learning process. As result, more unlabeled tail class samples under T = lead to more severe performance degradation. On the contrary, our L2AC can climinate the influence of the training bias and predict high-quality pseudo- labels for unlabeled data, such that it achieves significant performance gain in both settings.
4.2 RESULTS ON CIFAR- AND STL-10
Dataset: To make a more comprehensive comparison, we further evaluate L2AC on CIFAR-100 Krizhevsky et al. 2009) and STL-10 (Coates et al. 2011). For CIFAR-100, create labeled and unlabeled sets in the same way as described in Sec. 4.1 and set N1 = 150 and Mj = 300. STL-10 is more realistic SSL task that has no distribution information for unlabeled data. In our experiments, we set N, = 450 to construct the imbalanced labeled set and adopt the whole unknown unlaheled set (i.e., M 100k). It is worth noting that the unlabeled set of STL-10 is noisy it contains samples that do not belong to any of the classes in the labeled set. Results: As shown in Table L2AC achieves the best performance over GM and competitive performance over bACC compared to current state-of-the -ott----rtt--ttt ABC (Lee et al.] 2021) and DASO (Oh et al.) (022). This implies that our approach obtains a relatively balanced classification performance towards all the classes. While for SLT-10, more realistic noisy dataset without distribution information for unlabeled data, L2ACC significantly outperforms ABC and DASO on both bACC and GM, which demonstrates that has greater potential to be applied in the practical SSL scenarios.
4.3 RESULTS ON LARGE-SCALE SUN-397
Dataset: SUN397 (iao et 2010) is an imbalanced real-world scene classification dataset, which originally consists of 108,754 RGB images with 397 classes. Following the experimental setups in
7

---

Published as a conference paper at ICLR 2023
Table 4: Performance (bACC/ GM) on CIFARIO-LT and STL-10 under various label ratio S.
CIFAR-I0 (71 7u 100)
B
=
8 10
B=20
30
FixMatch L2AC (ours)
54.9 16.5
65.1 / 35.5
69.0/53.9
72.0/ / 62.2
76.5 / 74.3 83.6 83.2
62.8/ 55.8
75.9 74.1
79.3/78.4
80.8 79.9
STL-10 (y = 10, N/A)
==
B == 10
3 = 20
B=40
| = 60
FixMatch
46.5 19,9
27.0
58.2 39.8
6 60.7
69.2 67.6
w/ L2AC (ours)
02.8/57.1
66.5/ 62.99
72.6/ 70.6
77.0 75.7
78.8 / 77.9
Kim et al. (220a)), we hold-out 50 samples per each class for testing because no official data split provided. We then construct the labeled and unlabeled dataset according M 2. The
comparison methods includes: Vanilla, cRT (Kang et al. 2020). FixMatch (Sohn et al. 2020). DARP
(Kim et (0220a) and ABC (Lee al. (021). More training details are presented in Appendix [D.2
Results: The experimental results are summarized in T Compared to the baseline FixMatch (Sohn al. 2020), our proposed L2AC results in about 4% performance gain over all evaluation metrics, and outperforms all the SOTA methods. This further verifies the efficacy of our proposed method toward the real-world imbalanced SSL applications.
4.4 DISCUSSION
What about the performance under various label ratios? To answer this question, we vary the ratios of labeled data (denoted as B) on CIFAR-10 10 and STL-10 to evaluate the proposed method. For CIFAR-10, 10, define 3 = N1 (NN1 1 + M1) and set the imbalance ratio y 100. For STL-10, since it does not provide annotations for unlabeled data, we re-sample the labeled set from labeled data by = / and set the imbalance ratio nn 10. As shown in Table 4 our I.2AC consistently improves the baseline across different amounts of labeled data on both CIFAR-10 and 100. For example, STL-10 with
5% contains very scarce labeled data, where only 25 and 3 labeled samples belong to the most majority and minority classes, respectively. In such an extremely biased scenario, our L2AC significantly improves FixMatch by around 16% over bACC and 37% over GM.
How does L2AC perform on the majority/minority classes? To explain the source of perfor- mance improvements. we further visualize the confusion matrices on the test set of CIFAR-IO with 100. Noting that the diagonal vector of confusion matrix represents per-class recall. As shown Fig. 3 our 1.2AC provides a relatively balanced per-class recall compared with the baseline Fix- Match (Sohn et al. (020) and other imbalanced SSL methods, e.g., DARP (Kim et al.) (020a) and ABC (Lee et al. 2021p. It can also be observed that FixMatch easily tends to misclassify the samples of the minority classes into the majority classes, while our L2AC largely alleviates this bias. These results reveal that our L2AC provides an unbiased linear classifier for the test stage.
Could L2AC improve the quality of pseudo-labels? Qualitative and quantitative experiment sults have shown that the proposed L2AC can improve the performance of pscudo-labeling SSL methods under different settings. We attribute this the fact that L,ACC generate unbiased pseudo-labels during training. To validate this, we show per-class recall of pseudo-labels for CIFAR- 10 with " Ya 100 and n 100, Tu 100 (reversed) in Fig. 4 It is clear that L2AC significantly raises the final recall of the minority classes. Especially for the situation where the distribution of labeled and unlabeled data are severely mismatched, shown in Fig, 4b), our L2AC considerably improves the recall of the most minority class by around 60% upon FixMatch. Such high-quality pseudo-label estimation probably benefits from more unbiased classifier with basi- cally equal preference for each class.
How about the learnt línear classifier and feature extractor? We revisit the toy experiment in Section [ where the whole class-balanced training set is used to train an unbiased upper bound model. Instead, the baseline FixMatch and our L2AC are trained under the standard imbalanced SSL setups, and we compare the predicted class distributions on the test set with that of the upper bound model. As shown in Fig. [c), the classifier learned by L2AC approximates the predicted class distribution of the upper bound model and much better than FixMatch, indicating that L2AC results in a relatively unbiased classifier, For the feature extractor, we further visualize the representations of training data through t-SNE (Van der Maaten & Hinton 2008) CIFAR-10 10 with 7 100, 7a
8

---

Published as a conference paper at ICLR 2023
FicMatch
DARP
ABC
abel
Reditted bbnl
Predeted bacel
Figure 3: Confusion matrices of FixMatch (Sohn et al. 2020), DARP (im et al. 2020a), ABC Lee 2021, and ours CIFAR-10 under the imbalance ratio y 100.
et al.
1.0-
Rocall
Class index
Class
7/4
100
% 100 (reversed)
(a) FixMatch
(b) L.ACC (ours)
Figure Pseudo-label per-class recall of Fix- Match and ours on CIFAR-10 with n = 100 and (a) %o 100; (b) 7u- 100 (reversed).
Figure 5: t-SNE visualization of training data for (a) FixMatch and (b) L2AC. L2AC helps to dis- criminate tail classes from majority ones.
1. As shown in Fig. 5 the features of the tail classes from FixMatch are scattered to the majority classes. However, L2AC can help the model to effectively discriminate the tail classes (e.g.. Class 8, 9) from majority classes (e.g.. Class O, 1). Such high-quality representation learning also benefits from an unbiased classifier during training. In Appendix [E.T] we further present the unbiasedness of our L2AC by evaluating it various imbalanced test sets.
Ablation analysis. We conduct an ablation study to explore the contribution of each crit- ical component in L2AC. We experiment with FixMatch CIFAR-10 under 'N Yu = 100 and n = 100, To = 100 (reversed). 1) We first verify whether the bias attractor is helpful. To this end, we apply the proposed bias attractor to FixMatch. It can be seen from Table 5 that the bias attractor helps improve the performance to certain extent, indicating that the bias attrac- tor is effective yet not significant. As analyzed in Section [3.2] there is no prior knowledge for the bias attractor to assimilate the training bias in this plain training manner. 2) Next, we study how important the role bi-level optimization plays in our method. Instead of using a bi-level learning framework, we disengage the hierarchy structure of our upper-level loss and lower-level loss, and reformulate a single level optimization problem as C + X The results in Table show that such degraded version of L2AC provides substantial performance gain over FixMatch w/ bias attractor while still inferior to our L2AC, This demonstrates the effect of the proposed bias adaptive classifier and livee learning framework on protecting the linear classifier from the training bias.
Table 5: Ablation study.
CIFAR- 10 (1r = 100)
Methods
= 100
1/100 (reversed)
FixMatch
71.5 / 68.8
65.5 / 26.0
FixMatch w/ bias attractor
73.9/ 70.7
66.6 44.8
L2AC w/o bi-level training L2AC (ours)
76.6
79.3 78.0
82.1 /81.5
82.2/ 81.7
5 CONCLUSION
In this work, we propose a bias adaptive classifier to deal with the training bias problem in imbal- anced SSL tasks. The bias adaptive classifier is consist of linear classifier to predict unbiased labels and a bias attractor to assimilate the complicated training bias. It is learned with a bi-level optimization framework. With such a tailored classifier, the unlabeled data can be fully used by online pseudo-labeling to improve the performance of pseudo-labeling SSL methods. Extensive experiments show that our proposed method achieves consistent improvements over the baselines and current state-of-the the-arts.. We believe that our bias adaptive classifier can also be used for more complex data bias other than class imbalance.
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank the anonymous reviewers for their constructive suggestions on improving this paper. This research was supported by National Key R&D Program of China (2020YFA0713900) the Macao Science and Technology Development Fund under Grant 0612020A2, The Major Key Project of PCL (PCL2202A12), the China NSFC projects under contract 61721002 and 61906144.
REFERENCES
Eric Arazo, Diego Ortego. Paul Albert, Noel O' D'Connor, and Kevin McGuinness, Pscudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2020.
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmenta- tion anchoring. In ICLR, 2019a.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. NeurlPS, 32, 2019b.
Paula Branco, Luís Torgo, and Rita P Ribeiro. A survey of predictive modeling on imbalanced domains. ACM Computing Surveys (CSUR), 1-(2):1-50, 2016.
Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249-259, 2018.
Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In ICML, pp. 872-881. PMLR, 2019.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with aabel-distrbuttin-aaaar margin loss. In NeurIPS, pp. 1567-1578, 2019.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning (chapelle, o. et al., eds.; 2b0o)]book reviews]. IEEE Transactions on Neural Networks, 20(3):542-542, 2009.
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16:321-357, 2002.
Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. Feature space augmentation for long-tailed data. In ECCV, pp. 694-710, 2020.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215-223 JMLR Workshop and Conference Proceedings, 2011.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR, pp. 702-703, 2020.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effecttve number of samples. In CVPR. pp. 9268-9277, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICMI., pp. 1126-1135. PMLR, 2017.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In NeurlPS, pp. 529-536, 2004.
Lan-Zhe Guo and Yu-Feng Li, Class-inbalanced semi-supervised learning with adaptive threshold-
ing. In International Conference Machine Learning, pp. 8082-8094, PMLR, 2022.
Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 29): 1263-12844 2009.
10

---

Published as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, pp. 770-778, 2016.
Kurt Horik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural networks, 2(5):359-366) -665966666-6 1989.
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In CVPR, pp. 5375-5384, 2016.
Minsung Hyun, Jisoo Jeong, and Nojun Kwak. Class-imbalanced semi-supervised learning. arXiv preprint arXiv:2002.068/5, 2020.
Muhammad Abdullah Jamal, Matthew Brown, Ming-Hsuan Yang, Liqiang Wang, and Boqing Gong. Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective. In CVPR, pp. 7610-7619, 2020.
Nathalie Japkowicz. The class imbalance problem:
Significance and strategies. In Proceedings of the International Conference on Artificial Intelligence (ICAI), volume 56. Citeseer, 2000.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2020.
Salman H Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous A Sohel, and Roberto Togneri. Cost-sensitive learning deep feature representations from imbalanced data. IEEE transactions neural networks and learning systems, 29(8):3573-3587 2017.
Jachyung Kim, Youngbum Hur, Sejun Park, Eunho Yang. Sung Ju Hwang, and Jinwoo Shin. Distri- bution aligning refinery of pseudo-label for imbalanced semi-supervised learning. NeurIPS, 33, 2020a.
Jachyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major-to- minor translation. In CVPR, pp. 13896-13999 13905, 2020b.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Miroslav Kubat, Stan Matwin, al. Addressing the curse of imbalanced training sets: one-sided selection. In ICML, volume 97, pp. 179-186. Citeseer, 1997.
Zhengfeng Lai, Chao Wang, Sen-ching Cheung, and Chen-Nee Chuah. Sar: Self-adaptive refine- ment on pseudo labels for multiclass-mmbalancee semi-supervised learning. CVPR, pp. 4091- 4100, 2022.
Dong- Hoyg-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. Workshop on challenges in representation learning. ICML., 2013.
Hyuck Lee, Seungjae Shin, and Heeyoung Kim. Abe: Auxiliary balanced classifier for class-
imbalanced semi-supervised learning. NeurlPS, 34:7082-7094, 2021.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. ICCV, pp. 2980-2988, 2017.
Jialun Liu, Yifan Sun, Chuchu Han, Zhaopeng Dou, and Wenhui Li. Deep representation learning on saigttiiled data: A learnable embedding augmentation perspective. In CVPR, pp. 2970-2979, 2020.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boqing Gong, and Stella Yu. Large- scale long-tailed recognition in open world. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. In NeurlPS, pp. 136,144, 2016.

---

Published as a conference paper at ICLR 2023
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi suneri-ueeeeiee learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(8):1979-19) 2018.
Youngtack Oh, Dong-Jin Kim, and In So Kweon. Daso: Distribution-aware semantics-oriented pseudo-label for imbalanced semi-supervised learning. In CVPR, pp. 9786-9796, 2022.
Avital Oliver, Augustus Odena, Colin Raffel, Ekin D Cubuk, and Ian J Goodfellow. Realistic evalu- ation of deep semi-supervised learning algorithms. NeurIPS, pp. 3239-3250, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. NeurIPS, 32:8026-8037, 2019.
Mengye Ren, Wenyuan Zeng, Bin Yang. and Raquel Urtasun. Learning to reweight examples for robust deep learning. In ICML., pp. 4334 4343. PMLR, 2018.
Mehdi Sajjadi, Mchran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transforma- tions and perturbations for deep semi-supervised learning. NeurIPS, 29:1163-1171, 2016.
Li Shen, Zhouchen Lin, and Qingming Huang. Relay backpropagation for effective learning of deep convolutional neural networks. In ECCV, pp. 467-482. Springer, 2016.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta- weight-net: Learning an explicit mapping for sample weighting. Advances in Neural Information Processing Systems, 32:1919-1930, 2019.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying emmi-supervised learning with consistency and confidence. NeurIPS, 33, 2020.
Jingru Tan, Changbao Wang. Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie
Yan. Equalization loss for long-tailed object recognition. In CVPR, pp. 11662-11671, 2020.
Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. Advances in Neural Information Processing Systems, 33, 2020.
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averaged consis- tency targets improve sepe-supe deep learning results. In NeurIPS, pp. 1195-1204, 2017.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Renzhen Wang, Kaiqin Hu, Yanwen Zhu, Jun Shu, Qian Zhao, and Deyu Meng. Meta feature modulator for long-taile's recognition. arXiv preprint arXiv:2008.03422 2020.
Renzhen Wang, Yichen Wu, Huai Chen, Lisheng Wang, and Deyu Meng. Neighbor matching for semi-supervised learning. In Medical Image Computing and Computer Assisted Intervention, pp- 439-449 449. Springer, 2021.
Xudong Wang, Zhirong Wu, Long Lian, and Stella Yu. Debiased learning from naturally im- balanced pseudo-labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14647-14657, 2022.
Yu-Xiong Wang. Deva Ramanan, and Martial Hebert. Learning to model the tail. In NeurIPS, pp. 7032-7042, 2017.
Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang. CReST: A class-rehalancing self-training framework for imbalanced semi-supervised learning. In CVPR, pp. 10857-10866, 2021.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey zoo. In CVPR, pp. 3485-3492, IEFE, 2010.
12

---

Published as a conference paper at ICLR 2023
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoe Le. Unsupervised data augmentation for consistency training. NeurIPS, 33, 2020a.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In CVPR, pp. 10687-10698, 2020b.
Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving dass-imballaced learning- In NeurlPS, 2020.
Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learn- ing for face recognition with under-represented data. In CVPR, pp. 5704-5713, 2019.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo la- beling, NeurIPS, 34, 2021a.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Pazz mixup: Beyond empirical risk minimization. In ICLR, 2018.
Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, and Jian Sun. Distribution alignment: A unified framework for long-tail visual recognition. In CVPR, pp. 2361-2370, 2021b.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. BBN: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In CVPR, pp. 9719-9728, 2020.
13

---

Published as a conference paper at ICLR 2023
A
ALGORITHM
We give the training algorithm of the proposed I.2AC method in Algorithm
Algorithm learning adapt classifier during training
Input: labeled/ unlabeled training data Da, labeled unlabeled batch size m, max iterations
T
Output: classification network parameters {0, Φ)
1: Initialize {0", )0} <- {0,0) Φ} and w <- w.
2: for = 0 to T do
3: Dj = [x,
+ SampleMimiBatch((P..n,))
4: D. [x, f SampleMiniBatch (D.., m.)
5: B {x, y. 12 + SampleMiniBatchh(D.... n).
6:
Estimate pscudo-label y, for x; E De..
7:
Compute lower-level loss L by Eq.
8:
Update network parameters {
by Eq. 4).
9:
Compute upper-level loss Lbal by Eq. 5
10: Update bias attractor parameters w by Eq. 66
end for
B PROOF PROPOSITION 3.1
Proof B.1 The update of ó is as:
aL, (0, o, w) Os
160.
(8)
consequence, we have
w w' 17 Caaba (A, (0.0*+1))) )lt
8 (0, ())
lar achml(0, 62+1
w' na n
0odw
OΦ
(9)
aDf:
(0)))(0,))) (b)
Ocbal
w' + nj
lus
1os
lo+t1
n
Ow
000Af.
0Q
Denote by E,
(0.(0.0) 8Af.
then Eq.
becomes
0Df.
w
w²
rja
C
Ow
w.
(10)
1=1
where
aC (0, ()
G
10111
06
06
As the truining loss is
L)),, log kml where e; is the class label of the i-th sample. Therefore
Ai ties
Afren
(11)
k: # c;
Bi,4
1
(12)
1, =c
Meanwhile, the upper level loss is defined
pbal (0,
L
bal
(0, Φ):
(13)
m
this finishes the proof.
14

---

Published as a conference paper at ICLR 2023
C COMPARISON METHODS
To comprchensively evaluate the proposed method, we compare it with the Vanilla model that merely trained with labeled data and three other lines of methods: 1) re-balancing methods where only the class-imbalanced labeled data are used for training, including: Re-sampling (Japkowicz) 2000), LDAM-DRW (Cao al., 2019) and cRT (Kang al., (2020). 2) pseudo-labeling based SSL methods where both labeled and unlabeled data is used (without considering class-imbalance), in- cluding: Pseudo-labels (Lee et al. 2013), MixMatch (Berthelot al. 20196) and FixMatch (Sohn) et al. 2020). 3) imbalanced semi-supervised learning methods that consider class-imbalance and unlabeled data simultaneously, including: DARP (Kim al., 2020a), CReST+ (Wei et al. 2021]
ABC (Lee et al. 2021), SaR (Lai et al., 2022) and DASO (Oh et 2022).
We herein give brief introduction for all the comparison methods.
Vanilla, a plain classification network, e.g., Wide ResNet-28-2 OOlveer et al. 2018). trained with imbalanced labeled data by cross-entropy loss.
Re-sampling, re-balancing method that uses re-sampling strategy to balance the distribu- tion of training data.
LDAM-DRW, i.c., Label-distribution-aaaa margin, re-weighting method where the classifier encourage maintain large margin for tail classes.
cRT, i.c., Classifier re-training, a two-stage training method that first pretrains the entire network with all the imbalanced training data and re-train the classifier with a balanced objective.
MixMatch, SSL method which combines pseudo-labeling and consistency regularization techniques via Mixup augmentation (Zhang et al. 2018
FixMatch, a pseudo-labelling based SSL method of which the strongly augmented unla- beled samples (whose pscudo labels are generated from their weakly augmented versions) are used to train the network.
DARP, recent state-of-te the-art imbalanced SSL method that refines raw pseudo-labels via a convex optimization for alleviating distribution bias arisen by imbalanced and unlabeled training data.
based imbalanced SSL method that combines re-balancing and distribution alignment techniques to alleviate the training bias. The method assumes that labeled and unlabeled data have roughly the same distribution.
CReST,
ABC, which equips with two parallel linear classifiers with one fitting the imbalanced data and the other fitting the re-balanced data, and adds the consistency regularization further improve the performance.
SaR, i.e., self-adaptive refinement, which proposes the concept of mitigating vector that refines the soft labels of unlabeled data before generating the one-hot pseudo labels to alleviate the confirmation bias brought about by unlabeled samples.
DASO, for an unlabeled sample, which combines its pseudo-label from the linear classifier with that from similarity -bsed classifier to leverage their complementary properties in terms of bias. Moreover, a semantic alignment loss proposed to balance the biased feature representation.
For fair comparison, we use the same code base as DARP Kim et al. 2020a). As the training evaluation protocols of CReST (Wei al. 2021). ABC Lee et al. 2021) and DASO (Oh et al. 2022] are different from that of DARP, we reproduce their results according to the official codes [5] and DASO released by the authors. Note that the results
(i.e.. CReST [ ABC
CIFAR-100 in DARP (Kim et al. 2020a) are achieved under N, = 300, Mj = 150, while this paper keeps N1 150, M₁ 300 for satisfying the common assumption that the amount of unlabeled data
larger than that of labeled data.
15

---

Published as a conference paper at ICLR 2023
D EXPERIMENTAL SETUPS
D.1 IMPLEMENTATION DETAILS ON CIFAR AND STL-10
All our experiments are implemented with the Pytorch platform (Paszke et al. 2019) and follows the experimental settings in Kim et al. (2020a). We use Wide ResNet-28-2 (Oliver et al. 2018) as our backbone network. During training, the model is trained with Adam optimizer (Kingma & Ba 2015) under the default parameter setting, i.e., $1 = 0.9, B2 0.999, and = 10 The learning rate is set as 2 x 10-3 and the batch size is set as 64. The total number of training iterations are 2.5 10s in Kim et al. (2020a). To evaluate the model, (019b) and use an exponential moving average (EMA) of its parameters with decay rate of 0.999 each iteration. We also follow the standard evaluation protocols in Berthelot et al. (2019b) that evaluates the performance at every 500 iterations and reports the average test accuracy of the last 20 evaluations.
follow the setting in [Berthelot et al.
Bias attractor: As aforementioned in Section 3.2 the bias attractor is a light- network, i.e., a multi multi-layer pereeptron with one hidden layer in this paper. The input of the bias attractor the prediction scores output by the linear classifier, so the input dimension is the same as the number of classes. We normalize the input through its L2 norm or a softmax activation. Concretely, We use softmax operator on CIFAR-10 and STL-10, and L2 norm on CIFAR- 100 and SUN-397 due to its better and more stable performance than softmax operator. Note that the gradients of the input of the bias attractor are stopped in the training stage. The hidden layer dimension of the bias attractor is fixed as 256, which keeps stable and sound results through all our experiments. The parameters of bias attractor updated by Eq. (6). where the learning rate 1) is set as 1 x 10-4
Baselines: We evaluate our L2AC based on two recent popular SSL methods, i.e., MixMatch (Berth- elot al.) (019b) and FixMatch (Sohn et al. 2020). Both the two baselines are the cornerstone of current state-of-the-arr imbalanced SSL methods, such as DARP (Kim et al. 2020a). CReST (Wei) al.) 202T) and DASO (Oh et al., 2022). Note that the two baselines can be uniformly formulated as Eq. I For MixMatch, the pscudo-label of one unlabeled example is produced by temperature sharpening to the the average prediction of its different augmented versions, and the objective func- tion of unlabeled data is adopted as meaa-squared loss (MSE) function. The threshold 7 is kept as O, and An is dynamically updated by a linear ramp-up strategy, i.e., Au linearly increases from 0 to 75 during training. For FixMatch, unlabeled data are augmented by weak and strong augmentations via RandAugment (Cubuk et al., 2020). In particular, the weekly augmented data are used to generate leeudo-labels for the strongly augmented data, and these strongly augmented data are then used to compute the unlabeled loss in Eq. (I. We set T as 0.95, and X, as without applying linear ramp-up strategy.
D.2 IMPLEMENTATION DETAILS ON SUN397
SUN397 (iao al. 2010) is an imbalanced real-world scene classification dataset, which originally consists of 108,75 RGB images labeled with 397 classes. Following the experimental setups in
Kim et al. 2020a), we hold-out 50 samples per each class for testing because no official data split
is provided. We then artificially construct the labeled and unlabeled dataset using the remaining
dataset according to N₂
Mr 2. The comparison methods includes: Vanilla, classifier retraining (cRT)
KKng et al. 2020 FixMatch (Sohn et al. 2020). DARP (Kim et al. 2020a) and ABC (Lee et al., 2021
Training details. For pre- pre-processing. we randomly crop and rescale to 224 x 224 size all labeled and unlabeled training images before applying augmentation. We use standard ResNet-34 (He al,, 2016) as our backbone network. During training, the model is trained with Adam optimizer (Kingma & Ba 2015) with a batch-size of 128 labeled samples and 256 unlabeled samples, and a initial learning rate of 0.002 for 300 training epochs. For fair comparison, all the experiments are based on FixMatch (Sohn et al. 2020). We set unlabeled loss weight X as 1.0 and confidence threshold 0.6, and utilize exponential moving average technique with decay rate 0.99. Following Kim jet al. (2020a), we adopt RandAugment with random magnitude (Cubuk et al. 2020) for strong augmentation and random horizontal flip for weak augmentation.
16

---

Published as a conference paper at ICLR 2023
E ADDITIONAL EXPERIMENTS
E. EVALUATION ON IMBALANCED TEST SETS
Real-world test data is not always following an uniform distribution, and we herein simulate several imbalanced test sets to further study the generalization of the proposed method. As shown in Fig. we construct three imbalanced test sets: (a) Test-1, which follows long-tailed class distribution with an imbalance ratio of 10; (b) Test-2, which follows reversed long-tailed class distribution; (c) Test- 3. which follows a random class distribution. We evaluate the proposed L2AC upon FixMatch (Sohn) al. 2020) and compare it with the following methods: DARP (Kim al. 2020ap. CReST+ (Wei et al. 2. ABC (Lee et al. 2021). All these models are trained on CIFAR-I0 10 with imbalanced ratio ?
7 = Tu 100. As the evaluation metrics bACC and GM are insensitive class
imbalance of test sets, we add ACC to measure the recognition accuracy for all samples.
The results are summarized Tab. can be observed that the network learned with our L2AC achieves the best or the second best performance across all the test settings, which further indicates that the proposed bi-level learning framework provides a relatively unbiased classifier compared to other comparison methods.
1.0
1.0
1.0
Samples (k)
0.8
Samples
0.8
Samples
0.6
0.6
#
0 1 2
7
0
2
3
4
5
6
3
4
5
6
8
9
Class index
Class
index
Class
index
(a) Test-1
(b) Test-2
(c) Test-3
Figure 6: Class distributions of three typical imbalanced test sets.
Table 6: Imbalanced test set results. ACC: accuracy for all samples.
Test-1
Test-2
Test-3
Methods
bACC
GM
ACC
bACC
GM
ACC
bACC
GM
ACC
FixMatch
Sohn et al.
2020
72.4
86.1
72.7
66,9
56.6
72.3
65.7
DARP
Kim ct at.
2020a
72.5
86.3
75.5
73.2
63.9
75.6
73.3
w/ CReST
Wei et al. 2021
77.8
76.5
86.3
77.2
74.8
68.9
77.5
76.3
80.3
w/ ABC
(Lee
al. 2021
80.2
79.2
88.1
80.2
79.0
71.7
80.1
79.0
82.8
L2AC (ours)
82.6
82.0
87.2
82.4
81.8
78.6
82.7
82.1
83.9
E.2 EVALUATION ON STEP IMBALANCE SETUPS
In the main text, we have evaluated the proposed L2AC under long-tailed imbalanced settings with various imbalance ratios. Here, we further validate its generalization in the case of step imbalance on CIFAR-10 under two typical setups, namely 7/ = 7u 100 and 7=100, 7h. = As shown in Fig. step imbalance assumes a more severely imbalanced class distribution than the long-tailed imbalance setups, as there are very scare data for half of the classes. The experimental results are summarized in Tab. 7 We can that the proposed L2AC achieved the best performance compared with all the comparison methods for all settings. Especially in the case distribution mismatch between labeled and unlabeled data, L.ACC brings significant performance gain.
17

---

Published as a conference paper at ICLR 2023
5.0
Labbr ac data
Urtabe ld data
Labe 45 deta
Jrllaoe ad Cata
4.0
Samples
3.0
2.0
* 1.0
Class index
Class index
(a)
(h)
Figure 7: Class distributions of CIFAR-10 under step imbalance setups with (a) 71 = 7h 100; (b)
? 100, 7a
Table 7: Performance (bACC/ / GM) on CIFAR-10 under step imbalance setups.
Methods
?/1
7u 100
7: 100, 7o
FixMatch (Sohn et al.
2020
55.0
/24.442.07
60.1_1.97 18.3223..6
w/ DARP (Km
2020a
58.6 44 /3..0:1.27
w/ ABC
Ice ct al.
75.1 667-510.97
75.8 /6..2
w/ 1.2AC (ours)
76.7 74.5
81.8 0 /805
E.3 DETAILED ANALYSIS ON THE QUALITY OF PSEUDO-LABELS.
Case of n Yu- Fig. visualizes the confusion matrices of pseudo-labels of Fixmatch (Sohn et al., 2020) and our L2AC under the imbalance ratio "it % 100. Note that this requires use true labels that are hidden in the training stage. We can observe that the original pseudo- labels are highly biased toward majority classes of the labeled dataset. In contrast, our L2AC tends to relatively equal per-class recall, especially on minority classes the performance significantly improved compared to FixMatch (Sohn et al. 2020). This suggests that the proposed method readily improves the quality of pscudo-labels.
Fixtatth
Ours
3.0
3.0
0A2 8:00 OUR SEC R0C
OEB 9CD BIC
CAL 0E2
DO0 800 030
DOC 900
30C
OUR
900
300
AA
TrUR lahel
5.6
Thun
028 K. 810
22.022 nrd
000
900
#1
050
900 900 8:000000000000000000000000000800 111633
C2
Predicted
Predicted label
(a)
(b)
Figure 8: Pseudo-label confusion matrices of (a) FixMatch and (b) Ours on CIFAR-10 under % = Yu 100.
Case of 71 * Yu- Fig. 9 visualizes the confusion matrices of pseudo-lahels for Fixmatch (Sohn et al. 2020) and our L.2AC under the imbalance ratio = 100 while this setting, labeled set is unchanged and the total number of the training data remains the same compared "t Tu 100. Unlabeled data follow reversed class distribution, which means that more samples are added to the training set for the minority classes. According to the confusion matrix in Fig. FixMatch (Sohn et al.) 2020) remains to achieve high recall on the majority classes and low recall on the minority classes. Most samples of the last two tail classes are mislabeled the most majority classes. These findings reveal that the pseudo-labeling process becomes
100 (Reversed). Under
18

---

Published as a conference paper at ICLR 2023
more biased the mismatch of the distributions of labeled and unlabeled data become severe. The confusion matrix of L2AC clearly shows that it can exactly assimilate the training bias through the proposed bias attractor such that the linear classifier is able to predict correct label.
Fixtatch
Ours
0:00
DEB
Thue label
04B 043
048
048
060
True
23 905
X00 100 can oze 300
<44 04B act 900 801 030 oce
Predicted label
Predicted label
(b)
Figure 9: Pseudo-label confusion matrices of (a) FixMatch and (b) Ours on CIFAR- 10 with "
100, = 100 (reversed).
E.4 TRAINING CONVERGENCE VERIFICATION
To verify the convergence of our proposed L2AC approach, Fig. visuullize the training curves of the lower-level loss Eq. 3 and the upper-level loss Eq. (5) with training iteration increasing from 0 to 2.5 x 10 We can see that both lower-level loss and upper-level loss convergence fast within the first 100 epochs (5 x 10 iterations), and the test accuracy curve increases fast at the same time. When the test accuracy reached the peak value, our L2AC roughly remains the same test accuracy until termination, which verifies the robustness of the proposed method.
0.5
Uzperited lao
Lovenlawe isa
Training loss
Training loss
6,3
lest
Training Iterations
Training Iterations
8.0
0x
50c
Training Iterations
100k
150k
200k
2506
Curves of Left: lower-level and upper-level losses and Right: test accuracy durning training of our ap- proach on long-tailed CIFAR-10 under "Y = The 100.
Figure 10:
Figure 11: Curves of lpper-level loss (over B3) and lower-level loss over la- beled set Dr
Since the balanced set B is dynamically sampled from the labeled set D;, a nature question is whether the ueper-level loss (over B) has the same convergence rate with the lower-levell loss (over D;) during training. To investigate this, we further visualize the curves of these two losses in Fig. [I and we can observe that the two losses decrease differently and converge to values of different magnitudes at different iterations.
E.5 RUNNING COST ANALYSIS
In Section 3.2 we conduct complexity analysis of the training algorithm of our L2AC, which shows that our method is very efficient theory. To verify this, we herein measure floating point operations per second (FLOPS) using NVIDIA GeForce RTX 3090 quantify the training cost. We compare our proposed algorithm with the baseline model FixMatch (Sohn et al. 2020) and two state- of-the-art methods (Kim et al. 2020a) and (Lee et al. 2021). L.2AC uses the same code base with
19

---

Published as a conference paper at ICLR 2023
these two methods. Besides, also provide the training cost of L2AC (traditional), the algorithm that unrolls the gradient of the whole classification network to compute the second-order gradient of the bias attractor. just like most gradient-bad based bi-level optimization algorithm (Finn et al. 2017
Ren et al.) 2018). It can be seen that: (1) our 1.2AC is much faster than L.2AC (traditional ) (2) The training cost of our I.2AC is comparable the current state-of-the-art method ABC (Lee al. 2021
Table 8: Training cost analysis on CIFAR-10 10 and CIFAR-100.
FLOPS
Methods
Params
CIFAR-10
CIAFR-100
FixMatch (Sohn et al.
2020)
1.47 M
19.6 iter/sec
19.6 iter/sec
w/ DARP (Kim et al.
2020a)
1.47 M
18.2 iter/sec
7.5 iter/sec
w/ ABC (Lee al. (2021)
1.47 M
15.1 iter/sec
14.9 iter/sec
w/ 1.2.C (traditional)
1.48 M
9.9 iter/sec
9.7 iter/sec
w/ L2AC (ours)
1.48 M
14.2 iter/sec
13.9 iter/sec
can be observed that the computation cost increment of our proposed L2AC is nearly negligible compared with the baseline models especially considering the significant improvement performance of L2AC. And it is worth noting that our proposed L2AC is more efficient than traditional second- order optimization, we think owing two reasons: (1) the bias attractor only adds very small number of parameters (about 0.68% of the total number of parameters); (2) To calculate the second- order gradient of these parameters, we only need to unroll the gradient of the linear classifier as shown in Eq. (4)
Note that the test stage our L2AC requires no extra overhead compared with the baseline model FixMatch.
E.6 FEATURE VISUALIZATION UNDER OTHER SETUPS
In Fig. visualize the representations of training data through t-SNE (Van der Maaten & Hinton
2008 on CIFAR-10 with " = 100, 7a = Under such a setting, each class roughly has the same
number of samples, which ensures good visual visualization for each class. To verify that the
proposed L2AC can generally obtain a high-quality representation, we further visualize the t-SNE of training data under other experimental setups, including " = Yu 100 and 7/ 100, 7a 100 (reversed). As shown in Fig. [2 and Fig. [3] compared with FixMatch (Sohn et al., 2020). our L2AC certainly improves the separability of the tail classes from the head classes. This verifies that the result in Fig. is is not owing to the specific choice of the experimental setups.
(a) FixMatch
(b) L2AC (ours)
(a) FixMatch
(b) L2AC (ours)
Figure 12: t-SNE visualization of unlabeled data for (a) FixMatch and (b) L2AC on CIFAR-10
Figure 13: t-SNE visualization of unlabeled data for (a) FixMatch and (b) L2AC on CIFAR-10 with = 100, 7a = 100 (reversed).
with 72 = 100.
20

---

Published as a conference paper at ICLR 2023
F CONVERGENCE ANALYSIS OF ALGORITHM
We prove that our algorithm to minimize Lbal converges at rate of Ö ( which meets the conver- gence results of similar work such Ren et al. (2018).
Theorem F.1 Assume that Chat in Eq. 5 is L-smooth with p- p-bounded gradients on the samples of balanced set B, and the bias attractor function (- is disserentiable with a 8-bounded gradient and twice differentiable with bounded Hessian. Let 0e in Eq. (4) satisfies ar = + S T. and qu in
Eq. is set int = _3, for >> O such that critical point at the rate 0 (T) as
ov. < L. Then the loss function Lbat converges to
min
V (o (W(W)) S
(14)
OSEST
Proof F.1 The parameter w updated by stochastic gradient descent as wtl = w t1 V chol (ø (ls. where B, is mini-batch of class-balanced data. We can write Eq. 15) as
(15)
w = w' vi chal (ø' (w')) + +
(16)
where 5 is the random gradient noise with zero mean and finite variance a. Observe that ( (w) C (Q'(w))
(17)
=(Chal (0tt) ())+)) cuai (0' (w) + (
<ral (B'(w)))
According to L-smoothness of Chal with respect to o. we have
(w())( C. (0'(w'
S(VoCaa ( (w' / (w (w(()))) + L (100t)(u ( d'(w+ti)13.
(18)
meanwhile
+etl (w - 8 (w(tt) = aat volol(((u 0000000001
(19)
we have
(o (w)t)) (o'((o(
</ V (o (w0((w)))))) - a VoClo(w (00 ",,
( (20)
Saep' + L 2 2 jaip'
the second inequality holds due the assumption V Lbal Ch (o'(w(+1))) S p.
S p and
at V (,((((t)))0))ll)oo
According to the L-smoothness of Chat with respect to w, we have
- ( cbal (6- (w²))
<(V < bal (6(ww)) ++1 - w²) + L
". -|w't2 -w'12
2
bal (00 (w'))) - V__ bool (0' (w²)) + 5²)) + Lui
Lif
cat (o'(w)) + e'11
(21)
Luf m L (o' (1
Ludt
1 + (Ln) (V bal (b (w'(w))) (w'(w))),
2
Then Eq. 17 becomes
(w]tt)) (bal(d' (w())
Sa,p² + L 2
Lnk
L 1E' (IT (v chol (o'(w')))
= +
1 I. (o' (w0)))
2
(22)
21

---

Published as a conference paper at ICLR 2023
Rearranging Eq. [22 we have
(T Line
Il Ww Cha (0' (w 112))
(0' Crai (8) ( ( asp' + 20000=
Lof 2
11E' (Ln) no)(v w (6
(w (23)
E
Summing up the above inequalities, we obtain
T
LiE 2
Σ
| Cha (6' (w (w'))11
T Lif
Schoal (0' (w=))) cbal ( )+
aap (1 jaa))
+
Σ 11
t=1
(24)
+ Σ (L - (v) chal (6' (d(()))
T Lin
T E - (V
sCaa (o'
aap 2 a.) +
Σ
|
(8'w"))
2
Taking expectations with respect to g on Eq.
24
we can obtain
T
Life
Σ
V Coal (d' (w))))1
2
T
T
T
Schaa
(0) (w((w) +
Σ
aap (1 (1 + L
au)
Σ
+
Σ
(Tml
Ege(E"))) (25)
1=1
(=1
(o'
aip a2) +
2
The first inequality holds due to Further more, we have
E| = o² and the second equality holds due to Eg: [5*] = 0.
min Fl[[ V.
bat
(0 (w))))
T
2 20 (0'(w")) (w')) Σ app Lan) +
Σ
Lnio²
Σi= (2
Lm
T
2 (w) atp
Lax)
L
2t=1
(26)
1
2chal (0' (w) 1p
at + 0
1.
m
Σ Th
c2
(6 (w))) 4p log(T) o² log(T)
VT
-
VT
The second inequality hods due to Σ (21) Lapz 2 2I T/. The third inequality holds due
to S 3. As a4 + and Nu 22 we have Σt= = (VT), Σt= ni llggTT) and
ΣT a4 = log(TT) thus the last equality holds, this finish proof.
22


Reviews:
[{"review_id": "OGOjjdLQcp", "reviewer": "Reviewer_ihj1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a semi-supervised learning (SSL) method, especially for imbalanced classification. Since standard SSL methods based on pseudo labels often assign labels of head classes to unlabeled data, the obtained pseudo labels tend to have a small number of labels for tail classes, resulting in low classification performance. To cope with the problem, the bias adaptive classifier is proposed to reduce bias from imbalanced data. To train neural networks with the bias adaptive classifier, a bi-level optimization approach is employed. Through extensive numerical experiments, the proposed method outperformed various recent SSL methods. Also, the ablation study showed the usefulness of the proposed bi-level training with the bias adaptive classifier.", "summary_of_the_review": "This paper proposed the bias adaptive classifier for imbalanced semi-supervised learning. The idea looks novel, and the empirical results support its effectiveness. The reproducibility seems high. If the clarity of this paper is improved, the overall score will be increased.", "strength_and_weaknesses": "##### Strengths\n- The idea of the proposed bias adaptive classifier is novel and has an impact on imbalanced SSL.\n- The bi-level training is important to train a bias adaptive classifier and its effectiveness is carefully investigated through an ablation study.\n- The superior performance of the proposed method was demonstrated on extensive numerical experiments and various recent SSL methods were compared with the proposed method.\n\n##### Weaknesses\n- The current explanation of the proposed method is not really clear. The presentation of the proposed method could be improved. \n- It seems that the proposed method requires intensive computation compared with the other methods because the proposed method requires a second-order gradient. If the computation time is provided during the rebuttal phase, I will reconsider this point.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "B_gmsZw62z", "reviewer": "Reviewer_juK4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel imbalanced SSL method with pseudo-labeling.\nThe authors introduce a bias-adaptive module that can be trained to minimize\nclassification loss on a balanced dataset with bi-level optimization.", "summary_of_the_review": "The paper is overall good-looking, but the method does not make much sense to me.", "strength_and_weaknesses": "### Pros\n\n1. The paper is easy to follow and well-written.\n2. The problem is well-motivated and intuitively makes sense.\n3. The reported performance exceeds the baselines significantly.\n\n### Cons\n\n1. The balance loss L_bal is just a subset of the original pseudo-labeling loss L \n   (balanced dataset B is a subset of the labeled dataset D_l).\n   Thus, if L is minimized, L_bal will also be minimized (if we are overfitting L).\n   The effect of introducing a balanced dataset here is unclear.\n   If L_bal is a hold-out dataset (instead of \"dynamically sampled from the labeled training set\"), \n   there may be some differences.\n\n2. A much simpler method is to keep the bias-adaptive module fixed when updating other parameters\n   and update the bias-adaptive module on the balanced dataset. \n   I suggest the authors compare this baseline method and discuss the difference compared with\n   the proposed method. The bi-level optimization may be overkill.\n\n3. Theoretical analysis of the convergence of such a bi-level optimizing method is necessary,\n   which may help understand the problem in 2.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2JDdA0B74gs", "reviewer": "Reviewer_5NiX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a semi-supervised learning algorithm for unbalanced data. The key component is a bias adaptive classifier to alleviate the problem that the model is usually biased towards the majority classes. The bias adaptive classifier, together with the regular classifier, are learned through a bi-level training protocol, where the model us updated first and a sampled balanced set is used to update the bias attractor parameters. Experiments on several long-tailed dataset show the outstanding performance compared to existing solutions.\n", "summary_of_the_review": "In summary, this is overall a well-written, well-organized paper and the proposed method is technically sound. Despite its simplicity, the proposed method consistently outperforms existing works on SSL for class imbalance problem, and the improvement is quite significant. There are a few unclear pieces regarding motivation of the bias attractor parameters and missing references.", "strength_and_weaknesses": "This paper is well written and easy to follow. The idea of using another classifier to rectify bias introduced in the original classifier is intuitively sound. Moreover, the formula and design is actually very straightforward yet effective, allowing others to easily reproduce the proposed method and add the proposed module into any SSL framework. Technically, the approach of bi-level training makes a lot of sense to me, when considering the cost function to be enforcing the similarity between gradient of a sample and the average gradient of a sampled balanced set. In addition, this architecture seems works pretty well under various imbalance conditions. \n\nThis work relaxes the assumption used by some existing works that unlabeled data and labeled data should follow similar distribution, and experiments show that it can handle both this case and reversed case (unlabeled data distribution is reversed from labeled data distribution). Apart from consistent gain over compared algorithms, this paper also provides solid ablation study to highlight a few important observations on how the proposed algorithm performs regarding different perspectives, such as pseudo-label quality improvement, capability of handling extreme label ratios. Such study is valuable and helps evaluate the contribution of this work, making this paper convincing.\n\nThere are several minor weakness, listed as follows:\n\n- Although the related works section covers most of recent SSL algorithms for class imbalanced problem, there are more papers that should be cited:\n\n   -  [1] Class-Imbalanced Semi-Supervised Learning with Adaptive Thresholding, 2022\n   -  [2] Debiased Learning from Naturally Imbalanced Pseudo-Labels, 2022\n\n  In particular, [2] also talks about debasing for pseudo-labels under class imbalance scenario, which is quite related to this paper. It would be good to include and discuss about them.\n\n-  The motivation of the proposed bias adaptive classifier is still a bit vague to me, although the structure looks simple. I understand that it takes into account the second order gradient when computing the upper training loss $L_{bal}$, but could not draw a crystal clear connection between updating bias attractive parameters and enforcing model to make more prediction towards the minority classes. How such architecture, i.e., residual connection, linear and non-linear classifier design, and bi-level training helps alleviate classifier bias? More intuitive explanations would help. Additionally, how does the sampled balanced set affect performance of the bias adaptive classifier?\n\nSome minor grammatical errors:\n\n- Note that CReST+ (Wei et al., 2021) fail in … -> fails in\n- In summary, such an nested training… -> such a\n- design a be-level learning algorithm … -> a bi-level", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster