InPL: Pseudo-labeling the Inliers First for Imbalanced Semi-supervised Learning
Authors: Zhuoran Yu
Yin Li
Yong Jae Lee
Venue: ICLR 2023 poster (2023)
Keywords: imbalanced semi-supervised learning
energy-based model

Abstract: Recent state-of-the-art methods in imbalanced semi-supervised learning (SSL) rely on confidence-based pseudo-labeling with consistency regularization. To obtain high-quality pseudo-labels, a high confidence threshold is typically adopted.  However, it has been shown that softmax-based confidence scores in deep networks can be arbitrarily high for samples far from the training data, and thus, the pseudo-labels for even high-confidence unlabeled samples may still be unreliable.  In this work, we present a new perspective of pseudo-labeling for imbalanced SSL. Without relying on model confidence, we propose to measure whether an unlabeled sample is likely to be "in-distribution''; i.e., close to the current training data. To decide whether an unlabeled sample is "in-distribution'' or "out-of-distribution'', we adopt the energy score from out-of-distribution detection literature. As training progresses and more unlabeled samples become in-distribution and contribute to training, the combined labeled and pseudo-labeled data can better approximate the true class distribution to improve the model. Experiments demonstrate that our energy-based pseudo-labeling method, InPL, albeit conceptually simple, significantly outperforms confidence-based methods on imbalanced SSL benchmarks. For example, it produces a 4-6% absolute accuracy improvement on CIFAR10-LT when the imbalance ratio is higher than 50. When combined with state-of-the-art long-tailed SSL methods, further improvements are attained. In particular, in one of the most challenging scenarios, InPL achieves a 6.9% accuracy improvement over the best competitor.

Published as a conference paper at ICLR 2023
INPL: PSEUDO-LABELING THE INLIERS FIRST FOR IM- BALANCED SEMI-SUPERVISH LEARNING
Zhuoran Yu
Yin Li
Yong Jae Lee
University of Wisconsin M
{zhuoran. yin. 11 @ewisssscs...................iss edu yongjaelee@css wis.s..wisc... edu
ABSTRACT
Recent ofute-of-the-art methods in imbalanced semi-supervised learning (SSL) rely on confidence-based pseudo-labeling with consistency regularization. To ob- tain high-quality pseudo-labels, a high confidence threshold is typically adopted. However, has been shown that softmax-based confidence scores in deep net- works can be arbitrarily high for samples far from the training data, and thus, the pseudo-labels for even high-confidence unlabeled samples may still be unreliable. In this work, we present new perspective of pseudo-labeling for imbalanced SSL. Without relying on model confidence, we propose to measure whether an unlabeled sample is likely be "in-distribution": i.e., close the current training data. To decide whether an unlabeled sample is "in-distribution" or "out-of-distribution" we adopt the energy score from iitoo-ddistribution detection literature. As training progresses and more unlabeled samples become in-distribution and contribute to training, the combined labeled and pseudo-labeles data can better approximate the true class distribution to improve the model. Experiments demonstrate that our energy- -ased cantly outperforms confidence-basee methods on imbalanced SSL benchmarks. For example, produces around 3% absolute accuracy improvement on CIFAR I0-LT. When combined with state-of- fi-th-errtt long-tailed SSL methods, further improve- ments are attained. In particular, in one of the most challenging scenarios, InPL achieves 6.9% accuracy improvement over the best competitor.
method, InPL, albeit conceptually simple, signifi-
INTRODUCTION
In recent years, the frontier of semi-supervised learning (SSL) has seen significant advances through pseudo-labeling (Rosenberg et al., 2005; Lee et al., 2013) combined with consistency regulariza- tion (Laine & Aila, 2017; Tarvainen & Valpola, 2017; Berthelot et al., 2020; Sohn et al., 2020; Xie et al., 2020a). Pseudo-labeling, a type of self-training (Scudder, 1965; McLachlan, 1975) technique, converts model predictions unlabeled samples into soft or hard labels as optimization targets, while consistency regularization (Laine & Aila, 2017; Tarvainen & Valpola, 2017; Berthelot et al., 2019; 2020; Sohn et al., 2020; Xie al., 2020a) trains a model to produce the same outputs for two different views (c.g, strong and weak augmentations) of an unlabeled sample. However, most methods are designed for the balanced SSL setting where each class has similar number of training samples, whereas most real-world data are naturally imbalanced, often following a long-tailed distribution. To better facilitate real-world scenarios, imbalanced SSL has recently received increasing attention.
State-of-the imbalanced SSL methods (Kim et al., 2020; Wei et al., 2021; Lee et al., 2021) are build upon the pseudo-labeling and consistency regularization frameworks (Sohn et al., 2020; Xie et al., 2020a) by augmenting them with additional modules that tackle specific imbalanced issues (e.g., using per-class balanced sampling (Lee et al., 2021; Wei et al., 2021)). Critically, these methods still rely on ceafiddene-bbseed thresholding (Lee et al., 2013; Sohn et al., 2020; Xie et al., 2020a; Zhang et al., 2021) for pseudo-labeling. in which only the unlabeled samples whose predicted class confidence surpasses very high threshold (e.g., 0.95) are pseudo- labele- for training.
Confidence-based pseudo-labeling, despite its success in balanced SSL, faces two major drawbacks in the imbalanced, long-tailed setting. First, applying high confidence threshold yields significantly

---

Published as a conference paper at ICLR 2023
[x]
0.97
Labeled Data
In-distribution Unlabeled Data
Out-distribulion Unlabeled Data
(a) Beginning stage of training
(b) Later stage of training
Figure I: We illustrate the idea of InPI. with a toy example with one head class (green) and one tail class (red). (a) At the beginning of training, only a few unlabeled samples are close enough to the training distribution formed by the initial labeled data. Note that with confidence-basee the diamond unlabeled sample would be added as pseudo-label for the green class since the model's confidence for it is very high (0.97). Our InPL. instead ignores it since its energy score is too high and thus considered autoo--distribution at this stage. is evolved by both the initial labeled data and the pseudo-labeled "in-distribution" unlabeled data, and more unlabeled data can be included in training. In this example, with our approach InPL. the diamond sample would eventually be pseudo-labeled as the red class.
approach,
(b) As training progresses, the training distribution
lower recall of pseudo-labels for minority classes (Wei et al., 2021), resulting in an exacerbation of class imbalance. Lowering the threshold can improve the recall for tail classes but at the cost of reduced precision for other classes (see analysis in Section 4.4). Second, prior studies (Szegedy et al., 2014; Nguyen et al., 2015; Hein et al., 2019) show that softmax-based confidence scores in deep networks can be arbitrarily high even -itto--distrrbution samples. Thus, under long-tailed scenarios where the model is generally biased towards the majority classes, the model can predict high confidence scores for the head classes even if the instances are actually from the tail classes, resulting in low precision for head classes. Given the drawbacks of using the confidence score as the pseudo-label criterion, we seek to design a better approach to determine if an unlabeled sample should be pseudo-labeled.
In this work, we present a novel approach for pseudo-labeling that addresses the drawbacks of seafiddenee-based pseudo-labeling in imbalanced SSL. Instead of relying on a model's prediction confidence to decide whether to pseudo-label an unlabeled instance or not, we propose to view the pscudo-labeling decision as evolving in-distribution problem' Initially, only the ground -ruth human-labeled samples are considered "in-distribution" because they the only training examples. In each ensuing training iteration, the unlabeled samples that are close to the current "in "in-distribution" samples are pseudo-labeled and contribute to training, which in turn gradually expands the "in-distribution". Thus, any Nout-of-distribution unlabeled samples from previous iterations may become "in-distribution" in later iterations, as the distribution of the pseudo-labeled training data is continuously updated and expanded. An illustrative example of this process can be found in Figure 1.
-istrr--ittiibttion classification
To identify the "inliers", we leverage the energy score (LeCun et al., 2006) for its simplicity and good empirical performance. The energy score is a non-probabilistic scalar that is derived from a model's 's logits and theoretically aligned with the probability density of a data energy reflects data with higher/lower likelihood of occurrence following the training distribution, and has been shown to be useful for conventional out-of-distiibution (OOD) detection (Liu et al., 2020). In our imbalanced SSL setting, at each training iteration, we compute the energy score for each unlabeled sample. If an unlabeled sample's 's energy is below a certain threshold, we pseudo-label it with the predicted class made by the model. To the best of the authors' knowledge, our work is the first to consider pscudo-labeling in imbalanced SSL from an in-distribution vs. out out-distribution perspective and is also the first work that performs aveudo-labeling without using softmax scores. We refer to our method as Inlier Pseudo-Labelle (InPL) in the rest of this paper.
To evaluate the proposed InPL, we integrate it into the classic FixMatch (Sohn et al., 2020) framework and the recent state-of-the-art imbalanced SSL framework ABC (Lee et al., 2021) by replacing their vanilla confidence-hasse pseudo-labeling with our energy-based pseudo-labeling. InPL. significantly
Note that our definition of nut-of-distributiin is different from the typical definition from the Out-of- Distribution literature that constitutes unseen classes.
2

---

Published as a conference paper at ICLR 2023
outperforms the confidence-basse counterparts in both frameworks under long-tailed SSL scenarios. For example, InPL produces around 3% absolute accuracy improvement on CIFARIO-LT over vanilla FixMatch. With the ABC framework, our approach achieves 6.9% accuracy improvement in one of the most challenging evaluation scenarios. Our analysis further shows that pseudo-labels produced using InPL. achieve higher overall precision, and significant improvement in recall for the tail classes without hurting their precision much. This result demonstrates that the pseudo-labeling process becomes more reliable with InPL in long-tailed scenarios. InPL also achieves favourable improvement on large-scaled datasets, shows high level of robustness against the presence of real OOD examples in unlabeled data, and attains competitive performance on standard SSL benchmarks, which are (roughly) balanced.
RELATED WORK
Semi-Supervised Learning. While SSL research (Scudder, 1965; Mcl Lachlan, 1975) has been extensively studied in the balanced setting in which all categories have (roughly) the same number of instances, class-imbalan imbalancedd SSL is much less explored. A key challenge is to avoid overfitting to the majority classes while capturing the minority classes. State-of-the-an imbalanced SSL methods build upon recent advances in balanced SSL that combine aseudo-labeling (Lee et al., 2013) with consistency regularization (Laine & Aila, 2017; Tarvainen & Valpola, 2017; Berthelot et al., 2019; 2020; Xie et al., 2020b; Sohn et al., 2020; Xie et al., 2020a; Zhang et al., 2021; Xu et al., 2021), leveraging standard frameworks such as FixMatch (Sohn al., 2020) to predict pseudo-labels on weakly-augmented views of unlabeled images and train the model to predict those pseudo labels on strongly-augmented views. DARP (Kim et al., 2020) refines the pseudo-labels through convex optimization targeted specifically for the imbalanced scenario. CReST (Wei et al., 2021) achieves class-rebalancing by pseudo-labeling unlabeled samples with frequency that is inversely proportional to the class frequency. Adsh (Guo & Li, 2022) extends the idea of adaptive thresholding (Zhang et al., 2021; Xu et al., 2021) to the long-tailed scenario. ABC (Lee et al., 2021) introduces an auxiliary classifier that is trained with class-balanced sampling whereas DASO (Oh et al., 2022) uses similarity- classifier to complement pseudo-labeling
Class-Imbalances
Our method is in parallel to these developments it can be easily integrated into these prior approaches by replacing their confidence- based thresholding with our energy-based one; for example, when plugged into ABC. InPL achieves significant performance gains. Importantly, ours is the first to view the pseudo-labeling process for perspective.
imbalanced SSL from an "in-distribution vs. out-of-distributie
Distr--Disttibttion Detection. OOD detection aims to detect outliers that are substantially different from the training data, and is important when deploying ML models to ensure safety and reliability in real-world settings. The softmax score was used as baseline for OOD detection in (Hendrycks & Gimpel, 2017) but has since been proven to be unreliable measure by subsequent work (Hein et al., 2019; Liu et al., 2020). Improvements have been made in OOD detection through temperatured softmax (Liang et al., 2018) and the energy score (Liu et al., 2020). Robust SSL methods (Chen et al., 2020; Saito et al., 2021; Guo et al., 2020; Yu et al., 2020) aim to improve the model's 's performance when OOD examples are present in the unlabeled and/or test data under SSL scenarios. For example, D3SL (Guo et al., 2020) optimizes a meta network to selectively use unlabeled data; OpenMatch (Saito et al., 2021) trains an outlier detector with soft consistency regularization. These robust SSL methods are used as additional baselines for evaluation on imbalanced SSL in Section 4.1. Exploring new methods in OOD detection not the focus of our work. Instead, we show that leveraging the concept of OOD detection, and in particular, the energy score (LeCun et al., 2006; Liu et al., 2020) for pseudo-labeling, provides a new perspective imbalanced SSL that results in significant improvement under imbalanced settings.
3 APPROACH
Our goal is to devise more reliable way of pseudo-labeling in imbalanced SSL, which accounts for whether an unlabeled data sample can be considered "in-distribution" or "out-of-diisrribttion" based on the existing set of (pseudo-)labeled samples. To help explain approach, Inlier Pseudo-Labeling (InPL), we first overview the consistency regularization with confidence- based beeuo-llbbeling framework (Sohn al., 2020; Zhang et al., 2021; Xie et al., 2020a) that sfte-of--heean SSL methods build upon, as our method simply replaces one step - the pseudo-labeling criterion. We then provide detailed explanation of InPl..
3

---

Published as a conference paper at ICLR 2023
max pla X))
E(x,
Model
0.7
poftmax
0.01
?
Model
Eneray
-8.25
?
0.7
0.01
0.7
E(x,
1.6
0.02
1.6
f(x)
p(yix)
Label
f(x)
Laeudo-Label
(a)
Confidence-based Pseudo-Labeling (Prior work)
(b) Inlier Pseudo-Labeling (Ours)
Figure 2: Overview of Confidence-bauee Sheudo-Labeling vs. Inlier Pseudo-Labeling
3.1 BACKGROUND: CONSISTENCY REGULARIZATION WITH CONFIDENCE-BB E-BASED PSFUDO-LLABFLING
The training of pscudo-labeling SSL methods for image classification involves two loss terms: the supervised loss L, computed on human-labeled data and the unsupervised loss L,, computed unlabeled data. The supervised loss typically the standard multi-class cross-entropy loss computed
(oby.b))ja on stukly-augmenter views (e.g.. flip and crop) of labeled images. Let X' {(xb.b))) [b)) be the labeled set where and y denote the data sample and its corresponding one-hot label, respectively, Denote p[y|w(()))) = (x(())) as the predicted class distribution on input Xb. where w is a weakly- augmenting transformation and is classifier often realized deep network. Then at each iteration, the supervised loss for a batch B, of labeled data is given by
1 B,
C.
((yb,p(y(wxx)))
B b=
where H. is the cross-entropy loss.
Mainstream research in SSL focuses on how to construct the unsupervised loss. One dominating approach is consistency regularization (Laine & Aila, 2017; Tarvainen & Valpola, 2017; Berthelot et al., 2019; 2020), which regularizes the network to be less sensitive to input or model perturbations by enforcing consistent predictions across different views (augmentations) of the same training input, through an MSE or KL-divergence loss. Self-training (Lee al., 2013; Rosenberg et al., 2005; Rizve al., 2021; Xie al., 2020b) converts model predictions to optimization targets for unlabeled images. In particular, pseudo-labeling (Lee et al., 2013) converts model predictions into hard pseudo- labels. To ensure high quality of pseudo-labels, high confidence threshold is often used.
The recently introduced weak-strong data augmentation paradigm (Sohn et al., 2020; Xie et al., 2020a) be viewed the combination of these directions. When combined with confidence- based pseudo-labeling (Sohn et al., 2020; Zhang et al., 2021), at each iteration, the process can be summarized as follows:
1. For each unlabeled data point x, the model makes prediction p(ylw(xx) = fw((x)) on its weakly-augmentt view wxx).
2. Confidence thresholding is applied and a pseudo-label is only produced when the maximum predicted probability max; p(y (xx) of x is above a threshold To (typically, T. = 0.95).
3. The model then trained with its coongly-augmente ment (Cubuk et al., 2020) and CutOut (DeVries & Taylor, 2017)) along with its one-hot thresholded pseudo-label (x)) obtained on the weakly-augmented view.
view 2(x) (e.g., RandAug-
With batch size B, for unlabeled data, the unsupervised loss is formulated as follows:
Ba
C.
B 1 (max(p(x)w(xx)) {
where [[:] is the indicator function.
The final loss at each training iteration is computed by C= L, + XC,, with À as a hyperparameter to balance the loss terms. The model parameters are updated with this loss after each iteration.
4

---

Published as a conference paper at ICLR 2023
Confidence Energy
Confidence Energy
Energy
Entrg
Correct Pscudo Labels
Correct Psoudo Labels
Incorrect Pseudo-Labels
Incorrect Pseudo-Labels
0.6
0.9
(a) Confidence-based Pscudo-labeling
(b) Inlier Pscudo-labeling
The shaded region shows the unlabeled samples that are pseudo-labeled. Inlier Pseudo-Labeling can produce correct pseudo-labels for many low-confident unlabeled samples, increasing recall while filtering out many false positives.
Figure 3: Visualization: confidence vs energy score:
3.2 CONSISTENCY REGULARIZATION WITH INLIER PSEUDO-LABELING
Confidence-bil based thresholding (step in the above process) typically produces pseudo-label, with high precision in the balanced SSL setting, yet often leads to low precision for head classes and low recall for tail classes in the long-tailed scenario (Wei et al., 2021). Moreover, softmax-based confidence scores in deep networks are oftentimes overconfident (Guo et al., 2017), and can be arbitrarily high for samples that are far from the training data (Nguyen al., 2015; Hein et al., 2019). The implication in the imbalanced SSL setting is that the pseudo- label for even high-confidence unlabeled samples may not be trustworthy if those samples are far from the labeled data.
To address these issues, our method tackles the imbalanced SSL peeedo- process from different perspective: instead of generating pseudo-labels for high-confidence samples, we produce pseudo-labels only for unlabeled samples that are close to the current training distribution we call these "in-distribution" samples. The rest are confidences are deemed unreliable. The idea is that, training progresses and more unlabeled samples become m--iisrrbbution and contribute to training, the training distribution will better approximate the true distribution improve the model, and in turn, improve the overall reliability of the pseudo-labels (see Figure 1).
samples, for which the model's
To determine whether unlabeled sample is in-distribution fil-of--dstributionn score (LeCun et al., 2006) derived from the classifier f. The energy score is defined as:
the energy
E(x, f(x)) - log
(((x)TT)
where x is the input data and fi(x) indicates the corresponding logit value of the i-th class. K is the total number of classes and a tunable temperature.
When used for conventional OOD detection, smaller/higher energy scores indicate that the input is likely to be iredistributioonoutt---ddssiisib
Indeed, a discriminative classifier implicitly defines a free energy function (Grathwohl et al., 2020; Xie et al., 2016) that can be used to characterize the data distribution (LeCun et al., 2006; Liu et al., 2020). This is because the training of the classifier, when using the negative logeliild loss, seeks minimize the energy of in-distribution data points. Since deriving and analyzing the energy function not the focus of our paper, we refer the reader to Liu et al. (2020) for detailed connection between the energy function and OOD detection.
In our Inlier Pseudo-L Labulig framework, we compute the energy score for each unlabeled sample and only generate pseudo-label when the corresponding energy score less than pre-defined threshold Te. which indicates that the unlabeled sample is close to the current training distribution. The actual pseudo-label obtained by converting the model prediction on the augmendd view of (xb) to a one-hot labul.-label. Formally, the unsupervised loss is defined as:
1 L.
B Σ
([E(((x)) re)
We illustrate the key difference between confidence- pseudo- and energy- pseudo- labeling in Figure 2. It is worth noting that the energy score is not the only possible metric to implement our InPL approach and it is possible other OOD detection metries for the objective.
5

---

Published as a conference paper at ICLR 2023
CIFARIO-LT
CIFARIO0-LT
T 50
7 = 100
7 200
Y= 50
7 == 100
UDA (Xie et al., 2020a)
80.2 1:049
72.19
63.3241677
46.791076
41.471097
FixMatch (Sohn et al., 2020)
80.84+00.0
72.95
63.25
46,99
41.49
FixMatch-UPS (Rizve et al., 2021)
81.75
73.17:1163
64.38
FixMatch-InPL w/o AML (ours)
83.36
76.05:10.4
66.47
48.03
42.53
FixMatch-DDebias + AMI (Wang et al., 2022)
83.53-20.67
76.92+1.72
67.70
50.24:0046
44.124081
Fi.Matc--ImPLLours)
83.92
1.14
68.47
49.96110.6
44.33
OpenMatch (Saito al., 2021) FixMatch-D3SL (Guo al., 2020)
81.01
13.15:1.03
63.22:1186
46,92.1028
40.761081
11.20.10.33
72.71
65.09
46.83
41.22.1099
Table 1: Top-1 aceuracy FixMatch variants on CIFAR 10-LT/100-LT. For CIFARIO-LT and CIFAR 100-LT,, we use 10% and 30% data as labeled sets, respectively. We use Wide ResNet-28- (Zagoruyko Komodakis, 2016) for CIFAR 10-LT and WRN-28-8 for CIFAR 100-LT. All methods trained with the default FixMatch training schedule (Sohn et al., 2020). Results are reported with the mean and standard deviation over 3 different runs.
We choose the energy score for its simplicity and easy integration to existing SSL frameworks such as FixMatch and ABC, which does not incur additional model parameters or significant computation cost (apart from the computation of the energy score).
To further tackle the model bias in long-tail scenarios, we can also optionally replace the vanilla cross-entropy loss (H(()) in the unsupervised loss computation with margin-aware loss (Li et al., 2019; Khan et al., 2019; Cao et al., 2019; Wang et al., 2022). We adopt the adaptive margin loss (Wang et al., 2022) with the margin determined through the moving average of the model's prediction:
(9(x)))) A.
LAMe. llg ()(()())))) + Ebzp efa((((x)))-A)
(5)
where (1(()))) represents the corresponding logits of class on
input (2(x)).
The margin is computed by A, A llog( J where P is the averaged model prediction updated at each iteration through an exponential moving average. We shall see in the experiments that InPL counterparts regardless of whether we use vanilla
outperforms the vanilla confidence-bart cross-entropy loss or the adaptive margin loss.
EXPERIMENTS
In this section, first compare our energy-based pseudo-labeling approach confidence-based pseudo-labeling approaches that build upon FixMatch (Sohn et al., 2020) and ABC (Lee et al., 2021) framework and compare state-of-the-art imbalanced SSL approaches. We evaluate our method on both standard imbalanced SSL benchmarks and the large-scale ImageNet dataset. Next, we provide analyses on why InPL works well in the imbalanced setting, and then evaluate InPL's robustness to real OOD samples in the unlabeled data. All experiments are conducted with multiple runs using different random seeds and we report the mean and standard deviation of the final accuracy. Following prior suggestion (Oliver et al., 2018) on SSL, we implement our method and baseline methods in the same codebase. Unless otherwise specified, results for each method are generated with the same codebase, same random seeds, same data splits, and same network architecture for the best possible fair comparison. Implementation details of all experiments can be found in Appendix A.2.
4.1 ENERGY-BASED PSEUDO-LABELING VS. CONFIDENCE-BASER PSEUDO LABELING
We first evaluate the effectiveness of InPL's energy-based pseudo labeling over standard confidence- based pseudo labeling. For this, we integrate InPL into the Match framework (Sohn et al., 2020) (denoted as FixMatch-InPL), and compare with the following FixMatch variants, which all use safidence-bbsed pseudo-labeling UDA (Xic al., 2020a), FixMatch- -ebiaed (Wang et al., 2022), and UPS (Rizve et al., 2021). UDA uses soft pseudo-labels with temperature sharpening: FixMatch- Debiased (Wang al., 2022) adds debiasing module before computing the confidence score in

---

Published as a conference paper at ICLR 2023
Dataset
CIFARIO-LT
CIFARIOO-LT
Imbalance Ratio
100
150
200
7 20
FixMatch (Sohn et al, 2020)
72.32001 /53.8
68.5 / 45.8
66.32009
/42.42044444
51.0z0.0/ / 32.8
w/ DARP+cRT (Kim ct at., 2020)
78.1 666.611555
73.2+085 / 17.1
=0.7 / 4
w/ CReST+ (Wei et al., 2021)
76.62046 61.4
70.0 4:.45
51.6=0299 / 36,4404
w/ ABC (Lee et al,, 2021)
81. 21.1 / 72.0
77.1 / 64.4
73.9 58.1 1:2722
56.32019/ 43,4g0-4
w/ ABC-IaPL (ours)
82.9 / 16.4
79.7 / 70.8
76.4=109/
63.7.2208
57.7 / 46,4
RemixMatch (Berthelot al, 2020)
73.7 / 55.9
69.92u2s / 48. 2..40
68.2 445 4.070
24.0.0.2 / 37.14
w/ DARP+<RT (Kim al., 2020)
78.5*asi / 66.441169
73.9 s0 / 45
=0.1 / $3.66
w/ CReST+ (Wei et al., 2021)
75.7 / 59.6600.5
71.3 / 50.8
54.6=0.45 / 38. 1
w/ ABC (Lee et al,, 2021)
82,4 75.77
80.6 / 77.1 72.1
78.8_uz7
69.9 29.9
27.6 / 44.7 46.74
w/ ABC-InPl
83.62045 / 88.77009
81.3 / 76.8:0.38
78.8 74.5
58.4 48,9
Table 2: Top-1 accuracy on long-tailed CIFAR10/100 following ABC et al., 2021) evaluations. We use 20% labeled data for CIFARIO-LT and 40% labeled data for CIFARIOO-LT, We report both the overall accuracy (before "f"") and the accuracy of minority classes (after "/").
pseudo-labeling: UPS (Rizve et al., 2021) uses an uncertainty estimation with MC-Dropout (Gal & Chahramani, 2016) extra criteria in pseudo-labeling. We also compare with recent robust SSL methods, OpenMatch (Saito et al., 2021) and D3SL (Guo et al., 2020), as they connect SSL to OOD detection. Since UPS and D3SL do not use strong augmentations, for fair comparison, we integrate them into the Fix.Mtach framework, denoted by FixMatch- UPS and FixMatch-D33L. We evaluate on CIFARIO-LT and CIFAR (Krizhevsky & Hinton, 2009), which are long-tailed variants of the original CIFAR datasets. We follow prior work in long-tail SSL (Wei et al., 2021; Lee et al., 2021), and use an exponential imbalance function (Cui et al., 2019) to create the long-tailed version of CIFARIO and CIFARIOO. Details of constructing these datasets are in Appendix A.I.
InPL outperforms confidence-based pseudo-labeling approaches for imbalanced SSL. Our FixMatch-InPL shows a significant improvement over FixMatch variants with confidence-based pseudo-labeling by a large margin (e.g., 3% absolute percentage over FixMatch for CIFARIO-LTY) see Table 1. For CIFARI00-LT, InPL reaches 48.03% and 42.53% average accuracy when 7 = 50 and 7 100 respectively, outperforming other methods. UPS is the closest competitor as it tries to measure the uncertainty in the pseudo-labels. Using UPS in FixMatch improves the performance over vanilla FixMatch, yet it still cannot match the performance of InPL on CIFARIO- Moreover, UPS requires forwarding the unlabeled data 10 times to compute its uncertainty measurement, which is extremely expensive in modern SSL frameworks such as FixMatch. In comparison to FixMatch- UPS, InPL provides strong empirical results on long-tailed data and remains highly efficient. Comparing InPL. and FixMatch-debiasee improvement on CIFARIO-LY We also outperform the robust SSL methods, OpenMatch and D3SL. Overall, these results show the efficacy of InPL over standard confidence- pseudo-labeling for imbalanced SSL.
(both use AML), our approach again shows consistent
4.2 COMPARISON TO STATE-OF-THE AART IMBALANCED SSL APPROACHES
Next, we compare InPL. to state-of-the-ar imbalanced SSL approaches. We integrate InPL into ABC (Lee et al., 2021), recent framework designed for imbalanced SSL (denoted as ABC-InPL), by replacing its confidence-base pseudo-labeling with our energy-based pseudo-labeling, We compare our ABC-InPL with vanilla ABC well as other state-of-the-art imbalanced SSL methods, DARP (Kim et al., 2020), CREST (Wei et al., 2021), Adsh (Guo & Li, 2022) and DASO (Oh et al., 2022). The results are shown in Tables 2 and 3, which correspond different ways of constructing the imbalanced SSI. datasets. See implementations details in Appendix A. and A.2.
InPI, improves the state-of-the-art for imbalanced SSL. Table shows that ABC-InPL consistently outperforms the confidence- counterparts on CIFAR10-LY and CIFAR100-LT under the ABC framework. For FixMatch base, when the imbalance ratio 7 = 150 and 7 = 200 on CIFARI IO-LT, InPL shows a 2.6% and 2.5% improvement, respectively (FixMatch w/ ABC vs. FixMatch w/ ABC-InPL). Further, we demonstrate the flexibility of InPL by incorporating it into RemixMatch, a consistency-based SSL method that uses Mixup (Zhang et al., 2018). With RemixMatch, InPL again consistently outperforms confidence-basee methods. Importantly, ABC- achieves significantly higher accuracy for minority classes across all experiment protocols (numbers reported after the "/"
7

---

Published as a conference paper at ICLR 2023
CIFARIO-LT
CIFARI00-LT
7
7= 20
N; 500
N2 1500
N1 500
N; 1500
N1 50
Ni 150
Ni 50
No 150
Mi 1000
= 3000
Mi 4000
M. 3000
Mi 100
Mi 300
Mi 200
M, 300
FixMatchi (Sohn et al., 2020)
71.5
62.9
22.4
Adshf 2019)
76.3
78.1
67.5+0045
73.7a0.34
FixMarch (Sohn et 2020)
67.8
77.5++1.22
62.9
72.4+108
45.2
56.5+000
40.0=a..
=0.7
DARP (Kim al,
77.81003
67.2.49.22
49.4
58.1
52.2:wes
w CREST+ (Wei et al., 2021)
=6.3
78.1+
67 67.5++045
73.7-0034
44,5=0034
57.4=au
40.1+122
50.1=021
DASO al, 2022)
76.0=037
19.1.
8:.1
49.8-0024
59.24333
43.6aam
52.9=041
w/ ARC (Les ct 202))
78.9
83.8
66.511978
8D. 1s
47.5
59.1
41.6 11665
53.7,05
ABC--ASSO (Oh 2022)
80.1
83.410m
70.6.1000
80.4:006
50.2.003
60.01am
25.3
ABC-IaPP (Ours)
81.4
84.4
77,5+1.57
80,9:082
51.8-129
61.0
44.6
55.1=n3i
Table 3:
Top-1 accuracy on long-tailed CIFARI0/100 compared with SSL-LT methods following DASO (Oh et al., 2022) evaluations. N₁ and M1 represent the number of instances from the most- majority class. tWe use the reported results in Adsh (Cui et al., 2019) due adaptation difficulties.
in Table 2). This indicates that our "in-distribution out-of-distribution" perspective leads more reliable pseudo-labels when labeled data is scarce, compared to confidence- based selection.
Table 3 shows more results under different combinations of labeled instances and imbalance ratios. Across almost all settings, InPL consistently outperforms the baselines, sometimes by a very large margin. For example, the most challenging scenario (y 150 and N1 500), ABC-InPL achieves 77.5% absolute accuracy, which outperforms the state-of-the- art ABC-DASO baseline by 6.9% absolute accuracy. These results demonstrate that InPL achieves strong performance on imbalanced data across different frameworks and evaluation settings.
4.3 RESULTS ON IMAGENET
We evaluate InPL on ImageNet- 127 (Huh et al., 2016), large-scale dataset where the 1000 ImageNet classes are grouped into 127 classes based on the WordNet hierar- chy. This introduces class imbalance with ratio 286. Due to limited computation, we unable to large batch sizes in FixMatch (Sohn et al., 2020) (1024 for la- beled data and 5120 for unlabeled data). Instead, we use a batch size of 128 for both labeled and unlabeled data following the recent SSL benchmark (Zhang et al., 2021). As shown in Table 4, InPL outperforms the vanilla FixMatch with confidence-bast pseudo-labeling by large margin, which further demonstrates the efficacy of our approach on large-scaled datasets.
ImageNet- 127 (Imhulanced)
ImageNet (Balameed)
FisMatch
51.96
56.34
FixMatch-IPPL (ours)
$4.82
57.92
Table 4: Results on ImageNet- 127 and ImageNet. We use sample 10% data as the labeled set for ImageNet- 127 and use 100 labels per class for ImageNet. Our approach outperforms the confidence-baut baseed counterpart in FixMatch on both datasets.
While designed to overcome the limitations of confidence-basse pseudo-labeling on imbalanced data, InPL does not make any explicit assumptions about the class distribution. Thus, a natural question is: how does InPL. perform on standard SSL benchmarks? To answer this, we further evaluate InPI. on ImageNet with standard SSL settings in Table 4 (results on other standard SSI. datasets can be found in Appendix A.8) and InPL still shows favourable improvement. This shows that the energy-based pseudo-labeling also has potential to become general solution to SSL problems.
4.4
WHY DOES INPL WORK WELL ON IMBALANCED DATA?
To help explain the strong performance of InPL on imbalanced data, we provide a detailed pseudo- label precision and recall analysis on CIFARIO-LT Here, we refer to the three most frequent classes as head classes, the three least frequent classes tail classes, and the rest as body classes.
Figure 4 shows the precision and recall of our model's predicted pseudo-labels over all classes (a,c) and also for the tail classes (b,d). The analysis for the head and body classes can be found in Appendix A.4. Compared with FixMatch, InPL achieves higher precision for overall, head, and body pseudo-labels. Importantly, it doubles Fitchssh's recall of tail pseudo-labels without hurting
8

---

Published as a conference paper at ICLR 2023
1.00
1.0
0.8
0.0
0.6
3.95
0.6
0.8
0.4
0.2
2.85
50k 100k 150k 200k 250k
0.0
0 5ox 100K 150k 200K 250k
0.4
0
50k 100k 150K 2008 2506
0.0
50k 100k 150K 200K
(a) Precision: Overall
(b) Precision: Tail
(c) Recall: Overall
(d) Recall: Tail
Figure 4: Precision-Recall Analysis: We compare pseudo-label precision and recall between InPL and FixMatch. Orange and green curves denote FixMatch with threshold 0.95 and 0.6 respectively, InPL is shown in blue, which achieves improved recall for tail classes and better overall precision.
the precision much. This shows that InPL predicts more true positives for the tail classes and also becomes less biased to the head classes. Trivially lowering the confidence threshold for FixMatch can also improve the recall for tail pseudo-labels. However, doing significantly hurts its precision and does not improve overall accuracy. For example, as shown in Figure 4 (a) and (d), although FixMatch with lower threshold of 0.6 achieves higher recall for tail labels, the overall precision is significantly hurt, which results minimal improvement or degradation in overall accuracy. We further investigate various lower confidence thresholds for FixMatch and find that none of them leads to improvement in accuracy to match the performance level of InPL (see Appendix A.5).
4.5 ADDITIONAL ANALYSES
InPL is robust against OOD examples. In real-world scenarios, unlabeled data may not share the same of classes labeled data. To simulate this, evaluate our method with out-of-disrrbuttion data presenting in the unlabeled data. Here, we consider the scenario where unlabeled data contain examples from different datasets. We sample 4 labeled examples per class from CIFAR10 and use the rest of CIFAR 10 and SVHN as the unlabeled data. Another OOD robustness experiment under the scenario of class mismatch unlabeled data (Oliver et al., 2018) 8) can be found in Appendix A.6.
Although motivated by imbalanced SSL, InPI. also achieves strong performance under the aforementioned re- alistic conditions where unlabeled instances from different data domains are present the unlabeled set. We compare InPL with the confidence-bart based methods UDA (Xie et al., 2020a) and FixMatch (Sohn et al., 2020). When a large amount of OOD examples are in the unlabeled set, the overall performance of both methods decreases, but InPL. demonstrates significant advantage. Specifically, InPL outperforms UDA and FixMatch by 4.21% and 4.42% ab- solute accuracy, respectively. Moreover, InPL consistently includes less OOD examples in training (see Appendix A.6 for details). This experiment shows the robustness of InPL to true outliers.
Accuracy
UDA
62.81
FixMatch
62.60
FixMatch-InPL. (ours)
67.02
Table 5: Results when OOD samples from SVHN appear the unlabeled set when training a model on CIFAR10 10 clas- sification.
Other ablation studies. Additional ablation studies can be found in the Appendix including the choice of thresholds and temperatures, impact of adaptive margin loss, and more analysis.
DISCUSSION AND CONCLUSION
In this work, we presented novel "in-distribution vs. out-distribution" perspective for pseudo-labeling in imbalanced SSL, in tandem with our energy-based pseudo-labeling method (InPL). Importantly, our method selects unlabeled samples for pseudo-labeling based on their energy scores derived from the model's 's output. We showed that our method can be easily integrated into state-of-the-an imbalanced SSL approaches, and achieve strong results. We further demonstrated that our method renders robustness against out-of-distributio samples, and remains competitive on balanced SSL benchmarks. One limitation is the lack of interpretability of the energy scores; the energy score has a different scale and is harder interpret. Devising ways to better understand it would be interesting future work. Overall, we believe our work has shown the promise of energy- based approaches for imbalanced SSL, and hope that it will spur further research in this direction.

---

Published as a conference paper at ICLR 2023
Acknowledgements. This work was also supported in part by NSF CAREER ISS2150012, Wiscon- sin Alumni Research Foundation, and NASA 80N95C
REFERENCES
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raf- fel. MixMatchh A holistic approach to semi-supervised learning. Advances in Neural Information Processing Systems, 32, 2019.
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. ReMixMatch: Semi-supervised learning with distribution alignment and augmentation anchoring. In International Conference on Learning Representations, 2020.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced
datasets with label-distribution--wwae margin loss. Advances in neural information processing systems, 32, 2019.
Yanbei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong. Semi-supervised learning under class distribution mismatch. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3569-3576, 2020.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. RandAugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition Workshops, pp. 702-703, 2020.
Yin Cui, Menglin Jia, Tsung-Y Yi Lin, Yang Song. and Serge Belongie. blaasebalanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9268-9277, 2019.
Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 170.. 1705.0455, 2017.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, pp. 1050-1059.
PMLR, 2016.
Will Grathwohl, Kuan-Chich Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In International Conference on Learning Representations, 2020.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMLR, 2017.
Lan-Zhe Guo and Yu-Feng Li. Class-imbalanced suni-sueervii learning with adaptive thresholding. In International Conference on Machine Learning pp. 8082-8094, PMLR, 2022.
Lan-Zhe Guo, Zhen Yu Zhang, Yuan Jiang, Yu-Feng Li, and Zhi-Hua Zhou. Safe deep semi- supervised learning for unseen-class unlabeled data. In International Conference on Machine Learning, pp. 3897-3906. PMLR, 2020.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why ReLU networks yield high- confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 41-50, 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and duttof--istriiutiin examples in neural networks. In International Conference on Learning Representations, 2017.
Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What makes imagenet good for transfer learning? arXiv preprint /60.:vvv...0.. 04 2016.
10

---

Published as a conference paper at ICLR 2023
Salman Khan, Munawar Hayat, Syed Waqas Zamir, Jianbing Shen, and Ling Shao. Striking the right balance with uncertainty. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 103-112, 2019.
Jaebyung Kim, Youngbum Hur. Sejun Park, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin. Distri- bution aligning refinery of pseudo-label for imbalanced semi-super learning. Advances in Neural Information Processing Systems, 33:14567-14579, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprini arXiv: 14/2.6980 2014.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In International Conference on Learning Representations, 2017.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial energy-based learning. Predicting structured data, 1()), 2006.
Dong-Hyun Lee al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICMI., volume 3, pp. 896. 2013.
Hyuck Lee, Seungjae Shin, and Heeyoung Kim. ABC: Auxiliary balanced classifier for class- imbalanced semi-supervised learning. Advances in Neural Information Processing Systems, 34, 2021.
Zeju Li, Konstantinos Kamnitsas, and Ben Glocker. Overfitting of neural under class imbalance: Analysis and improvements for segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 402-410 410. Springer, 2019.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-off -iii-rr-iittitttionn image detection in neural networks. International Conference on Learning Representations, 2018.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based -fitt--iittinnttioon detection. Advances in Neural Information Processing Systems, 33:21 211-2464-21455, 2020.
Geoffrey J McLachlan. Iterative reclassification procedure for constructing an asymptotically optimal rule of allocation in discriminant analysis. Journal of the American Statistical Association, 70 (350):365-369, 1975.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. Workshop on Deep Learning and Unsupervised Feature Learning, NIPS, 2011.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 427-436, 2015.
Youngtaek Oh, Dong-Jin Kim, and In So Kweon. Daso: Distribution--wwar semantics-oriented pseudo-label for imbalanced supervupe learning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 9786-9796, 2022.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Real- istic evaluation of deep semi-supervised learning algorithms. Advances in Neural Information Processing Systems, 31, 2018.
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo- labeling: An uncertainty- aware pseudo-label selection framework for semi-supervised learning. In International Conference Learning Representations, 2021.

---

Published as a conference paper at ICLR 2023
Chuck Rosenberg. Martial Hebert, and Henry Schneiderman. Semi-supervised self-training of object detection models. In Seventh IEEE Workshops on Applications of Computer Vision (WACVMOTTOON 05)-oouume 1, 2005.
Kuniaki Saito, Donghyun Kim, and Kate Saenko. Openmatch: Open-set consistency regularization for semi-supervised learning with outliers. arXiv preprint
2021.
Henry Scudder. Probability of error of some adaptive pattern-recognition machines. IEEE Transac- tions on Information Theory, 113))363-371, 1965.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang. Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. FixMatch: Simplifying semi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems, 33:596-600 2020.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference on Learning Representations, 2014.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
Xudong Wang. Zhirong Wu, Long Lian, and Stella X Yu. Debiased learning from naturally imbalanced pscudo-labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14647-14657, 2022.
Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang. CReST: A class- self-training framework for imbalanced semi-supervised learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition, pp. 10857-10866, 2021.
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning, pp. 2635-2644 PMI.R, 2016.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems, 33:6256-6268, 2020a.
Qizhe Xie, Minh-Thang Thang Luong, Eduard Hovy, and Quoc V Le. Self Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 10687-10698, 2020b.
Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. International Conference on Machine Learning, pp. 11525-11536. PMLR, 2021.
Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Multi-task curriculum framework for open-set semi-supervised learning. In European Conference on Computer Vision, pp. 438-454. Springer, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference. British Machine Vision Association, 2016.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexh:tch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 34, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
This document complements the main paper by describing: (1) the construction of long-tailed datasets for semi-supervised learning (Appendix A.1); (2) training details of each experiment in the main paper (Appendix A.2); (3) ablation studies on choices of wewphh-pprameeee parameters (Appendix A.3); (4) more precision and recall analysis for pseudo-labels (Appendix A.4); (5) top-1 accuracy of FixMatch with different thresholds (Appendix A.5); (6) more results on realistic evaluations where real OOD examples are presented in the unlabeled sets (Appendix A.6); (7) contribution of adaptive margin loss in InPL (Appendix A.7); (8) additional results on standard SSL benchmarks (Appendix A.8); (9) theoretical comparison between confidence score and energy score.
For sections, figures, tables, and equations, we use numbers (e.g.. Table 1) to refer to the main paper and capital letters (e.g.. Table A) refer to this supplement.
A.1 CONSTRUCTION OF LONG-TAILED CIFAR
In this section, we describe the construction of the long-tailed version of CIFAR10/100 for semi- supervised learning. For experiments in Table 3, we follow prior work (Kim et al., 2020; Oh et al., 2022) and first set N1 and M1 to be the number of instances for the most frequent class in labeled sets and unlabeled sets, respectively. Next, with imbalance ratio % the number of instances for class in labeled sets is computed as N& N1 7 ---111//KK-1, where K is the total number of classes and N1 is the number of labels for the most frequent class. Likewise, the number of instances in unlabeled sets is computed as M2-- = M1 - (k- ()/(K1)),
For experiments in Table is similar to the aforementioned strategy except that M1 = D - N₁ where D is the total number of instances of each class. For CIFARIO I0-LT and C0-AR, D is 5000 and 500, respectively. For the experiments in Table 2, 20% labeled data for CIFAR 10-LT corresponds to N₁ = 1000 and Mi = 4000. 40% labeled data for CIFARI00-LE corresponds to N₁ 200 and Mj- 330.
we follow the construction strategy in ABC (Lee et al., 2021), which
A.2 TRAINING DETAILS OF LONG-TAIL EXPERIMENTS
For results in Table 1, we follow the default training settings of FixMatch (Sohn et al., 2020), yet reduce the number of iterations to x 26 following CREST (Wei et al., 2021). This is because models already converge within the reduced schedule. All other settings exactly the same as FixMatch. Please refer to the original paper for more details.
For results in Table 2, we follow the training settings of ABC (Lee et al., 2021). Specifically, for all the results, we use Adam (Kingma & Ba, 2014) as the optimizer with a constant learning rate 0.002 and train models with 25000 iterations. Exponential moving average of momentum 0.999 is used to maintain an ensemble network for evaluation. The strong augmentation transforms include Ragdenttent and CutOut, which is consistent with the standard practice in SSL. Batch size of both labeled set and unlabeled set is set to 64 and two grrongl--augmente views of each unlabeled sample are included in training. Following ABC, WideResNNet228. is used for both CIFARIO-LI and CIFAR 1 100-II for results in Table 2. Results are reported as the mean of five runs with different random seeds and the best accuracy of each run recorded.
For results in Table 3, we follow the training settings of DASO (Oh et al., 2022). Most of the settings are the same as ABC (Lee et al., 2021) except that SGD used as the optimizer. The base learning rate is set to 0.03 and evaluation is conducted every 500 iterations with total number of training iterations being 250000. The median value in last 20 evaluations of each single run is recorded and the results reported as the mean of recorded values over three different runs.
Note that when integrating InPL into the ABC framework, our energy-based pseudo-labeling is only applied the auxiliary blass-balanced classifier. The vanilla classifier is still trained with confidence-burl based pseudo-labeling because empirically we find no benefit of using energy-based based pseudo-labeling for both.
For all of our experiments, we simply set the weight of unsupervised loss to 1.0 without further hyper-parameter tuning.
13

---

Published as a conference paper at ICLR 2023
70
R5
76
Accuracy
78
Accurary
Accuraoy
88.8
74
82
70
Threshold
Threshold
Temperature
(a) Threshold: FixMatch Frame-
(h) Threshold: ABC Framework
(c) Temperature: FixMatch
work CIFARIO-LT
CIFARIO-LT
Framework on CIFARIO-LT
Figure A: Ablation study: (a) and (b): Effects different energy thresholds on FixMatch framework and ABC framework under CIFARIO-LT with imbalance ratio 100. For ABC framework, the ReMixmatch base is used. (c): Effects of the temperature parameter the energy function on CIFAR10-LT under Fix.Match framework.
A.3 ABLATION STUDIES ON HYPERPARAMETERS
We conduct ablation studies to better understand approach InPL., where we integrate it into the framework of FixMatch (Sohn et al., 2020) and ABC (Lee et al., 2021). Unless otherwise noted, experiments are conducted on CIFARIO-LT with imbalance ratio 100. For FixMatch framework, use 10% of labeled data and for ABC framework, we use 20% of labeled data.
Effect of energy threshold. The most important hyperparameter of our method is the energy threshold. Unlike confidence scores that range from 0 to I, energy scores are unbounded with their scale proportional to the number of classes. Our thresholds in the experiments are chosen via cross-validation with a separate sampling seed. We further experiment with different thresholds.
As shown Figure (a), within the FixMatch Framework, we observe a steady increase in the performance when the threshold goes from -7.5 to -9.5 and starts to decrease with even lower thresholds. This is because, with very low thresholds, the model becomes very conservative and only produces pseudo-labels for unlabeled samples that are very close to the training distribution; i.e., the recall pscudo-labels takes a significant hit. As for the ABC framework (Figure (b)), our method not very sensitive to the energy threshold. Most threshold values result in good performance but slightly higher threshold -5.5 achieves the best accuracy. Since ABC already performs balanced sampling, using higher energy threshold helps improve its recall of pseudo-labels without making the model more biased.
Effect of temperature for the energy score. Recall that the energy function (LeCun et al., 2006) has a tunable temperature T (Equation 3). Here, we empirically evaluate the impact of this parameter. As shown in Figure (c), the simplest setting of T 1 gives the best performance. We also experiment with much larger temperature T = 10, which leads to major drop in performance to 72.72% accuracy (not shown in the plot). As noted by prior work (Liu et al., 2020), larger T results in smoother distribution of energy scores, which makes it harder to distinguish samples. We find that T also results slightly worse performance. Therefore, we simply set temperature parameter reduce the effort of hyperparameter tuning for our method.
and omit the
A.4 PRECISION AND RECALL FOR HEAD AND BODY PSEUDO-LABELL
Our ablation study, presented in Figure of the main paper, compared the precision and recall of pseudo-labels for all classes and the tail classes. In this section, we further report the results (precision and recall of pseudo-labels) for the head and body classes. As shown in Figure B, pseudo-labee lauels produced by InPL consistently achieve higher precision with slightly lower recall across head and body classes. This further suggests that the model trained with InPL is less biased towards the frequent classes in comparison to seffidenne-baseed pseudo-labeling
A.5 ACCURACY OF FIXMATCH WITH DIFFERENT TURESHOLDS
As was shown in Section 4.4, trivially lowering the confidence threshold significantly hurts the pseudo-label precision. We further demonstrate the corresponding overall accuracy in Table A. We see that none of these confidence thresholds for FixMatch is able to match the performance of our approach InPI. in the long-tailed scenario, which further shows the effectiveness of our approach.
14

---

Published as a conference paper at ICLR 2023
1.00
1.0
1.0
0.9
2.95
0.0
0.3
2.90
50k 100k 150k 200k 250k
0.6
50k 100k 150k 200K 250k
0.7
0
50k 100k 150K 2008 2506
0.0
50k 100k 150k 2002 250k
(a) Precision: Head
(b) Precision: Body
(c) Recall: Head
(d) Recall: Body
Figure B: Precision-Recal Analysis on Head and Body Classes: Orange and green curves denote
FixMatch with threshold 0.95 and 0.6 respectively. InPL is denoted by blue curves. InPL consistently achieves higher pseudo-label precision with slightly lower recall compared with the confidence- pseudo-labeling baselines.
FixMatch (Sohn et al., 2020)
InPL (ours)
Confidence threshold
7 0.95
T = 0.8
T == 0.7
T = 0.6
Accuracy
73.73
74.12
71.53
73.55
77.03
Table A: Comparison to FixMatch with various confidence thresholds on CIFARIO-LT. Results are generated with one run of 10% labeled data and imbalance ratio 100 with the same random seed.
A.6 MORE RESULTS ON ROBUSTNESS TO REAL OOD SAMPLES
We present experimental results of InPL under the scenario of class mismatch in unlabeled data (Oliver al., 2018). Specifically, in labeled data and only evaluate the model on these classes. The unlabeled data comes from four classes with different mismatch ratios. For example, with 50% mismatch ratio, only two of the four classes in the unlabeled data are among the six classes in the labeled data.
six animal classes (bird, cat, deer, dog, frog, horse) of CIFAR 10
As shown in Figure Ca, when the mismatch ratio is low, all methods with strong augmentation perform similarly. However, when the mismatch ratio becomes larger, InPL consistently achieves better performance. When unlabeled data contains OOD examples from different datasets (Table 5), InPL significantly outperforms FixMatch. Figure Cb also shows that InPL consistently pseudo- less OOD examples in training.
Dataset
CIFARIO-LT
CIFAR100-LA
Imbalance Ratio
? = 100
? == 150
7 200
= 20
ReMixmatch ABC-InPL w/o AMI.
83. 10.8
80.8+11003
78.9
57.5
ReMixmatch + ABC-InPL
83.6+00.5
81.3+008
7
58.4=0.25
Table B: Impact of AML the ABC framework.
A.7 CONTRIBUTION OF ADAPTIVE MARGIN LOss
Finally, we investigate the contribution of the adaptive margin loss (AML) over vanilla cross-entropy loss in our approach InPL. Within the FixMatch framework, the impact of AML can be derived from Table I where AML brings an additional 1 1-2% point improvement in overall accuracy. For the ABC framework, we use ReMixmatch + ABC-InPL as an example. As shown in Table B, AML generally brings in less than 1% point accuracy improvement on top of the energy-based pseudo-labeling.
A.8 RESULTS ON STANDARD SSL BENCHMARKS
While designed to overcome the limitations of confidence- based pseudo-labeling imbalanced data, InPL does not make any explicit assumptions about the class distribution. Thus, a natural question is: how does InPI. perform balanced SSL benchmarks? To answer this, we integrate InPI. into
15

---

Published as a conference paper at ICLR 2023
CIFAR10, 6 Animal Classes, 40 Labels Each
Number of OOD Examples Being Pseudo- -ablle
UDA FiaMatch InPL
36
300
Test Error (%)
34
200
100
0
75%
5K
50K
9SK
140K
185K
230K
275K
320K
365K
Class Mismatch Ratio
(a) Comparison of test error CIFAR-10 10 (six ani- mal classes) with different class mismatch ratio.
(h) Number of true OOD examples included
Figure C: (a): Comparison of test error on CIFAR-10 (six animal classes) with different class mismatch ratio. For example, "50%" " indicates that two of the four classes in the unlabeled data not present in the labeled data. InPL outperforms other state-of-the-ard methods when the class mismatch ratio is high. (b) Number of true OOD examples pseudo-labeled. InPL. consistently includes less true OOD examples compared with FixMatch and FlexMatch.
FixMatch (Sohn et al., 2020) and evaluate it on the balanced CIFAR1O (Krizhevsky & Hinton, 2009), SVHN (Netzer et al., 2011), and STL-10 (Coates et al., 2011) benchmarks using the default FixMatch training settings.
Table C summarizes the results. When the amount of labeled data is small (CI- FAR N = 40 and SVHN N = 40), InPL shows noticeable improvement over FixMatch, UDA, and DASH (Xu et al., 2021). This shows that the energy-based pseudo-labeling can still be useful in low- data regimes where confidence estimates may be less reliable even if the training data balanced over classes. In other settings with more data (STL-10 N 1000), InPI. does not significantly outper- form confidence-batt based methods such FixMatch, but still shows competitive per- formance.
CIFARIO
SVHN
STL-
N = 40
N = 40
N = 1000
UDA
89.384175
94.881422
93.362017
DASH
86.78
96.97
92.741040
FixMatch
92.53.10.28
16.1911.18
93.751003
FixMatch-InPLL (ours)
94.58
97.78
93.82
Table C: Top-1 accuracy on CIFAR10, SVHN, and STL- 10. Duue to adaptation difficulties, report the results of DASH from its original paper (Xu et al., 2021), 1), which uses a different codebase. All methods (including ours) use the same backbone across experiments.
A.9 THEORETICAL COMPARISON BETWEEN CONFIDENCE SCORE AND ENERGY SCORE
We borrow the analysis from prior work (Liu et al., 2020) show the theoretical comparison between confidence scores and energy scores. The energy score for an in-distribution data point gets pushed down when the model is trained with the negative likeliieldoood (NLL) loss (LeCun al., 2006). This can be justified by the following derivation. For an in-distribution data x with label y, with the energy definition from Equation 3, the gradient can be expressed as:
0Lmu (x, 0)
1 (E(L,
1 8E(r, v)
e E(EN)/T
08
T 00
T
00
OF(x. u)
(F(,,))
(1 p(Y = 7/2)
20 p(Y =
00
Au
energy pushed down for
1 emergy pulled up for other labels
16

---

Published as a conference paper at ICLR 2023
The energy for label y gets pushed down whereas the energy for other labels are pulled up. Since the energy is dominated by the label y, during training, the overall energy is pushed down by the NLL loss for in-distribution data.
The softmax score has connections to the energy score. The log of max softmax confidence can be decoupled as the energy score and the maximum logit value:
log max ply (t) = B(z; [(x)
f (x:)
in-dist
For in in-dist
The energy score gets pushed down during training whereas the maximum logit term gets pulled up. These conflicting trends make the softmax score unreliable distinguish in-distribution and out-of-distribution examples.
In this work, we propose to treat the pseudo-labeling process as an evolving in in-distribution and out-of-distributtin classification problem where in-distribution examples are defined jointly with human-labeled data and pseudo-labeled data. Following this direction, we choose the energy-score as our pseudo-labeling criterion.
17


Reviews:
[{"review_id": "qCapd3I9Sen", "reviewer": "Reviewer_EW6x", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a new pseudo-labeling method based on the energy instead of a confidence for its application to imbalanced semi-supervised learning. Experimental results show improvement over existing methods.", "summary_of_the_review": "The paper presents a simple trick that works well across SSL benchmarks. The paper could be improved with more in-depth analysis on why it works better, but overall I think it is a simple method that would benefit the community.", "strength_and_weaknesses": "* Strength\n  - The paper is written clearly. \n  - The proposed method is very simple and shown to be effective. While simple, I tend to believe it is more than incremental, as it demonstrates consistent improvement across multiple SSL settings.\n\n* Weakness\n  - The comparison between the energy-based scoring and the confidence-based scoring should be more in-depth. Especially, the negative energy score $T \\log \\sum_{i=1}^{K} \\exp(\\frac{f_{i}(x)}{T})$ approximates a max logit $\\max_{i=1}^{K} f_{i}(x)$ when there is a dominant one. Given such similarity, it is somewhat surprising that this simple change leads to such a huge difference.\n    - The energy formulation is not invariant w.r.t the addition of values to logits, i.e., $E(f(x)) \\neq E(f(x) + \\mathrm{constant})$, while confidence is, i.e., $\\mathrm{confidence}(f(x)) = \\mathrm{confidence}(f(x) + \\mathrm{constant})$. Would this difference be one of the reasons why energy formulation is better than the confidence, as the magnitude of the embedding is taken into account?\n    - One interesting control experiment is to use a max-logit-based pseudo labeling, as it is similar to energy score in the sense that it is variant to the addition to logits.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "ddNZmpqX5zy", "reviewer": "Reviewer_b7AL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a new perspective of pseudo-labeling for imbalanced SSL.\nWithout relying on model confidence, they propose to measure whether an unlabeled sample is likely to be “in-distribution,”; i.e., close to the current training data. \nTo decide whether an unlabeled sample is “in-distribution” or “out-of-distribution,” the authors adopt the energy score from out-of-distribution detection literature. \nAs training progresses and more unlabeled samples become in-distribution and contribute to training, the combined labeled and pseudo-labeled data can better approximate the accurate class distribution to improve the model. \n", "summary_of_the_review": "Despite several concerns about the experiments, the observation in this work is interesting, and the proposed method is novel.", "strength_and_weaknesses": "Strength\n- The authors propose a new perspective to tackle the imbalanced semi-supervised learning problem, and the proposed solution is original.\n- The proposed method shows favorable performance compared to the recent methods. \n\nWeakness\n- It would be better to improve the explanation of the motivation. For example, Fig. 3 says that the energy-based criterion selects fewer incorrect samples for the pseudo-labels. However, it is hard to see that in the figure. It would be better to provide a quantitative measure to show precisely how many incorrect samples are selected with which criterion.\n- Also, instead of discarding the fixmatch-based criterion, how about using both criteria? In other words, use the pseudo-labels when the confidence is above a certain threshold, and the energy is below a certain threshold.\n- More importantly, there is no analysis of the design choice for the proposed method. Instead of simply borrowing the popular energy function, it would be better to analyze how to modify the loss function or what is the best hyper-parameter (at least about the energy threshold) for the best performance.\n- Finally, there is no evaluation result on realistic datasets such as iNaturalist or ImageNet datasets. Otherwise, it is hard to verify that the proposed method is helpful for realistic scenarios.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OLbpMcRIYRI", "reviewer": "Reviewer_YGBu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles an imbalanced semi-supervised learning (SSL) problem by proposing a new way to construct the pseudo labels of unlabeled samples, coined INlier Pseudo-Labeling (INPL). Unlike the previous approaches, which use the softmax confidence from the training classifier, INPL instead uses the energy score calculated by logsumexp of logits. As INPL only changes the score function for unlabeled samples, it is compatible with the existing imbalance or semi-supervised learning methods. Also, it does not incur additional computational costs. Significant empirical improvements on widely used imbalanced SSL benchmarks (CIFAR-10-LT and CIFAR-100-LT) and more general scenarios (existence of OOD samples in unlabeled data or balanced class distribution) demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "As aforementioned, this paper is quite limited in the aspect of technical novelty. However, the empirical results are significant, and the presentation is clear. Hence, I believe that the acceptance of this work will be a useful addition to the ICLR community.", "strength_and_weaknesses": "**Pros.**\n\n- **Simple and effective solution for an important problem**. Considering the class imbalance scenario is an essential step for applying SSL in a more realistic scenario but has yet to be explored. The proposed method can be used with a simple modification of the existing baseline, and it shows significant empirical gain; hence, it has the potential to be widely used to mitigate this problem without additional burdens.\n\n**Cons.**\n\n- **Omitted baselines in robust semi-supervised literature**. The proposed idea, which considers the imbalanced SSL under the OOD detection framework, is interesting and somewhat new. However, at the same time, there are concerns about its technical novelty since many relevant works in robust SSL literature have yet to be mentioned and compared [1,2,3,4]. Overall, these works detect the OOD samples in unlabeled data using their score functions; while there are no works that use energy score, but using the energy score to detect the OOD samples is not a new one [5]. Hence, in some sense, the proposed method could be viewed as a combination of existing techniques for the new problem. So, it seems to be necessary to compare with the previous robust SSL methods as well. \n- **Absence of large-scale experiments.** Previous imbalanced SSL works have proposed the experimental results on large datasets such as ImageNet-127 [6] or LSUN [7]; however, INPL has only been demonstrated under relatively small benchmarks (CIFAR-LT-10 and 100). For a real-world application, experiments on such large benchmarks are essential.\n\n[1] Nair et al., RealMix: Towards Realistic Semi-Supervised Deep Learning Algorithms., arXiv:19.12\n\n[2] Guo et al., Safe Deep Semi-Supervised Learning for Unseen-Class Unlabeled Data., ICML 20\n\n[3] Saito et al., OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers., NeurIPS 21\n\n[4] Park et al., OpenCoS: Contrastive Semi-supervised Learning for Handling Open-set Unlabeled Data., ECCV 22 (w)\n\n[5] Liu et al., Energy-based Out-of-distribution Detection., NeurIPS 20\n\n[6] Wei et al., CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning., CVPR 21\n\n[7] Lee et al., ABC: Auxiliary Balanced Classifier for Class-imbalanced Semi-supervised Learning., NeurIPS 21", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JdpPur9QSwb", "reviewer": "Reviewer_x1wG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies imbalanced semi-supervised learning. SOTA imbalanced semi-supervised learning methods are often based on pseudo-labeling and consistency regularization, which still relies on confidence thresholding. In this paper, the authors formulate the pseudo-labeling problem as a classification problem, i.e., using the energy score to determine where an instance is in distribution or out of distribution. The authors claim that the proposed approach is more reliable compare to existing ones. Empirical results demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "Generally speaking, this paper proposes a simple method for imbalance semi-supervised learning, but the proposed method is not new. The authors did not give a sufficient discussion about the proposed method.", "strength_and_weaknesses": "Strengths\n\n1.\tThe idea is simple and clear, and the paper is easy to understand and follow.\n\n2.\tThis paper studies an important problem in imbalanced semi-supervised learning, i.e., the reliability of the pseudo labels.\n\nWeaknesses\n\n1.\tThe authors claim that the confidence-based methods are not reliable in imbalance semi-supervised learning, but it is not clear why the proposed method is reliable. The proposed method and the existing methods both depend on pre-defined thresholds.\n\n2.\tThe motivation of the method is not very strong, and the contribution of this work is limited. The authors only change the criterion to pseudo-labeling an instance with energy score without proposing a new SSL approach.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster