Unifying Distribution Alignment as a Loss for Imbalanced Semi-supervised Learning
Authors: Justin Lazarow
Kihyuk Sohn
Chun-Liang Li
Zizhao Zhang
Chen-Yu Lee
Tomas Pfister
Venue: ICLR 2022 Submitted (2022)
Keywords: semi-supervised learning
imbalanced learning

Abstract: While remarkable progress in imbalanced supervised learning has been made recently, less attention has been given to the setting of imbalanced semi-supervised learning (SSL) where not only is a few labeled data provided, but the underlying data distribution can be severely imbalanced. Recent works require both complicated sampling-based strategies of pseudo-labeled data and distribution alignment of the pseudo-label distribution to accommodate this imbalance. We present a novel approach that relies only on a form of a distribution alignment but no sampling strategy where rather than aligning the pseudo-labels during inference, we move the distribution alignment component into the respective cross entropy loss computations for both the supervised and unsupervised losses. This alignment compensates for both imbalance in the data as well as the eventual distributional shift present during evaluation. Altogether, this provides a single, unified strategy that offers both significantly reduced training requirements and improved performance across both low and richly labeled regimes and over varying degrees of imbalance. In experiments, we validate the efficacy of our method on SSL variants of CIFAR10-LT, CIFAR100-LT, and ImageNet-127. On ImageNet-127, our method shows 1.6% accuracy improvement over the previous best method with 80% training time reduction.

Under review as a conference paper at ICLR 2022
UNIFYING DISTRIBUTION ALIGNMENT AS A LOSS FOR IMBALANCED
LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
While remarkable progress in imbalanced supervised learning has been made re- cently, less attention has been given the setting of imbalanced sumirsupre learning (SSL) where not only is few labeled data provided, but the underlying data distribution can be severely imbalanced. Recent works require both compli- cated sumpling-based strategies of aseudo-labelle, data and distribution alignment the pscudo-label distribution to accommodate this imbalance. We present a novel approach that relies only form of distribution alignment but no sam- pling strategy where rather than aligning the pseudo-labels during inference, we move the distribution alignment component into the respective cross entropy loss computations for both the supervised and unsupervised losses. This alignment compensates for both imbalance in the data as well as the eventual distributional shift present during evaluation. Altogether, this provides a single, unified strategy that offers both significantly reduced training requirements and improved perfor- mance across both low and richly labeled regimes and over varying degrees of imbalance. In experiments, we validate the efficacy of our method on SSL vari- ants of CIFARIO-LT, CIFARIO0-LT, and ImageNet-1277 On 1227 our method shows 1.6% accuracy improvement over the previous best method with 80% training time reduction.
INTRODUCTION
Semi-supervised learning (SSL) uses large pool of unlabeled data to learn classifier despite having access to only small amount of labeled data. Recently, techniques have been introduced (Berthelot et al., 2019; 2020; Sohn et al., 2020) which simplify the process while at the same time pushing performance to new levels. However, these approaches have focused on the cases where the class distributions are balanced for both the labeled and unlabeled data.
At the same time, work within the supervised learning community has shown renewed focus on imbalanced sometimes long-tailed learning owing to the fact that most data in the real world is not well-balanceed A variety of methodologies for this setting have been introduced (Kang et al., 2020; Menon et al., 2021; Ren al., 2020; Hong al., 2021). For instance, many have observed the bias that ordinary supervised learning techniques suffer from - favoring head classes over the less numerous tail classes. Kang al. (2020) show that softmax-based classifiers often produce classification weights which correlate with class frequency and thus reduces the class-balanced per- formance of the model. Resampling strategies (Chawla et al., 2002: He & Garcia, 2009; Buda al., 2018; Byrd & Lipton, 2019) (which sample from the pool based on the desired distribution) are often effective as well. By shifting from sampling the data distribution for the majority of training to more class-balanced regime at the end, one can learn a good representation while mitigating the aforementioned bias at the final classification layer. In addition, distributional shift (Hong et al., 2021) can be observed within existing protocols where training occurs an imbalanced dataset yet evaluation done with respect to a balanced one.
In this work, we study the combined setting of semi-supervised and imbalanced learning. In par- ticular, we consider both FixMatch (Sohn et al., 2020) and MixMatch (Berthelot et al., 2019) as base semi-supervised learners. Both employ two losses: cross entropy loss the labeled data, and an unsupervised loss that relies on consistency between the classifier outputs among augmented versions of unlabeled examples.

---

Under review as a conference paper at ICLR 2022
75
Figure 1: Large-scale performance on ImageNet-127 with 10% of labels available. The proposed UDAL trained for only 1/6 of the epochs of CReST+ (Wei et al., 2021) further closes the gap in accuracy to fully supervised baseline and heavily improves on lower bound supervised baseline trained on 10% of the labels.
65
FixMatch
DA, min 0.5
60
CReST+
UDAL (ours) Supervised (10% labels)
50
Supervised (100% labels)
45
1000
2000
3000
4000
5000
5000
Total Training Epochs
Therefore, vulnerability to bias from the imbalance can happen in three ways: the supervised loss itself, the quality of pseudo- labels derived from the classifier unlabeled examples, and the pseudo- labels themselves can bias the classifier even if they are perfectly predicted. Furthermore, confirma- tion bias within semi-supervised learning is already a worrisome factor even without any imbalance between the classes (Arazo et al., 2020). Within the balanced setting. the ReMixMatch (Berthelot al., 2020) approach to cemi-supervised learning introduces strong regularization through distri- bution alignment (Bridle et al., 1992) (i.e. modifying the prediction by the ratio of the desired dis- tribution to model distribution) that distribution alignment also confers benefits in the imbalanced setting and aims to progressively rebalance the distribution of pseudo-labels. Similar to resampling approaches (Kang et al., 2020), imbalanced learning considers shift from random sampling to class-balanced sampling, CReST at- tempts to align the pseudo- labels themselves to a more balanced distribution as training progresses.
help mitigate this. Recently CReST (Wei et al., 2021) has shown
However, distribution alignment is not the only technique that CReST relies upon to achieve good performance. In addition, CReST requires a generational approach to self ceff-training which accumu- lates relatively balanced subset of confident pseudo-labels to augment the labeled set with as form of self- self-training. As the generations proceed, this subset is re-sampled to become more and more bal- anced. Each generation re-initializes the classifier's network, and therefore, the only "state" retained is through these accumulated pseudo-labels (now treated as ordinary supervised labels). This pro- cess can be extremely costly with respect to training time, and hypothesize that it is not optimal because it fails to directly address the imbalance in the labeled data. Instead, we seek a simpler solution to imbalanced supervueeee learning through distribution alignment alone.
We ask the question: is this disjoint methodology truly necessary? Can single. central approach be devised address imbalance in semi-supervised learning? In this work, as our contributions, we identify an affirmative answer to this question by connecting the ideas of progressive distribution alignment from Berthelot et al. (2020); Wei et al. (2021) and the method of logit adjustment from fully supervised, imbalanced learning (Menon et al., 2021; Ren et al., 2020; Hong et al., 2021). Furthermore, this approach can be implemented with only a few lines of code, has significantly reduced training time requirements, and generally outperforms previous work. Finally, it shows significantly better performance characteristics as more labeled data becomes available and readily scales larger datasets achieving 1.6% increase in accuracy compared to the best existing method on ImageNet-127.
2 PREREQUISITES
We present background information on both problem settings. First, we formally define the problem setting of Imbalanced Semi-Supervised Learning (SSL). Second, we outline the idea of distribution alignment (Berthelot et al., 2020; Wei et al., 2021) to improve pseudo-label quality within both the balanced and imbalanced settings of SSL. We revisit recent method, CReST (Wei et al., 2021), which also attempts to address imbalanced semi-supervised learning.
2.1 CLASS-IMBALANCEE SEMI-SUPERVISED LEARNING
Semi-supervised learning relies on two sources of data: a labeled set X ((x 11)) where each I, is a training example and y/ is the corresponding target. Since classification is the focus of this work, we consider yi as a class label within = (1....,C) C'} with a total number of C' classes. In imbalanced learning, we expect varying numbers of training examples across classes. Therefore,
2

---

Under review as a conference paper at ICLR 2022
denote the number of examples in our labeled set corresponding to class C as Ne such that Ee No N. We assume that the classes are ordered with respect frequencies and in descending manner i... No 2 Ne+1- It is often useful to characterize the degree of imbalance by the ratio N1 /Nc and we refer to this as the imbalance ratio of the dataset. We use Pdat (y) and g(y) to denote the marginal distributions of the data and the model. When there is no ambiguity, we drop as Pdata and q to simplify presentations.
M Additionally, we have an unlabeled set of examples 24 = { 1 for which we have no correspond- ing target. While we expect that this set is also imbalanced, we additionally make the common assumption (Wei et al., 2021) that it follows the same class distribution and thus shares the same im- balance (ratio) as the labeled set. Finally, an important measure 3 = N+M of overall examples that are labeled.
N considers the percentage
2.2 DISTRIBUTION ALIGNMENT
Distribution alignment (DA) was re-introduced in the setting of semi-supervised learning within ReMixMatch (Berthelot et al., 2020). To mitigate the tendencies of semi-supervised learning to suffer from confirmation bias, regularization can be added to the pseudo-label inference step. In particular, if we assume the labeled and unlabeled data both come from the same distribution Paata (although, we do not know particular labels from the unlabeled set), we would expect that our model should produce pseudo-labels that follow the same distribution. This marginal distribution of the model (z) can be estimated by moving average, which we denote as q(v) or q. as in Berthelot al. (2020). If we denote our current model's predictions on unlabeled examples as (yy[zx), these predictions can be re- re-scd through dividing by & and multiplying by Pulta- After normalization, we have:
i (z) Normalize (((yla)
Pdata
q
In equation I, we assume element- wise operations beween ((v( Paata and é. Normalize(p) ensures p as a probability distribution which sums 1.
As noted in Wei et al. (2021), it is not always optimal to align the predictions directly to Pdata when Pdam is imbalanced. Rather, a smoothed form which (elementwise) exponentiates the distribution by a factor of a before normalization:
Pa Normalize (Pām)), 0 S a 1.
(2)
is used instead of paat and found to both regularize the predictions as well combat bias. As a - 0, this approaches an alignment against a more uniform distribution.
2.3 LOGIT ADJUSTMENT
While DA is clearly applicable to pseudo-label inference during training, it has no direct effect on the labeled portion. Since a semi-supervised approach relies on both labeled and unlabeled losses, is critical to address the problem of imbalance at the supervised level well. For this, examine popular technique within supervised learning, often known as logit adjustment (Menon et al., 2021), balanced softmax (Ren al., 2020), LADE (Hong al., 2021). These methods modify the loss computation to compensate for the class imbalance found in the data distribution. Notably, when a data distribution dlas--mbalannced, we attempt to minimize the classification loss with respect to this data distribution. However, at evaluation time, we either evaluate on a balance dataset or produce class-balance error by averaging the per-class accuracies. This is a shift distribution which can cause poor performance. Therefore, this shift is integrated into the cross entropy loss:
LLA (y, f(x)) = Loe (y, f(x) + log Pdata - log (Unif(C")) = Lo (y. f(x) + log Pasta) (3)
where Unif(C) is the discrete uniform distribution over C' classes, y is the true label of x, f(x) is the vector-valued output of the classifier, and Pdata is the marginal class distribution of the data as a vector. As elaborated within Menon et al. (2021), this has the effect that instead of optimizing f(x) directly, we optimize h(r) f(x) + log Paata which aligns the source distribution of f(x) correctly to the uniform class distribution seen during evaluation. Menon et al. (2021) also discusses an inference time procedure which attempts account for this shift without modifications to the training procedure. Since this is "for free",, we include results combined with it in Table as "LA (Inf)" for the inference time procedure.
3

---

Under review as a conference paper at ICLR 2022
SSL Learer
Model
Pseudo-
Labels
SSL Leamer
Supervised
Supervised
Distributio Alignment
Model
Unsupervised
Sampling
Unsupervised
Label Refinement
(a) Previous work
(b) Ours
Figure 2: Given an off-the-shelf Semi-Supervised Learning (SSL) learner, previous works address the data imbalanced issue the unsupervised loss by iterative (a) class-rebalancing sampling (Wei et al., 2021) or pseudo-label refinement (Kim et al., 2020). In this work, we propose to tackle the imbalance issue on (b) both supervised and unsupervised losses by directly performing distribution- aligned learning.
2.4 CREST
We briefly examine CReST (Wei et al., 2021), an approach to imbalanced semi-supervised learning using Fix and MixMatch base semi-supervised learner. They re-introduce generational self-training approach where after each generation (usually 64 epochs), confidently pseudo-laie laado-laheeed examples from the unlabeled set are added X. Ordinarily, this would exacerbate class imbal- ance. Wei et al. (2021) introduce a re-sampling strategy according to a more balanced form of the marginal class distribution of the data to overcome. However, it requires to reinitialize and train the network from scratch after each generation.
Additionally, for optimal performance, CReST requires distribution alignment improve pseudo- label quality. Specifically, as defined in Section 2.2, a schedule for is chosen so that the strength of re-balancing can be altered over the course of training. Given hyperparameter Oninn which defines the minimal value should take (corresponding the largest re-balancing of the data distribution), they choose a linear schedule such that for generation iteration t:
0t 1.0 (1.0 -0min)
where T is the number of generations over the course of training. This is referred to as progressive distribution alignment and is essential to strong performance.
3 PROPOSED METHOD
We describe our approach, Unifying Distribution Alignment as Loss (UDAL). First, we connect Sections 2.2 (distribution alignment) and 2.3 (logit adjustment). Then, examine how this allows us to apply the same mitigation to both the supervised and unsupervised components of modern consistency-bast based SSL methods (Berthelot al., 2019; Sohn et al., 2020) providing unified manner in which imbalance can addressed with respect to labeled and unlabeled data. Figure shows the direct comparison between the proposed UDAL and previous work.
3.1 A UNIFIED APPROACH
We argue that the essence of progressive distribution alignment as seen in Sections 2.2 and 2.4 sufficient for strong performance imbalanced cumisuupre learning if viewed through the lens of logit adjustment from Section 2.3. For the sake of presentation, we will use FixMatch (Sohn et al., 2020) as base semi-supervised learner, but our method is equally applicable other methods, including MixMatch (Berthelot et al., 2019). FixMatch consists of a cross entropy loss on the labeled data and cross entropy loss on unlabeled data with respect to inferred pseudo-labels:
(5) where f(x) the network's outputs, Plabel is labeled data distribution, Puntab is the unlabeled data distribution, PL produces hard pseudo-labels for confident predictions, while u strong and u weak are strong and weakly augmented versions of the unlabeled example u, respectively.
LFixMonch = En.y-Pins [CE (x))] + Eur-pan [Ce (u f
As described in Section 2.2 and 2.4, applying progressive distribution alignment modifies predic- tions of unlabeled data by aligning them target distribution. While this aligns the predictions
4

---

Under review as a conference paper at ICLR 2022
of the unsupervised branch, we must also align the supervised branch. However, distribution align- ment modifies the inference process and is unable to affect the known (true) labels of the supervised branch. Nonetheless, logit adjustment (Menon et al., 2021) demonstrates that logit adjustment can be used to align a supervised classifier to the uniform distribution Unif(C), Therefore, we examine whether it can simply replace the usage of distribution alignment during pseudo-label inference. Since we are interested in aligning to an arbitrary target distribution like in DA, we consider a gen- eralization of Equation that aligns Pa. instead of Unif(C'):
LCE (u+ (x) + log (Pdaaa log (fa.)
6
Defining h(z) = fxx + log Ron Poce
and solving for f(x). this implies that we're optimizing:
Pas
f(x) - [(x)) + log
Pdota
or rather, in the softmax space:
Pa.
Softmax([(x)) = Softmax
(((x))
(8)
Pdata
This is exactly the form of applying distribution alignment to fxx) using P... Therefore, we hy- pothesize that not only can we align the supervised branch in this manner, but that we can also apply this exact same form to the unsupervised branch:
(
Lce y. fxx) + log
Pas T
(9)
where we divide by the estimated model's class distribution through moving average, rather than Pdata since the unsupervised branch is learned from pseudo-labels drawn directly from the model rather than the data distribution. This allows us to use the same, unified approach for each branch by simply replacing the distribution for the supervised branch and the moving average of pseudo-labels for the unsupervised branch. Altogether, this results loss:
aligning from: the marginal class distribution of the data
LUDAL = yrrpara [CCE(U: [(x + log (Peat)) - log (fa)))
+ Euwppttts [ACE (PI()( I (uiinuss + log (9) log (Fa,))) )))]
where a₁ is generalized from Equation 4 as:
t T
a₂ 1.0 (1.0 Crmin)
(10)
to allow for a rate of alignment k and where po, is computed as in Equation 2.
We find that sharing this exact same form between branches has performance benefits outlined in Section 1. At the same time, unlike distribution alignment, this form does not directly modify the PL inference procedure. One might wonder whether progressive form of alignment is even necessary for the supervised branch. We observe heavily degraded performance when progressive alignment is not used in Table under "LA (Sup)".
For MixMatch, an Le loss function is used between predictions and soft targets. We find that apply- ing the same adjustment procedure as in FixMatch but under the L2 loss continues work as long as m'nin is changed to compensate.
We provide pseudocode for this approach in Algorithm 1. This constitutes only a few lines of code with no alterations to the training scheme or time compared to the base FixMatch/MizMMtchh learner. Additionally, no resampling is required.
EXPERIMENTS
We follow the experimental settings outlined in CReST (Wei et al., 2021). As done in Wei et al. (2021), we make the assumption that the marginal class distributions of labeled and unlabeled data are equal. We show the performance of UDAL over long-tailed versions of CIFAR10 and CIFAR100, as well as CIFARIO in the "DARP" (Kim et al., 2020) regime. Finally, we perform large-scale experiments the naturally long-tailed ImageNet- 127 dataset.
5

---

Under review as a conference paper at ICLR 2022
Algorithm UDAL Pseudocode, TensorFlow-ish
P_data: class di distribution in of 1abel ad data
p_model: moving average
model's predictions on unlabeled data
$ current_epose (q) current epoch of training lout of max_epoot total]
which e_min is approsched
mmin: minimum value of alpha for distribution alignment
def compute_adjusteeee adjossmentttttddieee
(current saarrent_dist): factor 1.0 11.0 s_min) (current_epoch
max_epoch)
normalize ensures the argument aums to
target_diat normalize (p_mal (p_data factor)
return ourrent_dist (target_dist 1e-9)
compute supervised (labeled) loss on examples x_1
E: classifier
y_l: labels
1oss_1
SoftmaxCR(y_ f(x_1) + log (compute_ate adwpsmdent_ndddddeo dist (p_data)))
compuse unsupervised (unlabeled) loss on examples x_u
y_u: PLs predicted From weakly-sugnent x_u after zonf didece thresholding adjustment (compute_adjustneeeettteee
loss_u SoftmaxCE ty_u, I(x_s) Log (compute
dist (p m=_
Method
50
7= 100
7 = 200
MixMatch Berthelot et al. (2019)
69.1 11:1
60.4
54.5
w/ CReST+ Wei et al. (2021)
76.7
66.1 1079
57.611.0
w/ UDAL (ours)
77.8
68.4
58.6
FixMatch (Sohn et al., 2020)
80. +0.44
67.3
59.7
w/ DA (Berthelot et al., 2020) (aimii 0.5)
82.4
73.6=063
63.7
DA (Oain = 0.5) +
(Sup)
83.5 +0.19
75.7=1.56
65.7
w/ LA (Inf) (Menon et al., 2021)
83,2
70.4=290
62.4+1.24
w/ CReST+ (Wei et al., 2021)
84.2
78.1 18014
67.7+1.39
w/ CReST+ & LA (Inf) (Wei al., 2021)
85.6
81.2
71.94324
w/ UDAL (ours)
85.3
80.2.2059
68.611132
w/ UDAL & LA (Inf) (ours)
86.3
82. 12037
72.91121
Table I: We compare baselines CIFARIO-LT 3 10%, from the simplest approaches that involve no accommodations for class imbalance to stronger baselines which attempt to address it. We note that include two types of logit adjustment (Menon et al., 2021) (LA) here: one that involves modifying the loss to the supervised branch ("LA Sup") to account for distributional shift and another that does modify training and applies adjustment factor at inference time ("LA (Inf)").
4.1 CIFAR-LT
4.2 DATASET CREATION
CIFARIOLT and CIFARI00-LT (Cao al., 2019; Cui et al., 2019) modifications of CI- PARIO/CIFRR 100 with a long-tailed (Zipfian) distribution. Given a desired imbalance ratio y and some class ordering Ci 1 < < C, sample N, examples from the dataset for class C₁ accord-
ing to N; N1 T- For CIFAR10, N1 5000. C 10 while N₁ 500 and C = 100 for C0F In order to create suitable splits for semi-supervised learning, labeled subsets are randomly sampled according to 8 10% and 30%. Imbalance ratios of y E {50, 100, 200} are explored for CIFARIO-LE while y E (50, 100} for CIFAR100-LT, We evaluate with respect to the original, balanced test which ensures that
ordinary accuracy metrics.
4.3 TRAINING
We follow the same model guidelines in CReST (Wei et al., 2021). Therefore, we use Wide ResNet- 28 (with width 2) as a backbone. FixMatch (Sohn et al., 2020) and MixMatch (Berthelot et al., 2019) are used as base supervspe learners. With respect to FixMatch, the unlabeled ratio (7) and confidence threshold (0.95) are untouched from the original paper's ' settings. The same cosine learning rate schedule is adopted in Wei et al. (2021). Notably different from (Wei et al., 2021), however, is our ability to use vastly shorter training times. Generally, we find only 64 epochs (216 iterations batch size of 64) is necessary reach optimal performance, and therefore all CIFAR

---

Under review as a conference paper at ICLR 2022
models are trained for 64 epochs constituting 5x reduction in training time compared to CReST. Like CReST (Wei et al., 2021), ) we find it useful to train MixMatch models slightly longer and use a less aggressive Omin therefore, MixMatch models are trained for 128 epochs and use mmin 0.5.
Apart from those of the base semi-supervised learner, our method introduces only two hyperparam- eters: Omin which controls the final strength of the re- balannnning for Pa. and k, which controls the rate at which we approach O'min- We use Omnin = 0.10 and h = 2 which allows most of training to be aligned to Pdata and spend the very last stages of training aligning to a relatively balanced class distribution. This is supported by other work (Kang et al., 2020) which empirically finds that a when training long-tailed, fully-supervised models, the bulk of training should be done with respect to random sampling, and only a small amount of time of class-balances balanced sampling is necessary near the end of training. We provide an ablation of these hyperparameters in Section 4.5.2.
Training is carried out over random folds of the data. We report final test accuracy along with standard deviation using the exponential moving average of the model's parameters. We use the same underlying codebase, written in TensorFlow (Abadi et al., 2015), and data splits CReST as recommended (Oliver et al., 2018) to prevent framework or dataset dependent uncertainty in performance.
4.4 RESULTS
4.4.1 CIFAR
CIFARIO-LT and CIFAR100-LT results are summarized in Table 2. Across the board, we find
UDAL is competitive with CReST+ (Wei et al., 2021) and is only outperformed in a single setting. Notably, we find that CReST+ especially struggles in the more label rich regimes. In particular, despite a heavy imbalance of y E {100, 200). UDAL significantly outpeforms CReST+ when / = 30% for both CIFAR (up to a 4 point increase) and CIFAR100-LT (up to 1.7 point increase). We hypothesize that CReST+ is unable to efficiently use more labeled data since it only weakly addresses imbalance in the supervised branch. The imbalance in the supervised branch can only be indirectly alleviated by adding pseudo-labeled to the dataset under some amount of resampling- While this could have high impact when there is scarce labeled data to begin with, it appears to have much less of an effect when more labeles are available. UDAL, however, directly aligns the supervised branch to a more balanced distribution over the course of training. Not only would this improve the balanced performance of the classifier in the fully supervised setting, but additionally aids the feedback loop provide more balanced pseudo-labels in the unsupervised branch.
CIFARIO-LT
CIFARIOI-LT
3= 10%
= 30%
0 10%
3 30%
Method
50
7= 100)
7=200
7=30
7= 100
7= 200
70=50
7= 100
7=
7= 100
FixMatch (Sohn et al., 2020)
79.4
66.3.x
59.74034
81.9
73.1
64.7
33.7
43.1 28.3
38.6
w/ CReST+ (Wei et al., 2021)
84.2 x
67.7gin
84.9_am
79.2
70,5g4%
38.8
34,6g00
46.7g3
42.0amm
w/ (ouns)
85.3
80.2.as
68.61112
86.7
82.4
74.5
39.8
34.3
48.0
43.7
Table 2: Classification accuracy (%) over CIFAR10-LT and CIFAR100-LT under variety of label fractions and imbalance ratios y, each averaged over folds.
DARP We show results within the DARP (Kim et al., 2020) setting in Table 3. While DARP also produces an imbalanced dataset from CIFARIO, it has slight differences. Whereas the CReST (Wei et al., 2021) setting assigns every example from the original CIFAR dataset to either the labeled or unlabeled dataset, DARP uses an unlabeled labeled ratio of 2:1, which may or may not utilize all data points. One can consider the DARP setting as a B = 33% but with only 95% of the original dataset.
Nonetheless, UDAL continues to significantly outperforms CReST+ in all settings and shows the largest gains of all in the most imbalanced settings - capable of taking advantage of the increased amount of labeled data.
4.4.2 IMAGENET-127
We additionally provide experimental results on ImageNet-127 in Table 4. As presented in (Wei et al., 2021), ImageNet-127 is coarser version of ImageNet (Krizhevsky al., 2012), containing
7

---

Under review as a conference paper at ICLR 2022
Method
? y= 50
Y 100
Y = 150
FixMatch (Sohn al., 2020)
79. 79.2=0.3
71.5
68.4-ua5
w/ DARP (Kim et al., 2020)
81.8 =.28
2005
70.4_0.25
w/ CReST+ (Wei et al., 2021)
83.9 $3.9=our
77.4+0.3
72.8 -
w/ UDAL (ours)
86.5
81.4
77.9
Table 3: Classification accuracy (%) under DARP's protocol (Kim et al., 2020) for CIFAR10.
the same number of examples but with 127 class groupings rather than 1000. This grouping results in an imbalance ratio y 286. We conduct experiments with 10% along with all base training settings identical to CReST+, with exception to the duration of training, where we observe that only 1000 epochs (1 (1/6 the total training time of CReST+) necessary for convergence (Figure 1). For UDAL, we continue to find & 2 to be optimal, however, a less aggressive O'min E [0.5, 0.6) was found to be more effective in this setting. Overall, we find UDAL provides quite healthy increases in performance over CReST and is only 0.7 absolute points worse than fully supervised baseline with access to 100% of the labels.
Epochs
Method
1000
2000
4000
6000
Supervised (100% labels)
75.8
Supervised (10% labels)
46.0
FixMatch (10% labels)
64.9
65.8
DA (amin 0.5)
68.0
69.1
w/ CReST+
68.3
70.7
73.7
w/ UDAL (ours, Ogain = 0.5)
74.5
74.3
w/ UDAL (ours, Ogain 0.55)
75.1
74.8
UDAL (ours, Again 0.6)
73.2
75.3
Table 4: Evaluating the proposed method on ImageNet127 with imbalance factor y = 286 where 10% samples are labeled. Supervised models are trained for 300 epochs with 100% labeled data or equivalently 3000 epochs with 10% labeled data.
4.5
ABLATION STUDIES
We carry out ablation studies for our method using the CIFARIO-LT dataset with y = 100 and 10%. We plot all graphs with respect to the same scale so that they may be readily compared.
4.5.1 REPLACING CREST'S S SAMPLING PROCESS
We explore whether it would be sufficient to simply replace CReST's generational sampling process with the supervised component of our method. The essential difference between this and our method would be the usage of distribution alignment within the pseudo-labe label inference (i.e., this affects the pseudo- labeling process directly) versus enforcing a modified loss computation without any change to the pseudo-labe inference process. We consider variety of schedules in Figure 3a. While this approach performs well, and can even slightly outperforms CReST+, it still falls short UDAL, which uses the same approach for both supervised and unsupervised losses without the need to alter the pseudo-label inference process.
4.5.2 SCHEDULE PARAMETERS
Since UDAL uses form of progressive distribution alignment in similar way CReST, intro- duces two hyperparameters: l and Q which dictates the extent and rate at which the alignment approaches the uniform distribution. An ablation of these values on CIFAR10- is shown in Figure 3b which supports that a quadratic schedule and relatively low amin are optimal.
4.5.3 IS SCHEDULE NECESSARY?
We investigate whether schedule is necessary for good performance, This is equivalent to a "sched- ule" with the distribution alignment hyperparameter k 0. We include variety of settings for Omin
8

---

Under review as a conference paper at ICLR 2022
adding
61 spocte
1229 spects
250 epects
Alcurrayy
26
74
74
74
0.0
0.5
(a) Evaluating hybrid ap-
(h) Analysis of sensitivity to (c) Removing the progressive schedule hyperparameters. We nature of our alignment shows
proach which uses the same su-
pervised loss changes UDAL
plot the classification perfor- a degradation in performance
but with a form of progressive
mance of our method as func- tion of schedule parameters & and
of UDAL when Ckain is fixed for the entirety of training (i.e. k 0).
DA on the pseudo-labely rather than the unsupervised loss.
Omin-
Figure 3: A variety of ablations concerning the approach of our method.
and the duration of training in Figure 3c to ensure we adequately explore the trends in performance. We observe significant loss in performance compared to the progressive version of UDAL pre- sented in the paper. We attribute this to the fact that the supervised branch is immediately pushed produce marginal distribution of pseudo-labels that are more balanced. This, however, goes against empirical evidence (Kang et al., 2020) that representations are best learned from data sam- pled according the data distribution, even if it is imbalanced.
RELATED WORK
supervvper learning (Grandvalet al., 2005; Lee, 2013; Laine & Aila, 2017: Berthelot al., 2019; 2020; Sohn et al., 2020) has recently seen strong advances in performance. This can be attributed to the success of pseudo-labeling (Lee, 2013) combined with consistency of predictions (Berthelot al., 2019; Sohn et al., 2020) among varying types of augmentations of unlabeled data.
While supervised learning has progressed significantly, a large amount of work has tackled the setting of class imbalanced learning (Kang et al., 2020; Jamal et al., 2020; Cui et al., 2019; Menon et al., 2021; Ren et al., 2020; Hong et al., 2021; Tang et al., 2020; Khan et al., 2017). These range from modifications to the loss formulation (Jamal et al., 2020; Menon et al., 2021; Ren et al., 2020; Hong et al., 2021: Khan et al., 2017) to decoupling representation from the classifier (Kang et al., 2020) and even modifying the optimization process itself (Tang et al., 2020).
By combining the two previous settings, we consider imbalanced, semi-supervised learning. Al- though still relatively unexplored, previous attempts (Hyun et al., 2020; Kim et al., 2020; Wei al., 2021) to combat imbalance in the semi-supervised setting have been made. Both Hyun et al. (2020); Kim et al. (2020) modify changes to the loss formulation of a base semi-supervised learner com- bat bias within majority classes, while Wei et al. (2021) involves hybrid, generational approach that progressively aligns pscudo-labels predictions as well as augmenting the labeled set with rebal- anced, confident pseudo-labels.
6 CONCLUSION
We presented Unifying Distribution Alignment as a Loss (UDAL) which addresses the issue of class- imbalance within semi-supervised learning. By connecting the ideas of progressive distribution alignment to logit adjustment, we provide loss that can be applied to both the supervised and unsupervised branches rather than previous disjoint approaches. Furthermore, this approach incurs no additional training time on top of the underlying semi-supervised learner, achieves improved performance across multiple imbalanced settings and datasets, and scales to larger, more realistic datasets like ImageNet- - all while requiring only a few lines of code. Future work includes applying UDAL in other settings, e.g. object detection where both imbalance and missing labels are pervasive.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Our work is concerned with building more data-efficient classification models while reducing bias present in the distribution of data in the wild. While our work may be subject to similar claims of fairness any supervised or partially supervised model, we do not believe it introduces harm and, anything, attempts to mitigate effects that might be seen in biased datasets.
REPRODUCIBLITY
Our work is based off of an open source release CReST available here. The modifications we make self-contained and Pythonic code is provided within the paper (e.g., Algorithm 1). All hyperparameters we introduce are documented within the paper.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangging Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learn- ing on heterogencous systems, 2015. URL https: /www tensorflow org/ Software available from tensorflow.org.
Eric Arazo, Diego Ortego, Paul Albert, Noel O' O'Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In International Joint Conference on Neural Networks (IJCNN), 2020. 2
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. MixMatch: A holistic approach to semi-supervised learning. In NeurIPS, 2019. I, 4, 6, 9
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. ReMixMatch: Semi-supervised learning with distribution matching and augmenta- tion anchoring. In ICLR, 2020. 1, 2, 6,9
John Bridle, Anthony JR Heading, and David JC MacKay. Unsupervised classifiers, mutual information and' phantom targets'. In NIPS, 1992. 2
Mateusz Buda, Atsuto Maki, and Macicj A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249-259, 2018.
Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning?
In ICML, 2019. 1
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-disributiinnaaaaa margin loss. arXiv preprint arXiv: 190600749 2019. 6
Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Yin Cui, Menglin Jia. Tsung Yi Lin. Yang Song, and Serge Belongie. Class-balanced loss based on effecctve number of samples. In CVPR, 2019. 6, 9
Yves Grandvalet, Yoshua Bengio, et al. Semi-supervised learning by entropy minimization. CAP, 367:281-296, 2005.
Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering, ((9):1263-1288) 2009. I
10

---

Under review as a conference paper at ICLR 2022
Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang. Disentangling label distribution for
visual recognition. In CVPR, 2021. 1,2,
Minsung Hyun, Jisoo Jeong, and Nojun Kwak. Class-imbalanced semi-supervised learning. arXiv preprint arXiv:2002.068// 2020. 9
Muhammad Abdullah Jamal, Matthew Brown, Ming- Yang. Liqiang Wang, and Boqing Gong- Rethinking class-balanced methods for long-tailed visual recognition from domain adaptation perspective. CVPR, 2020.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In ICLR, 2020. 1, 2, 7,9
Salman H Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous A Sohel, and Roberto Togneri. Cost-sensitive learning of deep feature representations from imbalanced data. IEEE transactions on neural networks and learning systems, 29(8):3573-3587 2017. 9
Jachyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin. Dis- tribution aligning refinery of pseudo-label for imbalanced semi-supervised learning. NeurIPS, 2020. 4, 5, 7, 8,
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. NIPS, 2012.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In ICLR, 2017.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. ICML Workshop, 2013.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In ICLR, 2021. 1, 3, 6,
Avital Oliver, Augustus Odena, Colin Raffel, Ekin D Cubuk, and Ian J Goodfellow. Realistic evalu- ation of deep semi-supervised learning algorithms. In NeurlPS, 2018.
Jiawei Ren, Cunjun Yu, Shunan Sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Bal- anced meta-softmax for long-tailed visual recognition. In NeurIPS, 2020. 1, 1,
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. FixMatch: Simplifying semi-supervised learning with consistency and confidence. In NeurIPS, 2020. 1, 4, 6, 8,
Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. In NeurIPS. 2020. 9
Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang. Crest: A class-rebalancing self-training framework for imbalanced semi-supervised learning. In CVPR, 2021. 2, 3, 4, 5, 6, 7, 8, 9

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 ALIGNMENT OF PSEUDO-LABELS OVER THE COURSE OF TRAINING
We examine the KL-divergence of the pseudo-labels (as moving average) over the course of CIFARIO-LT training using UDAL.
20k
50k
12


Reviews:
[{"review_id": "uL8DljZw5Y", "reviewer": "Reviewer_WjMc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros.**\n\n- **Clarity**. Overall, the writing is clear and easy to follow.\n- **Important problem with a simple and effective solution**. Considering the class imbalance scenario is an essential step for applying SSL in a more realistic scenario but has been less explored. The proposed method can be used with a simple modification of the existing baseline and it shows significant empirical gain; hence, it has the potential to be widely used to mitigate this problem without additional burdens. Although the proposed method can be viewed as a simple combination of the existing methods, I believe that the simplicity and effectiveness of the proposed method will be interesting to the reader in ICLR.\n\n**Cons.**\n\n- **More general imbalanced semi-supervised learning scenarios.** Although I agree that a considered scenario (i.e., distributions of both labeled and unlabeled are same) is most natural, I wonder that the proposed method can be applicable to more generic scenarios (i.e.,  distributions of both labeled and unlabeled are different). Is there any way to apply the proposed UDAL to such a challenging scenario?\n- **Effect of the strategy in (Kang et al., 2020).** In Section 4.3., the authors present that they use a deferred re-sampling strategy which is introduced by Kang et al., 2020 (\"... spend the very last stages of training aligning to a relatively balanced class distribution..\") Can the authors give the details about this? Also, is this necessary for empirical improvement?\n\nKang et al., Decoupling representation and classifier for long-tailed recognition., In *ICLR*, 2020", "summary_of_the_paper": "- This paper tackles the class-imbalanced problem with a semi-supervised learning scenario. Unlike the previous approaches, which require a complicated sampling strategy and multiple training pipelines, the authors provide a simple and unified framework, UDAL, by connecting the ideas of progressive distribution alignment (proposed for imbalanced semi-sup) to logits adjustment (proposed for imbalanced sup). This approach incurs no additional training time on top of the underlying semi-supervised learner. Significant empirical improvement on widely used benchmarks (CIFAR-10-LT, CIFAR-100-LT, and ImageNet-127) demonstrates the effectiveness of the proposed method.", "summary_of_the_review": "Although the proposed UDAL can be viewed as a simple combination of the existing methods (progressive distribution alignment and logits adjustment), I believe that the simplicity and empirical effectiveness of the proposed method will be interesting to the reader in ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fHHLWCQSaP", "reviewer": "Reviewer_inVm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## strengths\n1. This paper studies an important problem which is underexplored in literature;\n2. The proposed method is simple yet effective, which shows substantial performance gains on multiple datasets;\n3. This paper is well written and easy to understand;\n4. The proposed method is reasonable that aligns the distribution of both supervised and unsupervised losses.\n\n## weaknesses\n1. The novelty is limited. As mentioned, the proposed method is pretty much based on existing works, i.e., DA and LA, which are used and demonstrated to be helpful in class-imbalanced tasks. Importantly, the improvements of the methodologies are limited. For example, it is very natural to extend LA to unsupervised loss by using pseudo-labels. In this regard, this paper does not examine the quality of the generated pseudo-labels. Does the distribution of pseudo-label make sense? For the DA in Eq. (10), it immediately follows previous work, with a minor difference by introducing an additional parameter $k$;\n2. More baselines should be compared. In the current version, this paper mainly compares its proposal with CReST+. I am aware that CReST+ is a strong baseline, but it would be more convincing if more baselines could be compared, such as [1,2,3];\n3. Hyperparameters. From Figure 3, it is observed that the performance of the model is quite sensitive to the choice of $\\alpha_{min}$;\n4. In Table 4, the results for Supervised models are not informative. To better quantify the gap, LA or other techniques that handles class imbalance, should be also applied.\n\n\n[1] DISTRIBUTION-AWARE SEMANTICS-ORIENTED PSEUDO-LABEL FOR IMBALANCED SEMI-SUPERVISED LEARNING.\n\n[2] ABC: Auxiliary Balanced Classifier for Class-imbalanced Semi-supervised Learning. NeurIPS 2021.\n\n[3] Unsupervised Semantic Aggregation and Deformable Template Matching for Semi-Supervised Learning. NeurIPS 2020.", "summary_of_the_paper": "This paper studies class-imbalanced semi-supervised learning. To handle this problem, a unified approach is proposed by combing distribution alignment (DA) and logits adjustment (LA). In particular, this paper proposes to apply DA and LA to both supervised and unsupervised losses, which is new. This method shows significant improvement over baselines on three datasets. ", "summary_of_the_review": "My main concern is that the novelty is limited, which makes this paper like an incremental work. Therefore, I recommend rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "C7rxen70t_l", "reviewer": "Reviewer_aN2N", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ This paper addresses the topic of semi-supervised learning in cases where the underlying data distribution is severely imbalanced. This is an underexplored problem that is relevant to the AI/ML community and has practical real-world impact.\n+ In contrast to other papers addressing this problem, the proposed approach relies only on diistribution alignment without sampling-based strategies. This is done by moving the distribution into the cross-entropy loss computations of recent semi-supervised learning algorithms: FixMatch and MixMatch.\n+ This work connects ideas of progressive distribution alignment with logit adjustment from the fully supervised, imbalanced learning setting.\n+ By considering distribution alignment alone, training time is reduced (5x reduction from CReST).\n+ The method achieves significantly better performance characteristics as more labeled data becomes available, and the approach scales to larger datasets, outperforming the best existing method on ImageNet-127 (in terms of accuracy).\n+ The proposed method is well-framed in the existing literature.\n+ Section 2 provides a nice overview of the prerequisities needed to understand the method.\n+ Eqs. 8 and 9 are interesting insights.\n+ The approach is simple, requiring only a few additional lines of code to implement with no alteration to the training scheme and no increased training time.\n+ The approach only has two hyperparameters which need to be tuned.\n+ Experiments are conducted over three benchmark vision datasets of varying complexity: long-tailed versions of CIFAR10, CIFAR100, and ImageNet-127.\n+ Experimental setup seems reasonable, and follows a similar procedure to CReST.\n+ Experimental results are competitive or exceeds other methods on the tested datasets.\n+ Ablations are run to validate each component of the proposed approach.\n\nWeaknesses:\n- Imbalanced semi-supervised learning, while an important problem, is not a new problem, somewhat limiting novelty.\n- The proposed approach is a combination of two existing approaches, somewhat limiting novelty.\n- Experiments are conducted on only a single network architecture. It would be useful to show it works on a wide range of neural network architectures.\n- It would be useful to see if the results are statistically significant.\n\nQuestions:\n- What happens if the assumption that the unlabeled set of examples is imbalanced in a different way than the training examples?\n- Please define \"strong\" and \"weakly\" augmented versions of the unlabeled example.\n- Are there any significant drawbacks to this approach compared to others?\n- What happens in the balanced setting; does it fall back to a standard semi-supervised learning approach?", "summary_of_the_paper": "This paper addresses the topic of semi-supervised learning in cases where the underlying data distribution is severely imbalanced. This approach combines distribution alignment with logit adjustment, resulting in an efficient method for solving the aforementioned problem while improving performance in the test setting. Unlike existing state-of-the-art approaches such as CReST, the approach involves no sampling state, and imbalance mitigation is achieved just by modifying the loss functions of the model. Experiments are conducted over three benchmark vision datasets of varying complexity: long-tailed versions of CIFAR10, CIFAR100, and ImageNet-127. Experimental results are competitive with or exceed other methods on the tested datasets with 5x training speedup compared to CReST.", "summary_of_the_review": "This paper addresses the topic of semi-supervised learning in cases where the underlying data distribution is severely imbalanced. This is an underexplored problem that is relevant to the AI/ML community and has practical real-world impact. In contrast to other papers addressing this problem, the proposed approach relies only on diistribution alignment without sampling-based strategies. This is done by moving the distribution into the cross-entropy loss computations of recent semi-supervised learning algorithms: FixMatch and MixMatch. The paper is generally well-structured and clearly written. The experimental set up is sound, and experimental results show the promise of the model. The approach is well-grounded in the existing literature, and its motivation is clear. As it stands, I do not see any major flaws with the approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "02NFYMQewcw", "reviewer": "Reviewer_qKfL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper focuses on class-imbalanced semi-supervised learning and proposes a new method that combines distribution alignment and logit adjustment techniques. The proposal is simple. However, the proposed method relies on the assumption that the imbalance distribution between labeled and unlabeled datasets is the same. This is a very strong prior knowledge and in reality, we can not obtain this knowledge.  Thus, the application of the proposal is limited. Moreover, the new proposal is mainly based on existing techniques, although combining these techniques can achieve a performance improvement, the novelty and contribution are limited.", "summary_of_the_paper": "This paper focuses on class-imbalanced semi-supervised learning and proposes a new method that combines distribution alignment and logit adjustment techniques. Experiments show that the proposal can achieve performance improvement.", "summary_of_the_review": "This paper proposes a new method for class-imbalanced semi-supervised learning. However, in my view, the assumption of the proposal is too strong to satisfy and the novelty of the proposal is limited.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks for your submission to ICLR.

Reviews were fairly mixed on this paper, with two reviewers advocating for accepting the paper and two advocating for rejecting the paper.  There were some concerns raised by the reviewers, most notably novelty and some issues with the experiments.  After rebuttal, the negative reviewers maintained their scores and the positive reviewers were somewhat less enthusiastic.  In the end, the paper is quite borderline and could really go either way, but it seems that the paper could use another round of reviewing, particularly to make sure the issues raised by the reviewers are adequately addressed.

Please do keep in mind their comments when preparing a future version of the manuscript.
Decision: Reject