Semi-supervised Long-tailed Recognition using Alternate Sampling
Authors: Bo Liu
Haoxiang Li
Hao Kang
Nuno Vasconcelos
Gang Hua
Venue: ICLR 2022 Submitted (2022)

Abstract: Main challenges in long-tailed recognition come from the imbalanced data distribution and sample scarcity in its tail classes. While techniques have been proposed to achieve a more balanced training loss and to improve tail classes data variations with synthesized samples, we resort to leverage readily available unlabeled data to boost recognition accuracy. The idea leads to a new recognition setting, namely semi-supervised long-tailed recognition. We argue this setting better resembles the real-world data collection and annotation process and hence can help close the gap to real-world scenarios. To address the semi-supervised long-tailed recognition problem, we present an alternate sampling framework combining the intuitions from successful methods in these two research areas. The classifier and feature embedding are learned separately and updated iteratively. The class-balanced sampling strategy has been implemented to train the classifier in a way not affected by the pseudo labels' quality on the unlabeled data. A consistency loss has been introduced to limit the impact from unlabeled data while leveraging them to update the feature embedding. We demonstrate significant accuracy improvements over other competitive methods on two datasets.

Under review as a conference paper at ICLR 2022
SEMI-SUPERVISED LONG-TAILED RECOGNITION US- ING ALTERNATE SAMPLING
Anonymous authors Paper under double-blind review
ABSTRACT
Main challenges in long-tailed recognition come from the imbalanced data distri- bution and sample scarcity in its tail classes. While techniques have been proposed achieve a more balanced training loss and improve tail classes data varia- tions with synthesized samples, we resort to leverage readily available unlabeled data boost recognition accuracy. The idea leads new recognition setting. namely semi semi-supervised long-tailed recognition. We argue this setting better re- sembles the real-world data collection and annotation process and hence can help close the gap to real-world scenarios. To address the semi-supervised long-tailed recognition problem, we present an alternate sampling framework combining the intuitions from successful methods in these two research areas. The classifier and feature embedding are learned separately and updated iteratively. The class- balanced sampling strategy has been implemented train the classifier in way not affected by the pseudo labels' quality on the unlabeled data. A consistency loss has been introduced limit the impact from unlabeled data while leverag- ing them to update the feature embedding. We demonstrate significant accuracy improvements over other competitive methods on two datasets.
INTRODUCTION
Large-scale datasets, which contain sufficient data in each class, has been major factor to the success of modern deep learning models for computer vision tasks, such as object recognition. These datasets are usually carefully curated and balanced to have uniform data distribution over all classes. This balanced data distribution favors model training but could be impractical in many real world applications, where the frequency of samples from different classes can be imbalanced, leading to a long-tailed data distribution. As shown in Figure [bb,, several highly populated classes take up most of the labeled samples, and some of the classes only have very few samples during training.
The long-taile tailed recognition problem has been widely studied in the literature. One major challenge in this setting (Liu et al.] (2019): Kang et al.] (2020); Zhou et al. (2020)) to deep learning model training is the tendency of under-fitting in less-populated classes. The root causes of this under- fitting are the imbalanced training data distribution as well as the scarcity of data samples in the tail classes.
More specifically, with an imbalanced training data distribution, when several head classes take up most of the training samples, tail classes contribute little in the training loss. The model is such that biased towards head classes. Prior works (Lin et al. (2017); Cao et al. (2019): Kang et al. (2020); Zhou et al. (2020)) tried to mitigate the issue by re-sampling the training data to be balanced dis- tribution or calibrating the sample weights in calculating the loss. However, still the scarcity of tail class data samples limits the intra-class variations and overall recognition accuracy. Methods focus- ing on few-shot learning have been introduced to address this problem through data augmentation and data synthesis (Wang et al. (2018); Hariharan & Girshick (20h))c)())1)).....) Liu et al. (2020)).
In this work, resort to different path to leverage massive unlabeled real data in training to help improve the tailed recognition accuracy. Since data collection is much cheaper and accessible comparing to data annotation, additional unlabeled real data could readily be available in many real- world scenarios. This semi- cemi-supervised learning setting has been intensively studied in the literature
[Laine & Aila (2016): |Rasmus et al. (2015). Tarvainen & Valpola (2017); Berthelot al. 2019);

---

Under review as a conference paper at ICLR 2022
(a) Semi-supervised Recognition
(b) Long-tailed Recognition
of samplees
of samples
Classes
Head Class
Class
(c) Semi-supervised Long-tailed Recognition
# of samples class
Supervised
Unsupervised
Head Class
Tail Class
Figure 1: Comparison of different recognition paradigms: a) statistics of CIFAR-10 10 when used as a Semi-supervised Recognition benchmark; b) typical data distribution over classes in long-tailed recognition; c) the proposed Semi-super long-tail recognition setting, in which both labeled and unlabeled subsets follow the same underlying long-tailed data distribution.
Sohn et al. (2020)). However, as shown in Figure I [a), when we carefully look at the data distribution of the widely used benchmarks, observe well-balanced labeled subset and unlabeled subset. As discussed above, the manually curated balanced distribution, can lead to gap to real-world scenarios. This is especially true in unlabeled data. Without labels, people have no way to balance the data among classes.
In this paper, we propose a more realistic and challenging setting, namely semi-supervised long- tailed recognition. As shown in Figure[[cc,, we assume long-tailed data distribution of the overall dataset and both the labeled and unlabeled subsets of training data follow the same underlying long- tailed data distribution. This setting generally resembles realistic data collection and annotation workflow. After collecting the raw data, one has no knowledge of its class distribution before an- notation. As it is expensive to annotate the full corpus, common practice is to randomly sample subset for annotation under given labeling budget. When the raw data follows a long-tailed class distribution, we should expect the same in the labeled subset.
While this new recognition paradigm shares the challenges in both semi-supervised learning and long-tailed recognition, there is no readily naive solution to it. Methods in long-tailed recognition rely on class labels to achieve balanced training, which are not available in the unlabeled portion in the semi-supervised long-tailed recognition. Prior semi-supervised methods without considering the long-tailed distribution could fail as well.
Taking one of the competitive baseline methods for example, (Yang Xu (2000) proposed to firstly train a recognition model with the labeled subset to generate pseudo labels for the unlabeled subset, then the model fine-tuned with the full training dataset. However, when the labeled subset follows a long-tailed distribution, the pseudo labels are much less accurate for tail classes than head classes. As result, the overall pseudo labels quality could be too bad leverage (See Section 4.4] results in LIFAR-IO-SSE and ImageNet-SSLT).
To address the semi-supervised long-tailed recognition problem, we present a method designed specifically for this setting, We bring the successful class-balanced sampling strategy and com- bined it with model decoupling in an alternate learning framework to overcome the difficulty of balancing unlabeled training data.
Inspired by (Kang et al. (2020)), we decouple the recognition model into a feature embedding and classifier, and train them with random sampling and class-balanced sampling respectively. As
2

---

Under review as a conference paper at ICLR 2022
targeting semi-supervised setting, the classifier is only trained on labeled data to get around the difficulty of applying correctly class- balanca sampling on unlabeled data, aligning with the intuition that the classifier needs more robust supervision than the feature embedding.
After that, with the proposed alternative learning framework, we improve model by updating the feature embedding and the classifier iteratively. We assign pseudo labels with the up-to-date classi- fier and observed gradually improved accuracy of pseudo labels over iterations. The pseudo labels are then incorporated in fine-tuning the feature embedding with regularization term to limit its potential negative impacts. Similar iterative design has been proposed in semi-supervised learning literature (Laine Aila (2016): Tarvainen & Valpola (017)) but important implementation details differ.
To summarize, in this paper. 1) we resort to semi-supervised learning to help improve long-taile recognition accuracy and identify practical gap of current semi-supervised recognition datasets due to their well-balanced unlabeled subset; 2) we propose new recognition paradigm named semi- supervised long-tailed recognition better resembling real-world data collection and annotation work- flow; 3) we propose a new alternative sampling method to address the semi-supervised long-tailed recognition and demonstrate significant improvements on several benchmarks.
RELATED WORK
Long-tailed recognition has been recently studied lot (Wang al. (217); Oh Song et al. (016));
Lin et al. (2017); Zhang et (017)); Liu et al. )2; Wang & Hebert (2016)) Several approaches have been proposed, including metric learning (Oh Song et al. (2016) Zhang et al. (2017). loss weighting (Lin et al. (017)) ) and meta-learning (Wang & Hebert (2016)) Some methods design dedicated loss functions to mitigate the data imbalanced problem. For example, lift loss (Oh Song et al. (2166) introduces margins between many training samples. Range loss (Zhang et al. (2017) encourages data from the same class to be close and different classes to be far away the embedding space. The focal loss (Lin et al. (2017)) dynamically balances weights of positive, hard negative, and easy negative samples. As reported by (Liu et al. (2019)). when applied to long-tailed recognition, many of these methods improved accuracy of the few-shot group, but at the cost of lower accuracy over the many-shot classes.
Other methods, e.g. LDAM-DRW (Cao al. (2019) replace cross-entropy loss with LDAM loss. This adds a calibration factor to the original cross-entropy loss. When combined with loss re- weighting, improves the accuracy in all splits in long-tailed recognition. However, can be easily generalized to semi-supervised learning. Because both the calibration factor and the loss weight are calculated based the number of samples of each class.
In face recognition and person re-identification, the datasets are mostly with long-tailed distribu- tion. LEAP (Liu et al. (2020)) augmented data samples from tail (few-shot) classes by transferring intra-class variations from head (many-shot) classes. Instead of data augmentation, we introduce unsupervised data improve the performance of long-tailes recognition.
A recent work (Ynng& & Xu (2020)) rethinks the value of labels in imbalance learning. As part of the discussion, semi-supervised learning included. However, only the basic pseudo label solution and simple datasets, such as CIFAR and SVHN, are discussed.
More recent works (Kang et al. 2020); : Zhou et al. (2020) with improved long-tailed recogni- tion share the observation that feature embedding and the classifier should be trained with different sampling strategies. this work, we adopt our method on this observation to learn the feature em- bedding model with random sampling and train the classifier with dass-balanced sampling. This design is further closely compatible with semi sumi-supervieed learning under alternate learning.
Semi-supervised learning has been extensively discussed in recognition discipline (Laine & Aila (2016); Rasmus et al. (2015); Tarvainen & Valpola (2017)). One common observation is to optimize the traditional cross-entrope loss together with regularization term that regulates the perturbation consistency of unlabelled data.
Ladder net (Rasmus al. (2055) is introduced to minimise the reconstruction loss between the network outputs from a given sample and its perturbation. It is then simplified in (Laine Aila 2016)) two temporal modules: TT-Model and Temporal Ensembling. The Temporal Ensembling
3

---

Under review as a conference paper at ICLR 2022
Stage
Stage
Stage 3
Random Sampling
Clos-Balanced Samping
DOU'
Lge
Asup
(a) Initialization procedure. A recogni- tion model is first trained with random sampling. After that the feature em- classi-
(b) Diagram alternate learning. CNN modules green line only used in forwarding. Those in red are fine-tuned with the corre- sponding Joss. In Stage 1, samples from U are forwarded through f and U' consists of samples from U. and pseudo labels acquired from g. In Stage 2. and g are trained on the combination of and 4' Stage 3, only the classifier g is trained. f is fixed only used in forwarding.
bedding is used to train fier with (lass-balances sampling. the diagram, CNN components that are up- dated during training are highlighted in red.
encourages the output of the network from unlabeled data to be similar to its counterpart from previous training epoch. More recently, Mean Teacher (Tarvainen & Valpola (2017) extends it by assembling along training. Instead of storing previous predictions, they assemble a Teacher model by calculating the moving average of the training network, i.e. the Student. The Teacher then used to provide the consistency of predictions to the Student.
In addition to that, MA-DNN (Chen et al. 2018 introduces a memory module maintain the category prototypes and provide regularization for learning with unlabeled data. Label propagation aL. (2018) is also considered with the help of label graph. More recently, Mixmatch (Berthelot et al. (2019) and Fixmatch (Sohn et al. (2020)) improve the performance by introducing powerful data augmentations and perturbation consistencies.
All the semi semi-supervised methods above do not separate labeled data during semi-supervised training. In fact, it is beneficial to combine labeled data and unlabeled data certain proportion (Laine & Aila (2016); Tarvainen & Valpola (2017)). However, without further knowledge, we have no insight how to deal with this combination when long-tailed distribution is included. Furthermore, long-tailed learning methods require calibration or re-sampling based on the class distribution. This combination of labeled and unlabeled data makes the distribution unstable. In result, this is not suitable for long-tailed recognition.
Recently, Salsa (Rebuffi et al. (2020)) proposes to decouple the supervised learning from semi- supervised training. Our method follows the alternate training scheme from it. because it is sur- prisingly compatible with long-tailed learning. In practice, our method differs from Salsa in the following aspects.
sampling in supervised learning to deal with the long-tailed distribu- tion. Second, we use supervised learning instead of self-supervised learning as initialization. We find that self-supervised learning results in inferior performance in long-tailed scenario. Third, the re-initialization is not needed. Because our initialization already from supervised learning, there is not specific starting point to re- re-initialize the model. In fact, this enhances the soft constraint between the two stages in (Rebuffi et al. (2020))
First, we adopt
With the models continuously optimized along alternate learning, our method achieves superior performance while maintains the same amount of training epochs as fine-tuning on pseudo labels.
METHOD
In this section, we will introduce the proposed method to semi-supervised long-tailed recognition. The semi-supervised long-tailed recognition problem is first defined, and some notations are clari- fied. The decoupling strategy of long-tailed recognition then discussed. This also the initializa- tion phase of our method. After that, the alternate learning scheme with 3 stages is fully discussed.
3.1 SEMI-SUPERVISED LONG-TAILED RECOGNITION
We start by defining the semi- cemi-supervised Long-tailed recognition problem. Consider an image recog- nition problem with labeled training set D {(x;, V 1,..., N), where x; is an example
4

---

Under review as a conference paper at ICLR 2022
and ys {1....,CC C} its label, where C' is the number of classes. For semi-supervised learning, there is also an unsupervised training subset U/ = {x i 1....,M,
Although the labels of data in U are not available, every sample has its label from {L...., C). For class j. we have n samples from D and m, samples from 2. With the assumption that supervised and unsupervised data follow the same distribution, have the fact NN Mr Vj.
The testing set, on the other hand, in order to evaluate the performance on every class without bias, is balanced sampled on all classes in f1....,C) C).
3.2 MODEL DECOUPLING AND DATA SAMPLING
A CNN model combines feature embedding = f(x; E Rd, and classifier g E [0, 1Jc. Embedding f [x; 0) implemented by several convolutional layers of parameters 0. The classifier operates the embedding to produce class prediction
arg max; (z). In this work, we adopt the popular linear classifier g(z) = v (Wx + b), where is the softmax function.
Standard (random sampling) training of the CNN lies mini-batch SGD, where each batch is randomly sampled from training data. A class j of ny training examples has probability * of being represented in the batch. Without loss of generality, cardinality, i.e. n4 I ng. Vi > j. In the long-tailed setting, where n1 > nc: the model not fully trained on classes of large index j (tail classes) and under-fits. This can be avoided with recourse to non-uniform sampling strategies, the most popular of which is class-balances sampling. This samples each class with probability t. over-sampling tail classes.
assume classes sorted by decreasing
Kang et al. (020):: Zhou et al. 2020) shows that while classifier benefits from class-balard sam- pling, feature embedding is more robust in random sampling. Practically, Kang et al. al.] (2020) achieves this by decoupling the training into two stages, and train the feature embedding with random sam- pling in the first stage, and classifier the second with dlss-balaaceed sampling.
3.3 INITIALIZATION
The initialization of the proposed method follows the decoupling from (Kang et al. (2220). The
two-stage initialization is illustrated in Figure 2(a) A CNN model is first trained with random sampling. A feature embedding z f(x:; E R. and classifier g'(z) [0, 1] are acquired. After convergence, the classifier is re-initialized and trained with blassebalanced sampling, with the feature embedding fixed. This results in balancal classifier g(z) E [0, 1]c. Both the feature embedding and the classifier are trained on the supervised training subset D.
3.4 ALTERNATE LEARNING
After obtaining an initialized model, most semi-supervised learning methods fine-tune the model on combination of supervised and unsupervised samples. This is, however, incompatible with our long-tailed recognition model. When applied on unsupervised data, we have no ground truth for balanea sampling. One can make sacrifice by relying on pseudo labels assigned by the initialized model. But the effectiveness will depend on the accuracy of pseudo labels.
is even worse when considering the fact that long-tailed models usually have better performance on highly populated classes and worse on few-shot classes. Class-balanced sampling over- samples few-shot classes, while down-samples mhny-shot. This means, in general, the worse part of pseudo labels contributes more the training loss than it should be, while the better part contributes less.
Another difficulty is the model compatibility when combining the long-tailed model to semi- supervised learning methods. Many semi-supervised learning methods evolve the model and pscudo labels at the same time. For example, Mean Teacher (Tarvainen & Valpolal (2017)) assembles the teacher model by moving average and trains the student with consistency loss. When it comes to long-tailed model, it is not clear when we should update the feature embedding or classifier. And is also difficult incorporate both random and class-balane balanced sampling.
Inspired by (Rebutfi et al. 20200), ) which separates supervised learning apart from semi-supervised learning, we propose alternae learning scheme. The supervised training on data D, and semi-
5

---

Under review as a conference paper at ICLR 2022
supervised training data DUU are carried out in an alternate fashion together with model decou- pling and different data sampling strategies.
In practice, after initialization,
have feature embedding fx; f), classifier g'(z) trained with random sampling, and a classifier g(z) trained with blass-balanced sampling. In (Kang et al. 22200), only g(2) is used in testing. However, we keep the randomly trained classifier g (z) for turther usage. The training scheme iterates among stages for N loops, which are shown in Fig- ure 2(b)
Stage I: Label assignment. In this stage, pseudo labels are assigned for the unsupervised subset U. The feature embedding fxx:; 0) and class-balanced classifier g(z) used. The choice of classifier is equivalent to the long-tailed model when tested for better overall accuracy. The unsupervised subset with pseudo labels is 27 {(x,, 8:); ........... M). where y; are pseudo labels.
Stage 2: Semi-supervised training. After label assignment, we have pseudo labels for all unsu- pervised data. The model is the fine-tuned on the combination of true and pseudo labels, i.c. on D U.. In this stage, random sampling is used to update the feature embedding f(x; 0) and the randomly-trained classifier g' g()) The classification is optimized by cross-entropy loss:
LCE
Σ
-log d'g (f(x;)))
(X..UEEDUL
where oy, is the y--th element of
In semi-supervised learning literature, regularization loss is usually applied to maintain the con- sisteney for unlabeled data. This consistency loss captures the fact that data points in the neighbor- hood usually share the same label. We adopt this idea and implement the temporal consistency from (Laine& & Aila (016)). In practice, the class probabilitics are acquired from g'. Given the class prob- ability p--1 from epoch 1, and the class probability pf from epoch the loss is KL-divergence between the two.
Coonsist
Σ
ΣΦj
P
log
Pj
Pf
(EDU
where p and pf the j-th element of p--1 and p" respectively. Overall, the semi-supervised learning loss is the combination of the two.
Loemi LCE + ACcomeist.
(3)
Stage 3: Supervised training. We update the class-balanced classifier g(z) based on the refined feature embedding, which is fine-tuned with semi-supervised learning Stage 2. Specifically, the fine-tuning is applied with class-balanced sampling and only on the supervised subset D. In this stage, only classifier is updated. The feature embedding is fixed and only used forwarding. Given the class-balanced version of supervised subset D', the cross-entropy loss for classification is
Cxup
Σ
- log 9v. (f(;;; 0)),
(4)
(K.D)E
where Jy, is the yt element of g.
3.5 INSIGHT OF THE DESIGN
Feature embedding is trained with random sampling and semi-supervised learning. This is consis- tent with long-tailed model the sampling scheme. It also follows the fact that feature embedding is less prone to noisy labels. Actually, in self-supervised learning literature (Gidaris et al. (2018); He et al. (2020): Chen et al. 22020), the feature embedding can even be learned without labels.
Classifier is learned with class-balance sampling and only supervised data. This is again the same as the supervised version. And by avoiding fitting the classifier on pseudo labels, we prevent the wrong labels from propagating through the whole training process. Given the fact that the pseudo labels are provided by the classifier, if classifier is still optimize on those, wrong labels can be easily maintained in the fine-tuned version of the classifier.

---

Under review as a conference paper at ICLR 2022
Table 1: Left: Results(Accuracy in %) on CIFAR-IO-SSITT. ResNet-18 18 is used for all methods;
Right: Results(Accuracy in %)) on ImageNet-SSLT. ResNet- 18/50 are used for all methods.
Imbulance facior=]00| (00
ResNet-1885 18650
Method
Many-Shot Medium--Stot
Few-Shox
Overal
Many-Stut
Medium-Shet
Few-Shoi
LDAM-DRW
29.7/70.3
54.2/36.3
68.1/356
21.3/349
42,6/51.2
27.0/31.1
8.69999
Pacudo-Label
55.1/39.3
80.2/36.0
17.623.9
22,4/42.0
Mean Teacher L.
09.9/18.5
69.7755.2
57.3/41.4
79/4/329
21.5/2..
41.8/99.1
28.1/318
7.6/11.7
Desoupling
64.044588
91.1/86.5
63.047.22
44.4/14.4
24.8/27.2
53.958855
31,1/34.2
8.79.88
Pscudo-Label
08.9146.5
92.77290
70.8/47.0
49.814.2
25.322277
47.6/52.2
32.1/34.7
11.1/124
Ours
71 3/66.7
89. 19.5/44.4
67.7/69.4
60.2/51.4
26.5/79.0
52,0/57.1
33,9/36.5
10.7/12.3
Training the classifier only on labeled data also avoids the dilemma of class-balaneing on unlabeled data. Without ground truth labels, balanced sampling can only rely on pseudo labels, which are not perfect. And the fact that pseudo labels have more errors on few-shot classes is specially not suitable for class- balancea sampling. Because when few-shot classes are over-sampled those errors are also scaled up during training.
4 EXPERIMENTS
4.1 DATASETS
We manually curate two semi-supervised long-tailed recognition benchmarks.
CIFAR-1O-SSLT. For easy comparison and ablation, we compose a lightweight semi-supprrised long-tailed dataset based CIFAR-10 10 (Krizhevsky al. (2009).. Following (Cao et al. (2019)), randomly sample the training set of CIFAR-10 10 under an exponential function with imbalance ratios in {100, 1000} (the ratio of most populated class least populated). The unsupervised subset is collected from Tiny Images (oorrllaa et al. (2008) following the strategy introduced in (Yang & Xu (2020)) The class distribution of unlabeled data is always the same as the labeled one, with 5 times larger. For better description and comparison, we assign the 10 classes into 3 splits: many-shot, medium-shot, few-shot, with many-shot the most populated classes, medium the medium 3, and few-shot the least 4 classes.
ImageNet-SSLT. To evaluate the effectiveness of semi-supervised long-tailed recognition methods on large-scale datasets, assemble challenging dataset from ImageNet (ILSVRC-2012) (Deng) ctt al. (2009)). The supervised subset is sampled with Lomax distribution with shape parameter a 6, scale parameter À = 1000. It contains 41, 134 images from 1000 classes, with the maximum of 250 images per class and the minimum of 2 samples. The unsupervised subset is sampled under the same distribution with an unsupervised factor i.e. [14| = A/D]. The 1000 classes are divided into splits based on the amount of labeled data n: many-shot (n > 100), medium-shot (10 < n S 100), few-shot (n < 10). result, the dataset has 140 many-shot, 433 medium-shie and 427 few-shot classes. Methods are evaluated under all classes and each class split.
4.2 NETWORK ARCHITECTURE
ResNet-18 18 (He et al. (0166) is used on both CIFAR-10-SSLE and ImageNet-SSLL for fast exper- iments and comparison. ResNet-50 is used on ImageNet-SSL.T to show how methods scale up to larger networks.
4.3 COMPARISON METHODS
To our best knowledge, there is no available method designated for semi-supervised long-tailed recognition. We explore typical long-tailed recognition methods and sumi-supervised recognition methods, and combine them as baselines.
Long-tailed Recognition. We consider two long-tailed methods, one for loss calibration and the other for samping. LDAM-DRW (Cao et al. (2019) converts cross-entropy loss to LDAM loss with calibration factors based class counts. It further regulates the loss with loss weight also from class counts. Decoupling (Kang et al. (2020) decouples the training of embedding and classi- fier with different sampling strategies. This is also the initialization in our method.
7

---

Under review as a conference paper at ICLR 2022
Semi-supervised Recognition. Pseudo-Label is basic semi-supervised learning algorithm and can be easily combined with other models. It contains two phases. The first phase is initialization, the recognition model is trained on labeled data. Predictions of the initialized model assigned on unlabeled data, i.e. pseudo labels. The initialized model is then trained or fine-tuned on the combination of labeled and unlabeled data. In practice, combine Pseudo-Labe Label method with the two long-tailed recognition models to create two semi-supervised long-tailed recognition baselines. Pscudo-Label combined with LDAM-DRW the method used in (Yang & Xu (2020)).
Mean Teacher (Tarvainen & Valpola (2017) is a well-known semi-supervised learning method. It contains a Student model that is trained with SGD and Teacher model that is updated with moving average of the Student. It is, however, unclear how to train it with Decoupling. We only implement I.DAM loss with Student training.
4.4 RESULTS
CIFAR-IO-SSLT results are shown in Table I with imbalance ratio 100 and 1000. Our methods outperforms all other methods in overall accuracy.
Our initialized model is equivalent to Decoupling, which shows the worst performance among all methods. Alternate learning improves the overall performance more than 7% when imbalance factor 100, and 20% with imbalance factor 1000. Most of the improvement is from medium and few- shot classes. The larger improvement on the more imbalanced distribution shows that our method is more effective with more skewed dataset.
When Pseudo-Label is added upon Decoupling, around 5% improvement is achieved with imbalance factor 100. But this improvement diminishes when the data more imbalanced. This implies the fact that Pseudo-Label is more sensitive to bad tail class labelling.
With the improvement upon Pseudo-Label, our method has the same amount of training epochs on unsupervised data. The extra calculation in our methods compared to Pseudo-Label is from Stage and 2. However, the classifier training is only on supervised data, and only the linear classifier is updated. And label assignment does not involve any back pack-propagation. The extra time on these two stages are trivial compared to the training of the whole model on the whole dataset.
LDAM-DRW provides very competitive results without any semi-supervised learning methods when imbalance factor is 100. However, it scales up bad when combined with semi- supervisd techniques. By adding Pseudo-Label, it only improves 2% of overall accuracy. After looking at the splits results, we find that it improves the few-shot performance at the cost of many-shot. We believe this is because the wrong balancing factor introduced LDAM loss. It does match the true distribution. and skews the training process. Mean Teacher makes little difference from Pseudo-Label on LDAM- DRW.
ImageNet- results are shown in Table | Our methods outperforms all baseline methods with both ResNet-18 and -50 architectures. The ImageNet-SSLT setting really challenging that all of the methods give below 30% overall accuracy. In fact, our method is the only one that improves the few-shot performance while maintains the many-shot accuracy.
On ImageNet-SSLLe SSLT, Pseudo-Label based methods lose efficacy, because it improves few-shot perfor- mance with sacrifice on many-shot. This sacrifice is sometimes big, such as Lauuoo-Lael+LDAM- DRW with ResNet-18. This is not observed when Pseudo- Label is used on CIFAR-IO-SSLT, where improves the many-shot performance. This may be due to the bad many-shot pseudo-label quality on ImageNet-SSLT. Unlike CIFAR- many-shot, many-shot performance on ImageNet-SSLT is only around 50%. These wrong labels mislead the training and lower the performance of Seeudo-Labell methods. Our method, on the other hand, updates the pseudo labels iteratively, and is less prone to this problem.
where the initialized model has 90% of accuracy
Specifically, adding Pseudo-Label on ADAM-DRW decreases the overall performance. This can be explained by the fact that the balancing factor in it does not match the true distribution. Mean Teacher improves upon I.DM-DRWW when ResNet-50 is used. But it is still not as good ours.
8

---

Under review as a conference paper at ICLR 2022
Table Ablation results(Accuracy in %) on CIFAR-IO-SSIE Imbalance factor 100 used. Sam- pling methods are denoted as R for random, and C for class-balanced. The last two method names shows where the embedding is trained.
Method
Overall
Many-Shot
Medium-Shie
Few-Shot
R+R
50.9
93.0
57.8
14.1
C+
61.2
91.3
62.6
37.6
C+C
63.3
91.2
64.4
41.6
D D'
70.1
89.6
68.7
56.5
D
63.3
61.9
4.5 ABLATIONS
We further study the training choices of alternate learning. This consists of two parts, i.e. the sampling choices and semi-supervised learning choices. Results on CIFAR-IO-SSIA with imbalance factor 100 are listed in Table
Sampling choice. Currently, during alternate learning we use random sampling in Stage 2 and class- balance sampling Stage 3. This is consistent with long-tailed recognition (Kang al. (2020). However, other combinations naming format: {sampling in Stage (samlling in Stage 3}. In method names, "R" stands for random sampling and "C" stands for hass-balanced.
possible. Results are listed in the first 3 lines of Table 2 with
None of the alternatives can beat the initialized model (Decoupling). This expected. When the classifier is randomly trained ("R+R" and "C+R"), the model performs bad on few-shot classes. This will turn harm the training of embedding by pseudo labels on unsupervised subset. "C+C" trains the feature embedding with class-balanced sampling. However, it is balancing on pseudo labels, which can be wrong. The results show that this balancing yields inferior feature embedding.
Semi-supervised learning choice. We train feature embedding with the whole dataset, i.e. DUW' and the classifier with labeled subset D. Other combinations can also be investigated. The classifier can also be semi-supervise trained, i.e. on D U At the same time, feature embedding trained with or without 4'. We show the results in the last 2 lines of Table 2 In these two experiments, the classifier is always trained on UZl' The difference whether UI used for embedding learning.
Compared to the regular setting, where the classifier is trained on D, when we train it on D U the performance is slightly lower. This can explained by the fact that wrong pseudo labels in II' can be propagated through loops if the classifier is optimized on them. This is especially true for few-shot classes, where the accuracy is low. Because class-balanced sampling, the impact of few-shot classes is amplified. When compared to Table the main performance drop is from few-shot classes. This confirms our assumption.
However, when further remove the unsupervised training of embedding, the performance drops a lot. is even worse than the initialized model (Decoupling). In this case, the feature embedding should be equivalent that of the initialization. The only difference is the classifier. This further proves the fact that fine-tuning classifier on pseudo-labels harms the performance.
CONCLUSION
This work introduces the supessupe long-tailed recognition problem. It extennds the long-tailed problem with unsupervised data. With the property labeled and unlabeled data obeying the same distribution, this problem setting follows the realistic data collection and annotation workflow.
A method based on alternate learning is proposed. By separating supervised training from semi- supervised and decoupling the sampling methods, it incorporates the decoupling training scheme in loig-tailed recognition with semi-supereised learning.
Experiments show that the proposed method outperforms all current baselines. When results split based on class cardinality, the method exhibits its robustness to defective pseudo labels. This especially true for few-shot classes.

---

Under review as a conference paper at ICLR 2022
REFERENCES
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint
arXiv:/905.02249 2019.
Kaidi Cao. Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-awarr margin loss. arXiv preprint arXiv:1906.0744 07411,, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Yanbei Chen, Xiatian Zhu, and Shaogang Gong. Semi-supervised deep learning with memory. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 268-283, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-244222 255. leee, 2009.
Spyros Gidaris. Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1803.07728, 2018
Bharath Hariharan and Ross Girshick. Low-shot visual recognition by shrinking and hallucinating features. In Proceedings the IEFE International Conference on Computer Vision, pp. 3018- 3027, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yan- nis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In Eighth International Conference on Learning Representations (ICLR), 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised leaming. arXiv preprint 16Xiv:1610.02244 2016.
Qimai Li, Xiao-Ming Wu, and Zhichao Guan. Generalized label propagation methods for semi- supervised learning. 2018
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pp- 2980-2988, 2017.
Jialun Liu, Yifan Sun, Chuchu Han, Zhaopeng Dou, and Wenhui Li. Deep representation learning on long-tailed data: A learnable embedding augmentation perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2970-2979, 2020.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
Hyun Oh Song. Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4012-4012, 2016.
10

---

Under review as a conference paper at ICLR 2022
Antti Rasmus, Harri Valpola, Mikko Honkala, Mathias Berglund, and Tapani Raiko. Semi- supervised learning with ladder networks. arXiv preprint arXiv: 1507.0267,, 2015.
Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Semi-supervised learning with scarce annotations. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition Workshops, pp. 762-763, 2020.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.07668 2020.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: tency targets improve semi coni-supervised deep learning results. arXiv preprint urXiv.1703.01780, 2017.
Weight- averrahv consis-
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence, 30(1 19:1958-1970, 2008
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778, 2018.
Yu-Xiong Wang and Martial Hebert. Learning to learn: Model regression networks for easy small sample learning. In European Conference on Computer Vision, pp. 616-634. Springer, 2016.
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In Advances in Neural Information Processing Systems, pp. 7029-7039, 2017.
Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data, In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pp. 7278 7286, 2018.
Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalanced learning- arXiv preprint arXiv:2006.07529, 2020.
Xiao Zhang, Zhiyuan Fang, Yandong Wen, Zhifeng Li, and Yu Qiao. Range loss for deep face recognition with long-tailed training data. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5409-5418, 2017.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with cumulative learning for long-tailes visual recognition. In Proceedings of the IEFE/CVF Confer- on Computer Vision and Pattern Recognition, pp. 2020. 2020.
A TRAINING DETAIL.
In initialization, the feature embedding is trained with 200 epochs, and classifier is learned in 10 epochs after that. Stage 2 contains 40 epochs of fine-tuning of the embedding the whole dataset. In 5 loops of stages, it is in total 200 epochs embedding fine-tuning. There are also 10 epochs of classifier fine-tuning in Stage 3 per loop. In semi-supervised learning loss (3) A is used.
SGD optimizer with learning rate of 0.1 is used with cosine annealing during training in all stages. The momentum is 0.9, and weight decay is 0.0005.
All comparison methods are implemented with the authors used when available.
in their papers. The codes from
11

---

Under review as a conference paper at ICLR 2022
Table 3: Pseudo label accuracy on unlabeled training subset. CIFAR-IO-SSII with imbalance ratio 100 is used. Compared to testing set, the unsupervised subset is not balanced. In result, the overall accuracy is higher than that on testing set, because of the domination of many-shot classes. The results in many/nedium/fewwshot splits are more useful.
Loop
Overall
Many-Shor
Medium-Shi
Few-Shot
0
87.7
92.3
63.0
41.8
87.9
92.3
64.0
48.1
2
87.8
92.1
64.7
52.2
3
87.8
91.8
65.3
55.8
4
87.7
91.6
65.8
57.8
Table Results(Accuracy in %) on CIFAR-IO-SSLE imbalance factor 100. ResNet-32 is used for all methods.
Method
Overall
Many
Medium Shot
LDAM-DRW
77.34
91.8
74.3
73.7
Pseudo Label L
81.1
87.6
75.6
80.7
Decoupling
68.2
91.6
66.9
49.2
Ours
83.1
92.0
77.7
80.4
B MORE ABLATIONS
Accuracy unsupervised training subset. In Stage assign pseudo labels for all samples in U. 3 reveals how the accuracy changes along loops in all splits. Few-shot split performance improves much faster than others. This proves the effectiveness of our alternate learning scheme, and explains why our method outperforms the baselines by a large margin in few-shot classes.
The unsupervised subset has a long-tailed distribution, the overall performance is dominated by many-shot. However, alternate learning still gets benefits from the improvement on few-shot split. Accuracy on different splits is more useful when we analyze how the model evolves during training.
More results. Table compares our method to Salsa and FixMatch. The gains are significant for all data splits. And ResNet-32 results are shown in Table 4
C NATURALIST2018-SSUI
Dataset. We further curate benchmark for semi-supervised long-tailed recognition based iNat- uralist 2018 (Van Horn et al. (2018)). iNaturalist 2018 is a long-tailed dataset sampled from natural distribution. We follow the distribution in both of the labeled and unlabeled subset. More specifi- cally. Samples in each class is randomly wwwn-sampled one- -ifthiihh of the total number as labeled data, and the remains are assigned as unsupervised subset. Classes with less than 2 labeled samples are eliminated. In result, iNaturalist201 8-SSLT contains 8080 classes, with labeled samples from 200 to 2, and the unsupervised subset is times larger.
Classes are divided into three splits based on the number of labeled samples: many-shot (1100, +oo)), medium-shot (110, 100)). and few-shot ([2, 10)). It is extremely long-tailed dataset, with 134 many-shot classes, 1220 medium-shot classes, and 7010 few-shot classes.
Results. Results are shown Table 6 Our method the only one that improves the overall is different from our other benchmarks in the amount of few-shot classes. It has a very long tail taking up 87% of the label space. This makes the dataset especially hard when combined with unsupervised data.
performance upon baseline. NNaturalist201--S55
With the inferior quality of predictions, we see significant drop of Pseudo-Labe method in many- shot split. In fact, Pseudo-Label decreases the accuracy of baselines in all splits. Our method mitigates this problem, and improve the few-shot performance. Given the fact that most classes are in sho- split, our method the only one that increase the overall performance.
12

---

Under review as a conference paper at ICLR 2022
Table Results(Aceuracy in %) on CIFAR-1O-SSIL imbalance factor 100. ResNet-18 is used for all methods.
Method
Overall
Many-Shet
Mediom-Shoe Shot
Fow-Shot
Salsa
59.6
82.5
60.7
41.5
FixMatch
64.1
83.6
62.4
50.6
Ours 71.3
89.5
67.7
60.2
Table Results(Accuracy %) on INaturalis2018-SSSI ResNet-50 are used for all methods. For many-shot > 100, for medium-shot t E (10, 100), and for few-shot I 10, where is the number of labeled samples.
Method
Overall
Many-Shot
Medium-Shot
Few-Shot
Decoupling
27.9
54.1
41.7
24.8
Pseudo-Label + Decoupling
26.3
39.9
35.8
24.3
Ours
28.4
49.5
38.7
26.1
Comparison among benchmarks. From CIFAR-1O-SSLA to ImageNet-SSLIT and iNaturalist001 8- SSLT, the datasets have more and more classes and few-shot classes. result, they are more and more challenging. This challenge makes Pseudo-Label method ineffective. From CIFAR-IO-SSLI to ImageNet-SSLT, the shortcoming first appears many-shot splits. On ImageNet-SSLT, Pseudo- Label improves the few-shot performance with sacrifice of many-shot performance. Our method more robust to this difficulty. keeps the many-shot performance while improves the few-shot performance. On Naturalist2018-SS5 SSLT, the Pseudo-Labe improvement appears, and the drop on many-shot is big. Our method, however, can still improves the few-shot performance and control the drop of many-shot compared to the baseline.
few-shot split also dis-
All of these results show that semi-supervised long-tailed recognition challenging problem. Given the fact that this problem follows the natural workflow of data collecting, believe it de- serves more attention in the literature.
13


Reviews:
[{"review_id": "qCms1Fi6sd", "reviewer": "Reviewer_PFkd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The biggest weakness is the experiments. I am not sure why the authors choose ResNet18 as the backbone for Cifar10 experiments. In previous papers, people are using ResNet32, and even without the help of unlabeled data, the results on Cifar10-LT (imbalance 100) were already over 70% using ResNet32 (for example the baselines, not the actual method, reported in this paper [1]). It is hard to judge how good the proposed method is by the reported results, especially when the compared baselines are limited. Most of the existing long-tail methods after decoupling are not compared in the experiments (for example, BBN[2], logit-adjustment[3], and RIDE[4]). Although these methods were not designed for semi-supervised learning, simply adding a pseudo-label component to these baselines is not complicated. In addition, advanced semi-supervised learning methods (like FixMatch[5]) are not discussed either.\n\nBesides the experiments, I think the proposed method lacks novelty as well. Simply adding a pseudo-label component to the decoupling training procedure is too engineering. Since this paper is not a theoretical paper, the novelty of the method is a little more important. \n\n\n[1] Cai, J., Wang, Y., & Hwang, J. N. (2021). ACE: Ally Complementary Experts for Solving Long-Tailed Recognition in One-Shot. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 112-121).\n\n[2] Zhou, B., Cui, Q., Wei, X. S., & Chen, Z. M. (2020). Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9719-9728).\n\n[3] Menon, A. K., Jayasumana, S., Rawat, A. S., Jain, H., Veit, A., & Kumar, S. (2020). Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.07314.\n\n[4] Wang, X., Lian, L., Miao, Z., Liu, Z., & Yu, S. X. (2020). Long-tailed recognition by routing diverse distribution-aware experts. arXiv preprint arXiv:2010.01809.\n\n[5] Sohn, K., Berthelot, D., Li, C. L., Zhang, Z., Carlini, N., Cubuk, E. D., ... & Raffel, C. (2020). Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.07685.", "summary_of_the_paper": "This paper proposes a long-tailed semi-supervised learning setting where both labeled and unlabeled data from a long-tailed distribution exist. To address this setting, they use an iterative decoupling training method, which is based on Kang et al 2020. The only difference is after initialization using the original decoupling method, pseudo labels are obtained from unlabeled data and the feature extractor is updated with pseudo labels, then the classifier is updated using the new feature representations. Two new benchmarks are provided, and the methods had improved performance over selected baselines. ", "summary_of_the_review": "I think the proposed setting is interesting but the experiments are weak. Since the paper is not a theoretical paper, I think the novelty of methods should also be considered. However, the proposed method lacks sufficient novelty since the only additional component is a pseudo-label generation with a consistency loss. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "k5ejjpH76LM", "reviewer": "Reviewer_mLKc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The author propose the new setting of semi-supervised long-tailed recognition, where both labelled and unlabelled data are long-tailed. \n2. This paper is well-written, clear and easy to understand. The structure of this paper is also good.\n3. The proposed 3-staged method seems reasonable. I am not sure if the author is based on the findings that some balanced strategy, like class-balanced sampling, will hurt the quality of learned feature. Even they achieve better results. The improvement lies in classifier. Thus the  authors update the feature and classifier separately.\n\nWeaknesses:\n1. It is also about the new setting proposed. It is actually more like considering the characteristics of long tail in semi-supervised learning. It may not so significant by constraining that the unlabelled data also exhibit long-tailed distribution in long-tailed recognition. As claimed, the unlabelled data will help our classification, that means we may use as many as unlabelled data as possible, for example, the open-set data.\n\n2. Even we consider the long tailed case in unlabelled data, should we consider different long tailed versions? for example, the unlabelled data maybe uniformly distributed? The degree of skewness is less or more severe than the labelled data? Even worse, the many-, medium-, few- classes in unlabelled data may shift since the collected labelled data are easily biased. \n\n3. In Table 1, the overall results are the best. However, the few- accuracy of proposed methods seems lower than \"Pseudo-Label + L\" and \"Mean-teacher+L\". Why is the case? One motivation of proposed method is trying to improve the result when predicting the label of unlabelled data. It seems contradictory with the motivation. The proposed method did not improve the result of few-classes.\n\n", "summary_of_the_paper": "This paper propose a new setting, named semi-supervised long-tailed recognition. They consider both of the labelled data and unlabelled data exhibit long-tailed distribution. To solve the problem, they propose an alternate sampling framework which learns the feature and classifier separately and update them iteratively. On two datasets, the results validate the efficacy of proposed method.", "summary_of_the_review": "The author proposes a new setting of semi-supervised long-tailed classification and clearly demonstrated it. However, the new setting seems minor and not so novel. I thus give the score of 5. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cTUcjpt6vck", "reviewer": "Reviewer_Y7tQ", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n- This paper is easy to read, and the proposed method is easy to follow.\n\nWeakness\n1. The proposed framework is not new. Semi-supervised long-tailed recognition has already been studied with the name of imbalanced semi-supervised learning [1,2,3].\n\n2. The proposed method is also not novel. In Eq. (2), the authors propose to use temporal consistency loss for learning unlabeled data, which is similar to the method of [5]. This is already outdated in the semi-supervised learning literature where better loss functions [6,7] or pseudo-labeling-based methods [8] show the state-of-the-art performances.\n\n3. The main technical contribution, alternating sampling is also nothing but simple engineering that any long-tailed recognition or semi-supervised learning researchers do. In order to claim the significance of the proposed method, the authors should have at least provided a more comprehensive analysis on the proposed method (e.g., with different imbalance factors).\n\n4. The experiment is also too weak. The authors should have evaluated the proposed method on more training setups (at least one more imbalance factor) and more realistic datasets (iNaturalist [9] or other large-scale datasets) compared with stronger baseline methods (imbalanced semi-supervised learning [2,3], recent long-tailed recognition [10], or at least the FixMatch [8]).\n\n[1] Hyun et al., Class-imbalanced semi-supervised learning. arXiv 2020.\n\n[2] Kim et al., Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning. NeurIPS 2020.\n\n[3] Wei et al. CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning. CVPR 2021\n\n[4] Oh et al., Distribution-Aware Semantics-Oriented Pseudo-label for Imbalanced Semi-Supervised Learning. arXiv 21.\n\n[5] Laine et al., Temporal Ensembling for Semi-Supervised Learning. ICLR 2017\n\n[6] Tarvainen et al., Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. NIPS 2017\n\n[7] Berthelot et al., Mixmatch: A holistic approach to semi-supervised learning.. NeurIPS 2019.\n\n[8] Sohn et la., Fixmatch: Simplifying semi-supervised learning with consistency and confidence. NeurIPS 2020.\n\n[9] Su et al., The Semi-Supervised iNaturalist-Aves Challenge at FGVC7 Workshop. arXiv 2021\n\n[10] Menon et al., Long-tail learning via logit adjustment. ICLR 2021.\n", "summary_of_the_paper": "The authors tackle the problem of semi-supervised long-tailed recognition. To address the semi-supervised long-tailed recognition problem, the authors introduce an alternating sampling framework.", "summary_of_the_review": "There are various obvious weaknesses in this paper including the comprehensiveness of the experiments and the novelty of the problem definition and the proposed method. Therefore, recommend the authors significantly revise and re-submit this paper to the other venue.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "FbpkgBmfXY", "reviewer": "Reviewer_minL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1. The idea of alternating the sampling strategy is very simple yet powerful. \n2. The paper identifies an important problem for using pseudo-label in long-tailed learning: the problem with inaccurate pseudo-labels exacerbates when employing class-balanced sampling. The finding comes with solid empirical evidence (Table 2) and analysis (Sect 3.5).\n3. The empirical results are promising. The proposed method outperforms the existing methods by a significant margin (Table 1). Sufficient ablation studies are provided. Particularly, I find the loop-wise pseudo label accuracy shown in Table 3 insightful.\n\n\nWeakness:\n\n1. A semi-supervised feature learning baseline is missing.\n\n   This is my main concern about the paper. The key argument in the paper is that feature learning and classifier learning should 1) be decoupled, 2) use random sampling and class-balanced sampling respectively, 3) train on all labels and only ground-truth labels respectively. The authors, therefore, propose a carefully designed alternate sampling strategy.\n\n   However, a more straightforward strategy could be 1) train a feature extractor ($f$ in the paper) and a classifier ($g\\prime$ in the paper) using random sampling and any semi-supervised learning method on all data, then 2) freeze the feature extractor ($f$) and train a new classifier ($g$ in the paper) using class-balanced sampling on data with ground-truth labels. Compared with the alternate sampling strategy proposed in this paper, the semi-supervised feature learning baseline will take less implementation effort and is easier to combine with any semi-supervised learning methods.\n\n   The baseline seems to be missing in the paper. Although the naive baseline may not give the best performance, it should be compared to justify the sophisticatedly designed alternate sampling strategy.\n\n2. References are not up-to-date.\n\n   All references in the paper are in or before 2020. In fact, much research progress has been made since then. For example, some recent works [1, 2] study the class-imbalanced semi-supervised learning as well, a discussion on these methods should be necessary. A recent survey on long-tailed learning [3] could be a useful resource to help update the related works in the paper.\n\n3. Minor issues: \"The model is the fine-tuned on the combination of ...\" -> \"The model is then fine-tuned on the combination of ...\"\n\n[1] Su et al., A Realistic Evaluation of Semi-Supervised Learning for Fine-Grained Classification, CVPR 2021\n\n[2] Wei et al., CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning, CVPR 2021\n\n[3] Deep Long-Tailed Learning: A Survey, arXiv 2021", "summary_of_the_paper": "This paper proposes to alternate the sampling strategy in the semi-supervised setting and therefore fully exploit the advantage of decoupled learning.", "summary_of_the_review": "My overall attitude towards the paper is borderline. I appreciate the simplicity and effectiveness of the method provided in the paper. However, I believe that a more straightforward baseline should be compared in order to justify the design of the alternate sampling strategy.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-KpYXwzH6VU", "reviewer": "Reviewer_6A7n", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1)\tI believe this work is well motivated. The setting is more realistic than supervised imbalanced recognition in that imbalance usually comes together with unlabeled datasets. In this case, randomly sampling a part of the examples and labeling them manually is reasonable. In this sense, the proposed setting is more closed to what practitioners will encounter in the wild.\n2)\tThe manuscript is overall well-written and easy to follow. The authors introduce the setting and the motivations clearly. The contributions are also clearly summarized. \n3)\tThe intuition behind each stage of the method is stated clearly and emphasized. From my perspective, the proposed method is not a simple combination of pseudo-labeling with decoupling since the authors managed to associate each part of the networks with different training stages and take the effect of imbalance on pseudo-labels into consideration.\n\nCons:\n1)\tThe baselines are not strong enough. Although the proposed method outperforms the baselines by a large margin, it is still not clear how good the performance is. For supervised imbalanced recognition, the authors consider LDAM+DRW and decoupling as baselines. For standard semi-supervised learning, the authors compare with pseudo-labeling and mean teacher. However, these are not recent state-of-the-arts. It would be better to compare with leading methods on https://paperswithcode.com/sota/long-tail-learning-on-imagenet-lt and better semi-supervised methods such Fixmatch and Noisy student training.\n2)\tIt is unclear how the pseudo-labels are refined by stages 2 and 3. The authors should provide numbers or figures to demonstrate the effect of stages 2 and 3 separately.\n3)\tFor this setting, a more natural method could be: train the representations on all data with self-supervised learning, and fine-tune the representations with the supervised data, and train the final classifier with class aware-sampling. It could e interesting to compare with this baseline.\n", "summary_of_the_paper": "This paper proposes a new setting--semi-supervised long-tailed recognition. To harness the imbalanced unlabeled data, the authors combined the decoupling in long-tailed recognition and pseudo-labeling in semi-supervised learning, which formulates a three-stage method. Stage 1 generates pseudo-labels with a classifier trained with class-aware resampling. Stage 2 fine-tunes the feature extractor on with pseudo-labeling. Stage 3 trains the classifier with class-aware resampling on top of the refined feature extractor. Experiments indicate that the proposed method outperforms baselines.", "summary_of_the_review": "I like the setting and the method, but I still lean towards rejecting the paper, because the experiments are not convincing enough.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses semi-supervised learning with unbalanced class distribution, a.k.a long-tail. The main idea is to alternate learning of the representation and the classifier. 
Reviewers pointed out that several papers already addressed this learning setup, often under the name "imbalanced semi-supervised learning". No rebuttal was submitted. 

The paper should make direct comparison to recent papers listed by reviewers, both in terms of the technical approach and in terms of empirical experiments. It cannot be accepted tot ICLR.
Decision: Reject