Title: Continuous Contrastive Learning for Long-Tailed Semi-Supervised Recognition
Venue: NeurIPS 2024
Accepted: yes
Keywords: Semi-supervised learning

Long-tail learning

Weakly-supervised learning
Abstract: Long-tailed semi-supervised learning poses a significant challenge in training models with limited labeled data exhibiting a long-tailed label distribution. Current state-of-the-art LTSSL approaches heavily rely on high-quality pseudo-labels for large-scale unlabeled data. However, these methods often neglect the impact of representations learned by the neural network and struggle with real-world unlabeled data, which typically follows a different distribution than labeled data. This paper introduces a novel probabilistic framework that unifies various recent proposals in long-tail learning. Our framework derives the class-balanced contrastive loss through Gaussian kernel density estimation. We introduce a continuous contrastive learning method, CCL, extending our framework to unlabeled data using *reliable* and *smoothed* pseudo-labels. By progressively estimating the underlying label distribution and optimizing its alignment with model predictions, we tackle the diverse distribution of unlabeled data in real-world scenarios. Extensive experiments across multiple datasets with varying unlabeled data distributions demonstrate that CCL consistently outperforms prior state-of-the-art methods, achieving over 4% improvement on the ImageNet-127 dataset. The supplementary material includes the source code for reproducibility.

Continuous Contrastive Learning for Long-Tailed Semi-Supervised Recognition

Abstract

Long-tailed semi-supervised learning poses a significant challenge in training models with limited labeled data exhibiting a long-tailed label distribution. Current state-of-the-art LTSSL approaches heavily rely on high-quality pseudo-labels for large-scale unlabeled data. However, these methods often neglect the impact of representations learned by the neural network and struggle with real-world unlabeled data, which typically follows a different distribution than labeled data. This paper introduces a novel probabilistic framework that unifies various recent proposals in long-tail learning. Our framework derives the class-balanced contrastive loss through Gaussian kernel density estimation. We introduce a continuous contrastive learning method, CCL, extending our framework to unlabeled data using reliable and smoothed pseudo-labels. By progressively estimating the underlying label distribution and optimizing its alignment with model predictions, we tackle the diverse distribution of unlabeled data in real-world scenarios. Extensive experiments across multiple datasets with varying unlabeled data distributions demonstrate that CCL consistently outperforms prior state-of-the-art methods, achieving over 4% improvement on the ImageNet-127 dataset. Our source code is available at https://github.com/zhouzihao11/CCL .

1 Introduction

Semi-supervised learning (SSL) serves as a powerful approach for improving the generalization capabilities of deep neural networks (DNNs) in scenarios where labeled data is scarce [37, 59, 6, 23]. The core concept of SSL methods typically involves assigning pseudo-labels to unlabeled data and utilizing those with high confidence for model training [56, 71, 10]. However, many existing SSL algorithms presuppose a balanced label distribution across both labeled and unlabeled datasets. In real-world applications, datasets commonly exhibit a long-tailed label distribution [65, 28, 50, 67, 55], leading to biased pseudo-label generation favoring majority classes [40, 3, 68, 24]. This discrepancy challenges the effectiveness of SSL algorithms in addressing real-world datasets.

The exploration of long-tailed semi-supervised learning (LTSSL) has gained momentum to address the challenge of biased pseudo-label distribution arising from class imbalance in labeled and unlabeled data. Recent LTSSL approaches propose compensating for the learning of minority classes by

distribution alignment [33, 63], data rebalancing [22, 38], and logit adjustment [64, 43] to rectify the pseudo-label distribution. However, existing approaches often assume the equivalence of the unlabeled data distribution with the labeled data or rely on predefined anchor distributions to estimate the unlabeled data distribution [64, 43]. Furthermore, these methods primarily focus on correcting model outputs without delving into the role of representation learning in improving performance.

This paper explicitly introduces an approach to obtain effective representations for long-tail learning by adopting an information-theoretic view of DNNs. We present a probabilistic framework that utilizes the deep variational information bottleneck method [1] to learn good representations and demonstrate its unification of recent long-tail learning proposals, such as logit adjustment [44] and balanced softmax [51], through approximating the density of class-conditional distribution in different ways. Specifically, our framework encompasses class-balanced supervised contrastive learning [73, 15] via Gaussian kernel density estimation. We extend this framework to address LTSSL by adapting the supervised contrastive loss to unlabeled data using ' continuous pseudo-labels ', derived from model predictions and propagated labels, to mitigate confirmation bias. To account for varying label distribution of unlabeled data, we progressively estimate the label distribution through a moving average and adjust model predictions to align with the estimated distribution.

In summary, our contributions are as follows:

We propose a probabilistic framework which unifies many recent proposals in long-tail learning. Specifically, popular class-balanced contrastive learning methods can be seen as special cases of our framework when approximating the density using a Gaussian kernel.

We generalize the proposed framework to LTSSL and present a continuous contrastive learning method based on reliable and smoothed pseudo-labels to address confirmation bias and improve the quality of learned representations.

We conduct extensive experiments across several LTSSL datasets with diverse label distributions of unlabeled data. The results show that our proposal substantially outperforms previous state-ofthe-art methods.

2 A Probabilistic Framework for Long-Tail Learning

In this section, we first introduce a general framework for learning good representations. Then, we expand this framework to long-tail learning and illustrate how recent proposals can be regarded as specific instances of our framework through three ways for density approximation.

Problem setup of long-tail learning. We consider a C -class classification problem with instance space X and target space Y = { 1 , . . . , C } . Let P s and P t denote the source (training) and test distributions on ( X , Y ) , respectively. We denote by P s and P t the corresponding probability density (or mass) functions. Given a training dataset { ( x i , y i ) } N i =1 , where x i ∈ R d is the training sample and y i is the ground-truth label.

2.1 Learning good representations from information theoretical view

In this subsection, we rethink one of the most popular approaches to deal with representation learning, i.e., contrastive learning. We derive many recent proposals in this branch from an informationtheoretic view. Let Z denote the latent representation of X induced by the encoder enc( · ) parameterized by Θ . From an information-theoretic view, an optimal representation Z is maximally informative about the target Y , and minimally 'memorizes' X . The information bottleneck [60] adopts mutual information I ( · ) to measure information between two variables. Thus, optimal Z can be obtained by maximizing the following objective:

where δ ≥ 0 is a tradeoff parameter. The variational information bottleneck [1] solves the above objective by variational inference. Based on the definition of I ( · ) , we can rewrite Eq. (1) as:

where we omit Θ for simplicity. Since P ( y | z ) = ∫ d x P ( x | z ) P ( y | x ) is intractable in Eq. (2), let ̂ P ( y | z ) be a variational approximation to P ( y | z ) and considering that the Kullback-Leibler (KL)

divergence KL[ P ( y | z ) , ̂ P ( y | z )] is always positive, we have RHS of Eq. (2)'s lower bounded:

Suppose we use an encoder of the form P ( z | x ) ∼ N (enc( x ) , ε 2 I ) and P ( z ) ∼ N ( 0 , I ) , the second term of Eq. (3) equals the KL divergence KL[ P ( z | x ) , P ( z )] . Since P ( z | x ) and P ( z ) are normal distributions, it can be rewritten as: -δl ( 1 2 ε 2 -1 -2 log ε ) -δ ∥ enc( x ) ∥ 2 , where l is the dimension of z . For a deterministic model, z is almost unique for each x , thus assuming ε is a small constant close to 0. By integrating out P ( z | x ) and discarding constant terms, maximizing Eq. (3) can be approximated by minimizing:

In the following of this paper, we denote the output of enc( x ) as z (or z x for a particular sample x ) for simplicity. Minimizing Eq. (4) is equivalent to minimizing the following objective in the distribution of test data on each x :

Notably, Eq. (5) can be seen as a general framework for learning good representations. If P s ( Y ) = P t ( Y ) , P t ( Y = y | x ) can simply be substituted by the ground-truth labels of training samples. However, in long-tail learning, the class-probability function P t ( Y = y | x ) is different from that of the training data due to label distribution shift.

2.2 Probabilistic framework for long-tailed supervised learning

̸

Since P s ( Y = y | x ) = P t ( Y = y | x ) , we cannot directly solve Eq. (5). However, since long-tail learning typically assumes that P t ( Y ) is uniform and we work with the label shift assumption, i.e., P s ( x | Y = y ) = P t ( x | Y = y ) , we can obtain P t ( Y = y | x ) by Bayes' theorem:

Throughout the paper, we use the notation P ( x | Y = y ) to represent either P s ( x | Y = y ) or P t ( x | Y = y ) . In practice, ∥ z ∥ 2 can be omitted in optimization because normalization is commonly adopted in deep learning. In long-tail learning, minimizing Eq. (5) equals to minimizing:

According to Jensen's inequality, Eq. (7) attains its minimum value if and only if ̂ P t ( Y = k | x ) P s ( Y = k ) ∝ P s ( Y = k | x ) for k ∈ [ C ] . Hence, Eq. (7) can be replaced as follows:

where ̂ P ( Y = y | z ) = ̂ P t ( Y = y | z ) P ( Y = y ) ∑ k ∈ [ C ] ̂ P t ( Y = k | z ) P ( Y = k ) and P ( Y ) is an arbitrarily label distribution. Eq. (8)

can be seen as an extension of sample reweighting [52] and logit adjustment [44] using probabilistic labels rather than discrete labels when P ( Y ) is specified as P t ( Y ) and P s ( Y ) , respectively. Besides, Eq. (8) presents a unified framework that consolidates existing long-tail learning methods by estimating ̂ P ( z | Y = y ) or ̂ P t ( Y = y | z ) in different ways. In the following, we discuss three ways to estimate these terms.

Method 1: Explicitly specify ̂ P ( z | Y = y ) as a prior distribution such as vMF distribution [34] and Wrapped Cauchy Distribution [27].

Method 2: Approximate ̂ P ( z | Y = y ) using a learnable linear classifier. Let { w i , b i } C i =1 denote the parameters of a linear layer, which is followed by a softmax to obtain the normalized probability:

Method 3: Approximate ̂ P ( z | Y = y ) via the Gaussian kernel. A new sample from class y should be closer to all samples within class y and away from samples from other classes. Using the expected similarity among all samples within the class to measure distance, we derive:

where κ ( · , · ) represents the similarity between two samples, when we use Gaussian kernel κ ( z x , z x ′ ) = exp( z x · z x ′ ) and approximate expectation through empirical batch B = ∪ k ∈ [ C ] B k , that is E x ′ ∼ P ( ·| Y = y ) [ κ ( z x , z x ′ )] ≈ 1 |B y | ∑ x ′ ∈B y exp( z x · z x ′ ) , Eq. (10) can be instantiated as:

Interestingly, we observe that Eq. (11) resembles class-balanced contrastive loss. In the appendix, we also show that the Gaussian kernel approximation is identical to conventional supervised contrastive learning if the training data are class-balanced.

Notably, to ensure the computability of E x ′ ∼ P ( ·| Y = y ) [ κ ( z x , z x ′ )] in Eq. (10), it is essential to ensure that samples are available from each class. Existing methods address this by class-wise queues, class-wise centers, or class-wise vMF distribution, details of which are provided in the appendix.

Based on the above three density approximation methods, we find that many recent proposals in long-tail learning can be derived from our framework. In Table 1, we summarize existing methods based on the way they estimate the density, tackle the training/test label shift, and guarantee the computation of Eq. (10) in mini-batch training.

3 CCL: Continuous Contrastive Learning

In this section, we introduce the proposed algorithm CCL, which extends the class-balanced contrastive learning presented in Eq. (8) with Gaussian kernel estimation in Eq. (11) to LTSSL.

3.1 Problem setup of long-tailed semi-supervised learning

Let P l and P u denote the joint distribution ( X , Y ) for labeled data and unlabeled data, respectively. We denote by P l and P u the corresponding probability density (or mass) functions. We possess a labeled dataset { ( x l i , y l i ) } N i =1 of size N and an unlabeled dataset { x u j } j M =1 of size M , where x l i , x u j ∈ R d . The proportion of labeled data from the entire dataset is η = N M + N . Denote the number of labeled data for class i as N i , we have N 1 ≥ N 2 ≥ . . . ≥ N C if the classes are sorted by cardinality in decreasing order. The imbalance ratio of labeled data is given by γ l = N 1 N C , while the distribution of the label of the unlabeled data and its imbalance ratio γ u are unknown. The components of CCL include a feature extractor, two linear classifiers f s ( · ) , f b ( · ) and a contrastive feature projection head g ( · ) .

3.2 Balanced classifier training with estimated class prior

We develop our method based on FixMatch [56] following previous works [48, 64], and its objective is: ̂ L ssl = ̂ L l + ̂ L u , where ̂ L l is a traditional cross-entropy loss. For unlabeled data, the method operates by first generating pseudo-labels for unlabeled data using the model's predictions and selecting unlabeled data whose predicted maximum confidence is higher than a predefined threshold. The consistency regularizer ̂ L u is then applied to two views of each selected sample.

Balanced FixMatch for LTSSL. First, since the labeled data follow a long-tailed distribution, which we denote as π l , ̂ L l needs to be adjusted by logit adjustment [44] via Eq. (8):

Second, FixMatch is prone to fit wrong pseudo-labels with high predictive confidence during training [62, 3]. However, since the unlabeled data label distribution is inaccessible, pseudo-labels generated by the classifier may be sub-optimal for its adherence to a uniform distribution. By Bayes' theorem, with given estimated unlabeled data label distribution ̂ π u , the 'post-adjusted' model outputs for sample x u can be formulated as:

Given pseudo-label ̂ y = arg max k ∈ [ C ] ̂ P u ( Y = k | A w ( x u ); f b ) , where A w ( · ) denotes the weak data augmentation, ̂ L u is rewritten as:

̸

where M ( · ) denotes the sample mask to select reliable pseudo-labels. We progressively update ̂ π u using the exponential moving average (EMA) for each mini-batch by ̂ π u y = (1 -α ) ̂ π u y + α |B| ∑ x u ∈B ̂ P u ( Y = y | x u ; f b ) , where α is a momentum updating parameter and B denotes an unlabeled data subset. Directly using confidence selection can lead to a selected B with poor calibration due to model overconfidence [41, 45]. Thus, the energy score [36] is adopted to filter out reliable unlabeled data, which is defined as E ( x ) = -T · log ∑ k ∈ [ C ] e f k ( x ) /T , where T is the temperature and f ( x ) denotes the logits of x . We select reliable unlabeled data by M E ( x u ) := I ( E ( x u ) ≤ ζ ) using a predefined threshold ζ , and construct B = { x | x ∈ B u ∧ M E ( x ) = 0 } for the estimation of ̂ π u .

Dual-branches training. Based on the observation that class-balanced training can be harmful to representation learning, previous works [38, 64] have utilized another branch of the network for standard training. In contrast to the balanced branch, the standard branch, denoted as f s ( · ) , optimizes the cross-entropy loss without employing logit adjustment to fit the original training data distribution. We fuse the predictions of balanced and standard branches to reduce the confirmation bias of pseudo-labels by:

The rationale behind the equation is that the standard branch necessitates the elimination of imbalanced label prior and then compensates for unlabeled label prior when predicting pseudo-labels, which is achieved by defining ̂ π ∗ = ̂ π u π l + ̂ π u . Overall, the classification loss ̂ L cls is the combination of losses for learning f s ( · ) and f b ( · ) .

3.3 Continuous contrastive loss with reliable pseudo-labels

Apart from the classification loss and consistency regularizer, we aim to improve the quality of representations by extending the framework presented in Section 2 to LTSSL. To achieve the adaptation

of our framework, a primary obstacle must be addressed. The challenge arises from the unknown ground-truth label P u ( Y = y | x ) for unlabeled data, which results in the calculation of Eq. (8) infeasible. We propose to utilize the continuous pseudolabel ̂ P cls ( Y = y | x u ) as derived from the classifier in Eq. (15). Furthermore, E x ′ ∼ P ( ·| Y = y ) [ κ ( z x , z x ′ )] in Eq. (10) can be extended to unlabeled data and approximated by an empirical data subset B :

Plugging Eq. (16) into Eq. (10), we obtain the continuous pseudo-label ̂ P t ( Y = y | x u ; B ) for x u . Similar to Eq. (14), logit adjustment is used to handle label shift of unlabeled data by Bayes' theorem, and we can obtain:

̸

where ̂ P u ( Y = y | x u ; B ) = ̂ P t ( Y = y | x u ; B ) · ̂ π u y ∑ k ∈ [ C ] ̂ P t ( Y = k | x u ; B ) · ̂ π u k . So far, the generalized framework using continuous pseudo-labels for LTSSL is derived in Eq. (17), the critical issue is how to filter out a reliable unlabeled data subset B u such that the posterior estimation ̂ P cls ( Y = y | x ) in Eq. (16) is calibrated. Similarly, directly using confidence selection may lead to model overconfidence. To mitigate the confirmation bias in pseudo-labels produced by the learned classifier, we propose using the energy score for data selection to ensure model calibration [26]. Combining with labeled data, the loss ̂ L rpl is obtained with B = { x | x ∈ B l ∨ ( x ∈ B u ∧ M E ( x ) = 0) } .

3.4 Continuous contrastive loss with smoothed pseudo-labels

To further mitigate the impact of inaccurate pseudo-labels ̂ P cls ( Y = y | x u ) , we derive a complementary contrastive loss with smoothed pseudo-labels. Specifically, we propose aligning the representations of two views of a sample by imposing the weak-strong consistency regularization:

In this part, we aim to derive ̂ P ( Y = y | x u ) by propagating labels from nearby samples in the contrastive space. On the one hand, we take labeled data for B in Eq. (11) and construct the posterior for unlabeled data. Logit adjustment is employed for tackling label shift of unlabeled data:

Eq. (19) can be viewed as a process of propagating labels from labeled data to unlabeled data. On the other hand, we consider label propagation within unlabeled data, i.e., an unlabeled batch B u is used to estimate ̂ P ( Y = y | x u ; B ) . Assuming there is a sufficient amount of unlabeled data, we have 1 |B u | ∑ x u ∈B u ̂ P ( Y = y | x u ; B u ) ≈ ̂ π u y , hence the posterior can be approximated as:

Let P ( Y | X ; B ) represent a matrix stacked by [ P ( Y = 1 | x ) , . . . , P ( Y = C | x )] ⊤ of x from B , we can rewrite Eq. (20) in the form of matrix multiplication: ̂ P ( Y | X u ; B u ) = G · ̂ P ( Y | X u ; B u ) , where G is a similarity matrix and G ij = κ ( z x i , z x j ) ∑ x j ∈B u κ ( z x i , z x j ) . It can be interpreted that similar samples possess similar labels. By aggregating the predictions of labeled data and unlabeled data with a fixed hyperparameter β , we obtain:

Subsequently, Eq. (21) can be plugged into Eq. (18) for calculating ̂ L spl . To sum up, the total objective of CCL is:

where λ 1 and λ 2 are two hyperparameters.

̸

̸

4 Experiments

In this section, we conducted comprehensive experiments to verify the effectiveness of the proposed continuous contrastive learning method (CCL) on CIFAR10-LT, CIFAR100-LT, STL10-LT, and ImageNet-127 [29, 18] datasets. To simulate real-world unlabeled data, we tested our method on diverse label distributions of unlabeled data. Due to limited space, we defer the detailed experimental settings to the appendix.

4.1 Results on CIFAR10/100-LT and STL10-LT

For consistent ( γ l = γ u ) setting, results are presented in Table 2. From the results, we can see that CCL consistently outperforms all comparison methods by a large margin. In particular, CCL improves the previous state-of-the-art approach ACR by 2.8% on average. This verifies that the representations learned by our proposed contrastive losses are more discriminative because both CCL and ACR utilize a dual-branch network.

̸

For inconsistent ( γ l = γ u ) setting, we present the results in Table 3 and Table 4. Following prior works, we compare all methods using unlabeled data following uniform and reversed label distributions on CIFAR10/100-LT datasets. On the STL10-LT dataset, the underlying unlabeled data distribution is naturally inaccessible. In general, CCL achieves the best results in all settings. Particularly, CCL obtains an average performance gain of 1.6% over ACR without using predefined anchor distributions. The results indicate that our method is able to accurately estimate the unlabeled data distribution with calibrated pseudo-labels.

4.2 Results on ImageNet-127

ImageNet-127 is a naturally long-tailed dataset and has been used to test LTSSL methods in the recent literature [22, 64]. Following previous works, we downsample the original images to smaller sizes of 32 × 32 or 64 × 64 pixels using the box method from the Pillow library and randomly select 10% training samples to construct the labeled data. Learning discriminative representations and a balanced classifier is essential to achieve high performance. From the results in Table 5, we can see that CCLachieves superior results for image sizes of 32 × 32 and 64 × 64. It outperforms ACR by 4.3% and 4.2% in test accuracy.

4.3 Comprehensive evaluation of the proposed method

Understanding of balanced Fixmatch and dual-branch. First, balanced Fixmatch can be viewed as a separate EM algorithm process [17, 21], where the E-step involves estimating suitable pseudo-labels for unlabeled data through ̂ π u , and the M-step updates the model and obtains a new ̂ π u . As can be seen in Table 6, balanced Fixmatch achieves performance comparable to the recent state-of-the-art method ACR. Furthermore, dual-branch significantly enhances the performance of data under highly skewed long-tail distributions ( consistent setting), with an averaged 1 . 5% improvement.

0

100

200

300

400

500

Epoch

0.25

0.30

0.35

0.40

0.45

0.50

0.55

Estimation Error

confidence

energy

(a)

consistent

setting

0

100

200

300

400

500

Epoch

0.4

0.5

0.6

0.7

Estimation Error

confidence

energy

(b)

reversed

setting

0

100

200

300

400

500

Epoch

0.0

0.1

0.2

0.3

0.4

0.5

ECE Value

confidence

energy

(c)

consistent

setting

0

100

200

300

400

500

Epoch

0.0

0.2

0.4

0.6

ECE Value

confidence

energy

(d)

reversed

setting

How to estimate a relatively accurate ̂ π u ? Our method for estimating ̂ π u is equivalent to MLLS [53], which is an EM process. Accurate estimation of ̂ π u is only achievable when the model is calibrated [25]. Since the confirmation bias is induced by self-training, using confidence selection may result in overconfident but wrong pseudo-labels and hurt the calibration [45, 41]. In contrast, the energy score leverages the probability density of the predictions, exhibiting reduced vulnerability to overconfidence [39]. Thus, we propose energy selection for a reliable unlabeled data subset on which the model is calibrated, thereby enabling the accurate estimation of ̂ π u . We use expected

calibration error [26] (ECE) to assess model calibration. The tail of the curve of Figure 1c and 1d can be interpreted as overconfidence in false pseudo-labels caused by self-training. As can be seen in Figure 1a and 1b, the L 1 distance between the true class prior of unlabeled data and ̂ π u , estimated from data subset selected using energy, is significantly smaller compared to when confidence is used for selection, inducing a more balanced classifier training.

Continuous contrastive learning with reliable pseudo-labels. We carried out a comparative experiment by removing the continuous reliable pseudo-labels loss. The results reflect an averaged 0 . 8% drop on CIFAR10/100-LT, demonstrating its efficacy for learning high-quality representation. Moreover, we verified that the data subset filtered by energy selection obtains excellent model calibration. Figure 1c and 1d show energy achieves better calibration than confidence thresholding.

Continuous contrastive learning with smoothed pseudo-labels. Similarly, we conducted a comparative experiment by removing the continuous smoothed pseudo-labels loss. As can be seen in Table 6, the performance decreases in all three settings on CIFAR10/100-LT datasets, showing the necessity for a consistency regularization constraint for feature alignment within the contrastive learning space.

4.4 Results under more class distributions

Similar to ACR, to evaluate our method's effectiveness under more imbalanced settings, we conducted further experiments on CIFAR100LT, maintaining a fixed γ l = 20 and adjusting the imbalance ratio γ u of the unlabeled data from consistent to reversed. We set N 1 = 50 and M 1 = 400 (with M C = 400 in the reversed scenario) and compared the results with ACR as shown in Figure 2. The results demonstrate that our method consistently outperforms ACR in all scenarios.

5 Related Work

Long-tailed learning (LTL). Early strategies tackling LTL involve two aspects: resampling and reweighting. Resampling methods [9, 5, 8, 54] either undersample majority classes or oversample minority classes, which may result in information loss or overfitting. Reweighting methods [52, 16, 2, 12] assign different weights for each class or training sample. BBN [72] and Decoupling [31] claim that re-balancing can negatively impact representation. They propose a two-branch structure or a two-stage paradigm to address it. Logit adjustment methods [7, 44] learn larger margins for minority classes by obtaining optimal Bayesian classifiers. Recently, several methods [30, 15, 73, 20, 57] have been proposed to improve the representation learning based on supervised contrastive learning [32].

Long-tailed semi-supervised learning (LTSSL). Most semi-supervised learning (SSL) methods use unlabeled data by assigning pseudo-labels to unlabeled data [37, 6, 56, 71, 10] or aligning predictions of different views of the input by consistency regularization [59]. PAWS [4] leverages self-supervised representations derived from unlabeled data, and RoPAWS [46] further refines the model predictions using labeled data through kernel density estimation. However, most of these works assume a balanced class distribution of labeled and unlabeled data, which may be violated in real-world applications.

Recently, LTSSL has gained considerable attention due to its applicability in numerous real-life scenarios. Recent works mitigate pseudo-labels bias by distribution alignment or label refinement [33, 63, 69]. Some others focus on balanced classifier training to overcome long-tailed label distribution [38, 22, 66]. Regrettably, these methods simply assume an identical long-tailed distribution for labeled and unlabeled data, which may still be unrealistic. Considering the unknown unlabeled data distribution, which can be mismatched with the labeled distribution, DASO [48] mixes the outputs of linear and semantic classifiers to improve the quality of pseudo-labels. ACR [64] and CPE [43] refine consistency regularization or train multiple expert branches based on predefined anchor distributions. However, how to improve representation learning in LTSSL is ignored in most existing works.

 









20

1 40

1 60

1 80

1

100

1































5

1

10

1

15

1

20

1





















6 Conclusion

This paper presents a probabilistic framework that unifies many recent methods in long-tail learning. Our framework is equivalent to supervised contrastive learning when approximating the classconditional function using the Gaussian kernel. We further extend the contrastive learning objective to LTSSL based on continuous pseudo-labels to improve the learned representations. We utilize both reliable pseudo-labels generated by the model and smoothed pseudo-labels propagated from nearby samples to mitigate confirmation bias. Extensive experiments demonstrate that our proposed method achieves state-of-the-art performance in all settings. We hope that our work can motivate more research for LTSSL from the perspective of representation learning.

Acknowledgments and Disclosure of Funding

This work was supported by the National Science Foundation of China (62206049, 62225602), and the Big Data Computing Center of Southeast University. We would like to thank anonymous reviewers for their constructive suggestions.

References

Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. International Conference on Learning Representations , 2017.

Shaden Alshammari, Yu-Xiong Wang, Deva Ramanan, and Shu Kong. Long-tailed recognition via weight balancing. In IEEE Conference on Computer Vision and Pattern Recognition , pages 6897-6907, 2022.

Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In International Joint Conference on Neural Networks (IJCNN) , pages 1-8, 2020.

Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised learning of visual features by non-parametrically predicting view assignments with support samples. In IEEE Conference on Computer Vision and Pattern Recognition , pages 8443-8452, 2021.

Colin Bellinger, Roberto Corizzo, and Nathalie Japkowicz. Calibrated resampling for imbalanced and long-tails in deep learning. In Discovery Science: 24th International Conference, DS 2021, Halifax, NS, Canada, October 11-13, 2021, Proceedings 24 , pages 242-252. Springer, 2021.

David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. ArXiv Preprint ArXiv:1911.09785 , 2019.

Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. In Advances in Neural Information Processing Systems , volume 32, pages 1567-1578, 2019.

Nadine Chang, Zhiding Yu, Yu-Xiong Wang, Animashree Anandkumar, Sanja Fidler, and Jose M Alvarez. Image-level or object-level? a tale of two resampling strategies for long-tailed detection. In International Conference on Machine Learning , pages 1463-1472. PMLR, 2021.

Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of Artificial Intelligence Research , 16:321-357, 2002.

Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. Softmatch: Addressing the quantity-quality trade-off in semi-supervised learning. ArXiv Preprint ArXiv:2301.10921 , 2023.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning , volume 119, pages 1597-1607, 2020.

Xiaohua Chen, Yucan Zhou, Dayan Wu, Chule Yang, Bo Li, Qinghua Hu, and Weiping Wang. Area: adaptive reweighting via effective area for long-tailed classification. In IEEE International Conference on Computer Vision , pages 19277-19287, 2023.

Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 215-223. JMLR Workshop and Conference Proceedings, 2011.

Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Advances in Neural Information Processing Systems , 2020.

Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In IEEE International Conference on Computer Vision , pages 715-724, 2021.

Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In IEEE Conference on Computer Vision and Pattern Recognition , pages 9268-9277, 2019.

AP Dempter. Maximum likelihood from incomplete data via the em algorithm. Journal of Royal Statistical Society , 39:1-22, 1977.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009.

Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. ArXiv Preprint ArXiv:1708.04552 , 2017.

Chaoqun Du, Yulin Wang, Shiji Song, and Gao Huang. Probabilistic contrastive learning for long-tailed visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024.

Marthinus Christoffel Du Plessis and Masashi Sugiyama. Semi-supervised learning of class balance under class-prior change by distribution matching. Neural Networks , 50:110-119, 2014.

Yue Fan, Dengxin Dai, Anna Kukleva, and Bernt Schiele. Cossl: Co-learning of representation and classifier for imbalanced semi-supervised learning. In IEEE Conference on Computer Vision and Pattern Recognition , pages 14574-14584, 2022.

Kai Gan and Tong Wei. Erasing the bias: Fine-tuning foundation models for semi-supervised learning. In Proceedings of the 41st International Conference on Machine Learning , pages 14453-14470, 2024.

Kai Gan, Tong Wei, and Min-Ling Zhang. Boosting consistency in dual training for long-tailed semisupervised learning. arXiv preprint arXiv:2406.13187 , 2024.

Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. A unified view of label shift estimation. Advances in Neural Information Processing Systems , 33:3290-3300, 2020.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning , pages 1321-1330. PMLR, 2017.

Boran Han. Wrapped cauchy distributed angular softmax for long-tailed visual recognition. In International Conference on Machine Learning , pages 12368-12388. PMLR, 2023.

Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang. Disentangling label distribution for long-tailed visual recognition. In IEEE Conference on Computer Vision and Pattern Recognition , pages 6626-6636, June 2021.

Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What makes imagenet good for transfer learning? ArXiv Preprint ArXiv:1608.08614 , 2016.

Bingyi Kang, Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. In International Conference on Learning Representations , 2020.

Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations , 2020.

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems , 33:18661-18673, 2020.

Jaehyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin. Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning. In Advances in Neural Information Processing Systems , 2020.

Takumi Kobayashi. T-vmf similarity for regularizing intra-class feature distribution. In IEEE Conference on Computer Vision and Pattern Recognition , pages 6616-6625, 2021.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energy-based learning. Predicting Structured Data , 1(0), 2006.

Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop On Challenges In Representation Learning, ICML , 2013.

Hyuck Lee, Seungjae Shin, and Heeyoung Kim. Abc: Auxiliary balanced classifier for class-imbalanced semi-supervised learning. Advances in Neural Information Processing Systems , 34:7082-7094, 2021.

Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems , 33:21464-21475, 2020.

Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In IEEE Conference on Computer Vision and Pattern Recognition , pages 2537-2546, 2019.

Charlotte Loh, Rumen Dangovski, Shivchander Sudalairaj, Seungwook Han, Ligong Han, Leonid Karlinsky, Marin Soljacic, and Akash Srivastava. On the importance of calibration in semi-supervised learning. ArXiv Preprint ArXiv:2210.04783 , 2022.

Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. ArXiv Preprint ArXiv:1608.03983 , 2016.

Chengcheng Ma, Ismail Elezi, Jiankang Deng, Weiming Dong, and Changsheng Xu. Three heads are better than one: Complementary experts for long-tailed semi-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 14229-14237, 2024.

Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In International Conference on Learning Representations , 2021.

Shambhavi Mishra, Balamurali Murugesan, Ismail Ben Ayed, Marco Pedersoli, and Jose Dolz. Do not trust what you trust: Miscalibration in semi-supervised learning. ArXiv Preprint ArXiv:2403.15567 , 2024.

Sangwoo Mo, Jong-Chyi Su, Chih-Yao Ma, Mido Assran, Ishan Misra, Licheng Yu, and Sean Bell. Ropaws: Robust semi-supervised representation learning from uncurated data. ArXiv Preprint ArXiv:2302.14483 , 2023.

Yurii Nesterov. A method of solving a convex programming problem with convergence rate o (1 /k 2 ) . Doklady Akademii Nauk SSSR , 269:543, 1983.

Youngtaek Oh, Dong-Jin Kim, and In So Kweon. Daso: Distribution-aware semantics-oriented pseudolabel for imbalanced semi-supervised learning. In IEEE Conference on Computer Vision and Pattern Recognition , pages 9786-9796, 2022.

Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr Computational Mathematics and Mathematical Physics , 4:1-17, 1964.

Harsh Rangwani, Sumukh K Aithal, Mayank Mishra, et al. Escaping saddle points for effective generalization on class-imbalanced data. Advances in Neural Information Processing Systems , 35:22791-22805, 2022.

Jiawei Ren, Cunjun Yu, shunan sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and hongsheng Li. Balanced meta-softmax for long-tailed visual recognition. In Advances in Neural Information Processing Systems , volume 33, pages 4175-4186, 2020.

Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning , pages 4334-4343. PMLR, 2018.

Marco Saerens, Patrice Latinne, and Christine Decaestecker. Adjusting the outputs of a classifier to new a priori probabilities: a simple procedure. Neural Computation , 14(1):21-41, 2002.

Jiang-Xin Shi, Tong Wei, Yuke Xiang, and Yu-Feng Li. How re-sampling helps for long-tail learning? Advances in Neural Information Processing Systems , 36:75669-75687, 2023.

Jiang-Xin Shi, Tong Wei, Zhi Zhou, Jie-Jing Shao, Xin-Yan Han, and Yu-Feng Li. Long-tail learning with foundation model: Heavy fine-tuning hurts. In Proceedings of the 41st International Conference on Machine Learning , pages 45014-45039, 2024.

Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Advances in Neural Information Processing Systems , 2020.

Min-Kook Suh and Seung-Woo Seo. Long-tailed recognition by mutual information maximization between latent features and ground-truth labels. In International Conference on Machine Learning , pages 3277032782. PMLR, 2023.

Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International Conference on Machine Learning , pages 1139-1147. PMLR, 2013.

Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems , volume 30, pages 1195-1204, 2017.

Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. ArXiv Preprint Physics/0004057 , 2000.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research , 9(11), 2008.

Xudong Wang, Zhirong Wu, Long Lian, and Stella X Yu. Debiased learning from naturally imbalanced pseudo-labels for zero-shot and semi-supervised learning. ArXiv Preprint ArXiv:2201.01490 , 2022.

Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang. Crest: A class-rebalancing selftraining framework for imbalanced semi-supervised learning. In IEEE Conference on Computer Vision and Pattern Recognition , 2021.

Tong Wei and Kai Gan. Towards realistic long-tailed semi-supervised learning: Consistency is all you need. In IEEE Conference on Computer Vision and Pattern Recognition , pages 3469-3478, 2023.

Tong Wei and Yu-Feng Li. Does tail label help for large-scale multi-label learning? IEEE transactions on neural networks and learning systems , 31(7):2315-2324, 2019.

Tong Wei, Qian-Yu Liu, Jiang-Xin Shi, Wei-Wei Tu, and Lan-Zhe Guo. Transfer and share: semi-supervised learning from long-tailed data. Machine Learning , pages 1-18, 2022.

Tong Wei, Zhen Mao, Zi-Hao Zhou, Yuanyu Wan, and Min-Ling Zhang. Learning label shift correction for test-agnostic long-tailed recognition. In Proceedings of the 41st International Conference on Machine Learning , pages 52611-52631, 2024.

Tong Wei, Jiang-Xin Shi, Wei-Wei Tu, and Yu-Feng Li. Robust long-tailed learning under label noise. ArXiv Preprint ArXiv:2108.11569 , 2021.

Zhuoran Yu, Yin Li, and Yong Jae Lee. Inpl: Pseudo-labeling the inliers first for imbalanced semisupervised learning. ArXiv Preprint ArXiv:2303.07269 , 2023.

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference , pages 87.1-87.12, 2016.

Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems , 34:18408-18419, 2021.

Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In IEEE Conference on Computer Vision and Pattern Recognition , pages 9719-9728, 2020.

Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, and Yu-Gang Jiang. Balanced contrastive learning for long-tailed visual recognition. In IEEE Conference on Computer Vision and Pattern Recognition , pages 6908-6917, 2022.

A Comparison with Supervised Contrastive Learning

As we have derived in Eq. (10) and use the Gaussian kernel density estimation in Eq. (11), if we simply assume the data are class-balanced, it simplifies to:

The loss of supervised contrastive learning has two forms, i.e., ̂ L in scl and ̂ L out scl , which is distinguished by the position of summation over positive samples at the log( · ) . Thus, we get:

which is consistent with the original paper's derivation.

B Analysis of Existing Long-Tail Learning Methods

As we have derived before, here we dive deep into the analysis of existing methods and demonstrate that they all belong to our unified framework.

B.1 Discussion of Gaussian kernel estimation

Balanced contrastive learning [73] (BCL) was proposed to solve long-tailed problems with improved supervised contrastive learning. BCL involves two key techniques: class averaging and class complement. BCL averages out the contributions of different classes in the denominator to ensure class equal distribution, meanwhile, it takes nonlinear mapping of the classifier parameters to form a learnable class center to ensure that every class has at least one sample in a mini-batch:

However, BCL ignores the problem of the original long-tailed distribution in the training dataset, necessitating a reweighting operation. Let π denote the class prior, we have:

Gaussian mixture likelihood loss [57] (GML) initiates its approach from the concept of mutual information, employing the Gaussian kernel. GML integrates contrastive learning with logit adjustment to enhance its performance.

It proposes a class-wise queue Q = ∪ C k =1 Q k to ensure a balanced class occurrence in a mini-batch. However, it does not propose a unified framework, and the understanding is not deep enough.

Some other methods: In this section, we compare some other methods that use contrastive learning and analyze their mistakes.

K-positive contrastive learning [30] (KCL) is based on supervised contrastive learning, using K samples of the same class in the molecule to ensure balanced feature space. Putting in our unified framework, we can obtain the following:

Compared with the above, it regrettably still uses the same unprocessed denominator of SCL and cannot ensure that each mini-batch contains an equal number of samples from each class, nor that each class contributes equally, rendering it suboptimal for long-tailed learning. In addition, Eq. (28) is equivalent to the resampling technique.

Parametric contrastive learning [15] (PaCo) introduces a set of parametric class-wise learnable centers and uses adjustable parameters α for loss with respect to them. Our framework is used to derive its original form. First, we can obtain:

where β is a fixed coefficient. Then, we can derive the PaCo loss as follows.

where α = β P ( Y = y ) (1 -β ) |B y | . PaCo explicitly uses the parametric class center to ensure balanced class occurrence in a mini-batch. However, It ignores the class-equal contribution in loss computation, which can still be suboptimal.

Probabilistic contrastive learning [20] (Proco) simply assumes that the normalized features in contrastive learning follows a mixture of von Mises-Fisher (vMF) distributions on a unit ball, its probability density function has the following form:

where parameters ( µ y , ρ y ) need to be estimated. The advantage is that Proco can estimate ( µ y , ρ y ) using an online mini-batch, such that it can be derived as a closed form of expected contrastive loss. Despite the assumed vMF distribution, it still uses Gaussian kernel estimation:

where λ ( z x , y ) represents a fixed function related to z x , µ y , ρ y . Thus, the loss objective of Proco is:

However, the assumed distribution of Proco stills needs to be estimated ( µ y , ρ y ) using EMA of different batches, which is essentially a similar approach to the momentum queue used in GML. There still exist problems of inconsistent distribution of z in different mini-batches, and the strong assumption about P ( z | Y = y ) which may not follow the vMF distribution.

B.2 Discussion of explicitly assigning a specified distribution

Previous work mainly focused on P ( z | Y = y ) through modeling cos θ y = µ ⊤ y z ∥ µ y ∥ ∥ z ∥ .

T-vMF [34] models cos θ y as vMF distribution:

Due to the inherent properties of the exponential function, the posterior quickly converges to 0, despite a large ∥ ∥ z -µ y ∥ ∥ . Such a compact measuring function might hamper model training, since tailed samples hardly enjoy back-propagation due to vanishing gradient. To overcome this problem, T-vMF introduces a family of modeling methods as follows:

where s q ( ∥ ∥ z -µ y ∥ ∥ , ρ y ) = [ 1 -(1 -q ) 1 2 ρ y ∥ ∥ z -µ y ∥ ∥ ] 1 1 -q . Technically, T-vMF models ̂ P t ( Y = y | z ) as follows:

Thus, the loss objective of T-vMF is:

WCDAS [27] The accuracy of the posterior approximation is a crucial factor influencing the method's performance. Unlike t-vMF, which directly specifies the P ( z | Y = y ) with fixed parameters, WCDAS seeks an optimal parametric probability density function of P ( z | Y = y ) .

Modeling cos θ y as the Wrapped Cauchy distribution with trainable parametric ϑ = [ ϑ 1 , . . . , ϑ C ] , WCDAS models ̂ P t ( Y = y | z ) as follows:

Thus, the loss objective of WCDAS is:

C Mathematical Notations

To ensure clarity and precision throughout this paper, we provide a comprehensive list and definitions of the key mathematical symbols and terms used in this section. Each symbol is defined with its specific meaning and context to ensure consistency and accuracy across the document.

D Illustration of The Proposed Algorithm

D.1 Illustration of The Overall Proposed Algorithm

CCL consists of two parts: the classification part and the contrastive learning part. The classification part uses logit rectification of the classifier by class prior estimated with a dual-branch. For the contrastive learning part, the energy score is used to select reliable unlabeled data which are merged with labeled data for continuous contrastive loss to ensure calibration. Besides, information of labeled data and unlabeled are used in a decoupled manner while maintaining the constraints of aligning feature in the contrastive learning space, thereby forming a smoothed contrastive loss.

Continous

Dual-branch

reliablePL

Labeleddata

classifier

'ce

Energy Mask

ju

x

Z

g

D

Strong/Weak

EMA

Dataaugmentation

M

A"

Feature

A”

Feature

LogitFusion

spl

Label

Extractor

Extractor

nJ

Propagation

bal

π

Continous

Unlabeleddata

smoothed PL

bal

ProjectionHead

—> Forward of x?

→ Forward of A"

—→ Forward of A

.>Logit adjustment

Algorithm 1 Continous Contrastive Learning (CCL)

Input: labeled dataset and unlabeled dataset, standard branch f s and balanced branch f b , projection head g , class prior of labeled dataset π l , estimated unlabeled dataset class distribution ̂ π u , number of iterations in each epoch T , scaling parameter τ .

Require: Weak augmentation A w ( · ) , strong augmentation A s ( · ) , loss weight coefficients λ 1 , λ 2 . for t = 1 to T do

# Balanced classifier training with estimated class prior

Calculate classification loss ̂ L cls

Update estimated class distribution ̂ π u via EMA by energy score selection

# Continuous contrastive loss with reliable pseudo-labels

Merge reliable unlabeled data selected based on energy score with labeled data to construct B Calculate loss ̂ L rpl via Eq. (17) with B

# Continuous contrastive loss with smoothed pseudo-labels

Calculate G using unlabeled data

Compute posterior ̂ P ( Y | A w ( x u )) and ̂ P ( Y | A s ( x u )) using Eq. (21) 3

Calculate consistency regularization loss ̂ L spl via Eq. (18)

# Total Objective

Update f s and f b and g based on ∇L total using SGD end for

D.2 Illustration of Reliable Pseudo-labels and Smoothed Pseudo-labels

Continous

Labeleddata

Continous

smoothed PL

reliablePL

x

Construction

Graph

Z

Energy

score

E(x)

(x)

Du

Strong/Weak

Logit

u

Dataaugmentation

Fusion

W

Label

Feature

Au

Feature

Propagation

Extractor

Extractor

Z

S

x

Unlabeleddata

ProjectionHead

—>Forward of x/

—→Forward of A

Figure 4: Illustration of reliable pseudo-labels and smoothed pseudo-labels in CCL. To generalize the framework in Section 2 to LTSSL, the main challenge is unknown P u ( Y = y | x u ) , where x u denotes a sample in the unlabeled dataset. We first approximate P u ( Y = y | x u ) using the output of the calibrated and integrated classifier and use energy score to filter out reliable unlabeled data, ensuring the model's calibration, which constitutes the reliable pseudo-labels subset. Furthermore, we can also estimate the unknown P u ( Y = y | x u ) by leveraging the smoothness assumption. Specifically, we construct smoothed pseudo-labels by propagating labels from nearby samples using the Gaussian kernel density estimation.

E Pseudo Code of The Proposed Algorithm

Algorithm 1 summarizes the whole framework of the proposed CCL, which is clearly divided into three components: balanced classifier, continuous contrastive learning with reliable and smoothed pseudo-labels, respectively.

F Experimental Setup

Training datasets. Our experimental analysis uses a variety of commonly adopted SSL datasets, including CIFAR10-LT [35], CIFAR100-LT [35], STL10-LT [13], and ImageNet-127 [22] in various ratios of class imbalance γ and various ratios of the amount of labeled data η . To create imbalanced versions of these datasets, we consider the long-tailed imbalance where the frequency of data points decreases exponentially from the largest to the smallest class, that is, the number of samples in class c is N c = N 1 × γ -c -1 C -1 for labeled data and M c = M 1 × γ -c -1 C -1 for unlabeled data. We use Cutout [19] and Randaugment [14] for strong augmentation on unlabeled data, and we use SimAugment [11] on labeled data for continuous contrastive loss with smoothed pseudo-labels. Like recent LTSSL works, we consider three class distribution patterns for unlabeled data, namely, consistent , uniform , and reversed settings.

CIFAR10-LT : Following ACR [64], we conduct experiments with all comparison methods in settings where N 1 = 500 , M 1 = 4000 and N 1 = 1500 , M 1 = 3000 . We adopt imbalance ratios of γ l = γ u = 100 and γ l = γ u = 150 for consistent settings, while for uniform and reversed settings, we use γ l = 100 , γ u = 1 and γ l = 100 , γ u = 1 / 100 , respectively.

CIFAR100-LT : Like CIFAR10-LT, we perform experiments in configurations where N 1 = 50 , M 1 = 400 and N 1 = 150 , M 1 = 300 . For the consistent settings, we use imbalance ratios of γ l = γ u = 10 and γ l = γ u = 20 . In contrast, for the uniform and reversed settings, we apply γ l = 10 , γ u = 1 and γ l = 10 , γ u = 1 / 10 , respectively.

STL10-LT : Given the absence of ground-truth labels for the unlabeled data of the STL10 dataset, we manage the experiments by adjusting the imbalance ratio of the labeled data. Following ACR, we consider the labeled imbalance ratio of γ l = 10 or γ l = 20 .

ImageNet-127 : ImageNet127 was first introduced in an earlier research [29] and utilized in LTSSL by CReST. This dataset consolidates the 1000 classes [18] from ImageNet into 127 classes, grouping them according to the WordNet hierarchy. For ImageNet-127, we follow the original setting in CoSSL [22] ( γ l = γ u ≈ 286) .

Implementation details. Our experimental configuration largely aligns with Fixmatch [56] and ACR [64]. Specifically, we apply the Wide ResNet-28-2 [70] architecture to implement our method on the CIFAR10-LT, CIFAR100-LT and STL10-LT datasets; and ResNet-50 on ImageNet-127. We adopt the common training paradigm that the network is trained with standard SGD [47, 49, 58] for 500 epochs, where each epoch consists of 500 mini-batches, and a batch size of 64 for both labeled and unlabeled data. We use a cosine learning rate decay [42] where the initial rate is 0.03, we set τ = 2 . 0 for logit adjustment on all datasets, except for ImageNet-127, where τ = 0 . 1 . We set the temperature T = 1 and the threshold ζ = -8 . 75 for the energy score following [69], and we set λ 1 = 0 . 7 , λ 2 = 1 . 0 on CIFAR10/100-LT and λ 1 = 0 . 7 , λ 2 = 1 . 5 on STL10-LT and ImageNet-127 datasets for the final loss. We set β = 0 . 2 in Eq. (21) for smoothed pseudo-labels loss. To show the effectiveness of our approach, we perform a comparative analysis with several existing LTSSL algorithms, including DARP [33], CReST [63], DASO [48], ABC [38], and TRAS [66]. We also consider the most popular LTSSL methods ACR [64] and CPE [43]. The performance evaluation of these methods is based on the top-1 accuracy metric on the test set. We present the mean and standard deviation of the results from three independent runs for each method. In addition, our method is implemented using the PyTorch library and experimented on an NVIDIA RTX A6000 (48 GB VRAM) with an Intel Platinum 8260 (CPU, 2.30GHz, 220 GB RAM).

G In-depth Analysis

G.1 Sensitive analysis of hyperparameters

As outlined in figure 5a, CCL is relatively robust to the fluctuation of β from 0.1 to 0.4. However, when β is set to 0, the propagation within unlabeled data is ignored, resulting in a performance decrease of about 0 . 9% . Thus, the necessity of using Eq. (21) is verified. In addition, figures 5b and 5c both demonstrate that CCL is robust to loss weighting coefficients λ 1 and λ 2 within a certain range.

However, it is worth noting that when λ 1 = 1 . 0 , the proposed continuous reliable pseudo-labels loss is ignored, resulting in performance degradation.

0.0

0.1

0.2

0.3

0.4

Value of

in SPL loss

50

51

52

53

54

55

Top-1 accuracy (%)

Sensitivity of CCL on

(a) Sensitivity of CCL on β

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Value of

1

in total loss

50

51

52

53

54

55

Top-1 accuracy (%)

Sensitivity of CCL on

1

(b) Sensitivity of CCL on

λ

1

0.7

0.8

0.9

1.0

1.1

1.2

1.3

Value of

2

in total loss

50

51

52

53

54

55

Top-1 accuracy (%)

Sensitivity of CCL on

2

(c) Sensitivity of CCL on

λ

2

G.2 Time and Space Complexity Analyses

In this section, we conduct analyses of the time and space complexity of the proposed method. Denote feature space dimension D , batch size B and the number of classes C , the time and space complexity of CCL can be seen in Table 8 and analysis details as follows.

For time complexity, calculating L rpl requires two main parts, calculating kernel similarities by multiplying two matrix of size B × D and D × B with complexity O ( B 2 D ) , and calculating E x ′ ∼ P ( ·| Y = y ) [ κ ( z x , z x ′ )] by multiplying two matrix of size B × B and B × C with complexity O ( B 2 C ) . Calculating L spl requires a further calculating part compared to L rpl : inverse matrix I -β G in Eq.(21) with complexity O ( B 3 ) by utilizing the fast singular value decomposition.

For space complexity, calculating two losses requires two additional storage spaces, first sample pairwise kernel similarity requires space with complexity O ( B 2 ) , and ̂ P ( Y | X u ) requires space with complexity O ( BC ) .

Generally, given B = 64 , C = 100 and D = 256 . Compared to the scale of model parameters, CCL adds negligible overhead relative to the neural network's computational cost when computing loss. We further report the averaged mini-batch training time with a single 3090 GPU and the GPU memory usage in Table 9 and Table 10. As seen from these tables, the training time and space consumptions of CCL are comparable to the existing state-of-the-art method ACR when CCL applies additional data augmentations to labeled data for representation learning.

G.3 Confusion matrix

Figure 6 presents the confusion matrix on the test set generated by CCL and ACR, which is calculated on the CIFAR10-LT dataset under γ l = γ u = 100 and γ l = γ u = 150 settings. As we can see in the top row of the figure, ACR often misclassifies the minority class '7' and '8' into the majority class '4' and '0', respectively. In comparison, CCL effectively mitigates this misclassification phenomenon by achieving an average improvement of 7.5 % . Similarly in the

(a) ACR (

γ

l

=

γ

u

=100)

(b) CCL (

γ

l

=

γ

u

=10)

(c) ACR (

γ

l

=

γ

u

=150)

(d) CCL (

γ

l

=

γ

u

=150)

0.88

0.00 0.01 0.00 0.00 0.000.00 0.010.06 0.03

0.92

0.00 0.01 0.01 0.01 0.00 0.00 0.00 0.04 0.01

0.94

100000000000000100000000000

260

20000000000000000000100000

-0.000.960.00 0.00 0.00 0.00 0.00 0.00 0.00 0.03

-0.00

0.97

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02

-0.00

0.97

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.03

-0.00

0.97

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.03

N-0.040.000.84

0.01 0.03 0.02 0.04 0.01 0.00 0.01

N-0.03 0.00 0.840.01 0.02 0.03 0.04 0.01 0.00 0.00

0.8

N-0.050.000.83

0.03 0.03 0.02 0.04 0.01 0.00 0.00

N-0.040.000.82

0.01 0.03 0.03 0.05 0.01 0.01 0.00

8°0

-0.02 0.00 0.030.760.03 0.09 0.03 0.01 0.01 0.02

-0.01 0.00 0.020.75

0.030.090.050.020.010.01

-0.02 0.00 0.030.77

0.03 0.08 0.03 0.01 0.01 0.01

-0.01 0.00 0.030.690.04 0.12 0.05 0.03 0.02 0.01

True Label

-0.01 0.00 0.03 0.030.89

0.01 0.02 0.02 0.00 0.00

-0.000.000.030.01

0.88

0.010.020.040.000.00

0.6

q

0.01 0.02 0.03 0.01 0.00

寸

-0.00 0.00 0.02 0.020.880.01 0.03 0.03 0.01 0.00

0.6

-0.01 0.000.030.180.040.69

0.010.040.000.01

-0.00 0.00 0.03 0.15 0.040.72

0.01 0.04 0.00 0.00

-0.010.000.040.220.040.63

0.010.040.00 0.00

-0.000.000.020.150.040.71

0.02 0.04 0.01 0.01

o-0.010.00 0.05 0.050.02 0.01

0.850.010.00 0.00

-0.01 0.00 0.04 0.03 0.02 0.010.880.01 0.00 0.00

0.4

o -0.01 0.00 0.06 0.06 0.04 0.01

0.82

0.00 0.01 0.00

-0.01 0.00 0.04 0.04 0.030.01

0.860.00 0.01 0.00

0.4

-0.02 0.00 0.02 0.06 0.07 0.06 0.010.74

0.01 0.01

-0.01 0.00 0.01 0.04 0.04 0.05 0.010.83

0.010.01

-0.02 0.00 0.03 0.08 0.09 0.05 0.000.72

0.00 0.01

-0.01 0.00 0.02 0.04 0.05 0.09 0.010.76 0.01 0.02

00-0.08 0.04 0.01 0.01 0.00 0.00 0.01 0.000.80

0.06

00-0.05 0.03 0.01 0.00 0.00 0.00 0.01 0.00 0.86

0.04

0.2

00 -0.15 0.06 0.02 0.01 0.00 0.00 0.01 0.000.73

0.02

900000000000000000000000900-0

0.04

0.2

000000000000000000000000000-

0.77

-0.010.18 0.00 0.00 0.00 0.00 0.00 0.01 0.02

0.77

-0.07

0.470.00 0.02 0.00 0.00 0.00 0.00 0.030.41

0

1

2

3

Predicted Label

5

6

7

8

6

0

1

2

3

Predicted Label

4

5

6

7

8

6

-0.0

0

1

2

3

Predicted Label

4

5

6

8

6

0

1

2

3

Predicted Label

4

5

6

7

8

6

0.0

second row, CCL achieved an extraordinarily high accuracy of 78 % for class '9', which shows a significant gain of 37 % compared to ACR. CCL also achieves higher overall accuracy.

G.4 Precision and recall

To conduct a more in-depth analysis of the effectiveness of pseudo-labels generated by the proposed dual-branch fusion approach, we calculated the precision and recall of the pseudo-labels assigned to unlabeled data by ACR and CCL on CIFAR10-LT and CIFAR100-LT datasets. Specifically, we use γ l = γ u = 100 and γ l = γ u = 150 settings on CIFAR10-LT dataset and all three consistent , uniform , reversed settings on CIFAR100-LT dataset and we grouped the results of CIFAR100 into 10 categories, each category containing 10 classes, since CIFAR100 comprises 100 classes. As can be seen in figure 8, CCL achieves significantly improved precision of pseudo-labels for tailed classes '9' and '10' on CIFAR10 dataset, while also achieving better recall for head classes. Similarly in Figure 7, CCL achieves overall better precision and recall compared to ACR regardless of the distribution mismatch scenario. It clearly shows that the pseudo-labels generated by CCL are more capable of alleviating the confirmation bias of tailed classes without sacrificing the performance of head classes.

G.5 Visualization

Furthermore, we employ the t-distributed stochastic neighbor embedding (t-SNE) [61] to visualize the representations learned by the CCL method and contrast them with those from the previous ACR method. The comparative results on the test set, under consistent settings, are depicted in Figure 9. The figure demonstrates that the representations derived from CCL provide more distinct classification boundaries.

H Limitation

Our paper examines existing long-tailed learning methods through the lens of information theoretical view, proposing a unified framework. However, we have not established theoretical proof for the convergence of features within this framework. In the future, we intend to provide further theoretical analysis from the perspective of neural collapse.

1

2

3

4

5

6

7

8

9

10

Class index

0.0

0.2

0.4

0.6

0.8

Precision / Recall(%)

(a) ACR on

γ

l

=

γ

u

= 10

1

2

3

4

5

6

7

8

9

10

Class index

0.0

0.2

0.4

0.6

0.8

Precision / Recall(%)

(b) ACR on

γ

l

= 10

, γ

u

= 1

1

2

3

4

5

6

7

8

9

10

Class index

0.0

0.2

0.4

0.6

0.8

Precision / Recall(%)

(c) ACR on

γ

l

= 10

, γ

u

= 1

/

10

1

2

3

4

5

6

7

8

9

10

Class index

0.0

0.2

0.4

0.6

0.8

Precision / Recall(%)

(d) CCL on

γ

l

=

γ

u

= 10

1

2

3

4

5

6

7

8

9

10

Class index

0.0

0.2

0.4

0.6

0.8

Precision / Recall(%)

(e) CCL on

γ

l

= 10

, γ

u

= 1

1

2

3

4

5

6

7

8

9

10

Class index

0.0

0.2

0.4

0.6

0.8

Precision / Recall(%)

(f) CCL on

γ

l

= 10

, γ

u

= 1

/

10

1

2

3

4

5

6

7

8

9

10

Class index

0.0

0.2

0.4

0.6

0.8

1.0

Precision / Recall(%)

(a) ACR (

γ

l

=

γ

u

=100)

1

2

3

4

5

6

7

8

9

10

Class index

0.0

0.2

0.4

0.6

0.8

1.0

Precision / Recall(%)

(b) CCL (

γ

l

=

γ

u

=100)

1

2

3

4

5

6

7

8

9

10

Class index

0.0

0.2

0.4

0.6

0.8

1.0

Precision / Recall(%)

(c) ACR (

γ

l

=

γ

u

=150)

1

2

3

4

5

6

7

8

9

10

Class index

0.0

0.2

0.4

0.6

0.8

1.0

Precision / Recall(%)

(d) CCL (

γ

l

=

γ

u

=150)

I Broader Impact

The positive impacts of this work are two-fold: 1) It enhances the fairness of the classifier in semisupervised learning, preventing potential biases in deep models, such as an unfair AI primarily serving the majority, which could lead to discrimination based on gender, race, or religion; 2) It enables the easy collection of larger image datasets without the need for mandatory class-balancing preprocessing. For example, in training classifiers for real-world natural image scenes using the proposed method, we do not need to consider whether the distribution of unlabeled data matches that of the labeled data or if every class in the labeled data has an equal number of samples. However, negative effects might occur if the proposed long-tailed semi-supervised classification technique is misused. In the wrong hands, this approach could be exploited for unethical purposes, such as targeting or identifying minority groups for detrimental reasons.

Classification

Boundaries

(a) ACR on

γ

l

=

γ

u

= 100

Classification

Boundaries

(b) CCL (ours) on

γ

l

=

γ

u

= 100

Classification

Boundaries

(c) ACR on

γ

l

=

γ

u

= 150

Classification

Boundaries

(d) CCL (ours) on

γ

l

=

γ

u

= 150

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly articulated our contributions at the end of the abstract and the introduction. Additionally, the research scope of this paper is introduced at the beginning of both the abstract and the introduction.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of our method are analyzed in the appendix of this paper.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All experimental results were obtained by implementing the proposed method and running it on the dataset. All results are reproducible.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have made the source code publicly available via the link described in the abstract.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In the appendix, we provide a detailed description of the experimental setup, including the creation of the dataset, hyperparameter settings, as well as the pseudocode for our method and diagrams of the model structure.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: In the experimental section, we present statistical results, including the mean and variance of accuracy, ECE calibration metrics, and line graphs estimating prior errors.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computational resources we used are detailed in the appendix.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our code adheres to the NeurIPS Code of Ethics and does not violate any ethical guidelines.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss societal impacts in the appendix.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper does not employ high-risk data or models, thus posing no such risks.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The assets from several recent works that we utilized are all cited in our paper.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects

Peer reviews:
comments: SUMMARY:
The paper proposes a novel method to address the long-tail problem in semi-supervised learning by leveraging continuous contrastive learning on both labeled and unlabeled samples to improve model performance, particularly for minority classes.

STRENGTHS:
- Novelty: The paper introduces a novel approach to integrate contrastive learning with unlabeled samples into SSL frameworks, which is relatively unexplored in existing literature.
- Technical Soundness: The theoretical formulation of learning good representations and the unified framework of many existing long-tail learning methods are well-explained and logically sound.
- Experiments: The experiments are comprehensive, covering various imbalanced ratios and labeled ratios on CIFAR-10-LT and CIFAR-100-LT datasets. The results are statistically significant and show clear improvements over baseline methods.
- Clarity: The paper is well-structured and clearly written. The introduction, methodology, and experimental sections are easy to follow, and the figures and tables effectively illustrate the results.

WEAKNESSES:
- In line 189, it is unclear why selecting pseudo-labels using energy score can ensure better model calibration.

QUESTIONS:
A minor suggestion: the paper can include some demonstrations of unlabeled images with their associated continuous pseudo-labels, including both reliable and smoothed versions. This can help readers identify cases in which the model tends to make mistakes.

LIMITATIONS:
The authors have adequately addressed the limitations and potential negative societal impact of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-02

comments: SUMMARY:
The author proposes a probabilistic framework that unifies many recent proposals in long-tail learning. Specifically, for long-tailed semi-supervised learning,  a continuous contrastive learning method based on reliable and smoothed pseudo-labels to address confirmation bias and improve the quality of learned representations is proposed. The experiments demonstrate their effectiveness.

STRENGTHS:
1) The author notices an important problem that existing approaches often assume the equivalence of the unlabeled data distribution with the labeled data, which is more practical. 

2) The proposed probabilistic framework which unifies many recent proposals in long-tail learning is interesting.

3) The paper is well-written and easy to follow.

WEAKNESSES:
1) The author should discuss explicitly how the proposed method solves the unlabelled data  for diverse label distributions of unlabeled data and why other methods do not. 

2) How to interpret " Furthermore, these methods primarily focus on correcting model outputs without delving into the role of representation learning in improving performance."  line 33-34. 

3) The author may give an analysis of the relationship between  reliable pseudo-labels and smoothed pseudo-labels and how they play a different role in semi-supervised learning.

4) The time and space complexity should be analyzed in experiments between the proposed method and other methods.

QUESTIONS:
see weakness

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-04

comments: SUMMARY:
This paper proposes a novel contrastive learning method for long-tailed semi-supervised learning (LTSSL). The method is motivated by variational information bottleneck for learning good representations and extends to unlabeled data using continuous pseudo-labels. This paper showcases strong empirical results on multiple LTSSL datasets and label distributions.

STRENGTHS:
1.The proposed method is well-motivated. Previous works in LTSSL generally make contributions in learning balanced classifiers, while this paper examines representation learning.
2.This paper summarizes many existing methods for long-tail learning in a unified framework. It is interesting to show that the equivalence between supervised contrastive learning and Gaussian kernel estimation.
3.This paper proposes a new LTSSL method by optimizing the contrastive loss using two types of continuous pseudo-labels. The proposed method is novel in LTSSL.
4.Extensive experiments showcase that the proposed method achieves best results on all datasets. The paper also provides extensive in-depth studies to understand the effectiveness of the method.

WEAKNESSES:
1.In Table 1, the paper states that long-tail learning methods need to avoid none class samples in each min-batch during training. However, it is unclear how this issue is tackled in the proposed method CCL.
2.The training time of the proposed method is not reported. It is suggested to compare the training time with ACR.

QUESTIONS:
Please see Weaknesses.

LIMITATIONS:
The authors have adequately addressed the limitations and potential negative societal impact of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

comments: SUMMARY:
This paper tackles the long-tailed semi-supervised learning problem. It first reviews recent works with a novel probabilistic framework. Based on this, it proposes a continuous contrastive learning method, CCL, to extend the framework to unlabeled data with pseudo-labels. Experiments show that it outperforms all recent works on the popular long-tailed semi-supervised learning benchmarks.

STRENGTHS:
Overall, this paper is good.

1. With a framework from information theory perspective, it unified different approaches on the long-tailed learning topic.

2. It systematically studied the key ideas to tackle the long-tailed learning and semi-supervised learning tasks.

3. By proposing CCL, it unifies different ideas from long-tailed learning and semi-supervised learning, and the result is an effective approach to solve the LTSSL problem.

WEAKNESSES:
I only have one question:

In Eq. 22, it mentioned L_{cls}, which is also mentioned in Line 171. Is that the same as P^{cls} in Eq. 15?

QUESTIONS:
See weaknesses

In Eq. 22, it mentioned L_{cls}, which is also mentioned in Line 171. Is that the same as P^{cls} in Eq. 15?

LIMITATIONS:
Limitations and potential impacts are addressed in Appendix,
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-24