Continuous Contrastive Learning for Long-Tailed Semi-Supervised Recognition

Authors: zi-hao  zhou
siyuan  fang
zi-jing  zhou
tong  wei
yuanyu  wan
min-ling  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 16:18:26

Abstract: Long-tailed semi-supervised learning poses a significant challenge in training models with limited labeled data exhibiting a long-tailed label distribution. Current state-of-the-art LTSSL approaches heavily rely on high-quality pseudo-labels for large-scale unlabeled data. However, these methods often neglect the impact of representations learned by the neural network and struggle with real-world unlabeled data, which typically follows a different distribution than labeled data. This paper introduces a novel probabilistic framework that unifies various recent proposals in long-tail learning. Our framework derives the class-balanced contrastive loss through Gaussian kernel density estimation. We introduce a continuous contrastive learning method, CCL, extending our framework to unlabeled data using *reliable* and *smoothed* pseudo-labels. By progressively estimating the underlying label distribution and optimizing its alignment with model predictions, we tackle the diverse distribution of unlabeled data in real-world scenarios. Extensive experiments across multiple datasets with varying unlabeled data distributions demonstrate that CCL consistently outperforms prior state-of-the-art methods, achieving over 4% improvement on the ImageNet-127 dataset. The supplementary material includes the source code for reproducibility.

Main paper: 
 introduction: 
 Semi-supervised learning (SSL) serves as a powerful approach for improving the generalization capabilities of deep neural networks (DNNs) in scenarios where labeled data is scarce [37, 59, 6, 23] . The core concept of SSL methods typically involves assigning pseudo-labels to unlabeled data and utilizing those with high confidence for model training [56, 71, 10] . However, many existing SSL algorithms presuppose a balanced label distribution across both labeled and unlabeled datasets. In real-world applications, datasets commonly exhibit a long-tailed label distribution [65, 28, 50, 67, 55] , leading to biased pseudo-label generation favoring majority classes [40, 3, 68, 24] . This discrepancy challenges the effectiveness of SSL algorithms in addressing real-world datasets.The exploration of long-tailed semi-supervised learning (LTSSL) has gained momentum to address the challenge of biased pseudo-label distribution arising from class imbalance in labeled and unlabeled data. Recent LTSSL approaches propose compensating for the learning of minority classes by distribution alignment [33, 63] , data rebalancing [22, 38] , and logit adjustment [64, 43] to rectify the pseudo-label distribution. However, existing approaches often assume the equivalence of the unlabeled data distribution with the labeled data or rely on predefined anchor distributions to estimate the unlabeled data distribution [64, 43] . Furthermore, these methods primarily focus on correcting model outputs without delving into the role of representation learning in improving performance. This paper explicitly introduces an approach to obtain effective representations for long-tail learning by adopting an information-theoretic view of DNNs. We present a probabilistic framework that utilizes the deep variational information bottleneck method [1] to learn good representations and demonstrate its unification of recent long-tail learning proposals, such as logit adjustment [44] and balanced softmax [51] , through approximating the density of class-conditional distribution in different ways. Specifically, our framework encompasses class-balanced supervised contrastive learning [73, 15] via Gaussian kernel density estimation. We extend this framework to address LTSSL by adapting the supervised contrastive loss to unlabeled data using "continuous pseudo-labels", derived from model predictions and propagated labels, to mitigate confirmation bias. To account for varying label distribution of unlabeled data, we progressively estimate the label distribution through a moving average and adjust model predictions to align with the estimated distribution.In summary, our contributions are as follows: 1. We propose a probabilistic framework which unifies many recent proposals in long-tail learning.Specifically, popular class-balanced contrastive learning methods can be seen as special cases of our framework when approximating the density using a Gaussian kernel. 2. We generalize the proposed framework to LTSSL and present a continuous contrastive learning method based on reliable and smoothed pseudo-labels to address confirmation bias and improve the quality of learned representations. 3. We conduct extensive experiments across several LTSSL datasets with diverse label distributions of unlabeled data. The results show that our proposal substantially outperforms previous state-ofthe-art methods. 
a probabilistic framework for long-tail learning: 
 In this section, we first introduce a general framework for learning good representations. Then, we expand this framework to long-tail learning and illustrate how recent proposals can be regarded as specific instances of our framework through three ways for density approximation.Problem setup of long-tail learning. We consider a C-class classification problem with instance space X and target space Y = {1, . . . , C}. Let P s and P t denote the source (training) and test distributions on (X , Y), respectively. We denote by P s and P t the corresponding probability density (or mass) functions. Given a training dataset {(x i , y i )} N i=1 , where x i ∈ R d is the training sample and y i is the ground-truth label.In this subsection, we rethink one of the most popular approaches to deal with representation learning, i.e., contrastive learning. We derive many recent proposals in this branch from an informationtheoretic view. Let Z denote the latent representation of X induced by the encoder enc(•) parameterized by Θ. From an information-theoretic view, an optimal representation Z is maximally informative about the target Y , and minimally "memorizes" X. The information bottleneck [60] adopts mutual information I(•) to measure information between two variables. Thus, optimal Z can be obtained by maximizing the following objective:EQUATIONwhere δ ≥ 0 is a tradeoff parameter. The variational information bottleneck [1] solves the above objective by variational inference. Based on the definition of I(•), we can rewrite Eq. (1) as:EQUATIONwhere we omit Θ for simplicity. Since P(y .EQUATIONSuppose we use an encoder of the form P(z | x) ∼ N (enc(x), ε 2 I) and P(z) ∼ N (0, I), the second term of Eq. ( 3) equals the KL divergence KL[P(z | x), P(z)]. Since P(z | x) and P(z) are normal distributions, it can be rewritten as:-δl( 1 2 ε 2 -1 -2 log ε) -δ∥enc(x)∥ 2, where l is the dimension of z. For a deterministic model, z is almost unique for each x, thus assuming ε is a small constant close to 0. By integrating out P(z | x) and discarding constant terms, maximizing Eq. ( 3) can be approximated by minimizing:EQUATIONIn the following of this paper, we denote the output of enc(x) as z (or z x for a particular sample x) for simplicity. Minimizing Eq. ( 4) is equivalent to minimizing the following objective in the distribution of test data on each x: -EQUATIONNotably, Eq. ( 5) can be seen as a general framework for learning good representations EQUATIONEQUATION)Throughout the paper, we use the notationP(x | Y = y) to represent either P s (x | Y = y) or P t (x | Y = y).In practice, ∥z∥ 2 can be omitted in optimization because normalization is commonly adopted in deep learning. In long-tail learning, minimizing Eq. ( 5) equals to minimizing:-EQUATIONAccording to Jensen's inequality, Eq. ( 7) attains its minimum value if and only ifP t (Y = k | x)P s (Y = k) ∝ P s (Y = k | x) for k ∈ [C].Hence, Eq. ( 7) can be replaced as follows:EQUATIONwhereP(Y = y | z) = Pt(Y =y|z)P(Y =y) k∈[C] Pt(Y =k|z)P(Y =k) and P(Y ) is an arbitrarily label distribution. Eq. ( 8) can be seen as an extension of sample reweighting [52] and logit adjustment [44] using probabilistic labels rather than discrete labels when P(Y ) is specified as P t (Y ) and P s (Y ), respectively. Besides, Eq. ( 8) presents a unified framework that consolidates existing long-tail learning methods by estimating P(z | Y = y) or P t (Y = y | z) in different ways. In the following, we discuss three ways to estimate these terms.Method 1: Explicitly specify P(z | Y = y) as a prior distribution such as vMF distribution [34] and Wrapped Cauchy Distribution [27] .Method 2: Approximate P(z | Y = y) using a learnable linear classifier. Let {w i , b i } Ci=1 denote the parameters of a linear layer, which is followed by a softmax to obtain the normalized probability:EQUATION)Table 1 : A unified view of popular long-tail learning methods from our framework. "-" means that this method does not involve this issue and "×" indicates that the method has not resolved the issue.Label distribution shift Mini-batch computation of Eq. ( 10)BALMS [51] Linear layer Reweighting -LA [44] Linear layer Logit adjustment -BCL [73] Gaussian kernel Reweighting Class-wise center GML [57] Gaussian kernel Logit adjustment Class-wise queue KCL [30] Gaussian kernel Balanced resampling × PaCo [15] Gaussian kernel Logit adjustment Class-wise center Proco [20] Gaussian kernel Logit adjustment Class-wise vMF distribution T-vMF [34] T-vMF distribution Logit adjustment -WCDAS [27] Wrapped Cauchy distribution Logit adjustment -Method 3: Approximate P(z | Y = y) via the Gaussian kernel. A new sample from class y should be closer to all samples within class y and away from samples from other classes. Using the expected similarity among all samples within the class to measure distance, we derive:EQUATIONwhere κ(•, •) represents the similarity between two samples, when we use Gaussian kernelκ(z x , z x ′ ) = exp(z x • z x ′ ) and approximate expectation through empirical batch B = ∪ k∈[C] B k , that is E x ′ ∼P(•|Y =y) [κ(z x , z x ′ )] ≈ 1 |By| x ′ ∈By exp(z x • z x ′ ),Eq. ( 10) can be instantiated as:EQUATIONInterestingly, we observe that Eq. ( 11) resembles class-balanced contrastive loss. In the appendix, we also show that the Gaussian kernel approximation is identical to conventional supervised contrastive learning if the training data are class-balanced.Notably, to ensure the computability of E x ′ ∼P(•|Y =y) [κ(z x , z x ′ )] in Eq. (10) , it is essential to ensure that samples are available from each class. Existing methods address this by class-wise queues, class-wise centers, or class-wise vMF distribution, details of which are provided in the appendix.Based on the above three density approximation methods, we find that many recent proposals in long-tail learning can be derived from our framework. In Table 1 , we summarize existing methods based on the way they estimate the density, tackle the training/test label shift, and guarantee the computation of Eq. ( 10) in mini-batch training. 
ccl: continuous contrastive learning: 
 In this section, we introduce the proposed algorithm CCL, which extends the class-balanced contrastive learning presented in Eq. ( 8) with Gaussian kernel estimation in Eq. (11) to LTSSL.Let P l and P u denote the joint distribution (X , Y) for labeled data and unlabeled data, respectively. We denote by P l and P u the corresponding probability density (or mass) functions. We possess a labeled dataset {(x l i , y l i )} N i=1 of size N and an unlabeled dataset {x u j } M j=1 of size M , where x l i , x u j ∈ R d . The proportion of labeled data from the entire dataset is η = N M +N . Denote the number of labeled data for class i as N i , we have N 1 ≥ N 2 ≥ . . . ≥ N C if the classes are sorted by cardinality in decreasing order. The imbalance ratio of labeled data is given by γ l = N1 N C , while the distribution of the label of the unlabeled data and its imbalance ratio γ u are unknown. The components of CCL include a feature extractor, two linear classifiers f s (•), f b (•) and a contrastive feature projection head g(•).We develop our method based on FixMatch [56] following previous works [48, 64] , and its objective is: L ssl = L l + L u , where L l is a traditional cross-entropy loss. For unlabeled data, the method operates by first generating pseudo-labels for unlabeled data using the model's predictions and selecting unlabeled data whose predicted maximum confidence is higher than a predefined threshold. The consistency regularizer L u is then applied to two views of each selected sample.Balanced FixMatch for LTSSL. First, since the labeled data follow a long-tailed distribution, which we denote as π l , L l needs to be adjusted by logit adjustment [44] via Eq. ( 8):EQUATION)Second, FixMatch is prone to fit wrong pseudo-labels with high predictive confidence during training [62, 3] . However, since the unlabeled data label distribution is inaccessible, pseudo-labels generated by the classifier may be sub-optimal for its adherence to a uniform distribution. By Bayes' theorem, with given estimated unlabeled data label distribution π u , the "post-adjusted" model outputs for sample x u can be formulated as:EQUATIONGiven pseudo-label y = arg max k∈[C] P u (Y = k | A w (x u ); f b ),where A w (•) denotes the weak data augmentation, L u is rewritten as:EQUATIONwhere M(•) denotes the sample mask to select reliable pseudo-labels. We progressively update π u using the exponential moving average (EMA) for each mini-batch byπ u y = (1 -α) π u y + α |B| x u ∈B P u (Y = y | x u ; f b ), where α is a momentum updating parameter and B denotes an unlabeled data subset. Directly using confidence selection can lead to a selected B with poor calibration due to model overconfidence [41, 45] . Thus, the energy score [36] is adopted to filter out reliable unlabeled data, which is defined as E(x) = -T • log k∈[C] e f k (x)/T , where T is the temperature and f (x) denotes the logits of x. We select reliable unlabeled data byM E (x u ) := I(E(x u ) ≤ ζ) using a predefined threshold ζ, and construct B = {x | x ∈ B u ∧ M E (x) ̸ = 0} for the estimation of π u .Dual-branches training. Based on the observation that class-balanced training can be harmful to representation learning, previous works [38, 64] have utilized another branch of the network for standard training. In contrast to the balanced branch, the standard branch, denoted as f s (•), optimizes the cross-entropy loss without employing logit adjustment to fit the original training data distribution. We fuse the predictions of balanced and standard branches to reduce the confirmation bias of pseudo-labels by:EQUATION)The rationale behind the equation is that the standard branch necessitates the elimination of imbalanced label prior and then compensates for unlabeled label prior when predicting pseudo-labels, which is achieved by defining π * = π u π l + π u . Overall, the classification loss L cls is the combination of losses for learning f s (•) and f b (•).Apart from the classification loss and consistency regularizer, we aim to improve the quality of representations by extending the framework presented in Section 2 to LTSSL. To achieve the adaptation of our framework, a primary obstacle must be addressed. The challenge arises from the unknown ground-truth label P u (Y = y | x) for unlabeled data, which results in the calculation of Eq. ( 8) infeasible. We propose to utilize the continuous pseudolabel P cls (Y = y | x u ) as derived from the classifier in Eq. ( 15). Furthermore, E x ′ ∼P(•|Y =y) [κ (z x , z x ′ )] in Eq. ( 10) can be extended to unlabeled data and approximated by an empirical data subset B:EQUATION)Plugging Eq. ( 16) into Eq. ( 10), we obtain the continuous pseudo-label P t (Y = y | x u ; B) for x u . Similar to Eq. ( 14), logit adjustment is used to handle label shift of unlabeled data by Bayes' theorem, and we can obtain:EQUATIONwhereP u (Y = y | x u ; B) = Pt(Y =y|x u ;B)• π u y k∈[C] Pt(Y =k|x u ;B)• π u k. So far, the generalized framework using continuous pseudo-labels for LTSSL is derived in Eq. ( 17), the critical issue is how to filter out a reliable unlabeled data subset B u such that the posterior estimation P cls (Y = y | x) in Eq. ( 16) is calibrated. Similarly, directly using confidence selection may lead to model overconfidence. To mitigate the confirmation bias in pseudo-labels produced by the learned classifier, we propose using the energy score for data selection to ensure model calibration [26] . Combining with labeled data, the loss L rpl is obtained withB = {x | x ∈ B l ∨ (x ∈ B u ∧ M E (x) ̸ = 0)}.To further mitigate the impact of inaccurate pseudo-labels P cls (Y = y | x u ), we derive a complementary contrastive loss with smoothed pseudo-labels. Specifically, we propose aligning the representations of two views of a sample by imposing the weak-strong consistency regularization:EQUATIONIn this part, we aim to derive P(Y = y | x u ) by propagating labels from nearby samples in the contrastive space. On the one hand, we take labeled data for B in Eq. ( 11) and construct the posterior for unlabeled data. Logit adjustment is employed for tackling label shift of unlabeled data:EQUATIONEq. ( 19) can be viewed as a process of propagating labels from labeled data to unlabeled data. On the other hand, we consider label propagation within unlabeled data, i.e., an unlabeled batch B u is used to estimate P(Y = y | x u ; B). Assuming there is a sufficient amount of unlabeled data, we have1 |B u | x u ∈B u P(Y = y | x u ; B u ) ≈ π uy , hence the posterior can be approximated as:EQUATION) Let P(Y | X; B) represent a matrix stacked by [P(Y = 1 | x), . . . , P(Y = C | x)]⊤ of x from B, we can rewrite Eq. ( 20) in the form of matrix multiplication:P (Y | X u ; B u ) = G • P (Y | X u ; B u ) ,where G is a similarity matrix and G ij = κ(zx i ,zx j )x j ∈Bu κ(zx i ,zx j ) . It can be interpreted that similar samples possess similar labels. By aggregating the predictions of labeled data and unlabeled data with a fixed hyperparameter β, we obtain:P (Y | X u ) = βG • P (Y | X u ) + (1 -β) P Y | X u ; B l ⇒ P (Y | X u ) = (1 -β)(I -βG) -1 • P Y | X u ; B l . (21)Subsequently, Eq. ( 21) can be plugged into Eq. ( 18) for calculating L spl . To sum up, the total objective of CCL is:EQUATIONwhere λ 1 and λ 2 are two hyperparameters. γ = γ l = γ u = 100 γ = γ l = γ u = 150 γ = γ l = γ u = 10 γ = γ l = γ u = 20 Algorithm N 1 = 500 N 1 = 1500 N 1 = 500 N 1 = 1500 N 1 = 50 N 1 = 150 N 1 = 50 N 1 = 150 M 1 = 4000 M 1 = 3000 M 1 = 4000 M 1 = 3000 M 1 = 400 M 1 = 300 M 1 = 400 M 1 =CIFAR10-LT (γ l ̸ = γ u ) STL10-LT (γ u =N/A) γ u = 1 (uniform) γ u = 1/100 (reversed) γ l = 10 γ l = 20 Algorithm N 1 = 500 N 1 = 1500 N 1 = 500 N 1 = 1500 N 1 = 150 N 1 = 450 N 1 = 150 N 1 = 450 M 1 = 4000 M 1 = 3000 M 1 = 4000 M 1 = 3000 M = 100k M = 100k M = 100k M = 100k 
experiments: 
 In this section, we conducted comprehensive experiments to verify the effectiveness of the proposed continuous contrastive learning method (CCL) on CIFAR10-LT, CIFAR100-LT, STL10-LT, and ImageNet-127 [29, 18] datasets. To simulate real-world unlabeled data, we tested our method on diverse label distributions of unlabeled data. Due to limited space, we defer the detailed experimental settings to the appendix.For consistent (γ l = γ u ) setting, results are presented in Table 2 . From the results, we can see that CCL consistently outperforms all comparison methods by a large margin. In particular, CCL improves the previous state-of-the-art approach ACR by 2.8% on average. This verifies that the representations learned by our proposed contrastive losses are more discriminative because both CCL and ACR utilize a dual-branch network.For inconsistent (γ l ̸ = γ u ) setting, we present the results in Table 3 and Table 4 . Following prior works, we compare all methods using unlabeled data following uniform and reversed label distributions on CIFAR10/100-LT datasets. On the STL10-LT dataset, the underlying unlabeled data distribution is naturally inaccessible. In general, CCL achieves the best results in all settings. Particularly, CCL obtains an average performance gain of 1.6% over ACR without using predefined anchor distributions. The results indicate that our method is able to accurately estimate the unlabeled data distribution with calibrated pseudo-labels.ImageNet-127 is a naturally long-tailed dataset and has been used to test LTSSL methods in the recent literature [22, 64] . Following previous works, we downsample the original images to smaller sizes of 32×32 or 64×64 pixels using the box method from the Pillow library and randomly select 10% training samples to construct the labeled data. Learning discriminative representations and a balanced classifier is essential to achieve high performance. From the results in Understanding of balanced Fixmatch and dual-branch. First, balanced Fixmatch can be viewed as a separate EM algorithm process [17, 21] , where the E-step involves estimating suitable pseudo-labels for unlabeled data through π u , and the M-step updates the model and obtains a new π u . As can be seen in Table 6 , balanced Fixmatch achieves performance comparable to the recent state-of-the-art method ACR. Furthermore, dual-branch significantly enhances the performance of data under highly skewed long-tail distributions (consistent setting), with an averaged 1.5% improvement. How to estimate a relatively accurate π u ? Our method for estimating π u is equivalent to MLLS [53] , which is an EM process. Accurate estimation of π u is only achievable when the model is calibrated [25] . Since the confirmation bias is induced by self-training, using confidence selection may result in overconfident but wrong pseudo-labels and hurt the calibration [45, 41] . In contrast, the energy score leverages the probability density of the predictions, exhibiting reduced vulnerability to overconfidence [39] . Thus, we propose energy selection for a reliable unlabeled data subset on which the model is calibrated, thereby enabling the accurate estimation of π u . We use expected calibration error [26] (ECE) to assess model calibration. The tail of the curve of Figure 1c and 1d can be interpreted as overconfidence in false pseudo-labels caused by self-training. As can be seen in Figure 1a and 1b, the L 1 distance between the true class prior of unlabeled data and π u , estimated from data subset selected using energy, is significantly smaller compared to when confidence is used for selection, inducing a more balanced classifier training.γ u = 1 (uniform) γ u = 1/10 (reversed) Algorithm N 1 = 50 N 1 = 150 N 1 = 50 N 1 = 150 M 1 = 400 M 1 = 300 M 1 = 400 M 1 =Continuous contrastive learning with reliable pseudo-labels. We carried out a comparative experiment by removing the continuous reliable pseudo-labels loss. The results reflect an averaged 0.8% drop on CIFAR10/100-LT, demonstrating its efficacy for learning high-quality representation. Moreover, we verified that the data subset filtered by energy selection obtains excellent model calibration. Figure 1c and 1d show energy achieves better calibration than confidence thresholding.Continuous contrastive learning with smoothed pseudo-labels. Similarly, we conducted a comparative experiment by removing the continuous smoothed pseudo-labels loss. As can be seen in Table 6 , the performance decreases in all three settings on CIFAR10/100-LT datasets, showing the necessity for a consistency regularization constraint for feature alignment within the contrastive learning space. Similar to ACR, to evaluate our method's effectiveness under more imbalanced settings, we conducted further experiments on CIFAR100-LT, maintaining a fixed γ l = 20 and adjusting the imbalance ratio γ u of the unlabeled data from consistent to reversed. We set N 1 = 50 and M 1 = 400 (with M C = 400 in the reversed scenario) and compared the results with ACR as shown in Figure 2 . The results demonstrate that our method consistently outperforms ACR in all scenarios. 
related work: 
 Long-tailed learning (LTL). Early strategies tackling LTL involve two aspects: resampling and reweighting. Resampling methods [9, 5, 8, 54] either undersample majority classes or oversample minority classes, which may result in information loss or overfitting. Reweighting methods [52, 16, 2, 12] assign different weights for each class or training sample. BBN [72] and Decoupling [31] claim that re-balancing can negatively impact representation. They propose a two-branch structure or a two-stage paradigm to address it. Logit adjustment methods [7, 44] learn larger margins for minority classes by obtaining optimal Bayesian classifiers. Recently, several methods [30, 15, 73, 20, 57] have been proposed to improve the representation learning based on supervised contrastive learning [32] .Long-tailed semi-supervised learning (LTSSL). Most semi-supervised learning (SSL) methods use unlabeled data by assigning pseudo-labels to unlabeled data [37, 6, 56, 71, 10] or aligning predictions of different views of the input by consistency regularization [59] . PAWS [4] leverages self-supervised representations derived from unlabeled data, and RoPAWS [46] further refines the model predictions using labeled data through kernel density estimation. However, most of these works assume a balanced class distribution of labeled and unlabeled data, which may be violated in real-world applications.Recently, LTSSL has gained considerable attention due to its applicability in numerous real-life scenarios. Recent works mitigate pseudo-labels bias by distribution alignment or label refinement [33, 63, 69] . Some others focus on balanced classifier training to overcome long-tailed label distribution [38, 22, 66] . Regrettably, these methods simply assume an identical long-tailed distribution for labeled and unlabeled data, which may still be unrealistic. Considering the unknown unlabeled data distribution, which can be mismatched with the labeled distribution, DASO [48] mixes the outputs of linear and semantic classifiers to improve the quality of pseudo-labels. ACR [64] and CPE [43] refine consistency regularization or train multiple expert branches based on predefined anchor distributions. However, how to improve representation learning in LTSSL is ignored in most existing works. 
conclusion: 
 This paper presents a probabilistic framework that unifies many recent methods in long-tail learning. Our framework is equivalent to supervised contrastive learning when approximating the classconditional function using the Gaussian kernel. We further extend the contrastive learning objective to LTSSL based on continuous pseudo-labels to improve the learned representations. We utilize both reliable pseudo-labels generated by the model and smoothed pseudo-labels propagated from nearby samples to mitigate confirmation bias. Extensive experiments demonstrate that our proposed method achieves state-of-the-art performance in all settings. We hope that our work can motivate more research for LTSSL from the perspective of representation learning. 
 
 Appendix: As we have derived in Eq. ( 10) and use the Gaussian kernel density estimation in Eq. ( 11), if we simply assume the data are class-balanced, it simplifies to:EQUATION)The loss of supervised contrastive learning has two forms, i.e., L in scl and L out scl , which is distinguished by the position of summation over positive samples at the log(•). Thus, we get:EQUATION)which is consistent with the original paper's derivation.As we have derived before, here we dive deep into the analysis of existing methods and demonstrate that they all belong to our unified framework.Balanced contrastive learning [73] (BCL) was proposed to solve long-tailed problems with improved supervised contrastive learning. BCL involves two key techniques: class averaging and class complement. BCL averages out the contributions of different classes in the denominator to ensure class equal distribution, meanwhile, it takes nonlinear mapping of the classifier parameters to form a learnable class center to ensure that every class has at least one sample in a mini-batch:EQUATIONHowever, BCL ignores the problem of the original long-tailed distribution in the training dataset, necessitating a reweighting operation. Let π denote the class prior, we have:EQUATION)Gaussian mixture likelihood loss [57] (GML) initiates its approach from the concept of mutual information, employing the Gaussian kernel. GML integrates contrastive learning with logit adjustment to enhance its performance.Lgml (x, y) = -logEQUATION)It proposes a class-wise queue Q = ∪ C k=1 Q k to ensure a balanced class occurrence in a mini-batch. However, it does not propose a unified framework, and the understanding is not deep enough. Some other methods: In this section, we compare some other methods that use contrastive learning and analyze their mistakes. K-positive contrastive learning [30] (KCL) is based on supervised contrastive learning, using K samples of the same class in the molecule to ensure balanced feature space. Putting in our unified framework, we can obtain the following:EQUATIONEQUATIONCompared with the above, it regrettably still uses the same unprocessed denominator of SCL and cannot ensure that each mini-batch contains an equal number of samples from each class, nor that each class contributes equally, rendering it suboptimal for long-tailed learning. In addition, Eq. ( 28) is equivalent to the resampling technique.Parametric contrastive learning [15] (PaCo) introduces a set of parametric class-wise learnable centers and uses adjustable parameters α for loss with respect to them. Our framework is used to derive its original form. First, we can obtain:EQUATION)where β is a fixed coefficient. Then, we can derive the PaCo loss as follows.EQUATIONwhere α = βP(Y =y) (1-β)|By| . PaCo explicitly uses the parametric class center to ensure balanced class occurrence in a mini-batch. However, It ignores the class-equal contribution in loss computation, which can still be suboptimal.Probabilistic contrastive learning [20] (Proco) simply assumes that the normalized features in contrastive learning follows a mixture of von Mises-Fisher (vMF) distributions on a unit ball, its probability density function has the following form:EQUATIONwhere parameters (µ y , ρ y ) need to be estimated. The advantage is that Proco can estimate (µ y , ρ y ) using an online mini-batch, such that it can be derived as a closed form of expected contrastive loss. Despite the assumed vMF distribution, it still uses Gaussian kernel estimation:EQUATIONwhere λ(z x , y) represents a fixed function related to z x , µ y , ρ y . Thus, the loss objective of Proco is:EQUATIONHowever, the assumed distribution of Proco stills needs to be estimated (µ y , ρ y ) using EMA of different batches, which is essentially a similar approach to the momentum queue used in GML.There still exist problems of inconsistent distribution of z in different mini-batches, and the strong assumption about P(z | Y = y) which may not follow the vMF distribution. T-vMF [34] models cos θ y as vMF distribution:f (cos θ y ; ρ y ) = 1 C (ρ y )e ρy cos θy = C ′ (ρ y )e ρy∥z-µ y ∥ = C ′ (ρ y )s e ( zµ y , ρ y ).Due to the inherent properties of the exponential function, the posterior quickly converges to 0, despite a large zµ y . Such a compact measuring function might hamper model training, since tailed samples hardly enjoy back-propagation due to vanishing gradient. To overcome this problem, T-vMF introduces a family of modeling methods as follows: (36) wheref q (cos θ y ; ρ y ) = C ′ (ρ y ) 1 -(1 -q) 1 2 ρ y z -µ y 1 1-q = C ′ (ρ y )s q ( z -µ y , ρ y ),s q ( z -µ y , ρ y ) = 1 -(1 -q) 1 2 ρ y z -µ y 1 1-q .Technically, T-vMF models P t (Y = y | z) as follows:EQUATIONEQUATIONThus, the loss objective of T-vMF is:EQUATIONWCDAS [27] The accuracy of the posterior approximation is a crucial factor influencing the method's performance. Unlike t-vMF, which directly specifies the P(z | Y = y) with fixed parameters, WCDAS seeks an optimal parametric probability density function of P(z | Y = y).Modeling cos θ y as the Wrapped Cauchy distribution with trainable parametric ϑ = [ϑ 1 , . . . , ϑ C ], WCDAS models P t (Y = y | z) as follows:EQUATIONEQUATIONThus, the loss objective of WCDAS is:EQUATIONTo ensure clarity and precision throughout this paper, we provide a comprehensive list and definitions of the key mathematical symbols and terms used in this section. Each symbol is defined with its specific meaning and context to ensure consistency and accuracy across the document.D.1 Illustration of The Overall Proposed Algorithm CCL consists of two parts: the classification part and the contrastive learning part. The classification part uses logit rectification of the classifier by class prior estimated with a dual-branch. For the contrastive learning part, the energy score is used to select reliable unlabeled data which are merged with labeled data for continuous contrastive loss to ensure calibration. Besides, information of labeled data and unlabeled are used in a decoupled manner while maintaining the constraints of aligning feature in the contrastive learning space, thereby forming a smoothed contrastive loss. Algorithm 1 summarizes the whole framework of the proposed CCL, which is clearly divided into three components: balanced classifier, continuous contrastive learning with reliable and smoothed pseudo-labels, respectively.Training datasets. Our experimental analysis uses a variety of commonly adopted SSL datasets, including CIFAR10-LT [35] , CIFAR100-LT [35] , STL10-LT [13] , and ImageNet-127 [22] in various ratios of class imbalance γ and various ratios of the amount of labeled data η. To create imbalanced versions of these datasets, we consider the long-tailed imbalance where the frequency of data points decreases exponentially from the largest to the smallest class, that is, the number of samples in class c isN c = N 1 ×γ -c-1C-1 for labeled data and M c = M 1 ×γ -c-1 C-1 for unlabeled data. We use Cutout [19] and Randaugment [14] for strong augmentation on unlabeled data, and we use SimAugment [11] on labeled data for continuous contrastive loss with smoothed pseudo-labels. Like recent LTSSL works, we consider three class distribution patterns for unlabeled data, namely, consistent, uniform, and reversed settings.• CIFAR10-LT: Following ACR [64] , we conduct experiments with all comparison methods in settings where N 1 = 500, M 1 = 4000 and N 1 = 1500, M 1 = 3000. We adopt imbalance ratios of γ l = γ u = 100 and γ l = γ u = 150 for consistent settings, while for uniform and reversed settings, we use γ l = 100, γ u = 1 and γ l = 100, γ u = 1/100, respectively. • CIFAR100-LT: Like CIFAR10-LT, we perform experiments in configurations where N 1 = 50, M 1 = 400 and N 1 = 150, M 1 = 300. For the consistent settings, we use imbalance ratios of γ l = γ u = 10 and γ l = γ u = 20. In contrast, for the uniform and reversed settings, we apply γ l = 10, γ u = 1 and γ l = 10, γ u = 1/10, respectively. • STL10-LT: Given the absence of ground-truth labels for the unlabeled data of the STL10 dataset, we manage the experiments by adjusting the imbalance ratio of the labeled data.Following ACR, we consider the labeled imbalance ratio of γ l = 10 or γ l = 20. • ImageNet-127: ImageNet127 was first introduced in an earlier research [29] and utilized in LTSSL by CReST. This dataset consolidates the 1000 classes [18] from ImageNet into 127 classes, grouping them according to the WordNet hierarchy. For ImageNet-127, we follow the original setting in CoSSL [22] (γ l = γ u ≈ 286).Implementation details. Our experimental configuration largely aligns with Fixmatch [56] and ACR [64] . Specifically, we apply the Wide ResNet-28-2 [70] architecture to implement our method on the CIFAR10-LT, CIFAR100-LT and STL10-LT datasets; and ResNet-50 on ImageNet-127. We adopt the common training paradigm that the network is trained with standard SGD [47, 49, 58] for 500 epochs, where each epoch consists of 500 mini-batches, and a batch size of 64 for both labeled and unlabeled data. We use a cosine learning rate decay [42] where the initial rate is 0.03, we set τ = 2.0 for logit adjustment on all datasets, except for ImageNet-127, where τ = 0.1. We set the temperature T = 1 and the threshold ζ = -8.75 for the energy score following [69] , and we set λ 1 = 0.7, λ 2 = 1.0 on CIFAR10/100-LT and λ 1 = 0.7, λ 2 = 1.5 on STL10-LT and ImageNet-127 datasets for the final loss. We set β = 0.2 in Eq. ( 21) for smoothed pseudo-labels loss. To show the effectiveness of our approach, we perform a comparative analysis with several existing LTSSL algorithms, including DARP [33] , CReST [63] , DASO [48] , ABC [38] , and TRAS [66] . We also consider the most popular LTSSL methods ACR [64] and CPE [43] . The performance evaluation of these methods is based on the top-1 accuracy metric on the test set. We present the mean and standard deviation of the results from three independent runs for each method. In addition, our method is implemented using the PyTorch library and experimented on an NVIDIA RTX A6000 (48 GB VRAM) with an Intel Platinum 8260 (CPU, 2.30GHz, 220 GB RAM).As outlined in figure 5a , CCL is relatively robust to the fluctuation of β from 0.1 to 0.4. However, when β is set to 0, the propagation within unlabeled data is ignored, resulting in a performance decrease of about 0.9%. Thus, the necessity of using Eq. ( 21) is verified. In addition, figures 5b and 5c both demonstrate that CCL is robust to loss weighting coefficients λ 1 and λ 2 within a certain range.However, it is worth noting that when λ 1 = 1.0, the proposed continuous reliable pseudo-labels loss is ignored, resulting in performance degradation. In this section, we conduct analyses of the time and space complexity of the proposed method. Denote feature space dimension D, batch size B and the number of classes C, the time and space complexity of CCL can be seen in Table 8 and analysis details as follows. Generally, given B = 64, C = 100 and D = 256. Compared to the scale of model parameters, CCL adds negligible overhead relative to the neural network's computational cost when computing loss. We further report the averaged mini-batch training time with a single 3090 GPU and the GPU memory usage in Table 9 and Table 10 . As seen from these tables, the training time and space consumptions of CCL are comparable to the existing state-of-the-art method ACR when CCL applies additional data augmentations to labeled data for representation learning. second row, CCL achieved an extraordinarily high accuracy of 78% for class "9", which shows a significant gain of 37% compared to ACR. CCL also achieves higher overall accuracy.To conduct a more in-depth analysis of the effectiveness of pseudo-labels generated by the proposed dual-branch fusion approach, we calculated the precision and recall of the pseudo-labels assigned to unlabeled data by ACR and CCL on CIFAR10-LT and CIFAR100-LT datasets. Specifically, we use γ l = γ u = 100 and γ l = γ u = 150 settings on CIFAR10-LT dataset and all three consistent, uniform, reversed settings on CIFAR100-LT dataset and we grouped the results of CIFAR100 into 10 categories, each category containing 10 classes, since CIFAR100 comprises 100 classes. As can be seen in figure 8 , CCL achieves significantly improved precision of pseudo-labels for tailed classes "9" and "10" on CIFAR10 dataset, while also achieving better recall for head classes. Similarly in Figure 7 , CCL achieves overall better precision and recall compared to ACR regardless of the distribution mismatch scenario. It clearly shows that the pseudo-labels generated by CCL are more capable of alleviating the confirmation bias of tailed classes without sacrificing the performance of head classes.Furthermore, we employ the t-distributed stochastic neighbor embedding (t-SNE) [61] to visualize the representations learned by the CCL method and contrast them with those from the previous ACR method. The comparative results on the test set, under consistent settings, are depicted in Figure 9 . The figure demonstrates that the representations derived from CCL provide more distinct classification boundaries.Our paper examines existing long-tailed learning methods through the lens of information theoretical view, proposing a unified framework. However, we have not established theoretical proof for the convergence of features within this framework. In the future, we intend to provide further theoretical analysis from the perspective of neural collapse. The matrix inversion operation is implemented using torch.inverse(), which utilizes the fast singular value decomposition. The time complexity is O(|B u | 3 ). 


Decision: The paper initially presents a unified probabilistic framework for long-tail learning and further extends it to a semi-supervised setting through a novel continuous contrastive learning approach. The results demonstrate significant improvements across all benchmarks. The rebuttal effectively addresses the reviewers' concerns, leading to unanimous positive feedback. After reviewing the comments and feedback, the AC  recommends acceptance.

Reviews: review: paper_summary: paper_summary: This paper tackles the long-tailed semi-supervised learning problem. It first reviews recent works with a novel probabilistic framework. Based on this, it proposes a continuous contrastive learning method, CCL, to extend the framework to unlabeled data with pseudo-labels. Experiments show that it outperforms all recent works on the popular long-tailed semi-supervised learning benchmarks.
strength_weakness: strength_weakness: I only have one question:

In Eq. 22, it mentioned L_{cls}, which is also mentioned in Line 171. Is that the same as P^{cls} in Eq. 15?
questions: questions: See weaknesses

In Eq. 22, it mentioned L_{cls}, which is also mentioned in Line 171. Is that the same as P^{cls} in Eq. 15?
limitations: limitations: Limitations and potential impacts are addressed in Appendix,
paper_summary: paper_summary: This paper proposes a novel contrastive learning method for long-tailed semi-supervised learning (LTSSL). The method is motivated by variational information bottleneck for learning good representations and extends to unlabeled data using continuous pseudo-labels. This paper showcases strong empirical results on multiple LTSSL datasets and label distributions.
strength_weakness: strength_weakness: 1.In Table 1, the paper states that long-tail learning methods need to avoid none class samples in each min-batch during training. However, it is unclear how this issue is tackled in the proposed method CCL.
2.The training time of the proposed method is not reported. It is suggested to compare the training time with ACR.
questions: questions: Please see Weaknesses.
limitations: limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
paper_summary: paper_summary: The author proposes a probabilistic framework that unifies many recent proposals in long-tail learning. Specifically, for long-tailed semi-supervised learning,  a continuous contrastive learning method based on reliable and smoothed pseudo-labels to address confirmation bias and improve the quality of learned representations is proposed. The experiments demonstrate their effectiveness.
strength_weakness: strength_weakness: 1) The author should discuss explicitly how the proposed method solves the unlabelled data  for diverse label distributions of unlabeled data and why other methods do not. 

2) How to interpret " Furthermore, these methods primarily focus on correcting model outputs without delving into the role of representation learning in improving performance."  line 33-34. 

3) The author may give an analysis of the relationship between  reliable pseudo-labels and smoothed pseudo-labels and how they play a different role in semi-supervised learning.

4) The time and space complexity should be analyzed in experiments between the proposed method and other methods.
questions: questions: see weakness
limitations: limitations: yes
paper_summary: paper_summary: The paper proposes a novel method to address the long-tail problem in semi-supervised learning by leveraging continuous contrastive learning on both labeled and unlabeled samples to improve model performance, particularly for minority classes.
strength_weakness: strength_weakness: - In line 189, it is unclear why selecting pseudo-labels using energy score can ensure better model calibration.
questions: questions: A minor suggestion: the paper can include some demonstrations of unlabeled images with their associated continuous pseudo-labels, including both reliable and smoothed versions. This can help readers identify cases in which the model tends to make mistakes.
limitations: limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.

Comments: title: Keep the original ratings
comment: Thanks for your response. I would like to keep the original ratings.