Title: Combating Representation Learning Disparity with Geometric Harmonization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Long-tailed learning

self-supervised learning
Abstract: Self-supervised learning (SSL) as an effective paradigm of representation learning has achieved tremendous success on various curated datasets in diverse scenarios. Nevertheless, when facing the long-tailed distribution in real-world applications, it is still hard for existing methods to capture transferable and robust representation. The attribution is that the vanilla SSL methods that pursue the sample-level uniformity easily leads to representation learning disparity, where head classes with the huge sample number dominate the feature regime but tail classes with the small sample number passively collapse. To address this problem, we propose a novel Geometric Harmonization (GH) method to encourage the category-level uniformity in representation learning, which is more benign to the minority and almost does not hurt the majority under long-tailed distribution. Specially, GH measures the population statistics of the embedding space on top of self-supervised learning, and then infer an fine-grained instance-wise calibration to constrain the space expansion of head classes and avoid the passive collapse of tail classes. Our proposal does not alter the setting of SSL and can be easily integrated into existing methods in a low-cost manner. Extensive results on a range of benchmark datasets show the effectiveness of \methodspace with high tolerance to the distribution skewness.

Combating Representation Learning Disparity with Geometric Harmonization

Abstract

Self-supervised learning (SSL) as an effective paradigm of representation learning has achieved tremendous success on various curated datasets in diverse scenarios. Nevertheless, when facing the long-tailed distribution in real-world applications, it is still hard for existing methods to capture transferable and robust representation. Conventional SSL methods, pursuing sample-level uniformity , easily leads to representation learning disparity where head classes dominate the feature regime but tail classes passively collapse. To address this problem, we propose a novel Geometric Harmonization (GH) method to encourage category-level uniformity in representation learning, which is more benign to the minority and almost does not hurt the majority under long-tailed distribution. Specially, GH measures the population statistics of the embedding space on top of self-supervised learning, and then infer an fine-grained instance-wise calibration to constrain the space expansion of head classes and avoid the passive collapse of tail classes. Our proposal does not alter the setting of SSL and can be easily integrated into existing methods in a low-cost manner. Extensive results on a range of benchmark datasets show the effectiveness of GH with high tolerance to the distribution skewness. Our code is available at https://github.com/MediaBrain-SJTU/Geometric-Harmonization.

1 Introduction

Recent years have witnessed a great success of self-supervised learning to learn generalizable representation [7, 9, 15, 63]. Such rapid advances mainly benefit from the elegant training on the labelfree data, which can be collected in a large volume. However, the real-world natural sources usually exhibit the long-tailed distribution [50], and directly learning representation on them can lead to the distortion issue of the embedding space, namely, the majority dominates the feature regime [75] and the minority collapses [47]. Thus, it becomes urgent to pay attention to representation learning disparity, especially as fairness of machine learning draws increasing attention [28, 41, 68, 77].

Different from the flourishing supervised long-tailed learning [29, 46, 68], self-supervised learning under long-tailed distributions is still under-explored, since there is no labels available for the calibration. Existing explorations to overcome this challenge mainly resort to the possible tailed sample discovery and provide the implicit bias to representation learning. For example, BCL [77] leverages the memorization discrepancy of deep neural networks (DNNs) on unknown head classes and tail classes to drive an instance-wise augmentation. SDCLR [28] contrasts the feature encoder and its pruned counterpart to discover hard examples that mostly covers the samples from tail classes, and efficiently enhance the learning preference towards tailed samples. DnC [59] resorts to a divideand-conquer methodology to mitigate the data-intrinsic heterogeneity and avoid the representation

4

2

0

2

4

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

4

2

0

2

4

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

collapse of minority classes. Liu et al. [41] adopts a data-dependent sharpness-aware minimization scheme to build support to tailed samples in the optimization. However, few works hitherto have considered the intrinsic limitation of the widely-adopted contrastive learning loss, and design the corresponding balancing mechanism to promote the representation learning parity.

We rethink the characteristic of the contrastive learning loss, and try to understand 'Why the conventional contrastive learning underperforms in self-supervised long-tailed context?' To answer this question, let us consider two types of representation uniformity: (1) Sample-level uniformity . As stated in [62], the contrastive learning targets to distribute the representation of data points uniformly in the embedding space. Then, the feature span of each category is proportional to their corresponding number of samples. (2) Category-level uniformity . This uniformity pursues to split the region equally for different categories without considering their corresponding number of samples [49, 19]. In the class-balanced scenarios, the former uniformity naturally implies the latter uniformity, resulting in the equivalent separability for classification. However, in the long-tailed distributions, they are different: sample-level uniformity leads to the feature regime that is biased towards the head classes considering their dominant sample quantity and sacrifices the tail classes due to the limited sample quantity. By contrast, category-level uniformity means the equal allocation w.r.t. classes, which balances the space of head and tail classes, and is thus more benign to the downstream classification [17, 19, 37]. Unfortunately, there is no support for promoting category-level uniformity in contrastive learning loss, which explains the question arisen at the beginning.

In this study, we propose a novel method, termed as Geometric Harmonization (GH) to combat representation learning disparity in SSL under long-tailed distributions. Specially, GH uses a geometric uniform structure to measure the uniformity of the embedding space in the coarse granularity. Then, a surrogate label allocation is computed to provide a fine-grained instance-wise calibration, which explicitly compresses the greedy representation space expansion of head classes, and constrain the passive representation space collapse of tail classes. The alternation in the conventional loss refers to an extra efficient optimal-transport optimization that dynamically pursues the category-level uniformity. In Figure 1, we give a toy experiment 1 to visualize the distribution of the embedding space without and with GH. In a nutshell, our contributions can be summarized as follows,

To our best knowledge, we are the first to investigate the drawback of the contrastive learning loss in self-supervised long-tailed context and point out that the resulted sample-level uniformity is an intrinsic limitation to the representation parity, motivating us to pursue category-level uniformity with more benign downstream generalization (Section 3.2).

We develop a novel and efficient Geometric Harmonization (Figure 2) to combat the representation learning disparity in SSL, which dynamically harmonizes the embedding space of SSL to approach the category-level uniformity with the theoretical guarantee.

Our method can be easily plugged into existing SSL methods for addressing the data imbalance without much extra cost. Extensive experiments on a range of benchmark datasets demonstrate the consistent improvements in learning robust representation with our GH.

2 Related Work

Self-Supervised Long-tailed Learning. There are several recent explorations devoted to this direction [40, 28, 59, 41, 77]. BCL [77] leverages the memorization effect of DNNs to automatically drive an instance-wise augmentation, which enhances the learning of tail samples. SDCLR [28] constructs a self-contrast between model and its pruned counterpart to learn more balanced representation. Classic Focal loss [40] leverages the loss statistics to putting more emphasis on the hard examples, which has been applied to self-supervised long-tailed learning [77]. DnC [59] benefits from the parameter isolation of multi-experts during the divide step and the information aggregation during the conquer step to prevent the dominant invasion of majority. Liu et al. [41] proposes to penalize the sharpness surface in a reweighting manner to calibrate class-imbalance learning. Recently, TS [35] employs a dynamic strategy on the temperature factor of contrastive loss, harmonizing instance discrimination and group-wise discrimination. PMSN [2] proposes the power-law distribution prior, replacing the uniform prior, to enhance the quality of learned representations.

Hyperspherical Uniformity. The distribution uniformity has been extensively explored from the physic area, e.g. , Thomson problem [58, 53], to machine learning area like some kernel-based extensions, e.g. , Riesz s-potential [21, 43] or Gaussian potential [12, 4, 62]. Some recent explorations regarding the features of DNNs [49, 17, 47] discover a terminal training stage when the embedding collapses to the geometric means of the classifier w.r.t. each category. Specially, these optimal class means specify a perfect geometric uniform structure with clear geometric interpretations and generalization guarantees [78, 67, 31]. In this paper, we first extend this intuition into self-supervised learning and leverage the specific structure to combat the representation disparity in SSL.

3 Method

3.1 Problem Formulation

We denote the dataset D , for each data point ( x , y ) ∈ D , the input x ∈ R m and the associated label y ∈ { 1 , . . . , L } . Let N denote the number of samples, R = N max /N min denote the imbalanced ratio (IR), where N max , N min is the number of samples in the largest and smallest class, respectively. Let n i denote the number of samples in class i . In SSL, the ground-truth y can not be accessed and the goal is to transform an image to an embedding via DNNs, i.e. , f θ : R m → R d . In the linear probing evaluation, we construct a supervised learning task with balanced datasets. A linear classifier g ( · ) is built on top of the frozen f θ ( · ) to produce the prediction, i.e., g ( f θ ( x )) .

3.2 Geometric Harmonization

As aforementioned, most existing SSL methods in long-tailed context leverage the contrastive learning loss, which encourages the sample-level uniformity in the embedding space. Considering the intrinsic limitation illustrated in Figure 1, we incorporate the geometric clues from the embedding space to calibrate the current loss, enabling our pursuit of category-level uniformity. In the following, we first introduce a specific geometric structure that is critical to Geometric Harmonization.

Definition 3.1. (Geometric Uniform Structure). Given a set of vertices M ∈ R d × K , the geometric uniform structure satisfies the following between-class duality

̸

where ∥ M i ∥ = 1 , ∀ i ∈ { 1 , 2 , . . . , K } , K is the number of geometric vertices and C is a constant.

Above structure provides a characteristic that any two vectors in M have the same angle, namely, the unit space are equally partitioned by the vectors in M . This fortunately follows our expectation about category-level uniformity. Specially, if we use M as a constant classifier to involve into training, and have the oracle label y of the long-tailed data ( K = L ) to supervise the below prediction

then according to the neural collapse theory [49], the representation of all samples will fit the geometric uniform structure of M in the limit, namely, approach category-level uniformity. However,

𝜋

Q

Contrastive Learning

Q

�

From Sample-Level Uniformity to Category-Level Uniformity

Class index

# Instances

Head

Tail

Marginal distribution

constraint

Long-tailed data

:

Geometric Uniform Structure

:

Surrogate Label Allocation

Bi-Level Optimization

LInfoNCE

LGC

Q

min LInfoNCE + LGH

0,Q

the technical challenge is the complete absence of annotation y in our context, making directly constructing the objective min E [ y log q ] with Eq. (2) impossible to combat the representation disparity.

Surrogate Label Allocation. To address the problem of unavailable labels, we explore constructing the surrogate geometric labels ˆ q to supervise the training of Eq. (2). Concretely, we utilize the recent discriminative clustering idea [1] to acquire such geometric labels, formulated as follows

where ˆ q i refers to the soft assignment constrained in a K -dimensional probability simplex and π ∈ R K + refers to the distribution constraint. As we are dealing with long-tailed data, we propose to use the geometric uniform structure M to automatically compute the population statistic of the embedding space as π . Finally, Eq. (3) can be analytically solved by Sinkhorn-Knopp algorithm [14] (refer to Appendix D for Algorithm 1). Note that, the above idea builds upon a conjecture: the constructed surrogate geometric labels are mutually correlated with the oracle labels so that they have the similar implication to approach category-level uniformity. We will empirically verify the rationality of such an assumption via normalized mutual information in Section 4.4.

Overall Objective. Eq. (3) can be easily integrated into previous self-supervised long-tailed learning methods for geometric harmonization, e.g., SDCLR [28] and BCL [77]. For simplicity, we take their conventional InfoNCE loss [48] as an example and write the overall objective as follows

where w GH represents the weight of the geometric harmonization loss. Optimizing Eq. (4) refers to a bi-level optimization style: in the outer-loop, optimize min ˆ Q L GH with fixing θ to compute the surrogate geometric labels; in the inner-loop, optimize min θ L InfoNCE + L GH with fixing ˆ Q to learn the representation model. The additional cost compared to the vanilla method primarily arises from from the outer-loop, which will be discussed in Section 3.4 and verified in Section 4.4.

Compared with previous explorations [1, 7], the uniqueness of GH lies in the following three aspects: (1) Geometric Uniform Structure . The pioneering works mainly resort to a learnable classifier to perform clustering, which can easily be distorted in the long-tailed scenarios [17]. Built on

the geometric uniform classifier, our method is capable to provide high-quality clustering results with clear geometric interpretations. (2) Flexible Class Prior . The class prior π is assumed to be uniform among the previous attempts. When moving to the long-tailed case, this assumption will strengthen the undesired sample-level uniformity. In contrast, our methods can potentially cope with any distribution with the automatic surrogate label allocation. (3) Theoretical Guarantee. GH is theoretically grounded to achieve the category-level uniformity in the long-tailed scenarios (refer to Section 3.3), which has never been studied in previous methods. To gain more insights into our method, we further compare GH with discriminative clustering methods (SeLA [1], SwAV [7]) and investigate the impact of various components in GH. From the results in Table 1, we can see that GH consistently outperforms the vanilla discriminative clustering baselines in the linear probing evaluation. Notably, we observe that both GUS and the class prior π play a critical role in our GH.

3.3 Theoretical Understanding

Here, we reveal the theoretical analysis of GH on promoting the representation learning to achieve category-level uniformity instead of sample-level uniformity. Let us begin with a deteriorated case of sample-level uniformity under the extreme imbalance, i.e. , minority collapse [17].

Lemma 3.2. (Minority collapse) Assume the samples follow the uniform distribution n 1 = n 2 = · · · = n L H = n H , n L H +1 = n L H +2 = · · · = n L = n T in head and tail classes respectively. Assume d ≥ L and n H /n T → + ∞ , the lower bound (Lemma C.1) of L InfoNCE achieves the minimum when the class means of the tail classes collapse to an identical vector:

where µ l = 1 n l ∑ n l i =1 f θ ( x l,i ) denotes the class means and x l,i is the i -th data point with label l .

This phenomenon indicates all representations of the minority will collapse completely to one point without considering the category discrepancy, which aligns with our observation regarding the passive collapse of tailed samples in Figure 1. To further theoretically analyze GH, we first quantitatively define category-level uniformity in the following, and then theoretically claim that with the geometric uniform structure (Definition. 3.1) and the perfect aligned allocation (Eq. (3)), we can achieve the loss minimum at the stage of realizing category-level uniformity.

Definition 3.3. (Categorical-level Uniformity) We define categorical-level uniformity on the embedding space w.r.t the geometric uniform structure M when it satisfies

where µ ∗ k = 1 n k ∑ n k i =1 f ∗ θ ( x k,i ) represents the class mean for samples assigned with the surrogate geometric label k in the embedding space.

Theorem 3.4. (Optimal state for L ) Given Eq. (4) under the proper optimization strategy, when it arrives at the category-level uniformity (Definition 3.3) defined on the geometric uniform structure M (Definition 3.1), we will achieve the minimum of the overall loss L ∗ as

where J denotes the size of the collection of the negative samples and π y refers to the marginal distribution of the latent ground-truth labels y .

This guarantees the desirable solution with the minimal intra-class covariance and the maximal interclass covariance under the geometric uniform structure [49, 17], which benefits the downstream generalization. Notably, no matter the data distribution is balanced or not, our method can persistently maintain the theoretical merits on calibrating the class means to achieve category-level uniformity. We also empirically demonstrate the comparable performance with GH on the balanced datasets in Section 4.4, as in this case category-level uniformity is equivalent to sample-level uniformity.

3.4 Implementation and Complexity analysis

In Algorithm 2 of Appendix D, we give the complete implementation of our method. One point that needs to be clarified is that we learn the label allocation ˆ q in the mini-batch manner. In addition, the geometric prediction q and the adjusted ˆ q are computed at the beginning of every epoch as the population-level statistic will not change much in a few mini-batches. Besides, we maintain a momentum update mechanism to track the prediction of each sample to stabilize the training, i.e. , q m ← β q m + (1 -β ) q . When combined with the joint-embedding loss, we naturally adopt a cross-supervision mechanism min E [ ˆ q + log q ] for the reconciliation with contrastive baselines. The proposed method is illustrated in Figure 2 for visualization.

For complexity, assume that the standard optimization of deep neural networks requires forward and backward step in each mini-batch update with the time complexity as O ( B Λ) , where B is the minibatch size and Λ is the parameter size. At the parameter level, we add an geometric uniform structure with the complexity as O ( BKd ) , where K is the number of geometric labels and d is the embedding dimension. For Sinkhorn-Knopp algorithm, it only refers to a simple matrix-vector multiplication as shown in Algorithm 1, whose complexity is O ( E s ( B + K + BK )) with the iteration step E s . The complexity incurred in the momentum update is O ( BK ) . Since K,d and E s are significantly smaller than the model parameter Λ of a million scale, the computational overhead involved in GH is negligible compared to O ( B Λ) . The additional storage for a mini-batch of samples is the matrix Q m ∈ R K × B , which is also negligible to the total memory usage. To the end, GH incurs only a small computation or memory cost and thus can be plugged to previous methods in a low-cost manner. The empirical comparison about the computational cost is summarized in Table 17.

4 Experiments

4.1 Experimental Setup

Baselines. We mainly choose five baseline methods, including (1) plain contrastive learning : SimCLR [9], (2) hard example mining : Focal [40], (3) asymmetric network pruning : SDCLR [28], (4) multi-expert ensembling : DnC [59], (5) memorization-guided augmentation : BCL [77]. Empirical comparisons with more baseline methods can be referred to Appendix F.3.

Implementation Details. Following previous works [28, 77], we use ResNet-18 [23] as the backbone for small-scale dataset (CIFAR-100-LT [5]) and ResNet-50 [23] for large-scale datasets (ImageNet-LT [44], Places-LT [44]). For experiments on CIFAR-100-LT, we train model with the SGD optimizer, batch size 512, momentum 0.9 and weight decay factor 5 × 10 -4 for 1000 epochs. For experiments on ImageNet-LT and Places-LT, we only train for 500 epochs with the batch size 256 and weight decay factor 1 × 10 -4 . For learning rate schedule, we use the cosine annealing decay with the learning rate 0 . 5 → 1 e -6 for all the baseline methods. As GH is combined with baselines, a proper warming-up of 500 epochs on CIFAR-100-LT and 400 epochs on ImageNet-LT and Places-LT are applied. The cosine decay is set as 0 . 5 → 0 . 3 , 0 . 3 → 1 e -6 respectively. For hyper-parameters of GH, we provide a default setup across all the experiments: set the geometric dimension K as 100, w GH as 1 and the temperature γ GH as 0.1. In the surrogate label allocation, we set the regularization coefficient λ as 20 and Sinkhorn iterations E s as 300. Please refer to Appendix E.3 for more experimental details.

Evaluation Metrics. Following [28, 77], linear probing on a balanced dataset is used for evaluation. We conduct full-shot evaluation on CIFAR-100-LT and few-shot evaluation on ImageNet-LT and Places-LT. For comprehensive performance comparison, we present the linear probing performance and the standard deviation among three disjoint groups, i.e. , [many, medium, few] partitions [44].

4.2 Linear Probing Evaluation

CIFAR-100-LT. In Table 2, we summarize the linear probing performance of baseline methods w/ and w/o GH on a range of benchmark datasets, and provide the analysis as follows.

(1) Overall Performance. GHachieves the competitive results w.r.t the [many, medium, few] groups, yielding a overall performance improvements averaging as 2.32 % , 2.49 % and 1.52 % on CIFAR-100LT with different imbalanced ratios. It is worth noting that on the basis of the previous state-of-theart BCL, our GH further achieves improvements by 1.20 % , 1.82 % and 1.22 % , respectively. Our

GH consistently improves performance across datasets with varying degrees of class imbalance, demonstrating its potential to generalize to practical scenarios with more complex distributions. Specially, our method does not require any prior knowledge or assumptions about the underlying data distribution, highlighting the robustness and versatility to automatically adapt to the data.

(2) Representation Balancedness. In Section 1, we claim that GH helps compress the expansion of head classes and avoid the passive collapse of tail classes, yielding the more balanced representation distribution. To justify this aspect, we compare the variance in linear probing performance among many/medium/few groups, namely, their groupwise standard deviation. According to Table 2, our GHprovides [1.56 % , 2.35 % , 3.09 % ], [2.16 % , 2.44 % , 2.87 % ] and [1.26 % , 1.57 % , 1.74 % ] improvements w.r.t [many, medium, few] groups on CIFAR-100-LT-R100/R50/R10 with more preference to the minority classes for representation balancedness. Overall, GH substantially improves the standard deviation by [0.61, 0.38, 0.35] on different levels of imbalance.

ImageNet-LT and Places-LT. Table 2 shows the comparison of different baseline methods on largescale dataset ImageNet-LT and Places-LT, in which we have consistent observations. As can be seen, on more challenging real-world dataset, GH still outperforms other methods in terms of overall accuracy, averaging as 1.68 % , 0.68 % on ImageNet-LT and Places-LT. Specifically, our method provides [0.44 % , 2.23 % , 3.25 % ] and [0.55 % , 0.87 % , 0.45 % ] improvements in linear probing w.r.t. [many, medium, few] groups on ImageNet-LT and Places-LT. The consistent performance overhead indicates the robustness of our method to deal with long-tailed distribution with different characteristics. Moreover, the averaging improvement of standard deviation is 1.39 on ImageNet-LT, indicating the comprehensive merits of our GH on the minority classes towards the representation balancedness. However, an interesting phenomenon is that the fine-grained performance exhibits a different trend on Places-LT. As can be seen, the performance of head classes is even worse than that of tail classes. The lower performance of the head partition can be attributed to the fact that it contains more chal-

lenging classes. As a result, we observe that the standard deviation of our GH does not significantly decrease on Places-LT, which requires more effort and exploration for improvement alongside GH.

4.3 Downstream Finetuning Evaluation

Downstream supervised long-tailed learning. Self-supervised learning has been proved to be beneficial as a pre-training stage of supervised long-tailed recognition to exclude the explicit bias from the class imbalance [68, 41, 77].

To validate the effectiveness of our GH, we conduct self-supervised pre-training as the initialization for downstream supervised classification tasks on CIFAR-100-LT-R100, ImageNet-LT and PlacesLT. The state-of-the-art logit adjustment [46] is chosen as the downstream baseline. The combination of GH + LA can be interpreted as a compounded method where GH aims at the re-balanced representation extraction and LA targets the classifier debiasing. In Table 3, we can find that the superior performance improvements are achieved by self-supervised pre-training over the plain supervised learning baseline. Besides, our method can also consistently outperform other SSL baselines, averaging as 1.00 % , 0.63 % and 1.24 % on CIFAR-100-LT-R100, ImageNet-LT and Places-LT. These results demonstrate that GH are well designed to facilitate long-tailed representation learning and improve the generalization for downstream supervised tasks.

Cross-dataset transfer learning . To further demonstrate the representation transferability of our GH, we conduct more comprehensive experiments on the large-scale, long-tailed dataset CC3M [52] with various cross-dataset transferring tasks, including downstream classification, object detection and instance segmentation. Specifically, we report the finetuning classification performance on ImageNet, Places and fine-grained visual datasets Caltech-UCSD Birds (CUB200) [61], Aircrafts [45], Stanford Cars [33], Stanford Dogs [32], NABirds [60]. Besides, we evaluate the quality of the learned representation by finetuning the model for object detection and instance segmentation on COCO2017 benchmark [39]. As shown in Tables 4 and 5, we can see that our proposed GH consistently outperforms the baseline across various tasks and datasets. It further demonstrates the importance of considering long-tailed data distribution under large-scale unlabeled data in the pretraining stage. This can potentially be attributed to that our geometric harmonization motivates a more balanced and general emebdding space, improving the generalization ability of the pretrained model to a range of real-world downstream tasks.

4.4 Further Analysis and Ablation Studies

Dimension of Geometric Uniform Structure. As there is even no category number L available in SSL paradigm, we empirically compare our GH with different geometric dimension K on CIFAR100-LT-R100, as shown in Figure 3(a). From the results, GH is generally robust to the change of

10

50

100

150

200

400

800 16003200

Dimension

49

50

51

52

53

54

55

56

(a) Geometric Dimension

SimCLR

Focal

SDCLR

DnC

BCL

Method

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

NMI Score

Baseline

(b) Label Quality

0

0.2

0.4

0.6

0.8

1.0

N/A

Temperature

52.5

53.5

54.5

Prior

GH

(c) Label-Distribution Prior

Figure 3: (a) Linear probing performance w.r.t. the dimension K of the geometric uniform structure M (Appendix B) on CIFAR-LT-R100. (b) NMI score between the surrogate geometric labels and the ground-truth labels in the training stage on CIFAR-LT-R100. (c) Average linear probing and the error bars of the surrogate label allocation with variants of the label prior on CIFAR-LT-R100.

K , but slightly exhibits the performance degeneration when the dimension is extremely large or small. Intuitively, when K is extremely large, our GH might pay more attention to the uniformity among sub-classes, while the desired uniformity on classes is not well guaranteed. Conversely, when K is extremely small, the calibration induced by GH is too coarse that cannot sufficiently avoid the internal collapse within each super-class. For discussions of the structure, it can refer to Appendix B.

Surrogate Label Quality Uncovered. To justify the effectiveness of surrogate label allocation, we compare the NMI scores [54] between the surrogate and ground-truth label in Figure 3(b). We observe that GH significantly improves the NMI scores across baselines, indicating that the geometric labels are effectively calibrated to better capture the latent semantic information. Notably, the improvements of the existing works are marginal, which further verifies the superiority of GH.

Exploration with Other Label-Distribution Prior. To further understand π , we assume the ground-truth label distribution is available and incorporate the oracle π y into the surrogate label allocation. Comparing the results of the softened variants π y γ T with the temperature γ T in Figure 3(c), we observe that GH outperforms all the counterparts equipped with the oracle prior. A possible reason is that our method automatically captures the inherent geometric statistics from the embedding space, which is more reconcilable to the self-supervised learning objectives.

Uniformity Analysis. In this part, we conduct experiments with two uniformity metrics [62, 37]:

̸

̸

where j 1 , . . . , j k = i represent different classes. Specifically, U evaluates average distances between different class centers and U k measures how close one class is to its neighbors. As shown in Table 6, our GH outperforms in both inter-class uniformity and neighborhood uniformity when compared with the baseline SimCLR [9]. This indicates that vanilla contrastive learning struggles to achieve the uniform partitioning of the embedding space, while our GH effectively mitigates this issue.

Comparison with More SSL Methods. In Table 7, we present a more comprehensive comparison of different SSL baseline methods, including MoCo-v2 [24], MoCo-v3 [11] and various noncontrastive methods such as SimSiam [10], BYOL [20] and Barlow Twins [71]. From the results, we can see that the combinations of different SSL methods and our GH can achieve consistent performance improvements, averaging as 2.33 % , 3.18 % and 2.21 % on CIFAR-100-LT. This demonstrates the prevalence of representation learning disparity under data imbalance in general SSL settings.













wGH















On Importance of Bi-Level Optimization. In Table 8, we empirically compare the direct joint optimization strategy to Eq. (4). From the results, we can see that the joint optimization ( w/ or w/o the warm-up strategy) does not bring significant performance improvement over SimCLR compared with that of our bi-level optimization, probably due to the undesired collapse in label allocation [1]. This demonstrates the necessity of the proposed bi-level optimization for Eq. (4) to stabilize the training.

Qualitative Visualization. We conduct t-SNE visualization of the learned features to provide further qualitative intuitions. For simplity, we randomly selected four head classes and four tail classes on CIFAR-LT to generate the t-SNE plots. Based on the results in Figure 4, the observations are as follows: (1) SimCLR: head classes exhibit a large presence in the embedding space and heavily squeeze the tail classes, (2) GH: head classes reduce their occupancy, allowing the tail classes to have more space. This further indicates that the constructed surrogate labels can serve as the high-quality supervision, effectively guiding the harmonization towards the geometric uniform structure.

Sensitivity Analysis. To further validate the stability of our GH, We conduct empirical comparison with different weight w GH , temperature γ GH , regularization coefficient λ and Sinkhorn iteration E s on CIFAR-LT, as shown in Figures 5 and 7. From the results, we can see that our GH can consistently achieve satisfying performance with different hyper-parameter.

5 Conclusion

In this paper, we delve into the defects of the conventional contrastive learning in self-supervised long-tail context, i.e. , representation learning disparity, motivating our exploration on the inherent intuition for approaching the category-level uniformity. From the geometric perspective, we propose a novel and efficient Geometric Harmonization algorithm to counteract the long-tailed effect on the embedding space, i.e , over expansion of the majority class with the passive collapse of the minority class. Specially, our proposed GH leverages the geometric uniform structure as an optimal indicator and manipulate a fine-grained label allocation to rectify the distorted embedding space. We theoretically show that our proposed method can harmonize the desired geometric property in the limit of loss minimum. It is also worth noting that our method is orthogonal to existing self-supervised long-tailed methods and can be easily plugged into these methods in a lightweight manner. Extensive experiments demonstrate the consistent efficacy and robustness of our proposed GH. We believe that the geometric perspective has the great potential to evolve the general selfsupervised learning paradigm, especially when coping with the class-imbalanced scenarios.

References

Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In International Conference on Learning Representations (ICLR) , 2020.

Mido Assran, Randall Balestriero, Quentin Duval, Florian Bordes, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, and Nicolas Ballas. The hidden uniform cluster prior in self-supervised learning. In The Eleventh International Conference on Learning Representations , 2023.

Tanya Boone-Sifuentes, Asef Nazari, Imran Razzak, Mohamed Reda Bouadjenek, Antonio Robles-Kelly, Daniel Ierodiaconou, and Elizabeth S Oh. Marine-tree: A large-scale marine organisms dataset for hierarchical image classification. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management , pages 3838-3842, 2022.

Sergiy V Borodachov, Douglas P Hardin, and Edward B Saff. Discrete energy on rectifiable sets . Springer, 2019.

Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. Advances in Neural Information Processing Systems , 32:1567-1578, 2019.

Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV) , pages 132-149, 2018.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems , 33:9912-9924, 2020.

Mengxi Chen, Linyu Xing, Yu Wang, and Ya Zhang. Enhanced multimodal representation learning with cross-modal kd. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11766-11775, 2023.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning , pages 1597-1607. PMLR, 2020.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15750-15758, 2021.

Xinlei Chen*, Saining Xie*, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.02057 , 2021.

Henry Cohn and Abhinav Kumar. Universally optimal distribution of points on spheres. Journal of the American Mathematical Society , 20(1):99-148, 2007.

Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 715724, 2021.

Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems , 26, 2013.

Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision , pages 1422-1430, 2015.

Ziqing Fan, Yanfeng Wang, Jiangchao Yao, Lingjuan Lyu, Ya Zhang, and Qi Tian. Fedskip: Combatting statistical heterogeneity with federated skip aggregation. In 2022 IEEE International Conference on Data Mining (ICDM) , pages 131-140. IEEE, 2022.

Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences , 118(43):e2103091118, 2021.

Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann LeCun. On the duality between contrastive and non-contrastive self-supervised learning. In The Eleventh International Conference on Learning Representations , 2023.

Florian Graf, Christoph Hofer, Marc Niethammer, and Roland Kwitt. Dissecting supervised constrastive learning. In International Conference on Machine Learning , pages 3821-3830. PMLR, 2021.

Jean-Bastien Grill, Florian Strub, Florent Altch´ e, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems , 33:21271-21284, 2020.

DP Hardin and EB Saff. Minimal riesz energy point configurations for rectifiable ddimensional manifolds. Advances in Mathematics , 193(1):174-204, 2005.

John A Hartigan and Manchek A Wong. Algorithm as 136: A k-means clustering algorithm. Journal of the royal statistical society. series c (applied statistics) , 28(1):100-108, 1979.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9729-9738, 2020.

Feng Hong, Jiangchao Yao, Zhihan Zhou, Ya Zhang, and Yanfeng Wang. Long-tailed partial label learning via dynamic rebalancing. In The Eleventh International Conference on Learning Representations , 2023.

Chaoqin Huang, Haoyan Guan, Aofan Jiang, Ya Zhang, Michael Spratling, and Yan-Feng Wang. Registration based few-shot anomaly detection. In European Conference on Computer Vision , pages 303-319. Springer, 2022.

Chaoqin Huang, Qinwei Xu, Yanfeng Wang, Yu Wang, and Ya Zhang. Self-supervised masking for unsupervised anomaly detection and localization. IEEE Transactions on Multimedia , 2022.

Ziyu Jiang, Tianlong Chen, Bobak J Mortazavi, and Zhangyang Wang. Self-damaging contrastive learning. In International Conference on Machine Learning , pages 4927-4939. PMLR, 2021.

Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations , 2019.

Bingyi Kang, Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. In International Conference on Learning Representations , 2020.

Tejaswi Kasarla, Gertjan Burghouts, Max van Spengler, Elise van der Pol, Rita Cucchiara, and Pascal Mettes. Maximum class separation as inductive bias in one matrix. In Advances in Neural Information Processing Systems , volume 35, pages 19553-19566, 2022.

Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine-grained image categorization: Stanford dogs. In Proc. CVPR Workshop on Fine-Grained Visual Categorization (FGVC) , volume 2. Citeseer, 2011.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for finegrained categorization. In Proceedings of the IEEE international conference on computer vision workshops , pages 554-561, 2013.

A Krizhevsky. Learning multiple layers of features from tiny images. Technical Report TR2009, University of Toronto , 2009.

Anna Kukleva, Moritz B¨ ohle, Bernt Schiele, Hilde Kuehne, and Christian Rupprecht. Temperature schedules for self-supervised contrastive methods on long-tail data. In The Eleventh International Conference on Learning Representations , 2023.

Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsupervised representations. In International Conference on Learning Representations , 2021.

Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogerio S Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long-tailed recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6918-6928, 2022.

Jiye Liang, Liang Bai, Chuangyin Dang, and Fuyuan Cao. The k -means-type algorithms versus imbalanced data distributions. IEEE Transactions on Fuzzy Systems , 20(4):728-745, 2012.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll´ ar. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision , pages 2980-2988, 2017.

Hong Liu, Jeff Z HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. In International Conference on Learning Representations , 2021.

Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 212-220, 2017.

Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. Learning towards minimum hyperspherical energy. Advances in neural information processing systems , 31, 2018.

Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2537-2546, 2019.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Finegrained visual classification of aircraft. arXiv preprint arXiv:1306.5151 , 2013.

Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In International Conference on Learning Representations , 2021.

Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. Sampling Theory, Signal Processing, and Data Analysis , 20(2):1-13, 2022.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117(40):24652-24663, 2020.

William J Reed. The pareto, zipf and other power laws. Economics letters , 74(1):15-19, 2001.

Rasmus Rothe, Radu Timofte, and Luc Van Gool. Deep expectation of real and apparent age from a single image without facial landmarks. International Journal of Computer Vision , 126 (2-4):144-157, 2018.

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2556-2565, 2018.

Steve Smale. Mathematical problems for the next century. The mathematical intelligencer , 20 (2):7-15, 1998.

Alexander Strehl and Joydeep Ghosh. Cluster ensembles-a knowledge reuse framework for combining multiple partitions. Journal of machine learning research , 3(Dec):583-617, 2002.

Jianhua Sun, Yuxuan Li, Hao-Shu Fang, and Cewu Lu. Three steps to multimodal trajectory prediction: Modality clustering, classification and synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13250-13259, 2021.

Jianhua Sun, Yuxuan Li, Liang Chai, Hao-Shu Fang, Yong-Lu Li, and Cewu Lu. Human trajectory prediction with momentary observation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6467-6476, 2022.

Jianhua Sun, Yuxuan Li, Liang Chai, and Cewu Lu. Modality exploration, retrieval and adaptation for trajectory prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Joseph John Thomson. Xxiv. on the structure of the atom: an investigation of the stability and periods of oscillation of a number of corpuscles arranged at equal intervals around the circumference of a circle; with application of the results to the theory of atomic structure. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science , 7(39): 237-265, 1904.

Yonglong Tian, Olivier J Henaff, and A¨ aron van den Oord. Divide and contrast: Selfsupervised learning from uncurated data. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10063-10074, 2021.

Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 595-604, 2015.

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.

Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning , pages 9929-9939. PMLR, 2020.

Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE International Conference on Computer Vision , pages 27942802, 2015.

Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In International Conference on Learning Representations , 2021.

Chaoyi Wu, Feng Chang, Xiao Su, Zhihan Wu, Yanfeng Wang, Ling Zhu, and Ya Zhang. Integrating features from lymph node stations for metastatic lymph node detection. Computerized Medical Imaging and Graphics , 101:102108, 2022.

Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology. arXiv preprint arXiv:2308.02463 , 2023.

Yibo Yang, Liang Xie, Shixiang Chen, Xiangtai Li, Zhouchen Lin, and Dacheng Tao. Do we really need a learnable classifier at the end of deep neural network? arXiv preprint arXiv:2203.09081 , 2022.

Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalanced learning. Advances in neural information processing systems , 33:19290-19301, 2020.

Yuzhe Yang, Kaiwen Zha, Yingcong Chen, Hao Wang, and Dina Katabi. Delving into deep imbalanced regression. In International Conference on Machine Learning , pages 11842-11851. PMLR, 2021.

Jiangchao Yao, Bo Han, Zhihan Zhou, Ya Zhang, and Ivor W Tsang. Latent class-conditional noise model. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and St´ ephane Deny. Barlow twins: Selfsupervised learning via redundancy reduction. In International Conference on Machine Learning , pages 12310-12320. PMLR, 2021.

Fei Zhang, Lei Feng, Bo Han, Tongliang Liu, Gang Niu, Tao Qin, and Masashi Sugiyama. Exploiting class activation value for partial-label learning. In International Conference on Learning Representations , 2021.

Fei Zhang, Chaochen Gu, Chenyue Zhang, and Yuchao Dai. Complementary patch for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 7242-7251, 2021.

Ruipeng Zhang, Ziqing Fan, Qinwei Xu, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Grace: A generalized and personalized federated learning method for medical imaging. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 14-24. Springer, 2023.

Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Yuhang Zhou, Jiangchao Yao, Feng Hong, Ya Zhang, and Yanfeng Wang. Balanced destruction-reconstruction dynamics for memory-replay class incremental learning. arXiv preprint arXiv:2308.01698 , 2023.

Zhihan Zhou, Jiangchao Yao, Yan-Feng Wang, Bo Han, and Ya Zhang. Contrastive learning with boosted memorization. In International Conference on Machine Learning , pages 2736727377. PMLR, 2022.

Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems , 34:29820-29834, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors tackle the problem of contrastive learning in the context of imbalanced datasets where some classes have many more samples than other classes. Since no label information can be used, importance sampling is not an option and without any interventions, contrastive learning tends to overrepresent majority classes in the latent space, thereby hurting the readout accuracy on the minority classes. The authors propose a Geometric Harmonization technique in which they propose to estimate the class labels with clustering and then recalibrate the learned embeddings such that majority classes occupy the same feature space as minority classes. The authors show experimental results on CIFAR100-LT, ImageNet-LT and Places-LT.\n\nSTRENGTHS:\n- I appreciate the complexity analysis of the proposed method in Section 3.4.\n- The authors compare to several other benchmarks and show superior results with their method.\n- I think in general the method makes sense and it is intuitive that it should work in the proposed case.\n\nWEAKNESSES:\n- There are many typos and grammatical errors in the text which makes the paper hard to understand. Some mathematical definitions are wrong and inconsistent which makes me doubt the claims of the theoretical guarantees since there are even errors in the Lemmas / definitions:\n \n     - Line 94: “Let N denote the sample number and R = max_i n_i/min_i n_i denote the imbalanced ratio, where n_i denotes the number of samples in class i.” I am confused, the max and min are calculated per class i, which means that max_i n_i = min_i n_i = n_i, therefore R would be 1 for each class. Since I do not understand how R is defined, I cannot understand / judge the results in Table 2.\n     - Lemma 3.2 is inconsistent with itself. If n_L_H=n_H and n_L_H=n_T, then n_T = n_H and n_H / n_T -> infty is impossible because n_H / n_T = 1 by definition.\n    - Eq. 5: Using M both as the geometric structure and the size of the collection of negative samples is confusing and should not be done.\n\n- Considering the experimental results, I am not convinced that the authors have looked into the most important baselines. CIFAR100-LT, ImageNet-LT and Places-LT are well-established benchmarks and the baseline results that the authors aim to improve seem to be very low to me. I will elaborate on the different datasets in the following:\n     - For ImageNet-LT (https://paperswithcode.com/sota/long-tail-learning-on-imagenet-lt), the currently best number for a ResNet50 architecture is an accuracy of 70.1 (https://arxiv.org/pdf/2111.13579v4.pdf) with extra training data and 67.2 (https://arxiv.org/pdf/2111.14745v1.pdf) without extra training data. The best result the authors report in this paper is about 38% which is far below what is currently state of the art according to the benchmark list.\n     - For Places-LT, https://paperswithcode.com/sota/long-tail-learning-on-places-lt, the best numbers for a ResNet50 are above 45% while the authors here report numbers below 35%. \n\n-> I believe that in order for this method to be relevant to the community, the authors need to show results on the superior models with higher baseline accuracy. In the current state, it is not clear whether their results would generalize to the better models. \n\n- The motivation for the studied problem does not become clear to me from the introduction. The authors write: “However, the real-world natural sources usually exhibit the long-tailed distribution [31], and directly learning representation on them might lead to the distortion issue of the embedding space, namely, the majority dominates the feature regime [45] and the minority collapses [28].” Using the modal verb “might” here indicates a possibility but no further evidence is presented. Citing fairness research as an application is not enough in my opinion. From reading the introduction, I am not convinced that the problem the authors want to study actually exists in SSL. I would advise the authors to provide concrete examples where using SSL actually harms performance on minority classes. Currently, the main question of study posed in line 39 (“Why the conventional contrastive learning underperforms in self-supervised long-tailed context?”) does not seem well supported.\n\n- The first part of the first contribution is misleading and I believe wrong: “To our best knowledge, we are the first to investigate the drawback of the contrastive learning loss in self-supervised long-tailed context.” This drawback is investigated in several other papers which the authors discuss in their related work section, e.g. SDCLR specifically tackles this problem with a different method.\n\n- Figure 1: The spheres in the middle and right Figure look like ovals which is confusing. These should be spheres/ circles.\n\n- Abstract, line 5: „The attribution is that the vanilla SSL methods that pursue the sample-level uniformity easily leads to representation learning disparity, where head classes with the huge sample number dominate the feature regime but tail classes with the small sample number passively collapse.” -> This sentence is hard to parse and understand.\n\n\n\n\n- Table 1: What does IR stand for?\n\n- Line 119: “and π ∈ RK+ refers to the the marginal distribution constraint.” -> \"The\" twice and what do you mean under “marginal distribution constraint”?\n\n- The used datasets and models must be cited.\n\n\n- Line 162: “This phenomenon indicates all representations of the minority will collapse completely to one point without considering the category discrepancy, which corresponds to our observation regarding the passive collapse of tailed samples in Figure 1.” This statement is not accurate because the representations of the minority classes do not collapse to a single point in Figure 1. Lemma 3.2. covers an extreme case where there are infinitely more samples in the head classes compared to the tail classes, so Figure 1 does not represent this scenario and thus, using “corresponds” here is not accurate.\n\n\n\n- It is confusing that Table 7 comes before Tables 5+6, please fix.\n\n- Does Table 5 show the computational cost per epoch?\n\n- Line 218: “For hyper-parameters of GH, we provide a default setup across all the experiments: set the geometric dimension K as 100 and the temperature γGH as 0.1. In the surrogate label allocation, we set the regularization coefficient λ as 20 and Sinkhorn iterations Es as 300. Please refer to Appendix E.3 for more experimental details.” There are many hyperparameters that need to be set for this method, and it is not clear whether they were chosen on the test set, or how they were selected. It is also not clear how sensitive the algorithm is to these hyperparameters.\n\n- Line 223: “For comprehensive performance comparison, we present the linear probing performance and the standard deviation among three disjoint groups, i.e., [many, medium, few] partitions [25].” Please explain what the partitions into many/medium/few mean as it is not possible to understand the results otherwise.\n\n- Line 290: “To justify the intuition in Section 3.2” which intuition? Please be more specific.\n\n- I find the results in Fig. 3b unintuitive. The NMI scores show that GH is better aligned with ground truth labels compared to all other methods, but then why is the readout accuracy of GH only 1-2 percent points better compared to the other methods? Please discuss this as I am not sure how to interpret this result and whether it is meaningful to compare NMI scores between GH and the other methods.\n\nQUESTIONS:\n- How sensitive is GH to the choice of the hyperparameters?\n- Can you please comment on how the presented results on Places-LT and ImageNet-LT compare to the state-of-the-art results on https://paperswithcode.com/sota/long-tail-learning-on-places-lt and https://paperswithcode.com/sota/long-tail-learning-on-imagenet-lt?\n\nLIMITATIONS:\nThe authors could think a bit more about the impact of their work on applications related to long-tailed data distributions where minority and majority classes are present. How would applications situated in fairness research impacted by their work?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the SSL in a long-tailed distribution setting, where traditional contrastive learning may not work well.\nThe authors attribute that issue to the \"sample-level\" learning of SSL and propose Geometric Harmonization (GH) to regularize the \"category-level\" during training. GH can be a plug-and-play loss to the existing SSL frameworks without modification.\nEmpirical results of linear probing accuracy show the effectiveness of the proposed method in the long-tailed data over the baseline SimCLR.\n\nSTRENGTHS:\n+ Contribute to SSL capability in real-world scenarios with imbalanced classes which may be useful for unsupervised representation learning.\n+ Some theoretical analyses are provided to support the problem and method.\n+ The idea is reasonable\n+ Writing is clear enough\n\nWEAKNESSES:\n+ SSL has been developed with various methods to facilitate the representation learning for unlabelled data, this paper provided the experiments for one method, i.e. SimCLR, which is not a very strong baseline and not a state-of-the-art CL framework as claimed. \nFrom the presented results, SimCLR still works not too bad compared to SimCLR+GH in all settings. I would expect to see the effectiveness of GH in more strong SSL baselines including contrastive learning frameworks such as BYOL, MoCo-v3, etc, and maybe a more recent branch of SSL, namely mask autoencoder (MAE). To make the proposed method more complete, those baselines should be considered.\n+ The \"passive collapse\" of tail classes is mentioned in the paper but seems that except for some quantitative results showing GH improves baseline to some extent, there are no results (either visualization or quantitative metric) to show such collapse happens and that GH can mitigate or solve that problem.\n\nQUESTIONS:\n+ For transfer learning to downstream tasks, which dataset is pre-trained for SSL?\n+ From the concept/hypothesis in Fig. 2 bottom, when applying GH, the samples of each class are clustered well even though it is the imbalanced case. Is there visualization (t-SNE for example) for the real data that has been conducted in this paper to show that works as the figure of concept in Fig.2?\n\nLIMITATIONS:\nThey have discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The paper proposes to tackle the task of self-supervised learning on datasets sampled from long-tailed distributions. \n- Typical SSL approaches tend to perform a lot worse for the minority classes, thus hurting performance.\n- The proposed approaches termed Geometric Harmonization encourages category-level uniformity leading to better performance on various downstream tasks.\n- The approach is shown to be complementary to various existing SSL approaches.\n\nSTRENGTHS:\n- The paper tackles an important problem in SSL - that of training on data sampled from long-tailed distributions which is common in the real world. \n- The proposal of geometric harmonization is interesting. The idea is simple and easy to implement. While the approach is motivated by recent clustering-based works in SSL, the proposed approach seems novel.  \n- The paper is well written for the most part (some concerns are pointed out in other sections)\n- Authors perform extensive analyses and ablations to understand the working of the approach. \n- Experiments on various LT datasets show the benefit of using the approach.\n\nWEAKNESSES:\nFollowing are some of my concerns with the paper: \n- Clarity: Given that geometric harmonization is the key contribution of this work, I think the authors should give the readers more idea about the intuitions for the section on \"surrogate label allocation\". While the Figure 2 looks good, the caption is not very helpful to drive home the intuition. I see a similar issue with Definition 3.1 : meaning of K only becomes clear later in the paper. \n- Effect of batch-size : The authors have provided an analysis on the effect of batch size. I think such analysis will be especially interesting for datasets which have lot more labels : like ImageNet. \n- Discussion/comparison missing on some related works: [a], [b]\n\n\n[a] The hidden uniform cluster prior in self-supervised learning\n[b] Temperature Schedules for Self-Supervised Contrastive Methods on Long-Tail Data\n\nQUESTIONS:\n- Minor suggestions :\n    - L42 : reword \"as proof in [38]..\"\n    - \"sample number\" : I suggest using \"number of samples\" or a similar alternative instead to avoid any confusion. \n - Some more discussion and analysis on the observation in L298-300 would be really interesting. For this experiment, what happens if there is no support for a certain class in that mini-batch ?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper concentrates on the long-tailed distribution problem in SSL representation learning. To overcome the challenge in vanilla SSL methods, where head classes with the huge sample number dominate the feature regime, the authors propose the Geometric Harmonization (GH) method to encourage category-level uniformity. Specially, GH measures the population statistics of the embedding space, and then infer an fine-grained instance-wise calibration to constrain the space expansion of head classes while avoid the collapse of tail classes. Extensive results show the effectiveness of GH with high tolerance to the long-tailed distribution problem.\n\nSTRENGTHS:\n-1- This paper investigates the drawback of the contrastive learning loss in SSL long-tailed context, and shows that the resulted sample-level uniformity is an intrinsic limitation to the representation parity, as shown in Figure 1. The motivation of pursuing category-level uniformity is clear.\n\n-2- This paper is well-written and easy to follow. The proposed GH method that incorporates the geometric clues from the embedding space to calibrate the training loss, is interesting and solid.\n\n-3-  The proposed GH loss is versatile and can be easily plugged into existing SSL methods. The extensive experiments and ablation stidies demonstrate its effectiveness in learning robust representation.\n\nWEAKNESSES:\n-1- About the surrogate label allocation. What is the advantage to choose the discriminative clustering [1], what if we choose other clustering method such as K-means. How does the quality of the geometric label affect the final training results? When verifing the assumption that the constructed surrogate geometric labels are mutually correlated with the oracle labels, can the authors provide more intuitive results?\n\n-2- In E.q. 4, do you need to design how to balance these two losses?\n \n-3- Table 6 shows the results on class-balanced data. Can you explain why in some cases, incooperateing with GH will lead to the performance drop?\n\nQUESTIONS:\nPlease refer to *Weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied self-supervised representation learning with implicit class-imbalance data, in contrast to most prior works that assume class-balance. To do this, this paper proposed to augment the existing contrastive learning method with a novel geometric harmonization (GH). Intuitively, GH pulls samples toward the closest prototype in the embedding space, which differs from previous cluster-/prototype-based methods in the learnable prototypes and marginal prior. Through extensive experiments on several standard long-tail classification benchmarks, the authors demonstrated the efficacy of the proposed method with different contrastive learning methods as start points.\n\nSTRENGTHS:\n1. The paper is generally well-written and easy to follow.\n\n2. The paper is well-motivated. Class imbalance occurs in most real-world scenarios, and existing contrastive learning methods fail to take this into account. This paper provides a good exploration of this direction.\n\n3. The proposed geometric harmonization regularization is interesting. Though prototype-based constraints are already studied in contrastive learning [1, 5, 12], GH carefully considers the class imbalance by its design and is thus robust to such imbalance.\n\n4. The experiment results are strong compared with the baselines on imbalanced data. Moreover, the proposed method even performs on par with the baseline with balanced data.\n\nWEAKNESSES:\n1. The geometric harmonization relies heavily on the assumption that samples from the same class intrinsically lie in neighboring regions in the embedding space to estimate the assignment and prior. While this condition can be satisfied on curated datasets like CIFAR and ImageNet, it might be a different case for the uncurated datasets, e.g., YMCC or a subset of it. Some analysis on this end would further strengthen this paper.\n\n2. Lack of transfer learning experiments. In all the experiments, the models were pretrained and then finetuned on the same datasets. It is unclear whether the learned representations are general to transfer to other data distribution and different tasks. Possible options are adding cross-dataset experiments (e.g., YFCC-to-ImageNet-LT, ImageNet-Lt-to-CIFAR-LT) and transfer learning on MS-COCO object detection as in [5].\n\nQUESTIONS:\nSee the weakness section.\n\nLIMITATIONS:\nThe major limitation is that geometric harmonization relies heavily on the assumption that samples from the same class intrinsically lie in neighboring regions in the embedding space, which may not hold for real-world data.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the class-imbalance problem under the SSL setting. It shows that the Vanilla SSL methods pursuing the sample-level uniformity easily lead to representation learning disparity, where head classes with the huge sample number dominate the feature regime but tail classes with the small sample number passively collapse. It proposes Geometric Harmonization (GH) to encourage the category-level uniformity in representation learning. The GH loss can be easily integrated into existing SSL methods, and GH loss improve the performance over baselines on the imbalance class problem under the SSL setting, based on the experimental results.\n\nSTRENGTHS:\n1.This paper define the category-level uniformity in the embedding space that SSL learns.  The motivation to using category-level uniformity for imbalance class problem is clear. \n\n2.The proposed Geometric Harmonization (GH) loss improve the performance over baselines on the imbalance class problem under the SSL setting, based on the experimental results. \n\n3.The presentation of this paper is generally clear and easy to follow.\n\nWEAKNESSES:\n1. In the experiments of Section 4.3, it is better to conduct experiments on large-scale ImageNet pertaining using the proposed methods, and then transfer to other datasets (This is the so called “representation transferability”), rather than only: SSL pre-training and finetuning on the same datasets. \n\n2. I personally believe the theories and analyses both are specific to contrastive learning, rather than on other SSL methods (e.g., the non-contrastive SSL methods, BOYL, SiamSiam). Even though this paper conducts an experiment on other SSL methods in Section 4.4, it is not clear how the theory/analyses work for these SSL methods? Do the theoretic analyses still work for other SSL methods? This paper should at least mention it.   \n\n3. At last, I have concerns on whether the imbalance problem under the SSL is an important research topic. I definitely agree with that the class imbalance problem is very important. However, it is weird when considering the class imbalance problem under the SSL scenario. As everyone knows, SSL is considered as pre-training under large scale unlabeled data (class-free), and then transfer to downstream tasks (not specific for the classification problem). \nI believe the contributions are significant if this is the first paper to propose the imbalance problem under SSL setting. However, with several exists works as shown in the paper, I donot recognize the significance this paper contributes to the ML community. I think this paper will be stronger, if considering transfer to the class-imbalance object detection/semantic tasks.\n\nQUESTIONS:\nAddress Weakness 2 and 3.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-18"}