Combating Representation Learning Disparity with Geometric Harmonization
Authors: Zhihan Zhou
Jiangchao Yao
Feng Hong
Ya Zhang
Bo Han
Yanfeng Wang
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Long-tailed learning
self-supervised learning
TL;DR: We propose the novel Geometric Harmonization for self-supervised long-tailed learning, which overcomes the intrinsic limitation of the contrastive learning, i.e., sample-level uniformity, and progressively approaches the category-level uniformity.

Abstract: Self-supervised learning (SSL) as an effective paradigm of representation learning has achieved tremendous success on various curated datasets in diverse scenarios. Nevertheless, when facing the long-tailed distribution in real-world applications, it is still hard for existing methods to capture transferable and robust representation. The attribution is that the vanilla SSL methods that pursue the sample-level uniformity easily leads to representation learning disparity, where head classes with the huge sample number dominate the feature regime but tail classes with the small sample number passively collapse. To address this problem, we propose a novel Geometric Harmonization (GH) method to encourage the category-level uniformity in representation learning, which is more benign to the minority and almost does not hurt the majority under long-tailed distribution. Specially, GH measures the population statistics of the embedding space on top of self-supervised learning, and then infer an fine-grained instance-wise calibration to constrain the space expansion of head classes and avoid the passive collapse of tail classes. Our proposal does not alter the setting of SSL and can be easily integrated into existing methods in a low-cost manner. Extensive results on a range of benchmark datasets show the effectiveness of \methodspace with high tolerance to the distribution skewness.

Combating Representation Learning Disparity with Geometric Harmonization
Zhihan Zhou' Jiangehao Yao! Feng Hong' Ya Zhang Bo Han Yanfeng Wang" 122
CCooperative Medianet Innovation Center, Shanghai Jiao Tong University
Shanghai AI Laboratory 'Hong Kong Baptist University
(zhihanzhou, Sunarker, feng.hong, hong, ya_zhang, wangyanfeng" justu. edu. bhanm1@comp. hkbu. edu.hk hkbu.edu.h.
Abstract
Self--upervised learning (SSL) as an effective paradigm of representation learning has achieved tremendous success on various curated datasets in diverse scenarios. Nevertheless, when facing the long-tailed distribution in real-world applications, it is still hard for existing methods to capture transferable and robust representa- tion. Conventional SSL methods, pursuing sample-level uniformity, easily leads to representation learning disparity where head classes dominate the feature regime but tail classes passively collapse. To address this problem, we propose a novel Geometric Harmonization (GH) method encourage category-level uniformity in representation learning, which is more benign to the minority and almost does not hurt the majority under long-tailed distribution. Specially, GH measures the pop- ulation statistics of the embedding space on top of self-supervised learning, and then infer an fine-grained instance-wist calibration to constrain the space expan- sion of head classes and avoid the passive collapse of tail classes. Our proposal does not alter the setting of SSL and can be easily integrated into existing methods in low-cost manner. Extensive results on a range of benchmark datasets show the effectiveness of GH with high tolerance to the distribution skewness. Our code is available at
Harmmniiioiaa
Introduction
Recent years have witnessed great success of self-supervised learning to learn generalizable rep- resentation [7, 9, 15, 63]. Such rapid advances mainly benefit from the elegant training on the label- free data, which can be collected in large volume. However, the real-world natural sources usually exhibit the long-tailed distribution [50], and directly learning representation on them can lead distortion issue of the embedding space, namely, the majority dominates the feature regime [75] and the minority collapses [47]. Thus, becomes urgent to pay attention to representation learning disparity, especially as fairness of machine learning draws increasing attention [28. 41, 68, 77].
Different from the flourishing supervised long-tailed learning [29, 46, 68), self-supervised learning under long-tailed distributions is still under-explored, since there is no labels available for the cali- bration. Existing explorations to overcome this challenge mainly resort to the possible tailed sample discovery and provide the implicit bias to representation learning. For example, BCL [77] leverages the memorization discrepancy of deep neural networks (DNNs) on unknown head classes and tail classes to drive an instance-wise augmentation. SDCLR [28] contrasts the feature encoder and its pruned counterpart to discover hard examples that mostly covers the samples from tail classes, and efficiently enhance the learning preference towards tailed samples. DnC [59] resorts divide- and-conquer methodology to mitigate the data-intrinsic heterogeneity and avoid the representation
The corresponding authors are Jiangehao Yao and Yanfeng Wang.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Figure/ 1: Comparison of Geometric Harmonization and the plain SSL method on 2-D imbalanced synthetic dataset. (Left)) Visualization of the 2-D synthetic dataset. (Middle) The embedding distri- bution of each category learnt by the vanilla contrastive learning loss is approximately proportional to the number of samples, leading to the undesired representation learning disparity. (Right) GH mitigates the adverse effect of class imbalance and approaches to the category-level uniformity.
collapse of minority classes. Liu et al. [41] adopts a data-dependent sharpness-awan minimization scheme to build support to tailed samples in the optimization. However, few works hitherto have considered the intrinsic limitation of the widely-adopted contrastive learning loss, and design the corresponding balancing mechanism to promote the representation learning parity.
We rethink the characteristic of the contrastive learning loss, and try to understand "Why the con- ventional contrastive learning underperforms in self-supervised long-tailed context?" To answer this question, let consider two types of representation uniformity: (1) Sample-level uniformity. As stated in [62], the contrastive learning targets to distribute the representation of data points uni- formly in the embedding space. Then, the feature span of each category is proportional to their corresponding number of samples. (2) Category-
uniformity. This uniformity pursues to split the region equally for different categories without considering their corresponding number of sam- ples [49, 19]. In the class-balas balanced scenarios, the former uniformity naturally implies the latter uniformity, resulting in the equivalent separability for classification. However, in the long taigetalled distributions, they are different: sample-level uniformity leads to the feature regime that is biased towards the head classes considering their dominant sample quantity and sacrifices the tail classes due to the limited sample quantity. By contrast, category-level uniformity means the equal allocation w.r.t classes, which balances the space of head and tail classes, and is thus more benign the down- stream classification [17, 19, 37J. Unfortunately, there is no support for promoting category-level uniformity in contrastive learning loss, which explains the question arisen the beginning.
In this study, we propose a novel method, termed as Geometric Harmonization (GH) to combat rep- resentation learning disparity in SSL under long-tailed distributions. Specially, GH uses a geometric uniform structure to measure the uniformity of the embedding space in the coarse granularity. Then, surrogate label allocation is computed provide fineggrained instance-wiss calibration, which explicitly compresses the greedy representation space expansion of head classes, and constrain the passive representation space collapse of tail classes. The alternation in the conventional loss refers to an extra efficient optimal- formity. In Figure 1, we give a toy experiment' visualize the distribution of the embedding space without and with GH. In nutshell, our contributions can be summarized as follows,
optimization that dynamically pursues the category-level uni-
1. To our best knowledge, we are the first to investigate the drawback of the contrastive learn- ing loss in self-supervised long-tailed context and point out that the resulted sample-level uniformity is an intrinsic limitation to the representation parity, motivating to pursue -etegory-level uniformity with more benign downstream generalization (Section 3.2).
2. We develop a novel and efficient Geometric Harmonization (Figure 2) to combat the rep- resentation learning disparity in SSL, which dynamically harmonizes the embedding space SSL to approach the category-level uniformity with the theoretical guarantee.
3. Our method can be easily plugged into existing SSL methods for addressing the data im- balance without much extra cost. Extensive experiments on a range of benchmark datasets demonstrate the consistent improvements in learning robust representation with our GH.
For more details, please refer Appendix E.3.

---

2 Related Work
Self-Supervised Long-tailed Learning. There are several recent explorations devoted to this di- rection [40, 28, 59, 41, 77]. BCL [77] leverages the memorization effect of DNNs to automatically drive an instance-wise augmentation, which enhances the learning of tail samples. SDCLR [28] constructs a self-contrast between model and its pruned counterpart to learn more balanced repre- sentation. Classic Focal loss [40] leverages the loss statistics to putting more emphasis on the hard examples, which has been applied to self-supervise long-tailed learning [77]. DnC [59] benefits from the parameter isolation of multi-experts during the divide step and the information aggrega- tion during the conquer step to prevent the dominant invasion of majority. Liu et al. [41] proposes to penalize the sharpness surface in a reweighting manner calibrate class-imbalance learning- Recently, TS [35] employs dynamic strategy the temperature factor of contrastive loss, harmo- nizing instance discrimination and proup-wise discrimination. PMSN [2] proposes the power-law distribution prior, replacing the uniform prior, to enhance the quality of learned representations.
Hyperspherical Uniformity. The distribution uniformity has been extensively explored from the physic area, e.g., Thomson problem [58, 53]. to machine learning area like some bernel-based ex- tensions, e.g., Riesz s-potential [21. 43] or Gaussian potential [12, 62]. Some recent explorations regarding the features of DNNs [49, 17, 47] discover a terminal training stage when the embedding collapses to the geometric means of the classifier w.r.f. each category. Specially, these optimal class means specify perfect geometric uniform structure with clear geometric interpretations and gen- cralization guarantees [78, 67. 31]. In this paper, we first extend this intuition into self-supervised learning and leverage the specific structure combat the representation disparity in SSL.
3 Method
3.1 Problem Formulation
We denote the dataset DD for each data point (x.,) E D, the input E IR" and the associated label y E {L..... L}. Let V denote the number of samples, balanced ratio (IR), where Nmax, "mmas,Nmin Nimn is the number of samples in the largest and smallest class, respectively. Let n, denote the number of samples in class i. In SSL, the ground-truth y can not be accessed and the goal is to transform In the linear probing evaluation, we construct a supervised learning task with balanced datasets. A linear classifier g(-) is built on top of the frozen fo(.) to produce the prediction, i.e., g(fo(x)).
= Nmaz//N /Nmn denote the im-
image
embedding via DNNs, i.e., fo R + Rd
3.2 Geometric Harmonization
As aforementioned, most existing SSL methods in long-tailed context leverage the contrastive learn- ing loss, which encourages the sample-level uniformity in the embedding space. Considering the intrinsic limitation illustrated in Figure 1, we incorporate the geometric clues from the embedding space to calibrate the current loss, enabling pursuit of category-level uniformity. In the follow- ing, we first introduce a specific geometrie structure that is critical to Geometric Harmonization.
Definition 3.1. (Geometric Uniform Structure). Given a of vertices M E k the geometric uniform structure satisfies the following between-class duality
M. M, = C, Vi,j e (1, K}, i
(I)
where | = 1, vi (1, ... K), K is the number of geometric vertices and C' is constant.
Above structure provides a characteristic that any two vectors in M have the same angle, namely, the unit space are equally partitioned by the vectors in M. This fortunately follows our expectation about category-level uniformity. Specially, we use M as a constant classifier to involve into training, and have the oracle label y of the long-taine tailed data (K L) to supervise the below prediction
K
q = p= M) exp (M.. fo(x)//on)
Σ exp (M. Jollt)/ran)
(2)
then according to the neural collapse theory [49], the representation of samples will fit the geo- metric uniform structure of M in the limit, namely, approach category-level uniformity. However,
3

---

Long-tailed data
Contrastive Learning
Geometric Harmonization
π
Marginal distribution constraint
Head
LInfoNCE
LGC
Class index
Bi-Level Optimization
From Sample-Level Uniformity Category-Level Uniformity
Q
min IniaNCE Las #.9
Geoomerrc Unifarm Structure
jol Surrogate Label Allocation
Figure 2: Overview of Geometric Harmonization (w/ InfoNCE). To achieve harmonization with the category-level uniformity, we require ground-truth labels for supervision during training. How- ever, these labels are not available under the self-supervised paradigm. Moreover, estimating surro- gate labels directly from the geometric uniform structure is challenging and noisy, especially when the representation is not ideally distributed. To fullfiil this gap, we utilize the geometric uniform structure to measure the embedding space, and the captured population statistics are used for an instance-wis wise calibration by surrogate label allocation, which provides supervision feedback to counteract the sample-level uniformity. Specially, our method are theoretically grounded to ap- proach category-level uniformity the loss minimum. The additional model parameters incurred by GH are analytically and empirically demonstrated to trained in an efficient manner.
the technical challenge is the complete absence of annotation in our context, making directly con- structing the objective min Ely log q) with Eq. (2) impossible to combat the representation disparity.
Surrogate Label Allocation. To address the problem of unavailable labels, we explore constructing the surrogate geometric labels to supervise the training of Eq. (2). Concretely, we utilize the recent discriminative clustering idea [1] to acquire such geometric labels, formulated as follows
min Q=(q.....9N)
LGH
Σ WIND
qi log qı s.t. Q- N Q 1 1N,
(3)
D
where q: refers the soft assignment constrained in K -ddimensional probability simplex and π E I refers to the distribution constraint. As we are dealing with long-tailed data, we propose the geometric uniform structure M automatically compute the population statistic of the embedding space as л. Finally, Eq. (3) can be analytically solved by Sinkhorn-Knopy algorithm [14] (refer to Appendix D for Algorithm 1). Note that, the above idea builds upon conjecture: the constructed surrogate geometric labels are mutually correlated with the oracle labels so that they have the similar implication to approach category-level uniformity. We will empirically verify the rationality of such an assumption via normalized mutual information in Section 4.4.
Overall Objective. Eq. (3) can be easily integrated into previous supervupe long-tailed learning methods for geometric harmonization, e.g., SDCLR [28] and BCL [77]. For simplicity, we take their conventional InfoNCE loss [48] example and write the overall objective follows
min C LInfONCE + WCH 8.Q
where WCH represents the weight of the geometric harmonization loss. Optimizing Eq. (4) refers bi-level optimization style: in the outer-loop. optimize ming LGH with fixing compute the
surrogate geometric labels; in the inner-loop, optimize minş LIafoNCE + LCh with fixing Q to learn the representation model. The additional cost compared to the vanilla method primarily arises from from the loop,- which will be discussed in Section 3.4 and verified Section 4.4.

---

Compared with previous explo- rations [1, 71, the uniqueness of GH lies in the following three aspects: (1) Geometric Uniform Structure. The pioneering works mainly resort to learnable classifier to perform cluster- ing, which can easily be distorted in the long-tailed scenarios [17]. Built on
Table Linear probing of vanilla discriminative cluster- ing methods and variants of GH on CIFAR-100
IR
SimCLR
+SeLA
+SwAV
+GH
w/o GUS
w/o
100
50.7
50.5
52.2
54.0
53.3
53.1
50
52.2
52.0
53.0
55.4
54.6
54.4
10 55.7
56.0
56.1
57.4
56.7
56.4
the geometric uniform classifier, method is capable to provide high-quality clustering results with clear geometric interpretations. (2) Flexible Class Prior. The class prior л is assumed to be uniform among the previous attempts. When moving to the long-tailed case, this assumption will strengthen the undesired sample-level uniformity. In contrast, our methods can potentially cope with any distribution with the automatic surrogate label allocation. (3) Theoretical Guarantee. GH is theoretically grounded to achieve the category-level uniformity in the long-tailed scenarios (refer to Section 3.3), which has never been studied in previous methods. To gain more insights into method, we further compare GH with discriminative clustering methods (SeLA [1]. SWAV [7]) and investigate the impact of various components in GH. From the results in Table 1, we can see that
GH consistently outperforms the vanilla discriminative clustering baselines in the linear probing evaluation. Notably, we observe that both GUS and the class prior play critical role in our GH.
3.3 Theoretical Understanding
Here, we reveal the theoretical analysis of GH on promoting the representation learning to achieve category-level uniformity instead of sample-level uniformity. Let us begin with a deteriorated case of sample-level uniformity under the extreme imbalance, i.e., minority collapse [17].
Lemma 3.2. (Minority collapse) Assume the samples follow the uniform distribution nı n2 =
1LH n.H BLg+1 1L =
nt. =wr in head and tail classes respectively.
Assume d 2 and n n - +0C, the lower bound (Lemma C.I) of LIatONCE achieves the identical vector:
minimum when the class means of the tail classes collapse
lim ph: phj = Or, VLB Si SiSjj i.S. L,
where μ m Ei fo(2), denotes the class means and æi is the i-th data point with label I.
This phenomenon indicates all representations of the minority will collapse completely to one point without considering the category discrepancy, which aligns with observation regarding the pas- sive collapse of tailed samples in Figure 1. To further theoretically analyze GH, we first quantita- tively define category-level uniformity in the following. and then theoretically claim that with the geometric uniform structure (Definition. 3.1) and the perfect aligned allocation (Eq. (3)), we can achieve the loss minimum at the stage of realizing category-level uniformity.
Definition 3.3. (Categorical-level Uniformity) We define categorical-level uniformity on the em- bedding space w.r. the geometric uniform structure M when it satisfies
f = M3, Vk 1, ......... K,
where ph = + En (ax.;) represents the class mean for samples assigned with the surrogate geometric label ls in the embedding space.
Theorem 3.4. (Optimal state for L) Given Eq. (4) under the proper optimization strategy, when it arrives the category-level uniformity (Definition 3.3) defined on the geometric uniform structure M (Definition 3.1), we will achieve the minimum of the overall loss L as
L L' 2'= .. log (1/ + (K(+(K+K))) (K-1))xx(CC-
- 1)))
log (J/L).
(5)
where J denotes the size of the collection of the negative samples and n refers to the marginal distribution of the latent ground-truth labels y.
This guarantees the desirable solution with the minimal intra- claas covariance and the maximal inter- class covariance under the geometric uniform structure [49, 17], which benefits the downstream gen- eralization. Notably, no matter the data distribution is balanced or not, our method can persistently maintain the theoretical merits calibrating the class means to achieve category-level uniformity. We also empirically demonstrate the comparable performance with GH on the balanced datasets in Section 4.4, as in this case category-level uniformity is equivalent to sample-level uniformity.
5

---

3.4 Implementation and Complexity analysis
In Algorithm 2 of Appendix D, we give the complete implementation of our method. One point that needs to be clarified is that we learn the label allocation q in the mini- manner. In addition, the geometric prediction q and the adjusted q are computed at the beginning of every epoch as the population-level statistic will not change much in few mini-batches. Besides, we maintain momentum update mechanism to track the prediction of each sample to stabilize the training. i.e, qdd <- fqq + (1 B)q. When combined with the joint-embedding loss, we naturally adopt cross-supervision mechanism min i log q] for the reconciliation with contrastive baselines. The proposed method is illustrated in Figure 2 for visualization.
For complexity, assume that the standard optimization of deep neural networks requires forward and backward step in each bani-batch update with the time complexity as (BA),, where B is the mini- batch size and A is the parameter size. At the parameter level, we add an geometric uniform structure with the complexity as O ((Bd)), where K is the number of geometric labels and d is the embedding dimension. For Sinkhorn- -nopp algorithm, only refers to simple matrix-vector multiplication as shown in Algorithm 1, whose complexity is O (B + K + BK)) with the iteration step E,,- The complexity incurred in the momentum update is (BK). Since K, d and F, are significantly smaller than the model parameter A of a million scale, the computational overhead involved in GH is negligible compared to O(BA). The additional storage for a mini-batch of samples is the matrix Q" E K which is also negligible to the total memory usage. To the end, GH incurs only a small computation or memory cost and thus can be plugged to previous methods in a low-cost manner. The empirical comparison about the computational cost is summarized in Table 17.
4 Experiments
4.1 Experimental Setup
Baselines. We mainly choose five baseline methods, including (1) plain contrastive learning: Sim- CLR [9]. (2) hard example mining: Focal [40], (3) asymmelric network pruning: SDCLR [28), (4) multi-expert ensembling: DnC [59], (5) memorization- guided augmentation: BCL [77]. Empirical comparisons with more baseline methods can be referred to Appendix F.3.
Implementation Details. Following previous works [28, 771. we use ResNet-18 [23] as the backbone for small-scale dataset (CIFAR-100-LT [5]) and ResNet-50 [23] for large-scale datasets (ImageNet-LT [44], Places-LT [44]). For experiments
train model with the SGD optimizer, batch size 512, momentum 0.9 and weight decay factor 5 x 10-4 for 1000 epochs. For experiments on ImageNet-LT and Places-LT, we only train for 500 epochs with the batch size 256 and weight decay factor 1 x 10-4 For learning rate schedule, we use the cosine annealing decay with the learning rate 0.5 - 1e-6 for all the baseline methods. As GH is com- bined with baselines, a proper warming-up of 500 epochs on CIFAR- 100-LT and 400 epochs on ImageNet-LLT and Places-LT are applied. The cosine decay is set as 0.5 0.3, 0.3 1e-6 respec- tively. For hyper parameersssssrss of GH, we provide a default setup across all the experiments: set the geometric dimension K as 100, UGH as I and the temperature YGH as 0.1. In the surrogate label allocation, we set the regularization coefficient À as 20 and Sinkhorn iterations E, as 300. Please refer to Appendix E.3 for more experimental details.
CIFAR-100-LT,
Evaluation Metrics. Following [28,7] 77], linear probing on a balanced dataset is used for evaluation. We conduct full-shot evaluation on CIFAR- 100-LT and few-shot evaluation ImageNet-Ii LT and Places-LT. For comprehensive performance comparison, we present the linear probing performance and the standard deviation among three disjoint groups, i.e., [many, medium, few] partitions [44).
4.2 Linear Probing Evaluation
CIFAR-100-LT. In Table 2, we summarize the linear probing performance of baseline methods w/ and w/o GH range of benchmark datasets, and provide the analysis follows.
(1) Overall Performance. GH achieves the competitive results w.r. the [many, medium, few] groups, yielding overall performance improvements averaging as 2.32%, 2.49% and 1.52% on CIFAR-100 LT with different imbalanced ratios. It is worth noting that on the basis of the previous state-of-the- art BCL, our GH further achieves improvements by 1.20%, 1. 1.82% and 1 1.22%, respectively. Our

---

Table 2: Linear probing results on 1IFAR-100-LT with different imbalanced ratios (100, 50, 10), ImageNet-I.T and Places-LT. Many/Med/Few (1) indicate the average accuracy (%) w.r. fine-grained partitions according to the class cardinality. Std (L) means the standard deviation of the group-level performance and Avg (f) is the average accuracy (%) of the full test set. Following the previous work [28, 77J. Std represents balancedness measure quatity the variance among three specified groups. Improv. (f) represents the averaging performance improvements w.E.t. different baseline methods. We report the error bars with the multi-run experiments in Table 12 in Appendix F.
Dataset
SimCL.
+GH
Focal
+GH
SDCLR
+GH
DnC
+GH
BCL.
+GH
Improv.
CIFAR-RI00
Many
54.97
57.38
54.24
57.01
57.32
57.44
55.41
57.56
59.15
59.50
+1.56
Med
49.39
52.27
49.58
52.93
50.70
52.85
51.30
53.74
54.82
55.73
+2.35
Few
47.67
52.12
49.21
51.74
50.45
54.06
50.76
53.26
55.30
57.67
+3,09
Std
3.82
2.99
2.80
2.76
3.90
2.38
2.54
2.36
2.37
1.89
-0.61
Avg
50.72
53.96
51.04
53.92
52.87
54.81
52.52
54.88
56.45
57.65
+2.32
CTFAR R50
Many
56.00
58.88
55,40
57.97
57.50
58.47
56.03
59.04
59.44
60.82
42.16
Med
50.48
53,00
51.14
53.55
51.85
53.88
52.68
55.05
54.73
57.58
+2.44
Few
50.12
54.27
50.02
53.58
52.15
53.58
50.83
54.81
57.30
58.55
+2.87
Std
3.30
3.09
2.84
2.54
3.18
2.74
2.64
2.38
2.36
1.66
-0.38
Avg
52.24
55.42
52.22
55.06
53.87
55.34
53.21
56.33
57.18
59.00
+2,49
CIFAR R10
Many
57,85
59.26
58.18
60.06
58.47
59.21
59,82
61.09
60.41
61.41
+1.26
Med
55.06
56.91
55.82
56.79
54.79
56.06
56.67
58.33
57.15
59.27
+1.57
Few
54.03
55.85
54.64
57.24
52.97
55.58
56,21
57.33
59.76
60.30
+1.74
Std
1.98
1.75
1.80
1.77
2.80
1.97
1.96
1.95
1.73
1.07
-0.35
Avg
55.67
57.36
56.23
58.05
55.44
56.97
57.59
58.94
59.12
60.34
+1.52
Many
41.69
41.53
42.04
42.55
40.87
41.92
41.70
19
42.92
43.22
+0.44
ImageNet-L1
Med
33,96
36.35
35.02
36.75
33.71
36.53
34.68
36.63
35.89
38.16
+2.23
Few
31.82
35.84
33.32
36.28
32.07
36.04
33.58
35.86
33.93
36.96
+3.25
Std
5.19
3.15
4.62
3.49
4.68
3.26
4.41
3.45
4.73
3.32
-1.39
Avg
36.65
38.28
37.49
38.92
36.25
38.53
37,23
38.67
38.33
39.95
+1.68
Many
31.98
32.46
31.69
32.40
32.17
32.78
32.07
32.51
32.69
33.22
+0.55
Places-LT
Med
34.05
35.03
34.33
35.14
34.71
35.60
34.51
35.55
35.37
36.00
+0.87
Few
35.63
36.14
35.73
36.49
35.69
36.18
35.84
35.91
37.18
37.62
+0.45
Std 1.83
1.89
2.05
2.08
1.82
1.82
1.91
1.87
2.26
2.23
0.00
Avg 33.61
34.33
33.65
34.42
33.99
34.70
33.90
34.52
34.76
35.32
+0.68
GH consistently improves performance across datasets with varying degrees of class imbalance, demonstrating its potential to generalize to practical scenarios with more complex distributions. Specially, our method does not require any prior knowledge or assumptions about the underlying data distribution, highlighting the robustness and versatility to automatically adapt to the data.
(2) Representation Balancedness. In Section we claim that GH helps compress the expansion of head classes and avoid the passive collapse of tail classes, yielding the more balanced representation distribution. To justify this aspect, we compare the variance in linear probing performance among many/mediumm/few groups, namely, their groupwise standard deviation. According to Table 2, our GH provides [1.56%, 2.35%, 3.09%). 16. 2.44%, 2.87%] and [1 [1.26%, 1.57%. 1. 1.74%] improve- ments W.E. [many, medium, few] groups on CIFAR- SFFAR-10-LTER 100-LT-RII to the minority classes for representation balancedness. Overall, GH substantially improves the
WAL.TTROROOOOOOOA 10/R 10 with more preference standard deviation by [0.61, 0.38, 0.35] on different levels of imbalance.
ImageNet-LT and LL. Table 2 shows the comparison of different baseline methods large- scale dataset ImageNet-L.T" and Places-LT,, in which we have consistent observations. As can be seen, on more challenging real-world dataset, GH still outperforms other methods in terms of overall accu- racy, averaging as 1.68% 0.68% on ImageNet-LD LT and Places-LT. Specifically, our method provides [0.44%, 2.23%. 3.25%] and [0.55% 0.87%, 0.45%] improvements in linear probing w.E.f. [many, medium, few] groups on ImageNet-LD and Places-LT. The consistent performance overhead indi- cates the robustness of our method deal with long-tailed distribution with different characteristics. Moreover, the averaging improvement of standard deviation is 1.39 on ImageNet-Li: indicating the comprehensive merits of our GH on the minority classes towards the representation balancedness. However, an interesting phenomenon is that the fine-grained performance exhibits different trend on Places-LT. As can be seen, the performance of head classes is even worse than that of tail classes. The lower performance of the head partition can be attributed to the fact that it contains more chal-
7

---

Table 3: Supervised long-tailed learning by finetuning on CIFAR- 100-L-,, ImageNet-LT and Places- LT. We compare the performance of five self-supervised learning methods as the pre-training stage for downstream supervised logit adjustment [46] method. Improv. (f) represents the averaging per- formance improvements w.r.t. different baseline methods. Besides, the performance of logit adjust- ment via learning from scratch is also reported for comparisons.
Logit adjustment pretrained with the following SSL methods
Dataset
LA
Improv.
SimCLR
+GH
Focal
+GH
SDCLR
+GH
DnC
+GH
BCL
+GH
CIFAR-LT
46.61
49.81
50.84
49.83
51.04
49.79
50.73
49.97
50.84
50.38
51.32
+1.00
ImageNet-11
48.27
51.10
51.67
51.15
51.82
50.94
51.64
51.31
51.88
51.43
52.06
+0.63
Places-LT
27.07
32.63
33.86
32.69
33.75
32.55
34.03
32.98
34.09
33.15
34.48
+1.24
Table 4: Image classification on ImageNet, Places and fine-grained visual classification on various fine-grained datasets, pretrained on large-scale long-tailed CC3M and then finetuned.
Image Classification
Fine-Grained Visual Classification
ImageNet
Places
CUB200
Aireraft StanfordCars
StanfordDogs
NABirds
Average
SimCLR
52.06
37.65
44.61
65.89
57.63
50.99
46.86
53.20
+GH
53.39
38.47
45.76
68.08
60.24
52.88
47.58
54.91
lenging classes. As result, observe that the standard deviation of our GH does not significantly decrease on Places-LT, which requires more effort and exploration for improvement alongside GH.
4.3 Downstream Finetuning Evaluation
Downstream supervised long-tailed learning, Self-supervised learning has been proved to be beneficial as a pre-training stage of supervised long-tailed recognition to exclude the explicit bias from the class imbalance [68, 41, 77].
To validate the effectiveness of our GH, we conduct supf-supe pre-training as the initialization for downstream supervised classification tasks on LT. The sfate-of-the-ard logit adjustment [46] is chosen the downstream baseline. The combination of GH LA can be interpreted as compounded method where GH aims at the re-balanced repre- sentation extraction and LA targets the classifier debiasing. In Table 3, we can find that the superior performance improvements are achieved by self-supervised pre-training over the plain supervised learning baseline. Besides, our method can also consistently outperform other SSL baselines, av- eraging 1.00%, 0.63% and 1.24% results demonstrate that GH are well designed to facilitate long-tailed representation learning and improve the generalization for downstream supervised tasks.
ImageNet-LT and Places-
10FAR--000-L-RI 100, ImageNet-LT and Places-LT. These
Cross-dataset transfer learning. To further demonstrate the representation transferability of our GH, we conduct more comprehensive experiments on the large-scale, long-tailed dataset CC3M [52] with various cross-dataset transferring tasks, including downstream classification, object detection and instance segmentation. Specifically, we report the finetuning classification performance on Ima- geNet, Places and fine mne-grained visual datasets Caltech-
Birds (CUB200) [61], Aircrafts [45], Stanford Cars [33], Stanford Dogs [32], NABirds [60]. Besides, we evaluate the quality of the learned representation by finetuning the model for object detection and instance segmentation on COC02017 benchmark [39]. As shown in Tables 4 and 5, we can see that our proposed GH con- sistently outperforms the baseline across various tasks and datasets. It further demonstrates the importance of considering long-tailed data distribution under large-scale unlabeled data in the pre- training stage. This can potentially be attributed to that our geometric harmonization motivates a more balanced and general emebdding space, improving the generalization ability of the pretrained model to a range of real-world downstream tasks.
4.4 Further Analysis and Ablation Studies
Dimension of Geometric Uniform Structure. As there is even category number L available in SSL paradigm, we empirically compare our GH with different geometric dimension K on CIFAR- 100-LT-RI 100, as shown in Figure 3(a). From the results, GH is generally robust to the change of
8

---

Table 5: Object detection and instance segmentation with Table 6: Inter-class uniformity (t) and finetuned features on COC02017 benchmark, pretrained neighborhood uniformity (f) of pretrained on large-scale long-tailed CC3M.
features on CIFAR-LT.
Object Detoction
Instance Segmentation
Inter-class Uniformity
Noighborhood Uniformity
SimCLR
SimCLR
+GH
Aplioz
AP
Apmnaak
APggaa
AP
C100
1.00
2.80
0.72
2.00
SimCLR
31.7
51.0
33.9
30.2
49.8
32.1
C50
1.23
2.73
0.91
1.94
+GH
32.7
52.2
35.2
31.1
50.8
33.0
C10
1.18
2.60
0.85
1.83
56
0.7
Prior
Geometric Hamenization
54.5
Accurssy
53
0.4
53.5
53
0.2
52.5
NM
0.2
0.4
0.6
0.0 1.0
Bimension
Method
Temperature
(a) Geometric Dimension
(b) Label Quality
(c)
Prior
Figure 3: (a) Linear probing performance w.E. the dimension K of the geometrie uniform structure M (Appendix B) on CIFAR-LT-RIOO, (b) NMI score between the surrogate geometric labels and the ground-truth labels in the training stage CIFAR-LT-RI 100. (c) Average linear probing and the error bars of the surrogate label allocation with variants of the label prior on CIFAR-LT-R
K, but slightly exhibits the performance degeneration when the dimension is extremely large or small. Intuitively, when K extremely large, our GH might pay more attention to the uniformity among sub-classes, while the desired uniformity on classes is not well guaranteed. Conversely, when K is extremely small, the calibration induced by GH is too coarse that cannot sufficiently avoid the internal collapse within each super-class. For discussions of the structure, can refer to Appendix B.
Surrogate Label Quality Uncovered. To justify the effectiveness of surrogate label allocation, compare the NMI scores [54] between the surrogate and ground-truth label in Figure 3(b). We observe that GH significantly improves the NMI scores across baselines, indicating that the geomet- ric labels are effectively calibrated to better capture the latent semantic information. Notably, the improvements of the existing works are marginal, which further verifies the superiority GH.
Exploration with Other Gabel-Distribution Prior. To further understand π, we assume the ground-truth label distribution is available and incorporate the oracle म into the surrogate label allocation. Comparing the results of the softened variants ThE with the temperature "T in Fig- ure 3(C), we observe that GH outperforms all the counterparts equipped with the oracle prior. A possible reason is that our method automatically captures the inherent geometric statistics from the embedding space, which is more reconcilable to the self-supervised learning objectives.
Uniformity Analysis. In this part, we conduct experiments with two uniformity metrics [62, 37]:
h Σ
U L(L
Σ p. - Hill,, U² j=1,f$$
Σ min Jireoal
l|
Mjm 1l2,
Lk
where ,i, js A represent different classes. Specifically, U evvaluates average distances between different class centers and Uj measures how close one class is to its neighbors. As shown in Table 6, our GH outperforms in both inter-class uniformity and neighborhood uniformity when compared with the baseline SimCLR [9]. This indicates that vanilla contrastive learning struggles to achieve the uniform partitioning of the embedding space, while our GH effectively mitigates this issue.
Comparison with More SSL Methods. In Table 7, we present a more comprehensive compari- of different SSL baseline methods, including MoCo-v2 [24], MoCo-v3 [11] and various contrastive methods such as SimSiam [10]. BYOL [20] and Barlow Twins [71]. From the results, we can see that the combinations of different SSL methods and our GH can achieve consistent perfor- mance improvements, averaging as 2.33%, 38% and 2.21% on CIFAR- -00--- This demonstrates the prevalence of representation learning disparity under data imbalance in general SSL settings.

---

Table 7: Linear probing of more SSL variants CIFAR-100-LT with different imbalanced ratios.
Dataset
SimSiam
+GH
Barlow
+GH
BYOL
+GH
MoCo-v2
+GH
MoCo-v3
+GH
CIFAR-RION CIFAR-RSS R50
49.01
51.43
48.70
51.23
51.43
52.87
51.49
53.53
54.08
55.82
48.98
53.54
49.29
51.95
52.04
53.84
52.68
55.01
55.34
56.45
CIFAR-RIO
55.51
57.03
53.11
56.34
55.86
57.28
58.23
60.11
59.10
60.57
SimCLR
GH
100
54-
75
Accuracy
53-
Dineraton
Dimersion
GHl SimCL8
52-
-25
51-
-75
Tel
-75
Tea
50
100
105
100
75
50
25
50
100
100
25
Dimension
Dimension
WGH
Figure 4: T-SNE visualization of the pretrained features from 8 ran- Figure 5: Ablations of domly selected classes w.zt CIFAR-LT training images.
WGH on CIFAR-LT.
On Importance of Bi-Level Optimization. In Ta-
Table 8: Linear probing of joint optimization on CIFAR-100-L3 with different IRs.
ble we empirically compare the direct joint opti- mization strategy to Eq. (4), From the results, we can see that the joint optimization (w/ or w/o the warm-up strategy) does not bring significant perfor- mance improvement over SimCLR compared with that of our bi-level optimization, probably due to the undesired collapse label allocation [1]. This demonstrates the necessity of the proposed bi-level optimization for Eq. (4) to stabilize the training.
Method
C100
C50
C10
SimCLR
50.72
52.24
55.67
+GH (Joint)
50.18
52.31
54.98
w/ warm-up
51.14
52.75
55.37
+GH (Bi-level)
53.96
55.42
57.36
Qualitative Visualization. We conduct t-SNE visualization of the learned features to provide fur- ther qualitative intuitions. For simplity, we randomly selected four head classes and four tail classes on CIFAR-LT to generate the t-SNE plots. Based on the results in Figure 4, the observations are as follows: (1) SimCLR: head classes exhibit a large presence in the embedding space and heavily squeeze the tail classes, (2) GH: head classes reduce their occupancy, allowing the tail classes have more space. This further indicates that the constructed surrogate labels can serve as the high-quality supervision, effectively guiding the harmonization towards the geometric uniform structure.
Sensitivity Analysis. To further validate the stability of our GH, We conduct empirical comparison with different weight WCH, temperature YGH, regularization coefficient À and Sinkhorn iteration E, on CIFAR-LT, as shown in Figures and From the results, we can see that our GH can consistently achieve satisfying performance with different hyper-parameter.
5 Conclusion
In this paper, we delve into the defects of the conventional contrastive learning in self-supervised long-tail context, i.e., representation learning disparity, motivating our exploration the inherent intuition for approaching the category-level uniformity. From the geometric perspective, we propose a novel and efficient Geometric Harmonization algorithm counteract the long-tailed effect on the embedding space, i.e, over expansion of the majority class with the passive collapse of the minority class. Specially, our proposed GH leverages the geometric uniform structure an optimal indicator and manipulate a fine-grained label allocation to rectify the distorted embedding space. We theoretically show that our proposed method can harmonize the desired geometric property in the limit of loss minimum. It is also worth noting that our method is orthogonal to existing self-supervised long-tailed methods and can be easily plugged into these methods in a lightweight manner. Extensive experiments demonstrate the consistent efficacy and robustness of our proposed GH. We believe that the geometric perspective has the great potential to evolve the general self- supervised learning paradigm, especially when coping with the class-imbaledd imbalanced scenarios.
10

---

Acknowledgement
This work was supported by the National Key R&D Program of China (No. 20222D0160702, STCSM (No. 22511106101, No. 22511105700, No. 10001000100, 111 plan (No. BP0719010) and National Natural Science Foundation of China (No. 62306178). BH was supported by the NSFC Young Scientists Fund No. 62006202, NSFC General Program No. 62376235, and Guangdong Basic and Applied Basic Research Foundation No. 2022A151501155..
References
[1] Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In International Conference on Learning Representa- lions (ICLR), 2020.
[2] Mido Assran, Randall Balestriero, Quentin Duval, Florian Bordes, Ishan Misra, Piotr Bo- janowski, Pascal Vincent, Michael Rabbat, and Nicolas Ballas. The hidden uniform cluster prior in self-supervised learning. In The Eleventh International Conference on Learning Rep- resentations, 2023.
[3] Tanya Robles-Kelly, Daniel lerodiaconou, and Elizabeth Oh. Marine-tree: A large-scale marine organisms dataset for hierarchical image classification. In Proceedings of the s ACM Inter- national Conference on Information & Knowledge Management, pages -8842, 2022.
Asef Nazari, Imran Razzak, Mohamed Reda Bouadjenek, Antonio
[4] Sergiy V Borodachov, Douglas P Hardin, and Edward B Saff. Discrete energy on rectifiable sets. Springer, 2019.
[5] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with tabel-distribution-aaaas margin loss. Advances Neural Information Processing Systems, 12:151715578 2019.
[6] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on com- puter vision (ECCV), pages 132-149, 2018.
[7] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912 9924, 2020.
[8] Mengxi Chen, Linyu Xing, Yu Wang, and Ya Zhang. Enhanced multimodal representation learning with cross-modal kd. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11766-11775, 2023.
[9] Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1597-1607. PMLR, 2020.
[10] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
[11] Xinlei Chen Saining Xie*, and Kaiming He. An empirical study of training superviedd vision transformers. arXiv preprint arXiv:2104.0205,, 2021.
[12] Henry Cohn and Abhinay Kumar. Universally optimal distribution of points spheres. Jour- nal of the American Mathematical Society, 20(1):99-144 2007.
[13] Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning- In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 715- 724, 2021.
[14] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.

---

[15] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learn- ing by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pages 1422-1430, 2015.
[16] Ziqing Fan, Yanfeng Wang, Jiangchao Yao, Lingjuan Lyu, Ya Zhang, and Qi Tian. Fedskip: Combatting statistical heterogeneity with federated skip aggregation. In 2022 IEEE Interna- tional Conference on Data Mining (ICDM), pages 131-140. IEEE, 2022.
[17] Cong Fang. Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 1 118(43):e2 :e143999 118, 18, 2021.
[18] Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann LeCun. On the duality between contrastive and non-contrastive supervidd learning. In The Eleventh In- ternational Conference on Learning Representations, 2023.
[19] Florian Graf, Christoph Hofer, Mare Niethammer, and Roland Kwitt. Dissecting supervised constrastive learning. In International Conference on Machine Learning, pages 3821-3830.
PMLR, 2021.
[20] Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Ad- vances in neural information processing systems, 33:21271-21284 2020.
[21] DP Hardin and EB Saff. Minimal riesz energy point configurations for rectifiable d- dimensional manifolds. Advances in Mathematics, 193(1):174-200 2005.
[22] John A Hartigan and Manchek A Wong. Algorithm as 136: A k-means clustering algorithm. Journal of the royal statistical society. series c (applied statisttics, 28(1): 1979.
[23] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 770-778, 2016.
[24] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738. 2020.
[25] Feng Hong, Jiangchao Yao, Zhihan Zhou, Ya Zhang, and Yanfeng Wang. Long-tailed partial label learning via dynamic rebalancing. In The Eleventh International Conference on Learning Representations, 2023.
[26] Chaoqin Huang, Haoyan Guan, Aofan Jiang, Ya Zhang, Michael Spratling, and Yan-Feng Wang. Registration based few-shot anomaly detection. In European Conference on Computer Vision, pages 303-33 309-319.500 Springer, 2022.
[27] Chaoqin Huang, Qinwei Xu, Yanfeng Wang, Yu Wang, and Ya Zhang. Self-supervised mask- ing for unsupervised anomaly detection and localization. IEEE Transactions on Multimedia, 2022.
[28] Ziyu Jiang, Tianlong Chen, Bobak Mortazavi, and Zhangyang Wang. Self-damaging con- trastive learning. In International Conference Machine Learning, pages 4927-4939. PMLR, 2021.
[29] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2019.
[30] Bingyi Kang, Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. In International Conference on Learning Representations, 2020.
[31] Tejaswi Kasarla, Gertjan Burghouts, Max van Spengler, Elise van der Pol, Rita Cucchiara, and Pascal Mettes. Maximum class separation as inductive bias in one matrix. In Advances in Neural Information Processing Systems, volume 35, pages 19553-19566, 2022.
12

---

[32] Aditya Khosla, Nityananda Jayaevvaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine-grained image categorization: Stanford dogs. In Proc. CVPR Workshop on Fine- Grainnd Visual Categorization (FGVC), volume 2. Citeseer, 2011.
[33] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[34] A Krizhevsky. Learning multiple layers of features from tiny images. Technical Report TR- 2009, University of Toronto, 2009.
[35] Anna Kukleva, Moritz Böhle, Bernt Schiele, Hilde Kuehne, and Christian Rupprecht. Tem- perature schedules for self-supervised contrastive methods on long-tail data, In The Eleventh International Conference Learning Representations, 2023.
[36] Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsupervised representations. In International Conference on Learning Representations, 2021.
[37] Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogerio S Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long-tailed recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6918 6928, 2022.
[38] Jiye Liang, Liang Bai, Chuangyin Dang, and Fuyuan Cao. The k-means-type algorithms versus imbalanced data distributions. IEEE Transactions on Fuzzy Systems, 20(4):728-7453 2012.
[39] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Pi- otr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Eisi 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[40] Tsung-Yl Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pages 2980-2988, 2017.
[41] Hong Liu, Jeff Z HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. In International Conference on Learning Representations, 2021.
[42] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 212-220, 2017.
[43] Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai. and Le Song. Learning towards minimum hyperspherical energy. Advances in neural information processing systems, 31, 2018.
[44] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailes recognition in an open world. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 2537-2546, 2019.
[45] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification aircraft. arXiv preprint arXiv: 130..5530..55, 2013.
[46] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. International Conference on Learning Representations, 2021.
[47] Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. Sampling Theory, Signal Processing, and Data Analysis, 20(2):1-13, 2022.
[48] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint 177.0.1807.03748, 2018.
13

---

[49] Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):245222444 24663, 2020.
[50] William J Reed. The pareto, zipf and other power laws. Economics letters, 74(1):15-19) 2001.
[51] Rasmus Rothe, Radu Timofte, and Luc Van Gool. Deep expectation of real and apparent age from a single image without facial landmarks. International Journal of Computer Vision, 126 (2-4):144-157,22)
2018 2018.
[52] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 2556-2565, 2018.
[53] Steve Smale. Mathematical problems for the next century. The mathematical intelligencer, 20 (2):7-15, 1998.
[54] Alexander Strehl and Joydeep Ghosh. Cluster ensembles knowledge reuse framework for combining multiple partitions. Journal of machine learning research, 3(Dec):583-61) 2002.
[55] Jianhua Sun, Yuxuan Li, Hao-Shu Fang, and Cewu Lu. Three steps to multimodal trajectory prediction: Modality clustering, classification and synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13250-13259, 2021.
[56] Jianhua Sun, Yuxuan Li. Liang Chai, Hao-Shu Fang, Yong-Lu Li, and Cewu Lu. Human tra- jectory prediction with momentary observation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6467-6476, 2022.
[57] Jianhua Sun, Yuxuan Li, Liang Chai, and Cewu Lu. Modality exploration, retrieval and adapta- tion for trajectory prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[58] Joseph John Thomson. Xxiv. on the structure of the atom: an investigation of the stability and periods of oscillation of a number of corpuscles arranged at equal intervals around the circumference of circle; with application of the results to the theory of atomic structure. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 7(39): 237-265, 1904.
[59] Yonglong Tian, Olivier J Henaff, and Aäron van den Oord. Divide and contrast: Self- supervised learning from uncurated data. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 10063-10074, 2021.
[60] Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 595-604, 2015.
[61] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[62] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. International Conference on Machine Learn- ing, pages 9939. PMLR, 2020.
[63] Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE International Conference on Computer Vision, pages 2794 2802, 2015.
[64] Yifei Wang. Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is ladder: new theoretical understanding of contrastive learning via augmentation overlap. In Interna- tional Conference on Learning Representations, 2021.
14

---

[65] Chaoyi Wu, Feng Chang, Xiao Su, Zhihan Wu, Yanfeng Wang. Ling Zhu, and Ya Zhang. Inte- grating features from lymph node stations for metastatic lymph node detection. Computerized Medical Imaging and Graphics, 101:102108, 2022.
[66] Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology. arXiv preprint N.Xi:2338800266.5 2023.
[67] Yibo Yang, Liang Xie, Shixiang Chen, Xiangtai Li, Zhouchen Lin, and Dacheng Tao. Do really need learnable classifier the end of deep neural network? arXiv preprint arXiv:2203.0900 09081, 2022.
[68] Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalanced learn- ing. Advances in neural information processing systems, 33:19290-19301, 2020.
[69] Yuzhe Yang, Kaiwen Zha, Yingcong Chen, Hao Wang, and Dina Katabi, Delving into deep im- balanced regression. In International Conference on Machine Learning, pages 11842-11851. PMLR, 2021.
[70] Jiangchao Yao, Bo Han, Zhihan Zhou, Ya Zhang, and Ivor W Tsang. Latent class-conditional noise model. IEEE Transactions Pattern Analysis and Machine Intelligence. 2023.
[71] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International Conference on Machine Learn- ing, pages 123 12310-12320. PMLR, 2021.
[72] Fei Zhang, Lei Feng, Bo Han, Tongliang Liu, Gang Niu, Tao Qin, and Masashi Sugiyama. Exploiting class activation value for partial-labe learning. In International Conference on Learning Representations, 2021.
[73] Fei Zhang, Chaochen Gu, Chenyue Zhang, and Yuchao Dai. Complementary patch for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7242-7251, 2021.
[74] Ruipeng Zhang, Ziqing Fan, Qinwei Xu, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Grace: A generalized and personalized federated learning method for medical imaging. In Interna- tional Conference on Medical Image Computing and Computer-Assisted Intervention, pages 14-24. Springer, 2023.
[75] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long- tailed learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[76] Yuhang Zhou, Jiangchao Yao, Feng Hong, Ya Zhang, and Yanfeng Wang. Balanced destraction-recoonstaciioo dynamies for memory-replay class incremental learning. arXiv preprint arXiv:2308.016988 2023.
[77] Zhihan Zhou, Jiangchao Yao, Yan-Feng Wang, Bo Han, and Ya Zhang. Contrastive learning with boosted memorization. In International Conference on Machine Learning, pages 27367- 27377. PMLR, 2022.
[78] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Infor- mation Processing Systems, 34:29820-29888 2021.
15


Reviews:
[{"review_id": "Uty01wYu48", "reviewer": "Reviewer_krnL", "summary": "This paper addresses the class-imbalance problem under the SSL setting. It shows that the Vanilla SSL methods pursuing the sample-level uniformity easily lead to representation learning disparity, where head classes with the huge sample number dominate the feature regime but tail classes with the small sample number passively collapse. It proposes Geometric Harmonization (GH) to encourage the category-level uniformity in representation learning. The GH loss can be easily integrated into existing SSL methods, and GH loss improve the performance over baselines on the imbalance class problem under the SSL setting, based on the experimental results.", "questions": "Address Weakness 2 and 3.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.This paper define the category-level uniformity in the embedding space that SSL learns.  The motivation to using category-level uniformity for imbalance class problem is clear. \n\n2.The proposed Geometric Harmonization (GH) loss improve the performance over baselines on the imbalance class problem under the SSL setting, based on the experimental results. \n\n3.The presentation of this paper is generally clear and easy to follow.", "weaknesses": "1. In the experiments of Section 4.3, it is better to conduct experiments on large-scale ImageNet pertaining using the proposed methods, and then transfer to other datasets (This is the so called “representation transferability”), rather than only: SSL pre-training and finetuning on the same datasets. \n\n2. I personally believe the theories and analyses both are specific to contrastive learning, rather than on other SSL methods (e.g., the non-contrastive SSL methods, BOYL, SiamSiam). Even though this paper conducts an experiment on other SSL methods in Section 4.4, it is not clear how the theory/analyses work for these SSL methods? Do the theoretic analyses still work for other SSL methods? This paper should at least mention it.   \n\n3. At last, I have concerns on whether the imbalance problem under the SSL is an important research topic. I definitely agree with that the class imbalance problem is very important. However, it is weird when considering the class imbalance problem under the SSL scenario. As everyone knows, SSL is considered as pre-training under large scale unlabeled data (class-free), and then transfer to downstream tasks (not specific for the classification problem). \nI believe the contributions are significant if this is the first paper to propose the imbalance problem under SSL setting. However, with several exists works as shown in the paper, I donot recognize the significance this paper contributes to the ML community. I think this paper will be stronger, if considering transfer to the class-imbalance object detection/semantic tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We gratefully thank you for your time and efforts devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.\n\n> **Q1:** Cross-dataset transferring experiments.\n\n**A1:** We appreciate the reviewer’s constructive advice and conduct more comprehensive experiments on various cross-dataset transferring tasks to evaluate representation transferability as follows:\n\n- Pretraining datasets\n    - CC3M [1]. CC3M is a **large-scale, long-tailed** dataset with more than 3 million images. We pretrain SimCLR [11] and SimCLR with GH on CC3M for comparison. \n- Transferring to downstream classification\n    - Curated datasets: ImageNet [2] and Places [3]. We randomly subsample a balanced subset of ImageNet and Places for downstream finetuning, with the number of images per class set as 100.\n    - Fine-grained datasets: CUB200 [4], AirCraft [5], Stanford Cars [6], Stanford Dogs [7], NABirds [8]. These datasets requires semantic features to distinguish each category on the fine-grained granularity.\n    - We report the average accuracy of our CC3M-pretrained weights finetuning on classification on these datasets based on ResNet50 backbone.\n- Transferring to downstream object detection\n    - COCO2017 [9]. We report the bounding box AP of our CC3M-pretrained weights finetuning on object detection on COCO2017 using Faster-RCNN based on ResNet50-FPN backbone.\n- Transferring to downstream segmentation\n    - COCO2017 [9]. We report the mask AP of our CC3M-pretrained weights finetuning on segmentation on COCO2017 using Mask-RCNN based on ResNet50-FPN backbone. \n\nFrom the results in **Table R3-R4** in the complementary PDF file, we can see that our proposed GH consistently outperforms the baseline across various tasks and datasets. It further demonstrates the importance of **considering long-tailed data distribution under large-scale unlabeled data in the pretraining stage**. This can potentially be attributed to that our geometric harmonization motivates a more balanced and general emebdding space, improving the generalization ability of the pretrained model to a range of real-world downstream tasks. \n\nBesides, for more complete comparisons under other baselines, we will update the results in the submission once we finish all experiments.\n\n> **Q2:** Analysis of GH on other SSL methods.\n\n**A2:** Thanks for the constructive suggestions. We agree with the reviewer that our theorem and analyses are specific to contrastive learning. In terms of other non-contrastive SSL methods, we empirically show the superiority of our method on long-tailed data distribution in Section 4.4. Although it might not be straightforward to extend the theory to non-contrastive SSL methods, an explanation about the consistent superiority is that some non-contrastive methods still exhibit similar representation disparity with their contrastive counterpart, and our proposed method can similarly reallocate the geometric distribution to counteract the distorted embedding space. Specially, the recent study [10] theoretically and emprically explore the equivalence between contrastive and non-contrastive criterion, which may shed light on the intrinsic mechanism of how our GH benefits non-contrastive paradigm. \n\nWe appreciate the reviewer’s question, and will include these discussions about the our theorem in the revision for clarity.\n\n> **Q3:** The importance of considering imbalancing learning in SSL scenarios, the contribution of GH and the transferring experiments on object detecion/segmentation tasks.\n\n**A3:** Thanks for the comments although it is a challenging point of view. Without offense, we might be persistent that considering imbalancing learning in SSL scenarios can also be critical to the representation generalization. Following the reviewer's advice in Q1, we conduct the representation learning on large-scale long-tailed CC3M and  verify that our GH inherently provides merits for the generalization on different downstream tasks, including detection and segmentation as suggested. As completely transferring all baselines in large-scale datasets is time-consuming and computationally expensive, we will update the comprehensive experiments in the submission as soon as possible.\n\nBesides, we would like to reclarify our taxonomy as described in **Lines 28-38** that all previous explorations target to improve self-supervised long-tailed learning, but from different aspects such as reweight/optimization techniques, architecture design and data augmentation. However, **none of these methods focus on the intrinsic limitation of the contrastive learning**. Concretely, we provide more detailed aspects in **Table G1** in the general response for clarity.\n\nFurthermore, to the best of our knowledge, we are the first to **explicitly point out the concept of representation disparity, which is the key drawback we have investigated**, in self-supervised long-tailed learning. We believe that the undesired disparity is the intrinsic limitation of contrastive loss to hurt the representation quality. "}, {"review_id": "kqKVQsCVtF", "reviewer": "Reviewer_M9kH", "summary": "This paper studied self-supervised representation learning with implicit class-imbalance data, in contrast to most prior works that assume class-balance. To do this, this paper proposed to augment the existing contrastive learning method with a novel geometric harmonization (GH). Intuitively, GH pulls samples toward the closest prototype in the embedding space, which differs from previous cluster-/prototype-based methods in the learnable prototypes and marginal prior. Through extensive experiments on several standard long-tail classification benchmarks, the authors demonstrated the efficacy of the proposed method with different contrastive learning methods as start points.", "questions": "See the weakness section.", "limitations": "The major limitation is that geometric harmonization relies heavily on the assumption that samples from the same class intrinsically lie in neighboring regions in the embedding space, which may not hold for real-world data.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is generally well-written and easy to follow.\n\n2. The paper is well-motivated. Class imbalance occurs in most real-world scenarios, and existing contrastive learning methods fail to take this into account. This paper provides a good exploration of this direction.\n\n3. The proposed geometric harmonization regularization is interesting. Though prototype-based constraints are already studied in contrastive learning [1, 5, 12], GH carefully considers the class imbalance by its design and is thus robust to such imbalance.\n\n4. The experiment results are strong compared with the baselines on imbalanced data. Moreover, the proposed method even performs on par with the baseline with balanced data.", "weaknesses": "1. The geometric harmonization relies heavily on the assumption that samples from the same class intrinsically lie in neighboring regions in the embedding space to estimate the assignment and prior. While this condition can be satisfied on curated datasets like CIFAR and ImageNet, it might be a different case for the uncurated datasets, e.g., YMCC or a subset of it. Some analysis on this end would further strengthen this paper.\n\n2. Lack of transfer learning experiments. In all the experiments, the models were pretrained and then finetuned on the same datasets. It is unclear whether the learned representations are general to transfer to other data distribution and different tasks. Possible options are adding cross-dataset experiments (e.g., YFCC-to-ImageNet-LT, ImageNet-Lt-to-CIFAR-LT) and transfer learning on MS-COCO object detection as in [5].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe gratefully thank you for your time and efforts devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.\n\n> **Q1:** The geometric harmonization relies heavily on the assumption that samples from the same class intrinsically lie in neighboring regions in the embedding space to estimate the assignment and prior. While this condition can be satisfied on curated datasets like CIFAR and ImageNet, it might be a different case for the uncurated datasets, e.g., YMCC or a subset of it. Some analysis on this end would further strengthen this paper.\n\n\n**A1:** This is a very insightful question and thank the reviewer for this point. We agree with the reviewer for the potential presence of scattered samples from the same classes across different regions (rather than neighboring regions) in the embedding space, particularly in the context of large-scale long-tailed data distributions. In this case, our method will leverage the inherent pattern or semantic cluster information of the data in the fine-grained region to relatively mitigate the disparity. Such potential provides the model a chance to learn more general-purpose representations correlated to the clusters in various levels of granularities, going beyond the observed labels, thus promoting the generality and transferability of the pretrained representations. We do appreciate the reviewer's advice and conduct experiments on **large-scale long-tailed and uncurated dataset CC3M** [1] to address the possible concern, as detailed in **A2**, and we will include the discussion into the submission.\n\n\n> **Q2:** Lack of transfer learning experiments. In all the experiments, the models were pretrained and then finetuned on the same datasets. It is unclear whether the learned representations are general to transfer to other data distribution and different tasks. Possible options are adding cross-dataset experiments (e.g., YFCC-to-ImageNet-LT, ImageNet-Lt-to-CIFAR-LT) and transfer learning on MS-COCO object detection as in [5].\n\n**A2:** We appreciate the reviewer’s constructive advice and conduct more comprehensive experiments on various cross-dataset transferring tasks with distinct characteristics as follows:\n\n\n- Pretraining datasets\n    - CC3M [1]. CC3M is a **large-scale, long-tailed** dataset with more than 3 million images. We pretrain SimCLR [11] and SimCLR with GH on CC3M for comparison. \n- Transferring to downstream classification\n    - Curated datasets: ImageNet [2] and Places [3]. We randomly subsample a balanced subset of ImageNet and Places for downstream finetuning, with the number of images per class set as 100.\n    - Fine-grained datasets: CUB200 [4], AirCraft [5], Stanford Cars [6], Stanford Dogs [7], NABirds [8]. These datasets requires semantic features to distinguish each category on the fine-grained granularity.\n    - We report the average accuracy of our CC3M-pretrained weights finetuning on classification on these datasets based on ResNet50 backbone.\n- Transferring to downstream object detection\n    - COCO2017 [9]. We report the bounding box AP of our CC3M-pretrained weights finetuning on object detection on COCO2017 using Faster-RCNN based on ResNet50-FPN backbone.\n- Transferring to downstream segmentation\n    - COCO2017 [9]. We report the mask AP of our CC3M-pretrained weights finetuning on segmentation on COCO2017 using Mask-RCNN based on ResNet50-FPN backbone.\n\n\n\n\nFrom the results in **Table R3-R4** in the complementary PDF file, we can see that our proposed GH consistently outperforms the baseline methods across various tasks and datasets. This indicates the superiority of GH to improve contrastive learning on imbalanced datasets, yielding better model generalization to a range of real-world scenarios.\n\nBesides, for more complete comparisons under different baselines, we will update the results in the submission once we finish all experiments."}, {"review_id": "gotQbwsbvA", "reviewer": "Reviewer_TBJn", "summary": "This paper concentrates on the long-tailed distribution problem in SSL representation learning. To overcome the challenge in vanilla SSL methods, where head classes with the huge sample number dominate the feature regime, the authors propose the Geometric Harmonization (GH) method to encourage category-level uniformity. Specially, GH measures the population statistics of the embedding space, and then infer an fine-grained instance-wise calibration to constrain the space expansion of head classes while avoid the collapse of tail classes. Extensive results show the effectiveness of GH with high tolerance to the long-tailed distribution problem.", "questions": "Please refer to *Weaknesses.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-1- This paper investigates the drawback of the contrastive learning loss in SSL long-tailed context, and shows that the resulted sample-level uniformity is an intrinsic limitation to the representation parity, as shown in Figure 1. The motivation of pursuing category-level uniformity is clear.\n\n-2- This paper is well-written and easy to follow. The proposed GH method that incorporates the geometric clues from the embedding space to calibrate the training loss, is interesting and solid.\n\n-3-  The proposed GH loss is versatile and can be easily plugged into existing SSL methods. The extensive experiments and ablation stidies demonstrate its effectiveness in learning robust representation.", "weaknesses": "-1- About the surrogate label allocation. What is the advantage to choose the discriminative clustering [1], what if we choose other clustering method such as K-means. How does the quality of the geometric label affect the final training results? When verifing the assumption that the constructed surrogate geometric labels are mutually correlated with the oracle labels, can the authors provide more intuitive results?\n\n-2- In E.q. 4, do you need to design how to balance these two losses?\n \n-3- Table 6 shows the results on class-balanced data. Can you explain why in some cases, incooperateing with GH will lead to the performance drop?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We gratefully thank you for your time and efforts devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.\n\n> **Q1:** About the surrogate label allocation. What is the advantage to choose the discriminative clustering [1], what if we choose other clustering method such as K-means. How does the quality of the geometric label affect the final training results? When verifing the assumption that the constructed surrogate geometric labels are mutually correlated with the oracle labels, can the authors provide more intuitive results?\n\n**A1:** We appreciate the reviewer's insightful questions. Below are our replies to each subquestion.\n\n**Advantage:** Our discriminative clustering approach offers the following advantages: (1) it enables a flexible class prior that can adapt to various distributions, (2) it refines assignments in an efficient manner, (3) it is theoretically grounded to achieve the desired category-level uniformity along with the geometric uniform structure, effectively addressing the challenges posed by the long-tailed effect. Note that, we would like to reclarify that straightforward applying the discriminative clustering SeLA [17] or the variant SwAV [18] cannot work, whose difference from ours is discussed in Lines 134-152, and empirically verified in Table 1 of the original submission.\n\n**Drawback of K-means:** K-means algorithm tends to generate clusters with relatively uniform sizes, which will affect the cluster performance under the class-imbalanced scenarios [19]. To gain more insights, we conduct empirical comparisons using K-means as the clustering algorithm and evaluate the NMI score with ground-truth labels and the linear probing accuracy on CIFAR-LT-R100. \n\n| Method | Accuracy | NMI score |\n|:--|:--|:--|\n| SimCLR [11] | 50.72 | 0.28 | \n| +K-means | 51.44 | 0.35 |\n| +GH | 53.96 | 0.50 |\n\nFrom the results, we can see that K-means generates undesired assignments with lower NMI score and achieves unsatisfying performance compared with our GH. This observation is consistent with previous studies [19].\n\n**More intuitive results:** To enhance the understanding of the proposed surrogate label allocation, we conduct both quantitative and qualitive analyses.\n\n(1) Quantitative analyses: \n\nWe conduct more thorough experiments with several metrics [28,29] on CIFAR-LT, including: \n\n- Inter-Class Uniformity: $U=\\frac{1}{L(L-1)}\\sum_{i=1}^L\\sum_{j=1, j\\neq i}^L||\\mu_i-\\mu_j||_2$, where we have $L$ classes and $\\mu$ denotes the class means. It evaluates average distances between different class centers.\n- Neighborhood Uniformity: $U_k=\\frac{1}{Lk}\\sum_{i=1}^L\\min_{j_1,\\cdots, j_k}(\\sum_{m=1}^k||\\mu_i-\\mu_{j_m}||_2)$, where $j_1, \\cdots, j_k \\neq i$ represent different classes. It measures how close one class is to its neighbors.\n\nBased on the results in **Table R5** in the complementary PDF file, the observations are as follows: our method exhibits significant improvements in both inter-class uniformity and neighborhood uniformity when compared with the baseline SimCLR [11]. This indicates that vanilla contrastive learning struggles to achieve the uniform partitioning of the embedding space, while our proposed method effectively mitigates this issue.\n\n(2) Qualitive analyses: \n\nWe conduct t-SNE visualization of SimCLR and SimCLR with GH on CIFAR-LT. For simplity, we randomly selected four head classes and four tail classes to generate the t-SNE plots. Based on the results in **Figure R1** in the complementary PDF file, the observations are as follows: (1) SimCLR: head classes exhibit a large presence in the embedding space and heavily squeeze the tail classes, (2) GH: head classes reduce their occupancy, allowing the tail classes to have more space. This further indicates that the constructed surrogate labels can serve as the high-quality supervision, effectively guiding the harmonization towards the geometric uniform structure.\n\n\n> **Q2:** In E.q. 4, do you need to design how to balance these two losses?\n\n**A2:** In the submission, we have not introduced a hyperparameter to balance two losses, instead, we defaultly set a weight 1.0 (termed as $w_{GH}$) on GH loss in Eq.4 across all the experimental results. To address the question regarding balancing contrastive loss and our GH loss, we conduct experiments with different weight $w_{GH}$ on CIFAR-LT-R100 (please refer to **Figure R2** in the complementary PDF file).\n\nFrom the results, we can see that our method generally achieves comparable performance across different configurations of weight $w_{GH}$. We appreciate the reviewer's question and will add the discussion and empirical comparisons in the revision.\n\n> **Q3:** Table 6 shows the results on class-balanced data. Can you explain why in some cases, incooperateing with GH will lead to the performance drop?\n\n**A3:** Thanks for your detailed question. We would like to kindly clarify that our GH is generally comparable with baseline methods, which aligns with our expectations. Specifically, our GH is designed to offer advantages on imbalanced datasets while avoiding unreasonable degradation to contrastive learning on balanced dataset. In Table 6, the proposed GH shows the minor improvements over Focal [12], SDCLR [14] and BCL [16], and performs worse than SimCLR  and DnC [15] for less than 0.3\\%. The minor decrease in performance could potentially be attributed to some random factors during training (like weight initialization and data augmentation) or the negligible effect of GH loss as it might not reach an absolute zero value. \n"}, {"review_id": "UQ877hM2O3", "reviewer": "Reviewer_pT3P", "summary": "- The paper proposes to tackle the task of self-supervised learning on datasets sampled from long-tailed distributions. \n- Typical SSL approaches tend to perform a lot worse for the minority classes, thus hurting performance.\n- The proposed approaches termed Geometric Harmonization encourages category-level uniformity leading to better performance on various downstream tasks.\n- The approach is shown to be complementary to various existing SSL approaches. ", "questions": "- Minor suggestions :\n    - L42 : reword \"as proof in [38]..\"\n    - \"sample number\" : I suggest using \"number of samples\" or a similar alternative instead to avoid any confusion. \n - Some more discussion and analysis on the observation in L298-300 would be really interesting. For this experiment, what happens if there is no support for a certain class in that mini-batch ? \n", "limitations": "The authors have adequately addressed the limitations and societal impact of their work. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles an important problem in SSL - that of training on data sampled from long-tailed distributions which is common in the real world. \n- The proposal of geometric harmonization is interesting. The idea is simple and easy to implement. While the approach is motivated by recent clustering-based works in SSL, the proposed approach seems novel.  \n- The paper is well written for the most part (some concerns are pointed out in other sections)\n- Authors perform extensive analyses and ablations to understand the working of the approach. \n- Experiments on various LT datasets show the benefit of using the approach. ", "weaknesses": "Following are some of my concerns with the paper: \n- Clarity: Given that geometric harmonization is the key contribution of this work, I think the authors should give the readers more idea about the intuitions for the section on \"surrogate label allocation\". While the Figure 2 looks good, the caption is not very helpful to drive home the intuition. I see a similar issue with Definition 3.1 : meaning of K only becomes clear later in the paper. \n- Effect of batch-size : The authors have provided an analysis on the effect of batch size. I think such analysis will be especially interesting for datasets which have lot more labels : like ImageNet. \n- Discussion/comparison missing on some related works: [a], [b]\n\n\n[a] The hidden uniform cluster prior in self-supervised learning\n[b] Temperature Schedules for Self-Supervised Contrastive Methods on Long-Tail Data", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We gratefully thank you for your time and efforts devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.\n\n> **Q1:** Clarity: Given that geometric harmonization is the key contribution of this work, I think the authors should give the readers more idea about the intuitions for the section on \"surrogate label allocation\". While the Figure 2 looks good, the caption is not very helpful to drive home the intuition. I see a similar issue with Definition 3.1 : meaning of K only becomes clear later in the paper.\n\n**A1:** We appreciate the reviewer's constructive suggestions. We will add additional clarifications regarding the \"surrogate label allocation\" section especially about how and why it works in self-supervised long-tailed context. Futhermore, we will thoroughly proofread the paper again to enhance its clarity and facilitate understanding including the caption of Figure 2 and $K$ in Definition 3.1.\n\n\n> **Q2:** Effect of batch-size : The authors have provided an analysis on the effect of batch size. I think such analysis will be especially interesting for datasets which have lot more labels : like ImageNet.\n\n**A2:** Thanks for the advice. We summarize the experiments under different batch sizes on ImageNet as follows.\n\n\n\n| Batchsize |  256  |  384  |  512  | 768 |\n|:---------:|:-----:|:-----:|:-----:|:---:|\n|  SimCLR [11]   | 36.65 |  36.97     |  37.85     | 38.04    |\n|    +GH    | 38.28 | 39.22 | 41.06 | 41.34    |\n \n\nFrom the results, we can see that under different batch sizes, our GH consistently outperforms the baseline method and a larger batch size relatively promotes better performance. We will include detailed experiments and discussions regarding the training batch size on large-scale datasets in the revision. Specially, we refer the reviewer to A5, which explains why a smaller batch size may hurt the performance under the long-tailed data from many categories.\n\n\n> **Q3:** Discussion/comparison missing on some related works: [a], [b]. \n\n> [a] The hidden uniform cluster prior in self-supervised learning \n> [b] Temperature Schedules for Self-Supervised Contrastive Methods on Long-Tail Data\n\n**A3:** We very appreciate the reviewer to recommend the concurrent works PMSN [19] and TS [20], and will add these works into related works with the proper discussion.\n\n\n\nRegarding the comparison, we conduct a range of experiments on CIFAR-LT with different imabalanced ratios to compare PMSN/TS with GH as shown in **Table R2** in the complementary PDF file.\n\n\nFrom the results, we can see that the proposed method consistently outperforms PMSN/TS across different imbalanced ratios on CIFAR-LT. Besides, we can observe that combining GH and TS consistently improves the performance of contrastive learning on CIFAR-LT. Due to the time constraints, we will provide more comprehensive experiments/discussions to strengthen their comparisons in the revision.\n\n\n> **Q4:** Minor suggestions\n\n> (1) L42 : reword \"as proof in [38]..\"\n> (2) \"sample number\" : I suggest using \"number of samples\" or a similar alternative instead to avoid any confusion.\n\n**A4:** Thanks for your detailed suggestions. We will carefully revise each of them and proofread the submission.\n\n> **Q5:** Some more discussion and analysis on the observation in L298-300 would be really interesting. For this experiment, what happens if there is no support for a certain class in that mini-batch?\n\n**A5:** This is a very insightful question and thank the reviewer for this point. Intuitively, it might easily generate biased estimation when there is no support for a certain class in the mini-batch.Then, the cluster quality might be affected by the probability of encountering missing class, which potentially correlates one important factor, i.e., batch size. Empirically, as demonstrated in Table 13 in Appendix F.4, we obeserve that the performance drops when reducing the batch size by a factor of 4 on CIFAR-LT. This can potentially be attributed to the higher probability of encountering situations where certain classes are missing under smaller batch size. We appreciate the valuable question and will include more detailed experiments and discussion in the revision.\n"}, {"review_id": "ZCeNC13PQP", "reviewer": "Reviewer_G4d9", "summary": "The paper investigates the SSL in a long-tailed distribution setting, where traditional contrastive learning may not work well.\nThe authors attribute that issue to the \"sample-level\" learning of SSL and propose Geometric Harmonization (GH) to regularize the \"category-level\" during training. GH can be a plug-and-play loss to the existing SSL frameworks without modification.\nEmpirical results of linear probing accuracy show the effectiveness of the proposed method in the long-tailed data over the baseline SimCLR. ", "questions": "+ For transfer learning to downstream tasks, which dataset is pre-trained for SSL?\n+ From the concept/hypothesis in Fig. 2 bottom, when applying GH, the samples of each class are clustered well even though it is the imbalanced case. Is there visualization (t-SNE for example) for the real data that has been conducted in this paper to show that works as the figure of concept in Fig.2?", "limitations": "They have discussed the limitations.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ Contribute to SSL capability in real-world scenarios with imbalanced classes which may be useful for unsupervised representation learning.\n+ Some theoretical analyses are provided to support the problem and method.\n+ The idea is reasonable\n+ Writing is clear enough", "weaknesses": "+ SSL has been developed with various methods to facilitate the representation learning for unlabelled data, this paper provided the experiments for one method, i.e. SimCLR, which is not a very strong baseline and not a state-of-the-art CL framework as claimed. \nFrom the presented results, SimCLR still works not too bad compared to SimCLR+GH in all settings. I would expect to see the effectiveness of GH in more strong SSL baselines including contrastive learning frameworks such as BYOL, MoCo-v3, etc, and maybe a more recent branch of SSL, namely mask autoencoder (MAE). To make the proposed method more complete, those baselines should be considered.\n+ The \"passive collapse\" of tail classes is mentioned in the paper but seems that except for some quantitative results showing GH improves baseline to some extent, there are no results (either visualization or quantitative metric) to show such collapse happens and that GH can mitigate or solve that problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We gratefully thank you for your time and efforts devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.\n\n> **Q1:** Empirical studies on BYOL, MoCo-v3 and maybe a more recent branch of SSL, namely mask autoencoder (MAE).\n\n**A1:** Thanks for the constructive suggestions. We would like to kindly clarify that many representative self-supervised learning methods have been empirically compared in the submission, such as SimCLR [11], SeLa [17], SwAV[18], MoCo-v2[22], SimSiam[23] and Barlow Twins[24] (See Table 1,4). The comprehensive comparison of different SSL baselines includes (1) contrastive learning: SimCLR and MoCo-v2, (2) unsupervised clustering: SeLa and SwAV and (3) some other non-contrastive methods: SimSiam, Barlow Twins. Besides, we do appreciate the reviewer’s advice about conducting comparisons on more SSL baselines, and conduct more experiments based on BYOL [25] and MoCo-v3 [26] on CIFAR-LT with different imbalanced ratios.\n\nFrom the results in **Table R1** in the complementary PDF file, we can see that GH can consistently improve BYOL/MoCo-v3 across different imbalanced ratios on CIFAR-LT on imbalanced datasets. Due to the time constraints, we will provide more comprehensive experiments/discussions to strengthen their comparisons with MAE [27] and on ImageNet in the revision.\n\n> **Q2&Q4:** Visualization or quantitative metric demonstrating the passive collapse of tail classes and t-SNE visualization to support the concept/hypothesis presented in Fig. 2 on read data.\n\n**A2&A4:** We appreciate the reviewer's constructive suggestions, and conduct both quantitative and qualitive analyses to provide further intuitions into the proposed method.\n\n**Quantitative analysis:**\n\nWe conduct more thorough experiments with several metrics [28,29] on CIFAR-LT as follows: \n\n- Inter-Class Uniformity: $U=\\frac{1}{L(L-1)}\\sum_{i=1}^L\\sum_{j=1, j\\neq i}^L||\\mu_i-\\mu_j||_2$, where we have $L$ classes and $\\mu$ denotes the class means. It evaluates average distances between different class centers.\n- Neighborhood Uniformity: $U_k=\\frac{1}{Lk}\\sum_{i=1}^L\\min_{j_1,\\cdots, j_k}(\\sum_{m=1}^k||\\mu_i-\\mu_{j_m}||_2)$, where $j_1, \\cdots, j_k \\neq i$ represent different classes. It measures how close one class is to its neighbors.\n\nIn **Table R5** in the complementary PDF file, we compare these metrics on CIFAR-LT with different imbalanced ratios, and have the following observations: our GH exhibits significant improvements in both inter-class uniformity and neighborhood uniformity when compared with the baseline SimCLR. This indicates that vanilla contrastive learning struggles to achieve the uniform partitioning of the embedding space, while our proposed method effectively mitigates this issue.\n\n**Qualitive analysis:**\n\nWe conduct t-SNE visualization of SimCLR and SimCLR with GH on CIFAR-LT-R100. For simplity, we randomly selected four head classes and four tail classes to generate the t-SNE plots. Based on the results in **Figure R1** in the complementary PDF file, the observations are as follows: (1) SimCLR: head classes exhibit a large presence in the embedding space and heavily squeeze the tail classes, (2) GH: head classes reduce their occupancy, allowing the tail classes to have more space. \n\nThis indicates that passive collapse of tail classes still persists in real-world data and our method can effectively mitigate the negative effect. Moreover, our method demonstrates the ability to promote clusters with higher quality in the presence of class-imbalanced data, surpassing the baseline SimCLR. We will include these discussion and empirical quantitive/quanlitive comparisons in the revision.\n\n> **Q3:** For transfer learning to downstream tasks, which dataset is pre-trained for SSL?\n\nThanks for the detailed question. In section 4.3, the SSL methods are pretrained on the same datasets as the downstream datasets, i.e., ImageNet-LT to ImageNet-LT and Places-LT to Places-LT. In section F.9 in the appendix, the SSL methods are pretrained on ImageNet and linear probing evaluated on the CUB200 [4] and Aircraft [5]. \n\nBesides, we also conduct more comprehensive experiments on various cross-dataset transferring tasks to evaluate representation transferability. Specifically, we pretrain our method on large-scale long-tailed dataset CC3M [1]. Subsequently, we apply our approach to various downstream classification datasets, including well-curated datasets such as ImageNet [2] and Places [3], fine-grained datasets such as CUB200 [4], AirCraft [5], Stanford Cars [6], Stanford Dogs [7] and NABirds [8]. Furthermore, we also conduct empirical comparisons on object detection and segmentation task using COCO2017. From the results in **Table R3-R4** in the complementary PDF file, we can see that our proposed GH consistently outperforms the baseline across various tasks and datasets. This indicates the superiority of GH to improve contrastive learning on imbalanced datasets, yielding better model generalization to a range of real-world scenarios. We appreciate the reviwer's question and will additional clarifications and empirical studies on transfer learning in the revision.\n\n\n\n"}, {"review_id": "NrVPkkOa31", "reviewer": "Reviewer_R4di", "summary": "The authors tackle the problem of contrastive learning in the context of imbalanced datasets where some classes have many more samples than other classes. Since no label information can be used, importance sampling is not an option and without any interventions, contrastive learning tends to overrepresent majority classes in the latent space, thereby hurting the readout accuracy on the minority classes. The authors propose a Geometric Harmonization technique in which they propose to estimate the class labels with clustering and then recalibrate the learned embeddings such that majority classes occupy the same feature space as minority classes. The authors show experimental results on CIFAR100-LT, ImageNet-LT and Places-LT.", "questions": "- How sensitive is GH to the choice of the hyperparameters?\n- Can you please comment on how the presented results on Places-LT and ImageNet-LT compare to the state-of-the-art results on https://paperswithcode.com/sota/long-tail-learning-on-places-lt and https://paperswithcode.com/sota/long-tail-learning-on-imagenet-lt?", "limitations": "The authors could think a bit more about the impact of their work on applications related to long-tailed data distributions where minority and majority classes are present. How would applications situated in fairness research impacted by their work?", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- I appreciate the complexity analysis of the proposed method in Section 3.4.\n- The authors compare to several other benchmarks and show superior results with their method.\n- I think in general the method makes sense and it is intuitive that it should work in the proposed case.\n", "weaknesses": "- There are many typos and grammatical errors in the text which makes the paper hard to understand. Some mathematical definitions are wrong and inconsistent which makes me doubt the claims of the theoretical guarantees since there are even errors in the Lemmas / definitions:\n \n     - Line 94: “Let N denote the sample number and R = max_i n_i/min_i n_i denote the imbalanced ratio, where n_i denotes the number of samples in class i.” I am confused, the max and min are calculated per class i, which means that max_i n_i = min_i n_i = n_i, therefore R would be 1 for each class. Since I do not understand how R is defined, I cannot understand / judge the results in Table 2.\n     - Lemma 3.2 is inconsistent with itself. If n_L_H=n_H and n_L_H=n_T, then n_T = n_H and n_H / n_T -> infty is impossible because n_H / n_T = 1 by definition.\n    - Eq. 5: Using M both as the geometric structure and the size of the collection of negative samples is confusing and should not be done.\n\n- Considering the experimental results, I am not convinced that the authors have looked into the most important baselines. CIFAR100-LT, ImageNet-LT and Places-LT are well-established benchmarks and the baseline results that the authors aim to improve seem to be very low to me. I will elaborate on the different datasets in the following:\n     - For ImageNet-LT (https://paperswithcode.com/sota/long-tail-learning-on-imagenet-lt), the currently best number for a ResNet50 architecture is an accuracy of 70.1 (https://arxiv.org/pdf/2111.13579v4.pdf) with extra training data and 67.2 (https://arxiv.org/pdf/2111.14745v1.pdf) without extra training data. The best result the authors report in this paper is about 38% which is far below what is currently state of the art according to the benchmark list.\n     - For Places-LT, https://paperswithcode.com/sota/long-tail-learning-on-places-lt, the best numbers for a ResNet50 are above 45% while the authors here report numbers below 35%. \n\n-> I believe that in order for this method to be relevant to the community, the authors need to show results on the superior models with higher baseline accuracy. In the current state, it is not clear whether their results would generalize to the better models. \n\n- The motivation for the studied problem does not become clear to me from the introduction. The authors write: “However, the real-world natural sources usually exhibit the long-tailed distribution [31], and directly learning representation on them might lead to the distortion issue of the embedding space, namely, the majority dominates the feature regime [45] and the minority collapses [28].” Using the modal verb “might” here indicates a possibility but no further evidence is presented. Citing fairness research as an application is not enough in my opinion. From reading the introduction, I am not convinced that the problem the authors want to study actually exists in SSL. I would advise the authors to provide concrete examples where using SSL actually harms performance on minority classes. Currently, the main question of study posed in line 39 (“Why the conventional contrastive learning underperforms in self-supervised long-tailed context?”) does not seem well supported.\n\n- The first part of the first contribution is misleading and I believe wrong: “To our best knowledge, we are the first to investigate the drawback of the contrastive learning loss in self-supervised long-tailed context.” This drawback is investigated in several other papers which the authors discuss in their related work section, e.g. SDCLR specifically tackles this problem with a different method.\n\n- Figure 1: The spheres in the middle and right Figure look like ovals which is confusing. These should be spheres/ circles.\n\n- Abstract, line 5: „The attribution is that the vanilla SSL methods that pursue the sample-level uniformity easily leads to representation learning disparity, where head classes with the huge sample number dominate the feature regime but tail classes with the small sample number passively collapse.” -> This sentence is hard to parse and understand.\n\n\n\n\n- Table 1: What does IR stand for?\n\n- Line 119: “and π ∈ RK+ refers to the the marginal distribution constraint.” -> \"The\" twice and what do you mean under “marginal distribution constraint”?\n\n- The used datasets and models must be cited.\n\n\n- Line 162: “This phenomenon indicates all representations of the minority will collapse completely to one point without considering the category discrepancy, which corresponds to our observation regarding the passive collapse of tailed samples in Figure 1.” This statement is not accurate because the representations of the minority classes do not collapse to a single point in Figure 1. Lemma 3.2. covers an extreme case where there are infinitely more samples in the head classes compared to the tail classes, so Figure 1 does not represent this scenario and thus, using “corresponds” here is not accurate.\n\n\n\n- It is confusing that Table 7 comes before Tables 5+6, please fix.\n\n- Does Table 5 show the computational cost per epoch?\n\n- Line 218: “For hyper-parameters of GH, we provide a default setup across all the experiments: set the geometric dimension K as 100 and the temperature γGH as 0.1. In the surrogate label allocation, we set the regularization coefficient λ as 20 and Sinkhorn iterations Es as 300. Please refer to Appendix E.3 for more experimental details.” There are many hyperparameters that need to be set for this method, and it is not clear whether they were chosen on the test set, or how they were selected. It is also not clear how sensitive the algorithm is to these hyperparameters.\n\n- Line 223: “For comprehensive performance comparison, we present the linear probing performance and the standard deviation among three disjoint groups, i.e., [many, medium, few] partitions [25].” Please explain what the partitions into many/medium/few mean as it is not possible to understand the results otherwise.\n\n- Line 290: “To justify the intuition in Section 3.2” which intuition? Please be more specific.\n\n- I find the results in Fig. 3b unintuitive. The NMI scores show that GH is better aligned with ground truth labels compared to all other methods, but then why is the readout accuracy of GH only 1-2 percent points better compared to the other methods? Please discuss this as I am not sure how to interpret this result and whether it is meaningful to compare NMI scores between GH and the other methods.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "While the reviewer raised so many questions, we sincerely appreciate the reviewer's time and effort on our submission. Probably due to the difference in small research directions, the reviewer has some misunderstanding in the setting, background and evaluation of this topic. We do hope that the following point-to-point responses can address the concerns raised by the reviewer. Any further comments are welcomed.\n\n> **Q1:** Typos, grammatical errors and notation usage.\n\n**A1:** Sorry for the misunderstanding due to some typos, the definition $R$ and the notation usage. We will carefully address each of then in the revision to ease the understanding. Specially, we would like to explain that $R = N_{max}/N_{min}$ denotes the imbalance ratio, where $N_{max}$,$N_{min}$ represents the sample number in the largest\\/smallest class, and will revise the mentioned notation as $n_{L_H+1}=n_{L_H+2}=\\dots=n_{L}=n_T$, and use $J$ denote the number of negative samples.\n\n> **Q2:** Supervised long-tailed benchmark.\n\n**A2:** Thanks for the question. We would like to kindly clarify the differences between self-supervised long-tailed (SSL-LT) learning and the recommended supervised long-tailed learning (S-LT). Firstly, SSL-LT does not rely on any label information in the training stage, while S-LT leverages the fully supervised data. Furthermore, SSL-LT aims to address the issue of representation disparity, pursuing more balanced embedding space. In contrast, S-LT focuses on the classification disparity, which involves rectifying both the representation and the last-layer classifier. This distinction encourages different evaluation and corresponding performance of two paradigms. Specially, in SSL-LT, linear probing is employed to evluate the quality of representation, while classification is adopted to evaluate the accuracy of the predictions in S-LT. Consequently, direct performance comparison between SSL-LT and S-LT may not be feasible or appropriate.\n\n> **Q3:** Motivation.\n\n**A3:** Thanks for the questions. We would like to kindly clarify on the evidence regarding the limitation of contrastive learning on long-tailed data as follows:\n\n- Linear probing evidence. Table 2 shows that many-shot classes outperforms few-shot classes by an average of 5.67\\%/9.87\\% on CIFAR-LT/ImageNet-LT with SimCLR [11]. This indicates that contrastive loss is not immune to class imbalance, leading to performance disparities. Our empirical findings are consistent with several recent SSL-LT explorations [14,16,20,21].\n- Cross-dataset transfer. We conduct experiments on large-scale long-tailed CC3M and evaluated on various downstream tasks (See **Table R3-R4**). The results reveals that representation disparity actually exists in real-world data and GH promotes better transferability.\n- Uniformity metric. **Table R5** reveals that contrastive learning struggles to achieve the class-uniform partition, while our method effectively mitigates this issue. \n- Qualitive analysis. **Figure 1,R1** show that contrastive learning leads to representation disparity, where head classes dominate the embedding space and tail classes exhibit passive collapse. \n\nWe will enhance the explanation about the difference of two sub-areas to avoid misunderstanding.\n\n> **Q4:** Clarification on first contribution.\n\n**A4:** Thanks for the question. We would like to reclarify our taxonomy as described in **Lines 28-38** that all previous explorations target to improve self-supervised long-tailed learning, but from different aspects like reweight/optimization techniques, architecture design and data augmentation. However, **none of these methods focus on the intrinsic limitation of the contrastive learning**. Concretely, we provide more detailed aspects in **Table G1** in the general response for clarity.\n\n\n\n> **Q5:** Minor issues: (1) Oval-like embedding space in Figure 1, (2) long sentence in line 5, (3) twice \"the\" in line 162, (4) cite dataset and model, (5) “correspond” in line 162 (6) Table 7 location. (7) intuitions in line 290.\n\n**A5:** Thanks for your detailed suggestions. We will carefully follow the reviewer's advice to revise each of them in the revision.\n\n> **Q6:** Questions: (1) IR in Table 1, (2) marginal distribution constraint in line 119, (3) computational cost in Table 5, (4) many/medium/few partitions\n\n**A6:** Thanks for your detailed questions. Below are explanations to each questions:\n\n- IR indicates the imbalanced ratio $R$.\n- It means the distribution prior to determine the marginal projections of matrix $\\hat{Q}$ onto its rows and columns.\n- Table 5 shows the time cost per mini-batch. (See caption in Table 5)\n- Please refer to line 692-696 in Section E.1 in appendix for partition details.\n\n> **Q7:** Sensitivity to the hyper-parameters.\n\n**A7:** Thanks for the question. We would like to kindly clarify that many important hyperparameters have been compared or discussed (See Figure 3(a) and Table 11-13, Figure 5). Besides, we follow reviewer's advice to conduct more experiments on temperature $\\gamma_{GH}$, coefficient $\\lambda$ and Sinkhorn iteration $E_s$ on CIFAR-LT. The results in **Figure R3** shows that GH consistently achieves satisfying performance with different hyperparameter.\n\n> **Q8:** NMI and linear probing.\n\n**A8:** Thanks for the question. Below are the detailed reasons:\n\n- Linear probing removes the projector and GUS, while NMI uses their predictions.\n- NMI evaluates on train set, while linear probing evaluates on test set.\n- NMI is the normalization of the mutual information, while linear probing reports accuracy. \n\nThese differences make the scale for the relative improvement in NMI cannot be compared with that for accuracy.\n\n> **Q9:** Applications in fairness research.\n\n**A9:** Thanks for the constructive suggestions. We will add more discussions about how GH can benefit the applications situated in fairness research, especially from the perspective of representation parity, and cite the related works."}]

Author rebuttal:
We gratefully thank all the reviewers for their devoted efforts and constructive suggestions on this paper. We are glad that the reviewers have some positive impressions of our work, including: 

- Explorations on an **important and useful** problem (M9kH, pT3P, G4d9).
- **The motivation is clear** (krnL, M9kH, TBJn).
- The method is **interesting, novel, solid, reasonable**, and can be **seamlessly** integrated into existing SSL methods (krnL, M9kH, TBJn, pT3P, G4d9, R4di).
- **Extensive and solid** experiments with **plenty of ablation studies** (krnL, M9kH, TBJn, pT3P). 
- The paper is **well-written and easy to follow** (krnL, M9kH, TBJn, pT3P, G4d9).

We have addressed the reviewers' comments and concerns in **individual responses to each reviewer**. The reviews allowed us to improve our draft, and the changes made in our responses are summarized below:


- To back up the importance of the studies on self-supervised long-tailed learning, we provide further evidence with both the quantitative and qualitative empirical studies, see Table R3-R5, Figure R1.
- To address the concerns on the cross-dataset transferability of the proposed GH, we add more empirical evidence on various cross-dataset tasks pretrained on large-scale long-tailed CC3M, see Table R3-R4.
- We incorporate our method to more self-supervised methods, including BYOL and MoCo-v3 (See Table R1). Besides, we add empirical comparisons with several concurrent studies, including PMSN and TS (See Table R2).
- We further expand the ablation studies with comprehensive experiments, including sensitivity analysis of hyperparameters (See Figure R3), training configurations (See A2 to Reviewer pT3P), t-SNE visualization (See Figure R1) and uniformity analysis (See Table R5).
- We provide a more in-depth and comprehensive analysis on several points, including difference between SSL-LT and S-LT, the limitation of vanilla contrastive learning, the contribution of GH to SSL-LT, analysis of GH on other SSL methods, analysis of assumption on neighboring samples in the same class and advantages of surrogate label allocation. 

**We appreciate all reviewers' great effort again!** We have tried our best to address your concerns and improve the paper following the suggestions. **Would you mind checking it and confirming if there are any unclear parts?**

**Tables:**

[**Table G1.** Taxonomy of self-supervised long-tailed methods.]
| Method | Aspect | Description         |
| ------ | ----------- | ------------------- |
| Focal [12]  | Sample Reweighting        | Hard example mining |
|     rwSAM [13]  |   Optimization Surface         |  Data-dependent sharpness-aware minimization                   |
|   SDCLR [14] | Model  Pruning    | Model pruning and self-contrast  |
| DnC [15]   | Model  Capacity   | Multi-expert ensemble            |
| BCL [16]   | Data  Augmentation  | Memorization-guided augmentation |
| GH     | Loss  Limitation      | Geometric harmonization          | 

**References:**

[1] Piyush Sharma et al. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. ACL 2018.

[2] Jia Deng et al. Imagenet: A large-scale hierarchical image database. CVPR 2009.

[3] Bolei Zhou et al. Places: A 10 million image database for scene recognition. TPAMI 2017.

[4] Catherine Wah et al. The caltech-ucsd birds-200-2011 dataset. 2011.

[5] Subhransu Maji et al. Fine-grained visual classification of aircraft. 2013.

[6] Jonathan Krause et al. 3d object representations for fine-grained categorization. ICCV workshop 2013.

[7] Aditya Khosla et al. Novel dataset for fine-grained image categorization: Stanford dogs. CVPR workshop 2011.

[8] Van Horn et al. Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. CVPR 2015.

[9] Tsung-Yi Lin et al. Microsoft coco: Common objects in context. ECCV 2014.

[10] Quentin Garrido et al. On the duality between contrastive and non-contrastive self-supervised learning. ICLR 2023.

[11] Ting Chen et al. A simple framework for contrastive learning of visual representations. ICML 2020.

[12] Tsung-Yi Lin et al. Focal loss for dense object detection. ICCV 2017.

[13] Hong Liu et al. Self-supervised learning is more robust to dataset imbalance. ICLR 2022.

[14] Ziyu Jiang et al. Self-damaging contrastive learning. ICML 2021.

[15] Yonglong Tian et al. Divide and contrast: Self-supervised learning from uncurated data. ICCV 2021.

[16] Zhihan Zhou et al. Contrastive Learning with Boosted Memorization. ICML 2022.

[17] Asano Yuki Markus et al. Self-labelling via simultaneous clustering and representation learning. ICLR 2019.

[18] Mathilde Caron et al. Unsupervised learning of visual features by contrasting cluster assignments. NeurIPS 2020.

[19] Jiye Liang et al. The K-means-type algorithms versus imbalanced data distributions. TFS 2012.

[20] Mahmoud Assran et al. The hidden uniform cluster prior in self-supervised learning. ICLR 2023.

[21] Anna Kukleva et al. Temperature schedules for self-supervised contrastive methods on long-tail data. ICLR 2023.

[22] Kaiming He et al. Momentum contrast for unsupervised visual representation learning. CVPR 2020.

[23] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. CVPR 2021.

[24] Jure Zbontar et al. Barlow twins: Self-supervised learning via redundancy reduction. ICML 2021.

[25] Jean-Bastien Grill et al. Bootstrap your own latent-a new approach to self-supervised learning. NeurIPS 2020.

[26] Xinlei Chen et al. An Empirical Study of Training Self-Supervised Vision Transformers. ICCV 2021.

[27] Kaiming He et al. Masked autoencoders are scalable vision learners. CVPR 2022.

[28] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. ICML 2020.

[29] Tianhong Li et al. Targeted supervised contrastive learning for long-tailed recognition. CVPR 2022.

Meta-review: Six experts reviewed the paper, and all were positive after the rebuttal phase. The reviewers liked the idea and implementations, and the authors' experiments under the transfer setting are well appreciated. Hence, the decision is to recommend the paper for acceptance. We hope the authors will revise the paper according to the reviewers' feedback and the rebuttals.
Decision: Accept (spotlight)