SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Implementation Details. Following previous works [28, 77], we use ResNet-18 [23] as the backbone for small-scale dataset (CIFAR-100-LT [5]) and ResNet-50 [23] for large-scale datasets (ImageNet-LT [44], Places-LT [44]). For experiments on CIFAR-100-LT, we train model with the SGD optimizer, batch size 512, momentum 0.9 and weight decay factor  $5 \times 10^{-4}$  for 1000 epochs. For experiments on ImageNet-LT and Places-LT, we only train for 500 epochs with the batch size 256 and weight decay factor  $1 \times 10^{-4}$ . For learning rate schedule, we use the cosine annealing decay with the learning rate  $0.5 \rightarrow 1e^{-6}$  for all the baseline methods. As GH is combined with baselines, a proper warming-up of 500 epochs on CIFAR-100-LT and 400 epochs on ImageNet-LT and Places-LT are applied. The cosine decay is set as  $0.5 \rightarrow 0.3$ ,  $0.3 \rightarrow 1e^{-6}$  respectively. For hyper-parameters of GH, we provide a default setup across all the experiments: set the geometric dimension  $K$  as 100,  $w_{\mathrm{GH}}$  as 1 and the temperature  $\gamma_{\mathrm{GH}}$  as 0.1. In the surrogate label allocation, we set the regularization coefficient  $\lambda$  as 20 and Sinkhorn iterations  $E_s$  as 300. Please refer to Appendix E.3 for more experimental details.
Evaluation Metrics. Following [28, 77], linear probing on a balanced dataset is used for evaluation. We conduct full-shot evaluation on CIFAR-100-LT and few-shot evaluation on ImageNet-LT and Places-LT. For comprehensive performance comparison, we present the linear probing performance and the standard deviation among three disjoint groups, i.e., [many, medium, few] partitions [44].
4.2 Linear Probing Evaluation
CIFAR-100-LT. In Table 2, we summarize the linear probing performance of baseline methods  $w/$  and  $w/o$  GH on a range of benchmark datasets, and provide the analysis as follows.
(1) Overall Performance. GH achieves the competitive results w.r.t the [many, medium, few] groups, yielding a overall performance improvements averaging as  $2.32\%$ ,  $2.49\%$  and  $1.52\%$  on CIFAR-100-LT with different imbalanced ratios. It is worth noting that on the basis of the previous state-of-the-art BCL, our GH further achieves improvements by  $1.20\%$ ,  $1.82\%$  and  $1.22\%$ , respectively. Our
Table 2: Linear probing results on CIFAR-100-LT with different imbalanced ratios (100, 50, 10), ImageNet-LT and Places-LT. Many/Med/Few ( $\uparrow$ ) indicate the average accuracy (\%) w.r.t fine-grained partitions according to the class cardinality. Std ( $\downarrow$ ) means the standard deviation of the group-level performance and Avg ( $\uparrow$ ) is the average accuracy (\%) of the full test set. Following the previous work [28, 77], Std represents a balancedness measure to quantify the variance among three specified groups. Improv. ( $\uparrow$ ) represents the averaging performance improvements w.r.t. different baseline methods. We report the error bars with the multi-run experiments in Table 12 in Appendix F.
Dataset	SimCLR	+GH	Focal	+GH	SDCLR	+GH	DnC	+GH	BCL	+GH	Improv.
CIFAR-R100	Many	54.97	57.38	54.24	57.01	57.32	57.44	55.41	57.56	59.15	59.50	+1.56
Med	49.39	52.27	49.58	52.93	50.70	52.85	51.30	53.74	54.82	55.73	+2.35
Few	47.67	52.12	49.21	51.74	50.45	54.06	50.76	53.26	55.30	57.67	+3.09
Std	3.82	2.99	2.80	2.76	3.90	2.38	2.54	2.36	2.37	1.89	-0.61
Avg	50.72	53.96	51.04	53.92	52.87	54.81	52.52	54.88	56.45	57.65	+2.32
CIFAR-R50	Many	56.00	58.88	55.40	57.97	57.50	58.47	56.03	59.04	59.44	60.82	+2.16
Med	50.48	53.00	51.14	53.55	51.85	53.88	52.68	55.05	54.73	57.58	+2.44
Few	50.12	54.27	50.02	53.58	52.15	53.58	50.83	54.81	57.30	58.55	+2.87
Std	3.30	3.09	2.84	2.54	3.18	2.74	2.64	2.38	2.36	1.66	-0.38
Avg	52.24	55.42	52.22	55.06	53.87	55.34	53.21	56.33	57.18	59.00	+2.49
CIFAR-R10	Many	57.85	59.26	58.18	60.06	58.47	59.21	59.82	61.09	60.41	61.41	+1.26
Med	55.06	56.91	55.82	56.79	54.79	56.06	56.67	58.33	57.15	59.27	+1.57
Few	54.03	55.85	54.64	57.24	52.97	55.58	56.21	57.33	59.76	60.30	+1.74
Std	1.98	1.75	1.80	1.77	2.80	1.97	1.96	1.95	1.73	1.07	-0.35
Avg	55.67	57.36	56.23	58.05	55.44	56.97	57.59	58.94	59.12	60.34	+1.52
ImageNet-LT	Many	41.69	41.53	42.04	42.55	40.87	41.92	41.70	42.19	42.92	43.22	+0.44
Med	33.96	36.35	35.02	36.75	33.71	36.53	34.68	36.63	35.89	38.16	+2.23
Few	31.82	35.84	33.32	36.28	32.07	36.04	33.58	35.86	33.93	36.96	+3.25
Std	5.19	3.15	4.62	3.49	4.68	3.26	4.41	3.45	4.73	3.32	-1.39
Avg	36.65	38.28	37.49	38.92	36.25	38.53	37.23	38.67	38.33	39.95	+1.68
Places-LT	Many	31.98	32.46	31.69	32.40	32.17	32.78	32.07	32.51	32.69	33.22	+0.55
Med	34.05	35.03	34.33	35.14	34.71	35.60	34.51	35.55	35.37	36.00	+0.87
Few	35.63	36.14	35.73	36.49	35.69	36.18	35.84	35.91	37.18	37.62	+0.45
Std	1.83	1.89	2.05	2.08	1.82	1.82	1.91	1.87	2.26	2.23	0.00
Avg	33.61	34.33	33.65	34.42	33.99	34.70	33.90	34.52	34.76	35.32	+0.68
GH consistently improves performance across datasets with varying degrees of class imbalance, demonstrating its potential to generalize to practical scenarios with more complex distributions. Specially, our method does not require any prior knowledge or assumptions about the underlying data distribution, highlighting the robustness and versatility to automatically adapt to the data.
(2) Representation Balancedness. In Section 1, we claim that GH helps compress the expansion of head classes and avoid the passive collapse of tail classes, yielding the more balanced representation distribution. To justify this aspect, we compare the variance in linear probing performance among many/medium/few groups, namely, their groupwise standard deviation. According to Table 2, our GH provides  $[1.56\%, 2.35\%, 3.09\%]$ ,  $[2.16\%, 2.44\%, 2.87\%]$  and  $[1.26\%, 1.57\%, 1.74\%]$  improvements w.r.t [many, medium, few] groups on CIFAR-100-LT-R100/R50/R10 with more preference to the minority classes for representation balancedness. Overall, GH substantially improves the standard deviation by [0.61, 0.38, 0.35] on different levels of imbalance.
ImageNet-LT and Places-LT. Table 2 shows the comparison of different baseline methods on large-scale dataset ImageNet-LT and Places-LT, in which we have consistent observations. As can be seen, on more challenging real-world dataset, GH still outperforms other methods in terms of overall accuracy, averaging as  $1.68\%$ ,  $0.68\%$  on ImageNet-LT and Places-LT. Specifically, our method provides  $[0.44\%, 2.23\%, 3.25\%]$  and  $[0.55\%, 0.87\%, 0.45\%]$  improvements in linear probing w.r.t. [many, medium, few] groups on ImageNet-LT and Places-LT. The consistent performance overhead indicates the robustness of our method to deal with long-tailed distribution with different characteristics. Moreover, the averaging improvement of standard deviation is 1.39 on ImageNet-LT, indicating the comprehensive merits of our GH on the minority classes towards the representation balancedness. However, an interesting phenomenon is that the fine-grained performance exhibits a different trend on Places-LT. As can be seen, the performance of head classes is even worse than that of tail classes. The lower performance of the head partition can be attributed to the fact that it contains more chal
Table 3: Supervised long-tailed learning by finetuning on CIFAR-100-LT, ImageNet-LT and PlacesLT. We compare the performance of five self-supervised learning methods as the pre-training stage for downstream supervised logit adjustment [46] method. Improv.  $(\uparrow)$  represents the averaging performance improvements w.r.t. different baseline methods. Besides, the performance of logit adjustment via learning from scratch is also reported for comparisons.
Dataset	LA	Logit adjustment pretrained with the following SSL methods	Improv.
SimCLR	+GH	Focal	+GH	SDCLR	+GH	DnC	+GH	BCL	+GH
CIFAR-LT	46.61	49.81	50.84	49.83	51.04	49.79	50.73	49.97	50.84	50.38	51.32	+1.00
ImageNet-LT	48.27	51.10	51.67	51.15	51.82	50.94	51.64	51.31	51.88	51.43	52.06	+0.63
Places-LT	27.07	32.63	33.86	32.69	33.75	32.55	34.03	32.98	34.09	33.15	34.48	+1.24
Table 4: Image classification on ImageNet, Places and fine-grained visual classification on various fine-grained datasets, pretrained on large-scale long-tailed CC3M and then finetuned.
Image Classification	Fine-Grained Visual Classification
ImageNet	Places	CUB200	Aircraft	StanfordCars	StanfordDogs	NABirds	Average
SimCLR	52.06	37.65	44.61	65.89	57.63	50.99	46.86	53.20
+GH	53.39	38.47	45.76	68.08	60.24	52.88	47.58	54.91
lenging classes. As a result, we observe that the standard deviation of our GH does not significantly decrease on Places-LT, which requires more effort and exploration for improvement alongside GH.
4.3 Downstream Finetuning Evaluation
Downstream supervised long-tailed learning. Self-supervised learning has been proved to be beneficial as a pre-training stage of supervised long-tailed recognition to exclude the explicit bias from the class imbalance [68, 41, 77].
To validate the effectiveness of our GH, we conduct self-supervised pre-training as the initialization for downstream supervised classification tasks on CIFAR-100-LT-R100, ImageNet-LT and PlacesLT. The state-of-the-art logit adjustment [46] is chosen as the downstream baseline. The combination of  $\mathrm{GH} + \mathrm{LA}$  can be interpreted as a compounded method where GH aims at the re-balanced representation extraction and LA targets the classifier debiasing. In Table 3, we can find that the superior performance improvements are achieved by self-supervised pre-training over the plain supervised learning baseline. Besides, our method can also consistently outperform other SSL baselines, averaging as  $1.00\%$ ,  $0.63\%$  and  $1.24\%$  on CIFAR-100-LT-R100, ImageNet-LT and Places-LT. These results demonstrate that GH are well designed to facilitate long-tailed representation learning and improve the generalization for downstream supervised tasks.
Cross-dataset transfer learning. To further demonstrate the representation transferability of our GH, we conduct more comprehensive experiments on the large-scale, long-tailed dataset CC3M [52] with various cross-dataset transferring tasks, including downstream classification, object detection and instance segmentation. Specifically, we report the finetuning classification performance on ImageNet, Places and fine-grained visual datasets Caltech-UCSD Birds (CUB200) [61], Aircrafts [45], Stanford Cars [33], Stanford Dogs [32], NABirds [60]. Besides, we evaluate the quality of the learned representation by finetuning the model for object detection and instance segmentation on COCO2017 benchmark [39]. As shown in Tables 4 and 5, we can see that our proposed GH consistently outperforms the baseline across various tasks and datasets. It further demonstrates the importance of considering long-tailed data distribution under large-scale unlabeled data in the pretraining stage. This can potentially be attributed to that our geometric harmonization motivates a more balanced and general embedding space, improving the generalization ability of the pretrained model to a range of real-world downstream tasks.
4.4 Further Analysis and Ablation Studies
Dimension of Geometric Uniform Structure. As there is even no category number  $L$  available in SSL paradigm, we empirically compare our GH with different geometric dimension  $K$  on CIFAR-100-LT-R100, as shown in Figure 3(a). From the results, GH is generally robust to the change of
Table 5: Object detection and instance segmentation with Table 6: Inter-class uniformity (↑) and finetuned features on COCO2017 benchmark, pretrained neighborhood uniformity (↑) of pretrained on large-scale long-tailed CC3M.
Object Detection	Instance Segmentation	Inter-class Uniformity	Neighborhood Uniformity
APbbox	APbbox50	APbbox75	APmask	APmask50	APmask75	SimCLR	+GH	SimCLR	+GH
SimCLR	31.7	51.0	33.9	30.2	49.8	32.1	C100	1.00	2.80	0.72	2.00
C50	1.23	2.73	0.91	1.94
+GH	32.7	52.2	35.2	31.1	50.8	33.0	C10	1.18	2.60	0.85	1.83
(a) Geometric Dimension
(b) Label Quality
Figure 3: (a) Linear probing performance w.r.t. the dimension  $K$  of the geometric uniform structure M (Appendix B) on CIFAR-LT-R100. (b) NMI score between the surrogate geometric labels and the ground-truth labels in the training stage on CIFAR-LT-R100. (c) Average linear probing and the error bars of the surrogate label allocation with variants of the label prior on CIFAR-LT-R100.
(c) Label-Distribution Prior
$K$ , but slightly exhibits the performance degeneration when the dimension is extremely large or small. Intuitively, when  $K$  is extremely large, our GH might pay more attention to the uniformity among sub-classes, while the desired uniformity on classes is not well guaranteed. Conversely, when  $K$  is extremely small, the calibration induced by GH is too coarse that cannot sufficiently avoid the internal collapse within each super-class. For discussions of the structure, it can refer to Appendix B.
Surrogate Label Quality Uncovered. To justify the effectiveness of surrogate label allocation, we compare the NMI scores [54] between the surrogate and ground-truth label in Figure 3(b). We observe that GH significantly improves the NMI scores across baselines, indicating that the geometric labels are effectively calibrated to better capture the latent semantic information. Notably, the improvements of the existing works are marginal, which further verifies the superiority of GH.
Exploration with Other Label-Distribution Prior. To further understand  $\pi$ , we assume the ground-truth label distribution is available and incorporate the oracle  $\pi^y$  into the surrogate label allocation. Comparing the results of the softened variants  $\pi_{\gamma_{\mathrm{T}}}^{y}$  with the temperature  $\gamma_{\mathrm{T}}$  in Figure 3(c), we observe that GH outperforms all the counterparts equipped with the oracle prior. A possible reason is that our method automatically captures the inherent geometric statistics from the embedding space, which is more reconcilable to the self-supervised learning objectives.
Uniformity Analysis. In this part, we conduct experiments with two uniformity metrics [62, 37]:
$$
\mathrm {U} = \frac {1}{L (L - 1)} \sum_ {i = 1} ^ {L} \sum_ {j = 1, j \neq i} ^ {L} \| \boldsymbol {\mu} _ {i} - \boldsymbol {\mu} _ {j} \| _ {2}, \quad \mathrm {U} _ {k} = \frac {1}{L k} \sum_ {i = 1} ^ {L} \min  _ {j _ {1}, \dots , j _ {k}} (\sum_ {m = 1} ^ {k} \| \boldsymbol {\mu} _ {i} - \boldsymbol {\mu} _ {j _ {m}} \| _ {2}),
$$
where  $j_1, \ldots, j_k \neq i$  represent different classes. Specifically, U evaluates average distances between different class centers and  $\mathrm{U}_k$  measures how close one class is to its neighbors. As shown in Table 6, our GH outperforms in both inter-class uniformity and neighborhood uniformity when compared with the baseline SimCLR [9]. This indicates that vanilla contrastive learning struggles to achieve the uniform partitioning of the embedding space, while our GH effectively mitigates this issue.
Comparison with More SSL Methods. In Table 7, we present a more comprehensive comparison of different SSL baseline methods, including MoCo-v2 [24], MoCo-v3 [11] and various noncontrastive methods such as SimSiam [10], BYOL [20] and Barlow Twins [71]. From the results, we can see that the combinations of different SSL methods and our GH can achieve consistent performance improvements, averaging as  $2.33\%$ ,  $3.18\%$  and  $2.21\%$  on CIFAR-100-LT. This demonstrates the prevalence of representation learning disparity under data imbalance in general SSL settings.
Table 7: Linear probing of more SSL variants on CIFAR-100-LT with different imbalanced ratios.
Dataset	SimSiam	+GH	Barlow	+GH	BYOL	+GH	MoCo-v2	+GH	MoCo-v3	+GH
CIFAR-R100	49.01	51.43	48.70	51.23	51.43	52.87	51.49	53.53	54.08	55.82
CIFAR-R50	48.98	53.54	49.29	51.95	52.04	53.84	52.68	55.01	55.34	56.45
CIFAR-R10	55.51	57.03	53.11	56.34	55.86	57.28	58.23	60.11	59.10	60.57
Figure 4: T-SNE visualization of the pretrained features from 8 ran- Figure 5: Ablations of domly selected classes w.r.t CIFAR-LT training images.  $w_{GH}$  on CIFAR-LT.
On Importance of Bi-Level Optimization. In Table 8, we empirically compare the direct joint optimization strategy to Eq. (4). From the results, we can see that the joint optimization  $(w / o$  or  $w / o$  the warm-up strategy) does not bring significant performance improvement over SimCLR compared with that of our bi-level optimization, probably due to the undesired collapse in label allocation [1]. This demonstrates the necessity of the proposed bi-level optimization for Eq. (4) to stabilize the training.
Table 8: Linear probing of joint optimization on CIFAR-100-LT with different IRs.
Method	C100	C50	C10
SimCLR	50.72	52.24	55.67
+GH (Joint)	50.18	52.31	54.98
w/ warm-up	51.14	52.75	55.37
+GH (Bi-level)	53.96	55.42	57.36
Qualitative Visualization. We conduct t-SNE visualization of the learned features to provide further qualitative intuitions. For simplicity, we randomly selected four head classes and four tail classes on CIFAR-LT to generate the t-SNE plots. Based on the results in Figure 4, the observations are as follows: (1) SimCLR: head classes exhibit a large presence in the embedding space and heavily squeeze the tail classes, (2) GH: head classes reduce their occupancy, allowing the tail classes to have more space. This further indicates that the constructed surrogate labels can serve as the high-quality supervision, effectively guiding the harmonization towards the geometric uniform structure.
Sensitivity Analysis. To further validate the stability of our GH, We conduct empirical comparison with different weight  $w_{\mathrm{GH}}$ , temperature  $\gamma_{\mathrm{GH}}$ , regularization coefficient  $\lambda$  and Sinkhorn iteration  $E_{s}$  on CIFAR-LT, as shown in Figures 5 and 7. From the results, we can see that our GH can consistently achieve satisfying performance with different hyper-parameter.
In this paper, we delve into the defects of the conventional contrastive learning in self-supervised long-tail context, i.e., representation learning disparity, motivating our exploration on the inherent intuition for approaching the category-level uniformity. From the geometric perspective, we propose a novel and efficient Geometric Harmonization algorithm to counteract the long-tailed effect on the embedding space, i.e., over expansion of the majority class with the passive collapse of the minority class. Specially, our proposed GH leverages the geometric uniform structure as an optimal indicator and manipulate a fine-grained label allocation to rectify the distorted embedding space. We theoretically show that our proposed method can harmonize the desired geometric property in the limit of loss minimum. It is also worth noting that our method is orthogonal to existing self-supervised long-tailed methods and can be easily plugged into these methods in a lightweight manner. Extensive experiments demonstrate the consistent efficacy and robustness of our proposed GH. We believe that the geometric perspective has the great potential to evolve the general self-supervised learning paradigm, especially when coping with the class-imbalanced scenarios.
This work was supported by the National Key R&D Program of China (No. 2022ZD0160702), STCSM (No. 22511106101, No. 22511105700, No. 21DZ1100100), 111 plan (No. BP0719010) and National Natural Science Foundation of China (No. 62306178). BH was supported by the NSFC Young Scientists Fund No. 62006202, NSFC General Program No. 62376235, and Guangdong Basic and Applied Basic Research Foundation No. 2022A1515011652.
[1] Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In International Conference on Learning Representations (ICLR), 2020.
[2] Mido Assran, Randall Balestriero, Quentin Duval, Florian Bordes, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, and Nicolas Ballas. The hidden uniform cluster prior in self-supervised learning. In *The Eleventh International Conference on Learning Representations*, 2023.
[3] Tanya Boone-Sifuentes, Asef Nazari, Imran Razzak, Mohamed Reda Bouadjenek, Antonio Robles-Kelly, Daniel Ierodiaconou, and Elizabeth S Oh. Marine-tree: A large-scale marine organisms dataset for hierarchical image classification. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 3838-3842, 2022.
[4] Sergiy V Borodachov, Douglas P Hardin, and Edward B Saff. Discrete energy on rectifiable sets. Springer, 2019.
[5] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. Advances in Neural Information Processing Systems, 32:1567-1578, 2019.
[6] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pages 132-149, 2018.
[7] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9924, 2020.
[8] Mengxi Chen, Linyu Xing, Yu Wang, and Ya Zhang. Enhanced multimodal representation learning with cross-modal kd. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11766-11775, 2023.
[9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1597-1607. PMLR, 2020.
[10] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
[11] Xinlei Chen*, Saining Xie*, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.02057, 2021.
[12] Henry Cohn and Abhinav Kumar. Universally optimal distribution of points on spheres. Journal of the American Mathematical Society, 20(1):99-148, 2007.
[13] Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 715-724, 2021.
[14] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.