Geometry of Long-Tailed Representation Learning: Rebalancing Features for Skewed Distributions
Authors: Lingjie Yi
Jiachen Yao
Weimin Lyu
Haibin Ling
Raphael Douady
Chao Chen
Venue: ICLR 2025 Poster (2025)
Keywords: contrastive learning
representation learning
long-tail recgonition
theory
neural collapse
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Deep learning has achieved significant success by training on balanced datasets. However, real-world data often exhibit long-tailed distributions. Empirical studies have revealed that long-tailed data skew data representations, where head classes dominate the feature space. Many methods have been proposed to empirically rectify the skewed representations. However, a clear understanding of the underlying cause and extent of this skew remains lacking. In this study, we provide a comprehensive theoretical analysis to elucidate how long-tailed data affect feature distributions, deriving the conditions under which centers of tail classes shrink together or even collapse into a single point. This results in overlapping feature distributions of tail classes, making features in the overlapping regions inseparable. Moreover, we demonstrate that merely empirically correcting the skewed representations of the training data is insufficient to separate the overlapping features due to distribution shifts between the training and real data. To address these challenges, we propose a novel long-tailed representation learning method, FeatRecon. It reconstructs the feature space in order to arrange features from different classes into symmetricial and linearly separable regions. This, in turn, enhances the model’s robustness to long-tailed data. We validate the effectiveness of our method through extensive experiments on the CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018 datasets.

Published as a conference paper at ICLR 2025
GEOMETRY OF LONG-TAILED REPRESENTATION LEARNING: REBALANCING FEATURES FOR SKEWED DISTRIBUTIONS
Lingjie ;i'; Jiachen Yao', Weimin Lyu', Haibin Ling', Raphael Douady", Chao Chen'
Stony Brook University, University Paris Pantheon-Sorbonne
ABSTRACT
Deep learning has achieved significant success by training on balanced datasets. However, real-world data often exhibit long-tailed distributions. Empirical studies have revealed that long-tailed data skew data representations, where head classes dominate the feature space. Many methods have been proposed empirically rectify the skewed representations. However, clear understanding of the under- lying cause and extent of this skew remains lacking. In this study, we provide a comprehensive theoretical analysis to elucidate how long-tailed data affect feature distributions, deriving the conditions under which centers of tail classes shrink together
collapse into single point. This results in overlapping feature distributions of tail classes, making features the overlapping regions inseparable. Moreover, demonstrate that merely empirically correcting the skewed repre- sentations of the training data is insufficient to separate the overlapping features due distribution shifts between the training and real data. To address these chal- lenges, we propose a novel long-tailed representation learning method, FeatRecon. It reconstructs the feature space to arrange features from different classes into symmetrical and linearly separable regions. This, in turn, enhances the model's 's robustness to long-tailed data. We validate the effectiveness of our method through extensive experiments on the CIFAR-IO-LT, CIFAR-100-LT, ImageNet-LTT, and iNaturalist 2018 datasets.
INTRODUCTION
Deep learning models have achieved significant success by training large-scabe artificially balanced datasets (i.c., ImageNet (Deng et al., 22009). However, in real-world scenarios, datasets often exhibit long-tailed distributions, characterized by highly imbalanced class distribution (i.e., the sample sizes of different classes). A few classes (called head classes) have a large number of samples, whereas many other classes (called tail classes) contain only few samples. Training such datasets distorts a model's 's feature representations and decision boundaries, limiting the model's generalization capability and performance on test data.
Our understanding of balanced data representation has advanced significantly. For example, using the powerful representation learning tool, contrastive learning (Khosla et al. 2020). it has been shown (Graf et al.] (021)) that for balanced data, when the supervised contrastive loss (SC loss) reaches its minimum, the feature representations of each class converge their respective class centers, and all class centers collectively form regular simplex (see Theorem aad Fig. Ta). This highly symmetrical configuration ensures separation among different classes, resulting in strong classification performance.
However, the optimal representation configuration for imbalanced data remains poorly understood. Empirical studies have revealed that, when data follow long-tailed distribution, the optimal represen- tations form an asymmetrical configuration, with head classes dominating the feature space. Although several methods (Zhu et al., 2022; Kang et al. 2021 Li et al. 2022 have attempted to correct this asymmetry, they primarily rely on empirical adjustments. Crucially, none of these methods
"Email: chris.yi@ chris.yi@stonnyheee stonybrook.euuu

---

Published as a conference paper at ICLR 2025
34
01
4
22
(a) p =
P<1
(0)==1
< R
(e) p 2 R
Figure 1: Geometry of the optimal representation configuration for four classes with different imbalance factors, p. Centers of the four classes (ž1.... 21) are positioned on a unit hypersphere. Assume classes 2, 3 and 4 have the same size, N2 = N3 = N4. Class I's size is their size multiplied by p. _2 = p.Vz. (a): when class is empty (N1 = 0), classes 2, 3 and 4 form a regular simplex. (b) to (e): As p increases and N1 increases, Σ2, 2अ and Σn are pushed away from the equator and eventually collapse. (c): when p = 1, all four classes form a regular simplex. R is the critical constant which the collapse happens (see Sec. for details). 01 is the angle between the head class center and tail class centers, and 02 is the angle between tail classes.
provides a theoretical explanation of why and to what extent head classes dominate the feature space. Understanding this phenomenon could offer deeper insights into learning better representations of long-tailed data and inspire novel methods.
In this paper, we study the long-tailed data representation. We begin by establishing the first theoretical framework (in Theorem 2 to analyze the geometry of the optimal representation configuration, i.e., the arrangement of class centers when the SC loss is minimized, under different class distributions. In particular, derive the analytical relationship between the imbalance factor, i.c., the ratio of sample sizes between head and tail classes, and the angles between different class centers at the optimal configuration. We show that the imbalance factor increases, the head class increasingly dominates the feature space, pushing centers of the tail classes closer together. Beyond a certain critical threshold, centers of the tail classes collapse into a single point. Fig. illustrates the optimal configuration of four classes. From Fig. a to Fig. Te as the imbalance factor continuously increases, the center of the head class (2, pushes the other three tail classes' centers (32, 23 and Σn) closer and eventually collapse (Fig. lc
Our theory provides insights into how long-tailed data hurts the representation learning. Without any mitigation strategy, tail classes are pushed close to one another or even collapse, resulting in overlapping feature distributions and poor separability between them. To address this issue, existing methods often readjust the empirical tail class centers symmetric configuration. However, due to the limited sample sizes of tail classes, these approaches may overcorrect the issue, forcing the true centers of the tail classes to be too close to the head class, leading to overlapping distributions and poor separability between head and tail classes.
In this paper, we introduce FeatRecon, novel method to learn long-tailed representations. To rebalance the skewed feature distributions, this method reconstructs the feature space so that features of different classes arranged into symmetrical and linearly separable regions. Inspired by theoretical analysis, our method addresses the center-skew issue by rebalancing the sample sizes across all classes. This is accomplished by generating synthetic features for tail classes and using them for representation learning. To ensure linear separability, the synthetic features of each class are restricted within an estimated confidence support. i.c., the feature space region covering most samples in that class. We derive the necessary conditions for the confidence supports to ensure that they do not overlap at the optimal configuration.
Estimation of confidence support is crucial to our method. Direct estimation of the feature distributions challenging due the non- non-Euulidean geometry of the normalized feature space and the limited sample size of tail classes. Instead, we estimate the confidence support using the center of each class and a single "central angle" parameter. Since the estimation for tail classes can be unreliable, the statistics of tail classes are regularized using those of head classes. By iteratively generating synthetic
For completeness, analysis encompasses the cases when class I's size, Nr, is smaller than the others'. Technically, class 1 longer the head class when N₂ N₂2 N₂- Ng.
2

---

Published as a conference paper at ICLR 2025
features to fill these confidence supports, adjusting representations, and re-estimating confidence supports, we can learn a feature space where both head and tail classes are equally separated, with no overlap between their confidence supports.
Our contributions are summarized as follows:
We study the geometry of long-tailed data representation and explain how long-tailed data skew representations and limit model's generalization capability.
We develop novel algorithm to generate synthetic features balance sample sizes among all classes. The synthetic features are constrained within confidence supports which are estimated with head classes regularization.
We propose an iterative approach to learn a symmetric and linearly separable feature space for long-tailed data. Our method iteratively generates synthetic features, adjusting representations, and re-estimates confidence support.
We validate our method with experiments on commonly used datasets. FeatRecon outper- forms SOTA performance compared to widely adopted long-tail long-tailed learning baselines.
2 RELATED WORK
2.1 LONG-TAILED RECOGNITION
Resampling (Byrd & Lipton 2019) and re-weighting (Cui et al. 2019 Jamal et al. 2020 Chen et al. 2023) are two classical methods in long-tailed learning. The former method balances the number of training samples across different classes by either oversampling the tail classes or downsampling the head classes. The latter method balances the per class contributions the loss function by assigning higher weights to classes with smaller gradients. Other methods emphasize adjusting decision boundaries through either post-hoc weight normalization (Dang et al., 2024) or margin adjustment (Cao et al. 2019 Menon et al., [021] Khan et al. 2019)) The former type adjusts the decision boundaries by modifying the weight norms of the matrix of classifiers, while the latter type increases the margins of the tail classes.
Recent works also explore ideas in data augmentation (Ahn et al. 2023 Gao et al. 2024). which adjusts the strength of class-wise augmentation to help learn class-balanced representations. Transfer learning (Chen & Su 2023] Zhang et al., 2023) improve tail class learning by leveraging information from the nearby head classes. common way for transfer learning is to assume that data follow multivariate Gaussian distribution and transfer distribution statistics. However, robust estimation of parameters (i.e., the x K covariance matrix) can be challenging given the small sample sizes of tail classes, and the distributional assumption does not hold for normalized features that contrastive learning deals with.
2.2 CONTRASTIVE LEARNING FOR LONG-TAILED DATA
Contrastive learning (He et al. 2020 Chen et al. 2020 Caron et al. 2020 Chen & He 2021; Grill
al., 22020; Wang Isola 2020) has made great progress a tool for representation learning. Supervised contrastive learning (SCL) (Khosla et al. 2020), by optimizing supervised contrastive loss, learns symmetrical feature space in which the representations of each class collapse the vertices of regular simplex (Graf et al. 2021).
Recent studies long-tailed tailed learning (LTL) (Wang et al. 2021 Cui al. 2021 2023
Xuan & Zhang
2024) incorporate an SCL module into the LTL framework, aiming to learn better representations and train most robust classifiers. However, directly using SCL is not ideal, as some (Li et al. 2022 Zhu
et al. 2022) have demonstrated that SCL skews the feature space when training on taigetailed data.
Many methods have since focused on empirically readjusting these skewed representations. TCL (Li
al. 2022) addresses this by predefining well-separated empirical centers. Other methods rebalance
the number of contrastive pairs in the SC loss, i.e, positive contrastive pairs (Kang et al. (021).
negative contrastive pairs (Zhu et al. 2022). Our method balances contrastive pairs by generating new features for the tail classes on the surface of a unit hypersphere which are constrained within different separable hyperspherical caps.
3

---

Published as a conference paper at ICLR 2025
3 THEORETICAL ANALYSIS: LONG-TAILED DATA SKEW CONTRASTIVE FEATURE REPRESENTATIONS
In this section, study how long-tailed data skew the feature space. To understand how varying class distributions influence representations, we provide theoretical framework (in Sec. 3.2 to investigate the optimal representation configuration when SC loss is minimized (see Fig.
We show (in Theorem D. for balanced data, the optimal representations form a regular simplex. This reveals that representations of different classes equally separated to the largest extent. However, for imbalanced data, the optimal representation configuration becomes far more complex. Therefore, we focus the simplified one-vs-all scenario. We adjust the sample size of class 1, while assuming that the remaining K classes have equal and fixed sample size. In Theorem dynamic of the geometry of the optimal representation configuration the imbalance factor changes.
2 we study the
3.1 PRELIMINARIES
Suppose that we have N training samples, X = (T IN) E (X) randomly drawn from K distinct classes, with labels Y (11, ................................ MNN (D))a' and y [k] (1..... K}. A unit hypersphere (in R*) defined 5 E E Rh : = 11.. An encoder is map 4p X + R² that extracts representations from data, denoted Z = (o ......... (.......)))))
In practice, contrastive learning conducted batch-wise due to memory limitations. To simplify our analysis, assume unlimited memory to train all samples in single batch. We denote the set of indices of all samples as B = [N] = ; ]., and the set of indices of samples of the class k Bg {i E B, y kj. Let N₂ be the number of samples from class k. N& [B and N Ek=1 Nu. The following definitions are necessary for the study.
Definition (Supervised contrastive loss (SC loss)). Let Z be an N N-point configuration (assuming that all 25 are normalized), Z (z1...., EN) E (S*-1)N, with labels Y (y) s...... .NN [[K])*
and 3 < K S h 1. The supervised contrastive loss Lsc(; (Sh-1)N + IR is defined as
K
exp (12,, 2;)
Lsc = ΣΣ
ki. so where C&g
Σ
log
(1)
N₂
EiEBv() exp (126 exp (1) /T)
1 BR
JEBLMB
Definition (Equidistan/reeglla simplex). Let h, K E and K S h K-point configuration
(SI.... .Kx) (S*- form the vertices equidistant simplex inscribed in the unit hypersphere if and only if the following conditions hold:
(1) Vi E [K], I1all =1
(2) 3d E R.ER j and siisj
(Gi, Sz)
form the verfices of regular simplex if and only (I), (2) and the following condition holds:
(3) Eie(K] =0
3.2 OPTIMAL REPRESENTATION CONFIGURATION
In this subsection, we assume sufficiently powerful encoder capable of realizing any representation configuration, and set the temperature parameter (in Eq. to 1.
Optimal Representation Configuration for Balanced Data. When the training data balanced, The- orem[] states that the SC loss attains its minimum if and only if the features of each class converge at their respective class centers, and the centers of all classes form a regular simplex.
Theorem 1. Let Z be an N --point configuration (assuming that all zs are normalized), Z (21,...,ZN) (Sh-IyN, with labels Y (y1...... UN) E ([K]),, and s K < h When Y balanced, hence Vi E [K]. N₂- N. holds that:
((7)
N K
-1)
N(K -1
K K
Lsc 2 Nlgg
exp
(2)
K
4

---

Published as a conference paper at ICLR 2025
where equality is attained if and onty if there exists configuration of 2 = :,.....2x1 (Sh-1)K such that:
(AI) E Bk, 2, = 2x.
(A2) 7. form a regular simplex inscribed in the unit hypersphere.
Remark 1. This theorem has been previously established in Graf al. (2021). In this paper, provide refined proof (Appendix B. that dose presume Lsc (in Eq. (11 be the same when k. varies, as was done in (S39) of Graj et al. (207)) This allows us to extend the analysis to more general center configurations, particularly laying the foundation for the imbalanced data case (Theorem 2
Optimal Representation Configuration for Imbalanced Data. When the training data imbal- anced, we first find the tight lower bound function f of Lsc, assuming all features having converged to their respective class centers. therefore only depends on the center configuration. We then determine the optimal representation configuration by minimizing f. In the one-vs-all scenario. Theorem 2 states that the SC loss is minimized if and only if the features of classes 2 to K converge to the vertices of an equidistant simplex while the features of class 1 converge to the point that is perpendicular to the equidistant simplex (more explanations in Appendix (A.2)
Theorem 2. Let Z be an N-point configuration (assuming thut all zs are normalized), Z (ZI,...;ZN) E (Sh-1)N, with labels Y (1/..... yNN) ([K])*, and S S 1. If Wk E (2..... K}, N₂ a2 2 4, and 3p 0 such that N1 01 pa2 1, it holds that:
Lsc 2 (ccos(()).co((.)))
(3)
where f(): R IR is defined as:
f(11, p02 log ((pa2 -1)+ - (K - 1) 02 exp (x1))
(4)
+ (K 1) Jaz log 1) ((K 2) 02 exp (122) + paz exp (21)))
and equality is attained if and only if there exists a configuration of = (3)...., 2x) E (Sh-1) such that:
(A3) E Bi, 21 2k.
(A4) (ž2...., (...,kk)fonnn form an equidistant simplex whose verlex-centee--verte. angle equals 02
(AS) vk E f2....,KK, K}. (Z1, Zx) = cos(f1) and cos(f2) (K-1) c0.²
K-2
The numerical relationship between p and 0₁ can be sunmarized as:
(Case 1) < 1: 01 E
(cos-'(- (-x=-))0)
(Case 2) p= 01 = cos -(- x-1)) ((-x-1).
(Case 3) 1 < p < R(K, a2): 01 (-TT,00s* 1 (----/)
(Case 4) 2> R(K, 02): 01 -
Let bj = be defined as:
-2e2)22 -2, b2 --((ee - - then R(K oz)
-b bi +b
a(), = 2(1 + 2)az
Detailed proof is provided in Appendix B.2 We show that O2 depends on 0, when the representation configuration reaches its optimal. Let (()))(01) = (cos(O))) (o=1))oo((0))-1) and we prove that is a convex function of (0)) Therefore, g has one and only one minimal value within a given domain. We also show that g' is increasing function of p. So as p increases, 0, increases and 0₂ decreases. Here, 0 measures the extent of dominance of the head class.
K-2
5

---

Published as a conference paper at ICLR 2025
: 0.86
cos -1 of = 180
of = 149.19
90x)
g
111149.10
111000
Pz
1.0
0.0 0.6 044
0.000
1.0 0.8 0.6 0.4
0.2
X
x
(a)
(b)
(c)
(d)
Figure Numerical example. The sample size ratio is 10:1:1 in Example A and 100:1:1 in
Example B. (a) (xx of Example A. (b) g(r) of Example B. (c) Representations of Example A. (d) Representations Example B. Stars are empirical centers. Circles are available samples, triangles are missing samples. Black boxes are overlapping regions.
Remark 2. R a2) in Eq. 5 can be roughly simplified as a linear function only respect to K:
R((K) = 11 - e -1--1)--++--
)6 + -1+t-22222 +(1+ 28 12.16(K 1) (6)
2(1+6-2)
(K) provides an approximate estimate to distinguish Case and Case in Theorem
Numerical Examples. To quantify the extent that long-tailed data skew the feature space, we consider examples with K classes. Assume that tail classes in both samples have N2 N33- 50 samples. In Example A, set = 10, and the head class having N 500 samples. In Example B. set pg 100 and N 5000. We have PA (3) < PB. All samples are mapped unit circle ( Then 8² = 149.19", 0¹ 61.63% BB = 180 and 0 0 can be found when is minimized. Fig. [ visualizes values of the lower bound function f and the empirical representations of both examples.
4 METHOD
4.1 CHALLENGES IN LONG-TAILED REPRESENTATION LEARNING
In this subsection, we discuss the challenges faced in long tailed representation learning.
Skewed Center Configuration. shrink or even collapse. We refer to this phenomenon as the skewed center configuration. This leads to the feature distributions of the tail classes partially (Fig. 2c fully (Fig. 2d] overlapping. As result, samples in the overlapping regions become inseparable and cannot be distinguished by a classifier.
errooremreveete that long-tailed data force tail classes' centers to
Distribution Gap. One may consider rearranging the center configuration back to the symmetrical one to sep- arate the overlapping features. This approach implicitly assumes that the distribution of training data, Purin is the same with the true distribution of the underlying data, Pinee However, due to the limited sample sizes of tail classes, discrepancy often exists between Pumin and Petre- We refer this phenomenon distribution gap. When it occurs, rearranging the empirical center configuration can sepa- rate the training data but cannot ensure the separation of testing data. And doing this may even causes overlapping distributions between head and tail classes (as depicted in Fig. 3)
Figure 3: long-tailed data representa- tions. Left: Before center correction. Right: After center correction with dis- tribution shifts. To save space, we defer the legend Fig. 4
4.2 FEATRECON
To address the problems discussed above, we designed our method to rebalance the skewed feature distributions of long-tailed data to be symmetric and linearly separable.

---

Published as a conference paper at ICLR 2025
Head
Tall Classes
Available Peatures
Missing Fratures
Empiriaal Centers
Synthanie Fextures
Tree
Confrisnc
(d)
Figure 4: One iteration of our algorithm. (a) Estimation of the confidence supports with training data. The supports are drawn in dashed magenta circles. (b) Regularization of the support by head class statistics. (c) Generating synthetic features to fill the supports (cross markers). (d) Optimization of the SC loss separates the tail classes and their supports.
Theorem 2 suggests that balancing the sample sizes across all classes can correct the center configu- ration. To achieve this, we directly generate synthetic features in the feature space. Since all features are normalized (i.e, Z E S it is reasonable to assume that the features of each class fall within hyperspherical cap on Sh-1, parameterized by center and "radius" fixed central angle. Thus, for each class, we estimate its confidence support as a hyperspherical cap that contains the majority of features. We then sample synthetic features from these supports. Each support filled with real and synthetic features, and features from nearby classes are pushed away as training progresses.
However, since tail classes have limited sample sizes, their estimated supports are unreliable. To prevent missing features of tail classes from falling outside their respective supports and overlapping with the features of the head classes, we regularize tail classes' estimation with the statistics of neighboring head classes. Since synthetic samples encourage the learned representations to be symmetrical, as long as the central angle of any confidence support is sufficiently small (at most cos (- - )), these confidence supports guaranteed to be linearly separable when training reaches the optimal.
The procedure illustrated in Fig. Fig. 4a we estimate the confidence support based on limited training samples. In Fig. 4b the confidence supports of tail classes are regularized using the head class statistics. Fig. 4c. we generate synthetie samples that fill these confidence supports. Finally, by minimizing the SC loss, class centers moved equal distance from each other, and the supports are guaranteed to be linearly separable (Fig. 4d). In practice, we repeat the procedure iteratively.
Confidence Supports Estimation. A hyperspherical cap can be characterized by its center and central angle. For class k, we estimate these parameters as follows:
Pk
A
24. and 0g = Qa(cos (2.
)b = k},
N
|
where Qa denotes the quantile.
Head Class Regularization. For tail classes, the statistics regularized using the statistics of Pure from head classes, which are estimated more accurately due to sufficient training samples. This improves the robustness of tail class parameter estimation. Specifically, for tail class k, we select the top g head classes (C,.) with the highest similarities to its class center ji:
C2 = fa fa E top, (Sx)), where S₁ {p: fu lie Cr),
(8)
and regularize its statistics using those from the selected head classes C% follows:
2/ = (1-11-)) Σ wEf + ypu and 0% = (1 Σ
wgd + 70
(9)
NeBrlg where wf Ejece Ny
is the regularization weight, and y is the regularization magnitude.
Feature Generation. The estimated confidence support of class à defined as:
2 = 1 Sh-1
1 T
AAI > cos(0h)
14
where 0. = min(0%, cos -1 K 1- (10)
7

---

Published as a conference paper at ICLR 2025
Table I: Top-1 accuracy of ResNet-32 CIFAR-10/100-LT datasets with different imbalance factors.
Dataset
CIFAR-IO-E
CIFAR-IOOA
Imhalance Ratio (p)
100
50
100
50
CR
70.4
74.8
86.4
38.3
43.9
55.7
Focal Loss Lin al.
70.4
76.7
86.7
38.4
44.3
CB-Focal CCi al.
2019
74,6
79.3
87.1
39.6
45.2
58.0
LDAM-DRW (Cao
2019
77.0
81.0
88.2
42.0
46.6
58.7
CB-DA-LDAM Jamall et al.
2020
80.0
82.2
87.4
44.1
49.2
58.0
CE-OTmix Gao al.
2024
78.3
83.4
90.2
46.4
40.7
61.6
DWR-OTmix (Cao et
2019:
Gao al. 2024]
83.1
90.6
48.0
52.60
62.7
SCL (Khosla 2020
42.1
45.2
54.8
Hybrid-SC Wang
2021
81.4
91.1
46.7
51.9
63.5
Hybrid-PSC Wang et
202
78.8
83.9
91.0
45.0
48.9
62.7
KCL
Kang al. 2021
77.6
81.7
88.0
42.8
46.3
57.6
TSC
el al. 2022
79.7
82.9
88.7
43.8
47.4
59.0
BCL
Zhu ct al.
2022
84.3
87.2
91.1
51.8
56,6
64.9
SBCL
Hou
44.9
48.7
$7.9
FearRecon
85.2
87.8
91.6
52.5
57.0
65.3
And we sample Ngen features for each class from the respective confidence support.
Temperature Adjustment. Previous works Wanng & Liu 2021) have revealed how temperature parameter affects the representation learning. larger T places more emphasis on inter-class separation, allowing head classes, which have more samples, learn more accurate class boundaries where tail classes should remain distant. Inspired by this, we adjust the temperature for class k as:
(T NNx Nmin
))
Tk
0.5
1 [+ cos
- -T)
(11)
'NNaxx - Nmin
where T+, T- denote the upper and lower bounds of r, respectively. Also, Khosla et al. (2020) shows that 7 also controls the gradient scale: the larger т, the smaller the gradient. Therefore, rebalance gradient scale of the samples by adjusting the weight of S with a scalar parameter -.
Training Framework. Our training framework follows (Zhu et al. 2022). The model consists of: 1) base encoder f X h that extracts latent embeddings: 2) a prediction head L h p that produces model predictions = l f(X); 3) projection head h + z that generates normalized representations g ((x).
The prediction head is optimized using the training data with cross entropy loss and logit compensa- tion (Menon al. 2021). Let Pyy) be class priors and 6, log Pg. Then the Cg is:
exp (py + 6,))
La f(x)) log Even exp (py' + 6gy)
(12)
The projection head is optimized using both real and synthetic features with supervised contrastive loss Lsc. The final objective is:
L= AgCo AcLsc
(13)
where X and 7₂ hyperparameters that control relative strength among different losses.
5 EXPERIMENTS
5.1 DATASET AND IMPLEMENTATION DETAILS.
Dataset. CIFAR-10-LT and CIFAR-100-LY are the imbalanced subsets of the original CIFAR-10 10
and CIFAR-100 (Krizhevsky et al. 2009), following (Kang et al. 2021 Li et al. 2022 Zhu et al.
2022
We set the imbalance factor p Nmax/ min to 100, 50, 10.
ImaaeNet-LLT (Liu et al. 2019) is the subset of the original ImageNet (Deng et al. 2009). with the training set sampled with Pareto distribution with power value a = 0.6 and testing set unchanged.
The imbalance factor is 256, with the most frequent class having 1280 samples and the least frequent one having samples.
8

---

Published as a conference paper at ICLR 2025
INaturalist 2018 (Van Horn et al., 2018) is a large-scale long-tailed dataset that contains 437.5K images from 8, 142 classes with an extremely imbalanced distribution.
Table 2: Top-1 accuracy ResNet-33 on CIFAR-100-LT with imbalance factor equaling 100.
Methods
Many
Medium
Few
Hybrid-SC (Wang al. 2021
46.7
Following (Zuu et al. 2022), we train our model on the long-tailed training sets and evaluate on the balanced test- ing sets. We divide the testing sets into three subsets: many (with more than 100 instances), medium (with 20 to 100 instances), and few (with less than 20 instances) splits.
400 epeens 200 epochs
DRO-LT Samuel & Chechik
2021
64.7
50.0
23.8
47.3
RIDE(3 experts Wang et al 2020
68.1
49.2
23.9
48.0
BCL Zhu ct al.
2022
67.0
52.9
33.2
51.8
FeatRecon (Ours)
68.2
53.3
34,0
52.5
Balanced Softmax (Ren et al.
2020
50.8
PaCo (Cui et al. 2021
52.0
GPaCo (Cuu
52.3
BCI. Zhu
69.6
53.8
33.3
53.4
FeatRezon (Ours)
70.0
53.9
35.5
54.0
Implementation Details. Our imple- mentation follows (Zhu et al. (022).
adopt the ResNet-32 as the backbone. The projection head is a 2-layer MLP that generates 22-diimessional embeddings. Dimension of the hidden layer is 512. Our model is trained for 200 epochs with a batch size of 256 and with a SGD optimizer. The mo- mentum 0.9 and the weight decay is 5e - The learning rate warms up 0.15 in the first 5 epochs and adopt AutoAug (Cubuk et al., 2019) and Cutout (Deesies & Taylor] 2017) for the classification head, and adopt SimAug (Chen set A 2.0, A 0.6, a 0.99, and 7- 0.1, T4 is scheduled by training epoch between 0 and using method in Kukleva et al. (2023). We also train our model for 400 epochs for finer comparisons on CIFAR 1EFAR-100-LT. In this case, the learning rate warms up in the first 10 epochs and decay at the 360th and 380th epochs.
For both CIFAR-10-L) and CIFAR-TO0-LE
decay by 0.1 the 160th and 180th epochs. For data augmentation,
al. 2020 for the projection head. For hyperparameters.
We adopt ResNet-50 (He al. 2016) as the model backbone for both ImageNet-LTT and iNaturalist 2018.
Table 3:
Top-1 accuracy of ResNet-50 on ImageNet-LT dataset and iNaturalist 2018 dataset.
The projection head is 2-layer MLP that generates (024-dimensional em- beddings. Dimension of the hidden layer is 2048. For data augmentation, switch the strategy for the projec- tion head to RandAug (Cubuk et al. 2020). Our model is trained for 90 epochs for ImageNet-LT and 100 for iNaturalist 2018 epochs with batch size of 256 and with SGD optimizer. The momentum 0.9 and the weight decay is 5e for ImageNet-LT and 1e-4 for iNaturalist 2018. The learn- ing rate is 0.1 for ImageNet-I.T and 0.2 for iNaturalist 2018 with a cosine scheduler. Additionaly, we train our model for 90 epochs using ResNeXt- 50-32x4d (Xie et al. (017) the backbone. For hyperparameters, we X 1, >r 0.35, a 0.99, and and using method in Kukleva et al. (023)
Methods
ImageNet-LT
iNaturalist 2018
CE
41.6
61.7
Focal Loss Li al.
2017
43.7
61.3
LDAM-DRW Cao
2019
cRT (Kang al. 2020
47.3
65.2
7-norm (Kang
2020
46.7
65.6
LWS Kang et al.
2020
47.7
65.9
Area (
Chen et
2023
49.5
68.4
CE-OT (Gan et al.
2024
52.0
69.5
DRW-OTmix (Gao et
2024
53.4
71.1
IWB (Dang et al. 2024
55.2
71.5
SCL
Khosla et al.
2020
49.8
66.4
KCL
Kang
2021
51.5
68.6
TSC
Li et al. 2022
52.4
69.7
BCL.
7hu al.
2012
56.0
71.8
SBCL (Hou et al.
2023
70.8
FeatRecon
56.8
72.9
0.07, T+ is scheduled by training epoch between 0.07
5.2 RESULTS
CIFAR-LT Tab. shows experiment results CIFAR-10/100-13 datasets with imbalance factor
varying among 10, 50, and 100. For baselines, we select methods that only work with classifiers (Lin et al. 2017 Cui et al. 2019 Cao et al., 2019 Jamal et al. 2020 Gao al. 2024) and methods that
work with both representations and classifiers (Khosla l et al. 2020; Wang et al. 2021; Kang et al.
2021 Li et al. 2022 Zhu al. 2022 Hou al. 2023). We
that FeatRecon outperforms

---

Published as a conference paper at ICLR 2025
baseline models in all settings. Moreover, model achieves larger performance gain the imbalance factor increases, proving the effectiveness of our method for long- long-tailed data. Additionally, in Tab. 2 provide shot-wise results CIFAR-100-LT data with imbalance factor of 100. The model is trained for both 200 epochs and 400 epochs for fair comparisons with baselines that are trained under different settings. The results demonstrate the superiority of our approach, especially for the few-shot classes.
ImageNet-LT Tab. 3 shows the results of the experiments on the ImageNet-LT dataset using ResNet-50 as the model back- bone. Tab. 4 shows the results of the ex- periments using ResNeXt-50 as the model backbone. We report the overall Top-1 ac- curacy as well as the T accuracy of Many-shot, Medium-shot, and Few-shot classes. Similarly to the experiments on CIFAR-LT, select methods that only work with classifiers (Lin et al. [0177] / Cao
Table 4: Top-1 accuracy of ResNeXt-50 on ImageNet- LT dataset.
Method
Many
Med
Few
All
Focal Loss (Lin ct al.
2017
64.3
37.1
8.2
43.7
7-norm (Kang
2020
59.1
46.9
30.7
49.4
LWS
Kang et al. 2020
60.2
47.2
30.3
49.9
IWB Dang ct al 2024
64.2
52.2
40.2
55.2
BCL (Zhu al.
2022
67.2
53.9
36.5
56.7
Ours
67.9
54.7
37.8
$7.5
al. 2019 Kang al.
2020 Chen al.
2023
Gao et al. 2024
Dang et al. 2024) and methods that work with both representations and
classifiers (Khosla
2020 Kang et 2021 [i al. [022]] Zhu et al. 2022] Hou al. (023)
for baselines. The results show that our method outperforms baselines in terms of overall accuracy, demonstrating the effectiveness of approach for learning classes with synthetic features.
NNtturalist 2018 Tab. 3 also lists the experiment results on iNaturalist 2018 dataset. Similarly to
results on ImageNet-LT, our method outperforms baselines on the accuracy of tail classes and overall accuracy, highlighting our model's ''s capability of learning from a few samples.
5.3 ABLATION STUDY
We evaluate the design of FeatRecon through ablation study on CIFAR-100-LA dataset, with an imbalance factor of 100. Each model runs for 200 epochs. The results are displayed
Table 5: Ablating model components.
Exp
LC
SC
Up Sam
Feat Gen
Temp Adj
Accuracy
A
50.2
2
/
/
51.1
+0,9
in Tab. 5 Exp. provides
51.2
+1.0
the baseline by training clas- sifier with logit compensation (LP) (Menon et al. 2021). Exp.
52.2
+2.0
5 V
/
52.5
42.3
2 introduces an additional projection head and trains feature representations with the SC loss (Khosla jtt al., 2020). This design brings a 0.9% performance improvement, underscoring the benefit of representation learning. In Exp. 3. we balance the sample size across different classes by naively upsampling (Up Sam) the existing features for representation learning. However, this approach has no significantly positive effect. It highlights the effectiveness of our synthetic feature generation method (Feat Gen), shown in Exp. 4, which brings a 2.0% performance gain. In Exp. 5, we validate the benefit of training with temperature adjustment (Temp Adj), which leads to a total 2.3% performance increase.
6 CONCLUSION
In this paper, we establish a theoretical framework to investigate the optimal representation configu- ration for long-tailed data. We prove that centers tail classes either shrink or collapse. Building on this analysis, we explore the challenges associated with tailed representations, identifying two key issues: skewed center configurations and distribution shifts. Inspired by our findings, we introduce a novel method for long-tailed representation learning. Our method reconstructs the feature space for long-tailed data, arranging the representations of each class into symmetric and linearly separable regions. We demonstrate the effectiveness of our approach on several benchmark datasets and the results show that our method achieves state-of-the-an performance.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This research was partially supported by the National Science Foundation (NSF) grant CCF-2144901, CPS-2128350, the National Institute of Health (NIH) Grants ROINS143143, R01CA227843, and the Stony Brook Trustees Faculty Award. Lingjie Yi is also supported in part by Bloomberg Data Science Fellowship.
REFERENCES
Sumyeong Ahn, Jongwoo Ko, and Se- So-Young Yun. Cuda: Curriculum of data augmentation for long-tailed recognition. ICLR, 2023.
Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In ICML, 2019.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with abbel-disrrbuttonn-awar margin loss. In NeurIPS, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.
Unsupervised learning of visual features by contrasting cluster assignments. In NeurlPS, 2020.
Jiahao Chen and Bing Su. Transfer knowledge from head to tail: Uncertainty calibration under long-tailed distribution. CVPR, 2023.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICMI,, 2020.
Xiaohua Chen, Yucan Zhou, Dayan Wu, Chule Yang, Bo Li, Qinghua Hu, and Weiping Wang. Area: adaptive reweighting via effective area for long- tailed classification. In CVPR, 2023.
Xinlei Chen and Kaiming He. Exploring simple Siamese representation learning. In CVPR, 2021.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment:
Learning augmentation strategies from data. In CVPR, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR workshops, 2020.
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In ICCV, 2021.
Jiequan Cui, Zhisheng Zhong, Zhuotao Tian, Shu Liu, Bei Yu, and Jiaya Jia. Generalized parametric contrastive learning. TPAMI, 2023.
Yin Cui, Menglin Jia, Tsung-Y Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on efffective number of samples. In CVPR, 2019.
Wenqi Dang, Zhou Yang, Weisheng Dong. Xin Li, and Guangming Shi. Inverse deep toiled learning. In AAAI, 2024.
for
Philippe Delsarte, Jean-Marie Goethals, and Johan Jacob Seidel. Spherical codes and designs. In Geometry and Combinatorics. Elsevier, 1977.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Terrance Drieies and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv, 2017.
Jintong Gao, He Zhao, Zhuo Li, and Dandan Guo. Enhancing minority classes by mixing: An adaptative optimal transport approach for long-tailed classification. NeurIPS, 2024.
Florian Graf, Christoph Hofer, Mare Niethammer, and Roland Kwitt. Dissecting supervised con-
trastive learning. In ICMI. 2021.
11

---

Published as a conference paper at ICLR 2025
Jean- Cean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent a new approach to self-supervised learning. In NeurIPS, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. In CVPR, 2020.
Chengkai Hou, Jieyu Zhang, Haonan Wang, and Tianyi Zhou. Subelass-balancing contrastive learning for long-tailed recognition. In ICCV, 2023.
Muhammad Abdullah Jamal, Matthew Brown, Ming-Hsuan Yang. Liqiang Wang, and Boqing Gong. Rethinking class-balanced methods for long-tailed visual recognition from domain adaptation perspective. In CVPR, 2020.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In ICLR, 2020.
Bingyi Kang, Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. In ICLR, 2021.
Salman Khan, Munawar Hayat, Syed Waqas Zamir, Jianbing Shen, and Ling Shao. Striking the right balance with uncertainty. CVPR, 2019.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sama,, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurlPS, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Anna Kukleva, Moritz Böhle, Bernt Schiele, Hilde Kuehne, and Christian Rupprecht. Temperature schedules for self-supervised contrastive methods on long-tail data. In ICLR, 2023.
Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogerio Feris, Piotr Indyk, and Dina
Katabi. Targeted supervised contrastive learning for long-tailed recognition. In CVPR, 2022.
Tsung-Yo Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella Yu. Large-scale tailed recognition in open world. In CVPR, 2019.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In ICLR, 2021.
Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long-tailed visual recognition. NewrIPS, 2020.
Dvir Samuel and Gal Chechik. Distributional
robustness robuuttts loss for tail learning. In CVPR, 2021.
Jianlin Su. Vae the hypersphere.
2021.
Laurens Van Der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. JMLR, 2008
Grant Van Hor, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In CVPR, 2021.
Peng Wang, Kai Han, Xiu-Shen Wei, Lei Zhang, and Lei Wang. Contrastive learning based hybrid networks for long-tailed image classification. In CVPR, 2021.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere. In ICMI., 2020.
12

---

Published as a conference paper at ICLR 2025
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu. Long-taile tailed recognition by routing diverse distribution-aww aware experts. arXiv, 2020.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks, In CVPR, 2017.
Shiyu Xuan and Shiliang Zhang. Decoupled contrastive learning for long-tailed recognition. In AAAI, 2024.
Manyi Zhang, Xuyang Zhao, Jun Yao, Chun Yuan, and Weiran Huang. When noisy labels meet long tail dilemmas: A representation calibration method. In CVPR, 2023.
Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, and Yu-Gang Jiang. Balanced contrastive learning for long-tailed visual recognition. In CVPR, 2022.
13

---

Published as a conference paper at ICLR 2025
A
APPENDIX A
A.1 PSEUDO ALGORITHMS
A.1.1 TRAINING PROCESS
In this section, we first present the training procedure of FeatRecon. FeatRecon is a heuristic method that iteratively generates synthetic features, adjust representations, and re-estimate confidence supports at each step of the training process.
Algorithm 1: FeatRecon Algorithm
Input: Available training samples D = {ar, Yo heB, from classes, the quantile parameter a, the regularization magnitude y and which controls the total number of synthetic features.
for I...., do
Sample mini-batch (a:, (x v1) from D for k= I,... K do
Estimate batch-wise support centers and update with epoch-wise estimation ; if class & is tail class then
Regularize all statistics as Eq. (8 and Eq. 19;;
end
Generate Ngen synthetic features for class k;;
end
Compute the cross entropy loss Lg (Eq. (2) with training data Compute the supervised contrastive loss Lsc (Eq. (I):
11
Update model with loss L = X XcLsc Estimate confident supports as Eq. 7 for all classes
13
14
end
Return Trained model
A.1.2 SYNTHETIC FEATURE GENERATION
Next, present the pseudo- alggorths of synthetic feature generation. This part follows the idea of Su (202I). Suppose that the angles between the samples class and their respective center y follow given distribution Pı (0; (1). To generate synthetic features within the confidence support of the class ki we first sample angles from (o: (44) but only retain samples that satisfy 0 S 0 < 0₃. So we have:
O = (0 ~ pa (0; 1 < 0x
(14)
Then we can generate samples whose angle to the center is via:
x cos(O) i4 + (f) x
(15)
where U and fi 0
Here, y are sampled from the uniform distribution on the surface of a h 2 hypersphere Sh- that is perpendicular to jig. And we can sample / via
NO IN)
(s. P
x 1
(16)
(s,
And verify that the samples generated via the above method fall within the confidence support and their angles to the class centers follow truncated version of px (0; (c).
The entire procedure can be summarized by Algorithm.
14

---

Published as a conference paper at ICLR 2025
Algorithm 2: Synthetic Feature Generation Algorithm
Input: Parameters of the estimated confidence support { kJ. angle distributions p& (0; p4))
numbers of generated samples Gk. where [K]
for == I..., K do
Sample Na iid. {0s} as described Eq.
Sample N₂ iid. {FR} } as described in Eg. T6
Calculate { } as described Eq. 15
end
Return (EE))
terms of generating synthetic features, we sample 0 from the normal distribution. This evolves to estimate two more parameters, the center and the variance of 0. We estimate both parameters once every epoch and regularize them at each mini-batch.
A.2 MORE EXPLANATION OF THEOREM
In this subsection, provide more detailed mathematical explanation with respect to Theorem
states the necessary and sufficient conditions on the representation configuration for the SC loss to attain its minimal. (A3) states thyat representations of each class converge the respective class centers. (A4) states that the centers of class 2 to K form K 2 equidistant simplex, the angles between whose vertices are equal 0z. (A5) states that the vector between the spherical center and the center of class is perpendicular to the equidistant simplex, and the angle between class l's center and other classes' center all equal 01. And cos(02) (K c))
K-2
The additional numerical relationship describes that the entire dynamic of the configuration of all centers as increases from 0 to +00. More specifically:
(Case 1) 0 < p < 1: 2 < 0 cos ( (-R-T)
< O2 < cos cos ((-=-2)
R-2)
(Case 2) p = 1: becomes a data balance case where 01 = 02 = cos -1--1- (- R_1) This indicates regular simplex.
that all class centers form
(Case 3) 1 < p < R(K, d2): as continues increase, it becomes long- problem. The head class (1*t) increasingly dominates the feature space as æ > 01 > cos
0g > 0. At this stage, centers of tail classes increasingly shrink together.
(Case 4) p R(K (,KK 02): centers of tail classes collapses with 0₂ and 0 æ
In both long-tailed cases, 0, measures the extent that a head class dominate the feature space. Also, Theorem is special case of Theorem (Case 2).
A.3 LIMITATIONS
While our theoretical framework opens a door to study long-tailed representation, it's currently limited to the simple one vs. all case. The solution for more general cases remains unsolved.
To solve this problem, it is crucial to identify the numerical relationships among the angles between different class centers. As suggested Lemma there are classes with distinct sample sizes, there will be (M distinct angles. Determining the numerical relationships among these angles poses significant challenge.
However, while our theory primarily addresses the one-vs-all scenario, it unveils a crucial insight into how the head classes dominate the feature space and provides guidance algorithm design.
15

---

Published as a conference paper at ICLR 2025
A.4 REPRESENTATION VISUALIZATION
In Fig. 5 we visualize the learned representations of CITAR-10 testing images with imbalance factor equaling 100 using I-SNE (Van Der Maaten & Hinton 2008 The results show that with the help of synthetic features generated by FeatRecon, the resulting testing distributions of different classes are more separated.
airplane (5000)
bird (1796)
deer (645)
frog (232)
ship (83) truck (50)
automobile (2997)
cat (1077)
dog (387)
horse (139)
Figure 1-SNE visualization of CIFAR-10 testing set. (left) Learned representations without synthetic features. (Right) Learned representations with synthetic features generated by FeatRecon. Numbers in the legend after class names represents the numbers of training samples from this class.
16

---

Published as a conference paper at ICLR 2025
B APPENDIX B
B.1 PROOF OF THEOREM
In this section, we provide proofs of Theorem I proposed in Sec. 3.2 Our proof is different from what is shown in (Graf et al. 2021 Zhu et al. 2022) in order to take into account the long-tailed distribution. For convenience in reading, let us recall some related notions and definitions.
h, N. K e N
2 = R
Sh-1 { E R² || =
. y {L.
K} [K]
B = {l...., ] = [N]
B₁ = (E:iE B, = k}
N₂ |Bll
Definition 1 (Supervised contrastive loss) Let Z be N-point configuration (assuming that all 28 are normalized), Z = (Z)..... .N) E (Sh~1)0 with labels Y = (V1...., YN) E ([K])> and K S h 1. Let B [N] By ii:ie B,y yi k} and N₂ |B The supervised contrastive loss Lscl:; Y): (5a-11 y + IR is defined as:
Lsc Σ Σ
ked sc where Lisc NR
1NN=11
Σ log
exp ((zi, ElEBY(S) exp ((z,, 21)1)))))))
K=
jEB.
Definition (Fquidistan//regguaar Simplex) Let h K E with < h 1. A K-point configuration
< .G......kk) E (S^-1)N forms the vertices of an equidistant simplex inscribed in the unit hypersphere if and only if all of the following conditions hold:
(1) E []] | =1
3E R, j and si<jsk, (G)
And form the vertices of regular simplex inscribed in the unit hypersphere if and only if (1), (2) and the following condition holds:
Eie[K Si =0 0
Theorem Let Z be a N-point configuration (assuming that all z8 are normalized), (21, (,, 2N) (Sh-1)N, with labels Y (M1,..., yN) ([K])*, and K h Let B [N].
B₁ (i:iE B, 3/ kt) and N₂ BBl.- When Y is balanced, hence Vi [K]. No. R. it holds
that:
Lsc 2 N log ((x
N ( + N(K 1)
1) -1)
exp (- K
(Kk-1)
K
K (
where equality is attained if and only if there exists configuration of 2 = (Z1,..., ,x) (Sh-1) such that:
(A1) i e Bg, 2; Eк.
(A2) Z form regular simplex inscribed in the unit-hyperspher.
17

---

Published as a conference paper at ICLR 2025
B.1.1 STEPS OF PROOF
First let's rewrite Sc and Lsc (assuming Ak E [K], No > 1).
SC
1
E
log
exp ((2s)
NA
EiEm(() exp ((2,) 21)
JEBA
EiEm( exp ((zas at)
E log
N&
exp ((zi; zy))
JEBA
Nk-1
(Eiem)() exp ((2g, 21))
N& log
[[JEBA(1) exp ((z1, 23))
(17)
ΣlEBYO exp ((2)
log
(SEBANE) (21 Na-1
exp
EiEm( exp ((2g, 21))
log
N4-1 1 ΣjeBa (2 23)
exp
and hence
Lsc Σ
ci. sc
Jmm 1 B
(18)
K
Lemmo 2
N₂ log
(Nx- () +e
Σ
No
exp exp ((3k, Eki))
WE(K
where 24 = No Eien, 24. When balanced, Vi E [K], N₂ = K
then
N
N (N
N
Lsc
log
-1)
Σ
exp ((3R, zx))
K
K
K'E[K]
(19)
k'
Lemma
N Nlg
N ((
K -1)
N(K - 1)
exp (3)
K
and equality is attained if and only if all of the following conditions hold:
(BI) E Bg. 21 Zu.
(B2) Vk [K] and k [K]][)) (2x, 3xr) B.
(B3) There exists a configuration of = (2,,..., xx) such that (B2) holds.
(Case 1) K h +
K-1 or ==
(Case 2) K < h+1:
1 k- < 51 <
When a, b > 0, f()) log (a + be") is strictly increasing function. And Eq. 19 suggests that the
lower bound Lsc is a strictly increasing function of 3. When reaches its miniaal value, so does LSc- When K 1, Bmin R-T, have:
18

---

Published as a conference paper at ICLR 2025
((x) 1)
N
N(K-K 11)
Lsc 2 N log
te-
exp
K
K K-1
(20)
NN - N(K -1)
K
N log
exp
K
K
When 3
K-I- Lemma shows that (B2) and (B3) imply 7 (Z1,..., 2x form regular
simplex. Thus, the conditions for equality can be summarized as: there exists configuration of
Z = (,,....,xx' E (h*-1) such that:
(A1) E Bg == Ex.
(A2) 7 form regular simplex inscribed the unit hypersphere.
B.1.2 LEMMAS PART
In this section, we provide definitions and proofs of lemmas that used for the proof of Theorem Lemma 1. Let Z be an K point configuration (assuming that all zs are normalized), Z (aa....,2]] (Sh-1yK If33 R, Vi,j E [K] and i such that all inner products (21,2))
are equal, then one of the the following cases holds:
(Case 1) K > h + 3 = 1.
N-1 or =1 (Case 2) K h + 1: B = -
(Case 3) K + - N S 11
1 And when 3 = K-T Z = (3l, ..... 2k) forms a regular simplex.
Proof. As explained in DDelartee et al. 1977). there are at most h + equidistant points Sh-1 (the size a spherical I-distance set S h 1). When > h+11 1, all N points collapse into a single point and B = 1, which is Case 1. When N
+ 1, these points either form regular simplex or collapse into a single point, which Case 2. When N < h + 1, these points form a regular/non-regular equidistant simplex collapse into single point, which is Case 3.
(S forms regular simplex, 3 K-1 S B S (case 3) and when Z = (21,..., .2k) E
Next, we will show why when < h + 1,
1
(Case 2). Given that
Σ
Σ KEEK
Σ RE(K]
2k
2k
KE(K]
Σ kE[K]
(Ehs ZEL +
Σ
(Znz 3m)
(21)
mE(K]\()
K + K(K - 1).3
2 0,
this shows -R-- S B. Since 3 is the dot product of two unit vectors, 3 S 1. Then we have:
1
N < SB
(22)
When Z = (k) E ( forms a regular simplex, we have ΣkE(K] 0. Then
K K(K 1)3 and K-
Now we prove when 3 = k-1 7 = (Z1,...,,2, 2x) forms a regular simplex. Recall that Vi,j E [K] and # have ||2||| = 1, and (2 27) B. When B -ー-- Eq. (2 shows ΣkE[K] 2 0. Then Z forms regular simplex.
19

---

Published as a conference paper at ICLR 2025
Lemma 2. Let Z be N-point configuration (assuming that all normalized), Z (i,...,;NN) E 15h-1)N, with labels Y (yi..... .NN E ([K])* Let i E B, = k). Ak E [K], EieB. S is bounded below by:
N Bk = {i
Σ sc 2 NA log iEBx
1N
Σ
No exp ((Ek, (x)))
(23)
K'E[K]
K/k
1 where &y No EieB, 24, and equality is attained if and only if the following condition holds:
(BI) Vi E B2, Ek.
Proof. According to Eq. 177
ElEBY exp ((z), m1)
SC
log
Nu-1 EjEBrM (2 (21,22)
cxp
EleBAM exp ((z,, 21)) Ex'(k EmEB, exp (122)
(24)
1 Wt
log
EjeBorn (z),
exp
No-I
There are three terms in Eq. (24). Let's check their lower bounds one by one. Applying Jensen's S inequity, the first term can be bounded below:
Σ exp ((2,, 2 (Nx 1) cxp
(2),2))
(25)
(N₂
LEB
IEB
where equality is attained if and only if all of the following conditions hold:
(CI) VA [K] and vi E Bg. Ba(k, i) such that Vjj E BBVii), all inner products (z4 2z) a(k, are equal.
Let 2い
1 No EieB, 24. Similarly, the second term can be bounded below:
Σ exp ((zi, zm))) 2 Nw exp
No
(Zis km)) mEB
Σ
= NN exp
Zio
N
Σ
2m
me BE
me Be
NN exp ((zg) Zw()
(26)
where equality is attained if and only if all of the following conditions hold:
(C) Vk E [K] and vi Be. Bo(k,, i, k') such that k' E [K]] (zi, 2m.) = a' (ks i, k') are equal. And a' (k, i, k') (2,, )x.).
(k) and m E Be all inner products
Using Cauchy-Schwarz inequality, the third
be bounded below:
1 exp NN EjeBoro (2,, (z),2))
No-1 EjeBoro || [13)
1
(27)
cxp
where equality is attained if and only if the following condition holds:
20

---

Published as a conference paper at ICLR 2025
(3) Ak [K] and Vi.j Bk, 2j Ek.
It is obvious to see that when condition (C3) holds, all samples from the same class collapse into their class center (denoted by &..) In this case, and thus condition (CI) and (C2) hold well, where akk, i) and a' (k, é, k') (2k: k So, (C3) is sufficient condition for (C1) and (C2). Now we have:
EkEBAY(i) exp ((2s, a)) + EKE(K] LiEB exp ((2)) 21))
Σ B
khss SC
log
K
v Bg
exp
N4-1 EjeBA (F 23))
(Na - 1)
1 (N EieRANo (Ets 2t) + Ex'eR EiERy exp ((za, mt) Kt
Eg
log
exp Nw-1 EyEBA (21
iE
EXeCK LIEB exp ((2), ai) k'
log
(Nx -1++
EjEBUMO (Ei, (i)
V Bg
exp
Nx-1
ΣN'(K] N/K
Nk exp ((2), Exr)
log
(Nk 1) +
EjEBA(() (26
KEB
exp
N
Eg 2
log
(N - + + Σ
Nk exp ((Zu: Zwil)
B
KElK] K
No log
(N₂ (Na-1)+e
Σ No exp ((ze, Zw))
KE(K]
K*k
(28)
where equality is attained if and only if the following condition holds:
(BI) E Bg, 2, = Zk.
Here, (B1) and (C3) express the same condition.
21

---

Published as a conference paper at ICLR 2025
Lemma 3. Let Z be an K point configuration (assuming all 2s normalized), Z (21, ;xk) (Sh-1yk and $ + 1, holds that:
K Σ
a log (a 1)-1)+0 +b
Σ exp Ex)) +e
2 Ka Kalog((a + b b(K 1) exp (3) + +c),
KE(K] K
(29)
where 1, b, O, and equality is attained if and only if all of the following conditions hold:
(B2) Vk [K] and k [K]][R), (Ek, ²x² B.
(B3) There exists configuration of 2
kx) such that (B2) holds.
(Case 1) K = h 1: 3 --T or 3 =
(Case 2) K < h N-1 S 1
Proof. Since f(x) exp(x) is a convex function, applying Jensen's 's inequality, we have
Σ
exp ((āk) mx() 2 (K 1) exp
(Ek,
K 1
KEEK] k $
1 E|K k #
(30)
exp (K - exp
Σ
3x
K
K KER k #
(K 1) exp (3x),
where equality is attained if and only if all of the following conditions hold:
(C4) vk E [K] and k E [K]](}), (Ex, 3x) = Bx-
(C5) There exists configuration of 7 = (Ei...., ãx) such that (C4) holds.
When > 1, c > 0, fx) = log ((a - 1) + b (exp (x) + c)) is also convex function. By Jensen's inequality, we have
K
K
Σ a log ((a - 1) + ((xpp (8k) + c)) 2 Kalog
(a - 1)+ exp
Σ
+
C
K
k=
1 K
(31)
Kalog (a- 1) + b
exp
3
+
K
k=1
Kalog ((a - 1) + b (exp (8) c)),
where equality is attained if and only if all of the following conditions hold:
(C6) vk E [K] and &' E [K]] 8g = 33 :3.
(C7) There exists configuration of 7 = (Z,,..., Exx) such that (C6) holds.
22

---

Published as a conference paper at ICLR 2025
Note that when (C6) and (C7) hold, (C4) and (C5) hold too, And according to Lemma when K & 1, Case and Case in Lemma satisfy (C7). And hence
log Σ
(a 1) +b
Σ
exp (EE,, +
Kaloo(aaaa + b((K 1) exp (3) + +c)),
KE(K] k/
(32)
where equality is attained if and only if all of the following conditions hold:
(B2) vk E [K] and k' E [K]] (Ex, ūx) = B.
(B3) There exists a configuration of = (3),..., _x) such that (B2) holds.
(Case 1) K = h+ - 1
N-1 or 3 = 1
(Case 2) K < h h+1: N-T S <
23

---

Published as a conference paper at ICLR 2025
B.2 PROOF OF THEOREM 2
In this section, we provide proofs for Theorem proposed in Sec. 3.2 For convenience in reading, let us recall some related notions and definitions.
h. N,N EN
Z Rh
. Sh-1 {2 E Rh. ||=|| = }}
. y {1,.... KJ = [K]
= {1...., N} = [N]
B3 (i:iE B, kk
. N [&l
Definition 1 (Supervised contrastive loss) Let Z be an N-point configuration (assuming that all 28 are normalized), Z = (Zi...., ,N) E (Sh-1)A with labels Y (y1...., UN) E ([K])* and
K < h 1. Let B [N]. B₃ fi: fi:ie B, y4 k} and N₁- = | Supervised contrastive loss Lsc(: ( (S + R is defined as:
Lsc = Σ
K
Σ
cke Scs where Lise
1(N,21) N₂ -
log exp ((2), (2),2z) ElEmGo exp (12,, 21))
k=1 B
JEBA((
Theorem 2 Let be an N-point configuration (assuming that all 28 are normalized), (Z1,...,,NN) (S4-1),, with labels Y (yi,..., .N) ([K])*, and 3 & K h 1. If
Vk E (2..... K}, N₂ 02 2 4, and 3p > such that N1 = =0 = pag > 1, holds that:
Lsc 2 f(cos(0)),cos
where ;(:):xx + R defined as:
R
f(11,02) = pag log ((pa - 1) + - (K - 1) 02 exp (f1))
+ (K- - 1)a2 log ((a2 - 1) + - ((K 2) exp (x2) + pa2 exp (æ1)))
and equality is attained if and only if there exists a configuration of = ( Ex) E (Sh-1) such that:
(A3) E BkG 24 = Ex.
(A4) (Z2,..., [xx form an equidistant simplex whose
vertex angle equals 022
(A5) Vk: E (2...., K), (Z,, =x) cos(O1) and cos (K cos?
The numerical relationship between p and can be summarized as:
(Case 1) p < 1: 01 cos foos-'((x20) ((-=-))0)
(Case 2) == 01 cos - (-k-1)
(Case 3) 1< p < R(K, a2): 01 E [-I,,o0 1 x-11)
(Case 4) 2 R(K, a2): 0 =
Let bj (K- () -e 2e 20²)aa 2, b2 8(1 (K)_21(aaKKK))))1 -e?), then R( K, o2)
be defined as:
-b +
0 + b
R(K,a2)
2(1 e 2)
24

---

Published as a conference paper at ICLR 2025
B.2.1 STEPS OF PROOF
Following Eq. Eq. 18 and Lemma 2 Appendix B.I.I have
Lemma
Csc
Z
No log
(Nvx - 1) Σ
No exp (12,, (x1)
(33)
K'e[K]
k' fk
where equality is attained and only if there exists a configuration of 2 = (Z1,..., ,x) ) (Sh-1) such that:
(A3) i E BRs 24 Zk.
When K < h 1, Vk (2,..., KJ, Nu 02 4, and 3p such that N₁ 01 p02
following Lemma 5 we have:
Lsc
log 1)+=" Σ
No exp ((Zk, 2x))
k=
k k'E(K] M'A
(34)
Lemma
f(cos(01), cos(8)))
where
R is defined as:
f(11, = pag log ((pa2 - 1) + - - (K - -1) a2 exp (xI))
(35)
+ (K 1)2z log ((22 1) + -1 ((K - 2) 02 cxp (x22) + pag exp (21))),
and equality is attained if and only if all of the following conditions hold:
(A4) (Z.,..., ;x) form an equidistant simplex whose vertex-center-vertes angle equals 02.
(Z1, Ex) cos(01) and cos(O2) (K (0))
(A5) Vk E The numerical relationship between / and 01 can be summarized as:
K-2
(Case 1) p < 1: 01 (cos (cos--(-R- (-)
(Case 2) == 1: 0 = cos (-x-T)-
(Case 3) < p < R(K, a2): 01 E -T, -, 008 -1 1 x-1-)
(Case 4) 2 R(K,ag a2): 01 -
Let bj = (K-1)(1+e -222 - 2, b2 = 8(11e 2)( 1 11)g - -)?). then R(K, 02) be defined as:
-bi + / b2
R(K,a2)=
2(1 + e 222)
25

---

Published as a conference paper at ICLR 2025
B.2.2 LEMMAS PART
In this section, we provide definitions and proofs of lemmas that used for the proof of Theorem 2
Lemma 4. Let Z be K point configuration (assuming that all 2s normalized), Z (Z,...., Σx) E (S (igh-)K,, and 3 < K - h + 1. 1f,k, k E (2..... K) and l + k such that
(2a, 2x² B2 and 31 min (21, 2x. (21, 2x²) = c). it holds that:
82 = K 1) 89 1
K K-2 2 where 1 1 31 < 0 and
S <
(36)
K -
Proof. Without loss of generality, we assume that ( ...,,...... 5x) forms an equidistant simplex in the
southern hemisphere of Sh-1 and then & is at the north pole. Let
x Ex= Zkll, we have
= 111) then
K
K
1141
2k
2k:
K
K
K
k=
k=
k==
1
Σ k=2
Zk (Zk, 2x3 +
{Zks 2md)
(K 1)2
(37)
k
1
(KK - 1) (K (K) (K
(K- 1)2
13112
so we have
82 (K - 1) 39
(38)
K-2
According to Lemma
K-2 S 82 and so -1 S.
Lemma 5. Let 7 be K point configuration (assuming all Es normalized), 7 = -21,..... Σk
(Sh-1yK and I K & h + 1. Let B = [N], B₁ = :i:i iic E B,y, = k} and N₂ B& Ler J(:) (Sh-1)k R is defined as:
K
J(2) = Σ N₂ log
- 1)
No exp ((2k, Ex())
(39)
A MEEK]
k/
1/tk (2..... K), N₁ a2 2 4, and 7p such that N, 01 = p02 > 1, holds that:
J(Z) 2 f(3i),
(40)
where f(.): R R defined as:
f(x) pap ((pa2 - + + [K - 11) (x))
-K (c221)++e) (K-2)) 222
1))2 (K--1)=-1)
pa2 exp (x))
K-2
41
and equality is attained if and only if all of the following condition holds:
26

---

Published as a conference paper at ICLR 2025
(B4) vk, k E (2...., K} and k + + (z),Et) 31 and (Ek, Ex) f3 (K (
K
Let 31 = cos the numerical relationship between and 01 can be summarized as: (Case p < 1: 01 E (- (=<==11-0 k)
(Case 2) p= 01 = - K-I
(Case 3) 1 1< p < R(K, a2): 01 (-1, - K-1
(Case 4) 2 R(K, 02): 01 =-1
Here (K 1) e --2e²)ay and by 8(1+ () (1) -²²). R(K, 02)
is given by:
-bi + v 03 b2
R(K,02)
(42)
2(1 c 2)22
Proof. When N1 01, Ak ( ,K,, No = a22 01 = p02, then
[(Z) Σ N& log
(Nx - 1)) +c
Σ
No exp ((Zu, Bxr))
K'EK] K'
K
=a log
(al - +e
02 exp (131, Zx²)
(43)
k==2
+
a2 log
(a2-1) - 1) -
02 exp ((3k, Ex)) + a₁ exp ((Ex, &11)
2m 2
K==2
According to Eq.
30 Lemma
3 the first term can be bounded low:
a1 log
(al 1) Σ
a2 exp ((51, Ex/))
k=
(44)
2 01 log ((a) - 1) 1 (K 1) 02 exp (B1)
fi1))
Similarly, the second term can be bounded low:
ag log (a2-1)+e" -e-
Σ
0g exp ((z,, Ex)) an exp ((zx, Z))
(45)
2 (K- l)az log ((a2 - 1) + ((K - 2) a2 exp (32) + a exp (B1))
f2()1)
Combining Eq.
44 Eq. 45 and Lemma 4
have
27

---

Published as a conference paper at ICLR 2025
[(2) 2 paz log ((paz e- 60-(( (K -1) exp2ex(((1))
(K ) paz exp (5:)) (31))
+ (K 1)02 log (02 (02- - 1)+ (K- 02 exp
K K-2
-11(31) f2((3) f(31),
(46)
where -15 I and equality is attained if and only if the following condition holds:
(C8) Wk (2,.... K} and k + k', (21, kx) 31 and (Ek, B2 (K -1)1))
To find the minimal value of f(r when -1 these value is difficult but can be found with scientific
O, need find the critical value of f'(x:
and the sign of (x)) Direct computation computation software once we know all parameters in specific case. For analytical purpose, investigate the general form. Let 3 S K < h + 1, > 0, pa2 1, a2 2 4 and - S x < 0.
We first study key properties of f(x) (P1) We start by analyzing the derivatives of f(t). The first and the second derivative of fi are:
De
[1 -K 1)a3 (pa2 - 1) + e- (K- 1)age* 0,
(47)
and
(pa2 pe
(x(x) e - 1)a2 ((02 1) + e 1)agez)2
0.
(48)
Here f1(z) and ff' are strictly positive because every term of them positive. The First derivative of f2(2) is:
(K (K-1)x²-1)
2(K exp
+ pe
K-2
e 1)ag
(49)
(a2 - (KK 2)222a2ee exp
[K-1]a2-] K-2
+
paze
The second derivative of f2(x) is difficult to calculate directly. We instead do it in another way. If variable, we have:
take y(x) exp( [K [K1])=--
K-2
dy(x) 2(K - 1). K- exp( (K 11)² < 0.
(50)
dx
K-2
It holds because every term but (negative) in dy(x)) is positive. And fz(x) can be written as:
pe 2(K 1)zy
=((x)=([=,))) e e - 11)a
(az - 1) + e- page.
+ e-1(K p= -2)azy
(51)
+ Ca/
=
c1 + czy
where C₁ (K - 1)a3
0, C2 pe c3 2(K 1)x, C4 (az 1) c- 'azee"
C5 t < - 2)02 and - < x < 0. Then the partial derivative of G to is:
(G(r,y)
C1
(cycs 02cs)
dy
(c1 + csy)²
(cs c ((2(K - 1)x - (K- - 2)) -1 paze" + +(zz "1paee"
Ci
11]2(K 1)x) (52)
< 0.
28

---

Published as a conference paper at ICLR 2025
Here aG is strictly negative because (2(K* 1)x (K 2)) and negative while all other terms are positive, Similarly, f2( can be written as:
2(K 1 1)yz pC
fá(z) = (lo)=)Gzzv)) = e (K 1))a
-a2-10 - 1) + e (K - 2)a2y e paze
(53)
Cg + C7e
C3 + cye#
where 6i = e - - 2(K 1)y, 07 = p. og -o) - 0g = e p02
and 5 x < 0. Here CI, co, c7, C,, > 0. Then the partial derivative of G to x is:
aG(z, v) dr
(cs + cge=)2 ((1 x)ceege" c70se" + cgcs) > 0.
(54)
8G is strictly positive because every term of it is positive. Combining Eq. 550 Eq. 52 Here du
and Eq. (54). we have:
0G(x, v)
aG(z. v)
dy(x) 0.
(x)
dx +
(55)
dy x
Thus, according to Eq. (48) and Eq.
55
the second derivative of f(x) is:
>
(56)
This reveals that ((x) is a convex function.
(P2) Next, we analyze how p affects (x)) we view / as variable instead of a constant, we have
fi(x) H1 p) c 1 1)aZe*
P
a2p + c- - 1)aze
(57)
02p + C2
where = (K - 1)a3e* > 0, = cK 1)aze _ 1. Then the partial derivative of H1 to p is given by:
(x,
C2
e-l(K 1) 1)az
0p
=
(azp + 02)2
(azp + 02)2 = C1
E -l (3 - 1) -1-1
2e U22
(58)
> 6)
= C
(02p + 02)
(a2p 02)
> 0.
When K > 3 and -1 S < 0, e- - 1)age* > 2e 22 So, as long as 02 2 4
7 holds,
C2 > holds. Similarly, we also have:
P+ 2(K - 1)ze "0xe- exp K-2
(K--))=-)
((()) = H2(x,p) =E - 1)a3e
e- a-laze + (az 1) c-I(K 2) exp
(K- (K-1)=--1
K-2
p C3
csp C5
(59)
29

---

Published as a conference paper at ICLR 2025
where 61 = (K - 1)aee* > 0, 03 = 2(K - 1)zee" " exp
(K 1)=1)=2=
= e age" and
k-2
C5 = (a2 - 1) + E ee'(K' - 2) exp
(K-11)=²11)
Then the partial derivative of Hg to p is:
K-2
(H2 p)
C5 0301
(a2 + t
02 (-Kr) exp
(K- (K-1)=²-1
3p
61
(cpp + cz)²
C1
(cap + 0x)2
(60)
holds because every term in atl, Op
is positive. Combining Eq. (57) to Eq. (60). we have
f((x) fi(z) + f2) HI, p) + p) = =H(x
(61)
and
OH (z. p)
(x.p))
0H2(2, p)
> 0.
(62)
0p
0p
0p
So f'x)== Hx, p) is an increasing function with respect to
With the above 2 key properties of fx) in hand, let us check some important values.
(VI). When x = 0, we have:
(K - 1)po3
e- -1)pa3
((()) (002 +) - 1)a2 (a2 - 1) + - (K 2)02 exp
K- + paa)
(63)
0.
holds because every term in f0( is positive. This case shows that, when samples from 1 equal-sized classes are well trained, they form a K regular simplex (31 0 and 3g = K-2
Once samples from the hthh class come, the original K 2 simplex starts shrink the loss goes down when 81 decreases and 32 increases.
(V2). When K- we have:
f'( K - = H(
K - (I.P)
e e1((K - 1)aze -
e e - 1)age x- (p-2)
+
(pa2 1) + e e 1)aze
(a2 1) + e a-1aze- x-1 ((K 2 + p)
(64)
and
1
H(- - 1- 1)= =
(65)
62 H R p) is increasing function with respect to p. Recalling that f(x) is a convex function, with Eq. (64) and Eq. (65). we can conclude that:
According Eq. 66 and Eq.
(C9) When p < 1: f( --) intermediate value theorem, there exists a critical point à E (- S,LT,0) 0), such that f'( = and f(z) attains its minimal value at r = å. If p increases, f((2) increases too. leads to f'( 0, then there comes critical point & E x-T, where f'(ā)
H -[-R__11)
O. Since f'(0) O, according to the
30

---

Published as a conference paper at ICLR 2025
(C10) When p 1: f((-x-)) H(- K-1 1) 0. So K-1 is the critical point and f(x)
attains its minimal value at x = =---I- -K-I
(C11) When p > 1: f'( R-1) > H(- k-1: 1) = And Vr [- 0],
0), f((x) > and
f(x) 2 f( f(-
(V3). When - 1, from (C7) and (C8) we know that f'( < 0f S 1. Now let's only consider the case when p >
(((1)) = e 3(K - 1)a3p
-2(K - 1) + E E2
+ +(K - +(K-1)))
11)a
(002 1) + e e22(K 1)02
(02 1) (K - 2) 022 + e e-2pa2
p- 2(2 -1)e²
= = 1)a3
02p + 2(K 1)02 -
E-2app (K- 1) 02
c (K- 1)02
p+ 02
(66)
02p +1
2020pp C3
e - 1)a3
(11 3)app + (c1 cs + +222))+++00
(02p + ( 02p++03
(K - 1)e 20
I(p),
(02p + +1)p+c)((e a2p + C3)
where 01 = e (K- 1 1, C2 = - - 1)e², C3 = (K - 02 - 1 and:
I(p) = (1 - yazp + (c1 + 03 - x2(2) c102
= e1+e azp² a + (K 1) + e- -2e?)a2 - 2)p 2(K - 1)((K 1)a2 )
(67)
When K 2 3. as long as 02 2 4 > 1/2 holds, c1 > 2e 20 > 0. Also c3 > 0, so we have (K-1)e 2ag
(Dap+o;)le-*on) Fo-pttt > 0, then (-1 1) 2 0 + Mpp 2 0. To solve this inequality, let us first take a
look at the value:
M (q + c3 + 22c2) 2 4(1 + c Ja2cic2
> 4(1 + 2azc102
(68)
= 8(1 +e )a2(K (K- - 1)e²c1
C > 0.
Let bi Ci + c3 + 0g02
(K 1)(1 + - - -2²))ag - 2 < 0, by = 9)2cie2
Jagce2
8(1+e (K - [)az(((K
1) 02 - e²)
> 0 and M = bi+ b2 > 0. Then the solution for L(p) > 0
and also f'(-1) is:
b / 1 + bo
b1 + / 13 + b2
p 2
or p
(69)
2(1 + e aa2
2(1 + 6 2)a²
Since by > O, then / + > وب and so b v bi + by i O. As we only consider the case
where p>> 1. We retain the right-hand part of Eq.
69
Combined with (C11), now we can conclude that: when K and 02 2 4, let
R(K, -bi b + b2
(70)
2(1 )a2
222)aa2 where by -(K-1)(1+e= - -2 )az - 2 0 and by = 8(1 +e - -K) -1)a -e2) 0
31

---

Published as a conference paper at ICLR 2025
(C12) When p R(K, a2): f(-1) < 0. Since Vr E 1- 0), fz) > 0, according to the intermediate value theorem, there exists a critical point & E (-1, - ) such that (i) = and f f() attains its minimal value when x = 2. If p increases, ( increases too. It leads (0) > 0, then there comes new critical point I E (-1, where f' = 0.
(C13) When p [KK, a2): (') 2 Then Vo E [-1,0], (x) 2 0. f(x attains its minimal value when x -1
Combining (C8) to (C13), we conclude that: J(Z) reach its minimal if and only if the following condition holds:
(B4) vk, E f,,..., K} and k + K', (31, Bk) 31 and (Ek, Ex) 332 [K 1
K
Let 31 = c0)( the numerical relationship between p and 01 can be summarized as:
(Case 1) p < 1: 01 (- R).
(Case 2) p 01 -
(Case 3) 1 p< R(K, 02): 01 E (-1, -R-)).
(Case 4) p2 R(K, a2): 0 -1
Here bj = =(K-1)(1 +e - -22 -22")a2 - 2 and 12 = 8(1 (K-+ - 11)aa(KKa1KK)))) -6 R(K, 02) given by:
-b1 + bb + b2
02),
(71)
2(1 + 2-2)
32

---

Published as a conference paper at ICLR 2025
B.3 PROOF OF REMARK 2
Proof. Recall that:
b1 + v B3 4
R(K, = 2(1 + c 22a22
(72)
where bj = (K - -- -
b, - 8(1 + 2) - 1)a2((K
- 1)a2 - -²²). bj and
b2 in Eq. (2) can roughly simplified as
= (K- - 1)(11+==-220 22 2 (K - 1)(1 + t --2-220 -222) = (K - -1)bi
02
(73)
b2
2
8(1 +e ) - 1)(( -1)
2 8(1 + 3K 1)² (K 1)2
03
02
where 8/- = (1 + e 2c²) and b/2 = 8(1 + 2)
Then we can roughly simplifies R(K, a2)) as
function only respect K as:
b +
bi ) by 2(1+6-2
-b, + b bb
R()
2(1 2)a2
-1/ + 1
z *K- 1)
2(1 + 22)
(74)
-(1 + c- - 26 + /(1 (1+e-2- - -2 2 8(1 +e-2)
+
(K 1) (K) s 12.16(K - 1)
2(1 +
33


Reviews:
[{"review_id": "gZSDP13kCM", "reviewer": "Reviewer_itNQ", "summary": "This paper presents a theoretical analysis and solution for the performance degradation of deep learning models when faced with long-tailed data distributions, introducing a method called FeatRecon. The authors provide a systematic exploration of how long-tailed data skews feature representations and detail the resulting limitations on the model's generalization capabilities. Their findings indicate that feature centers of tail classes may shrink and overlap, leading to inseparable representations across different classes.\n\nTo address this issue, the authors propose an algorithm that generates synthetic features to balance the sample sizes of all classes, constraining these synthetic features within confidence supports estimated from head class statistics. Furthermore, their iterative approach aims to create a symmetric and linearly separable feature space, thereby enhancing the model's robustness to long-tailed data. Extensive experiments conducted on multiple long-tailed datasets, including CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018, demonstrate the effectiveness of the proposed method.\n\nOverall, this work effectively bridges theoretical insights and practical applications, offering valuable contributions and innovative techniques for tackling long-tailed data challenges, thereby possessing significant research merit.", "questions": "1. Theoretical Aspects\n\nExpand the Theoretical Framework: Consider relaxing the one - vs - all assumption and researching more complex long - tailed data distribution situations to further enhance the theoretical framework and make it more versatile and practical.\n\nStrengthen the Connection between Theory and Method: In the method description section, more explicitly expound on how to design and optimize the method according to theoretical analysis. For instance, in the processes of synthetic feature generation and confidence support estimation, clarify how to utilize theoretical results to guide the selection and adjustment of parameters, thus better integrating theory and method.\n\n2. Experimental Aspects\n\nSupplementary Comparative Experiments: Pay attention to the latest research advancements in the field of  long - tailed learning and add comparative experiments with more recent methods to evaluate the performance advantages and innovativeness of the FeatRecon method more comprehensively.\nUnify Experimental Settings: Whenever possible, strive to unify the experimental settings across different datasets as much as possible, or provide a detailed explanation for the reasons behind different settings, in order to compare and analyze the experimental results more clearly.\n\n3. Writing Aspects\n\nSimplify the Expression: Simplify and elucidate the complex theoretical derivations and method descriptions. Employ more textual explanations to aid in understanding mathematical formulas and symbols, thus avoiding readers from getting lost while reading.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper has remarkable advantages. In terms of innovation, the theoretical contributions are prominent, including in-depth analysis of the impact of long-tailed data on feature representation and the construction of a theoretical framework for optimal representation configuration; at the same time, the methodological innovation is unique, and the FeatRecon method effectively solves related problems by reconstructing the feature space. The experimental design is elaborate, with extensive and comprehensive datasets, detailed and sufficient comparisons, and reasonable ablation studies. The result analysis shows significant performance advantages and in-depth and detailed discussions.", "weaknesses": "1. Theoretical Basis\n\nLimitations of Theoretical Assumptions: The theoretical analysis mainly relies on a simple one - vs - all scenario assumption, which involves adjusting the sample size of one category while keeping the sample sizes of the others equal and fixed. This assumption may have certain limitations in practical applications and fails to fully cover the complex distribution of long - tailed data.\n\nThe Inadequate Combination of Theory and Practice: Although a theoretical framework is proposed in the paper, in practical applications, the connection between some designs of the FeatRecon method and the theoretical analysis is not direct and close enough. For example, in the processes of synthetic feature generation and confidence support estimation, further elaboration is needed on how to better reflect the analysis of category centers and feature distributions in the theory.\n\n2. Experimental Comparison\n\nIncomplete Comparison with the Latest Methods: Although the paper has compared with numerous existing methods, in some cutting-edge research directions of long-tailed learning, there might be some of the latest methods not included in the comparison experiments, making it impossible to fully determine the absolute superiority of the proposed method in the entire research field.\n\nThe Issue of Consistency in Experimental Settings: There are some differences in the experimental settings for different datasets, such as different backbone networks, varying numbers of training epochs, and diverse data augmentation methods. Although the author may aim to adapt to the characteristics of different datasets, this inconsistency could have a certain impact on the comparison and analysis of the results, and the rationality of different settings requires further explanation.\n\n3. Writing Standards\n\nComplexity in Some Content Representation: In certain sections of theoretical derivations and method descriptions, the paper presents a relatively high level of complexity. It employs a substantial number of mathematical symbols and formulas, thereby augmenting the difficulty for readers to comprehend. For instance, during the proof of Theorem 2, some derivation steps could be elaborated more thoroughly to assist readers in better following the author's line of reasoning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q5YT3kDDST", "reviewer": "Reviewer_2QQJ", "summary": "This work proposes a method based on contrastive learning to address the long-tailed learning problem. By uniformly sampling the confidence supports of the corresponding classes, a consistent number of samples for each class is achieved for contrastive learning. Additionally, the method incorporates the LDAM loss to facilitate supervised learning with long-tail data.", "questions": "See Weaknesses 1 and 2.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors provide a comprehensive theoretical framework.\n2. offers an intriguing method for sampling features with a balanced number of categories and provides a solid theoretical foundation.", "weaknesses": "1. Feature Visualization Lacks Clarity: The feature visualization presented in the manuscript is insufficient. For instance, Figure 4 appears to be a diagram created by the authors. I would like to see whether the actual training results align with expectations. Could the authors provide additional visualization results, such as t-SNE, to illustrate the training process more clearly?\n\n2. Distribution Changes Despite Fixed Empirical Centers: In Figure 4, if the Empirical Centers remain unchanged, what accounts for the changes in distribution? In layman's terms, the class to which a sample belongs is ultimately determined by the \"distance\" of the sample from the class center.  Or maybe the expression regarding Real Distributions is somewhat unclear. I do not fully understand why the real distribution changes and how it should be aligned with confidence supports.  Specifically, what does this refer to? Is it the ground-truth distribution? If the results are as shown in Fig.4 (a), the training samples appear to be completely separated. If only the existing training data is used, the situation seems no different from Fig.4 (d), and the final classifier would not change. Could the authors provide further clarification on this matter?\n\n3. There is a lack of related work, and some important and latest work is missing. \nFor example, \n\nDecoupling methods:\n\n[1] B. Zhou, et al., BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition, in CVPR 2020.\n\n[2] Z. Zhong, et al. Improving Calibration for Long-Tailed Recognition, in CVPR 2021.\n\n[3] M. Li., et al., Feature Fusion from Head to Tail for Long-Tailed Visual Recognition, in AAAI 2024.\n\nLogit adjustment methods:\n\n[4] J. Ren. et al., Balanced Meta-Softmax for Long-Tailed Visual Recognition, in CVPR 2020.\n\n[5] M. Li., et al., Long-tailed visual recognition via gaussian clouded logit adjustment, in CVPR 2022.\n\nTransfer learning (learving pre-trained transformer-based backbone): \n\n[6] J. Shi et al. Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts, in ICML 2024.\n\nIt is recommended that the author conduct a comprehensive analysis of related work. Or conduct a more detailed analysis of the long-tail learning method based on contrastive learning. However, important work should not be lacking.\n\n4. Tiny issue:\nA. K. Menon, et al., Long-tail learning via logit adjustment. ICLR 2021. The work is publushed in ICLR 2021, not 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dpS2s9Tzli", "reviewer": "Reviewer_MXe4", "summary": "* _Theoretical contributions_: The paper introduces a theoretical framework to study how imbalanced/long-tailed (LT) data distributions distort class representations in feature space. It mathematically demonstrates that feature centers of underrepresented classes (tail classes) collapse or shrink, leading to overlapping regions that make class separation challenging. \n\n* _Technical contributions_: The paper proposes a new method, FeatRecon, which reconstructs the feature space to enforce symmetry and linear separability among classes. FeatRecon achieves this by generating _synthetic features_ for underrepresented classes and iteratively adjusting class representations to counteract the collapse effect. It introduces a novel approach to estimate and regularize \"confidence supports\", ie regions in feature space for tail classes using head class statistics.\n\n* _Experimental evaluation_: FeatRecon demonstrates state-of-the-art performance on the common LT benchmarks, with strong accuracy improvements in places, eg for the more challenging iNat dataset.", "questions": "Q1: In Sec 3.2 \"optimal\" representation is defined wrt to training loss. It is unclear to me how this correlates to performance for the case of LT, where at test time we care equally about accuracy over all classes, ie what is defined as  \"distribution shift.\" in 4.1. It would be nice to discuss in the theoretical part how optimality at training in this case is not the same as optimality for recognition accuracy, where the average is over classes.\n\nQ2: You now sample uniformly in the estimated confidence support region. Would it make sense to sample that support space better? e.g. methods like MoCHi [Hard negative Mixing, NeurIPS 2020] generate synthetic features for contrastive learning by interpolating neighbouring features in feature space. Interpolating tail class features using the neighbouring head class features would result in a similar generation (eg in Fig 4). Could be an interesting sampling to ablate.\n\nQ3: How sensitive is regularisation to the hyper parameters like q (ie top-q head classes selected)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper offers an analysis on imbalanced data feature space geometry for the case of the SCL loss. It presents a novel and technically sound theoretical framework that motivates a novel method. The analysis in Sec 2 is complemented with clearly drawn figures/ilustrations. The paper is clearly written and has strong experimental validation supporting the new method.", "weaknesses": "W1: In the proposed method,  the regularize tail classes’ estimation of statistics using the statistics of neighboring head classes. The intuition behind that choice is not clear to me.\n\nW2: There seems to be a typo in Table 5: loss in row 3 is relative to row 2, but gains in rows 4 and 5 are relative to row 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3MfzyTPagu", "reviewer": "Reviewer_nqrn", "summary": "This paper aims to address the long-tailed recognition problem.\nThe paper provides a sound theoretical analysis of contrastive learning from the perspective of neural collapse.\nBased on the analysis, they draw the conclusion that increasing the size of samples for tail classes can alleviate the collapse issue.\nThus, the proposed method is to synthesize sample features for tail classes.\nMoreover, a class-wise temperature schedule is used in their algorithm.\n\nExperiments are conducted on popular long-tailed benchmarks, like ImageNet-LT, iNaturalist, and CIFAR-LT. Some improvements are observed when compared with baselines, like KCL, TSC, and BCL.", "questions": "(1) Fair comparisons with current state-of-the-art methods, like GPaCo [ref1], on large-scale datasets including ImageNet-LT and iNautralist 2018.    \n\n(2) Discussion on the differences between the proposed method and previous augmentation-based methods, like ReMix and work[ref2].\n     What are the advantages of the proposed method over them?\n\n(3) Clarification on the temperature adjustment.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) Sound theoretical analysis on neural collapse for contrastive learning.     \n(2) The paper is well-organized and easy to follow.", "weaknesses": "(1) Lack of comparisons with current state-of-the-art methods, like GPaCo, especially on large-scale data including ImageNet-LT and iNaturalist 2018.\n\n[ref1] Generalized Parametric Contrastive Learning, TPAMI 2023.\n\n(2) Synthesizing sample features for tail classes is not a novel idea. The paper should discuss the differences between the proposed method and previous relevant works, like ReMix and work[ref2].\n\n[ref2] Feature Space Augmentation for Long-Tailed Data, ECCV 2020.\n\n(3) The connection between the proposed method, like feature augmentation and temperature adjustment, and the neural collapse theory is not strong. Increasing the samples for tail classes can obviously alleviate the long-tailed recognition problem. \n\n(4) The claim that \"Head classes benefit from a larger τ while tail classes benefit from a smaller one. We demonstrate\n(in Appendix A.3.2) it holds for SC loss too.\" seems to contradict Eq. (11) that the tail classes will be assigned with a large temperature.\n\n(5) Minor typo, \"N_min - N_min\" should be  \"N_max - N_min\" in Eq. (11).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)