Escaping Saddle Points for Effective Generalization on Class-Imbalanced Data
Authors: Harsh Rangwani
Sumukh K Aithal
Mayank Mishra
Venkatesh Babu Radhakrishnan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Long-Tailed Learning
Class-Imbalanced Learning
Saddle Points

Abstract: Real-world datasets exhibit imbalances of varying types and degrees. Several techniques based on re-weighting and margin adjustment of loss are often used to enhance the performance of neural networks, particularly on minority classes. In this work, we analyze the class-imbalanced learning problem by examining the loss landscape of neural networks trained with re-weighting and margin based techniques. Specifically, we examine the spectral density of Hessian of class-wise loss, through which we observe that the network weights converges to a saddle point in the loss landscapes of minority classes. Following this observation, we also find that optimization methods designed to escape from saddle points can be effectively used to improve generalization on minority classes. We further theoretically and empirically demonstrate that Sharpness-Aware Minimization (SAM), a recent technique that encourages convergence to a flat minima, can be effectively used to escape saddle points for minority classes. Using SAM results in a 6.2\% increase in accuracy on the minority classes over the state-of-the-art Vector Scaling Loss, leading to an overall average increase of 4\% across imbalanced datasets. The code is available at https://github.com/val-iisc/Saddle-LongTail.

Escaping Saddle Points for Effective Generalization on Class-Imbalanced Data
Harsh Rangwani"
Mayank Mishra R. Venkatesh Babu Video Analytics Lab, Indian Institute of Science, Bengaluru, India (harshr@iisc ac in, sumuthaithal66@mail.. com, mayankmishrer shra@i isc. ac.in, in, venkyšiisc. ac in}
Sumukh K Aithal
Abstract
Real-world datasets exhibit imbalances of varying types and degrees. Several techniques based on re-weighting and margin adjustment of loss are often used enhance the performance of neural networks, particularly on minority classes. In this work, we analyze the class-imbalanced learning problem by examining the loss landscape of neural networks trained with re-weighting and margin based techniques, Specifically, we examine the spectral density of Hessian of class- wise loss, through which we observe that the network weights converges to a saddle point in the loss landscapes of minority classes. Following this observation, we also find that optimization methods designed to escape from saddle points can be effectively used to improve generalization on minority classes We further theoretically and empirically demonstrate that Sharpness-Awart Minimization (SAM), recent technique that encourages convergence to a flat minima, can be effectively used to escape saddle points for minority classes. Using SAM results 6.2% increase in accuracy the minority classes over the state-of-the-ar Vector Scaling Loss, leading to an overall average increase of 4% across imbalanced datasets. The code is available at http:///gghuuu..oooooaaissssssie
Introduction
In recent years, there has been a lot of progress in visual recognition thanks to the availability of well- curated datasets [30, 39], which are artificially balanced in terms of the frequency of samples across classes. However, modern real-world datasets are often imbalanced (i.e. long-tailed etc.) [29, 42, 43] and suffer from various kinds of distribution shifts. Overparameterized models like deep neural networks usually overfit classes with high frequency of samples ignoring the minority (tail) ones [7, 43]. In such scenarios, when evaluated for metrics that focus on performance minority data, these models perform poorly. These metrics are an essential and practical criterion for evaluating models in various domains like fairness [12], medical imaging [48] etc.
Many approaches designed for improving the generalization performance of models trained on imbalanced data, are based the re- weightitnn of loss [14]. The relative weights for samples of each class are determined, such that the expected loss closely approximates the testing criterion objective [9]. In recent years, re-weighting techniques such Deferred Re- Weighting (DRW) [9], and Vector Scaling (VS) Loss [28] have been introduced, which improve over the classical re- re-weighting method of weighting the loss of each class sample proportionally the inverse of class frequency. However, even these improved re- re-weighting techniques lead to overfitting on the samples of tail classes. Also, it has been shown that use of re-weighted loss for training deep networks converges final solutions similar to the un-weighted loss case, rendering it to be ineffective [8].
* Equal Contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Imbalanced CIFAR-10
Spectral Head class (CE+DRW)
Landscape
Haae Cheossee
101
20
Merkun Cheessee
Eigerwalues
Mynma
class (CE-DRW)
59.01
Val Acc: 64.1
1011
Tai Caesaa
1079
Eigervaluer Lies
Sodale
Figure I: With class-wise Hessian analysis of loss, we observe that when deep neural networks are trained on class-imbalanced datasets, the final solution for tail classes reach region of large negative curvature indicating convergence to saddle point (bottom), whereas the head classes converge to minima (top). The properties of the loss landscape (saddle points or minima) can be observed by analyzing eigen spectral density (centre). 2
This work looks at the loss landscape in weight space around final converged solutions for networks trained with re- weighted loss. We find that the generic Hessian-based analysis of the average loss used prior works [19, 17], does not uncover any interesting insights about the sub optimal generalization on tail classes (Sec. 3). As the frequency of samples is different for each class due to imbalance, analyze the Hessian of the loss for each class. This proposed way of analysis finds that re- re-weighting cannot prevent convergence saddle points in the region of high negative curvature for tail classes, which eventually leads to poor generalization [16]. Whereas for head classes, the solutions converge to minima with almost significant presence of negative curvature, similar networks trained on balanced data. This problem of converging to saddle points has not received much traction in recent times, the negative eigenvalues disappear when trained on balanced datasets, indicating convergence to local minima [10, 19]. However, surprisingly our analysis shows that convergence to saddle points is still a practical problem for neural networks when they trained on imbalanced (long-tailed) data (Fig. 1).
In the community, lot of techniques optimization methods designed to be able to escape saddle points efficiently [18, 24, 25], some of which involve adding a component of isotropic noise to gradients. However, these methods have not been able improve the performance of deep networks in practice, as the implicit noise of SGD itself mitigates the issue of saddle points when trained on balanced data [15, 25]. However in the case of imbalanced datasets, find that the component of SGD along negative curvature (i.e., implicit noise) is insufficient to escape saddle points for minority classes. Thus, learning imbalanced data can be serve practical benchmark for optimization algorithms that can escape saddle points.
We further demonstrate that Sharpness- technique, with re- re-weighting can effectively enhance the gradient component along the negative curvature, allowing effective escape from saddle points which leads to improved generalization performance. We find that SAM can significantly improve the performance across various re- weighting and margin enhancing methods designed for long-tailed and Class-imbllance learning- The significant improvements are also observed on large-scale datasets of ImageNet-LT and iNaturalist 2018, demonstrating our resutls' applicability at scale. We summarize contributions below:
Minimization (SAM) [17] recent optimization
We propose class-wise Hessian analysis of loss which reveals convergence saddle points in the loss landscape for minority classes. We find that even loss re- re-weighting solutions converge to saddle point, leading sub-optimal generalization the minority classes.

---

We theoretically demostrate that SAM with re-weighting and high regularization factor significantly enhances the component stochastic gradient along the direction of negative curvature, that results in effective escape from saddle points.
We find that SAM can successfully enhance the performance of even state-of-the-ar tech- niques for learning imbalanced datasets which have re-weighting component (e.g. VS Loss and LDAM) across various datasets and degrees of imbalance.
2 Related Work & Background
In this work, we use g(x) to denote the output of model, Vg(æ) to denote the gradients with respect to parameters, and y denote the data and labels, respectively. We review the re-weighting methods used for training on imbalanced data with distribution shifts, followed by optimization techniques related work.
2.1 Long-Tailed Learning
Re-sampling [7] and Re-weighting [21] are the most commonly used methods to train on class- imbalanced datasets. Oversampling the minority classes [11] and undersampling the majority classes [7] are two approaches to re-sampling. Oversampling leads to overfitting on the tail classes, and undersampling discards a large amount of data, which inevitably results in poor generalization. Kang et al. [26] proposed to decouple representation learning and classifier training to improve performance with the same. Mixup Shifted Label- Aworee Smoothing model (MiSLAS) [5]] aims to improve the calibration models trained on long-tailed datasets by mixup and label-aware smoothing and thereby improve performance. RIDE [45] and TADE [50] are ensemble- based methods that achieve state-of-the-att on the long-tailed visual recognition. Samuel and Chechik [41] introduces a new loss, DRO-LT, based distributionally robust optimization for learning balanced feature representations. We explore the problem of training iasalannba datasets through the lens of optimization and loss landscape. We will now describe some representative recent effective methods in detail, which we will use as baselines. Additional discussion on long-tailed learning methods is present in App. H.
LDAM [9]: LDAM introduces optimal margins for each class based on reducing the error through generalization bound. It results in the following loss function where A, is the margin for each class:
g(x), _,,
CLDAM(y:g(Z)) = log ca(m)) A. + Eivn ca(m); where A, for j E {l....,k) (I)
nz
The core idea of LDAM to regularize the classes with low frequency (low i.e. nz) more, in comparison to the head classes with high frequency.
DRW [9]: Deferred Re- Weighting refers to training the model with average loss till certain epochs (K), then introducing weight Wi proportional to 1 1/n, to loss term specific to each class j at a later stage. This way of wewegghtting has been shown to be effective for improving generalization performance when combined with various losses such as Cross Entropy (CE), LDAM etc. We will be using CE+DRW method representative re-eeightnng method for our analysis. We define CE+DRW loss below for completeness:
1
Ccely: g(x)) Wiy (o)(e
po(stl)) where Wjj 1+ (nz - epoch2k
VS[28]: Vector Scaling loss is recently proposed loss function which unifies the idea of multiplica- tive shift (CDT shift [47]), additive shift (i.e Logit Adjustment [35]) and loss re- re-weighting. The final loss has the following form:
Lvs(u:g(a)) = wy log(e (v9()) (ol -y
(3)
Here the y and A, are the multiplicative and additive logit hyperparameters, respectively.
Figures for the minima and saddle point from [4] and used for illustration purposes only.

---

A) CIFAR-10 LT CE+
B) Class O- CCE+DRW
CJ Class 8 CE+DRW
Asae 13
Anee 22,
331
Ane 22.13
Asai 0.300
Acai 1.72
Anasi -2
Denety
10-1
Train Loss: 2.50
Val Acs: 15 81
Vall Acc:
Train Loos:
Tra 03
Acc: 32.1
L2-
1017
-107
10-1
-10P
10
Elgenvatues
Eigenvalues
Eigenvalues
CIFAR-10 LT CE+ DRW (SAM)
[) Class O- -CE+DRW (SAM)
F] Class
CE+DDW (SAM)
Ara- 15
Arwu 22.74
Arer 14.58
Amai -0 0.59
Anmi -1.35
Aneei 2.28
Density
20
Loss: 3.5ed1
Acc: 86
Density
Val Acc: 95.3
Train Look 00-00
Val Acci 72.8
Tra in Loss: 7.0e22
(0-
10-4
20-7
18-F
Eigervaaues
Eigenvalues
Eigenvalues
Figure 2: Eigen Spectral Density (Class-wise) on the head class (Class 0) and tail class (Class 8) with SGD and SAM. It can be observed that with the head classes, the validation accuracy with SGD (B) and SAM (E) are similar and the density of negative eigenvalues is not significant. On the tail class, SAM (F) escapes the saddle points (large density of negative eigenvalues) in SGD (C), leading to 20% increase in validation accuracy. A and D show the overall spectral density calculated across all samples in the dataset. Overall spectral density does not indicate the presence of saddles.
2.2 Loss Landscape
Saddle points are regions in loss landscape that usually depict a plateau region with some negative curvature. In the non-convex setting, it has been shown that there is an existence of an exponential number of saddle points in loss landscape and convergence to these points demonstrate poor generalization [16]. There has been a lot of effort in developing methods for effectively escaping saddle points which involve the addition of noise (e.g., Perturbed Gradient Descent (PGD) [18, 24, 25.). However, these algorithms have not received much attention in the deep learning community as it has been shown that the implicit noise in SGD can escape saddles easily and converge to local minima [15]. Also, it has been empirically shown that negative eigenvalues from the Hessian spectrum disappear after a few steps of training, indicating escape from saddle points when neural networks are trained on balanced datasets [2, 10, 40J. However, contrary this, we demonstrate that convergence to saddle points is prevalent in minority class loss landscapes and is practical problem that can serve as a practical benchmark for the development of algorithms that escape saddle points.
Saddle Points:
Flat Minima based Optimization methods: Empirically, it has been shown that converging to a flat minima in loss landscape for deep network leads to improved generalization in comparison sharp minima [23, 27]. Recent works have tried to exploit this connection between the geometry of the loss landscape and generalization to achieve lower generalization error. Sharpness-Aware Minimization (SAM) [17] is one such algorithm that aims to simultaneously minimize the loss value and sharpness of the loss surface. SAM has shown impressive generalization abilities across various tasks including Natural Language processing [5], meta-learning [1] and domain adaptation [38]. Low-Pass Filtering SGD (LPF-SGD) [6] is another recently proposed optimization algorithm that aims recover flat minima from the optimization landscape. LPF-
convolves the loss function with a Gaussian Kernel with variance proportional to the norm of the parameters of each filter in the network. In this work, we aim to explore the effectiveness of such algorithms for the task of escaping saddle points, which is a new direction for these algorithms.
3 Convergence to Saddle Points in Tail Class Loss Landscape
This section analyzes the dynamics of the loss landscape of neural networks trained on imbalanced datasets. We use the Cross Entropy (CE) loss ÉCE to denote the average cross entropy loss for each
class. For fine- -naiird analysis, we focus on average loss on each class Lee (g). We visualize the loss landscape of the head and tail classes through the computation of Hessian Eigenvalue Density [19].
The Hessian of the train loss for each class H Vi contains important properties about

---

0) Dynamics of training w/ SAM (CIFAR-10 LT)
vs Jets (CIFAR-10 LT Class 9)
Dynamice
LT)
Tear CC
Tol (C
12
Eprochs
0.1
0.2
93
84
65
05
37
16
Eporrs
Figure 3: A) In CE+DRW, the tail class loss landscapes show significant non-convexity as indicated by the large value of Ammi//marl Amarl whereas head classes (0,2) converge convex landscapes. B) When CE+DRW is trained with SAM, it avoids convergence to non-convex regions throughout the training. as indicated by the low value of /Amin/Amaz! C) With high p. the Amin increases, and the model converges to a point with low negative curvature (approx. minima).
the curvature of the classwise loss landscape. The Hessian Eigenvalue Density provides all suitable information regarding the eigenvalues of H. In this work, we focus on (max eigenvalue) and Amin(min eigenvalue), which depict the extent of positive and negative curvature present. We use the Lanczos algorithm introduced in Ghorbani al. [19] to compute the Hessian Eigenvalue Density (spectral density) tractably. We further calculate the validation accuracy of a particular class and its eigen spectral density for analysis. We provide more details for these experiments in the App. D.
Does the proposed class-wise analysis of loss landscape offer any additional insights? In prior works [19, 20, 32], the Hessian of the average loss is used to characterize the nature of the converged point in the loss landscape. However, we find that when particularly trained on imbalanced datasets like CIFAR-10 LT, the eigen spectral density of the Hessian of average loss (Fig. 2A) does not differ from that of head class loss (Fig. 2B), indicating convergence to a local minima. However, explicitly analyzing the Hessian for the tail class loss (Fig. 2C) gives the correct indication of the presence of negative eigenvalues (i.e., curvature), which is in contrast to average loss. Hence, our proposed class-wise wise analysis of Hessian is essential for characterizing the nature of the converged solution when the training data is imbalanced.
What happens when you train a neural network with CE-DRW method on CIFAR-10 LT? Fig. 2 shows the spectral density on samples from the head class (Class O with 5000 samples) and tail class (Class 8 with 83 samples) at the checkpoint with the best validation accuracy. The spectral density of the head class contains few negative eigenvalues. Most of the eigenvalues are centered around zero, as also observed when training on a balanced dataset [19]. On the other hand, for the tail class, there exists a large number of both negative and positive eigenvalues, indicating convergence to a saddle point. We find that at this point, the LCE(Y) is low along with the norm of gradient, which indicates stationary saddle point. We also observe that the spectral density of the tail class contains many outlier eigenvalues, and Xmaz is much larger compared to the head class indicating sharp curvature. These evidences show that the tail class solution converges a saddle point instead of local minimum. Merkulov and Oseledets [36] indicated the existence of stationary points with low error but poor generalization in the loss landscape of neural networks. Also, the existence of saddle points being associated with poor generalization has been observed for small networks [16]. However, in this work, we show that convergence to saddle points can specifically occur the loss landscape of tail classes even for the popular ResNet [22] family of networks, which is an important and novel observation to the best of our knowledge.
Dynamics of training on Long-Tailed Datasets: We analyze the |Amin/ Amazi for the head, mid and tail classes at various epochs (10, 50, 160, 170, 190, 200) across training understand the dynamics of optimization with CE+DRW on long-tailed data (Fig. 3A). AAmin/maan is a measure of non-convexity of the loss landscape [31], where high value of [Amin/ >mazl conveys non-convexity indicating convergence to points with significant negative curvature. The network converges to non-convex regions with negative curvature for tail classes, showing convergence to the saddle point. Also, we find that for the certain tail (Class 7, 8) and mid classes (Class 5), the network starts converging towards regions with negative curvature after applying loss re-weighting (DRW at 160th epoch). This indicates that DRW leads to convergence to saddle point rather than preventing it.
5

---

4 Escaping Saddle Points for Improved Generalization
In this section, we analyze the Sharpness- Minimization technique for escaping from saddle points in tail class loss landscape. In existing works [3, 33, 52]. the effectiveness of SAM in escaping saddle points has not been explored to the best of our knowledge.
Sharpness- Aware Minimization (SAM): Sharpness-Awar Minimization is a recent technique which aims to flatten the loss landscape by first finding a sharp maximal point in the neighborhood of current weight w. It then minimizes the loss at the sharp point (w + +).
min max f(u +) |el|sp
(4)
Here, f is any objective (eg. CE or LDAM loss function) and p is the hyperparameter that controls the extent of neighborhood. A high value of p leads to convergence to much flat loss landscape. The inner optimization in above objective is first approximated using a first order solution:
((w) mrg f(w) + e² f(w()) DV 1e
(5)
After finding E(w), the network weights are updated using the gradient V ff(w)) In recent work [3], it has been shown that the normalization of the norm of the gradient for É(w) calculation above leads oscillation which implies non-convergence theoretically. Also, has been empirically shown that the unnormalized version of the gradient with adjusted performs better than the normalized version. Hence, we use the approximation i.e. ((w) pV f(w) for our theoretical results. As we will be using the stochastic version of the gradient, we use 2 as the stochasticity parameter and denote the gradient as V (w)). With this, we now define the gradient with respect to that associated with SAM:
VJ SAM (ww) = f + ((w)) = f= + (z)(())
(6)
As we are using the same batch for obtaining the gradient to calculate the ((w) and loss, we can use the same the argument. We now analyze the component of the SAM gradient in the direction of negative curvature, which is required for escaping saddle points [15].
4.1 Analysis of SAM for Escaping Saddle Points
Our analysis is based on the Correlated Negative Curvature (CNC) assumption [15] that states that stochastic gradients have components along the direction negative curvature, which helps them escape from saddle points. This assumption has been shown to be theoretically valid for the problem of learning half-spaces and also has been empirically verified for a large number of neural networks of different sizes [15]. We now formally state the assumption below:
ASSUMPTION 1 (Correlated Negative Curvature [15]). Let Vw be the minimum eigenvector cor- responding to the minimum eigenvalue of the Hessian matrix V f(w). The stochastic gradient ff=(w) satisfies the CNC assumption if the second moment of the projection along the direction Vw uniformly bounded away from zero, i.e.
3y 2 0 s.l. Vw E(< Vw; ( > 27
It has also been emphasized that the value of y is shown to correlate with the magnitude of X This shows that with a high negative eigenvalue, there is large component of gradient along the negative curvature along Vnp This allows the SGD algorithms to escape the saddle points. However, we find that in the case of class imbalanced learning (Fig. 1) even stochastic gradients may have an insufficient component in the direction of negative curvature to escape the saddle points. We now show that SAM technique, which aims to reach flat minima, further amplifies the gradient component along negative curvature and can be effectively used escape the saddle point. We now formally state our theorem based on the CNC assumption below:
THEOREM 2. Let Vw be the minimum eigenvector corresponding to the minimum eigenvalue Amin of the Hessian matrix 2 f(w. The V FSAA (w) satisfies that it's second moment of projection in U is atleast (1 + pAmin) )2 times the original (component of V [w())):
By s.t. Vw: E[< Vw, V SAM
F.SAM
(w) > 2 (1 + pdmin)
(8)

---

Table 1:
Results on CIFAR- 10 LT and CIFAR- 100 LT with 3=100. SAM with re- re-weighting is able to avoid the regions of negative curvature, leading to major gain in performance on the mid and tail classes with CE, L.AMM and VS.
CIFAR-IO
CIFAR- 100 LT
Acc
Head
Mid
Tail
Acc
Hcad
Mid
Tail
CE CE + SAM
71.7 10.1
90.813.
71.910
52.3
38.5
64.5
36.811.
8.2110
73. 10.3
93.3
74.1
51.7
39.6.0.6
66.510.7
38.1
8.010.8
CE DRW [9]
25.5
91.620.4
74.1
61.420.9
41.0.2.6
61.311.
41.7
14.720.9
CE+ DRW SAM
80.6
91.4+0.3
78.0+0.4
73.1 +0.9
44.6.n.4
61.2
47.5-0.6
20.7
LDAM + DRW [9]
77.5
91.1
75.7
66.4_0.2
42.7
61.8
42.2-1.o
19.4
LDAM + DRW SAM
81.9
91.0=a.2
79.2+0.n
76.4 #1.1
45.4=m.1
64.4
46.2=002
20.8 =0.3
VS [28]
78.6
90.619.4
75.8
70.3
41.7
54.4
41.1
26.811.0
VS SAM
82.4
90.7.0.0
79.6
78.0.a1.2
46.6.2044
56.420.4
48.820.6
31.720.1
REMARK. The above theorem adds the factor (1 + pamin)² to increase the component in direction of negative curvature (7) when Amin I = Due to this increase, the model will be able to escape from directions with high negative curvature, leading to an increased Amin Also, as the factor =2 is inversely proportional to p. the high value of p aids in effectively increasing the minimum negative eigenvalue. To empirically verify this, we evaluate the Hessian spectrum for the CIFAR-10 LT dataset using CE-DRW method for different values of (Fig. 3C). We find that, as expected from the theorem, in practice, the high values of p lead to less negative values of Amin. This indicates escaping the saddle points effectively, hence avoiding convergence to regions having negative curvature in loss landscape. The proof of the above theorem and additional details is provided in Appendix B.
We also want to convey that theoretically, techniques like Perturbed Gradient Descent (PGD), and LPF-SGD (Low-Pass Filter SGD), which add Gaussian noise into gradient to escape saddle points can also used for mitigating negative curvature. Also it has been found that SGD [15] can also escape the saddle points and converges to solutions with flat loss landscape. Also, theoretically according to Theorem 2 in Daneshmand et al. [15] the SGD algorithm convergence to second-order stationary point depends on the y as O(y under some assumptions on f. As we find that SAM with high P enhances the component of SGD in direction of negative curvature (7) by (1 pdmmm)² is reasonable to expect that SAM is able to escape saddle points effectively and converge to solutions with significant less negative curvature quickly implying better generalization. We provide empirical evidence for this in Fig. 3B and Sec. 5.2.
What happens when you train neural network with SAM DRW? With SAM (high p). the large negative eigenvalues present in the loss landscape of the tail class get suppressed (Fig. 2F). In the spectral density for the tail class, can be seen that Amin much closer to zero for SAM compared to its counterpart with SGD. This aligns with the hypothesis that SAM escapes regions of negative curvature, leading improved accuracy on the tail classes. However, the spectral density of the head class does not change significantly compared to that of Empirical Risk Minimization (ERM), although the Amaz is much lower for SAM, indicating a flatter minima for the head class.
We also analyze the /mmin/ Amoz across multiple steps of training with SAM (Fig. 3B), where DAmin/ /mmaz| is a measure of non-convexity of the loss surface. We observe that SAM does not allow the tail classes to reach a region of high non-convexity. The values of Amin/ /mmaa is much lower for SAM compared SGD (Fig. 3A) throughout training, indicating minimal negative eigenvalues (i.e. more convexity) in the loss landscape, especially for the tail and medium classes. This clearly shows that SAM avoids regions of substantial negative curvature in the search of flat minima. Further, we note that once the re- ee-eighttig begins, SAM is able to avoid convergence to a saddle point (non- convexity decreases), which is contrary to what we observe with CE+DRW (with SGD). Theorem states that SAM consists of larger component in the direction of negative curvature which allows to reach a solution with minimal negative curvature. Empirically, Fig. 3B also supports the Theorem as we observe that SAM reaches minima (high convexity) for all the classes.
7

---

Table 2: Results on CI and CIFAR- 100 with Step
P vs Accuracy (CIFAR-10
Imbalance (8 = 100). SAM generalizes well across datasets with varied type of imbalance, resulting substantial gain in tail accuracy in all settings.
80.0
Accurayy
23.0
CIFAR-10
CIFAR-
Ace
Head
Tail
Acc
Hcad
Tail
40.5
CE CF SAM
65.1
83.6
41.7
38.6
76.3
00.9
66.1
92.9
39.4
39.3
78.6
00.0
TalArarecy
CE + DRW [91
72.2
93.1
51.2
45.8
73.9
17.8
CE DRW SAM
79.3
92.7
48.3
73.1
23.4
Figure 4: Impact of p (regulariza- tion factor) on Overall Accuracy and Tail Accuracy (CIFAR-10 LT).
LDAM + DRW [9]
77.6
89.2
66.0
45.3
70.3
20.4
LDAM + DRW + SAM
81.0
90.5
71.5
49.2
74.0
24.4
VS [28]
77.0
91.7
62.3
46.5
69.0
24.1
VS + SAM
82.0
91.7
72.3
48.3
70.4
26.2
5
Experiments
5.1 Class-Imbalanced Learning
We report our results on four long-tailed datasets: CIFAR-10 LT [9], CIFAR-IOK 100 LT [9]. ImageNet-LT [34], and iNaturalist 2018 [44]. a) CIFAR-10 LT and CIFAR-100 LT: The original CIFAR-10 and CIFAR- 100 datasets consist of 50,000 training images and 10,000 validation images, spread across 10 and 100 classes, respectively. We use two imbalance versions, i.e, long-tail imbalance and step imbalance, as followed in Cao et al. [9]. The imbalance factor, Numxx
Datasets:
Nmo denotes the ratio between the number of samples in the most frequent (Nmax) and least frequent class (Nmin). For both the imbalanced versions, we analyze the results with 3 = 100. b) ImageNet-L.T and iNaturalist 2018: We use the ImageNet-LT version as proposed by [34], which is an itassiimbllaneed version of the large-scale ImageNet dataset [39]. It consists of 1 115.8K images from 1000 classes, with 1280 images in the most frequent class and 5 images in the least. iNaturalist 2018 [44] is a real-world long-tailed dataset that contains 437.5K images from 8,142 categories. In the case of long-tail imbalance, we segregate the classes of all the datasets into Head (Many), Mid (Medium), and Tail (Few) subcategories, as defined in 1. For step imbalance experi- ments on CIFAR datasets, we split the classes into Head (Frequent) and Tail (Minority), as done in [9].
We follow the hyperparameters and setup as in Cao al. [9] for CIFAR-10 LT and CIFAR-100 LT datasets. We train a ResNet-32 architecture as the backbone and SGD with momentum of 0.9 as the base optimizer for 200 epochs. A multi-step learning rate schedule is used, which drops the learning rate by 0.01 and 0.0001 at the 160th and 180th epoch, respectively. For training with SAM, we set a constant P value of either 0.5 or 0.8 for most methods. For ImageNet-LT and iNaturalist 2018 datasets, we use the ResNet-50 backbone similar to [51]. An initial learning rate of 0.1 and 0.2 is set for iNaturalist 2018 and ImageNet-LT, respectively, followed by cosine learning rate schedule. We initialize the P value with 0.05 and utilize a step schedule to increase the p value during the course of training for SAM experiments. We run every experiment on long-tailed CIFAR datasets with three seeds and report the mean and standard deviation. Additional implementation details are provided the App. C. Algorithm for SRW+ is defined in App. G.
Experimental Details:
Baselines: a) Pross-Entropy (CE): CE minimizes the average loss across all samples, and thus. the performance of tail classes is much lower than that of head classes. b) CE + Deferred Re-Weighting (DRW) [9]: The re- weighting of CE loss inversely by class frequency is done in the later stage of training. e) LDAM DRW [9]: a margin- loss that encourages larger margins for less- frssufre classes. d) Vector Scaling (VS) Loss [28]: VS loss incorporates both additive and multiplicative logit adjustments to modify inter-class margins.
Margin (LDAM) proposes
Table 1 summarizes our results on CIFAR-10 LT and CIFAR- 100 LT with ,3 of 100. It can be observed that SAM with re-weighting significantly improves the accuracy on mid and tail classes while preserving the accuracy on head classes. SAM improves upon the overall performance
Results:
8

---

Table Results on iNaturalist 2018 and ImageNet-LT datasets with LDAM+DRW and comparison with other methods. The numbers for methods marked with are taken from [51].
Method
iNaturalist 2018
ImageNet-LT
Two stage
Ace
Head
Mid
Tail
Acc
Head
Mid
Tail
CE
60.3
72.8
62.7
54.8
42.7
62.5
36.6
12.5
cRT [26] 1
68.2
73.2
68.8
66.1
50.3
62.5
47.4
29.5
LWS [26] f
69.5
71.0
69.8
68.8
51.2
61.8
48.6
33.5
MiSLAS [51]
71.6
73.2
72.4
70.4
52.7
61.7
51.3
35.8
DisAlign [49]
69.5
61.6
70.8
69.9
52.9
61.3
52.2
31,4
DRO-LT [41]
69.7
73.9
70.6
68.9
53.5
64.0
49.8
33.1
CE + DRW
X
63.0
59.8
64.4
62.3
44.9
57.9
42.2
21.6
CE + DRW + SAM
65.3
60.5
66.2
65.5
47.1
56.6
45.8
28.1
LDAM + DRW
X
67.5
63.0
68.3
67.8
49.9
61.1
48.2
28.3
LDAM + DRW SAM
X
70.1
64.1
70.5
71.2
53.1
62.0
52.1
34.8
of CE+DRW by 5.1% on CIFAR-10 LT and 3.6% on CIFAR-100 LT datasets, with the tail class accuracy increasing by 11.7% and 7.7% respectively. These results empirically show that escaping saddle points with SAM leads to a notable increase in overall accuracy primarily due to the major gain in the accuracy on the tail classes. The addition of SAM recently proposed long-tail learning methods like LDAM and VS loss leads to a significant increase in performance, which indicates that the role of SAM is orthogonal the margin- margin-based methods. On the other hand, SAM without re-weighting (CE+SAM) improves accuracy on the head and mid classes rather than the tail class. This can be attributed to the fact that standard ERM minimizes the average loss across all the samples without re-weighting such that the weightage of tail class samples in the overall loss is minimal. This shows that naive application of SAM ineffective improving tail class performance, comparison to proposed combination of re- re-weighting methods with SAM. We also show improved results with various imbalance factors (5) in App. F.
We also show results with step imbalance (6 100) on CIFAR- 10 and CIFAR-IO0 datasets (Table 2). With step imbalance on CIFAR-10. the first five classes have 5000 samples each, and the remaining classes have 50 samples each. The addition of SAM improves the overall performance of CE+DRW on CIFAR-10 by 7.1 %, with the tail class accuracy increasing by 14.6%. We observe that on most tail classes, the density of negative eigenvalues in the spectral density is much lower with SAM. This indicates that despite multiple classes with few samples, SAM with DRW can avoid the saddle points. SAM systematically improves performance with LDAM and VS loss leading to state-of-the-art performance on both CI and CIFAR- 100 in the step imbalance setting.
Do these observations scale to large-scale datasets? We report the results on ImageNet-LT dataset in Table 3. We also compare with recent long-tail learning methods: cRT [26]. MisLAS [51], AisAlign [49] and DRO-LT [41]. The observations on CIFAR- 10 LT and CIFAR- -IFAR-100 LT hold good even on ImageNet-LT. For example, the accuracy on tail classes increases by 6.5% with the introduction of SAM on CE + DRW, which is similar to the gain observed in CIFAR-100 LT with CE + DRW. We observe that LDAM+DRW including MisLAS, cRT, LWS, and DisAlign. Compared to these two-stage methods, our method is single stage method and outperforms these two-stage methods. These observations point out that the problem of saddle points also exists in large datasets and convey that SAM is easily generalizable to large-scale imbalanced datasets without making any significant changes. On iNaturalist 2018 [44] too, the accuracy tail classes gets boosted by more than 3% with SAM (Table 3).
surpasses the performance of two-stage training methods
Comparison with SOTA: VS loss [28] is a recently proposed margin-based method that achieves state-of- of-the-arr art performance on iaas--iballanced datasets with single-stage training without strong augmentations [51], ensembles [50] or self sulesuupevv [46]. SAM significantly improves upon the performance of VS on both CIFAR- 10 LT and CIFAR- 100 LT. For the practitioners, we suggest using high p SAM with re-weighting or margin based methods for effective learning on long-tailed data. We also integrate SAM with more recent IB-Loss [37] and Parametric Contrastive Learning (PaCo) [13] methods and report the results App. E. We find that SAM is also effectively able to improve performance of these recent methods.

---

Table 4: Results on CIFAR- 10 LT and CIFAR-100 100 LT with various methods that escape saddle points.
CIFAR-10 10 LT
CIFAR- 1PRAR1100 LT
Acc
Head
Mid
Tail
Acc
Head
Mid
Tail
CE + DRW
75.5
91.6
74.1
61.4
41.0
61.3
41.7
14.7
CE DRW PGD [24]
77.2
92.0
75.2
65.0
42.2
63.0
41.6
17.0
CE + DRW + LPF-SGD [6]
78.5
90.8
77.7
67.2
42,9
64.0
43.7
15.8
CE DRW SAM
80.6
91.4
78.0
73.1
44.6
61.2
47.5
20.7
5.2 Ablation Studies
note on o value: We observe that as we increase the smoothness parameter (p) in SAM, the accuracy the tail classes increases significantly (Fig. 4). The accuracy tail classes increases from 63% for p= 0.05 73% for p 0.8 on CIFAR- LT with CE+DRW. This can be ascribed to the correlation between Amin and / as discussed in Sec. 4.1. As the p increases, the negative curvature in the tail classes disappears because SAM aims to find a flat minima with a large neighhorhood with low loss value. A very large P (0.8) leads drop in the head accuracy because it restricts the solution space of the head class, resulting in a drop in the overall accuracy. This also emphasizes that high p necessary for escaping saddle points and achieving the best results.
Other methods escape saddle points: In Table 4, we show that other methods developed to escape saddle points, such as PGD, be used for improving generalization tail classes. LPF-SGD, an algorithm promoting convergence to flat landscape, inherently adds Gaussian noise to the network parameters and could be considered similar PGD. We LPF-SGD to CE+DRR DRW leads to a substantial gain in the performance tail classes on CIFAR-10 LT and CIFAR-100 LT. It can also be observed that CE+DRW+SAN outperforms both PGD and LPF-SGD by 2% on average. This further highlights that various methods literature developed to escape saddle points efficiently be directly used to improve the performance of minority classes when training on class-imbalanced datasets.
that the addition of PGD and
6 Conclusion
In this work, we show that training on imbalanced datasets can lead to convergence to points with sufficiently large negative curvature in the loss landscape for minority classes. We find that this is quite common when neural networks are trained with loss functions that are re- re-weigt or modified to enhance the focus on minority classes. Due to the occurrence of saddle points, we observe that the network suffers from poor generalization on minority classes. We propose to use Sharpness- Aware Minimization (SAM) with high regularization factor regions negative curvature and enhance the generalization performance. We theoretically and empirically demonstrate that SAM with high is able escape saddle points faster than SGD and converge to better solutions, which a novel observation to the best of our knowledge. We show that combining SAM with
effective method to escape
techniques for learning with imbalanced data leads to significant gains in performance on minority classes. We hope that our work leads to further research in studying the effect of negative curvature in generalization as show they practical issue for class-ing iinalannbalneee learning using deep neural networks.
Acknowledgements: This work was supported in part by SERB-STAR Project (Project: Proojee.SSTR/20200000222 Govt. of India. Harsh Rangwani supported by Prime Minis- ter's Research Fellowship (PMRF). We are thankful for their support.
10

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Added in appendix.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] To the best of our knowledge, we do suspect any potential negative societal impacts of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
(b) Did you include complete proofs of all theoretical results? [Yes] Proof is added in the appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [Yes] Additional information added in Appendix C
(b) Did you specify all the training details (e.g., data splits, byperparameters, how they were chosen)? [Yes] See Section Additional information added in Appendix C.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See section 5. We report error bars for CIFAR 10-LT and CIFAR 100-LT dataset. For the large scale ImageNet-LT and iNaturalist 2018, we do not report error bars due to computational constraints.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Added in the Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] Added in Appendix I
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [N/A] We use publicly available assets for our work.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]

---

References
[1] Momin Abbas, Quan Xiao, Lisha Chen, Pin-Yu Chen, and Tianyi Chen. Sharp-maml: Sharpness-
aware model-agnostie meta leaming. arXiv preprint arXiv:2206.03990 2022. 4
[2] Guillaume Alain, Nicolas Le Roux, and Pierre-Antoine Manzagol. Negative eigenvalues of the hessian in deep neural networks. arXiv preprint arXin:1902.02366, 2019, 4
[3] Maksym Andriushchenko and Nicolas Flammarion. Understanding sharpness- aware minimiza- tion, 2022. URL https: pttttp:://opereeviiii http://wwwweeeorrrrrrraaaaaaaaae
6
[4] Raman Arora, SANJEEV Arora, Joan Bruna, NADAV Cohen, SIMON DU, RONG GE, SURIYA GUNASEKAR, C JASON LEE, TENGYU MA, et al. Theory of deep learning, 2020.
[5] Dara Bahri, Hossein Mobahi, and Yi Tay. Sharpness-aware minimization improves language model generalization. arXiv preprint arXiv:21 2021. 4
[6] Devansh Bisla, Jing Wang, and Anna Choromanska. Low-pass filtering sgd for recovering flat optima in the deep learning optimization landscape. arXiv preprint 1xXiv:2201.00025, 2022. 4,
10
[7] Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249-259, 2018. 1,
3
[8] Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning?
In International Conference on Machine Learning, pages 872-881 281. PMLR, 2019.
[9] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with aabel-distibbutionaawe margin loss. Advances neural information processing systems, 32, 2019. 1, 3, 7,8
[10] Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):
124018, 2019. 2.4
[11] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16: 321-357, 2002.
[12] Andrew Cotter, Maya Gupta, Heinrich Jiang, Nathan Srebro, Karthik Sridharan, Serena Wang, Blake Woodworth, and Seungil You. Training well-generalizing classifiers for fairness metrics and other data-dependent constraints. In International Conference on Machine Learning, pages 1397-1405. PMLR, 2019.
[13] Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 715-724, 2021.
[14] Yin Cui, Menglin Jia, Tsung-Yi Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9268-9277, 2019.
[15] Hadi Daneshmand, Jonas Kohler, Aurelien Lucchi, and Thomas Hofmann. Escaping saddles with stochastic gradients. In International Conference on Machine Learning, pages 1155-1164
PMLR, 2018. 2, 4, 6,
[16] Yann Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non- convex optimization. Advances in neural information processing systems, 27, 2014. 2, 4,
5
12

---

[17] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aae aware min- imization for efficiently improving generalization. In International Conference on Learning 2,
Representations, 2021. URI. hwtptttt//// pttttps://pperevviiie wtwpe//wwwwwee.eeett.rrereeee met//forummddd=STnmlmp
[18] Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points online stochastic gradient for tensor decomposition. In Conference on learning theory, pages 797-842.
PMIR, 2015. 4
[19] Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net opti- mization via hessian eigenvalue density. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2232-2241. PMLR, 09-15 Jun 2019. URL mlr attps://proceediin: wtt.:/ss....rpee.eppepeeee ghwggghorbnnin wetebaaniiib..o. html. 2, 4,
[20] Justin Gilmer, Behrooz Ghorbani, Ankush Garg, Sneha Kudugunta, Behnam Neyshabur, David Cardoze, George Dahl, Zachary Nado, and Orhan Firat. A loss curvature perspective on training instability in deep learning. arXiv preprint arXiv:2110.04369, 2021. 5
[21] Haibo He and Edwardo A. Garcia. Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 2119):1233-1288 2009. doi: 10. 10.100000 239. 3
[22] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016. 5
[23] Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1):1-42, 1997.
4
[24] Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M Kakade, and Michael I Jordan. How to escape saddle points efficiently. In International Conference on Machine Learning, pages 1732.
PMLR, 2017. 2, 4, 10
[25] Chi Jin, Praneeth Netrapalli, Rong Ge, Sham M. Kakade, and Michael I. Jordan. Stochastic gradient descent escapes saddle points efficiently. ArXiv, 1ttp//9022.481111 1, 2019. 2,4
[26] Bingyi Kang. Saining Xie, Mareus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representutions, 2020. URL https: pttttp:://ppereeveee //pp///ppeeee..e.e//
net/forum? id=r1gRTCVFvB.
[27] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On larce-batch training for deep learning: Generalization gap and sharp minima. arXiv preprini arXiv: (116666994488 2016. 4
[28] Ganesh Ramachandra Kini, Orestis Paraskevas, Samet Oymak, and Christos Thrampoulidis. Label-imbalanced and group-sensitive classification under overparameterrizatoo.. Advances in Neural Information Processing Systems, 34, 2021. 1, 3, 7, 8,
[29] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123()):32-73, 2017.
[30] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[31] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018. 5
[32] Xinyan Li, Qilong Gu, Yingxue Zhou, Tiancong Chen, and Arindam Banerjee. Hessian based analysis of sgd for deep nets: Dynamies and generalization. In Proceedings of the 2020 SIAM International Conference Data Mining, pages 190-198. SIAM, 2020. 5
13

---

[33] Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsich, and Yang You. Towards efficient and scalable sharpness-awart minimization. arXiv preprint arXiv:2203.027744 2022.
[34] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition an open world. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 2537-2546, 2019. 8
[35] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.07314 2020.
[36] DM Merkulov and Ivan V Oseledets. Empirical study of extreme overfitting points of neural networks. Journal of Communications Technology and Electronics, 64(12):1527-1534, 2019.
[37] Seulki Park, Jongin Lim, Younghan Jeon, and Jin Young Choi. suftuence-bllanee loss for imbalanced visual classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 735-744, October 2021. 9
[38] Harsh Rangwani, Sumukh K Aithal, Mayank Mishra, Arihant Jain, and R. Venkatesh Babu. A closer look at smoothness in domain adversarial training. In Proceedings of the 39th International Conference on Machine Learning, 2022. 4
[39] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3)2211-22) 252, 2015. 1,8
[40] Levent Sagun, Utku Evei, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of perepparaetrizee neural networks. arXiv preprint arXiv:/706.0444 04454, 2017.
[41] Dvir Samuel and Gal Chechik. Distributional robustness loss for long-tail learning. In Proceed- ings the IEEE/CVF International Conference Computer Vision (ICCV), pages 9495-9504, October 2021. 3,9
[42] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64-73, 2016.
[43] Grant Van Horn and Pietro Perona. The devil is in the tails: Fine-grained classification in the wild. arXiv preprint 17Xvv:170.001450 2017.
[44] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 8769-8778. 2018. 8,9
[45] Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella Yu. Laigetailed recogni- tion by routing diverse distribution aware experts. In International Conference on Learning Representations, 2021. URL https: //tttper//perreviier net fd 3
[46] Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class imbalanced learning. Advances in Neural Information Processing Systems, 33:19290-19301, 2020. 9
[47] Han-Jiz Ye, Hong- You Chen, De-Chuan Zhan, and Wei-Lun Chao. Identifying and compensat- ing for feature deviation in imbalanced deep learning. arXiv preprint arXiv:2001.01335, 2020.
3
[48] Chuanhai Zhang. Medical image classification under class imbalance. PhD thesis, Iowa State University, 2019.
[49] Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, and Jian Sun. Distribution alignment: A unified framework for long-tail visual recognition. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 2361-2370, 2021. 9
14

---

[50] Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. Test-agnostic long-tailed recognition by test- test-time aggregating diverse experts with sulervupevvsi arXiv preprint arXiv:2107.09249 2021. 3, 9
[51] Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recognition. In Proceedings of the /EEE/CVF conference on computer vision and pattern recognition, pages 16499, 2021. 3, 8, 9
[52] Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha C Dvomek,, sekhar tatikonda, James s Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-aware training. In International Conference on Learning Representations, 2022. URL https:/ htttp:://operreiiee wwwpearrrieee.nee/oormmmmmm
15


Reviews:
[{"review_id": "mTY16gHSkF", "reviewer": "Reviewer_oRk4", "summary": "The authors analyze the class-imbalanced learning problem through the lens of the loss landscape. \nSpecifically, they examine the spectral density of Hessian of class-wise loss, in which they observe that for the tail class loss landscape, the solution converges to a saddle point. \nThey further theoretically and empirically demonstrate that Sharpness Aware Minimization (SAM), a recent technique that aims to converge to flat minima, can be effectively used to escape saddle points for minority classes.\nBorrowing SAM results in an increase in accuracy in the minority classes.\n", "questions": " ", "limitations": "The authors addressed the limitations of the proposed method and the potential negative social impact of their work.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n- The idea to interpret the long-tailed distribution problem with respect to the loss of landscape is novel.\n- The authors validate the effect of the SAM for model training theoretically and visually.\n\nWeakness\n\nThe proposed solution for the long-tailed recognition problem is not novel enough. It would be better to propose a novel method by modifying the existing SAM method to be especially helpful for the tail classes. Simply borrowing the existing method, which is generally helpful, might not be enough for complete research. Also, empirically in Table 1,2,3, it is hard to see the SAM is especially helpful for the tail classes (especially helpful to alleviate the long-tailed distribution). \n\nThe experiments also seem to be weak. For example, in Table1, the authors should also evaluate their method with other imbalance ratios. \n\nAlso, they should compare with more recent state-of-the-art long-tailed recognition works [1,2,3,4]; the baselines shown in the paper are too weak. In order to claim the effectiveness of the proposed method, the authors should apply the proposed method to the state-of-the-art methods and see the improvements instead of only applying the method on simple baselines.\n\n[1] Self Supervision to Distillation for Long-Tailed Visual Recognition, ICCV 2021\n[2] Parametric Contrastive Learning, ICCV 2021\n[3] Influence-Balanced Loss for Imbalanced Visual Classification, ICCV 2021\n[4] Distilling Virtual Examples for Long-tailed Recognition, ICCV 2021\n\n========== ------- Comments after the rebuttal ------========\nI have read the other reviewer's comments and the author's rebuttal, which addresses most of my concerns. Therefore, I would like to raise my score.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QOQiZ85hQFR", "reviewer": "Reviewer_fv7S", "summary": "In this paper, the authors find that the key issue for imbalanced classification is that the training for minority classes can lead to convergence to saddle points of their loss landscape. This phenomenon cannot be avoided in the reweighting methods. Consequently, the network suffers from poor generalization on minority classes. To escape from the saddle points, they introduce Sharpness-Aware Minimization (SAM) with a high regularization factor $rho$ to enhance the generalization performance. They theoretically and empirically demonstrate that SAM with high $rho$  is able to escape saddle points faster than SGD and converge to better solutions.", "questions": "1. Can you explain why the negative eigenvalues appear on imbalanced datasets and disappear on balanced datasets? Do they appear because of the negative gradients from the majority classes? If so, please analyse the differences and compare with the equalization loss (Equalization loss for long-tailed object recognition, CVPR 2020).\n2. Is SAM only applied on the loss of the minority classes? I think providing an overall algorithm is better.", "limitations": "Does the saddle points exist in other methods, such as data augmentation and sample-level reweighting methods?", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. Analysis the problem of imbalanced classification from the persepective of optimaization peocedure is novel.\n2. The conclusion that saddle points cannot be avoided in reweighting methods is interesting.\n3. Sufficient theoretically and empirically analysis.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NsDNT-a5hQw", "reviewer": "Reviewer_hF5j", "summary": "The authors find that for the tail class loss landscape, the solution converges to a saddle point and the network thus suffers from poor generalization on minority classes. This work uses Sharpness-Aware Minimization (SAM) to escape saddle points and enhance the generalization performance, which has been theoretically and empirically demonstrated. Experimental results show that combining SAM with SOTA techniques leads to significant gains in performance on minority classes.", "questions": "How the sharp maximal point ϵ and weight w are updated in this work?\n\nWhat is the difference between  θ and w? ", "limitations": "1 The Algorithm should be presented and described in detail, which is helpful for understanding the proposed method.\n2 The background of Sharpness-Aware Minimization (SAM) shoud be described in detail. \n", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength \n1 The idea of studying the effect of negative curvature for class-imbalanced learning using deep neural networks is novel and interesting, which may pave a new way for further research on imbalanced classification.\n2  Experimental results empirically show that escaping saddle points with SAM leads to a notable in crease in overall accuracy primarily due to the major gain in the accuracy on the tail classes.\n\nweakness\n1 The Algorithm should be presented and described in detail.\n2 The background of Sharpness-Aware Minimization (SAM) shoud be described in detail. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Inadequate Data and Algorithm Evaluation", "I don’t know"]}, "extra_text": {}}, {"review_id": "LNoELmrLw1n", "reviewer": "Reviewer_s5kU", "summary": "In the real-world, many datasets are imbalanced, which can degrade the trained model’s performance. To tackle this problem, this paper analyzes the class-imbalance problem by examining the spectral density of Hessian of class-wise loss. From the observation that the tail class loss converges to a saddle point, this paper proposes to use sharpness aware minimization (SAM). This paper justifies the proposed method theoretically and empirically on multiple benchmark datasets.", "questions": "Please refer to the weakness.", "limitations": "The limitations and potential negative societal impact are addressed,", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- This paper justifies the proposed method with theoretical justification and various analyses.\n- The proposed method improves the baselines. \n- The paper is well written.\n\nWeaknesses\n- The comparison baselines and the prior literature on long-tailed learning is limited.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept