Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
Authors: Sicong Li
Qianqian Xu
Zhiyong Yang
Zitai Wang
Linchao Zhang
Xiaochun Cao
Qingming Huang
Venue: ICML 2025 poster (2025)
Keywords: Long-tailed Learning
Primary area: general_machine_learning->supervised_learning

Abstract: Real-world datasets often follow a long-tailed distribution, making generalization to tail classes difficult. Recent methods resorted to long-tail variants of Sharpness-Aware Minimization (SAM), such as ImbSAM and CC-SAM, to improve generalization by flattening the loss landscape. However, these attempts face a trade-off between computational efficiency and control over the loss landscape. On the one hand, ImbSAM is efficient but offers only coarse control as it excludes head classes from the SAM process. On the other hand,  CC-SAM provides fine-grained control through class-dependent perturbations but at the cost of efficiency due to multiple backpropagations. Seeing this dilemma, we introduce Focal-SAM, which assigns different penalties to class-wise sharpness, achieving fine-grained control without extra backpropagations, thus maintaining efficiency. Furthermore, we theoretically analyze Focal-SAM's generalization ability and derive a sharper generalization bound. Extensive experiments on both traditional and foundation models validate the effectiveness of Focal-SAM.

Focal-SAM: Focal Sharpness-Awar Minimization for Long-Tailed Classification
Sicong Li
Qianqian Xu 3 Zhiyong Yang* Zitai Wang
Linchao Zhang Xiaochun Cao Qingming Huang
Abstract
success often relies on carefully curated, balanced datasets. In real-world scenarios, data often exhibits long-tailed distribution, where a few categories have abundant sam- ples while most categories contain only a small number of examples. Long-tailed learning focuses on effectively training models on such imbalanced datasets (Zhang et al., 2023; 2024a). Numerous approaches have been proposed to address this challenge, including remmann (Buda et al., 2018), re-balancing (Cui et al., 2019; Ren et al., 2020; Wang et al., 2023), representation learning (Zhu et al., 2022; Cui et al., 2024), ensemble learning (Wang et al., 2021; Zhang et al., 2022), and fine-tuning foundation models (Dong et al., 2023; Shi et al., 2024).
Real-world datasets often follow long-tailed dis- tribution, making generalization to tail classes dif- ficult. Recent methods resorted to long-tail vari- ants of Sharpness- Aware Minimization (SAM), such as ImbSAM and CC-SAM, to improve gen- eralization by flattening the loss landscape. How- ever, these attempts face a trade-off between com- putational efficiency and control over the loss landscape. On the one hand, ImbSAM is efficient but offers only coarse control as it excludes head classes from the SAM process. On the other hand, CC-SAM provides fine-grained control through glassddependent perturbations but at the cost of ef- ficiency due to multiple backpropagations. Seeing this dilemma, we introduce Focal-SAM, which assigns different penalties class-wise achieving fine-grained control without extra back- propagations, thus maintaining efficiency. Fur- thermore, we theoretically analyze Focal-SAM's generalization ability and derive a sharper gener- alization bound. Extensive experiments on both traditional and foundation models validate the ef- fectiveness of Focal-SAM.
Recently, Rangwani al. (2022) visualized the loss land- scape of different classes and observed that tail classes of- ten suffer from saddle points. Since the loss landscape is closely related to the generalization of modern neural net- works (Keskar et al., 2017; Jiang et al., 2020), they apply Sharpness-Awarn Minimization (SAM) (Foret et al., 2021) to help tail classes escape from saddle points. Later, since the original SAM operates on all classes, ImbSAM (Zhou et al., 2023a) excludes the head classes to better focus flat- tening the landscape of the tail classes. However, when com- bined with popular re- b-ballnccin methods (Cao et al., 2019; Kini et al., 2021; Menon et al., 2021). I), this coarse-grained approach often overemphasizes the tail classes, leading to poor head and overall performance. To achieve fine- fine-grained control, CC-SAM (Zhou et al., 2023b) uses class-dependent perturbation. However, the per-class perturbation requires least C' additional backpropagations, where C denotes the number of classes, making it rather computationally expen- sive. This raises a natural question: Can we design a method that achieves both fine-graine control and computational efficiency?
sharpness,
1. Introduction
In the past decades, deep learning has achieved remarkable success in various fields, including image classification (He et al., 2016), medical image processing (Ronneberger et al., 2015), and object detection (Ren et al., 2015). However, this
'Institute of Information Engineering. CAS "School of Cy- ber Security, University of Chinese Academy of Sciences Key Lab. of Intelligent Information Processing, Institute of Computing Tech., CAS "School of Computer Science and Tech., University of Chinese Academy of Sciences Artificial Intelligence Institute of China Electronics Technology Group Corporation, "School of Cyber Science and Tech., Shenzhen Campus of Sun Yat-sen University "BDKM, University of Chinese Academy of Sciences. Correspondence to: Qianqian Xu <xuqianqian ict.ac.en>>, Qing- ming Huang <qmhuang @ ac.n..
Targeting this goal. we integrate the focal mechanism (Lin et al., 2017) with SAM, inducing novel approach named Focal-SAM. Specifically, we introduce the focal sharpness term, which is defined as the weighted sum of wise sharpness, where the weights decrease in a focal-like man- ner from head to tail classes. On the one hand, Focal-SA. controls the flatness of different classes in a fine-grained way, better balancing the performance between head and tail classes than ImbSAM, as shown in Fig. (a). On the other
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Focal-SANe Focal Sharpness-Aware Minimization for Long-Tailed Classification
Fine-grained Control (vs. ImbSAM)
(b)
Efficiency (vs. CC-SAM)
ImbSAM
Focal-SAM (Ours)
Ours
DiET Ls
max,
E El mx)
1.0
10
80-
CIFAR1O-LT
Penrlly
Truncated
Penally
Class-specific Weights
Ours
CIFAR1DO
Weights
70-
ImageNet-LT
Naluralist
Head
Head
Toil
Focal-SAM
60-
CC-SAM
Ours
Ours
50-
500
1000
1500
xSharp Head
Flat Tail
Flat Head
Fiat Tall
Training Speed (Images/s)
Figure 1: (a) ImbSAM applies the sharpness penalty only to tail classes, leading sharp loss landscape for head classes. In contrast, Focal-SAM assigns class-specific weights to the sharpness penalty, resulting in smooth loss landscapes for both head and tail classes. (b) Focal-SAM replaces per-class perturbations in CC-SAM with class-specific sharpness penalties, significantly enhancing computational efficiency while achieving better performance.
hand, Focal-SAN AMM replaces the per-class perturbations in CC- SAM with per-class sharpness penalties, making it much more efficient than CC-SAM, as illustrated in [b). Fur- thermore, we provide an informative generalization bound based the PAC-Bayesian theory. This bound not only decreases at faster rate than those of SAM and CC-SAM (Ó(1/)) vs. 0(1/v/ where is the number of training samples) but also demonstrates the influence of the hyper- parameters and trace of the Hessian.
We propose Focal-SAM, a simple yet effective method that provides fine-grate -e-iggrined control of loss landscape and maintains computational efficiency. Theoretical analysis further offers a sharp generalization bound of Focal-SAM
Extensive experiments validate the effectiveness of the proposed Focal-SAM, ranging from training ResNet mod- els from scratch to fine-tuning foundation models.
Finally, we conduct extensive experiments on various bench- mark datasets validate the effecttivneess of SAM, including training ResNet models from scratch and fine- tuning the foundation model CLIP (Radford et al., 2021). The results show that Focal-SAM consistently outperforms other baaee methods across multiple datasets and mod- els in long-tailed recognition tasks. Prior arts (Zhou et al., 2022; Khattak et al., 2023; Park et al., 2024) have demon- strated that fine-tuning CLIP often performs well on the target domain but struggles with domain shifts. Therefore, also assess model performance OOD test sets when fine-tuning foundation models, referred to as long-tailed domain generalization tasks. The results indicate that Focal- SAM improves performance by approximately 0.5%~44.% when combined with baselines on OOD test sets. These further suggest that Focal-SAM can enhance generalization, leading to better performance under domain shifts.
2. Related Work
2.1. Long-Tailed Learning
Several approaches address long-tailed learning challenges, such as re-sampling (Buda et al., 2018; Wang et al., 2019b; Liu et al., 2022), re-balancing (Cui et al., 2019; Ren et al., 2020; Wang et al., 2023; 2022; Han et al., 2024; Hou et al.,
2022; Lyu et al., 2025; Yang et al., 2023b;a; 2022; Zhao et al., 2024a; Dai et al., 2023; Shao al., 2023; Hong et al., 2024), data augmentation (Kim et al., 2020; Hong al., 2022; Ahn et al., 2023; Wang et al., 2024b;a), representa- tion learning (Cui et al., 2021; Zhu et al., 2022; Cui et al., 2024; Gao et al., 2023; Zhang et al., 2024b), ensemble learn- ing (Wang et al., 2021; Zhang et al., 2022; Li et al., 2022; Aimar et al., 2023; Yang et al., 2024; Zhao et al., 2024b), and fine-tuning foundation models (Dong et al., 2023; Shi et al., 2024). This paper focuses on loss modification, tech- nique that modifies the loss function to guide the model's attention towards tail classes, consequently improving their performance. Various methods have been proposed, such as LDAM (Cao et al., 2019), which enlarges the margin for tail classes to enhance their generalization performance. Cao et al. (2019) further introduce a training scheme called Deferred Re-weighting (DRW) used in conjunction with
In summary, our key contributions are follows:
Systematic studies illustrate the limitations of ImbSAM and CC-SAM. ImbSAM fails to flatten the loss landscape for head classes, while CC-SAM is highly computation- ally expensive.
2

---

Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
LDAM to improve model performance. However, Menon et al. (2021) argue that previous loss modification techniques sacrifice consistency in minimizing the balanced error. They propose the LA (Menon et al., 2021) loss, which introduces adjustments to the standard cross-entropy loss to ensure Fisher consistency for balanced error minimization. Build- ing on this work, the VS (Kini et al., 2021) loss further improves upon the LA loss by incorporating both additive and multiplicative adjustments, beneficial during the initial and terminal phases of training respectively. Most recently, Wang et al. (2023) provide comprehensive generalization analysis of these losses.
tal number of training samples. Let Doat denote the uni- form test distribution. Following prior work (Cao al., 2019; Hong et al., 2021), given class y, D and Doot share the same class-conditions distribution, denoted as Dy - P(ærly). We use My to represent the number of sam- ples in the y-th class and my = nnynn to denote the ratio of the y-th class in the training set. Without loss of generality, we assume n2 2 n2 2 ... 2 ncc with n1 >> nc.
The model parameters denoted by w. with total of k pa- rameters. The loss for sample (æ,, y) is defined as (ww; x. v). The training loss over dataset S is given by (s) & 1 Σi E(w;; T, U:)- Similarly, the loss specifically for samples from the y-th class within S is defined as (2(()) & n
n
In this paper, we leverage these loss functions while aiming to specifically improve their generalization ability for long- tailed classification tasks.
1 Σ Ey f(w;; Ti, Uz). We further define the expected loss
over D. Dbal and Dy as LD(W) - E ((as)) x, y),, Draai (w) A E( [f(w;; av,)] and LD, (w) EEDD, [,(w; æ. v)]. respectively. Our goal is to optimize pa- rameters w on dataset S such that L.D.. (w) is minimized, leading to good performance on the balanced test set.
2.2. Sharpness of Loss Landscape
Generalization in deep neural networks has always been a crucial focus in machine learning research. Recent stud- ies (Keskar et al., 2017; Jiang et al., 2020) have empirically and theoretically demonstrated that flatter minima in the loss landscape typically lead to better generalization. Inspired Minimization (SAM) (Foret et al.,
3.2. Limitations in ImbSAM and CC-SAM
ImbSAM. ImbSAM divides classes into head and tail groups denoted as H and T. It applies SAM only to the tail group to focus on flattening loss landscape for these classes. Its objective function is:
by this, Sharpness- 2021) is developed to find flatter minima, achieving superior performance across various tasks.
In the context of long-tailed learning, Rangwani al. (Rang- wani et al., 2022) suggest combining SAM with re- r-blalnniing techniques to help the model escape saddle points and im- prove generalization. Imbalanced SAM (ImbSAM) (Zhou et al., 2023a) incorporates class priors into SAM by di- viding classes into head and tail groups. It applies SAM exclusively to the tail classes while maintaining standard op- timization for head classes, aiming to specifically enhance the generalization of tail classes. Class-Conditional SAM (CC-SAM) (Zhou et al., 2023b) applies SAM to each class individually, using class-specific perturbation radii. These radii increase from head to tail classes, enabling fine-grained control over the loss landscape for each class.
L (w) & (2) + max L3 +e)
(1)
From (I),)), ImbSAM excludes all head classes from SAM. As a result, the loss landscape for head classes becomes sharper, which may reduce their generalization perfor- mance. To validate this, we analyze the spectral density of the Hessian H (Ghorbani et al., 2019), common mea- sure for the flatness of the loss landscape. We also consider two key metrics: the largest eigenvalue Xmaz and the trace Tr( Higher values of Amaz and Tr(H) generally indi- cate a sharper loss landscape. Following prior work (Rang- wani et al., 2022), we compute the eigen spectral density of the Hessian for head and tail classes on the CIFAR- 10 LT dataset using the VS loss function. The results are shown in Fig.2.
This work also extends the SAM framework for long-tailed classification. Our method aims achieve fine-grained con- trol over the loss landscape while maintaining computational efficiency.
A comparison between F(g. and Fig.2(f)) reveals that ImbSAM effectively reduces Tr(H) and Xmaz for the tail classes, suggesting a flatter loss landscape. However, when comparing Fig. 2(a) and F(b. we observe that with Imb- SAM. the values of Tr(H) and Amar for head classes are significantly higher. This indicates that ImbSAM's exclusion of head classes from SAM sharpens their loss landscape, potentially degrading their generalization perfor- mance.
3. Motivation
3.1. Problem Setup
We define the sample space as X' and the label space as y' = [1, .C.. In the long-tailed recognition task, the training set follows an imbalanced distribution DD and consists of data pairs denoted as S = {(x,, y:) Ji where
CC-SAM. CC-SAM applies SAM to each class individu- ally, using class-specific perturbation radii. The objective
y E 3 is the label for sample x, E X, and is the to-
3

---

Focal-SAMe Focal Sharpness-Aware Minimization for Loig-Tailed Classification
Density (Lag Scale)
10a-
Amaui 8.79
Density (tog Scale)
13.34 Tr(H)) 93.92
Demsity (Log Scale)
10a
Amasi 0.49
Density (Log Scale)
10a
Amuni B.
200.
THH]:: 52.09
109
100
Tr(H):
Tr(H): 58.77
10-2
10-2
10-2
10-2
10"4.
10"4
10"4
10-4
10-6
10
10-6
10-9
-10'-10¹ -1 0 1 Eigenvalue
10"
10"
-10"-10*
-1 1
10a
10
-10'-10'
-1 0
10
10
-10--10"
-1
0 1
10
10a
Eigenvalue
Eigenvalue
Eigenvalue
(a) SAM: Head Classes
(b) ImbSAM: Head Classes
(c) CC-SAM: Head Classes
(d) Focal-SAM: Head Classes
Density (Log Scale)
Anw: 33.61
Density (Log Scale)
Amari 30.09
Density (Log Scale)
Anari 32.43
Density (Log Scalej
Anar! 29.61 Tr(Hi: 189:51
100
T2(H): 22
200
Tr(H): 194.98
100
TK(H): 597.57
10-2
10-2
10-2
10-2
10
10-4
10-4
10-4
10"6
10-6
10-6
10-6
-10"-10'
-1 0
10
10
10"-10'
-1 1
20
10
-10"-10*
-1 0
10
10
-10--10'
-1
1
10
10'
Elgenvalue
Eigenvalue
Eigenvalue
Eigenvalue
(c) SAM: Tail Classes
(f) ImbSAM: Tail Classes
(g) CC-SAM: Tail Classes
(h) Focal-SAM: Tail Classes
Figure 2: Eigen Spectral Density of Hessian for head and tail classes of ResNet models trained with various SAM variants on CIFAR-10 LT using VS loss. A smaller Amaz and Tr(H) (H) generally indicate a flatter loss landscape.
Table 1: Average training time per epoch (in seconds) for different SAM variants across four long-tailed datasets using ResNet models. For CC-SAM, we follow its protocol by perturbing only the last few layers to improve its efficiency.
Methods
CIFAR-10 LT
CIFAR-100 LT
ImageNet-LT
iNaturalist
SAM
5.66s (1.00x)
4.81s (1.00x)
170.04s (1.00x)
831.67s (1.00x)
ImbSAM
7.80s (1.37x)
6.68s (1.39x)
293.11s (1.72x)
1088.61s (1.31
CC-SAM
11,61s (2.05x)
19.70s (0.10x)
1626.54s (9.57x)
12869.89s (15.47x)
Focal-SAM (Ours)
7.67s (1.36x)
6.71s (1.40x)
291.05s (1.71x)
1068.92s (1.29x)
function is defined as:
To confirm this, measure the average training time per epoch for various SAM variants across four datasets using ResNet models. For CC-SAM, we follow its protocol by perturbing only the last few layers to enhance efficiency. As shown in 1ab. despite perturbing fewer parameters, CC- SAM takes about 2~155x more time than SAM, depending on the dataset. The training time ratio of CC-SAM SAM grows with the number of classes in the batch. These high computational costs make CC-SAM particularly impractical for large-scale datasets or fine-tuning foundation models.
L (w) A max |elaspt m.
Lis(w +
(2)
The optimal perturbation (u( for each class is also class- wise and estimated as p (w)))( W L's(())) The model parameters are updated with the learning rate 7 as:
1 n
-
(wLis(W)) (w)
This fine-grained method flattens the loss landscape of head and tail classes more effectively, as shown in F(c. and Fig. However, CC-SAM much more computa- tionally demanding than SAM. According to E3), per parameters update requires computing the gradient for each class i's loss at w + ( ie.. Li (w))m++?((())) There- fore, CC-SAM requires at least C' backpropagations per update, whereas SAM only needs two. Thus, CC-SAM has a much higher computational cost than SAM. For details on the backpropagation requirements for SAM and ImbSAM, please see App.B.
4. Methodology
4.1. Focal Sharpness-Aware Minimization
Motivated by the analysis in Sec.3, we develop a new method called Focal-SAM. This approach achieves fine- grained control over the flatness between head and tail classes while maintaining computational efficiency, as shown in Tab. and Fig.2.
To this end, we first introduce the concept of class-wise
4

---

Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
8
25
180
1.5.
5
20
150
7
15
120
0.5
1.5.
10
100
0.4
0.6
0.8
1.0
0.80
0.85
0.90
0.95
1.00
0.92
0.94
0.96
0.98
1.00
0.97
0.98
0.99
1.00
(1 (1-R8
=
(1
=;)
(1
xz)
(1
(a) CIFAR-10 LT
(b) CIFAR-100 LT
ImageNet-LT
(d) iNaturalist
Figure 3: The probability density distributions of (1 mT)² for various y values on CIFAR-1O LT, CIFAR- 100 LT, ImageNet- LT, and iNturrllst.
sharpness, defined as the loss difference between the origi- nal model parameters w and the perturbed ones, to quantify the sharpmess of loss landscapes across different classes:
Using first-order Taylor expansion, approximate the solution of the inner maximization problem for L (ww):
V V V ((w) so argmax E (w.Li(() = P L. (w)ll ||||>SS
L w. c) ('s + c) ( . E
(4)
(7)
Next, we propose sharpness term called focal sharp- ness:
Then, we can substitute e and compute the gradients of L FS (w) to solve the outer minimization problem: S
L (w)) (w)
dellaS 11-=))is((..
(5)
ES (w) (w(Ls(w) + \ + É(w)) - [](w))) V L
where (1-m,)" is the focal weight that provides fine-grained control over class-wise sharpness, and y is a tunable hy- perparameter. When y increases, the distribution of focal weight (1 - ) will skew more to tail classes. Fig.3 illus- trates how the probability density distributions of (1 m,)" varies with respect y various long-tailed datasets.
(Ts(m)) XL)(w))) AV I. (,(()) w+
(8)
From (7) and (8), computing V (10) to update model parameters requires only three backpropagations: one for LL one for (Lg(Lww)) - w- and one for V. T. w+t(w)- Therefore, Focal-SAM is more computationally efficient than CC-SAM, making it more suitable for large-scale datasets fine-tuning foundation models.
Then, the objective of Focal-SAM is defined by the combi- nation of the training loss and the focal sharpness term:
5 (w) = (s(w) A 1.55(w) (w);
(6)
Overall, Alg.1 1 gives the pseudo-code to optimize the Focal- SAM objective, using SGD as the base optimizer.
where À is hyperparameter controlling the importance of focal sharpness. This formulation highlights how Focal- SAM overcomes ImbSAM's limitations. When and A = 1, (5)(5) penalizes the sharpness of each class equally, reverting to standard SAM. Conversely, when y sufficiently large, focal weights for head classes rapidly approach 0, while the weights for tail classes remain rel- atively large. In this scenario, Focal-SAM approximates ImbSAM. Typically, we select moderate 7,, such that the focal weights increase smoothly from head to tail classes. This fine-grained control over loss landscape improves the flatness of tail classes while maintaining that of head classes, ultimately enhancing generalization for both traditional and foundation models.
0
Algorithm -ooal--AAN algorithm
Input: Training set S, perturbation radius p. hyperparame-
ter X, learning rate
Output: Model trained with Focal-SAM
1: Initialize weights wp. t = O;
2: while not converged do Sample batch B {(x) (101,,)1... V/), (,,, (b)): Compute (ur); Compute V Vw and é(w) according to Eq.
6: Perturb w with é(wr), and compute gradient 91
V L (w))
7: Compute gradient g2 = V [[ - L, (w))
Update weights: w = w; - 1 + g22);
4.2. Optimizing the Focal-SAM Objective Function
t= + 1;
In this section, we discuss how to optimize the Focal-SAM
10: end while
objective S (w). Let (w)) A E mm)" (
5

---

Focal-SANe Focal Sharpness-Aware Minimization for Long-Tailed Classification
4.3. Generalization Ability of Focal-SAM
Focal-SAM enables more effective optimization of
TeDuac (w)) Specifically, we can reformulate Eq. to
Previous works have established the generalization bound for SAM (Foret et al., 2021) and CC-SAM (Zhou et al., 2023b). However, these bounds are relatively loose (with an order of nt) and could bias the training process. For example, the perturbation radius of CC-SAM (i.e., Pi in Eq.(2)) is set as the solution to minimizing its PAC-Ba -ay-ssasi bound. Since the generalization is not sharp enough, the estimated perturbation radius P3 could deviate from the optimal one, thus leading to inferior performance. In this section, we develop sharper generalization bound with order of 1/n for Focal-SAM.
D (w) + (II) S (I) + (III).
(10)
Typically, (II) tends to be large without SAM-based tech- niques. As a result, minimizing the right-hand side (RHS) of (q.( in such cases may not induce small D (w). In contrast, Focal-SAM reduces the trace tr [H ( (w)) by effectively flattening the loss landscape, leading to small (LI). This makes it more effective to minimize D (w) when we optimize the RHS of (10). This insight again validates the necessity of Focal-SAM.
We assume the loss function f(w; y) has an upper bound of B, which is a common and practical assumption. Then, we derive the following generalization bound based on the PAC-Bayesian theorem proposed in (Tolstikhin & Seldin, 2013). For conciseness, we present an informal formula- tion in the main content, leaving the formal one and the corresponding proof in App.A.
5. Experiments
This section evaluates the effectiveness of Focal-SAM through a series of experiments. Detailed experimental settings and additional results are provided in App.C and App.D due to space constraints.
Theorem 4.1 (Informal). Assume that ((, y) E D {(w; x. y) S B. For any p > O, any uniform distribution Doos and any distribution D, with high probability over the choice of the training set S D.
5.1. Experiment Protocols
Datasets. We use four widely adopted long-tailed datasets for long-tailed recognition tasks: CIFAR-10 LT (Cao et al., 2019), CIFAR-ION 100 LT (Cao et al., 2019), ImageNet-LTT (Liu et al., 2019) and iNaturalist (Horn al., 2018). The CIFAR- LT datasets include variants with imbalance ratios of (200, 100, 50, 10). In addition to evaluating model performance on ID test sets, we also assess it on OOD test sets, referred to as long-tailed domain generalization tasks. Specifically, we train the model on ImageNet-LT and evaluate on three OOD datasets: -ageeNet-Sketch (Wang et al., 2019a), Ima- geNetV2 (Recht et al., 2019), and ImageNet-C (Hendrycks & Dietterich, 2019). For more details, see App.C.1.
21. (w))
-0
Xp2
I.Doat (w) S
tt(H(())
CτC
+ In(n)
(II)
> log w ll3/p2) + W
+ Ō
(1I1)
where n = (S), V A parameters, H(w) represents the Hessian matrix of LbB (w) at point and tr()) represents the matrix trace.
T,, ,, k is the number of
Competitors. When training ResNet models on the CIFAR- LT dataset, we assess several loss functions. These meth- ods are further combined with SAM (Foret et al., 2021), I), ImbSAM (Zhou et al., 2023a), and CC-SAM (Zhou et al., 2023b) as baselines. For the ImageNet-LT and iNatural- ist datasets, we employ a range of representative meth- ods baseline methods. When fine-tuning the foundation model CLIP (Radford et al., 2021), we evaluate both full fine-tuning with LA loss (denoted as FFT) and parameter- efficient fine-tuning using the LIFT method (Shi et al., 2024), along with their performance when combined with different SAM variants. For more details, please refer to App.C.2.
From the theorem, we have the following insights:
The generalization bound consists of three components. Specifically, (I) is the empirical loss on the training set L (w), which can be minimized via large-scale mod- els. (II) reveals how the generalization performance is affected by multiple factors, including X. p. (H))) (IIT) decreases at faster rate of (11m).
The hyperparameters and y play a crucial role. On the one hand, a larger À can increase both components (LI) and (III) of the bound. Therefore, careful tuning of X can induce tighter bound. On the other hand, larger y leads to smaller V also leading to a tighter bound. This suggests that assigning greater weights to the sharpness of the tail classes can improve the overall generalization ability.
Evaluation Protocol. For long-tailed recognition tasks, we assess model performance using balanced accuracy (Menon et al., 2021). )). To provide deeper insights, we split the classes into three groups: Head, Medium, and Tail, and report ac- curacy for each group individually. For long-tailed domain generalization tasks, we evaluate performance on OOD bal- anced test sets, including top-1 accuracy and accuracy for
6

---

Focal-SANe Focal Sharpness-Aware Minimization for Long-Tailed Classification
Table 2: Performance comparison on CIFAR-100 LT datasets with various imbalance ratios (IR). FFT FFT denotes fully fine- tuning the foundation model with LA loss. Due to space limitations, additional CIFAR-100 LT results combining more methods, as well as the CIFAR-10 LT results, are shown in Tab.6 and Tab.5.
IR 100
IR200
IR50
IR10
Method
Head
Med
Tail
All
All
All
All
Training from scratch
CE
69.2
41.6
9.0
41.5
37.5
45.6
58.1
CE+SAM
72.7
41.8
7.0
42.2
38.9
46.8
59.7
CE+ImbSAM
68.5
46.0
9.6
43.0
38.7
47.8
60.1
CE+CC-SAM
70.1
44.2
9.0
42.7
39.1
47.4
60.0
CE+Focal-SAM
73.8
44.2
8.9
44.0
39.6
48.1
60.9
LA (Menon al., 2021)
61.3
42.3
28.6
44.9
41.8
50.3
59.4
LA+SAM
63.1
52.2
32.2
50.0
45.5
52.8
62.6
LA+ImbSAM
57.4
51.1
31.0
47.3
43.4
52.2
62.4
LA+CC-SAM
63.7
51.9
32.3
50.1
45.6
53.0
63.0
SA+
63.9
53.0
32.5
50.7
46.0
54.5
63.8
Fine-tuning foundation model
FFT
88.2
79.3
66.1
78.5
76.3
81.2
85.5
FFT+SAM
87.9
82.5
70.8
80.9
77.7
83.4
86.8
FFT+ImbSAM
87.5
82.0
70.2
80.4
77.2
81.9
86.7
FFT+CC-SAM
87.8
82.9
70.9
81.0
78.2
83.5
87.0
FFT+Focal-SAM
88.1
82.8
72.4
81.6
79.0
83.9
87.3
LIFT (Shi et al., 2024)
85.3
81.1
79.2
82.0
79.6
82.8
85,0
LIFT+SAM
85.0
81.5
79.4
82.1
79.6
83.0
85.1
LIFT+ImbSAM
84.7
81.9
78.9
82.0
79.8
83.1
85.2
LIFT+CC-SAM
84.8
81.8
79.0
82.0
79.7
83.1
85.2
LIFT+Focal-SAM 85.4
81.9
79.4
82.4
80.0
83.2
85.4
each class group. For more details of the evaluation protocol, please refer to App.C.3.
or surpassing ImbSAM on tail classes. Additionally, Focal- SAM generally outperforms CC-SAM on both head and tail classes, showing its ability to achieve a finer balance between head and tail classes performance.
Implementation Details. For CIFAR-LT datasets, we train ResNet models using ResNet-32 (He et al., 2016) as the backbone. For ImageNet-LT and iNaturalist datasets, we employ ResNet-50 (He et al., 2016). Training is conducted for 200 epochs. For fine-tuning foundation models, we follow the protocols outlined in LIFT (Shi et al., 2024). Specifically, we fine-tune the image encoder of CLIP (Rad- ford et al., 2021) with a ViT-B/16 (Dosovitskiy et al., 2021) backbone. The training lasts for 20 epochs. For further implementation details, please refer App.C.4.
Tab.3 presents results on the larger ImageNet-LT and iNatu- ralist datasets. Combining the baseline LA with Focal-SAM improves performance by approximately 1 1.9% when training ResNet models. Similarly, pairing the baseline FFT or LIFT with Focal-SAM yields a performance gain of 0.3%~2.49% when fine-tuning foundation models, outper- forming several competitors.
5.3. Long-tailed Domain Generalization
5.2. Performance Comparison
In Tab.4, we evaluate the model trained on the ImageNet-LT dataset across three OOD datasets. The results show the following: 1) SAM-based methods, when combined with performance gain
Tab.2 summarizes the experimental results on the CIFAR-LT datasets with different imbalance ratios. From these results, have the following observations: 1) Focal-SAM consis- tently performs better than SAM, ImbSAM, and CC-SAM across various loss functions. 2) Focal-SAM significantly outperforms ImbSAM on head classes, while maintaining
FFT or LIFT, generally achieve OOD datasets than the ID dataset (ImageNet-LT). This observation aligns with prior studies (Zhou et al., 2022; Khattak et al., 2023; Park et al., 2024), which suggest that

---

Focal-SANe Focal Sharpness-Aware Minimization for Long-Tailed Classification
Table 3: Performance comparison on ImageNet-LT and iNaturalist. The results for methods marked with f are taken from the original paper. "_" indicates that the original paper didn't 't report the corresponding results.
ImageNet-LT
iNaturalist Med
Method
Head
Med
Tail
All
Head
Tail
All
Training from scratch
CB (Cui et al., 2019)
39.6
32.7
16.8
33.2
53.4
54.8
53.2
54.0
cTT (Kang et al., 2020) y
61.8
46.2
27.3
49.6
69.0
66.0
63.2
65.2
DiVE (He et al., 2021)
64.1
50.4
30.7
49.4
70.6
70.0
67.6
69.1
DRO-LT (Samuel & Chechik, 2021)
64.0
49.8
33.1
53.5
69.7
DisAlign (Zhang et al., 2021)
61.3
52.2
31.4
52.9
69.0
71.1
70.2
70.6
WB (Alshammari et al., 2022)
62.5
50.4
41.5
53.9
71.2
70.4
69.7
70.2
CC-SAM (Zhou et al., 2023b) f
61.4
49.5
37.1
52.4
65.4
70.9
72.2
70.9
LA (Menon et al., 2021)
62.8
49.0
31.8
52.0
68.4
69.4
69.2
69.2
LA+SAM
63.1
51.7
33.1
53.6
68.3
70.8
71.9
71.0
LA+ImbSAM
62.6
50.3
32.6
52.6
68.0
70.2
70.2
69.9
LA+Focal-SAM
63.9
52.2
34.4
54.3
68.4
72.0
72.5
71.8
Vineituning foundation model
Decoder (Wang et al., 2024c) +
73.2
59.2
LPT (Dong et al., 2023) t
79.3
76.1
FFT
79.9
70.5
51.0
71.5
69.7
71.9
71.7
71.6
FFT+SAM
80.9
72.9
54.3
73.5
69.5
74.4
74.4
73.8
AFT+ImSSAM
80.6
72.6
52.2
72.9
68.5
73.4
73.8
73.1
FFT+CC-SAM
80.6
73.6
54.2
73.6
69.2
74.1
74.2
73.6
FFT+Focal-SAM
80.8
73.9
54.4
73.9
69.1
74.7
74.3
74.0
LIFT (Shi et al., 2024)
79.7
76.2
72.8
77.1
74.1
79.4
81.5
79.7
LIFT+SAM
79.9
76.4
72.7
77.2
73.5
79.7
81.6
79.8
LIFT+ImbSAM
79.8
76.4
72.5
77.2
73.2
79.5
81.4
79.6
LIFT+CC-SAM
79.8
76.4
73.3
77.3
74.0
79.4
81.5
79.7
LIFT+Focal-SAM
79.7
76.6
73.6
77.4
73.9
79.8
81.7
80.0
fine-tuning foundation models often perform well target (ID) datasets but struggles with unseen (OOD) datasets. 2) Focal-SAM achieves a performance improvement of 0.5% to 4.3%, surpassing SAM, ImbSAM, and CC-SAM. This is because Focal-SAM effectively enhances the model's generalization ability by flattening the loss landscape, which mitigates performance issues on OOD test sets.
of improving CC-SAM's efficiency.
5.5. Sharpness of Loss Landscape for Focal-SAM
To examine the impact of Focal-SAM on the loss landscape. Fig.2(d) and Fig. 2ih) show the eigenvalue spectrum of Hes- sian for head and tail classes of models trained with Focal- SAM CIFAR-10 LT using the VS loss function. Com- paring F(e. and Fig.2(h), the trace Tr(H) and the max- imum eigenvalue Amaz for tail classes in Focal-SAM are significantly lower than those SAM. Similarly, Fig.2(b) and Fig.2(d) reveal that Tr(H) and Xmar for head classes in Focal-SAM much smaller than those ImbSAM. These results suggest that Focal-SAM achieves fine- the-grained bal- ance in the flatness between head and tail classes.
5.4. Training Speed of Focal-SAM
To assess the computational efficiency of Focal-SAN we evaluate the training time per epoch across various long- tailed datasets, shown in Tab.1. Focal-SAM requires about 50% more running time than SAM and has similar running time to ImbSAM. Given that our method consis- tently outperforms SAM and ImbSAM, thus the compu- lational cost is acceptable for the performance gain. Fur- thermore, Focal-SAM is significantly faster than CC-SAM while delivering better performance, aligning with our goal
5.6. Ablation Study About and À
We analyze the influence of hyperparameters y and À to Focal-SAM the CIFAR-LT datasets.
8

---

Focal-SANe Focal Sharpness-Aware Minimization for Long-Tailed Classification
Table 4: Performance comparison for domain generalization. The source models are trained on the ImageNet-LT dataset and evaluated on -ftt--i-itiiiutioon datasets, including ImageNet-Sketch, ImageNetV2, and ImageNet-C.
SoggeNet-Skettle
ImageNetV2
ImageNet-C
Method
Head
Med
Tail
All
Head
Med
Tail
All
Head
Med
Tail
All
FFT
42.9
35.5
21.4
36.4
70.1
60.2
45.2
62.0
50.3
41.4
26.1
42.8
FFT+SAM
44.9
39.3
26.1
39.6
71.2
62.6
48.0
63.9
52.5
44.6
29.3
45.6
FFT+ImbSAM
45.2
39.5
24.8
39.7
71.0
62.2
46.5
63.5
52.0
44.7
28.3
45.2
FFT+CC-SAM
45.0
41.0
26.8
40.6
71.3
63.2
48.4
64.3
52.0
45.1
29.4
45,6
FFT+Focal-SAM
45.5
41.2
27.3
41.0
71.8
63.6
48.8
64.8
52.6
45.5
29.8
46.1
LIFT (Shi et al., 2024)
46.4
43.3
45.7
44.8
70.4
65.9
64.7
67.5
52.6
48.7
47.3
50.0
LIFT+SAM
46.9
43.5
46.4
45.2
70.4
66.0
65.5
67.6
52.9
49.2
48.1
50.5
LIFT+IbSAM
46.4
43.5
46.0
44.9
70.0
66.2
65.5
67.6
52.6
49.0
47.7
50.2
LIFT+CC-SAM
46.8
44.1
47.6
45.6
70.4
66.2
65.4
67.7
53.0
49.7
49.2
50.9
LIFT+Focal-SAM
46.9
44.7
49.4
46.2
70.0
66.8
66.9
68.0
53.1
49.9
49.8
51.1
50-
Accuracy (96
Accuracy (56)
48-
82
Accuracy (9)
Accuracy
81
48
80
LA
47
LDAM-DRW
VS
48
6.
50
1000
0.0
0.2
55
03
20
(a) CIFAR-IO
(b) CIFAR-100 LT
(a) CIFAR-IOLT
(b) CIFAR-100 LT
Figure 4: Ablation Study of Focal-SAM w.r.f
Figure Ablation Study of Focal-SAM W.E
Impact of 7: Fig.4 explores the effect of y. As increases, performance initially improves, suggesting that assigning greater weight to the class-wise sharpness of tail classes benefits performance. However, further increase in y leads to declining accuracy, indicating that assigning excessive weight to the class-wise sharpness of tail classes can harm performance.
Focal-SAM's generalization ability, deriving a tighter bound. Extensive experiments validate DAMalsSAM's effectiveness.
A Acknowledgements
This work was supported in part by the Fundamental Research Funds for the Central Universities, in part by the National Key R&D Program of China under Grant 201 8AAA0102000, in part by National Natural Science Foundation of China: 62236008, 62441232, U21B2038, U23B2051, 62122072 62025604, 62441619, 62206264 and 92370102, in part by Youth Innovation Promotion Association CAS, in part by the Strategic Priority Re- search Program of the Chinese Academy of Sciences, Grant XoBXD606000.. in part by the China National Postdoctoral Program for Innovative Talents under Grant BX20240384.
Impact of ): Fig.5 investigates the effect of As À in- creases, accuracy initially improves but subsequently de- creases. This indicates trade-off between minimizing the training loss and minimizing the sharpness of the loss land- scape.
6. Conclusion
This paper examines the limitations of ImbSAM and CC- SAM in long-tailed learning. ImbSAM excludes all head classes from SAM, often overemphasizing tail classes when combined with rebalancing methods. CC-SAM's per-class perturbation strategy provides fine-grained control over the loss landscape but is computationally costly. To address these issues, we propose Focal-SAM. method that effi- ciently balances loss landscape flatness between head and tail classes. Additionally, we offer theoretical analysis of
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences our work, none which we feel must be specifically highlighted here.

---

Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
References
N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Ahn, S., Ko, J., and Yun, S. CUDA: curriculum of data augmentation for taiged recognition. In International Conference on Learning Representations, 2023.
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
Aimar, E. S., Jonnarth, A., Felsberg, M., and Kuhlmann, M. Balanced product of calibrated experts for long-tailed recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19967-19977, 2023.
Gao, P., Xu, Q. Wen, P., Yang, Z., Shao, H., and Huang, Q. Feature directions matter: Long-tailed learning via rotated balanced representation. In International Conference on Machine Learning, pp. 27542-27555 2023.
Alshammari, S., Wang, Y., Ramanan, D., and Kong, S. Long-tailed recognition via weight balancing. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6887-6897, 2022.
Ghorbani, B., Krishnan, S., and Xiao, Y. An investigation into neural net optimization via hessian eigenvalue den- sity. In International Conference on Machine Learning, pp. 2232-2222222 2241, 2019.
Bernstein, S. On a modification of chebyshev's inequality and of the error formula of laplace. Ann. Sci. Inst. Sav. Ukraine, Sect. Math, pp. 38-38, 1924.
Buda, M., Maki, A., and Mazurowski, M. A. systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, pp. 249-259, 2018.
Han, B., Xu, Q.. Yang, Z., Bao, S., Wen, P., Jiang, Y., and Huang, Q. Aucseg: Auc-oriented pixel-level long-tail semantic segmentation. In Annual Conference on Neural Information Processing Systems, 2024.
Cao, K., Wei, C.,, Gaidon, A., Aréchiga, N., and Ma, T. Learning imbalanced datasets with label-distribution- aware margin loss. In Annual Conference Neural Information Processing Systems, pp. 1565-1576, 2019.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Cui, J., Zhong, Z., Liu, S., Yu. B., and Jia, J. Parametric con- trastive learning. In IEEE/CVF International Conference Computer Vision, pp. 695 705-704,227 2021.
He, Y., Wu, J., and Wei, X. Distilling virtual examples for long-tailed recognition. In IEEE/CVF International Conference on Computer Vision, pp. 235-244, 2021.
Cui, J., Zhong, Z., Tian, Z., Liu, S., Yu, B., and Jia, Generalized parametric contrastive learning. IEEE Trans. Pattern Anal. Mach. Intell., pp. 7463-7474, 2024.
Hendrycks, D. and Dietterich, T. G. Benchmarking neu- ral network robustness to common corruptions and per- turbations. In International Conference on Learning Representations, 2019.
Cui. Y., Jia, M., Lin, T., Song, Y., and Belongie, S. J. Class-balanced loss based on effective number of samples. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 9268-9277, 2019.
Hong, F., Yao, J., Lyu, Y., Zhou, 7., Tsang, I. W., Zhang, Y., and Wang, Y. On harmonizing implicit subpopulations. In International Conference on Learning Representations, 2024.
Dai, S., Xu, Q., Yang, Z., Cao, X., and Huang, Q. DRAUC: an -iiee distributionally robust AUC optimiza- tion framework. In Annual Conference on Neural Information Processing Systems, 2023.
Hong, Y., Han, S., Choi, K., Seo, S., Kim, B., and Chang, Disentangling label distribution for long-tailed visual recognition. In IEBE Conference on Computer Vision and Pattern Recognition, pp. 6626 6636, 2021.
Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 248 225, 2009.
Hong, Y., Zhang, J., Sun, Z., and Yan, K. Safa: Sample- adaptive feature augmentation for long-tailed image clas- sification. In European Conference on Computer Vision, pp. 587-603, 2022.
Dong, B., Zhou, P., Yan, S., and Zuo, W. LPT: long-tailed prompt tuning for image classification. In International Conference on Learning Representations, 2023.
Horn, G. V., Aodha, O. M., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S. J. The inaturalist species classification and detection dataset. In IEEE Conference on Computer Vision and Pattern Recognition, pp. $769-8778, 2018.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenbom, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby,
10

---

Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
Hou, W., Xu, Q., Yang. Z., Bao, S., He, Y., and Huang, Q- Adauc: End-to-end adversarial AUC optimization against long-tail problems. In International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 8903-8925, 2022.
Liu, B., Li, H., Kang, H., Hua, G., and Vasconcelos, N. Breadcrumbs: Adversarial lass-balanced sampling for long-tailed recognition. In European Conference on Computer Vision, pp. 637-653, 2022.
Liu, Z., Miao, Z., Zhan, X., Wang, J., Gong, B., and Yu, S. X. Large-scale long-tailed recognition in an open world. IEEE Conference on Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. Fantastic generalization measures and where to find them. In International Conference on Learning Representations, 2020.
Lyu, X., Xu, Q., Yang, Z., Lyu, S., and Huang, Q- SSE- SAM: balancing head and tail classes gradually through stage-wise SAM. In Association for the Advancement of Artificial Intelligence, pp. 19278-19286, 2025.
Kang, B., Xie, S., Rohrbach, M., Yan, Z., Gordo, A., Feng, J.. and Kalantidis, Y. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2020.
Menon, A. K., Jayasumana, S., Rawat, A. S., Jain, H., Veit, A., and Kumar, S. Long-tail learning via logit adjustment. In International Conference on Learning Representations, 2021.
Keskar, N. S., Mudigere. D., Nocedal, J., Smelyanskiy, M.. and Tang, P. T. P. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017.
Park, J., Ko, J., and Kim, J. Prompt learning via meta- regularization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26930-26940, 2024.
Khattak, M. U., Wasim, S. T., Naseer, M., Khan, S., Yang, M., and Khan, F. S. Self-regulating prompts: Foundational model adaptation without for- getting. In IEEE/CVF International Conference on Computer Vision, pp. 15144 15154, 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transfer- able visual models from natural language supervision. In International Conference Machine Learning, pp.
Kim, J., Jeong, J., and Shin, J. M2m: Imbal- anced classification via major-to to-iinor translation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13893-139022 2020.
8748 8763, 2021.
Rangwani, H., Aithal, S. K., Mishra, M., and R. V. B. Escaping saddle points for effective generalization class-imbalanee data. In Annual Conference on Neural Information Processing Systems, pp. 22791- 22805, 2022.
Kini, G. R., Paraskevas, O., Oymak, S., and Thrampoulidis, C. Label- iabblliibbllnced and group-sensitive classifica- tion under overparameterization.. In Annual Conference on Neural Information Processing Systems, pp. 18970- 18983, 2021.
Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do ima- genet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389-5400, 2019.
Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Master's 's thesis, Department of Computer Science, University of Toronto, 2009.
Ren, J., Yu, C., Sheng, S., Ma, X., Zhao, H., Yi, S., and Li, H. Balanced meta-softmax for long-tailed visual recognition. In Annual Conference on Neural Information Processing Systems, pp. 4175-4186, 2020.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Annual Conference on Neural Information Processing Systems, pp. 1 1106-1 14, 2012.
Ren, S., He, K., Girshick, R. B., and Sun, Faster R-CNN: towards real-time object detection with region proposal networks. In Annual Conference on Neural Information Processing Systems, pp. 91-99, 2015.
Li, J., Tan, Z., Wan, J., Lei, Z., and Guo, G. Nested col- laborative learning for loig-tailed visual recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6939-6948, 2022.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Convo- lutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention, volume 9351, pp. 234-241, 2015.
Lin, T., Goyal, P., Girshick, R. B., He, K., and Dollár, P. Fo- cal loss for dense object detection. In IEEE International Conference on Computer Vision, pp. 2999-3007, 2017.
Samuel, D. and Chechik, G. Distributional robustness loss for long-tail learning. In IEEE/CVF International Conference on Computer Vision, pp. 9475-9484, 2021.

---

Focal-SAM: Focal Sharpness-Aware Minimization for Loig-Tailed Classification
Shao, H., Xu, Q. Yang, Z.,, Wen, P., Gao, P., and Huang, Q Weighted ROC curve in cost space: Extending AUC to cost-sensitive learning. In Annual Conference on Neural Information Processing Systems, 2023.
Yang, Z., Xu, Q. Bao, S., Cao, X., and Huang, Q. Learning with multiclass AUC: theory and algorithms. IEEE Trans. Pattern Anal. Mach. Intell., 44(11):7747-7765 2022.
Yang, Z., Xu, Q., Bao, S., Wen, P., He, Y. Cao, X., and Huang. Q. Auc-oriented domain adaptation: From theory to algorithm. IEEE Trans. Pattern Anal. Mach. Intell., 45 (12):14161-144) 2023a.
Shi, J., Wei, T., Zhou, 7., Shao, J., Han, X., and Li, Y. Long- tail learning with foundation model: Heavy fine-tuning hurts. In International Conference on Machine Learning, pp. 45014 45039, 2024.
Yang, Z., Xu, Q., Hou, W. Bao, S., He, Y., Cao, X., and Huang. Q Revisiting auc-oriented adversarial training with loss-agnostic perturbations. IEEE Trans. Pattern Anal. Mach. Intell., 45(12) 15511, 2023b.
Tolstikhin, I. O. and Seldin, Y. Pac-bayes-empirrial- bernstein inequality. In Annual Conference on Neural Information Processing Systems, pp. 109-117, 2013.
Yang, Z., Xu, Q. Wang, Z., Li, S., Han, B., Bao, S., Cao, X., and Huang, Q. Harnessing hierarchical label distri- bution variations in test agnostic long-tail recognition. In International Conference Machine Learning, pp. 16624-56664 2024.
Wang, B., Wang, P., Xu, W., Wang, X., Zhang, Y., Wang. K., and Wang, Y. Kill two birds with one stone: Rethink- ing data augmentation for deep long-tailed learning. In International Conference on Learning Representations, 2024a.
Zhang, C., Almpanidis, G., Fan, G., Deng, B., Zhang, Y., Liu, J., Kamel, A., Soda, P., and Gama, J. A systematic review on long-tailed learning. CoRR, abs/2408.0043 2024a.
Wang, H., Ge, S., Lipton, Z. C., and Xing, E. P. Learning robust global representations by penalizing local predic- tive power. In Annual Conference on Neural Information Processing Systems, 2019a.
Zhang, S., Li, Z., Yan, S., He, X., and Sun, J. Distribu- tion alignment: A unified framework for long-tail visual recognition. In IEEE Conference Computer Vision and Pattern Recognition, pp. 2361-237677 2021.
Wang, P., Zhao, Z. Wen, H., Wang, F., Wang, B., Zhang, Q., and Wang, Y. Llm-autoda: Large language model-driven automatic data augmentation for long-tailed problems. In Annual Conference on Neural Information Processing Systems, 2024b.
Zhang, T., Zheng, H., Yao, J., Wang. X., Zhou, M., Zhang, Y., and Wang. Y. Long-tailes diffusion models with oriented calibration. In International Conference on Learning Representations, 2024b.
Wang, X.,
Lian, L.,, Miao, Z., Liu, Z., and Yu, S. X. Long-tailed recognition by routing diverse distribution- aware experts. In International Conference on Learning Representations, 2021.
Zhang, Y., Hooi, B., Hong, L., and Feng, J. Self- supervised aggregation of diverse experts for test-agnostic long-tailed recognition. In Annual Conference on Neural Information Processing Systems, pp. 34077- 34090, 2022.
Wang, Y., Gan, W., Yang, J., Wu, W., and Yan, J. Dy- namic curriculum learning for imbalanced data clas- sification. In JEEE/CVF International Conference on Computer Vision, pp. 5016-5025, 2019b.
Zhang. Y., Kang, B., Hooi, B., Yan, S., and Feng, J. Deep long-tailed learning: A survey. IEEE Trans. Pattern Anal. Mach. Intell., pp. 10795-10816 2023.
Wang. Y., Yu, Z., Wang, J., Heng, Q. Chen, H., Ye, W., Xie, R., Xie, X., and Zhang, S. Exploring vision-language models for imbalanced learning. Int. J. Comput. Vis., pp. 224 237, 2024c.
Zhao, Z., Wang, P., Wen, H., Xu, W., Lai, S., Zhang, Q. and Wang, Y. Two fists, one heart: Multi-objective opti- mization based strategy fusion for long-tailed learning. In International Conference Machine Learning, 2024a.
Wang, Z., Xu, Q.. Yang, Z., He, Y., Cao, X., and Huang, Q. Openauc: Towards auc-oriented open-set recognition. In Annual Conference on Neural Information Processing Systems, 2022.
Zhao, Z., Wen, H., Wang. Z., Wang, P., Wang, F., Lai, S., Zhang, Q., and Wang, Y. Breaking long-tailed learning bottlenecks: controllable paradigm with hypernetwork- generated diverse experts. In Annual Conference on Neural Information Processing Systems, 2024b.
Wang, Z., Xu, Q., Yang, Z., He, Y., Cao, X., and Huang, Q. A unified generalization analysis of re- re-weighting and logit-adjin adjustemtttt for imbalanced learning. In Annual Conference on Neural Information Processing Systems, pp. 14841748433 2023.
Zhong, Z., Cui, J., Liu, S., and Jia, J. Improving calibra- tion for long-tailed recognition. In IEEE Conference on
12

---

Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
Computer Vision and Pattern Recognition, pp. 16489- 16498, 2021.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Condi- tional prompt learning for vision- vasoun-language models. In IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pp. 16795-16804, 2022.
Zhou, Y., Qu, Y., Xu, X., and Shen, H. Imbsam: A
closer look at sharpness-aware minimization class- imbalanced recognition. In IEEE/CVF International Conference Computer Vision, pp. 11311-11321, 2023a.
Zhou, Z., Li, L.,, Zhao, P. Heng, P., and Gong, W. Class-conditional sharpness-awars minimization for deep long-tailed recognition. In JEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3499- 3509, 2023b.
Zhu, J., Wang, Z., Chen, J., Chen, Y. and Jiang, Bal- anced contrastive learning for long-tailed visual recogni- tion. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6898-6907, 2022.
13

---

Focal-SAM: Focal Sharpness-Aware Minimization for LongeTailed Classification
Appendix
Contents
A Proof of Theorem
15
A.1 Framework of the Proof
15
A.2 Proof of Lem.A.2
15
A.3 Proof of Lem.A.3 and Lem.A.4
16
A.4 Proof of Thm.4.1
20
Analysis of Backpropagation Requirements for SAM and ImbSAM
20
B.1 Backpropagation Requirements for SAM
20
B.2 Backpropagation Requirements for ImbSAM
21
More Experiment Protocols
21
Datasets
C.2 Competitors
21
C.3 Evaluation Protocol
22
C.4 Implementation Details
22
C.5 Experimental Hardware Setup
22
More Experiment Results
23
D.I Additional Results on CIFAR-LT Datasets
23
D.2 CE and mCE Metrics on ImageNet-C for Long-tailed tailed Domain Generalization
24
D.3 Results for Aligning Computational Cost
24
D.4 Visualization of Loss Landscape
25
D.5 Ablation Study About Perturbation Radius p
26
D.6 Additional Results for Eigen Spectral Density of Hessian
27
14

---

Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
A. Proof of Theorem
A.I. Framework of the Proof
Goal. To bound the balanced loss Drai (w) using our objective loss:
F (w) - (Lw - A [3(ww) + max ( +e) L
(11)
eellss
(a)
(b)
Framework of the proof.
1. Essentially, the generalization bound describes how empirical values (LS (w)) deviate from the expected one (L (w)). To bound such deviations, Bernstein's inequality (Bernstein, 1924) and PAC-Bayesian theorem (Tolstikhin & Seldin, 2013) are convenient tools. Notice that these tools require the empirical values to be sampled i.i.d. from the distribution on which the expectation is based. Since the training set ~ D, we first transform the distribution from Doot to D building on the work of Wang et al. (2023), i.e.,
L Diat (w) S (D))
split inko
(w) - X (7)
+ L (w)
(12)
(c)
(d)
Get (c) $ (a) via Bernstein's inequality (Lem.A.2).
Get (d) $ (b):
Bound (d) using a intermediate value (w +)] via Taylor expansion (Lem.A.4).
Bound E2 [4][ (w + E)] by (b) via PAC Bayesian bound (Lem.A.3).
Combine all, we get Thm.A.5 as follow:
L Dout (w) S (c) (d)
+ (b) L5 (ww)
(13)
A.2. Proof of Lem.A.2
We begin by introducing Bernstein's inequality to prove the first part.
Lemma A.1 (Bernstein's 's Inequality (Bernstein, 1924)). Let X1,... X,, be i.i.d. random variables, / = E[x]] and Vi, |X, pl <b b. Let 2 Var(X.). With probability at least 5,
402 log(
4b log(()
Xnn 1l S
(14)
3n
where X, 1 " Dim X;.
Employing Lem.A.1, we can derive the following lemma to bound Lo(w) A- LT)(w) by Ls(w) A L.s(w).
Lemma A.2. Assume that V(x y) E D,0 f(w; y) < B. With probability - over the choice of the training set
S~D D
40 (B AB') log(
(> < 2 (w0)
(15)
3n
where B' 2 C EE TB (w) A Lo(w) x (w) and (w) 2 Ls(w) A Lz(w).
Proof. Since v(x,y) y) D, V. E W. ww: x. < B, have
5 Ls(ur) s B,O S T. (. S
(16)
15

---

Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
and
C 0 - Lg(w) = - (1- (1-m))- - (1-
im1
1m1
(17)
0 - L (w) = ( - [1 11 B
i=1
By the above two inequalities, we can obtain
( - + XB'
(18)
(3 S B + XB'
Thus, we have
1 +d(w)) S (B AB")
(19)
To simplify the analysis. we assume (ww 2 0. This assumption is reasonable because our experiments in Sec.5.6 typically show that the best value for À slightly less than 1, where this assumption holds true. With this assumption, the variance of & (w) can be bounded as:
Var (w)) s ( 2 (B+ AB) 3
(20)
Using Lem.A.1, with probability least 1 8, we have
8 (B + XB') (2 log
$ (w) S (1)))
n
8 (B + XB') 10g(())
(21)
+
3n
1 /.
2 (w) + 20 (B + XB') log(?)
s (ww
3n
where the last inequality leverages the property that for any positive numbers and b, Vab S +
Reformulate the inequality, we can obtain that with probability at least 1 5,
4 (>w)) S ()w + 40 (B + AB') log(?)
(22)
3n
A.3. Proof of Lem.A.3 and Lem.A.4
The following lemmas utilize the PAC- MAC-Bayesian theorem to prove the second part. We first derive an intermediate result in the following lemma.
Lemma A.3. Assume that V(x, 3/) D,0 < f(w; x, S B. Then, for any p > 0 and any distribution D, with probability
1 8 over the choice of the training set S D
Be (L3 w + +e]] S max 2 +
St
(VTe + 22lnn) 2 2B' + 2k log (1+ |w| + 4k log
log 7²
2'VR(nB'41)
(23)
36
n
where n = 1S), k is the number of parameters, B' A D ull (1 n.) T and aQ
Vk- 22m0)))()))
Proof. Inspired by the proof technique SAM (Foret et al., 2021), we provide the following proof.
16

---

Focal-SANe Focal Sharpness-Aware Minimization for Long-Tailed Classification
Since V) E Veu W, luw w, < B, we have:
C [g(w)= = Σ - n. L' - Σ -1-T1
m B' 1=1
(24)
1=1
(
S (1- ) TTB
(25)
i=1
Thereby, the right-hand side of the bound in the theorem lower bounded by A log(1 +
1 1 kp
which greater than B'
when |/ |3 > kpp (exp(nB' /k) 1] and in this case the inequality holds trivially. Thereby, we only consider the case when || S kp [exp(xBB /k) 1] in the rest of the proof.
Using PAC-Baycsian generalization bound in (Tolstikhin & Seldin, 2013), for any fixed prior over parameters, with probability 5 over training S, for any posterior over parameters, the following generalization bound holds:
Ew (w s Ewn (w + eE KL(QIP) + log 2yn
+ KL(2IP) + log
2/n d
(26)
n
< 2E ( + KL(O + log 2/n
where the last inequality leverages the property that for any positive numbers a and b, vab S a + Following SAM (Foret et al., 2021), we assume P = NNur, and Q = Ni of J), then the KL divergence can be written as:
ko? + 11ppp- 1000-1Ql: mell 1 kL(O)))
+ k: log
(27)
2
ofp
Let T = (cexp((1 j)/k)li NJ be the predefined set of values for of. If for any j E N, the bounds holds with then by the union bound, all above bounds hold simultaneously with probability
probability 1 - dj with òn Zy..
66
1- E 2ン
Let aQ = vk+/21m(m))
nQ w and pe = 0. We have:
a + up mell3
nB' k:
S p² +
< p²
expl
(28)
k
Let = [1- & log(( + [l////////c)/ can ensure j E N by setting e = p² exx(xB'x') For o = cexp((1-)))) -
we have:
2 12
|| 12
< < S expl
(02 + =
(29)
k
k
17

---

Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
Building on (q.(28) and (2),, we can obtain an upper bound for the KL divergence:
ko +
pp
(T
Q
k + koog
(30)
2
0
k(p²
exp( (p²
80
12
$
k klog
(31)
1
1
2 Q
k
exp( 1
(0² +
k log
(32)
2
2
exp(#) + 1|
IV + 22))
k log
(33)
p2
ls klog (1+ 11w112 1
2k
(vk v2 (2
(34)
kp2
Given the bound that corresponds to j holds with probability - 6, for 63
x the log term can be bounded as:
2/n
2/n
log
log
log
(35)
6,
6
2/n
x 2
log(c/p²))2
$ log
log
(36)
6
2 7
π (1 + k (D()e)((nB//)))*
< log
log
(37)
6
2 2 n
π >² + nB²²
log
log
(38)
6
2 V nB²)²
log
(39)
35
where the first inequality is derived from the fact that j S 1 + A log(c/(0² + 1log(c/6/+/111//// $ 11 & log(c// p²).
Therefore, the generalization bound be written as:
Ex (e,o (w + E]] S 2E, 20.. E)] 2k log
Iwll + 4k log
(T+ + 22 + 4 log 36
(40)
n
Since ||e||2 has chi-square distribution, for any positive I, we have:
P([ee122 kož 2 20h + +2102) s sexp(-()
(41)
Therefore, with probability 1 - 1/n, we have:
|Ell s o² 1: + +2 k/n(n()) + 2ln(n)
(42)
2 $ oz Vh 22
(43)
< p²
(44)
Therefore, we have:
18

---

Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
2B'
E (c..((0.o))1133 + +E] S 2(1 1/n)
max |cllssp
[g + 6)
12
+ log
[
Iull Ep²
+ 4k log (VT + 22 + 4log
=² (n
36
+
(45)
< 2 + cc)
||e|>SSp
2+ 2B' + 2k log [
[w]3 kp
4k log
(Vi v2
4 4log"
=² ='(n(nB++1)
+
n
Combining the above lemma with the Taylor expansion, we can derive the following lemma to bound À (ww) by A maa Lg(w + t).
Lemma A4. Assume that vx, D20 S x(w;,,)) y) S B. Then, for any p > 0 and any distribution D, with probability
1 -- 8 over the choice of the training set S ~ D
p²
kp
(z(w)= S max |E
2L (w +
tr(H(w)) o (vT + 2l())))²
2(V + 2ln(n))²
(46)
2 2B' + 2k log (1+ +
|| lap
+ 4kloog
Vl + 22lnn + 4 /' +112
36
n1
where n = (S). le is the number of parameters, B' (i) T, H(w) represents the Hessian matrix of L (w) at point and Ir(.) represent the mutrix truce.
Proof. By expanding Ni., + E)] around w using a second-order Taylor Series expansion, we can obtain
1 E H(w)e o([|e|1)) 143 + +]] = Bem-(a.0))44300 + + (VL((w)) +
2
(47)
= Lww + Be. - H H(w)e] + (o(
where σQ & 22+
and (ww) represents the Hessian matrix of (zw at point w.
Thereby, we have:
E Be,N +e] (.((w) Pe H(w)e) + (o.
LLw + tr(H(w))
E 1e + ofk: E )
(48)
p²
kp²
= Lzw + 2(Vx+ (l(nn)
tr(H(w)) + of (Vh+ 2mn(n))
Combining (48) with Lem.A.3, with probability 1 - 8, we have
kp
Lb(w)= < max ||e|:S5
2L + 2(V1 + 22lu(n))2 tr(H(w)) ol (vT (nn))))
(49)
2 + 2B' + 2 log 1 +
||0|| 1pp
+ 4kg
VT: + v2lnn)
4 =² / +1)
71
19

---

Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
A.4. Proof of Thm.4.1
Combining the above two parts, we can finally derive the following theorem.
Theorem A.5 (Restate of Thm.4.1), Assume that Vax y) D,0 A(w; x. S B. For any p 0, any uniform distribution Doat and any distribution D, with probability 1 - 8 over the choice of the training set S ~ D
D (w) 1 2 (w)
40 (0-(B AB" 1og(())
Ap²
(H((W)
CTC
3n CтC
2(VK= + (2 CTC
2+ 2B' + +2
(1
||all kpp
4klog
VT+ V2
+4log 2x vn(nn'+1)
36
(50)
+ A.
n CTC
Akp²
o( (VT+ + 22l((n)) CTC
CCTC)
where n = (S), le is the number of parameters, B' & at point w and tr)-) represent the matrix trace.
((w) represents the Hessian matrix of L (w)
Proof. Combining Lem.A.2 and Lem.A.4 and using union bound, with probability at least - 8, we have
40 (B + XB") log(*)
Ap2
Lp(w)= s 22 (w) +
+t(H(ww)
3n
2(Vk / 2 )
+ A 2 2B' + 2k 2kglo (1
+ 4k log
Vk + 22
4
2x² V +1)²
kp
(51)
n
Akp²
ol (VT /2l 22)(
We further recognize that:
C Lp(w)= = Σ LLL,(w))) (w) 2
Σ
TCLD, (w0) C.T L (w)
(52)
1=1
Substituting (52) into (51( leads to Ahm.A.5.
B. Analysis of Backpropagation Requirements for SAM and ImbSAM
B.1. Backpropagation Requirements for SAM
SAM aims to find flatter minima, ensuring the entire neighborhood around the model parameters has consistently low training loss. The objective loss function is defined as:
L
SAM (w) A max Ls(w + E)
(53)
The optimal perturbation ÉSAM (w) for the inner maximization problem is estimated as follow:
( = VwLs(w)
(54)
L
Thus, the gradient of LSAM(w) can be approximated as:
V L. VwLL (w
(55)
To update parameters once using SAM, two backpropagations are required: one for Lswww, and another for V To w+e(ww
20

---

Focal-SAMe Focal Sharpness-Aware Minimization for Long-Tailed Classification
B.2. Backpropagation Requirements for ImbSAM
ImbSAM divides classes into head and tail groups, denoted as H. and T, and applies SAM only to the tail group. Its objective function is:
5 (w) - (w)) + max م
L5 +w)
(56)
The optimal perturbation (w)) for the inner maximization problem is estimated follow:
(w) Éms(w) ~ V Lg [+
(57)
Thus, the gradient of (ww) can be approximated as:
V w S
( (w)
V [1) (w) V 1.g (w) w+((ww
(58)
To update parameters once using ImbSAM, three backpropagations are required: one for V VwL](w), one for (wL)(w). and another for V 1.g (w) (0+((w)
C. More Experiment Protocols
C.1. Datasets
For long-tailed recognition tasks, conduct experiments on four widely used long-tailed datasets: CIFAR-10 LT, CIFAR- 100 LT, ImageNet-LT, and iNaturalist. For long-tailed domain generalization tasks, we train the model on ImageNet-LT and evaluate it on three OOD datasets: ImageNet-Sketch, ImageNetV2, and ImageNet-C. Below is detailed description of these datasets:
CIFAR-100 LT and CIFAR-10 LT (Cao al., 2019). The original CIFAR-100 (Krizhevsky Hinton, 2009) and CIFAR-10 10 (Krizhevsky & Hinton, 2009) datasets contain 50,000 training images and 10,000 testing images for 100 and 10 classes, respectively. We utilize their various long-tailed versions with different imbalance ratios of {100, 50, 10).
ImageNet--LT (Liu et al., 2019). The ImageNet-LT dataset derived from the ImageNet (Deng et al., 2009) dataset according to a Pareto distribution, containing 1000 categories. The dataset includes 115,846 training images and 50,000 test images. The dataset has an imbalance ratio of 256.
INaturalist (Horn et al., 2018). The iNaturalist dataset is real-world large-scale dataset, consisting of 8142 categories. The training set contains approximately 430,000 images, while the test set contains about 24,000 images. The dataset's imbalance ratio 500.
SSaeehhttSketch (Wang et al., 2019a). The ImageNet-Sketch dataset is an OOD test set derived from the ImageNet (Deng et al., 2009) dataset, comprising 50,000 images across 1000 classes. Each image is a sketch, introducing a domain shift relative to ImageNet.
Imag (Recht et al., 2019). The ImageNetV2 dataset consists of 10,000 images spanning the same 1000 classes as ImageNet. The images are sourced differently from the original ImageNet (Deng et al., 2009), resulting in slight domain shift.
ImageNet-C (Hendrycks & Dietterich, 2019). The ImageNet-C dataset includes the same 1000 classes ImageNet (Deng et al., 2009) but features corrupted versions of the original validation set. Each image undergoes one of 15 corruption types at 5 severity levels, resulting in 75 dataset variations.
C.2. Competitors
When training ResNet models from scratch, we evaluate serveral competitive methods on different datasets. For the CIFAR- LT dataset, we assess multiple loss functions, including CE loss, LDAM+DRW (Cao et al., 2019), LA loss (Menon et al., 2021), and VS loss (Kini et al., 2021). These methods are further combined with SAM (Foret et al., 2021), ImbSAM (Zhou al., 2023a), and CC-SAM (Zhou al., 2023b) as baseline comparisons. For the ImageNet-LTT and iNauuralist datasets, we
21

---

Focal-SANe Focal Sharpness-Aware Minimization for Long-Tailed Classification
employ range of representative methods, including CB (Cui et al., 2019) for class re-balancing, cRT (Kang et al., 2020) for decoupled training, DiVE (He et al., 2021) for transfer learning, DRO-LT (Samuel & Chechik, 2021) for representation learning, DisAlign (Zhang et al., 2021) for class re- re-balancing, and WB (Alshammari et al., 2022) for regularization. When fine-tuning the foundation model CLIP (Radford et al., 2021), we use Decoder (Wang et al., 2024c) and LPT (Dong et al., 2023) as baselines. We also evaluate both fully fine-tuning the models with LA loss (denoted as FFT), and parameter-efficient fine-tuning using the LIFT method (Shi et al., 2024), as well as their performance when combined with different SAM variants.
C.3. Evaluation Protocol
For long-tailes recognition tasks, we assess model performance using top-1 accuracy on balanced test sets. This ensures all classes contribute equally to the evaluation. To provide a more detailed analysis, we follow the approach in (Zhong et al., 2021; Liu et al., 2019) by splitting the classes into three subsets: Head, Medium, and Tail. Accuracy is then reported for each subset individually. For CIFAR-10 10 LT (IR- 100), the Head classes contain more than 1000 samples, the Medium classes have 200~1000 samples, and the Tail classes have less than 200 samples. For CIFAR-100 LT (IR = 100), ImageNet-LT. and INaturalist, the Head classes contain more than 100 samples, the Medium classes have 20~ 100 samples, and the Tail classes have less than 20 samples. Prior arts (Zhou et al., 2022; Khattak et al., 2023; Park et al., 2024) have demonstrated that fine-tuning CLIP (Radford et al., 2021) often performs well on the target domain but struggles with domain shifts, Therefore, when fine-tuning the foundation models, we also assess model performance on OOD test sets, referred to as long-tailed domain generalization tasks. Specifically, models are trained on the ImageNet-LT dataset and evaluated on dittof-diitrrbuutinn datasets, including ImageNet-Skette (Wang et al., 2019a), ImageNetV2 (Recht et al., 2019), and ImageNet-C C (Hendrycks & Dietterich, 2019). We evaluate model performance on these OOD balanced test sets, including top-1 accuracy and accuracy for each class subset.
C.4. Implementation Details
We follow the procedures described below to train ResNet models from scratch. For the CIFAR-LT datasets, we use ResNet-32 (He et al., 2016) as the backbone. We employ stochastic gradient descent (SGD) as the base optimizer, with an initial learning rate of 0.1, a batch size of 64, and momentum of 0.9. Training spans 200 epochs, using a cosine annealing scheduler to reduce the learning rate from 0.1 to 0 gradually. For the larger-scale ImageNet-LTT and iNaturalist datasets, we employ ResNet-50 (He et al., 2016) as the backbone. SGD is again used as the base optimizer with a momentum of 0.9. For ImageNet-L.T, the initial learning rate is set to 0.1, with a batch size of 256, while for iNaturalist, the initial learning rate is 0.2, and the batch size increased to 512. Training for these datasets also lasts 200 epochs with cosine annealing scheduler. Additionally, We employ step scheduler for p. following the approach of Rangwani et al. (2022). This scheduler initializes p until the 160th epoch and then increases its value towards the end of training.
For fine-tuning foundation models, we follow the protocols outlined in LIFT (Shi et al., 2024), A cosine classifier is added after the image encoder of CLIP (Radford et al., 2021), with its weights initialized using the text encoder, which is then discarded. We fine-tune the image encoder of CLIP with a ViT-B/16 (Dosovitskiy et al., 2021) 1) backbone. Stochastic gradient descent (SGD) is used as the base optimizer, with batch size of 128 and momentum of 0.9. The initial learning rate is 0.01 for parameter-efficient fine-tuning and 0.001 for full fine-tuning. Unlike LIFT (Shi et al., 2024), all models in our experiments are fine-tuned for 20 epochs across datasets and methods. In LIFT, models are trained for 10 epochs on the CIFAR-LT and the ImageNet-LLT datasets, and 20 epochs on the iNaturalist dataset. We extend the training to 20 epochs because the models do not fully converge under the original settings.
C.5. Experimental Hardware Setup
All the experiments are conducted on Ubuntu servers equipped with Nvidia(R) RTX 3090 GPUs and RTX 4090 GPUs. Fine-tuning the foundation models is performed using a single GPU for all datasets. The number of GPUs used for training the ResNet models from scratch varies based on dataset size: a single GPU for the CIFAT-LT datasets, 2 GPUs for the ImageNet-LT dataset, and GPUs for the iNaturalist dataset.
22

---

Focal-SAMe Focal Sharpness-Aware Minimization for Long-Tailed Classification
D. More Experiment Results
D.I. Additional Results on the CIEAR-LT Datasets
In this section, we show additional results on the CIFAR-LT datasets. Specifically, 6 presents additional experimental results on the CIFAR-ION 100 LT dataset with more combined methods. Tab.5 provides the experimental results on the CIFAR- 10 LT dataset. The results suggest that Focal-SAM consistently outperforms SAM, ImbSAM, and CC-SAM across all methods and datasets, regardless of whether ResNet models trained from scratch or foundation models are fine-tuned. This indicates that Focal-SAM offers better fine-grained control over the loss landscape for both head and tail classes, leading to improved overall performance. This further highlights the effectiveness of Focal-SAM.
Table 5: Performance comparison on CIFAR-10 LT datasets with various imbalance ratios (IR). FFT denotes fully fine-tuning foundation model with LA loss.
IR100
IR200
IR50
IR10
Method
Head
Med
Tail
All
All
All
All
Training from scratch
CE
87.0
73.6
54.0
73.1
68.6
78.3
87.4
CE+SAM
89.5
73.9
56.7
75.0
69.8
79.6
88.8
IE+ImbSAM
88.0
79.0
60.1
76.9
72.6
81.1
89.3
CE+CC-SAM
88.9
74.1
61.3
76.2
71.3
80.0
89.2
CE+Focal-SAM
89.3
75.4
62.9
77.2
71.7
82.0
90.0
LDAM+DRW (Cao al., 2019)
85.5
74.6
69.0
77.3
73.8
80.8
87.3
LDAM+DRW+SAM
88.9
78.3
73.2
81.0
78.6
84.5
89.4
LDAM-DRW-ImbSAM
86.5
79.7
73.7
80.6
77.3
84.0
88.9
LDAM-DRW-CCC-SAM
88.4
79.2
73.3
81.1
78.9
84.4
89.4
LDAM-DRW
88.7
79.5
74.2
81.6
79.2
84.5
89.5
LA (Menon al., 2021)
87.6
72.6
70.1
77.9
74.3
81.6
87.8
LA+SAM
86.7
80.6
78.2
82.3
78.9
85.4
90.2
LA+ImbSAM
84.1
81.6
80.1
82.2
78.6
84.7
89.5
LA+CC-SAM
86.6
80.8
78.5
82.5
79.1
85.5
90.2
LA+Focal-SAM
86.9
81.2
79.2
82.9
79.6
85.5
90.5
VS (Kini al., 2021)
88.1
77.1
68.4
78.9
74.7
81.5
88.3
VS+SAM
85.6
82.7
76.6
82.0
79.0
85.4
90.3
IS+ImbSAM
85.3
82.1
77.3
81.9
78.7
84.8
90.0
VS+CC-SAM
85.6
82.0
78.2
82.3
79.3
85.5
90.4
VS+Focal-SAM
87.7
80.6
78.8
82.9
79.5
85.8
90.7
Fine-tuning foundation model
FFT
97.9
95.8
95.9
96.7
95.7
97.1
97.9
FFT+SAM
97.5
96.5
97.0
97.0
96.6
97.5
98.0
ImT+M
97.0
97.0
97.4
97.1
96.5
97.7
97.9
FFT+CC-SAM
97.6
96.2
97.0
97.0
96.6
97.6
98.0
FFT+Focal-SAM
97.5
96.4
97.5
97.2
96.6
97.5
98.2
LIFT (Shi et al., 2024)
96.6
95.7
97.4
96.6
96.3
96.8
97.2
LIFT+SAM
96.6
95.6
97.8
96.7
96.4
96.7
97.0
LIFT+IbSSAM
96.5
95.9
97.7
96.7
96.4
96.7
97.2
LIFT+CC-SAM
96.5
95.6
97.9
96.6
96.4
96.7
97.3
LIFT+Focal-SAN
96.6
95.6
98.1
96.8
96.4
96.9
97.3
23

---

Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
Table 6: Performance comparison on CIFAR-I00 LT with more combined methods
IR100
IR200
IR50
IR10
Method
Head
Med
Tail
All
All
All
All
Training from scratch
LDAM+DRW (Cao et al., 2019)
63.1
44.4
18.6
43.2
40.3
46.1
57.3
LDAM-DRW+SAM
67.6
51.7
25.9
49.5
45.8
52.6
61.1
LDAM-DRW+ImbSAN
62.5
48.8
26.4
46.9
42.5
51.3
59.8
LDAM-DRW-CC-SAM LDAM-DRW+FFcal-SAR
66.5
52.2
26.2
49.4
45.7
52.3
61.0
67.9
52.7
26.9
50.3
46.2
53.8
62.3
VS (Kini et al., 2021)
58.3
43.8
31.1
45.1
41.6
49.3
59.4
VS+SAM
62.7
52.0
29.3
49.0
45.5
53.5
62.5
VS+ImbSAM
56.1
53.3
29.9
47.2
44.7
52.6
62.6
VS+CC-SAM
62.2
52.2
30.3
49.1
45.2
53.7
62.9
VS+Focal-SAM
62.7
52.6
31.0
49.7
45.8
54.5
63.7
D.2. CE and mCE Metries on ImageNet-C for Long-tailed Domain Generalization
ImageNet-C (Hendrycks & Dietterich, 2019) contains the corrupted versions of ImageNet (Deng et al., 2009) dataset, with 15 corruption types applied severity levels, resulting in 75 dataset variations. In addition the model's average accuracy across these 75 datasets, as shown in Tab.4, ImageNet-C introduces two additional metrics: Corruption Error (CE) and Mean Corruption Error (mCE). These metrics systematically assess the robustness of models against image corruption. CE measures the accuracy drop of model on specific type and severity of corruption compared to baseline model, typically AlexNet (Krizhevsky et al., 2012). mCE aggregates the CE values across all corruption types and severity levels, providing a single robustness score for the model. Tab.7 presents the CE and mCE results on the ImageNet-C dataset when fine-tuning the foundation models. The results show that Focal-SAM generally achieves significantly lower CE and mCE values across entire dataset and for each corruption type. This further demonstrates Focal-SAM's effectiveness in improving generalization.
Table 7: The CE and mCE values for different methods the ImageNet-C dataset. The source models are trained ImageNet-LT and evaluated on ImageNet-C. Lower values indicate better performance.
Blur
Noise Impul
Digital
Weather
Method
mCEJ
Motion
Defoc
Glass
Zoom
Gauss
Shot
Contr
Blast
JPEG
Pixel
Bright
Snow
Pog
Frost
FFT
72.6
72.8
74.6
78.2
79.4
73.6
74.1
75.1
66.7
80.0
78.4
71.3
63.7
66.6
65.7
68.5
+SAM
69.0
69.2
72.2
76.9
76.9
68.9
69.7
70.7
63.6
77.5
74.5
66.8
59.2
62.6
60.6
65.7
+lmbAAM
69.5
69.4
72.2
76.8
77.1
70.2
70.7
72.0
62.9
77.6
76.4
68.5
59.6
62.5
60.2
65.7
-CC-SAM
69.0
69.0
74.1
76.6
77.8
69.5
69.3
71.2
64.0
76.6
75.2
67.0
58.1
61.3
59.6
65.0
+Focal-SAM
68.3
68.2
72.7
76.6
76.6
68.3
68.8
70.1
63.2
76.2
74.2
66.0
58.0
61.5
59.7
65.1
LIFT
63.6
61.7
67.6
75.9
72.4
61.0
61.2
62.7
54.1
80.8
72.7
60.3
52.1
$7.0
52.2
62.1
+SAM
63.0
61.1
67.2
75.7
71.5
60.4
60.6
62.0
53.6
80.6
72.1
59.4
51.7
56.3
51.7
61.7
AIbbAA
63.4
61.6
67.5
75.8
72.1
60.9
60.9
62.6
53.8
80.6
72.5
59.9
51.9
56.7
51.9
61.8
+CC-SAM
62.5
61.0
66.5
75.3
70.9
59.5
$9.9
61.3
53.7
79.9
71.7
58.2
51.3
55.5
51.6
61.2
+Focal-SAM
62.2
66.3
75.2
71.1
59.3
59.9
53.1
79.6
71.0
58.2
50.7
55.3
51.1
61.0
D.3. Results for Aligning Computational Cost
Focal-SAM requires about 50% more training time than SAM. To fairly evaluate the benefit of Focal-SAM, we conduct experiments where we extend the training epochs of SAM to match Focal-SAM's total computational cost. Specifically, we increase the training epochs to 300 or 30 (1.5 x the original 200 or 20) for SAM, while keeping Focal-SAM at 200 or 20 epochs. In this setting, the total computational cost of SAM and Focal-SAM becomes comparable. We conduct these experiments on CIFAR-100 LTT, ImageNet-L.T, and iNatualist datasets. The results are shown in Tab.8, Tab.9, and Tab. 10.
24

---

Focal-SAMe Focal Sharpness-Aware Minimization for Long-Tailed Classification
Table 8: Performance comparison on CIFAR-100 LT with aligned computational cost.
IR100
IR200
IR50
IR10
Method
Epoch
All
All
All
All
Training from scratch
CE+SAM
300
43.0
39.2
46.9
60.0
CE+Focal-SAM
200
44.0
39.6
48.1
60.9
LDAM+DRW+SAM
300
50.4
46.4
53.0
61.2
LDAM-DRR-FFOalSSAN
200
50.3
46.2
53.8
62.3
VS+SAM
300
49.2
45.5
53.0
63.3
VS+Focal-SAM
200
49.7
45.8
54.5
63.7
LA+SAM
300
50.1
45.5
53,8
63.0
SAA
200
50.7
46.0
54.5
63.8
Fine-tuning foundation model
FFT+SAM
30
81.2
78.3
83.6
86.9
FFT+Focal-SAM
20
81.6
79.0
83.9
87.3
LIFT+SAM
30
82.1
80.2
83.1
85.2
LIFT+Focal-SAM
20
82.4
80.0
83.2
85.4
Table Performance comparison on ImageNet-LT with aligned computational cost.
ImageNet-LT Medium
Method
Epoch
Head
Tail
All
Training from scratch
LA+SAM
300
63.2
51.6
34.8
53.8
LA+Focal-SAM
200
63.9
52.2
34.4
54.3
Fine- Mine-tuning foundation model
FFT+SAM
30
80.6
73.1
56.1
73.6
FFT+Focal-SAM
20
80.8
73.9
54.4
73.9
LIFT+SAM
30
79.8
76.1
73.5
77.2
LIF++Focal-SAM
20
79.7
76.6
73.6
77.4
Table 10: Performance comparison iNaturalist with aligned computational cost.
iNaturalist Medium
Method
Epoch
Head
Tail
All
Training from scratch
LA+SAM
300
68.0
71.4
72.4
71.5
LA+Focal-SAM
200
68.4
72.0
72.5
71.8
D.4. Visualization of Loss Landscape
Fig.6 and 7 visualize the loss landscape for head and tail classes of ResNet models trained with SAM, ImbSAM, CC-SAM, and Focal-SAM on the CIFAR-100 LT and CIFAR-10 LT datasets using VS loss respectively. From the results, we can observe that the loss landscape for tail classes with ImbSAM generally appears flatter and smoother than with SAM, suggesting that ImbSAM better flattens the loss landscape for tail classes. However, the head class loss landscape with
25

---

Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
ImbSAM is generally sharper than with SAM, indicating that ImbSAM's exclusion of all head classes from the SAM term sharpen the loss landscape for head classes, which might reduce their generalization performance. In contrast, CC-SAM and Focal-SAM provide fine-grained class-wise control, leading to flatter loss landscape for both head and tail classes.
(a) SAM: Head
(b) ImbSAM: Head
(c) CC-SAM: Head
Focal-SAM: Head
(e) SAM: Tail
(1) ImbSAM: Tail
(g) CC-SAM: Tail
(h) Focal-SAM: Tail
Figure 6: Visualization of loss landscape for head and tail classes of ResNet models trained with SAM, ImbSAM, CC-SAM, and Focal-SAM CIFAR-100 LT using VS loss respectively.
(a) SAM: Head
(b) ImbSAM: Head
(c) CC-SAM: Head
(d) Focal-SAM: Head
(e) SAM: Tail
(I) ImbSAM: Tail
(g) CC-SAM: Tail
(h) Focal-SAM:
Figure 7: Visualization of loss landscape for head and tail classes of ResNet models trained with SAM, ImbSAM, CC-SAM, and Focal-SAM CIFAR-10 LT using VS loss respectively.
D.5. Ablation Study About Perturbation Radius p
Fig.8 illustrates the impact of the hyperparameter p the performance of Focal-SAM when combined with LDAM+DRW. LA, and VS methods on the CIFAR-LT datasets during ResNet models training. As increases, Focal-SAM's performance initially improves but then declines. This indicates a trade-off between achieving flatter minima and reducing training loss.
The optimal value for Focal-SAM is approximately 0.3, which is higher than the commonly optimal value for SAM on balanced training datasets, as reported by Foret et al. (2021). This observation consistent with Rangwani et al. (2022), who suggest that larger p can enhance performance long-tailed learning.
26

---

Focal-SAMe Focal Sharpness-Aware Minimization for Loin-Tailed Classification
50
49-
Accuracy (%)
81
48-
80
Accuracy
47
48
45
LA
78-
LDAM-DRW
44
LDAM-DRW
0
0.0
(a) CIFAR-IO
(b) CIFAR-100 LT
Figure 8: Ablation Study of Focal-SAM w.E.I. P
D.6. Additional Results for Eigen Spectral Density of Hessian
This section presents additional results of the spectral density of hessian for ResNet models trained with SAM, ImbSAM, CC-SAM, and Focal-SAM. We analyze models trained on CIFAR-10 LT and CIFAR-100 LT datasets using VS and CE loss functions. The results are visualized in Fig.9, Fig. 10 and Fig.
The results indicate that the largest eigenvalue Amaz and the trace (H( of the Hessian for tail classes are generally smaller with ImbSAM than with SAM. This suggests that ImbSAM flattens the loss landscape for tail classes more effectively, However, Amaz and tr H)) for head classes are typically larger with ImbSAM than with SAM, indicating that ImbSAM's coarse-grained strategy of excluding all head classes from SAM terms sharpens the loss landscape for those classes In contrast, CC-SAM applies finer control over the loss landscape by using class-dependent perturbation radii, generally achieving lower Amar and tr tr( for head and tail classes. Overall, both Amar and tr(H) for head and tail classes relatively lower with Focal-SAM than other SAM-based methods. This further suggests that Focal-SAM provides fine-grained control over the loss landscape, leading flatter landscape for both head and tail classes.
10º
10²
10'-
Density (Log Scale)
Anar: 9.89
Density (Log Scale)
Anw: 11.91
Density (Log Scale)
Amar 10. A3
Density (Log Scale)
Amer: 10.30
10a
Ti(H): 162.39
10a
256
10
THH):: 147.41
10a
THH:: 163.63
10
10-2
10"2
10-2
10-1
10-1
10-4
10
10-t
10-4
-10--00
1
10a
10a
-10--10-
-1
10a
10a
-102-10¹
-1
10a
10a
-10--10²
10a 10a
Elgenvalue
Elgenvalue
Elgenvalue
Eigenvalue
(a) SAM: Head Classes
(b) ImbSAM: Head Classes
(c) CC-SAM: Head Classes
(d) Focal-SAM: Head Classes
Density (Log Scale)
2mow: 29.11
Density (Log Scale)
Arau 27.47
Density (Log Scale)
Arcui 26 64 THH::
Density (Log Scale)
Amwo 25.32
10a
477.77
10a
10a
10a
10-2
10-2
10
10
10
10
10-+
100-10'
11 Eigenvalue
10a
10a
-10-100
1
10a
10a
110-10"
-1
10a
10a
-10--00 -1
D
10a 10a
Elgenvalue
Eigenvalue
Eigenvalue
(c) SAM: Tail Classes
ImbSAM: Tail Classes
(g) CC-SAM: Tail Classes
(h) Focal-SAM, Tail Classes
Figure 9: Eigen Spectral Density of Hessian for head and tail classes of ResNet models trained with SAM, ImbSAM, CC-SAM, and Focal-SAM on CIFAR- 100 LT using VS loss respectively. A smaller Amaz and Tr(H) generally indicate flatter loss landscape.
27

---

Focal-SAM: Focal Sharpness-Aware Minimization for Loin-Tailed Classification
Density (Log Scale!
107
8.19
Density (Log Scale
10P
Avar 10.12 Tr(H): 89 21
Density [Log Scalel
Arcus 8.57
Density (Log Scale!
10a
8.76
10a
TI(H:
109
109
T((H): 58 41
109
THH): 36.91
10"2
10-2
10"2
10"2
10
10
10-4
10
10
10
10a
10a
10a
10a
10a
10a
10*
10
Eigenvalue
Elgenvalue
Elgenvalus
Eigenvalue
(a) SAM: Head Classes
(b) ImbSAM: Head Classes
(c) CC-SAM: Head Classes
(d) Focal-SAM: Head Classes
Density (Log Scale)
Amar: 35.72
Density (Lay Scale)
Amar 26.76
Density (Log Scale)
Ares: 27.6
Density (Lag Scale)
Ama: 24.07
10a
THH:: 217.15
109
TM|H: 159.73
10"
THH 180.47
10a
Tr(H): 113.31
10-2
10-2
10-2
10-2
10-1
10-4
10-4
10-4
10-4
10-8
10-
10-3
10a
10a
102
104
10
10
10
10a
10t
10
10a
Eigenvalue
Elgenvalue
Elgenva
Eigenvalue
(c) SAM: Tail Classes
ImbSAM: Tail Classes
CC-SAM: Head Classes
(h) Focal-SAM: Tail Classes
Figure 10: Eigen Spectral Density of Hessian for head and tail classes of ResNet models trained with SAM, ImbSAM, CC-SAM, and Focal-SAM on CIFAR-10 LT using CE loss respectively. A smaller Amaz and Tr(H) generally indicate flatter loss landscape.
Density (Log Scale!
Anw: 7.94
Density (Log Scale!
102
10.16 7(H);
Density (Log Scale!
Amari 8.67 THH):
Density (Log Scale!
102
Amasi 8.43
10a
TN(H):
10
263 61
10"
109
[HH]:
10-2
10-2
10-2
10-2
10
10
10
10
10.5
10
10**
10"*
-10--10-
10
-10--10t
101
-10
10c
-10--10-
Eigenvalue
Eigenvalue
Eigenvalis
Eigenvvauee
(a) SAM: Head Classes
(b) ImbSAM: Head Classes
(c) CC-SAM: Head Classes
(d) Focal Secal-SAMM: Head Classes
Density (Log Scale)
Amaxi 26,55 98
Density (Lay Scale)
Amasi 24, 3b
Density (Log Scale)
Aras 25 05
Density (Log Scale)
Amas 25.01
109
TrHH): 652.76
10a
TrHH; 381.37
10a
THHH: 486. o8
1.0
T(H): 561.13
10-
10
10-1
10-1
10-1
10-1
10-4
10-6
10-4
10-6
10
104
102
102
-10 10a
Eigenvalue
Elgenvaluu
Eigenvalue lue
Eigenvvalue
(e) SAM: Tail Classes
( ImbSAM: Tail Classes
CC-SAM: Tail Classes
(h) Focal-SAM: Tail Classes
Figure 11: Eigen Spectral Density of Hessian for head and tail classes of ResNet models trained with SAM, ImbSAM, CC-SAM, and Focal-SAM on CIFAR- 100 LT using CE loss respectively. A smaller Xmaz and Tr(H) generally indicate flatter loss landscape.
28


Reviews:
[{"review_id": "mYHBXrVMjo", "reviewer": "Reviewer_n3JG", "summary": "For long-tailed (imbalanced) classificaton, the authors propose\nFocal-SAM that aims at class-wise SAM so that flatter minima are\nfound.  They first show that imbSAM, which applies SAM only to tailed\nclasses, can increase sharpness in the head classes.  They then show\nthat CC-CAM could be computational expensive due to one\nbackpropagation for each class.\n\nThe proposed Focal-SAM has class-wise SAM, however, not class-wise\nperturbation (epsilon) for SAM.  That is, they have one perturbation\nfor all classes, which requires only xone backpropagation for all\nclasses.  For each class, they first calculate L(w+epsilon) - L(w).\nThen, for the focal-SAM loss, they weight each class loss with (the\nweight component of) focal loss, which is then maximized over epsilon.\nThe overall loss is the oringial loss plus the focal-SAM loss.  They\ndiscussed that only 3 backpropagations are needed.\n\nFor empirical evaluation, they use 4 datasets and compare with 3\nexisting SAM-based techniques, combine with and 3 methods for\nimbalanced data.  Empirical results indicate that the proposed method\ncompares favorably.\n\n## update after rebuttal\n\nAfter reading and responding to the authors' rebuttal, I decided to maintain my rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1:  On computational overhead, how different are the perturbations between the two methods?\n\n**A1:**  Thanks for your question! \n\nThe perturbation in Focal-SAM is computed as:\n$$\n\\hat{\\epsilon}(w) = \\rho \\frac{\\nabla_w L_S^\\gamma(w)}{|| \\nabla_w L_S^\\gamma(w) ||_2}\n$$\n\nwhere $L_S^\\gamma(w) = \\sum_{i=1}^C (1 - \\pi_i)^\\gamma L_S^i(w)$.\n\nIn contrast, CC-SAM computes a class-specific perturbation for each class $i$ as:\n$$\n\\hat{\\epsilon}_i(w) = \\rho_i^* \\frac{\\nabla_w L_S^i(w)}{|| \\nabla_w L_S^i(w) ||_2}\n$$\n\n**In terms of computational overhead:**\n\nEach perturbation requires one backpropagation (BP) to compute the corresponding gradient:  \n\n- Focal-SAM requires only **one BP** to compute perturbation to compute $\\nabla_w L_S^\\gamma(w)$.\n- CC-SAM requires total **$C$ BPs**, one for each class, to compute $\\nabla_w L_S^i(w)$.\n\n**The difference between the two perturbation are twofold:**\n\n- CC-SAM employs class-specific radii $\\rho_i^*$ to finely control the sharpness penalty for each class. In contrast, Focal-SAM uses a unified perturbation radius $\\rho$.\n- CC-SAM computes the gradient of class-wise loss $L_S^i(w)$ individually, while Focal-SAM computes the gradient of the weighted loss $L_S^\\gamma(w)$. \n\nIf we **disregard the difference in scaling across perturbations**, *i.e.*, assume:\n$$\n\\frac{\\rho}{|| \\nabla_w L_S^\\gamma(w) ||_2} = \\frac{\\rho_i^*}{|| \\nabla_w L_S^i(w) ||_2}, \\forall i\n$$\n\nthen the Focal-SAM perturbation can be rewritten as:\n$$\n\\hat{\\epsilon}(w) = \\sum_{i=1}^C (1 - \\pi_i)^\\gamma \\hat{\\epsilon}_i(w)\n$$\n\nThis suggests that the perturbation in Focal-SAM can be viewed as a **weighted aggregation of the class-wise perturbations in CC-SAM**. Therefore, Focal-SAM implicitly considers the contribution of all classes and adjusts the emphasis across classes by tuning $\\gamma$.\n\n\n\n\n\n> Q2: The second term in Eq.4 does not seem to be in ImbSAM or CC-SAM.\n\n**A2:** Thank you for your questions! \n\nIn fact, we can **reformulate the objective functions of ImbSAM and CC-SAM to make the second term, $L(w)$, explicitly appear**. Furthermore, we can rewrite the objectives using the class-wise sharpness term (Eq.4). \n\nFor ImbSAM:\n$$\nL_S^{IS}(w) = L_S^H(w) + L_S^T(w) + \\max _{|| \\epsilon ||_2 \\le \\rho} [L_S^T(w+\\epsilon) - L_S^T(w)] = L_S(w) + \\max _{|| \\epsilon ||_2 \\le \\rho} \\sum _{i \\in T} \\tilde{L}_S^i(w, \\epsilon) \\quad  (1)\n$$\n\nFor CC-SAM:\n$$\nL_S^{CS}(w) = \\sum_{i=1}^C \\frac{1}{\\pi_i} \\cdot L_S^i(w) + \\sum_{i=1}^C \\max_{|| \\epsilon || \\le \\rho_i^*} \\frac{1}{\\pi_i} \\cdot [L_S^i(w + \\epsilon) - L_S^i(w)] = \\sum_{i=1}^C \\frac{1}{\\pi_i} \\cdot L_S^i(w) + \\sum_{i=1}^C \\max_{|| \\epsilon ||_2 \\le \\rho_i^*} \\frac{1}{\\pi_i} \\cdot \\tilde{L}_S^i(w, \\epsilon) \\quad (2)\n$$\n\nEq.(1) shows that ImbSAM's objective includes the class-wise sharpness only for tail classes. This design specifically focuses on flattening the loss landscape for tail classes but may lead to poor generalization for head classes.\n\nEq.(2) shows that CC-SAM considers the class-wise sharpness for all classes, using class-specific perturbation radii $\\rho_i^*$. This allows it to more effectively flatten the loss landscape for both head and tail classes.\n\n\n\n\n\n> Q3: Fig.4: Gamma seems to be quite small (close to zero) for the highest accuracy. \n\n**A3:** Thanks for your questions! The reason is that **even a small $\\gamma$ is sufficient to create a relatively skewed** $(1 - \\pi_i)^\\gamma$ from head to tail classes. \n\nAs stated in the paper, we assume that $\\pi_1 \\ge \\pi_2 \\ge \\cdots \\ge \\pi_C$. For example, in the CIFAR-10 LT dataset ($C = 10$ classes), when $\\gamma = 0.8$ (the value at which the LA method achieves the best performance, as shown in Fig.4), we calculate that $(1 - \\pi_{10})^\\gamma$ is approximately $0.99$, whereas $(1 - \\pi_1)^\\gamma$ is around $0.66$. This indicates that even a relatively small $\\gamma$ can result in a highly skewed $(1 - \\pi_i)^\\gamma$. \n\nTherefore, despite $\\gamma$ being small, the focal component still plays a contributing role.\n\n\n\n\n\n> Q4: Figure 3: A lower $\\pi$ has a higher $(1 - \\pi_i)^\\gamma$. However, this is not the case in the figure?\n\n**A4:** Thanks for your question! We apologize if our description of Fig.3 is unclear and potentially misleading. \n\nTo clarify, Fig.3 presents the distribution of $(1 - \\pi_i)^\\gamma$ across various $\\gamma$ and datasets. Specifically,  for a given $\\gamma$ and dataset, we plot the distribution of  $\\mathcal{P} = \\\\{ (1 - \\pi_i)^\\gamma \\\\} _ {i = 1} ^C$, where $C$ is the number of classes. Since the datasets **contain many tail classes with similarly small sample sizes** (*i.e.*, small $\\pi_i$), the distribution **tends to peak at relatively large values of $(1 - \\pi_i)^\\gamma$**. Hence, Fig.3 differs from the scenario you expected, where it should plotting $\\mathcal{P}' = \\\\{(1 - \\pi_{y_n})^\\gamma \\\\}_{n = 1}^N$, with $N$ being the number of samples.", "claims_and_evidence": "The emiprical results indicate that Focal-SAM can achieve higher\naccuracy with less computation.", "methods_and_evaluation_criteria": "The proposed method is based on insights on the limitatons of imbSAM\nCC-SAM.  By reducing sharpness for all classes and the number of\nbackpropagations, Focal-Sam can outperform imbSAM and CC-SAM.  Focal\nloss is an existing method.  The evaluation criteria are reasonable.", "theoretical_claims": "Due to not being familiar with some of the terms, I did not check the\ntheoretical claim and proofs in the appendix.", "experimental_designs_or_analyses": "Due to not being familiar with some of the terms, I did not check the\ntheoretical claim and proofs in the appendix.", "relation_to_broader_scientific_literature": "The key contributions are limited to SAM-based methods for long-tailed\nclassifications.", "essential_references_not_discussed": "I am not aware of essential references related to SAM within the\nimbalanced context that are not discussed.", "other_strengths_and_weaknesses": "The proposed method on class-wise SAM with fewer backpropagations is\ninteresting.  The empirical results indicate performance improvements\nin accuracy`and speed.  The paper is generally well written.", "other_comments_or_suggestions": "see questions below", "questions_for_authors": "1.  On computational overhead, since a key difference is one\nperturbation (epsilon) for all classes (Focal-SAM) vs for each class\n(CC-SAM), how different are the perturbations between the two methods?\n\n2.  Eq 4: L(w + epsilon) - L(w).  The second term does not seem to be\nin imbSAM or CC-SAM.  Can you provide some discussion on the\ndifference?\n\n3.  Figure 4: Gamma seems to be quick small (close to zero) for the\nhighest accuracy. That is, the focal component seems to be an\ninsignificant contributor. Any further insights on why?\n\n4.  Figure 3: A lower $\\pi_i$ has a higher $(1-\\pi_i)^\\gamma$, which implies a higher $(1-\\pi_i)^\\gamma$ (x-axis) has a lower probability density (y-axis).  However, this is not the case in the figure?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "n/a"}}, {"review_id": "5JUsJiG4m7", "reviewer": "Reviewer_JBZt", "summary": "This paper proposes a learning mechanism named Focal Sharpness-Aware Minimization (Focal SAM), which is an exquisite extension of SAM theories over long-tailed classification tasks. Compared with existing methods, the proposed Focal SAM excels at keeping the flatness of landspaces of both head and tail classes. Extensive experiments show that Focal SAM outperforms existing methods on most datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1:  (1) From Table. 2, I observe a significant performance enhancement with the original SAM, but the improvements from SAM to Focal-SAM are not obvious. (2) I notice that the running time of Focal-SAM is 50% more than the original SAM. This dilates the necessity of employing Focal SAM.\n\n**A1**: Thanks for your question!\n\n**(1) Performance Gains:** It is true that the improvement from SAM to Focal-SAM is generally smaller than that from the baseline to SAM. This is expected, as SAM itself is already an effective method, and Focal-SAM is proposed as a **refinement of SAM**. Therefore, the additional improvement is naturally smaller but **still consistent and meaningful**. We also conduct multiple runs and statistical significance analysis to further confirm this. Please see `A1` for Reviewer `bvws` for details.\n\n\n\n**(2) Computational Cost:** While Focal-SAM increases training time by around 50% compared to SAM, this **does not mean it is unnecessary**. To fairly assess its benefit, we conduct experiments where we extend the training epochs of SAM to match Focal-SAM's total computational cost. Specifically, we increase the training epochs to **300 or 30** (1.5$\\times$ the original 200 or 20) for SAM, while keeping Focal-SAM at **200 or 20** epochs. **In this setting, the total computational cost of SAM and Focal-SAM becomes comparable.** All other experimental settings remain the same as in the main paper. We conduct these experiments on CIFAR-100 LT, ImageNet-LT, and iNaturalist datasets. The results are summarized below, from which we can observe **consistent improvement**.\n\n-----\n\n**CIFAR-100 LT**\n\n|Method|Epoch|IR100|IR200|IR50|IR10|\n|-|-|-|-|-|-|\n|CE+SAM|300|43.0|39.2|46.9|60.0|\n|**CE+Focal-SAM**|200| **44.0**|**39.6**|**48.1**|**60.9**|\n|LDAM+DRW+SAM|300|**50.4**|**46.4**|53.0|61.2|\n|**LDAM+DRW+SAM**|200|50.3|46.2|**53.8**|**62.3**|\n|VS+SAM|300|49.2|45.5|53.0|63.3|\n|**VS+Focal-SAM**|200|**49.7**|**45.8**|**54.5**|**63.7**|\n|LA+SAM|300|50.1|45.5|53.8|63.0|\n|**LA+Focal-SAM**|200|**50.7**|**46.0**|**54.5**|**63.8**|\n|FFT+SAM|30|81.2|78.3|83.6|86.9|\n|**FFT+Focal-SAM**|20|**81.6**|**79.0**|**83.9**|**87.3**|\n|LIFT+SAM |30| 82.1|**80.2**|83.1|85.2|\n|**LIFT+Focal-SAM**|20|**82.4**|80.0|**83.2**|**85.4**|\n\n-----\n\n**ImageNet-LT**\n\n|Method|Epoch|Head|Medium|Tail|All|\n|-|-|-|-|-|-|\n|LA+SAM|300|63.2|51.6|**34.8**|53.8|\n|**LA+Focal-SAM**|200|**63.9**|**52.2**|34.4|**54.3**|\n|FFT+SAM|30|80.6|73.1|**56.1**|73.6|\n|**FFT+Focal-SAM**|20|**80.8**|**73.9**|54.4|**73.9**|\n|LIFT+SAM|30|**79.8**|76.1|73.5|77.2|\n|**LIFT+Focal-SAM**|20|79.7|**76.6**|**73.6**|**77.4**|\n\n-----\n\n\n**iNaturalist**\n\n|Method|Epoch|Head|Medium|Tail|All|\n|-|-|-|-|-|-|\n|LA+SAM|300|68.0|71.4|72.4|71.5|\n|**LA+Focal-SAM**|200|**68.4**|**72.0**|**72.5**|**71.8**|\n\n\n\n\n\n\n\n\n\n\n> Q2: I think that a quite amount of theoretical contributions are provided in the appendix. This makes the conclusion of your theorem a little abrupt. I suggest the authors move several lemmas to the main paper.\n\n**A2**: Thanks for your valuable suggestion!\n\nDue to space constraints in the anonymous submission version, we included the proof sketch and the lemmas supporting Theorem 3.1 in the appendix. We acknowledge that this may make the conclusion of Theorem 3.1 appear abrupt. **In the future version, we will revise the manuscript and relocate the proof sketch along with several key lemmas, such as Lemma B.2 and Lemma B.4, into the main text.**\n\n\n\n\n\n> Q3: Meanings that Figure 1.b and Table. 1 try to convey seems repeated.\n\n**A3**: Thanks for your question!\n\nWe acknowledge that there is some overlap between Figure 1(b) and Table 1 regarding the part of computational cost. However, **the key messages we intend to convey through them are different**. In Figure 1(b), our primary goal is to directly compare Focal-SAM and CC-SAM to demonstrate the **effectiveness of Focal-SAM**, showing that it achieves better performance with higher efficiency. In contrast, Table 1 is used to **highlight the limitations of CC-SAM and to motivate the development** of our method. Specifically, we compare CC-SAM other SAM-based methods—including SAM, ImbSAM, and Focal-SAM—and show that CC-SAM incurs higher computational costs than the others. This observation motivates our proposal of Focal-SAM as an efficient alternative.", "claims_and_evidence": "This paper proposes Focal SAM, which is motivated by several shortcomings of ImbSAM and CC-SAM. The authors give empirical analysis and visualizations of these.  This paper also gives the generalization bound of Focal SAM based on a theorem, and concrete proofs to this theorem are attached to the appendix. Conclusively, all claims are well validated.", "methods_and_evaluation_criteria": "Yes. The proposed Focal SAM is a successful application of SAM to long-tailed classification with the combination of class-wise scaling control. This paper may be of interest to the research community,  bringing new insights and broadening horizons.", "theoretical_claims": "This paper proposes a theorem to show the generalization bound of their Focal SAM. To the best of my knowledge, I found no technical flaws in the proof. I'm not quite sure about the part concerning PAC-Bayesian generalization.", "experimental_designs_or_analyses": "Yes. The experiment design is appropriate. The authors conduct experiments over four datasets. Experiments show that their Focal-SAM outperforms existing methods. The authors also make necessary explanations for this.", "relation_to_broader_scientific_literature": "I think this paper contributes much to the research community. Although the methodology is intuitive, the authors give detailed proofs and experiments to validate its superiority. This makes their methods solid and easy to implement, illuminating the research community.", "essential_references_not_discussed": "I found no essential references missed in this paper.", "other_strengths_and_weaknesses": "- Strengths:\n1. The paper is well-written and easy to follow.\n2. The motivation is clear, where authors analyze the shortcomings \n3. The idea is quite simple, easy to implement and also superior. The authors give concrete theoretical and experimental analysis of their method.\n4. The experiments are extensive and well-designed.\n\n- Weaknesses:\n1. I notice that the running time of Focal-SAM is 50% more than the original SAM. The authors think that it's negligible given the strong performance of Focal SAM, while I can't agree.  From Table. 2, I observe a significant performance enhancement with the original SAM, but the improvements from SAM to Focal-SAM are not obvious. This dilates the necessity of employing Focal SAM. \n2. I think that a quite amount of theoretical contributions are provided in the appendix. This makes the conclusion of your theorem a little abrupt. I suggest the authors move several lemmas to the main paper.\n3. Meanings that Figure 1.b and Table. 1 try to convey seems repeated.", "other_comments_or_suggestions": "It seems that Focal SAM may not be very efficient. Also, I'm not quite sure about some proof details concerning PAC-Bayesian generalization. Considering all of the above, I tend to give this paper a weak reject. If the author can answer my doubts well, I am willing to raise my score.", "questions_for_authors": "Can you provide more explanations for the efficiency of Focal SAM? With a trade-off between performance and efficiency, I feel that Focal SAM has no explicit advantage compared with SAM.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hs12mV5AtI", "reviewer": "Reviewer_ag1Z", "summary": "The paper introduces Focal-SAM, a new variant of Sharpness-Aware Minimization (SAM) designed for long-tailed classification. It aims to improve generalization for both head and tail classes by integrating the focal mechanism with SAM. Compared with baselines like ImbSAM and CC-SAM, Focal-SAM efficiently achieves flatness in both head and tail classes. The authors present a theoretical generalization bound with improved convergence rates. Moreover, they validate Focal-SAM’s effectiveness through extensive experiments.\n\n##Update\n\nThe authors' response addresses my concerns.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: Why don't use $\\tilde{L}_S^{FS} (w)$ (eq 5) as the main objective? Specifically, why manage $\\tilde{L}_S^{FS} (w)$ as a penalty of standard loss as in eq 6, which will add an additional hyperparameter $\\lambda$?\n\nThanks for your questions! \n\nThe motivation behind SAM-based methods is that the geometry of the loss landscape is closely related to a model's generalization ability. Specifically, **flatter minima** in the loss landscape generally leads to better generalization compared to sharper ones. Therefore, SAM-based methods aim to **simultaneously minimize both the loss value and the sharpness** to seek out flatter minima.\n\nIn our Focal-SAM method, $\\tilde{L}_S^{FS}(w)$ is a term which **reflects the sharpness of loss landscape** at $w$. If we only minimize $\\tilde{L}_S^{FS}(w)$ , we can generally obtain a model parameter $w$ with a flat loss landscape. However, this **does not necessarily minimize the actual loss value** $L_S(w)$, which may remain large and degrade model performance.\n\nTherefore, we incorporate the sharpness term $\\tilde{L}_S^{FS}(w)$ as a penalty term to the standard loss value $L_S(w)$, weighted by a hyperparameter $\\lambda$. This enables **simultaneously minimization of both sharpness and the loss value**, where $\\lambda$ controls the importance of the sharpness term. \n\n- If $\\lambda$ is very large, the objective function primarily minimizes $\\tilde{L}_S^{FS}(w)$, neglecting $L_S(w)$.\n- If $\\lambda=0$, the method reduces to minimizing only the standard loss $L_S(w)$. \n\nIn Fig.5 of our paper, we conduct an ablation study on $\\lambda$ to explore these scenarios. The result show that **both excessively small and large values of $\\lambda$ lead to suboptimal performance**, while a moderate $\\gamma \\approx 0.8$ achieves the best results. This highlights the importance of balancing loss minimization and sharpness control.\n\n\n\n\n\n> Q2: Why perturbation is computed only on $\\tilde{L}_S^{FS} (w)$ rather than $L_S^{FS}(w)$ in eq 7?\n\nThanks for your questions! \n\nTo clarify, let's first recall the definition of the objective function $L_S^{FS} (w)$:\n$$\nL_S^{FS} (w) = L_S(w) + \\lambda \\cdot \\tilde{L}_S^{FS} (w)\n$$\n\nwhere \n$$\n\\tilde{L}_S^{FS} (w) = \\max _{|| \\epsilon ||_2 \\le \\rho} \\sum _{i=1}^C (1 - \\pi_i)^\\gamma \\tilde{L}_S^i(w, \\epsilon)\n$$\n\nAs shown, the perturbation $\\epsilon$ appears **only** in the second term $\\tilde{L}_S^{FS}(w)$ and is **independent** of the first term $L_S(w)$.\n\n\nTherefore, when optimizing $L_S^{FS}(w)$, the perturbation is **computed exclusively over the inner maximization problem in $\\hat{\\epsilon}(w)$**. Specifically, we solve for the optimal perturbation $\\hat{\\epsilon}(w)$ as:\n$$\n\\hat{\\epsilon}(w) = \\arg\\max _{|| \\epsilon ||_2 \\le \\rho} \\sum _{i=1}^C (1 - \\pi_i)^\\gamma \\tilde{L}_S^i(w, \\epsilon)\n$$\n\n\n\n> Q3: What is the computational cost of baselines such as LA, FFT, and LIFT? If their cost is approximately half that of SAM variants, a fairer comparison would be to evaluate them with twice the number of training steps.\n\n**A3**: Thank you for your valuable suggestion! \n\nIndeed, the computational cost of baselines such as LA, FFT and LIFT is approximately half that of SAM-based variants. To provide a fairer comparison, we conduct additional experiments in which we extend the training epochs of these baseline methods to roughly match the total computational cost of the SAM-based methods. Specifically, we double the training epochs to **400 or 40** (2$\\times$ compared to the original 200 or 20) for these baselines, while keeping the SAM-based methods at **200 or 20** epochs. All other experimental settings remain consistent with those reported in the paper. These experiments are conducted on both CIFAR-100 LT and ImageNet-LT datasets. The results are summarized below.\n\nThe results clearly show that, under comparable computational budgets, Focal-SAM consistently outperforms the baseline methods across different datasets. This further demonstrates that Focal-SAM effectively enhances model generalization in long-tailed learning scenarios.\n\n\n\nBelow are the results:\n\n-----\n\n**CIFAR-100 LT**\n\n|Method|Epoch|IR100|IR200|IR50|IR10|\n|-|-|-|-|-|-|\n|CE|400|41.0|37.7|45.8|57.0|\n|**CE+Focal-SAM**|200|**44.0**|**39.6**|**48.1**|**60.9**|\n|LA|400|45.2|42.3|49.9|58.7|\n|**LA+Focal-SAM**|200|**50.7**|**46.0**|**54.5**|**63.8**|\n|FFT|40|76.4|74.2|79.8|85.1|\n|**FFT+Focal-SAM**|20|**81.6**|**79.0**|**83.9**|**87.3**|\n|LIFT|40|82.0|**80.3**|82.9|85.1|\n|**LIFT+Focal-SAM**|20|**82.4**|80.0|**83.2**|**85.4**|\n\n\n-----\n\n\n**ImageNet-LT**\n\n|Method|Epoch|Head|Medium|Tail|All|\n|-|-|-|-|-|-|\n|LA|400|61.6|45.7|31.1|49.8|\n|**LA+Focal-SAM**|200|**63.9**|**52.2**|**34.4**|**54.3**|\n|FFT|40|78.9|69.1|51.9|70.5|\n|**FFT+Focal-SAM**|20|**80.8**|**73.9**|**54.4**|**73.9**|\n|LIFT|40|79.4|75.9|72.1|76.7|\n|**LIFT+Focal-SAM**|20|**79.7**|**76.6**|**73.6**|**77.4**|", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "**Method:** \n- Why don't use $\\tilde{L}_S^{FS}(\\mathbf(w))$ (eq 5) as the main objective? Specifically, why manage $\\tilde{L}_S^{FS}(\\mathbf(w))$ as a penalty of standard loss as in eq 6, which will add an additional hyperparameter $\\lambda$? \n- Why perturbation is computed only on $\\tilde{L}_S^{FS}(\\mathbf(w))$ rather than $L_S^{FS}(\\mathbf(w))$ in eq 7?\n\n**Evaluation:** Empirical evaluation makes sense to me.", "theoretical_claims": "Yes, I checked Appendix B.", "experimental_designs_or_analyses": "Yes, I check Section 4 and Appendix D, E.\n\n - What is the computational cost of baselines such as LA, FFT, and LIFT? If their cost is approximately half that of SAM variants, a fairer comparison would be to evaluate them with twice the number of training steps.", "relation_to_broader_scientific_literature": "The paper proposes a SAM variant specifically for long-tailed classification tasks.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "(Summarize from above)\n\n\n1. Why don't use $\\tilde{L}_S^{FS}(\\mathbf(w))$ (eq 5) as the main objective? Specifically, why manage $\\tilde{L}_S^{FS}(\\mathbf(w))$ as a penalty of standard loss as in eq 6, which will add an additional hyperparameter $\\lambda$? \n2. Why perturbation is computed only on $\\tilde{L}_S^{FS}(\\mathbf(w))$ rather than $L_S^{FS}(\\mathbf(w))$ in eq 7?\n3. What is the computational cost of baselines such as LA, FFT, and LIFT? If their cost is approximately half that of SAM variants, a fairer comparison would be to evaluate them with twice the number of training steps.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yOJAVgEiyT", "reviewer": "Reviewer_bvws", "summary": "The paper proposes a new sharpness aware minimization (SAM) algorithm suited for long-tailed classification. It works by formulating SAM as a regularizer, and then applying different regularization strengths for each class. The authors show that this approach can be more computationally efficient than baselines. The authors also show theoretical generalization bounds on their approach. Experiments on standard benchmarks empirically validate the effectiveness of the method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: Please include standard errors over multiple runs for numerical results tables. The performance of Focal-SAM is often quite close to baselines, so establishing this statistical significance will be important.\n\n**A1**: Thanks for your valuable suggestion! \n\nIn the original paper, we report the results using the same fixed random seed for all methods to ensure a fair and consistent comparison. Following your advice, we have now conducted additional experiments using **three different random seeds** to further assess the effectiveness of our method. Specifically, we report the **average accuracy** along with the **standard error** across these runs. All other experimental settings remain the same as those described in the paper. These additional experiments are conducted on the CIFAR-100 LT datasets.\n\nFurthermore, we perform a **statistical significance** analysis using the **Mann-Whitney U test** to compare the performance of Focal-SAM against baseline methods:\n\n- Baselines marked with $^{**}$ indicate that Focal-SAM outperforms the corresponding baseline with a p-value $\\le 0.05$.\n- Baselines marked with $^*$ indicate that Focal-SAM outperforms the corresponding baseline with a p-value $\\le 0.10$.\n\nThe results shows that the average performance of Focal-SAM consistently surpasses other SAM-based methods across multiple runs. This further supports the effectiveness of the proposed Focal-SAM approach.\n\n-----\n\n**CIFAR-100 LT**\n\n| Method             | IR100               | IR200               | IR50                | IR10                |\n| ------------------ | ------------------- | ------------------- | ------------------- | ------------------- |\n| CE                 | 41.2$\\pm$0.3$^{**}$ | 37.7$\\pm$0.3$^{**}$ | 45.3$\\pm$0.4$^{**}$ | 57.9$\\pm$0.2$^{**}$ |\n| CE+SAM             | 41.9$\\pm$0.2$^{**}$ | 38.9$\\pm$0.2$^{**}$ | 46.5$\\pm$0.4$^{**}$ | 59.8$\\pm$0.1$^{**}$ |\n| CE+ImbSAM          | 42.8$\\pm$0.2$^{**}$ | 38.8$\\pm$0.3$^{**}$ | 47.7$\\pm$0.2        | 60.2$\\pm$0.3$^{**}$ |\n| CE+CC-SAM          | 43.0$\\pm$0.3$^{*}$  | 39.1$\\pm$0.1$^{**}$ | 47.5$\\pm$0.2$^{*}$  | 60.2$\\pm$0.2$^{**}$ |\n| **CE+Focal-SAM**   | **43.7**$\\pm$0.3    | **39.7**$\\pm$0.2    | **48.0**$\\pm$0.3    | **60.7**$\\pm$0.1    |\n| LA                 | 44.8$\\pm$0.2$^{**}$ | 42.0$\\pm$0.3$^{**}$ | 50.2$\\pm$0.2$^{**}$ | 59.3$\\pm$0.3$^{**}$ |\n| LA+SAM             | 49.6$\\pm$0.4$^{**}$ | 45.1$\\pm$0.4$^{*}$  | 53.1$\\pm$0.2$^{**}$ | 62.6$\\pm$0.2$^{**}$ |\n| LA+ImbSAM          | 47.6$\\pm$0.3$^{**}$ | 43.6$\\pm$0.2$^{**}$ | 52.4$\\pm$0.1$^{**}$ | 62.0$\\pm$0.3$^{**}$ |\n| LA+CC-SAM          | 50.1$\\pm$0.1$^{**}$ | 45.4$\\pm$0.2        | 53.4$\\pm$0.4$^{**}$ | 62.9$\\pm$0.2$^{**}$ |\n| **LA+Focal-SAM**   | **50.6**$\\pm$0.2    | **45.8**$\\pm$0.3    | **54.3**$\\pm$0.2    | **63.5**$\\pm$0.2    |\n| FFT                | 78.7$\\pm$0.2$^{**}$ | 76.1$\\pm$0.2$^{**}$ | 81.3$\\pm$0.1$^{**}$ | 85.5$\\pm$0.1$^{**}$ |\n| FFT+SAM            | 81.0$\\pm$0.1$^{**}$ | 77.6$\\pm$0.1$^{**}$ | 83.4$\\pm$0.2$^{**}$ | 86.7$\\pm$0.1$^{**}$ |\n| FFT+ImbSAM         | 80.4$\\pm$0.2$^{**}$ | 77.3$\\pm$0.1$^{**}$ | 81.7$\\pm$0.1$^{**}$ | 86.5$\\pm$0.2$^{**}$ |\n| FFT+CC-SAM         | 81.1$\\pm$0.1$^{**}$ | 78.2$\\pm$0.1$^{**}$ | 83.5$\\pm$0.1$^{**}$ | 87.0$\\pm$0.1$^{*}$  |\n| **FFT+Focal-SAM**  | **81.6**$\\pm$0.1    | **78.8**$\\pm$0.2    | **83.9**$\\pm$0.1    | **87.3**$\\pm$0.2    |\n| LIFT               | 81.9$\\pm$0.1$^{*}$  | 79.7$\\pm$0.1$^{**}$ | 83.0$\\pm$0.2$^{*}$  | 85.1$\\pm$0.1$^{*}$  |\n| LIFT+SAM           | 82.0$\\pm$0.2        | 79.7$\\pm$0.1$^{**}$ | 83.1$\\pm$0.1        | 85.2$\\pm$0.2        |\n| LIFT+ImbSAM        | 81.9$\\pm$0.1$^{*}$  | 79.8$\\pm$0.2        | **83.2**$\\pm$0.1    | 85.2$\\pm$0.1        |\n| LIFT+CC-SAM        | 82.0$\\pm$0.1        | 79.8$\\pm$0.1$^{*}$  | 83.1$\\pm$0.1        | 85.2$\\pm$0.1        |\n| **LIFT+Focal-SAM** | **82.2**$\\pm$0.1    | **80.1**$\\pm$0.2    | **83.2**$\\pm$0.1    | **85.4**$\\pm$0.2    |", "claims_and_evidence": "Yes, all claims are sufficiently supported both theoretically and empirically.", "methods_and_evaluation_criteria": "Methods and evaluation criteria are well chosen; the authors evaluate on 4 different benchmarks including ImageNet-LT.", "theoretical_claims": "Theoretical claims (specifically Theorem 3.1) appear correct and rely on standard techniques.", "experimental_designs_or_analyses": "The experimental protocol described in section 4.1 seems valid.", "relation_to_broader_scientific_literature": "The two key related works are ImbSAM and CC-SAM. ImbSAM divides classes into head and tail groups, unlike Focal-SAM which does not divide the classes. CC-SAM applies SAM individually to each class, making it much more computationally intensive than Focal-SAM which applies a single perturbation to the parameters.", "essential_references_not_discussed": "No; as far as I know all relevant literature is discussed.", "other_strengths_and_weaknesses": "The paper is overall well-written and presented with solid empirical and theoretical results.\n\nOn the other hand, the idea proposed in this paper is a fairly straightforward combination of existing ideas (focal-loss and SAM). Moreover, this paper will likely be of interest mostly to the community working on long-tailed classification. Both of these limit the overall potential impact of the paper.", "other_comments_or_suggestions": "Please include standard errors over multiple runs for numerical results tables. The performance of Focal-SAM is often quite close to baselines, so establishing this statistical significance will be important.", "questions_for_authors": "1. How much do the numerical results vary between runs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes Focal-SAM, a class-wise variant of Sharpness-Aware Minimization tailored for long-tailed classification. The method combines the focal loss weighting scheme with a shared perturbation strategy to efficiently approximate class-wise SAM using only three backpropagations. The reviewers appreciated the clear motivation, simplicity of the approach, and its strong empirical performance across multiple datasets. While the improvement over standard SAM is modest and the theoretical contributions are partly relegated to the appendix, the method is well developed  and offers practical value for imbalanced classification settings. I recommend acceptance.
Decision: Accept (poster)