Tilted Sharpness-Aware Minimization
Authors: Tian Li
Tianyi Zhou
Jeff Bilmes
Venue: ICML 2025 poster (2025)
Keywords: sharpness-aware optimization
exponential tilting
generalization
Primary area: optimization
TL;DR: We propose a theoretically principled optimization objective along with its algorithms that generalizes SAM and achieves better performance.

Abstract: Sharpness-Aware Minimization (SAM) has been demonstrated to improve the generalization performance of overparameterized models by seeking flat minima on the loss landscape through optimizing model parameters that incur the largest loss within a neighborhood. Nevertheless, such min-max formulations are computationally challenging especially when the problem is highly non-convex. Additionally, focusing only on the worst-case local solution while ignoring potentially many other local solutions may be suboptimal when searching for flat minima. In this work, we propose Tilted SAM (TSAM), a smoothed generalization of SAM inspired by exponential tilting that effectively assigns higher priority to local solutions that incur larger losses. TSAM is parameterized by a tilt hyperparameter $t$ and reduces to SAM as $t$ approaches infinity. We show that TSAM is smoother than SAM and thus easier to optimize, and it explicitly favors flatter minima. We develop algorithms motivated by the discretization of Hamiltonian dynamics to solve TSAM. Empirically, TSAM arrives at flatter local minima and results in superior test performance than the baselines of SAM and ERM across a range of image and text tasks.

Tilted Sharpness-Aware Minimization
Tian Li
Tianyi Zhou?
Jeffrey A. Bilmes
Abstract
be defined
min L(0) _ Σ Ma,;; 0). [E(M]
(1)
Sharpness- Aware Minimization (SAM) has been demonstrated to improve the generalization per- formance of overparameterized models by seek- ing flat minima on the loss landscape through op- timizing model parameters that incur the largest loss within a neighborhood. Nevertheless, such min-max formulations are computationally chal- lenging especially when the problem is highly non-convex. Additionally, focusing only on the worst-case local solution while ignoring poten- tially many other local solutions may be subopti- mal when searching for flat minima. In this work, we propose Tilted SAM (TSAM), a smoothed generalization of SAM inspired by exponential tilting that effectively assigns higher priority to local solutions that incur larger losses. TSAM parameterized by a tilt hyperparameter and re- duces SAM approaches infinity. We show that TSAM is smoother than SAM and thus easier to optimize, and it explicitly favors flatter min- ima. We develop algorithms motivated by the discretization of Hamiltonian dynamics to solve TSAM. Empirically, TSAM arrives at flatter local minima and results in superior test performance than the baselines of SAM and ERM across a range of image and text tasks.
In overparameterized models, however, minimizing ERM may arrive at a bad local minimum. To address this, one line of work focuses on minimizing the sharpness of final solutions, ensuring that the losses of parameters around local minima are uniformly small. One popular formulation is sharpness aware minimization (SAM), that optimizes over the worst-case loss over perturbed parameters (Foret et al., 2020). For perturbing region ||e|| S p where E IR,, the canonical SAM objective is defined as
min (²) max L(0 + c). #
(2)
|e||S
Typically, SAM is optimized by alternating between running gradient ascent (to find the max loss) and gradient descent steps (to minimize the max loss) on model parameters (e.g.. Foret et al., 2020; Andriushchenko & Flammarion, 2022). However, is difficult for such algorithms (and its variants) that rely on one or few steps of gradient ascent to find the exact perturbation that incurs the true max loss, the loss landscape can be highly non-convex and potentially non- smooth. Despite recent advancements on approximately solving the SAM objective (e.g.. Liu et al., 2022b;a; Zhuang et al., 2022), the min-max formulation itself overlooks many neighborhood regions that may also result large losses, leaving some loss surface near local minima sharp. For instance, we have computed the average loss across the neighborhoods of SAM solutions, and find that it is still higher than the ones obtained by our approach (Section 5.2).
1. Introduction
To this end, we propose generalized and smoothed variant of SAM inspired by exponential tilting and its widespread usage in probability and statistics. In optimization literature, it has also been used as an efficient min-max smoothing operator (Kort & Bertsekas, 1972). Tilted SAM (TSAM), parameterized by tilt scalar 2 0, is defined as
Empirical risk minimization (ERM) is classic framework for machine learning that optimizes for the average per- formance of the observed samples. For n training sam- ples { ]ie[m] (which may also contain label information), model parameters f Rd, and a loss function I(-), let ERM
t log (fclotoduu)))
1
'Uiiversity of Chicago University of Maryland, College Park Unniversity of Washington. Correspondence to: Tian Li <litian& uchicago.cdu>.
min L'(0) = #
= log (Eute) (c1110te)
1
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
where L(e + c) is defined in Eq. (1), jule) denotes an un- certainty probability measure for E Rd that can represent

---

Tilted Sharpness-Aware Minimization
uniform balls such as ||l| p (but other measures are pos-
sample the model perturbation (Section 4). We empirically demonstrate that TSAM results in flatter solutions and supe- rior generalization performance than SAM and its variants for deep neural networks including transformers on both image and text datasets (Section 5).
sible well). When t + oo and ju(e) takes ||e|| S p. (0) reduces to the SAM objective (0)... When & = 0, (0) reduces to the average loss over the perturbed neighborhood uu(e), i.e., E l + +)] where the expectation is taken with respect to the randomness of e (Formally proved in Ap- pendix B). When we set =0 0 and p - O, the TSAM loss is reduced the classic average empirical risk L(0). We may
2. Related Work
use [u))) E,, and E interchangeably when the meaning is clear from the context.
Sharpness-Aware Minimization. SAM regularizes over- parameterized models by considering adversarial data points that have large training errors (Foret et al., 2020; Zheng et al., 2021). The SAM variants, training dynamics, and applications in different models have been extensively stud- ied in prior work (Long & Bartlett, 2023; Foret et al., 2020;
TSAM provides smooth transition between min- max optimization (Eq. (2)) and min-avg optimization ming [L + <)]. The min-avg optimization has ap- peared in prior works known as pretage-eerturbed sharp- ness (Wen et al., 2022), noise-perturbed loss (Zhang al., 2024), or random smoothing (Duchi et al., 2012). The smoothness parameter of the TSAM objective increases as the value of increases, which suggests that it is easier to optimize than SAM (Section 3). As we formalize later, TSAM reweights gradients of neighboring solutions based on their loss values, which can be viewed as a soft version of SAM that assigns all the weights one single worst minimum. In addition to the benefits in optimization, rigor- ously considering many, opposed to one, neighbourhood parameters that incur large losses can result in improved generalization. We provide both theoretical characterization and empirical evidence showing that TSAM solutions are flatter than those of ERM and SAM. One line of the mostly related works have explored tilted risks to reweight different data points (Li al., 2023; Robey et al., 2022). In this work, we use the TSAM framework to assign varying priority to local minima in the parameter space.
[L
Bartlett et al., 2023; Andriushchenko & Flammarion, 2022:
Chen et al., 2024; Kwon et al., 2021; Chen et al., 2021;
Liu al., 2022b; Zhou et al., 2021; Du et al., 2022; Back et al., 2024; Zhao et al., 2022; Mi et al., 2022; Zhuang et al., 2022; Mueller et al., 2023; Xie et al., 2024; Tahmasebi et al., 2024). Notably, TSAM objective can be viewed as a spe- cial case of general sharpness-aware approach (Tahmasebi et al., 2024), though Tahmasebi et al. (2024) do not specifi- cally study TSAM. Some work aim to improve efficiency of the SAM algorithm studying different relaxations (Du et al., 2022; Liu et al., 2022a). Zhao et al. (2022) use linear interpolation between normal gradients and SAM outer gradients evaluated at the max-loss parameter, which does not take into account the possibly many bad local min- ima for highly non-convex problems. Zhou et al. (2021) perform sample-wise reweighting for SAM, as opposed to parameter-wise reweighting proposed herein. Liu et al. (2022b) improve the inner max optimization by adding random perturbation to the gradient ascent step to smoothen its trajectory. Li & Giannakis (2023) leverage a moving average of stochastic gradients in the ascent direction to reduce the gradient variance'. Our goal is not to better ap- proximate the inner max or develop algorithms for solving the min-max SAM formulation, but rather, to solve dif- ferent TSAM objective that reweights many local minima to seek flat solutions. Nevertheless, we still compare with more advanced algorithms for the SAM objective and show the superiority of TSAM solutions (Section 5). As TSAM is a new objective, in principle, we can readily apply many existing optimization techniques (that can be potentially ap- plied SAM as well) such as variance reduction (Johnson & Zhang, 2013), acceleration (Nesterov, 1983), adaptiv- ity (Streeter & McMahan, 2010; Duchi et al., 201 I; Kingma & Ba, 2014) on top of the tilted stochastic gradients to gain further improvement.
To solve TSAM, we need to estimate the integral over ju(e) (Eq. (3)), or equivalently, to estimate the full gradient of the objective, which is a tilted aggregation of gradients evalu- ated at L(0+ + +c. Both require sampling the perturbation with probability proportional to (( for the integration. Naively sampling at random to obtain L'( would be inefficient, as it is likely that [. + <) under the sampled is small and therefore we need many samples to converge to the true distribution. On the other hand, methods based on Hamiltonian Monte Carlo (HMC) (Leimkuhler & Re- ich, 2004) are guaranteed to arrive at the exact distribution. Inspired by the Euler's rules for HMC, we develop an algo- rithm to efficiently sample e's and estimate the true gradient of T. (O)) Our contributions are summarized as follows.
Contributions. We propose TSAM, new sharpness- aware optimization objective that reweights the parameters around local minima via exponential tilting. We rigorously study several properties of TSAM, showing that it always fa- vors flatter solutions as f increases (Section 3). To optimize TSAM, we adapt a specific HMC algorithm to efficiently
'Note that the notion of 'variance' in VASSO (Li & Giannakis, 2023) refers to the variance of stochastic gradients compared with full gradients; whereas in TSAM, we examine the loss variance around the neighborhood regions, where the randomness in vari- ance (Definition 3.4) comes from the perturbation e.
2

---

Tilted Sharpness-Aware Minimization
There are also works exploring why SAM leads better gen- eralization or theoretically studying what SAM (and its im- plementation) is effectively minimizing (Chen et al., 2024; Andriushchenko & Flammarion, 2022; Long & Bartlett, 2023; Wen et al., 2022). In this work, we prove that TSAM (and SAM) encourages flatter models for a class of prob- lems including generalized linear models, where flatness (or sharpness) is characterized by the variance of the losses around the minima (Definition 3.5). Our proposed TSAM framework is particularly suitable for problems where flat- ness helps generalization. The various notions of sharpness, along with theoretical relations between sharpness and gen- eralization still remain an open problem (Andriushchenko et al., 2023; Wen et al., 2024; Ding et al., 2024; Tahmasebi et al., 2024), which is outside the scope of our paper.
prove that there exists ( E (0, oo) that result in the tightes bound (Section 3.3). All properties discussed in this section hold regardless of the distributions of e (i.e., choice of u(e)), unless otherwise specified.
3.1. Convexity and Smoothness
In this part, we connect the convexity and smoothness of TSAM with the convexity and smoothness of the ERM loss. We provide complete proofs in Appendix B. We first define a useful quantity (tilted weights) that will be used throughout this section.
Definition 3.1 (t-tilted weights). For a perturbed model parameter 0 + c, we define its corresponding t-tilted weight as w (0 + e) := .1((014)
Tilting in Machine Learning. Exponential tilting, used to shift parametric distributions, has appeared in previous literature in importance sampling, optimization, control, and information theory (e.g., Siegmund, 1976; Kort & Bert- sekas, 1972; Dembo, 2009; Aminian et al., 2024; Whittle, 2002; 1981). Recently, the idea of tilted risk minimization (which exponentially reweights different training samples) has been explored in machine learning applications such as enforcing fairness and robustness, image segmentation, and noisy label correction (Li et al., 2023; Robey et al., 2022: Zhou et al., 2020; Szabó et al., 2021; Aminian et al., 2024). A closely-related LogSumExp operator is often used to as an smooth approximation to the max, which is always considered more computationally favorable (Kort& & Bert- sekas, 1972; Calafiore & El Ghaoui, 2014; Shen & Li, 2010; Li et al., 2023). One application of tilted risks applied to the adversarial training problem is to balance worst-case robustness (i.c., adversarial robustness) and average-case robustness in the data space (Robey et al., 2022), among other approaches that can also achieve a transition between worst-case and average-case errors (Rice et al., 2021). Our work is similar conceptually, but we consider reweighting adversarial model parameters, instead of adversarial data points. Compared with SAM (optimizing the largest loss), the TSAM framework offers additional flexibility of optimiz- ing over quantiles of losses given the connections between exponential tilting and quantile approaches (Rockafellar et al., 2000; Li et al., 2023).
The weight of parameter + is exponentially proportional to the loss evaluated this point. The expectation is with respect to the randomness of constrained by p(e). When
= O, O-tilted weights are uniform. When t - co, w (0+c) focuses on the max loss among all possible (0 + <). Such weights have appeared in previous literature on importance sampling (Siegmund, 1976), but they are only applied to reweight sample-specific losses, as opposed to perturbation- specific parameters. Given tilted weights in Definition 3.1, we can present the TSAM gradients and Hessian as follows.
Lemma 3.2 (Gradient and Hessian for TSAM). Assume L(.) is continuously differentiable. The full gradient of TSAM (Objective (3)) is
(0) Ele VI." + e)] Blw ()+)) +]].
Bleth10+))
The Hessian of TSAM V2L is
t {E [w'(0 + ())(+++((((++++ (
-E
T E [w'(0 + c) + +I)
+ E [=((0+c\[<(++e]]
(4)
The gradient of TSAM can be viewed as reweighting the gradients of V + c) by the loss values clL((++) Exam- ining the Hessian, we note that the first term & multiplied by positive semi-definite matrix, and the second term can be viewed as reweighted Hessian of the original loss L evaluted at f + c.
3. Properties of TSAM
It is not difficult to observe that if T-(0) is p-Liischitz with respect to 0, then (0()) is p-Lipschitz with respect to 0. If L is p-strongly convex, then L' is also p-strongly convex (proof in Appendix B). Next, we show that the smoothness of TSAM scales linearly with t.
In this section, we discuss properties of the TSAM objec- tive. We first state the convexity and smoothness of TSAM (Section 3.1). We then show that as increases, the gap between less-flat and more-flat flat solutions of the TSAM ob- jective becomes larger. In other words, optimizing TSAM would give flatter solutions increases (Section 3.2). Fi- nally, we discuss the generalization behavior of TSAM and
Lemma 3.3 (Smoothness of TSAM). Let L(.) be B-smooth and 3 is bounded. Then L' L'(0) is ((t)-smooth, where 3(1)
satisfies 0 < <lim f < +00.
B(()
3

---

Tilted Sharpness-Aware Minimization
That is, =(1)=(()) The proof is deferred to Appendix B. In Lemma 3.3, we connect the smoothness of the tilted ob- jective with the smoothness of the original ERM objective. We see that for any bounded f. the smoothness parameter is bounded. As increases, TSAM becomes more difficult to optimize as the loss becomes more and more non-smooth. When 1 + xo, 3(1) + oc. If we have access to unbiased gradient estimates at each round, it directly follows that the convergence of SAM (TSAM with t - oo) objective is slower than that of tilted SAM following standard argu- ments (Nesterov, 2013). To further visualize this, we create a one-dimensional toy problem in Appendix A, where we obtain the globally optimal solutions for each objective. We show that both SAM and TSAM are able to arrive at flat solutions; but the SAM objective is non- non-smooth, hence more difficult to optimize.
obtained from different objectives, and observe that TSAM achieves smaller sharpness values (measured in terms of standard variance) than ERM and SAM. Proper definitions of sharpness is generally still an open problem, and other options are possible such as the trace of Hessian, gradient norms, and other functions over Hessian (Wen et al., 2022; Tahmasebi et al., 2024).
Theorem 3.6 (TSAM prefers flatter models as & increases).
Assume L( is given by Eq. (6) and ((0) is continuously dif- ferentiable. For any 01, 02 E IR,, let (01. 82) == L'()))
L' (02).. 1f is t-sharper than 0z, then 8g" (8' .02) 2 0.
8t
For some 0 sharper than Oz. it is possible that T((O:) L(02), which implies that ERM is not able to distin- guish between the two solutions, while TSAM can. Fur- thermore, Theorem 3.6 indicates that as 1. increases, the TSAM objective favors 0g more aggressively, as the gap between ('():) and L' (02) grows larger. We explore a oinedimensional toy problem with many local minima: (0)) = sin ( + 0.005 1)2 and focus on the area 0 E (0.2, 2.5) to visualize this behavior. We take p = 0.2 and a fixed learning rate 0.005 for all objectives. Each run starts from the same initialization & = 0.5. In Figure I, we see that as t increases, TSAM leads to flatter solutions, despite having larger objective values measured by L()). As a side note, we prove that for any 0 the objec- tive value of L (0) is monotonically increasing as f increases (Appendix B). Next, we discuss special case when t is close to O, where we provide another perspective on the TSAM behavior.
3.2. TSAM Prefers Flatter Models as Increases
In this subsection, we focus on a specific class of models including generalized linear models (GLMs), where the loss function (zz;; 0) carries the form of
((x;; 0) = A(0) - 0 T(x;).
(5)
1 Σ T(xi) n iE[m)
L(0) = A(0) TT
(6)
For GLMs, A(O) is convex function, and EiE(n] (x.) > 0 (Wainwright & Jordan, 2008). Our results in this section apply to loss functions defined in Eq. (6). which subsume linear models. Before introducing the main theorem, we define two important quantities that will be used throughout this section, and in the experiments.
Discussions for t - O. All the results above hold for the small-t regime, where sharpness reduces to standard variance when - 0 (Definition 3.4). It still follows that 0g' (0).0))) 0t t vide another interpretation of TSAM when i is close to zero Similar statements have also appeared in prior works in a dif- ferent context (e.g., Liu & Theodorou, 2019; Li et al., 2023). For very small t. it holds that 1 log (E [ct1(++0]) z E + c)] + 2 var (I.(0 + c)) + ott)). We provide a proof in Appendix B. Hence, optimizing TSAM is approximately equivalent to optimizing for the mean plus variance of the losses under the perturbed parameters. When t = 0, it re- duces to only optimizing for E E + c)]. In other words, TSAM with close to is directly minimizing O-sharpness (standard variance). For any 0, and 0g such that 01 sharper than 02. we have
Definition 3.4 (t-weighted mean and variance). We define [ "|exxT Simi-
2 0 if 0, is sharper than 0z. Here, we pro-
I-weighted mean of random variable X as larly, we define t-weighted variance of a random variable X as Blexx x2 B[e
When f = O, these definitions reduce to standard mean Similar tilted statistics definitions have also appeared in prior work (Li et al., 2023). We leverage weighted variance to define sharpness below.
[[X] and variance
Definition 3.5 (t-sharpness). We say that model param- eter is 02 is t-sharper than 6g if the t-weighted variance of [( + é) (which is a random variable of loss distribution under model parameters perturbed by e) is larger than the (-weighted variance of [(02- + +)).
g' 02) R ([ + +c)] + var (T.(0: + +c))
Given the definition of sharpness above based on weighted variance, we are ready to prove that TSAM encourages flatter local minima as the increase of t. Empirically, in Section 5, we also plot the O-sharpness of the solutions
t
E[L((22 + c]] 2
var (L(0g + c)),
(0' Øt 2
(var (L(6) + c)) var (L(Ag + c))) 2 0.
4

---

Tilted Sharpness-Aware Minimization
ERM
TSAM (bw4)
TSAM (t=20)
TSAM (t=100)
1-
15
2.0
0.5
1.0
2.
255
1.5
2.0
2.5
100
Figure 1. Optimization traecctoiies of different objectives (orange) with final solutions marked in star. The local minima gets flatter from left to right in each subfigure. We see that TSAM favors flat minima as increases. ERM solution by gradient descent converges to a sharp minimal. We solve TSAM on this one-dimensional problem by sampling thousands of e's estimate the gradient of L' (0) each step, which is infeasible for real problems.
This is special case of Theorem 3.6 for ( + O. It suggests that as we increase t from 0 for a small amount, the standard variance of neighborhood loss would reduce. Note that some recent works propose a noise-injected loss similar to TSAM with f - (Zhang et al., 2024). The proposed algorithm therein explicitly minimizes the trace of Hessian, which aligns with our arguments that the TSAM objective can lead to flatter solutions.
where c= (L. is a constant unrelated to L.
We defer the proof to Appendix B, where we build upon ex- isting generalization results of a related objective (Aminian et al., 2024). From Theorem 3.7, we see that when the sam- ple space of is empty, our result reduces to E 2)] <
L(0) + Mv log(2/5) scaling at rate of 1 consistent Y
2n
with standard uniform bound on the average risk (Shalev- Shwartz & Ben-David, 2014). When 1 + oo and we de- fine ple) to be ||el| I p over some distribution, the re- sult gives an upper bound on the generalization of SAM:
So far, we study properties of TSAM regarding convexity, smoothness of the objective, and sharpness of the resulting solutions. We note that these properties of the objective are independent of the actual optimization algorithms used to optimize TSAM. Though Theorem 3.6 implies similar benefits of both TSAM and SAM relative ERM (assum- ing optimize TSAM and SAM perfectly), Lemma 3.3 shows the superiority of the TSAM objective over SAM with unbounded smoothness parameters, as TSAM is eas- ier to optimize. The performance of practical applications depends on both the properties of the objectives and the approximation algorithms used solve them.
Ez [10 Z)] - ma L(0 + c) S MV
log(2/8) 2m
+
Additionally, denote (TSAM and @ERM as optimal solu- tions for TSAM (Eq. (3)) and ERM (Eq. (I)), respec- tively. For modest values of t, due to the negativity of var, (01219 22 the upper bound of the linear population 2teaest
risk Ez EZ(U(TTSMM, 2)] can be smaller than that of the lin-
ear risk Ez ((OERMM .Z)], as long log E, [e ( +00g-
var, (e122C015MM I8AM +.))
< ((OERM). This implies that by solving
3.3. Generalization of TSAM
TSAM, we can obtain a solution that results in a smaller upper bound of the linear population error than that of ERM.
In this section, we give a uniform bound on the gener- alization error of our TSAM objective. By solving the tilted objective empirically, during test time, we are ulti- mately interested in evaluating the linear population risk Eg (6;; Z)] where Z denotes the underlying data distribu- tion and Jejm ~ Z. We define generalization error as the difference between population risk and our empirical objective value [z; Z)] - t log E, [e (L))+))), bounded as follows.
4. Algorithms
In this section, we describe the algorithms we use to solve TSAM. The main challenge in solving TSAM is to sample to get a good estimator of L'(O), or equivalently, V (O). We first describe a general approach where we use estimated tilted gradients (given sampled c's) to update the model (Section 4.1.. Then, we discuss how to sample e's via a specific Hamiltonian Monte Carlo algorithm and present our method and implementation (Section 4.2).
Theorem 3.7 (Generalization of TSAM). Assume losses are bounded as 0 < L(.) < M. Suppose we have n training data points. For any E θ and t 2 0, with probability 1 1-ô, the difference between population risk and empirical TSAM risk gen, := [0;; 2)] 1t2 log E, (e2100))) satisfies
4.1. General Algorithm
To solve TSAM, the primary challenge is to estimate the integral & llog (f
log(2/8)
its full gradient assuming madient-bbsed methods and B B.
gen, S M
+ +c,
(7)
2n
21e21M
5

---

Tilted Sharpness-Aware Minimization
the differentiable loss L. A naive way is first sample from ja(e) following the pre-defined distribution (e.g., Gaus- sian or uniform) over uue), and then perform tilted aggrega- tion with weights proportional to e c(1.(0+)), However, this approach may be extremely inefficient, as there could be an infinite set of perturbed model parameters with relatively small losses, which are not informative. In Figure 5 in the appendix, we empirically show that even when we sample much larger number of c's, the resulting accuracy is still worse than our proposed method. Instead, we propose to sample 8 number of e's from distribution ( (denoted as fe()) where 0 S. 5 S t. We then use these JEl] to obtain an empirical gradient estimation with weights pro- portional to (e(t=8)((0+s)) JE[e] as the full gradient a tilted average of the original gradient on L(-). To improve sample efficiency, we use gradient-basee methods such as Hamiltonian Monte Carlo (HMC) that simulates Hamilto- nian dynamics (Leimkuhler & Reich, 2004). The structure of our proposed method is in Algorithm I. Note that in principle, after estimating the tilted stochastic gradients, we can further apply existing optimization techniques such as variance reduction (Johnson & Zhang, 2013), accelera- tion (Nesterov), or adaptivity (Streeter & McMahan, 2010; Duchi et al., 201 1) to gain further improvement, which we leave for future work.
Euler's s approach as follows. We first initialize eo from an Lg ball that satisfies ||c]| I p. and initialize the momentum po R from some Gaussian distribution, i.e. Po _N(0, Note that the negative log probability density of the energy function (e)) is - log(e = -5T(0 + e). At each sampling step, we run the following steps for N iterations with a small step-size 3 to obtain a candidate e:
p + p+ 35V. + t), 6 1-e+1 Bp/o².
(8)
After obtaining a candidate c, we accept with probability min e ( -0- / Lzall 20 }. If the candidate
is not accepted, we set (p, e) the initial point before the N iterations. Repeating the above for enough times would give us sample from the exact distribution.
Algorithm 3 Sampling from eAT(00++() where |<|| - P
Require: 80, total samples s. uncertainty ball radius p I: for j = 0.... do
2: Perturb 8 with random 8, sampled from Gaussian
or uniform distribution: 03 < 0' + 6y
3: Run normalized SGD on the mini- batch data at f):
VL()))
8 0} +
9 0 8¹
VL (8)ll
Algorithm 1 Tilted SAM Solver
4: end for
5: Return (+)))(())
Require: f, 80, learning rate 17, total iterations T, total
number of samples 0 S 6 2st
1: for = 0, ...... T - do
Generating one e via HMC requires at least 2N gradient evaluations, which is infcasible for large-scale problems. Hence, we set N = in all the main experiments, and meanwhile accept the generated e with probability 1. For completeness, evaluate the effects of increasing N in HMC in Appendix C. We observe that using N > does not significantly improve the performance. Running equations in Eq. (8) for one step. if p is initialized as p = 0, we have E e+ L + e). where 3' is a constant. We adopt this updating rule in our problem, and run the aforementioned procedure in parallel for s times to get s samples. Our method is presented in Algorithm 3. Though Algorithm 3 does not guarantee the e, 's result in a consistent estimator of the TSAM integral, we empirically showcase its effec- tiveness on non-convex models including transformers in the next section.
2: Sample random perturbations {er}je(s) from distri- bution ( under the constraint characterized by ple) via some HMC algorithm (Algorithm 3)
3: Update& 8 with the estimated gradient evaluated the mini-batch:
g++1 0¹ 17 EjEts] el (t-8))(0++ +5) VL + +ez)
EjE(t) c((-6)(((++))
4: end for
5: Return 0r
4.2. Sampling
There could be potentially different algorithms for sampling e where p(e) x (0((e)) Here we propose an approximate and cheap sampler based on discretization of Hamiltonian dynamics. Our method is inspired by one of the best-known way to approximate the solution to a system of differential equations, i.e., Fuler's method or its modification (Neal et al., 2011). A more accurate solver like the Icap-frog method might be more popular for HMC, but these come at an increased expense (Neal et al., 2011). As our goal to minimize computational cost, we stick with the cheaper
5. Experiments
In this section, we first describe our setup. Then we present our main results, comparing TSAM with the baselines of ERM (Eq. (L)). SAM (Eq. (2)), and SAM variants on both image and text data (Section 5.1). We explain TSAM's su- perior performance by empirically examining the flatness of local minima in Section 5.2. In Section 5.3, we discuss the
6

---

Tilted Sharpness-Aware Minimization
effects of hyperparameters. Our code is publicly available giihuh.com/llttag6//SS.
along the gradient ascent directions and pick the one incur- ring the biggest loss. Then evaluate the gradient under that step size to be applied to the original model parame- ters. We call these expensive SAM baselines ESAMI, and ESAM2, respectively. We also evaluate two more advanced sharpness-aware optimization methods: PGN that combines normal gradients and SAM gradients (Zhao et al., 2022), and Ramdom SAM (RSAM) which adds random perturbations before finding the adversarial directions (Liu et al., 2022b). We let PGN and RSAM run the same amount of time as TSAM on the same computing platform. On all the datasets, we tuned t values via grid search from {0, 1, 5, 20, 100}.
Tasks and Datasets. We study image classification tasks with CNNs and transformers, and language modeling tasks with BERT. First, we explore training ResNet18 8 (He et al., 2016) and WideResNet16-8 (Zagoruyko, 2016) on CI- FARI00 (Krizhevsky et al., 2009). Previous works show that SAM is robust label noise (Foret et al., 2020; Back et al., 2024); and we investigate this setting by training the same models on on CIFAR100 with label noise generated by substituting 20% of the true labels uniformly at random to other labels. Since vision transformers (ViTs) (Dosovit- skiy et al., 2020) have been shown to result in sharper local minima than CNNs (Chen et al., 2021), we study the perfor- mance of finetuning ViTs (pretrained on ImageNet (Deng et al., 2009)) on auttof-distribution Describable Tex- ture Dataset (DTD) (Cimpoi et al., 2014), where the task is 47-class classification. We also use WedeResNet16-88 to train DTD from scratch. Additionally, we evaluate a 200-class classification task for Tinylmagenet (Le& & Yang, 2015) with & and ResNet34 (He et al., 2016) mod- els. Lastly, for text data, we study finetuning& a pretrained DistilBERT (Sanh, 2019) model on the GLUE benchmark including both classification and regression problems.
Our results are shown in Table 1. The performance for all I's on three image datasets and different model architectures are reported in Section 5.3. For the GLUE benchmark, we report the standard metrics for each dataset in GLUE TSAM consistently achieves higher test performance than ERM and variants of SAM. We provide corresponding convergence plots of ERM, vanilla SAM, and TSAM in Appendix C.
5.2. Flatness of TSAM Solutions
In this part, we take more detailed look into the properties of TSAM solutions compared with the ones of ERM and SAM on the CIFAR100 dataset trained by ResNet18 from scratch. In Figure 2, we plot the loss mean and variance over the neighborhood areas around local minima obtained by different objectives, i.e., [[ + +c] and var [(0 + +)], where E ~ (0,, 5²). and 0 denotes the different solutions of any objective (with a slight abuse of notation). These measurements have appeared in prior works named average- loss sharpness (Wen et al., 2024; Chen et al., 2021), and are consistent with our sharpness definition (Definition 3.5) mentioned before. In Figure 2, for all 5 values, we see that AM consistently result in flatter local minima than ERM and SAM measured by both the mean and variance of losses around the minima. In addition, we evaluate sharpness following other common notions by investigating the top-5 eigen values of Hessian (e.g., Foret et al., 2020). Under the same model setup, the top-5 eigenvalues are (342.11 I, 304.72, 260.71, 252.92, 210.88} for ERM, {232.60, 198.35, 182.61, 153.74, 145.76) for SAM, and (140.91, 1 113.38, 105.90, 92.94, 89.55) for TSAM (t=20). We see that TSAM achieves the smallest max eigenvalues among the three.
Hyperparameter Tuning. We take pac) to be ||<|| S p for all TSAM experiments, and tune the parameters separately from {0.05, 0.2) for relevant methods. For TSAM, we tune from {0, 1, 5, 20, 100) and select the best one based on the validation set. We also report the performance for all t's in the next sections. We use s=3 or s=5 sampled c's for all datasets and find that it works well. For some SAM variants that introduce additional hyperparameters, we tune those via grid search as well. The batch size is 64 for all the datasets and methods and a constant learning rate is tuned from {0.0003, 0.001, 0.003, 0.01, 0.03, 0.1} for each algorithm. Despite the existence of adaptive methods for SGD and SAM (Kingma& & Ba, 2014; Kwon et al., 2021), we do not use adaptivity for any algorithm for fair comparison. See Appendix C for details on hyperparameter tuning.
5.1. TSAM Leads to Better Test Performance
We compare the performance of various objectives and algo- rithms in Table 1. ERM denotes minimizing the empirical average loss with batch SGD. SAM is the vanilla SAM implementation with one step of gradient ascent and one step of gradient descent at each iteration (Foret et al., 2020). Note that TSAM requires more gradient evaluations per iter- ation. Hence, we include two additional baselines of SAM under the same computational budget TSAM runs. (1) We simply run the vanilla SAM algorithm for more iterations until it reaches the same runtime as TSAM. (2) We try an- other SAM approximation by exploring different step sizes
We further report the training and test performance of best- tuned ERM, SAM, and TSAM in Table 3 in the appendix. We show that ERM solutions have lower training losses but higher test losses than SAM and TSAM when evaluated on the average test performance (i.e., the 'ERM' column in the right table). This is due to the fact that ERM does not generalize as well as SAM or TSAM, and there exist bad sharp local minima around ERM solutions. On the other hand, while TSAM's average training loss is the highest
7

---

Tilted Sharpness-Aware Minimization
Table TSAM achieves higher test performance relative ERM and different variants of SAM across image datasets and the GLUE benchmark with both CNNs and transformers. TSAM (or SAM) is particularly suitable for applications with distribution shifts (DTD and noisy CIFARIO0 datasets), which also consistent with observations prior works (Foret al.. 2020; Back al., 2024). TSAM also results in lower test loss, discussed in detail in the next section.
CIFARI00
DTD
Noisy CIFAR100
Tiny Imagentt
Methods
R8
WideResNett
ViT
WideResNet
ResNet18
WideResNet
Re
ResNer33
ERM
71.39 (.2)
73.22 (:2)
66.38 (.2)
16.97 (.3)
61.01 (.2)
57.03 (.3)
71.10 (.2)
74.90 (.1)
SAM
76.52 (.1)
78.44 (11)
67.87 (.2)
17.45 (.2)
69.00 (.1)
68.02 (22)
72.43 (.1)
76.75 (.2)
ESAMI
(.1)
(1)
68.18 (.2)
17.67 (.2)
69.20 (.)
69.79 (11)
(.1
77.41 (1)
ESAM2
77.52 (.1)
79.03
68.35 (.1)
17.71 (1)
67.27 (.1)
66.83 (1)
(.126
77.40 (.1)
PGN
77.45 (.1)
78.58 (.2)
67.76 (.1)
18.23 (22)
65.68 (.1)
64.02 (2)
73.18 (.1)
77.80 (.1)
RSAM
77.35 (.))
79.02 (.2)
68.35 (.2)
17.66 (.3)
69.31 (.1)
65.93 (.2)
73.57 (.1)
77.72 (1)
TSAM
77.78 (.1)
80.85 (.2)
68.82 (.1)
18.63 (.2)
69.98 (.1)
70.26 (.I)
73.55
77.79 (.)
Objectives
CoLA
WNLI
SST-2
MNLI
QNLI
RTE
MRPC
QQP
STSB
AVG
ERM
52/80.34
54.93
90.48
79.6
87.72
60.65
83.82
86.32
86.6/86.3
77.15
SAM
52/80.48
56.34
91.74
81.1
86.42
58.84
85.29
87.71
87.0/86.5
77.56
ESAMI
52/80.44
56.34
91.63
81.1
86.18
59.02
85.31
87.69
87.1/86.7
77.59
ESAM2
52/80.53
56.34
91.63
81.2
86.29
59.25
85.80
87.47
86.8/86.5
77.62
ISAM
52/80.81
56.34
91.86
81.1
87.81
60.65
85.05
88.77
87.1/86.6
78.01
charphess imsanaad by ABIE" 451
aharanass www.soadddyyyaaaoooooo +10
report the test accuracies the training proceeds under multiple values of t's for all the three tasks. Results are plotted in Figure 7 in the appendix. We see that there are range of t's that result in faster convergence or higher accuracies than SAM. There also exists an optimal t that leads to the best test performance. This is consistent with our previous generalization bound (Section 3.3). Though Theorem 3.6 captures the benefits of both SAM and TSAM, note that the final empirical performance does not only depend on the properties of the objectives. But rather, it also relies on the choice of approximation algorithms. Results in Theorem 3.6 assume that the objectives are optimized perfectly, which infeasible in high-dimensional settings.
ERM
SAM
15AM
ISAM
10-
0.2
0.34 0.5
0.55
9.52
300
3.20
Figure 2. Sharpness of the solutions found by ERM, SAM, and TSAM on CIFARI00 with ResNet18. We empirically measure sharpness by both [[ + <)] and var[L(0" + e)] where e ~ (0, )²). 8 denotes different optimal model parameters obtained from the three objectives. These metrics (especially variance) are also consistent with Definition 3.5 with
We see that TSAM solutions have a flatter neighborhood compared with the other two.
Additionally, we empirically study the effects of scheduling during optimization. Increasing from 0 to fixed value effectively switches from weighting local minima uniformly to rewighting them based on the loss values, and vice versa. We experiment with two options: linearly decreasing and linearly increasing t on the noisy CIFAR 100 dataset trained by ResNet18. The convergence curves are shown in Figure We see that using a fixed t throughout training does not have significant difference from scheduling L. Hence, stick to fixed f's for our TSAM experiments.
(which is expected because it does not directly optimize over ERM), the test losses of TSAM for both the average-case performance and worst-case performance are lower than the other two baselines. While we show better generalization of TSAM empirically, rigorous understandings between generalization and flatness remains an open area of research.
5.3. Sensitivity to Hyperparameters
Effects of the Tilting Hyperparameter t. One critical hyperparameter in TSAM is f. When t 0, TSAM objec- tive reduces to the average-case perturbed objective. When t. + 0o, the TSAM objective (Eq. (3)) recovers SAM (Eq. 2). But the TSAM algorithm (Algorithm 3) do not exactly recover SAM's alternating updating approximation when po. See Section for detailed discussion. Here,
Effects of the Number of c's. One may wonder if we need to sample a large number of perturbations for the algorithm to be effective. In Table 2, we show that we usually only need s = 3 or 5 number of c's to achieve significant im- provements relative to SAM. The models in three columns correspond to ResNet18, VIT, and ResNet18, respectively.
8

---

Tilted Sharpness-Aware Minimization
Andriushchenko, M., Croce, E, Müller, M., Hein, M., and Flammarion, N. A modern look at the relationship between sharpness and generalization. arXiv preprint arXiv:2302.00011, 2023.
CI
DTD
noisy CIFARI00
SAM
76.52
67.87
69.00
TSAM, s=3 77.40
68.82
69.55
TSAM, 5=5
77.78
68.70
69.98
Baek, C., Kolter, Z., and Raghumathan, A. Why is sam robust to label noise? arXiv preprint orXiv:2405.03677 2024.
Table 2. Test accuracies of SAM and TSAM with different number of sampled c's (denoted as 8: see Algorithm 3). Empirically, we do not need many samples from the tilted distribution.
Bartlett, P. L., Long, P. M., and Bousquet, O. The dynam- ics of sharpness-aware minimization: Bouncing across ravines and drifting towards wide minima. Journal of Machine Learning Research, 24(316):1-33) 2023.
6. Conclusion
In this work, we have proposed a tilted sharpness-awee aware min- imization (TSAM) objective, which leverages exponential tilting (parameterized by () to reweight potentially many lo- cal minima in the neighborhoods. TSAM is more smooth problem relative to SAM with bounded r, and it explicitly encourages flatter solutions as increases. We have pro- posed a practical algorithm motivated by HMC to sample from the tilted distribution c+(()+)) Through experiments on different models and datasets including label-noise set- tings, we have demonstrated that TSAM consistently outper- forms SAM and its variants both image and text datasets.
Boucheron, S., Lugosi, G., and Massart, P. Concentration Inqqualities: A Nonasymptotic Theory of Independence. Oxford University Press, 2013.
Calafiore, G. C. and El Ghaoui, L. Optimization Models. Cambridge University Press, 2014.
Chen, X., Hsieh, C.-J., and Gong, B. When vision trans- formers outperform resnets without pre-training or strong data augmentations. arXiv preprint arXiv:2106.01545 2021.
Acknowledgements
Chen, Z., Zhang, J., Kou, Y., Chen, X., Hsieh, C.-J,, and Gu, Q. Why does sharpness- aware minimization gener- alize better than sgd? Advances in Neural Information Processing Systems, 36, 2024.
We thank Ahmad Beirami, Behrooz Tahmasebi, and Manzil Zaheer for helpful discussions. We thank the anonymous reviewers for their feedback.
Cimpoi, M.. Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Proceed- ings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Impact Statement
In this work, we propose tilted sharpness-aware minimiza- tion (TSAM) that leverages exponential tilting to reweight sharp local minima and achieve better generalization. It could potentially improve the test performance of a wide range of models including those designed address soci- etal problems such as faimess, robustness, and privacy. As the algorithmic framework is general, we do not anticipate our objective or algorithm to directly cause any negative impacts. However, TSAM is applied in an undesirable manner, c.g., to some malicious attacking approaches, it may strengthen the capabilities of those models.
Dembo, A. Large deviations techniques and applications. Springer, 2009.
Deng, J., Dong, W., Socher, R., L.-J,, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-25525 Ieee, 2009.
Ding, L., Drusvyatskiy, D., Fazel, M., and Harchaoui, Z.
Flat minima generalize for low-rank matrix recovery. In- formation and Inference: Journal of the IMA, 13(2): iaae009, 2024.
References
Dosovitskiy, A., Beyer, L.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image worth 16x16 words: Transformers for image recognition at scale. arXiv 2020.
Aminian, G., Asadi, A. R., Li, T., Beirami, A., Reinert, G., and Cohen, S. N. Generalization error of the tilted empirical risk. arXiv preprint arXiv:2409. 19431, 2024.
preprint arXiv:2010.
Andriushchenko, M. and Flammarion, N. Towards under- standing sharpness-ave aware aware minimization. In International Conference on Machine Learning, pp. 639-668. PMLR, 2022.
Du, J.. Zhou, D., Feng. J., Tan, V., and Zhou, J. T. Sharpness- aware training for free. Advances in Neural Information Processing Systems, 15:23439-234511 2022.

---

Tilted Sharpness-Aware Minimization
Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Liu, Y., Mai, S., Chen, X., Hsieh, C.-J., and You, Y. Towards efficient and scalable sharpness-aware minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12360-123700 2022a.
Duchi, J. C., Bartlett, P. L.., and Wainwright, M. J. Ran- domized smoothing for stochastic optimization. SIAM Journal on Optimization, 22(2):674-701, 2012.
Liu, Y., Mai, S., Cheng, M., Chen, X., Hsieh, C.-J,, and You, Y. Random sharpness-aware minimization. Advances in Neural Information Processing Systems, 35:24543- 24556, 2022b.
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-awarr minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.
Long, P. M. and Bartlett, P. L. Sharpness-aware min- imization and the edge of stability. arXiv preprint arXiv:2309. 12488, 2023.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual leamn ing for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Mi, P., Shen, L.., Ren, T., Zhou, Y., Sun, X., Ji, R., and Tao, D. Make sharpness-award minimization stronger: A sparsified perturbation approach. Advances in Neural Information Processing Systems, 35:30950-30962, 2022.
Johnson, R. and Zhang, T. Accelerating stochastic gradient descent using predictive variance reduction. Advances in neural information processing systems, 26, 2013.
Mueller, M., Vlaar, T. J., Rolnick, D., and Hein, M. Normal- ization layers are all that sharpness-aware minimization needs. In Thirty-seventh Conference Neural Informa- tion P Processing Systems, 2023.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6900 2014.
Neal, R. M. et al. Mome using hamiltonian dynamics. Hand- book of markov chain monte carlo, 2(11):2, 2011.
Kort, B. W. and Bertsekas, D. P. A new penalty function method for constrained minimization. In Proceedings of the 1972 ieee conference on decision and control and 11th symposium on adaptive processes, pp. 162-166. IEEE, 1972.
Nesterov, Y. A method of solving a convex programming problem with convergence rate of 1/k². Proceedings of the USSR Academy of Sciences, 269:3.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Nesterov, Y. method for solving the convex programming problem with convergence rate o (1/k2). In Dokl akad nauk Sssr, volume 269, pp. 543, 1983.
Kwon, J., Kim, J., Park, H., and Choi, I. K. Asam: Adaptive sharpness- aware minimization for scale-invariant learn- ing of deep neural networks. In International Conference on Machine Learning, pp. 5905-5914, 14. PMLR, 2021.
Nesterov, Y. Introductory lectures convex optimization: A basic course, volume 87. Springer Science & Business Media, 2013.
Rice, I.., Bair, A., Zhang, H., and Kolter, J. Z. Robustness between the worst and average case. Advances in Neural Information Processing Systems, 34:27840-27851, 2021.
Le, Y. and Yang, X. Tiny imagenet visual recognition chal- lenge. CS 231N, 7(7):3, 2015.
Leimkuhler, B. and Reich, S. Simulating hamiltonian dy- namics. Number 14. Cambridge university press, 2004.
Robey, A., Chamon, L., Pappas, G. J., and Hassani, H. Probabilistically robust learning: Balancing average and worst-case performance. In International Conference on Machine Learning, pp. 18667-18686. PMLR, 2022.
Li, B. and Giannakis, G. Enhancing sharpness-awart aware op- timization through variance suppression. Advances in Neural Information Processing Systems, 36, 2023.
Rockafellar, R. T., Uryasev, S., et al. Optimization of condi- tional value-at-risk. Journal of risk, 2:21-42 2000.
Li, T., Beirami, A., Sanjabi, M., and Smith, V. On tilted losses in machine learning: Theory and applications. Journal of Machine Learning Research, 24914 2023.
Sanh, V. Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.0 108, 2019.
Liu, G.-H. and Theodorou, A. Deep learning theory review: An optimal control and dynamical systems per- spective. arXiv preprint arXiv: 1908. 10920, 2019.
Shalev-Shwarte S. and Ben-David, S. Understanding ma- chine learning: From theory to algorithms. Cambridge university press, 2014.
10

---

Tilted Sharpness-Aware Minimization
Shen, C. and Li, H. On the dual formulation of boosting algorithms. IEEE Transactions Pattern Analysis and Machine Intelligence, 32(12):2216-2231 2010.
Zheng, Y., Zhang, R., and Mao, Y. Regularizing neural works via adversarial model perturbation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 8156-888888 8165, 2021.
Siegmund, D. Importance sampling in the monte carlo study of sequential tests. The Annals of Statistics, pp. 673 363-64, 1976.
Zhou, T., Wang, S., and Bilmes, J. Robust curriculum learning: from clean label detection noisy label self- conrection. In International Conference Learning Representations, 2020.
Streeter, M. and McMahan, H. B. Less regret via online conditioning. arXiv preprint arXiv: urXiv:1002.4800 1002.486, 2010.
Zhou, W., Liu, F., Zhang, H., and Chen, M. Sharpness- aware minimization with dynamic reweighting. arXiv preprint arXiv:2112.0877 2021.
Szabó, A., Jamali-Rad, H., and Mannava, S.-D. Tilted cross-entropy (tce): Promoting fairness in semantic seg- mentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2305- 2310, 2021.
Zhuang. J., Gong, B., Yuan, L.., Cui, Y., Adam, H., Dvornek, N., Tatikonda, S., Duncan, J., and Liu, T. Surrogate gap minimization improves sharpness-award training. arXiv preprint arXrr:2203.08066 2022.
Tahmasebi, B., Soleymani, A., Bahri, D., Jegelka, S., and Jaillet, P. A universal class of sharpness-aware minimiza- tion algorithms. ICML, 2024.
Wainwright, M. and Jordan, M. Graphical models, ex- ponential families, and variational inference. Foundations and Trends Machine Learning, 2008
Wen, K., Ma, T., and Li, Z. How does sharpness-awe aware minimization minimize sharpness? arXiv preprint arXiv:2211.05729, 2022.
Wen, K., Li, Z., and Ma, T. Sharpness minimization algo- rithms do not only minimize sharpness to achieve better generalization. Advances in Neural Information Process- ing Systems, 36, 2024.
Whittle, P. disk-sensitive inear/quadraticggaussian control. Advances in Applied Probability, 13(4):764-777, 1981.
Whittle, P. Risk sensitivity, a strangely pervasive concept. Macroeconomic Dynamics, 6(1):5-18, 2002.
Xie, W., Latorre, F., Antonakopoulos K., Pethick, T., and Cevher, V. Improving SAM requires rethinking its opti- mization formulation. In Proceedings of the 41st Interna- tional Conference on Machine Learning, 2024.
Zagoruyko, S. Wide residual networks. arXiv preprint 1rXiv:1605.07146. 2016.
Zhang. H. R., Li, D., and Ju, Noise stability optimization for finding flat minima: A hessian-based regularization approach. Trunsactions Machine Learning Research, 2024. ISSN 2835-8856.
Zhao, Y., Zhang, H., and Hu, X. Penalizing gradient norm for efficiently improving generalization in deep learning. In International Conference on Machine Learning, pp. PMLR, 2022.

---

Tilted Sharpness-Aware Minimization
A. Additional Toy Problems
In Figure in Section I, we present a specific toy problem where TSAM arrives at more flat solutions as t increases. Though the TSAM objective will recover SAM when - oo, we note that TSAM can be easier to solve due smoothness. To illustrate this, we create another toy problem in Figure and below. We see that SAM always leads non-smooth optimization problem for 0.
0.0
p= 0.12
p 0.23
p 0.34
2
2
(0)57
1580
p 0.46
p 0.57
P 0.69
p
2
2
2
L9(8)
(500)
1368
2
4
0
2
4
0
2
2
4
0
0
Figure SAM losses as p increases. The original loss function (shown in the blue lines across all figures) a one-dimensional problem I(O)- 00- 1| 0.01 i s 2, and L(0) (0 - 3) otherwise. Note that 0 = 3 is more flat solution than 0 = 1, though L(1) L(3). The SAM objective ming | L(B + <). shown in the red lines, where the values of p's increase from 0 to 0.8. When 0. the objective reduces ERM. For p 0, the SAM objectives (red lines) non-smooth, and the global minima (marked in orange) are achieved at flat region in L(-). The SAM objective visualization holds regardless of the usage of any existing SAM algorithms or implementation.
0.0
0.12
P 0.23
p 0.34
L(
L'(6)
L9(8)
2
4
0
0
0
p= 0.46
p= 0.57
p = 0.69
p= 0.8
2
2
2
L'(8)
L47)
(16)
0
4
0
2
4
4
0
Figure 4. TSAM (t=0.01) losses p increases. The TSAM objective (red lines) is & log (Bp(e) [elt(ate]) where ju(e) U[(cc] S p) defines uniform distribution constrained a ball with radius p. The larger p is, the darker the redness becomes. TSAM with small t is able to find flat solutions.
12

---

Tilted Sharpness-Aware Minimization
B. Complete Proofs
B.1. Proofs for Section 3.1
Proof for the Case of + 0, (0 +) + + Note that if L(-) is continuously diffcrentiable, then (L continuous w.r.t. E R.d also continuous w.r.t. t E R. When -
lim (0) lim log (ʃ (u(ee) 1-2
1
(9)
2-0
/
Sellloto
(1 L(0 (e(++)dp(e)
(10)
/ L(e + c)dp(c)
(11)
E[ + ()]-
(12)
Proof for Lipschitzness. 0. This follows from
First observe that L(0) p-Lipschitz with respect to 0, then L'(0) is p-Lipschitz with respect
- L'(())-L''s
log
(T(000))))
1
log (12(t))))
(13)
t
t
E [ctt(0)++)]
log
(14)
E [ett((
7 log epl -oll
1
E (l200
(15)
E
pl00 0211-
(16)
Proof for Strong Convexity, Assume is continuously differentiable.
j-strongly convex, then L' is also u-strongly
convex. This is because of the Hessian in Eq. (17), which can be written
eLL(0+e)
L2I'(0) = t. +E
(17)
Elet((++))
where M is positive semi-definite matrix. We note that due the jesttrong convexity of L, the second term satisfies (2)
L c)] > μI. Hence, L') > μI.
E
Proof for Smoothness.
From Eq. (17), we know that
(((+e)
(2)((0) M 7 Ejets100+c]]
E
LL (0++)))
(18)
As E
Blett(00)]]
1. and the max eigenvalue Amax ( ( +c) S B, we have
min - Amax < + +)) +oc.
(19)
B.2. Proofs for Section 3.2
In the following, denote E,, Define (() as
g(t) (,,) (,2)
(20)
log (ʃ (:2(0
-1og log (ʃ ((iietttddlle)
(21)
13

---

Tilted Sharpness-Aware Minimization
Assume / has the specific form of
(x;00) A(O) 0 (xx)),
(22)
( - = A(0) A(0))-TT(()) TT)()).
1 "
(()) A(O) OT
(23)
Under this form, we have that
(0) log (ʃ e (fetmono-Ono T(o))(
(24)
(-w0'T()) f eeraletos e r(t))(()de
log
(25)
-TTT((x> + + 1oog (ferseno- p(e)dd)
(26)
Define
(²(() :=log ( (1
(27)
Then we have
1 (0) t
log (R (B(0(A00t000000) - - + r'))
(28)
Define
n²( cla(0tto-
(29)
M²(0) = (n) + Tx))]
(30)
E [[)
m'(0) -1
(31)
We have that
On'(0) n² (0)(A((++)) -
(32)
We know
(0)
E
(lA(0+6) -e T(O) c) TT()))
= -0)(
(33)
01
E [eeA((04)- T(x)]
aL'
1 [')))
1 (0)
1 -
1
[0) Ot 82
() = m0)
(34)
Ot
B.3. ('( is monotonically non-inereasing We would like to prove the sign of aL'(0) m (0), is non-negative. The sign of m m²(0) is the same as the sign of (2mm (0).
We have
(0) = +h
(35)
m(t
-1'00 hh()0) (h-(0)
Oh'(0)
(36)
Ø1. 0t.
and
ah'(0) Eln'(0)AA((+ +c) 1
(37)
0t
(E (n))))))
Elr'(0)(A(0++)) - ( + - TT()))
(38)
(E (0)))
14

---

Tilted Sharpness-Aware Minimization
Let random variables X (n(0)(A((++)))]7T)) and Y (n Following the fact [[X])]]]1Y=](((XXX]))) 2 0 gives
(0
E| - EXXEEEXX 0. [E (0)))
(39)
06
Therefore,
a(t'm' Ot 2 0.
(40)
We note that lime+0 (0) lim ( + th' (0) 0. Hence t'm 2 0. Therefore, m' 2 0. we have shown that the tilted SAM loss (0) is monotonically non-decreasing as the increase of t, for any 0.
B.4. t-SAM prefers flatter models increases
Next, we examine g01, 82) (0)) - L' (92). First, we have
0g' 00,,01, 02)
(01)
(l2)
(41)
Øt
Ol
1
- 1 (02)))
(01) )+ + - (0,) + (02)
(42)
12
12
= =m'(01) m (02).
(43)
Similarly, it holds that
((t2m'(O)))
m(t'm'(02))
(O1)
(h2(
(44)
Ot
Ot
Ot
Ot
For 2 0, we have
(01)
(h2))))
@(tmm((1))) a(t2m'(02))
sign
sign
(45)
Øt sign (m)(11) m (02)) sign ((mm((1) I'm (02)).
Øt
Øt.
Øt
(46)
Let the random variable L, denote A(0I + c) - T(x), and random variable L2 denote A(22 + cT(r). Then
(h,)
E [et L E [e (E [eth 11))²
E e L3] E [etta]
EEEEt L1] B[ethh]
2
(47)
Ot
(E (2
Oh 0h'(01)
(02)
EE
teth LZ
Beltth L1)
IE Tettlo
[B[e [B L2]
(48)
Øt
Øt
E ethh
Elethi]
E [ethi]]
Elethig)
Given random variables L1 and L2, the exponentially reweighted losses can be defined as eh Lı and et Ly. The weighted second moment is Ble"ha Li]
and the I-weighted mean is B Hence, Bfet1113 (zlett can be B
B viewed t-weighted variance. As 0 is t-sharper than 82, we have (h'( ah'(02) 2 O. Therefore m01) -Imm (82) 0g (0, 0. dt
B Beeta]
is non-decreasing as increases. takes value of 0 when t = 0. which implies that m' (01) t2m (82)
Proof for the Discussions on L + 0. Recall that exp and log functions can be expanded
exp(x) +
I
+
x²
(49)
2
k=1
30
22
log(x 1)
(50)
2
k=1
15

---

Tilted Sharpness-Aware Minimization
For very small S
log E (cll(ote))
(51)
1 (1(000)
log
E 1 + 1T(0 + +c)
(52)
2
t² (0 +0)
EEa U1(0 (1(0+e) +c)+ e [1(00+))
E
UL(O + E) +
- -
(53)
2
12
12
[B[TT(0 + o)
E T.2 +o]
+ + +((1)
(54)
2
=|[ + o) + 2
E
[T + +0] - E [L + +)) 0(
(55)
E [ + c)] var (L(0 + c))
(56)
Hence, proposed objective can be viewed as optimizing for the mean plus variance of the losses in the neighborhood regions when & is very close to 0. When / = 0, it reduces to only optimizing for [[ + c)) for E j(c). For any 0₁ and 02 such that 02 is sharper than 02, we have
g(t) 101 + c)] + var (L(01 + <)) ([ + c)] var (L(62 + +)),
(57)
g((t) (var (L(01 + +)) var (L + +)))
(58)
Recall that sharpness is defined as standard variance when f + 0 (Definition 3.5), and we have that g(()) 2 0.
B.5. Proof for Theorem 3.7
We first state some useful lemmas.
Lemma B.1 (Stated and proved Aminian et al. (2024)). Let X be a random variable. Suppose <<<<x < to0, we have
var(X)
var(X)
< log(E[X]) [[log(X)] <
(59)
202
2a
The Lemma directly follows from existing results in Aminian al. (2024). For completeness, we include the proof here.
Proof. As & daz (log(z) + 3r²) = _ + 233 the function log(z) + Bx² is concave for = 262 and convex for 3
1
2n2
Hence, by Jensen's inequality,
X²
X2
1
Ellog(X] (X)] E log(X)
log log(E[[X]) +
21,2 E[X
E[X
(60)
212
212
212
Loog(EE())) [[[X)
log
var(X).
(61)
212
which completes the proof of the lower bound. A similar approach can be applied to derive the upper bound.
Proof for Theorem 3.7. We can now proceed with the detailed proof below.
Proof. Examine the following decomposition of the generalization error:
Ez (00, 2) log E. (CA(00))) = (1,, 2) - E. +) + R. +c]] log R. [ett00))))
(62)
Based on Lemma B.1, let X be (l((+c) and 1 < clT((++) < cM (assuming positive and bounded losses and non-nooneeee non-negative 1), we have that
var(e'L(0+e)) 21e2tM
1 log E. - +c) (ar(e
(63)
2t
16

---

Tilted Sharpness-Aware Minimization
and
var(elL(0+e)) 2te
2 var(elL(0+e))
2 [T + +c] { log( (B, ( ((
(64)
2t
For the term Bz]((0, [ E. EL[L(O + e)], we further decompose it into
Ez [10 / E. [ <)] Ez (00 2) I((0) I((0) Tee [T + c)]-
(65)
Recall that I(-) denote the empirical average loss based training samples (Eq. (1)), applying Hoeffding Inequal- ity (Boucheron et al., 2013) gives
12g(2/5)
Ez Ez 110, 2)] - L(0) < MV
(66)
2n
Combining (64) (65), and (66), we have the desired bound
Ez 2)] log R. (c2 < My 1og((/5)
(a((e))(0+e)
+ [(0)) - R.[T,(0 + c)]-
(67)
2n
21e
To investigate the impacts of / on the generalization bound, we can leave the last term T((0) E.[ + c)] as it is since it is independent of t.
Additionally, to bound L(Ø)- - E.[L(0 + e)] (the gap between empirical average losses and its randomly-smoothed version) we note that it related to the curvature I.(0). we further assume that the loss is ju-strongly convex, then it holds that
L( E[ < L(0) L(0) -
2 1eel S 1 - [[[el
(68)
Combining all the above results gives
log(2/8)
var(e
Ez Z)] log E. (e S My
[]
(69)
2n
21e
2
17

---

Tilted Sharpness-Aware Minimization
C. Additional Experimental Details
C.1. Hyperparamter Tuning
For all the three datasets and all methods, we tune learning rates from {0.0001, 0.0003, 0.001,0.003, 0.01, 0.03, 0.13. We use a fixed batch size of 64, label smoothing of 0.1 (for smooth cross-entropy loss), momentum with a parameter 0.9, and L2 weight decay 0.0005 for all runs. For vanilla SAM, we p from (0.05, 0.1, 0.5, 1, 15) and found that the best values 0.05, 0.1, 0.1 for CIFAR 100, DTD, noisy CIFAR 100, respectively. For SAM variants, we tune p parameters in the same way. The PGN baseline (Zhao et al., 2022) introduces another hyperparamette--tte coefficients for the linear combination between normal gradients and SAM gradients, and we tune that from {0.3, 0.5, 0.7, 0.9}. We follow the recommendations of X and hyperparameters in the original Random SAM paper (Liu et al., 2022b). For TSAM, we set = in Algorithm 1, set p to be 20, and a Algorithm 3 to be 0.995 across all datasets. We tune from {0, 1, 5, 20, 100), and the best t's are 20, 5. for the three image datasets. The number sampled e's (the s hyperparameter Algorithm 1) are chosen from {3, 5). We show the effects of t and in detail in Section 5.3 in the main text.
C.2. Naive Sampling
As discussed in Section 4, one naive approach estimate L( for in uniformly distributed over puc) is to first uniformly sample e's over plc), and then perform tilted aggregation, as follows:
Bfe
و EiE(s) E Eiels el(0+c)
- μ(c).
(70)
We demonstrate convergence function of on the CIFARI00 dataset in the figure below. We see that as s increases, the performance increases. However, when s 10, which means that we need 10 gradient evaluations per model updates, the accuracy is lower than that of TSAM with the proposed algorithm.
CIFAR100
test accuracy
0.5
naive sampling (s=3)
naive sampling (s=10)
TSAM, proposed alg.
0.2
20
40
80
epochs
Figure 5. TSAM (t 200 under the proposed algorithm compared with first uniformly sampling and then performing tilted aggregation.
C.3. Additional Results
Convergence Curves. In Table present the final test accuracies of TSAM and the baseline. In Figure 6, show the convergence of these methods on three datasets. We see that TSAM achieves the fastest convergence, and arrives at better solution. This is consistent with our argument that TSAM with bounded is more smooth objective than the original SAM formulation (Section 3).
18

---

Tilted Sharpness-Aware Minimization
CIFAR100
DTD
noisy CIFAR100
0.68
0.70
test accuracy
0.66
test accuracy
0.64
0.60
0.62
0.55
ERM
test
0.60
ERM
0.50
ERM
SAM
SAM
SAM
TSAM
TSAM
0.40
TSAM
0.56
25
50
75
100
125
150
175
200
10
20
30
50
100
150
200
250
300
epochs
epachs
epochs
Figure 6. Convergence curves on three image datasets showing test accuracies.
Training and Test Loss Comparisons under Different Objectives. In Table 3, we show that TSAM generates better than ERM and SAM by comparing training and test losses.
Table 3. To fürther understand TSAM behavior, we report the losses of models trained by (ERM, SAM, TSAM} and evaluated on (ERM, SAM, TSAM) objectives, respectively. The left table shows training losses and the right one shows test losses. We see that (1) every objective achieves the smallest training loss if directly being optimized (diagonal entries, left table). (2) Though SAM and TSAM incurs larger training losses than ERM (last two rows, left table), they lead to smaller test losses (last two rows, right table), i.c., better generalization. (3) Optimizing TSAM results in the smallest test loss across all the three metrics (last row, right table).
evaluated on
ERM
SAM
TSAM
evaluated on
ERM
SAM test loss
TSAM
trained on
training loss
trained on
ERM
0.1283
1.35
3.48
ERM
0.9302
2.05
3.54
SAM
0.1489
0.22
0.60
SAM TSAM
0.7414
0.91
1.34
TSAM
0.1763
0.27
0.46
0.7163
0.90
1.08
Effects of in HMC. Recall that Algorithm 3 follows HMC updates in Eq. (8) by running it for = 1 step. In principle, we can run Eq. (8) for more than step to generate each candidate perturbation e. This require additional gradient evaluations, which can be very expensive. We report results N and N Table below.
configurations
CIFAR100
DTD
N 1 (s 3)
0.7740
0,6882
N 1 (s 5)
0.7778
0.6870
N 2 (s 3)
0.7745
0.6883
N = 3
0.7752
0.6880
Table > requires times more gradient evaluations to generate a single perturbation (Eq. (8), without resulting significantly better model accuracies.
Runtime Comparisons. We report the runtime of TSAM and other baselines in Table 5 below.
Effects of the Tilting Hyperparameter t. One critical hyperparameter in TSAM is t. When t = 0, TSAM objective reduces to the average-case perturbed objective. When f - co, the TSAM objective (Eq. (3)) recovers SAM (Fq. 2). Here, we report the test accuracies as the training proceeds under multiple values of t's for the three tasks. In Figure 7, we see that there are a range of I's that result in faster convergence or higher accuracies than SAM.
Effects of Scheduling t. We also evaluate the performance when we schedule t dynamically during training. In Figure 8, we see that there is no significant difference between using fixed increasing / linearly, or decreasing it lienarly on the noisy CIFAR100 data.
19

---

Tilted Sharpness-Aware Minimization
Table Runtime comparisons (in minutes) averaged over different hyperparameter sets. ERM and SAM run the same numbers epochs
TSAM. All baselines explained in Section 5. ESAMI denotes the baseline of letting SAM run longer until it reaches the same computation budget as TSAM.
datasets
ERM
SAM
ESAM1
ESAM2
PGN
RSAM
TSAM
CIFARTO0
32
65
325
310
333
320
333
DTD
18
17
15
15
Noisy CIFARION
45
92
350
315
302
312
337
SAM
t=0
t=1
t=5
t=20
t=
100
CIFAR100
DTD
noisy CIFAR100
0.70
0.77
0.69
test accuracy
cost accuracy
test accurac
0.08
0.75
accuracy
G.775
Tinal Sccurcy
0,688
0.60
0.635
0.75
0.00
0.666
0.5
0.690
0.693
0.765
Final
0.608
0.11
0.55
20
100
300
0.655
0.7
0.55
100
125
275
19
20
50
100
150
230
epochs
# apochs
epochs
Figure Test accuracies of SAM and TSAM for various values of when number sampled e's 3 for each dataset. We select the best SAM and TSAM runs based on the final accuracies on validation data. The results suggest that (1) there multiple / values that give superior performance than SAM; and (2) we typically need to manually tune a best via grid search. The empirical performance under different values of relies tradeoffs between optimization efficiency (Section 3.1, Section 4), flatness (Section 3.2), and generalization (Section 3.3), and is difficult to determine an optimal prior to training. Note that in the label noise regime (left subtigure), one might think that SAM performance could be further improved via smaller learning rate early stopping: however, observe that SAM with a smaller learning rate does not give better accuracy. With early stopping, SAM accuracy is 0.6918, which is still 0.4% lower than that of TSAM without early stopping.
noisy CIFAR100
0.72
fix
test accuracy
increase
0.68
decrease
0.64
0.60
50
100
150
200
250
300
# epachs
Figure 8, Linearly decreasing or increasing the tilting hyperparameter with the epochs does not differ from the results of a fixed
20


Reviews:
[{"review_id": "AD6XVIbGHC", "reviewer": "Reviewer_Sf29", "summary": "This paper proposes Tilted SAM (TSAM) as a smoothed version of SAM using exponential tilting. Its smoothness enbles an easier optimization and a better generalization.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and valuable feedback. \n \n**[connections between flat local minima and improved generalization]** Note that we do not intend to claim ‘flat local minima leads to a better generalization in all cases’ by claiming “TSAM arrives at flatter local minima and results in superior test performance…” The exact relations between the two for deep learning is still an open question, as pointed out in our submission citing several related works (Section 2). Our study is partly motivated by the empirical success of a series of SAM works under which sharpness of local minima is significantly reduced (under various notions of sharpness), and we aim to develop a better formulation along this line by considering and reweighting multiple local minima in a principled framework. We will further clarify this in the next version.\n\n**[runtime of TSAM]** We agree that TSAM requires more gradient evaluations per iteration,  as discussed and acknowledged in the paper. However, we have evaluated TSAM and all the baselines with the same computational budget (Section 5), e.g., by letting baselines run 3$\\times$ or 5$\\times$ longer than TSAM, or optimizing for the inner max of SAM in a more complicated way with more gradient computation (ESAM2). We see that TSAM still outperforms those approaches on a variety of tasks and models. The trend still holds even if we further increase the runtime of ERM (SGD) or SAM. In addition, TSAM is inherently easy to parallelize since all the perturbations of model parameters are independent of each other. If we optimize our implementation in this way, TSAM would incur the same per-iteration runtime as the vanilla SAM algorithm. \n\n**[other questions]** (a) The performance of both TSAM and SAM would improve very slightly after 200 epochs, but TSAM still outperforms SAM. (b) When $N=1$, we are running gradient ascent once to locate the regions with (relatively) large losses, where the first-order gradient information serves as a strong signal to guide the areas to sample from. Additionally, the gap between losses can be magnified by using a large value of $t$ during reweighting. We will explain the intuition in more detail in the next version.", "claims_and_evidence": "All the claims are clear to me except one: \n\"Empirically, TSAM arrives at flatter local minima and results in superior test performance than the baselines of SAM and ERM across a range of image and text tasks.\"\n\nIt's not clear to me the connection between a flatter local minima and a superior test performance. It would be nice to refer to some papers or add explanation when it's first claimed.", "methods_and_evaluation_criteria": "Yes. They use test accuracy for classification tasks and sharpness when comparing to ERM and SAM.", "theoretical_claims": "Yes, I checked theorem 3.6 in the main text.", "experimental_designs_or_analyses": "Yes. I checked all of their experiments. I have a question on one experiment in appendix:\n\nIn figure 7, the CIFAR100 test accuracy of t=20 seems still increasing . What happens after 200 epochs?", "relation_to_broader_scientific_literature": "This paper propose applies exponential tilting to smooth SAM, which is a novel combination.", "essential_references_not_discussed": "See \"Claims And Evidence\"", "other_strengths_and_weaknesses": "Strengths:\nThe theoretical part is solid.\n\nWeakness:\n[limited impact] According to Table 5, the running time of TSAM of cifar100 is 10 times larger than running ERM and 5 times larger than SAM. Meanwhile, the improvement by TSAM is not significant (~1-2%) as shown in table1 for various tasks.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Do you have insights on why the algorithm still works even with N=1 and \"accept the generated ϵwith probability 1\"?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SITlb4JFGL", "reviewer": "Reviewer_F33F", "summary": "The authors propose TSAM, which is a version of SAM where, instead of taking a max over the loss around a point in parameter space during training, the authors propose to take a weighted average. Since the method is not tractable for weight functions, the authors develop an approximation by sampling. It is then shown that this leads indeed to (modest) gains in generalization performance.  The authors also put some effort in theoretical analysis.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and positive assessment of our work! We would like to address the remaining questions/concerns as follows.\n\n**[experimental improvements]** We observe statistically significant improvements of TSAM compared with the baselines. In terms of the test loss, TSAM outperforms baselines by a large margin (Table 3 in Appendix C.3). Furthermore, as TSAM is a new objective function minimizing a weighted combination of bad losses, we can further improve the current algorithm by combining it with other optimization techniques such as applying the idea of variance reduction to obtain better estimated gradients, or incorporate adaptivity to precondition the estimated gradients. We leave more fine-grained exploration of solvers as future work.\n\n**[other measures of sharpness and related work]** Thanks for the suggestions and for pointing out the related work. In our work, we have explored two metrics of sharpnesses, as discussed in Section 5.2. The first sharpness measure (results visualized in Table 2) is closely related to the local entropy notion mentioned in the related work (Pittorino et. al.) and we agree the formulations are conceptually related as well. We will add discussions around the connections with Pittorino et. al. Our formulation is inspired by exponential tilting, which inherently has rich connections with different areas such as information theory, applied probability, and optimization [1]. We are certainly interested in developing other sharpness measures based on the properties of our objectives leveraging connections with prior works, and we will add discussions on this in the next version.\n\n[1] On Tilted Losses in Machine Learning: Theory and Applications, JMLR.", "claims_and_evidence": "The claim that TSAM leads to flatter minima and better generalization peformance is well-supported by experiments and theoretical analysis. However, the gains in performance are very small and and not entirely consistent (see Table 1).", "methods_and_evaluation_criteria": "The authors compare to several other methods and over a wide range of datasets. This part of the paper is sufficient.", "theoretical_claims": "While I did not follow all mathematical analysis in detail, the general methods and results seems to be in line with the existing literature.", "experimental_designs_or_analyses": "No.", "relation_to_broader_scientific_literature": "The work presents an incremental improvement over SAM. Together with the theoretical foundations I think this is not a problem, however.", "essential_references_not_discussed": "Some important papers in the field are not cited, especially Pittorino et. al: \"Entropic gradient descent algorithms and wide flat minima \", which seems to have the same setup as the authors (see below).", "other_strengths_and_weaknesses": "- I think the basic idea is novel and the execution in the paper is good.\n- The experiments are well done and wide-ranging, even though the results are not entirely convincing (see above)\n- The paper is well written\n- The theoretical aspect is well developed", "other_comments_or_suggestions": "I would suggest exploring different measures of flatness to make the work stronger.", "questions_for_authors": "The central definition for TSAM (eq. 3) seems to be mathemetically equivalent to the local entropy in [1] (see first equation there and replace w' by w + epsilon), up to the choice of measure/distance function. The paper [1] has and the preceding work has some extensive analysis of this kind of setup and I think it would be good to relate these streams (just to be clear, the work in [1] is quite different after that basic definition, so there is no issue in being too similar, but at the same time it's clearly related and it is not cited/discussed).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LJ9nSUdtaD", "reviewer": "Reviewer_WTKN", "summary": "Authors propose Tilted SAM (TSAM) that  builds upon SAM in order to smooth out the optimization process using exponential tilting. Unlike SAM, which focuses on the worst-case loss within a neighborhood, TSAM reweights local solutions based on their loss values, favoring flatter minima. Authors claim that this makes TSAM easier to optimize and improves generalization performance across image and text tasks. They develop an algorithm using Hamiltonian Monte Carlo to efficiently estimate gradients for TSAM. Empirical results show that TSAM achieves better test performance and flatter minima compared to SAM and ERM.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and valuable comments.\n\n**[expensive computation]** We would like to clarify that the runtime numbers in Table 5 are reported for the same number of iterations of all methods (to illustrate the per-iteration cost), while the final test performance in the experiment section (Section 5) is obtained by letting all methods **run the same amount of wall-clock time**. Across all datasets, we consistently observe that under the same runtime, TSAM outperforms other algorithms.\n\nIn addition, TSAM is inherently easy to parallelize since all the perturbations of model parameters are independent of each other. If we optimize our practical implementation in this way, TSAM would incur the same per-iteration runtime as the vanilla SAM algorithm (consisting of one-step of gradient ascent and one step of gradient descent each iteration). \n\n**[scheduling $t$]** We note that scheduling $t$ with a fixed constraint set (defining the neighborhood of model parameters) does not change final performance significantly as long as we start from or end with the same values of $t$. When $t$ is zero, TSAM reduces to an average-perturbation-based sharpness-aware objective, which still accounts for some notions of sharpness of the loss surface. In the future, we plan to further investigate scheduling $t$ together with the radius parameter of the constraint set where it is possible to cover vanilla ERM with a mini-batch SGD optimizer.", "claims_and_evidence": "There are several claims made in the paper which are well-supported, including that TSAM leads to flatter solutions by averaging over multiple perturbations. While TSAM is backed by strong theoretical analysis, the claim about TSAM achieving better generalization performance is still quite strong whereas the gap in performance seems marginal in most experiments. Authors also show that the smoothness introduced by TSAM reduces optimization difficulties, however, it still takes more time (at least 3x) to compute as compared to ERM and SAM.", "methods_and_evaluation_criteria": "* Overall, the methodology is described clearly and backed by detailed analysis on several toy examples.\n* The authors propose sampling multiple $\\epsilon$’s from a distribution to obtain an empirical gradient estimation, with the idea that the full gradient is a tilted average of the original gradient. While the idea is interesting, it introduces significant computational challenges, as efficiently sampling and computing gradients from this distribution can be resource-intensive and slow. Are there other techniques to avoid this issue?", "theoretical_claims": "The proposed method is well-grounded in theory (with proofs provided for major claims).", "experimental_designs_or_analyses": "- The experiments involve different types of datasets and models and TSAM shows marginal improvements in all the cases.\n- The additional analysis on hyper-parameters and ablations are quite insightful.", "relation_to_broader_scientific_literature": "The idea behind TSAM is to improve SAM by smoothing the optimization process using exponential tilting to prioritize higher-loss perturbations. This is related to several earlier works on flat minima for improved generalization. It also leverages Hamiltonian Monte Carlo (HMC) for efficient gradient estimation. TSAM extends ideas from average-perturbed sharpness and noise-perturbed loss techniques to improve generalization and optimization stability.", "essential_references_not_discussed": "I think the literature review is sufficient, and the authors have effectively supported their ideas with relevant references and prior research.", "other_strengths_and_weaknesses": "Please refer to my previous comments.", "other_comments_or_suggestions": "Please refer to my previous comments.", "questions_for_authors": "Please refer to my previous comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HTSGhN9k3X", "reviewer": "Reviewer_YVio", "summary": "This paper introduces Tilted Sharpness-Aware Minimization (TSAM), a novel extension of Sharpness-Aware Minimization (SAM) designed to further enhance generalization in deep learning models. While SAM aims to minimize the worst-case local solutions, it overlooks many neighboring regions that may also contribute to significant losses. TSAM addresses this limitation by exponentially tilting the loss landscape, assigning greater weight to neighbors with higher losses. The authors theoretically demonstrate that as the tilt scalar $t$ increases, TSAM favors flatter minima. Additionally, they prove that TSAM achieves a tighter generalization bound than SAM for modest $t$. Extensive experiments validate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and valuable comments. We hope our response below can address the reviewer's concerns.\n\n**[more experiment results]** We appreciate the reviewer’s suggestions for adding more baselines. We have cited the papers mentioned by the reviewer in related work. We did not compare them in our original submission mainly because they are proposed to solve a SAM-like minimax style objective, where TSAM proposes a new objective function than SAM along with an algorithm to solve it. As plotted in Figures 3 and 4 in the appendix, even if we can solve the minimax SAM objective perfectly (in that particular toy problem, we did a brute-force search of the optimal parameters in the one-dimensional space), TSAM solutions still demonstrate more desirable properties than SAM solutions (more smooth). \n\nHowever, we agree more results will strengthen our experiments, and we compare with both GAM [3] and GSAM [2]. We leave out ASAM because GSAM achieves better performance than it (Figure 5 of the GSAM paper). For GSAM [2], we tune all the hyperparameters including the alpha parameter from a grid of {0.01, 0.02, 0.03} as suggested by the paper.  The GAM algorithm [3] can be expensive as it requires to compute Hessian vector products. We implemented its non-expensive approximation, as explained in the GAM paper and actually implemented in its open-sourced code. \n\nResults are shown in the table below. We report the best performance of each baseline on each dataset. For instance, in noisy CIFAR100, both GSAM and GAM overfit to noise at the later stage of training and we apply early stopping to obtain the best test performance.\n\n| Method   | CIFAR100 w/ WideResNet | CIFAR100 w/ ResNet18 | noisy CIFAR100 w/ ResNet18 | noisy CIFAR100 w/ WideResNet | DTD w/ ViT | DTD w/ WideResNet |\n|----------|-------------------------|------------------------|------------------------------|-------------------------------|------------|--------------------|\n| SGD      | 73.22                   | 71.39                  | 61.01                        | 57.02                         | 66.38      | 16.97              |\n| TSAM     | 80.85                   | 77.78                  | 69.98                        | 70.26                         | 68.82      | 18.63              |\n| GSAM [2] | 78.21                   | 77.95                  | 66.84                        | 70.35                         | 68.67      | 18.13              |\n| GAM [3]  | 78.01                   | 77.55                  | 67.26                        | 65.32                         | 68.28      | 17.97              |\n\n[1] ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks, ICML 2021.\n\n[2] Surrogate Gap Minimization Improves Sharpness-Aware Training, ICLR 2022.\n\n[3] Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization, CVPR 2023.\n\n**[compute cost]** We agree that TSAM requires more gradient evaluations per iteration,  as discussed and acknowledged in the paper. However, we have evaluated TSAM and all the baselines with the same computational budget (Section 5), e.g., by letting baselines run 3$\\times$ or 5$\\times$ longer than TSAM, or optimizing for the inner max of SAM in a more fine-grained way using more gradient computation (ESAM2). We see that TSAM still outperforms those approaches on a variety of tasks and models. The trend still holds even if we further increase the runtime of ERM (SGD) or SAM. In addition, TSAM is inherently easy to parallelize since all the perturbations of model parameters are independent of each other. If we optimize our implementation by sampling perturbations in parallel, TSAM would incur the same per-iteration runtime as the vanilla SAM algorithm. \n\n**[typos]** Thanks for pointing out the typos in the submission. We will fix them in the next version.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "- In Alg. 1, at each training step, the method samples $s$ random perturbations via HMC algorithm (Alg. 3). This process requires one backpropagation to compute the gradient $\\nabla L(\\theta^i_j)$ for each $j$, resulting in a total of $s$ backpropagations. Additionally, updating $\\theta_i$ requires another $s$ backpropagations to compute $\\nabla L(\\theta^i + \\epsilon_j)$ for each $j$.  In total, a single parameter update requires $2s$ backpropagations, leading to a high computational cost that may be unbearable.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "- Some popular SAM variants, such as ASAM [1], GSAM [2], GAM [3], are not included as baselines in the experiments.\n- The experiments are primarily conducted on small-scale datasets. Larger-scale datasets, such as ImageNet, should be used to further validate the effectiveness of the proposed method. \n\n-----\n\n[1] ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks, ICML 2021\n\n[2] Surrogate Gap Minimization Improves Sharpness-Aware Training, ICLR 2022\n\n[3] Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization, CVPR 2023", "relation_to_broader_scientific_literature": "The proposed method contributes to the broader scientific literature by introducing the idea of leveraging exponential tilting to reweight local minima, thereby addressing the drawbacks of SAM.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "**Strengths**\n\n- The idea of leveraging exponential tilting to reweight local minima is novel.\n\n- The paper provides a series of theoretical analyses to further validate the effectiveness of the proposed method.", "other_comments_or_suggestions": "- There are some typos. For example, the title should be corrected from \"Reweighting Local Mimina with Tilted SAM\" to \"Reweighting Local Minima with Tilted SAM\". Additionally, in line 26, the phrase \"such algorithms (and its variants) that reply on one or few steps\" should be \"such algorithms (and its variants) that rely on one or few steps\".", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a generalized and smoothed variant of SAM called Tilted SAM (TSAM). While SAM optimizes the worst-case loss within a parameter neighborhood, it struggles to find the true maximum loss point and ignores other potentially sharp regions. Instead of focusing solely on the worst perturbation, TSAM reweights all neighborhood points based on their exponentiated loss, effectively considering multiple high-loss regions. A tilt scalar t parameterizes this process, yielding a smooth objective that interpolates between minimizing the average neighborhood loss and recovering SAM's min-max objective. The authors theoretically show and empirically demonstrate that TSAM leads to flatter solutions and better generalization bounds than SAM for modest t. To overcome the challenge of optimizing TSAM, which requires sampling perturbations from a complex distribution, they adapt an efficient Hamiltonian Monte Carlo algorithm for gradient estimation.

Reviewers generally acknowledged the novelty of applying exponential tilting to smooth the SAM objective and appreciated the solid theoretical analysis provided, which demonstrated properties like favoring flatter solutions and achieving potentially tighter generalization bounds than standard SAM. Despite that, reviewers raised concerns on the practical trade-offs. A major recurring point across multiple reviews (Sf29, WTKN, YVio) was the substantially higher computational cost per iteration compared to ERM and SAM, as evidenced by runtime tables. While acknowledging the method finds flatter minima and shows performance improvements, reviewers questioned whether the observed gains (often described as marginal or modest, ~1-2%) justified the increase in training time or gradient evaluations per step. Less critical concerns included comparisons against more recent SAM variants like ASAM, GSAM, and GAM (YVio), the need for validation on larger datasets like ImageNet (YVio), and missing discussion of closely related work (F33F).

In their rebuttal, authors clarified that while the per-iteration cost is higher, their main experimental results were obtained using an equal total computational budget for all methods (meaning baselines like SAM/ERM were run for proportionally longer). They also argued that TSAM is inherently parallelizable, potentially matching SAM's per-iteration time with optimized implementations. Furthermore, they provided new experimental results showing TSAM outperformed requested baselines GSAM and GAM under equal compute budgets. They also provided clarifications on the flatness-generalization connection, cited the related work pointed out, and addressed other minor technical questions. 

The final sentiment remained somewhat mixed; while reviewers acknowledged the clarifications, some concerns about the cost-benefit ratio persisted, leading to borderline recommendations ranging from accept to weak reject (averaging to weak accept). Considering the authors' responses, particularly their clarification on the algorithm's parallelizability, which addresses a major concern and enhances practical utility, and their inclusion of new comparisons against baselines like GSAM and GAM, I believe this paper offers significant enough contributions to be useful for the community. Therefore, I recommend acceptance.
Decision: Accept (poster)