SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dongkuk Si Chulhee Yun
{dongkuksi, chulhee.yun}@kaist.ac.kr
Sharpness-Aware Minimization (SAM) is an optimizer that takes a descent step based on the gradient at a perturbation  $\pmb{y}_t = \pmb{x}_t + \rho \frac{\nabla f(\pmb{x}_t)}{\|\nabla f(\pmb{x}_t)\|}$  of the current point  $\pmb{x}_t$ . Existing studies prove convergence of SAM for smooth functions, but they do so by assuming decaying perturbation size  $\rho$  and/or no gradient normalization in  $\pmb{y}_t$ , which is detached from practice. To address this gap, we study deterministic/stochastic versions of SAM with practical configurations (i.e., constant  $\rho$  and gradient normalization in  $\pmb{y}_t$ ) and explore their convergence properties on smooth functions with (non)convexity assumptions. Perhaps surprisingly, in many scenarios, we find out that SAM has limited capability to converge to global minima or stationary points. For smooth strongly convex functions, we show that while deterministic SAM enjoys tight global convergence rates of  $\tilde{\Theta} \left( \frac{1}{T^2} \right)$ , the convergence bound of stochastic SAM suffers an inevitable additive term  $\mathcal{O}(\rho^2)$ , indicating convergence only up to neighborhoods of optima. In fact, such  $\mathcal{O}(\rho^2)$  factors arise for stochastic SAM in all the settings we consider, and also for deterministic SAM in nonconvex cases; importantly, we prove by examples that such terms are unavoidable. Our results highlight vastly different characteristics of SAM with vs. without decaying perturbation size or gradient normalization, and suggest that the intuitions gained from one version may not apply to the other.
Modern neural networks are armed with a large number of layers and parameters, having a risk to overfit to training data. In order to make accurate predictions on unseen data, generalization performance has been considered as the most important factor in deep learning models. Based on the widely accepted belief that geometric properties of loss landscape are correlated with generalization performance, studies have proved theoretical and empirical results regarding the relation between sharpness measures and generalization [14, 17, 19, 21, 29]. Here, sharpness of a loss at a point generally refers to the degree to which the loss varies in the small neighborhood of the point.
Motivated by prior studies that show flat minima have better generalization performance [19, 21], Foret et al. [16] propose an optimization method referred to as Sharpness-Aware Minimization (SAM). A single iteration of SAM consists of one ascent step (perturbation) and one descent step. Starting from the current iterate  $\boldsymbol{x}_t$ , SAM first takes an ascent step  $\boldsymbol{y}_t = \boldsymbol{x}_t + \rho \frac{\nabla f(\boldsymbol{x}_t)}{\|\nabla f(\boldsymbol{x}_t)\|}$  to (approximately) maximize the loss value in the  $\rho$ -neighborhood of  $\boldsymbol{x}_t$ , and then uses the gradient at  $\boldsymbol{y}_t$  to update  $\boldsymbol{x}_t$  to  $\boldsymbol{x}_{t+1}$ . This two-step procedure gives SAM a special characteristic: the tendency to search for a flat minimum, i.e., a minimum whose neighboring points also return low loss value. Empirical results [3, 9, 16, 20, 25] show that SAM demonstrates an exceptional ability to perform well on different models and tasks with high generalization performance. Following the success of SAM,
Table 1: Convergence of SAM with constant perturbation size  $\rho$  after  $T$  steps. C1, C2, C3, and C4 indicate  $\beta$ -smoothness,  $\mu$ -strong convexity, convexity, and Lipschitzness, respectively. C5 indicates the bounded variance of the gradient oracle. See Section 2.3 for definitions of C1-C5.
Optimizer	Function Class	Convergence Upper/Lower Bounds	Reference
Deterministic SAM	C1,C2	mint∈{0,...,T}f(xt)−f* = O(exp(−T)+1/T2)	Theorem 3.1
Deterministic SAM	C1,C2	mint∈{0,...,T}f(xt)−f* = Ω(1/T2)	Theorem 3.2
Deterministic SAM	C1,C3	1/T ∑t=0T-1||∇f(xt)||2 = O(1/T+1/√T)	Theorem 3.3
Deterministic SAM	C1	1/T ∑t=0T-1||∇f(xt)||2 ≤ O(1/T)+β2ρ2	Theorem 3.4
Stochastic SAM	C1,C2,C5	Ef(xT)−f* ≤ O(exp(−T)+[σ2−β2ρ2]+/T) + 2β2ρ2/μ	Theorem 4.1
Stochastic SAM	C1,C3,C5	1/T ∑t=0T-1E||∇f(xt)||2 ≤ O(1/T+√[σ2−β2ρ2]+/√T) + 4β2ρ2	Theorem 4.3
Stochastic n-SAM	C1,C5	1/T ∑t=0T-1E||∇f(xt)||2 ≤ O(1/T+1/√T) + β2ρ2	Theorem 4.5
Stochastic m-SAM	C1,C4,C5	1/T ∑t=0T-1E[(||∇f(xt)||−βρ)2] ≤ O(1/√T) + 5β2ρ2	Theorem 4.6
numerous extensions of SAM have been proposed in the literature [6, 15, 18, 23, 24, 26-28, 30-32, 34, 35].
On the theoretical side, various studies have demonstrated different characteristics of SAM [1, 2, 4, 5, 12, 13, 22, 33]. However, comprehending the global convergence properties of practical SAM on a theoretical level still remains elusive. In fact, several recent results [2, 18, 27, 31, 35] attempt to prove the convergence guarantees for SAM and its variants. While these results provide convergence guarantees of SAM on smooth functions, they are somewhat detached to the practical implementations of SAM and its variants. They either (1) assume decaying or sufficiently small perturbation size  $\rho$  [18, 27, 31, 35], whereas  $\rho$  is set to constant in practice; or (2) assume ascent steps without gradient normalization [2], whereas practical implementations of SAM use normalization when calculating the ascent step  $y_{t}$ .
1.1 Summary of Our Contributions
To address the aforementioned limitations of existing results, we investigate convergence properties of SAM using gradient normalization in ascent steps and arbitrary constant perturbation size  $\rho$ . We note that to the best of our knowledge, the convergence analysis of SAM have not been carried out under the two practical implementation choices. Our analyses mainly focus on smooth functions, with different levels of convexity assumptions ranging from strong convexity to nonconvexity. We summarize our contributions below; a summary of our convergence results (upper and lower bounds) can also be found in Table 1.
- For deterministic SAM, we prove convergence to global minima of smooth strongly convex functions, and show the tightness of convergence rate in terms of  $T$ . Furthermore, we establish the convergence of SAM to stationary points of smooth convex functions. For smooth nonconvex functions, we prove that SAM guarantees convergence to stationary points up to an additive factor  $\mathcal{O}(\rho^2)$ . We provide a worst-case example that always suffers a matching squared gradient norm  $\Omega(\rho^2)$ , showing that the additive factor is unavoidable and tight in terms of  $\rho$ .
- For stochastic settings, we analyze two versions of stochastic SAM ( $n$ -SAM and  $m$ -SAM) on smooth strongly convex, smooth convex, and smooth (Lipschitz) nonconvex functions. We provide convergence guarantees to global minima or stationary points up to additive factors  $\mathcal{O}(\rho^2)$ . In case of  $m$ -SAM, we demonstrate that these factors are inevitable and tight in terms of  $\rho$ , by providing worst-case examples where SAM fails to converge properly.
1.2 Related Works: Convergence Analysis of SAM
Recent results [18, 27, 31, 35] prove convergence of stochastic SAM and its variants to stationary points for smooth nonconvex functions. However, these convergence analyses are limited to only smooth nonconvex functions and do not address convex functions. Also, as already pointed out, all the proofs require one crucial assumption detached from practice: decaying (or sufficiently small) perturbation size  $\rho$ . If  $\rho$  becomes sufficiently small, then the difference between  $f^{\mathrm{SAM}}(\pmb{x}) = \max_{\| \pmb{\epsilon} \| \leq \rho} f(\pmb{x} + \pmb{\epsilon})$  and  $f(\pmb{x})$  becomes negligible, which means that they undergo approximately the same updates. Due to this reason, especially in later iterates, SAM with negligible perturbation size
would become almost identical to gradient descent (GD), resulting in a significant difference in the convergence behavior compared to constant  $\rho$ .
As for existing proofs that do not require decaying  $\rho$ , Andriushchenko and Flammarion [2] prove convergence guarantees for deterministic SAM with constant  $\rho$  on smooth nonconvex, smooth Polyak-Lojasiewicz, smooth convex, and smooth strongly convex functions. Moreover, the authors prove convergence guarantees of stochastic SAM, this time with decaying  $\rho$ , on smooth nonconvex and smooth Polyak-Lojasiewicz functions. However, their analyses are also crucially different from the practical implementations of SAM because their proofs are for a variant of SAM without gradient normalization in ascent steps, with updates in the form of  $\pmb{x}_{t + 1} = \pmb{x}_t - \eta \nabla f(\pmb{x}_t + \rho \nabla f(\pmb{x}_t))$ . This form can be considered as vanilla SAM with an "effective" perturbation size  $\rho \| \nabla f(\pmb{x}_t)\|$ , which indicates that even with constant  $\rho$ , the effective perturbation size will become smaller and smaller as the algorithm converges towards a stationary point. As in the case of decaying  $\rho$  discussed above, this can make a huge difference in the convergence behavior.
Figure 1 illustrates a comparison between SAM with and without gradient normalization, highlighting the disparity between their convergence points. As we predicted above, it is evident that they indeed reach
Figure 1: Trajectory plot for a function  $f(x,y) = (xy - 1)^2$ . The green line indicates global minima of  $f$ . SAM and USAM indicates SAM with and without gradient normalization, respectively, and GD indicates gradient descent. When USAM approaches the green line, it converges to a nearby global minimum, which is similar to GD. In contrast, SAM travels along the green line, towards the flattest minimum (1,1).
entirely distinct global minima, exhibiting different convergence characteristics.
Discussions on the related works on the relationship between sharpness and generalization, as well as other theoretical properties of SAM, are provided in Appendix A.
The euclidean norm of a vector  $\pmb{v}$  is denoted as  $\| \pmb{v} \|$ . We let  $\mathbb{R}_+^* \triangleq \{x \in \mathbb{R} \mid x > 0\}$ , and use  $[\cdot]_+$  to define  $\max \{\cdot, 0\}$ . We use  $\mathcal{O}(\cdot)$  and  $\Omega(\cdot)$  to hide numerical constants in our upper and lower bounds, respectively. We use  $\tilde{\mathcal{O}}(\cdot)$  to additionally hide logarithmic factors. When discussing rates, we sometimes use these symbols to hide everything except  $T$ , the number of SAM iterations. When discussing additive factors that arise in our convergence bounds, we use these symbols to hide all other variables except the perturbation size  $\rho$ . For an objective function  $f$  and initialization  $\pmb{x}_0$  of SAM, we define  $f^* \triangleq \inf_{\pmb{x}} f(\pmb{x})$  and  $\Delta \triangleq f(\pmb{x}_0) - f^*$ .
2 Sharpness-Aware Minimization: Preliminaries and Intuitions
Before presenting the main results, we first introduce deterministic and stochastic SAM, and present definitions of function classes considered in this paper. We next introduce virtual gradient map and virtual loss that shed light on our intuitive understanding of SAM's convergence behavior.
2.1 Sharpness-Aware Minimization (SAM)
We focus on minimizing a function  $f: \mathbb{R}^d \to \mathbb{R}$ , where the optimization variable is represented by  $\pmb{x} \in \mathbb{R}^d$ , using the Sharpness-Aware Minimization (SAM) optimizer. Instead of minimizing the vanilla objective function  $f(\pmb{x})$ , SAM [16] aims to minimize the SAM objective  $f^{\mathrm{SAM}}(\pmb{x})$ , where
$$
\min  _ {\boldsymbol {x}} f ^ {\mathrm {S A M}} (\boldsymbol {x}) = \min  _ {\boldsymbol {x}} \max  _ {\| \boldsymbol {\epsilon} \| \leq \rho} f (\boldsymbol {x} + \boldsymbol {\epsilon}). \tag {1}
$$
For  $\rho \in \mathbb{R}_+^*$ , the SAM loss  $f^{\mathrm{SAM}}(\pmb{x})$  outputs the worst function value in a  $\rho$ -ball  $\{\pmb{w} \in \mathbb{R}^d \mid \| \pmb{x} - \pmb{w} \| \leq \rho\}$  around  $\pmb{x}$ . Assuming "sufficiently small"  $\rho$ , the inner maximization of (1) can be solved by Taylor-approximating the objective:
$$
\arg \max  _ {\| \boldsymbol {\epsilon} \| \leq \rho} f (\boldsymbol {x} + \boldsymbol {\epsilon}) \approx \arg \max  _ {\| \boldsymbol {\epsilon} \| \leq \rho} f (\boldsymbol {x}) + \langle \boldsymbol {\epsilon}, \nabla f (\boldsymbol {x}) \rangle = \rho \frac {\nabla f (\boldsymbol {x})}{\| \nabla f (\boldsymbol {x}) \|} \triangleq \hat {\boldsymbol {\epsilon}} (\boldsymbol {x}).
$$
Using this approximate solution  $\hat{\epsilon}(\pmb{x})$ , we can define the approximate SAM objective  $\hat{f}^{\mathrm{SAM}}$  as  $\hat{f}^{\mathrm{SAM}}(\pmb{x}) \triangleq f(\pmb{x} + \hat{\epsilon}(\pmb{x}))$ . In order to run gradient descent (GD) on  $\hat{f}^{\mathrm{SAM}}(\pmb{x})$ , one needs to calculate its gradient; however, from the definition of  $\hat{\epsilon}(\pmb{x})$ , we can realize that  $\nabla \hat{f}^{\mathrm{SAM}}(\pmb{x})$  has terms that involve the Hessian of  $f$ . Here, SAM makes another approximation, by ignoring the Hessian term:
$$
\nabla \hat {f} ^ {\mathrm {S A M}} (\pmb {x}) \approx \nabla f (\pmb {x}) | _ {\pmb {x} + \hat {\epsilon} (\pmb {x})}.
$$
From these approximations, one iteration of SAM is defined as a set of two-step update equations:
$$
\left\{ \begin{array}{l} \boldsymbol {y} _ {t} = \boldsymbol {x} _ {t} + \rho \frac {\nabla f (\boldsymbol {x} _ {t})}{\| \nabla f (\boldsymbol {x} _ {t}) \|}, \\ \boldsymbol {x} _ {t + 1} = \boldsymbol {x} _ {t} - \eta \nabla f (\boldsymbol {y} _ {t}). \end{array} \right. \tag {2}
$$
As seen in (2), we use  $\pmb{x}_t$  to denote the iterate of SAM at the  $t$ -th step. We use  $T$  to denote the number of SAM iterations. We refer to the hyperparameter  $\rho \in \mathbb{R}_{+}^{*}$  as the perturbation size and  $\eta \in \mathbb{R}_{+}^{*}$  as the step size. Note that in (2), the perturbation size  $\rho$  is a time-invariant constant; in practice, it is common to fix  $\rho$  as a constant throughout training [3, 9, 16].
According to the SAM update in (2), the update cannot be defined when  $\| \nabla f(\pmb{x}) \| = 0$ . In practice, we add a small numerical constant (e.g.,  $10^{-12}$ ) to the denominator in order to prevent numerical instability. In this paper, we ignore this constant and treat  $\frac{\nabla f(\pmb{x}_t)}{\|\nabla f(\pmb{x}_t)\|}$  as 0 whenever  $\| \nabla f(\pmb{x}_t) \| = 0$ .
2.2 SAM under Stochastic Settings
To analyze stochastic SAM, we suppose that the objective is given as  $f(\pmb{x}) = \mathbb{E}_{\xi}[l(\pmb{x};\xi)]$ , where  $\xi$  is a stochastic parameter (e.g., data sample) and  $l(\pmb{x};\xi)$  indicates the loss at point  $\pmb{x}$  with a random sample  $\xi$ . Based on the SAM update in (2), we can define stochastic SAM under this setting:
$$
\left\{ \begin{array}{l} \boldsymbol {y} _ {t} = \boldsymbol {x} _ {t} + \rho \frac {g (\boldsymbol {x} _ {t})}{\| g (\boldsymbol {x} _ {t}) \|}, \\ \boldsymbol {x} _ {t + 1} = \boldsymbol {x} _ {t} - \eta \tilde {g} (\boldsymbol {y} _ {t}). \end{array} \right. \tag {3}
$$
We define  $g(\pmb{x}) = \nabla_{\pmb{x}} l(\pmb{x}; \xi)$  and  $\tilde{g}(\pmb{x}) = \nabla_{\pmb{x}} l(\pmb{x}; \tilde{\xi})$ . Here,  $\xi$  and  $\tilde{\xi}$  are stochastic parameters, queried from any distribution(s) which satisfies  $\mathbb{E}_{\xi} l(\pmb{x}; \tilde{\xi}) = \mathbb{E}_{\xi} l(\pmb{x}; \xi) = f(\pmb{x})$ .
There are two popular variants of stochastic SAM, introduced in Andriushchenko and Flammarion [2]. Stochastic  $n$ -SAM algorithm refers to the update equation (3) when  $\xi$  and  $\tilde{\xi}$  are independent. In contrast, practical SAM algorithm in Foret et al. [16] employs stochastic  $m$ -SAM algorithm, which follows the update equation (3) where  $\xi$  is equal to  $\tilde{\xi}$ . We will consider both versions in our theorems.
2.3 Function Classes
We state definitions and assumptions of function classes of interest, which are fairly standard.
Definition 2.1 (Convexity). A function  $f: \mathbb{R}^d \to \mathbb{R}$  is convex if, for any  $\pmb{x}, \pmb{y} \in \mathbb{R}^d$  and  $\lambda \in [0,1]$ , it satisfies  $f(\lambda \pmb{x} + (1 - \lambda) \pmb{y}) \leq \lambda f(\pmb{x}) + (1 - \lambda) f(\pmb{y})$ .
Definition 2.2 ( $\mu$ -Strong Convexity). A differentiable function  $f: \mathbb{R}^d \to \mathbb{R}$  is  $\mu$ -strongly convex if there exists  $\mu > 0$  such that  $f(\pmb{x}) \geq f(\pmb{y}) + \langle \nabla f(\pmb{y}), \pmb{x} - \pmb{y} \rangle + \frac{\mu}{2} \| \pmb{x} - \pmb{y} \|^2$ , for all  $\pmb{x}, \pmb{y} \in \mathbb{R}^d$ .
Definition 2.3 ( $L$ -Lipschitz Continuity). A function  $f: \mathbb{R}^d \to \mathbb{R}$  is  $L$ -Lipschitz continuous if there exists  $L \geq 0$  such that  $\| f(\boldsymbol{x}) - f(\boldsymbol{y}) \| \leq L \| \boldsymbol{x} - \boldsymbol{y} \|$ , for all  $\boldsymbol{x}, \boldsymbol{y} \in \mathbb{R}^d$ .
Definition 2.4 ( $\beta$ -Smoothness). A differentiable function  $f: \mathbb{R}^d \to \mathbb{R}$  is  $\beta$ -smooth if there exists  $\beta \geq 0$  such that  $\| \nabla f(\boldsymbol{x}) - \nabla f(\boldsymbol{y}) \| \leq \beta \| \boldsymbol{x} - \boldsymbol{y} \|$ , for all  $\boldsymbol{x}, \boldsymbol{y} \in \mathbb{R}^d$ .
Next we define an assumption considered in the analysis of stochastic SAM (3).
Assumption 2.5 (Bounded Variance). The gradient oracle of a differentiable function  $f: \mathbb{R}^d \to \mathbb{R}$  has bounded variance if there exists  $\sigma \geq 0$  such that
$$
\mathbb {E} _ {\boldsymbol {\xi}} \| \nabla f (\boldsymbol {x}) - \nabla l (\boldsymbol {x}; \boldsymbol {\xi}) \| ^ {2} \leq \sigma^ {2}, \quad \mathbb {E} _ {\tilde {\boldsymbol {\xi}}} \| \nabla f (\boldsymbol {x}) - \nabla l (\boldsymbol {x}; \tilde {\boldsymbol {\xi}}) \| ^ {2} \leq \sigma^ {2}, \quad \forall \boldsymbol {x} \in \mathbb {R} ^ {d}.
$$
2.4 SAM as GD on Virtual Loss
Convergence analysis of SAM is challenging due to the presence of normalized ascent steps. In order to provide intuitive explanations of SAM's convergence properties, we develop tools referred to as
virtual gradient map and virtual loss in this section. These tools can provide useful intuitions when we discuss our main results.
In order to define the new tools, we first need to define Clarke subdifferential [10], whose definition below is from Theorem 6.2.5 of Borwein and Lewis [7].
Definition 2.6 (Clarke Subdifferential). Suppose that a function  $f: \mathbb{R}^d \to \mathbb{R}$  is locally Lipschitz around a point  $\pmb{x}$  and differentiable on  $\mathbb{R}^d \setminus W$  where  $W$  is a set of measure zero. Then, the Clarke subdifferential of  $f$  at  $\pmb{x}$  is  $\partial f(\pmb{x}) \triangleq \mathrm{cvxhull}\{\lim_{t\to \infty}\nabla f(\pmb{x}_t)\mid \pmb{x}_t\to \pmb{x},\pmb{x}_t\notin W\}$ , where cvxhull denotes the convex hull of a set.
Clarke subdifferential  $\partial f(\pmb{x})$  is a convex hull of all possible limits of  $\nabla f(\pmb{x}_t)$  as  $\pmb{x}_t$  approaches  $\pmb{x}$ . It can be thought of as an extension of gradient to a nonsmooth function. It is also known that for a convex function  $f$ , the Clarke subdifferential of  $f$  is equal to the subgradient of  $f$ .
Using the definition of Clarke differential, we now define virtual gradient map and virtual loss of  $f$ , which can be employed for understanding the convergence of SAM.
Definition 2.7 (Virtual Gradient Map/Loss). For a differentiable function  $f: \mathbb{R}^d \to \mathbb{R}$ , we define the virtual gradient map  $G_f: \mathbb{R}^d \to \mathbb{R}^d$  of  $f$  to be  $G_f(\pmb{x}) \triangleq \nabla f\big(\pmb{x} + \rho \frac{\nabla f(\pmb{x})}{\|\nabla f(\pmb{x})\|}\big)$ . Additionally, if there exists a function  $J_f: \mathbb{R}^d \to \mathbb{R}$  whose Clarke subdifferential is well-defined and  $\partial J_f(\pmb{x}) \ni G_f(\pmb{x})$  for all  $\pmb{x}$ , then we call  $J_f$  a virtual loss of  $f$ .
If a virtual loss  $J_{f}$  is well-defined for  $f$ , the update of SAM (2) on  $f$  is equivalent to a (sub)gradient descent update on the virtual loss  $J_{f}(\pmb{x})$ , which means,  $\pmb{x}_{t+1} = \pmb{x}_t - \eta G_f(\pmb{x}_t)$ . The reason why we have to use Clarke subdifferential to define the virtual loss is because even for a differentiable and smooth function  $f$ , there are cases where the virtual gradient map  $G_{f}$  is discontinuous and the virtual loss  $J_{f}$  (if exists) is nonsmooth; see the discussion below Theorem 3.1.
Note that if the differentiable function  $f$  is one-dimensional ( $d = 1$ ), the virtual loss  $J_{f}(x)$  is always well-defined because it can be obtained by simply (Lebesgue) integrating  $G_{f}(x)$ . However, in case of multi-dimensional functions ( $d > 1$ ), there is no such guarantee, although  $G_{f}$  is always well-defined. We emphasize that the virtual gradient map  $G_{f}$  and virtual loss  $J_{f}$  are mainly used for a better intuitive understanding of our (non-)convergence results. In formal proofs, we use them for analysis only if  $J_{f}$  is well-defined.
Lastly, we note that Bartlett et al. [4] also employ a similar idea of virtual loss. In case of convex quadratic objective function  $f(\pmb{x})$ , the authors define  $\pmb{u}_t = (-1)^t\pmb{x}_t$  and formulate a (different) virtual loss  $\tilde{J}_f(\pmb{u})$  such that a SAM update on  $f(\pmb{x}_t)$  is equivalent to a GD update on  $\tilde{J}_f(\pmb{u}_t)$ .
3 Convergence Analysis Under Deterministic Settings
In this section, we present the main results on the (non-)convergence of deterministic SAM with constant perturbation size  $\rho$  and gradient normalization. We study four function classes: smooth strongly convex, smooth convex, smooth nonconvex, and nonsmooth Lipschitz convex functions.
3.1 Smooth and Strongly Convex Functions
For smooth strongly convex functions, we prove a global convergence guarantee for the best iterate.
Theorem 3.1. Consider a  $\beta$ -smooth and  $\mu$ -strongly convex function  $f$ . If we run deterministic SAM starting at  $x_0$  with any perturbation size  $\rho > 0$  and step size  $\eta = \min \left\{\frac{1}{\mu T} \max \left\{1, \log \left(\frac{\mu^5 \Delta T^2}{\beta^6 \rho^2}\right)\right\}, \frac{1}{2\beta}\right\}$  to minimize  $f$ , we have
$$
\min _ {t \in \{0, \dots , T \}} f (\pmb {x} _ {t}) - f ^ {*} = \tilde {\mathcal {O}} \left(\exp \left(- \frac {\mu T}{2 \beta}\right) \Delta + \frac {\beta^ {6} \rho^ {2}}{\mu^ {5} T ^ {2}}\right).
$$
The proof of Theorem 3.1 can be found in Appendix B.2. As for relevant existing studies, Theorem 1 of Dai et al. [13] can be adapted to establish the convergence guarantee  $f(\boldsymbol{x}_T) - f^* = \tilde{\mathcal{O}}\left(\frac{1}{T}\right)$ , by selecting the step size  $\eta = \min \left\{\frac{1}{\mu T} \max \left\{1, \log \left(\frac{\mu^2 \Delta T}{\beta^3 \rho^2}\right)\right\}, \frac{1}{\beta}\right\}$  and employing a similar proof technique as ours. We highlight that Theorem 3.1 achieves a faster convergence rate compared to the concurrent bound by Dai et al. [13]. Moreover, Theorem 11 of Andriushchenko and Flammarion [2] proves the convergence guarantee for this function class:  $\| \boldsymbol{x}_T - \boldsymbol{x}^* \|^2 = \mathcal{O}(\exp(-T))$ , but assuming
SAM without gradient normalization, and the boundedness of  $\rho$ . Here, we get a slower convergence rate of  $\tilde{\mathcal{O}}\left(\exp(-T) + \frac{1}{T^2}\right)$ , but with any  $\rho > 0$  and with normalization.
By viewing SAM as GD on virtual loss, we can get an intuition why SAM cannot achieve exponential convergence. Consider a smooth and strongly convex function:  $f(x) = \frac{1}{2} x^2$ . One possible virtual loss of  $f$  is  $J_{f}(x) = \frac{1}{2} (x + \rho \mathrm{sign}(x))^{2}$ , which is a nonsmooth and strongly convex function, for which the exponential convergence of GD is impossible [8].
Indeed, we can show that the sublinear convergence rate in Theorem 3.1 is not an artifact of our analysis. Interestingly, the  $\tilde{\mathcal{O}}\left(\exp (-T) + \frac{1}{T^2}\right)$  rate given in Theorem 3.1 is tight in terms of  $T$ , up to logarithmic factors. Our next theorem provides a lower bound for smooth strongly convex functions.