SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bingcong Li
Georgios B. Giannakis
University of Minnesota - Twin Cities Minneapolis, MN, USA {lixx5599, georgios}@umn.edu
Sharpness-aware minimization (SAM) has well documented merits in enhancing generalization of deep neural networks, even without sizable data augmentation. Embracing the geometry of the loss function, where neighborhoods of 'flat minima' heighten generalization ability, SAM seeks 'flat valleys' by minimizing the maximum loss caused by an adversary perturbing parameters within the neighborhood. Although critical to account for sharpness of the loss function, such an 'over-friendly adversary' can curtail the outmost level of generalization. The novel approach of this contribution fosters stabilization of adversaries through variance suppression (VaSSO) to avoid such friendliness. VaSSO's provable stability safeguards its numerical improvement over SAM in model-agnostic tasks, including image classification and machine translation. In addition, experiments confirm that VaSSO endows SAM with robustness against high levels of label noise. Code is available at https://github.com/BingcongLi/VaSSO.
Despite deep neural networks (DNNs) have advanced the concept of "learning from data," and markedly improved performance across several applications in vision and language (Devlin et al., 2018; Tom et al., 2020), their overparametrized nature renders the tendency to overfit on training data (Zhang et al., 2021). This has led to concerns in generalization, which is a practically underscored perspective yet typically suffers from a gap relative to the training performance.
Improving generalizability is challenging. Common approaches include (model) regularization and data augmentation (Srivastava et al., 2014). While it is the default choice to integrate regularization such as weight decay and dropout into training, these methods are often insufficient for DNNs especially when coping with complicated network architectures (Chen et al., 2022). Another line of effort resorts to suitable optimization schemes attempting to find a generalizable local minimum. For example, SGD is more preferable than Adam on certain overparameterized problems since it converges to maximum margin solutions (Wilson et al., 2017). Decoupling weight decay from Adam also empirically facilitates generalizability (Loshchilov and Hutter, 2017). Unfortunately, the underlying mechanism remains unclear, and whether the generalization merits carry over to other intricate learning tasks calls for additional theoretical elaboration.
Our main focus, sharpness aware minimization/optimization (SAM), is a highly compelling optimization approach that facilitates state-of-the-art generalizability by exploiting sharpness of loss landscape (Foret et al., 2021; Chen et al., 2022). A high-level interpretation of sharpness is how violently the loss fluctuates within a neighborhood. It has been shown through large-scale empirical studies that sharpness-based measures highly correlate with generalization (Jiang et al., 2019). Several works
have successfully explored sharpness for generalization advances. For example, Keskar et al. (2016) suggests that the batchsize of SGD impresses solution flatness. Entropy SGD leverages local entropy in search of a flat valley (Chaudhari et al., 2017). Different from prior works, SAM induces flatness by explicitly minimizing the adversarially perturbed loss, defined as the maximum loss of a neighboring area. Thanks to such a formulation, SAM has elevated generalization merits among various tasks in vision and language domains (Chen et al., 2022; Zhang et al., 2022). The mechanism fertilizing SAM's success is also theoretically investigated based on arguments of implicit regularization; see e.g., (Andriushchenko and Flammarion, 2022; Wen et al., 2023; Bartlett et al., 2022).
The adversary perturbation, or adversary for short, is central to SAM's heightened generalization because it effectively measures sharpness through the loss difference with original model (Foret et al., 2021; Zhuang et al., 2022; Kim et al., 2022). In practice however, this awareness on sharpness is undermined by what we termed friendly adversary. Confined by the stochastic linearization for computational efficiency, SAM's adversary only captures the sharpness for a particular minibatch of data, and can become a friend on other data samples. Because the global sharpness is not approached accurately, the friendly adversary precludes SAM from attaining its utmost generalizability. The present work advocates variance suppressed sharpness aware optimization  $(\mathrm{VaSSO}^{1})$  to alleviate 'friendliness' by stabilizing adversaries. With its provable stabilized adversary, VaSSO showcases favorable numerical performance on various deep learning tasks.
All in all, our contribution is summarized as follows.
We find that the friendly adversary discourages generalizability of SAM. This challenge is catastrophic in our experiments – it can completely wipe out the generalization merits.
A novel approach, VaSSO, is proposed to tackle this issue. VaSSO is equipped with what we termed variance suppression to streamline a principled means for stabilizing adversaries. The theoretically guaranteed stability promotes refined global sharpness estimates, thereby alleviating the issue of friendly adversary.
A side result is tighter convergence analyses for VaSSO and SAM that i) remove the bounded gradient assumption; and ii) deliver a more flexible choice for hyperparameters.
$\diamond$  Numerical experiments confirm the merits of stabilized adversary in VaSSO. It is demonstrated on image classification and neural machine translation tasks that VaSSO is capable of i) improving generalizability over SAM model-agnostically; and ii) nontrivially robustifying neural networks under the appearance of large label noise.
Notation. Bold lowercase (capital) letters denote column vectors (matrices);  $\| \mathbf{x}\|$  stands for  $\ell_2$  norm of vector  $\mathbf{x}$ ; and  $\langle \mathbf{x},\mathbf{y}\rangle$  is the inner product of  $\mathbf{x}$  and  $\mathbf{y}$ .  $\mathbb{S}_{\rho}(\mathbf{x})$  denotes the surface of a ball with radius  $\rho$  centered at  $\mathbf{x}$ , i.e.,  $\mathbb{S}_{\rho}(\mathbf{x})\coloneqq \{\mathbf{x} + \rho \mathbf{u}\mid \| \mathbf{u}\| = 1\}$ .
2 The known, the good, and the challenge of SAM
This section starts with a brief recap of SAM (i.e., the known), followed with refined analyses and positive results regarding its convergence (i.e., the good). Lastly, the friendly adversary issue is explained in detail and numerically illustrated.
2.1 The known
Targeting at a minimum in flat basin, SAM enforces small loss around the entire neighborhood in the parameter space (Foret et al., 2021). This idea is formalized by a minimax problem
$$
\min  _ {\mathbf {x}} \max  _ {\| \boldsymbol {\epsilon} \| \leq \rho} f (\mathbf {x} + \boldsymbol {\epsilon}) \tag {1}
$$
where  $\rho$  is the radius of considered neighborhood, and the nonconvex objective is defined as  $f(\mathbf{x})\coloneqq \mathbb{E}_{\mathcal{B}}[f_{\mathcal{B}}(\mathbf{x})]$ . Here,  $\mathbf{x}$  is the neural network parameter, and  $\mathcal{B}$  is a random batch of data. The merits of such a formulation resides in its implicit sharpness measure  $\max_{\| \epsilon \| \leq \rho}f\bigl (\mathbf{x} + \boldsymbol {\epsilon}\bigr) - f(\mathbf{x})$ , which effectively drives the optimization trajectory towards the desirable flat valley (Kim et al., 2022).
${}^{1}$  Vasso coincides with the Greek nickname for Vasiliki.
Algorithm 1 Generic form of SAM
1: Initialize:  $\mathbf{x}_0, \rho$
2: for  $t = 0, \dots, T - 1$  do
3: Sample a minibatch  $\mathcal{B}_t$ , and define stochastic gradient on  $\mathcal{B}_t$  as  $\mathbf{g}_t(\cdot)$
4: Find  $\epsilon_t \in \mathbb{S}_\rho(\mathbf{0})$  via stochastic linearization; e.g., (4) for VaSSO or (3) for SAM
5: Calculate stochastic gradient  $\mathbf{g}_t(\mathbf{x}_t + \pmb{\epsilon}_t)$
6: Update model via  $\mathbf{x}_{t + 1} = \mathbf{x}_t - \eta \mathbf{g}_t(\mathbf{x}_t + \pmb {\epsilon}_t)$
7: end for
8: Return:  $\mathbf{x}_T$
The inner maximization of (1) has a natural interpretation as finding an adversary. Critical as it is, obtaining an adversary calls for stochastic linearization to alleviate computational concerns, i.e.,
$$
\boldsymbol {\epsilon} _ {t} = \underset {\| \boldsymbol {\epsilon} \| \leq \rho} {\arg \max } f (\mathbf {x} _ {t} + \boldsymbol {\epsilon}) \overset {(a)} {\approx} \underset {\| \boldsymbol {\epsilon} \| \leq \rho} {\arg \max } f (\mathbf {x} _ {t}) + \left\langle \nabla f (\mathbf {x} _ {t}), \boldsymbol {\epsilon} \right\rangle \overset {(b)} {\approx} \underset {\| \boldsymbol {\epsilon} \| \leq \rho} {\arg \max } f (\mathbf {x} _ {t}) + \left\langle \mathbf {g} _ {t} (\mathbf {x} _ {t}), \boldsymbol {\epsilon} \right\rangle \tag {2}
$$
where linearization  $(a)$  relies on the first order Taylor expansion of  $f(\mathbf{x}_t + \epsilon)$ . This is typically accurate given the choice of a small  $\rho$ . A stochastic gradient  $\mathbf{g}_t(\mathbf{x}_t)$  then substitutes  $\nabla f(\mathbf{x}_t)$  in  $(b)$  to downgrade the computational burden of a full gradient. Catalyzed by the stochastic linearization in (2), it is possible to calculate SAM's adversary in closed-form
$$
\boxed {\mathbf {S A M}: \quad \boldsymbol {\epsilon} _ {t} = \rho \frac {\mathbf {g} _ {t} (\mathbf {x} _ {t})}{\| \mathbf {g} _ {t} (\mathbf {x} _ {t}) \|}.} \tag {3}
$$
SAM then adopts the stochastic gradient of adversary  $\mathbf{g}_t(\mathbf{x}_t + \boldsymbol{\epsilon}_t)$  to update  $\mathbf{x}_t$  in a SGD fashion. A step-by-step implementation is summarized in Alg. 1, where the means to find an adversary in line 4 is presented in a generic form in order to unify the algorithmic framework with later sections.
2.2 The good
To provide a comprehensive understanding about SAM, this subsection focuses on Alg. 1, and establishes its convergence for (1). Some necessary assumptions are listed below, all of which are common for nonconvex stochastic optimization problems (Ghadimi and Lan, 2013; Bottou et al., 2016; Mi et al., 2022; Zhuang et al., 2022).
Assumption 1 (lower bounded loss).  $f(\mathbf{x})$  is lower bounded, i.e.,  $f(\mathbf{x}) \geq f^{*}, \forall \mathbf{x}$ .
Assumption 2 (smoothness). The stochastic gradient  $\mathbf{g}(\mathbf{x})$  is  $L$ -Lipschitz, i.e.,  $\| \mathbf{g}(\mathbf{x}) - \mathbf{g}(\mathbf{y})\| \leq L\| \mathbf{x} - \mathbf{y}\|$ ,  $\forall \mathbf{x},\mathbf{y}$
Assumption 3 (bounded variance). The stochastic gradient  $\mathbf{g}(\mathbf{x})$  is unbiased with bounded variance, that is,  $\mathbb{E}[\mathbf{g}(\mathbf{x})|\mathbf{x}] = \nabla f(\mathbf{x})$  and  $\mathbb{E}[\| \mathbf{g}(\mathbf{x}) - \nabla f(\mathbf{x})\|^2 |\mathbf{x}] = \sigma^2$  for some  $\sigma >0$ .
The constraint of (1) is never violated since  $\| \pmb{\epsilon}_t\| = \rho$  holds for each  $t$ ; see line 4 in Alg. 1. Hence, the convergence of SAM pertains to the behavior of objective, where a tight result is given below.
Theorem 1 (SAM convergence). Suppose that Assumptions 1-3 hold. Let  $\eta_t \equiv \eta = \frac{\eta_0}{\sqrt{T}} \leq \frac{2}{3L}$  and  $\rho = \frac{\rho_0}{\sqrt{T}}$ . Then with  $c_0 = 1 - \frac{3L\eta}{2}$  (clearly  $0 < c_0 < 1$ ), Alg. 1 guarantees that
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} \right] \leq \mathcal {O} \left(\frac {\sigma^ {2}}{\sqrt {T}}\right) \quad a n d \quad \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \| \nabla f (\mathbf {x} _ {t} + \boldsymbol {\epsilon} _ {t}) \| ^ {2} \right] \leq \mathcal {O} \left(\frac {\sigma^ {2}}{\sqrt {T}}\right).
$$
The convergence rate of SAM is the same as SGD up to constant factors, where the detailed expression hidden under big  $\mathcal{O}$  notation can be found in Appendix D. Our results eliminate the need for the bounded gradient assumption compared to existing analyses in (Mi et al., 2022; Zhuang et al., 2022). Moreover, Theorem 1 enables a much larger choice of  $\rho = \mathcal{O}(T^{-1/2})$  relative to (Andriushchenko and Flammarion, 2022), where the latter only supports  $\rho = \mathcal{O}(T^{-1/4})$ .
A message from Theorem 1 is that any adversary satisfying  $\epsilon_t \in \mathbb{S}_{\rho}(\mathbf{0})$  ensures convergence. Because the surface  $\mathbb{S}_{\rho}(\mathbf{0})$  is a gigantic space, it challenges the plausible optimality of the adversary and poses a natural question - is it possible to find a more powerful adversary for generalization advances?
(b)
Figure 1: (a) A friendly adversary erases the generalization merits of SAM; (b)  $m$ -sharpness may not directly correlate with variance since noisy gradient degrades generalization; and (c)  $m$ -sharpness may not hold universally. Note that test accuracies in (a) and (b) are normalized to SGD.
2.3 The challenge: friendly adversary
Adversary to one minibatch is a friend of others. SAM's adversary is 'malicious' for minibatch  $\mathcal{B}_t$  but not necessarily for other data because it only safeguards  $f_{\mathcal{B}_t}(\mathbf{x}_t + \boldsymbol{\epsilon}_t) - f_{\mathcal{B}_t}(\mathbf{x}_t) \geq 0$  for a small  $\rho$ . In fact, it can be shown that  $f_{\mathcal{B}}(\mathbf{x}_t + \boldsymbol{\epsilon}_t) - f_{\mathcal{B}}(\mathbf{x}_t) \leq 0$  whenever the stochastic gradients do not align well, i.e.,  $\langle \mathbf{g}_t(\mathbf{x}_t), \mathbf{g}_{\mathcal{B}}(\mathbf{x}_t) \rangle \leq 0$ . Note that such misalignment is common because of the variance in massive training datasets. This issue is referred to as friendly adversary, and it implies that the adversary  $\boldsymbol{\epsilon}_t$  cannot accurately depict the global sharpness of  $\mathbf{x}_t$ . Note that the 'friendly adversary' also has a more involved interpretation, that is,  $\mathbf{g}_t(\mathbf{x}_t)$  falls outside the column space of Hessian at convergence; see more discussions after (Wen et al., 2023, Definition 4.3). This misalignment of higher order derivatives undermines the inductive bias of SAM, thereby worsening generalization.
To numerically visualize the catastrophic impact of the friendly adversary, we manually introduce one by replacing line 4 of Alg. 1 as  $\tilde{\epsilon}_t = \rho \tilde{\mathbf{g}}_t(\mathbf{x}_t) / \| \tilde{\mathbf{g}}_t(\mathbf{x}_t)\|$ , where  $\tilde{\mathbf{g}}_t$  denotes the gradient on  $\tilde{\mathcal{B}}_t$ , a randomly sampled batch of the same size as  $\mathcal{B}_t$ . This modified approach is denoted as SAM-db, and its performance for i) ResNet-18 on CIFAR10 and ii) ResNet-34 on CIFAR100<sup>2</sup> can be found in Fig. 1(a). Note that the test accuracy is normalized relative to SGD for the ease of visualization. It is evident that the friendly  $\tilde{\epsilon}_t$  in SAM-db almost erases the generalization benefits entirely.
Source of friendly adversary. The major cause to the friendly adversary attributes to the gradient variance, which equivalently translates to the lack of stability in SAM's stochastic linearization  $(2b)$ . An illustrative three dimensional example is shown in Fig. 2, where we plot the adversary  $\epsilon_{t}$  obtained from different  $\mathbf{g}_t$  realization in  $(2b)$ . The minibatch gradient is simulated by adding Gaussian noise to the true gradient. When the signal to noise ration (SNR) is similar to a practical scenario (ResNet-18 on CIFAR10 shown in Fig. 2 (e)), it can be seen in Fig. 2 (c) and (d) that the adversaries almost uniformly spread over the norm ball, which strongly indicates the deficiency for sharpness evaluation.
Friendly adversary in the lens of Frank Wolfe. An additional evidence in supportive to SAM's friendly adversary resides in its connection to stochastic Frank Wolfe (SFW) that also heavily relies on stochastic linearization (Reddi et al., 2016). The stability of SFW is known to be vulnerable – its convergence cannot be guaranteed without a sufficient large batchsize. As thoroughly discussed in Appendix A, the means to obtain adversary in SAM is tantamount to one-step SFW with a constant batchsize. This symbolizes the possible instability of SAM's stochastic linearization.
2.4 A detailed look at friendly adversaries
The gradient variance is major cause to SAM's friendly adversary and unstable stochastic linearization, however this at first glance seems to conflict with an empirical note termed  $m$ -sharpness, stating that the benefit of SAM is clearer when  $\epsilon_{t}$  is found using subsampled  $\mathcal{B}_t$  of size  $m$  (i.e., larger variance).
Since  $m$ -sharpness highly hinges upon the loss curvature, it is unlikely to hold universally. For example, a transformer is trained on IWSLT-14 dataset, where the test performance (BLEU) decreases with smaller  $m$  even if we have tuned  $\rho$  carefully; see Fig. 1(c). On the theoretical side, an example is provided in (Andriushchenko and Flammarion, 2022, Sec. 3) to suggest that  $m$ -sharpness is not
(a)  $\mathrm{SNR} = 5$
(b)  $\mathrm{SNR} = 1$
(c)  $\mathrm{SNR} = 0.1$
(d)  $\mathrm{SNR} = 0.01$
Figure 2: (a) - (d) SAM's adversaries spread over the surface; (e) SNR is in [0.01, 0.1] when training a ResNet-18 on CIFAR10, where the SNR is calculated at the first iteration of every epoch.
(e) SNR in practice
necessarily related with sharpness or generalization. Moreover, there also exists specific choice for  $m$  such that the  $m$ -sharpness formulation is ill-posed. We will expand on this in Appendix B.
Even in the regime where  $m$ -sharpness is empirically observed such as ResNet-18 on CIFAR10 and ResNet-34 on CIFAR100, we show through experiments that  $m$ -sharpness is not a consequence of gradient variance, thus not contradicting with the friendly adversary issue tackled in this work.
Observation 1. Same variance, different generalization. Let  $m = 128$  and batchsize  $b = 128$ . Recall the SAM-db experiment in Fig. 1(a). If  $m$ -sharpness is a direct result of gradient variance, it is logical to expect SAM-db has comparable performance to SAM simply because their batchzises (hence variance) for finding adversary are the same. Unfortunately, SAM-db degrades accuracy. We further increase the variance of  $\tilde{\mathbf{g}}_t(\mathbf{x}_t)$  by setting  $m = 64$ . The resultant algorithm is denoted as SAM-db-m/2. It does not catch with SAM and performs even worse than SAM-db. These experiments validate that variance/stability correlates with friendly adversary instead of  $m$ -sharpness.
Observation 2. Enlarged variance degrades generalization. We explicitly increase variance when finding adversary by adding Gaussian noise  $\zeta$  to  $\mathbf{g}_t(\mathbf{x}_t)$ , i.e.,  $\hat{\epsilon}_t = \rho \frac{\mathbf{g}_t(\mathbf{x}_t) + \zeta}{\|\mathbf{g}_t(\mathbf{x}_t) + \zeta\|}$ . After tuning the best  $\rho$  to compensate the variance of  $\zeta$ , the test performance is plotted in Fig. 1(b). It can be seen that the generalization merits clearly decrease with larger variance on both ResNet-18 and ResNet-34. This again illustrates that the plausible benefit of  $m$ -sharpness does not stem from increased variance.
In sum, observations 1 and 2 jointly suggest that gradient variance correlates with friendly adversary rather than  $m$ -sharpness, where understanding the latter is beyond the scope of current work.
3 Variance-suppressed sharpness-aware optimization (VaSSO)
This section advocates variance suppression to handle the friendly adversary. We start with the design of VaSSO, then establish its stability. We also touch upon implementation and possible extensions.
3.1 Algorithm design and stability analysis
A straightforward attempt towards stability is to equip SAM's stochastic linearization with variance reduced gradients such as SVRG and SARAH (Johnson and Zhang, 2013; Nguyen et al., 2017; Li et al., 2019). However, the requirement to compute a full gradient every a few iterations is infeasible and hardly scales well for tasks such as training DNNs.
The proposed variance suppression (VaSSO) overcomes this computational burden through a novel yet simple stochastic linearization. For a prescribed  $\theta \in (0,1)$ , VaSSO is summarized below
$$
\boxed {\mathbf {V a S S O}: \quad \mathbf {d} _ {t} = (1 - \theta) \mathbf {d} _ {t - 1} + \theta \mathbf {g} _ {t} (\mathbf {x} _ {t})} \tag {4a}
$$
$$
\boldsymbol {\epsilon} _ {t} = \underset {\| \boldsymbol {\epsilon} \| \leq \rho} {\arg \max } f (\mathbf {x} _ {t}) + \left\langle \mathbf {d} _ {t}, \boldsymbol {\epsilon} \right\rangle = \rho \frac {\mathbf {d} _ {t}}{\| \mathbf {d} _ {t} \|}. \tag {4b}
$$
Compared with (2) of SAM, the key difference is that VaSSO relies on slope  $\mathbf{d}_t$  for a more stable stochastic linearization as shown in (4b). The slope  $\mathbf{d}_t$  is an exponentially moving average (EMA) of  $\{\mathbf{g}_t(\mathbf{x}_t)\}_{t}$  such that the change over consecutive iterations is smoothed. Noticing that  $\epsilon_t$  and  $\mathbf{d}_t$  share the same direction, the relatively smoothed  $\{\mathbf{d}_t\}_{t}$  thus imply the stability of  $\{\epsilon_t\}_{t}$  in VaSSO. Moreover, as  $\mathbf{d}_t$  processes information of different minibatch data, the global sharpness can be captured in a principled manner to alleviate the friendly adversary challenge.
To theoretically characterize the effectiveness of VaSSO, our first result considers  $\mathbf{d}_t$  as a qualified strategy to estimate  $\nabla f(\mathbf{x}_t)$ , and delves into its mean square error (MSE).
Theorem 2 (Variance suppression). Suppose that Assumptions 1 - 3 hold. Let Alg. 1 equip with  $i)\epsilon_{t}$  obtained by (4) with  $\theta \in (0,1)$ ; and, ii)  $\eta_t$  and  $\rho$  selected the same as Theorem 1. VaSSO guarantees that the MSE of  $\mathbf{d}_t$  is bounded by
$$
\mathbb {E} \left[ \| \mathbf {d} _ {t} - \nabla f (\mathbf {x} _ {t}) \| ^ {2} \right] \leq \theta \sigma^ {2} + \mathcal {O} \left(\frac {(1 - \theta) ^ {2} \sigma^ {2}}{\theta^ {2} \sqrt {T}}\right). \tag {5}
$$
Because SAM's gradient estimate has a looser bound on MSE (or variance), that is,  $\mathbb{E}[\| \mathbf{g}_t - \nabla f(\mathbf{x}_t)\|^2 ]\leq \sigma^2$ , the shrunk MSE in Theorem 2 justifies the name of variance suppression.
Next, we quantify the stability invoked with the suppressed variance. It is convenient to start with necessary notation. Define the quality of a stochastic linearization at  $\mathbf{x}_t$  with slope  $\mathbf{v}$  as  $\mathcal{L}_t(\mathbf{v})\coloneqq \max_{\| \boldsymbol {\epsilon}\| \leq \rho}f(\mathbf{x}_t) + \langle \mathbf{v},\boldsymbol {\epsilon}\rangle$ . For example,  $\mathcal{L}_t(\mathbf{d}_t)$  and  $\mathcal{L}_t\big(\mathbf{g}_t(\mathbf{x}_t)\big)$  are quality of VaSSO and SAM, respectively. Another critical case of concern is  $\mathcal{L}_t\big(\nabla f(\mathbf{x}_t)\big)$ . It is shown in (Zhuang et al., 2022) that  $\mathcal{L}_t\big(\nabla f(\mathbf{x}_t)\big)\approx \max_{\| \boldsymbol {\epsilon}\| \leq \rho}f(\mathbf{x}_t + \boldsymbol {\epsilon})$  given a small  $\rho$ . Moreover,  $\mathcal{L}_t\big(\nabla f(\mathbf{x}_t)\big) - f(\mathbf{x}_t)$  is also an accurate approximation to the sharpness (Zhuang et al., 2022). These observations safeguard  $\mathcal{L}_t(\nabla f(\mathbf{x}_t))$  as the anchor when analyzing the stability of SAM and VaSSO.
Definition 1 ( $\delta$ -stability). A stochastic linearization with slope  $\mathbf{v}$  is said to be  $\delta$ -stable if its quality satisfies  $\mathbb{E}\left[|\mathcal{L}_t(\mathbf{v}) - \mathcal{L}_t(\nabla f(\mathbf{x}_t))| \right] \leq \delta$ .
A larger  $\delta$  implies a more friendly adversary, hence is less preferable. We are now well-prepared for our main results on adversary's stability.
Theorem 3 (Adversaries of VaSSO is more stable than SAM.). Suppose that Assumptions 1 - 3 hold. Under the same hyperparameter choices as Theorem 2, the stochastic linearization is  $\left[\sqrt{\theta}\rho\sigma + \mathcal{O}\left(\frac{\rho\sigma}{\theta T^{1/4}}\right)\right]$ -stable for VaSSO, while  $\rho\sigma$ -stable in SAM.