SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 3.2. Suppose  $\frac{\beta}{\mu} \geq 2$ . For any choice of perturbation size  $\rho$ , step size  $\eta$ , and initialization  $\mathbf{x}_0$ , there exists a differentiable,  $\beta$ -smooth, and  $\mu$ -strongly convex function  $f$  such that
$$
\min  _ {t \in \{0, \dots , T \}} f (\boldsymbol {x} _ {t}) - f ^ {*} = \Omega \left(\frac {\beta^ {3} \rho^ {2}}{\mu^ {2} T ^ {2}}\right)
$$
holds for deterministic SAM iterates.
For the proof of Theorem 3.2, refer to Appendix B.3. By comparing the rates in Theorems 3.1 and 3.2, we can see that the two bounds are tight in terms of  $T$  and  $\rho$ . The bounds are a bit loose in terms of  $\beta$  and  $\mu$ , but we believe that this may be partly due to our construction; in proving lower bounds, we used one-dimensional quadratic functions as worst-case examples. A more sophisticated construction may improve the tightness of the lower bound, which is left for future work.
3.2 Smooth and Convex Functions
For smooth and convex functions, proving convergence of the function value to global optimum becomes more challenging, due to the absence of strong convexity. In fact, we can instead prove that the gradient norm converges to zero.
Theorem 3.3. Consider a  $\beta$ -smooth and convex function  $f$ . If we run deterministic SAM starting at  $\mathbf{x}_0$  with any perturbation size  $\rho > 0$  and step size  $\eta = \min \left\{\frac{\sqrt{2\Delta}}{\sqrt{\beta^3\rho^2T}}, \frac{1}{2\beta}\right\}$  to minimize  $f$ , we have
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} = \mathcal {O} \left(\frac {\beta \Delta}{T} + \frac {\sqrt {\Delta \beta^ {3} \rho^ {2}}}{\sqrt {T}}\right).
$$
The proof of Theorem 3.3 is given in Appendix B.4. As for relevant existing studies, Theorem 11 of Andriushchenko and Flammarion [2] proves the convergence guarantee for this function class:  $f(\bar{x}) - f^{*} = \mathcal{O}\left(\frac{1}{T}\right)$ , where  $\bar{x}$  indicates the averaged  $x$  over  $T$  iterates, while assuming SAM without gradient normalization and bounded  $\rho$ . Here, we prove a weaker result: a convergence rate of  $\mathcal{O}\left(\frac{1}{\sqrt{T}}\right)$  to stationary points, albeit for any  $\rho > 0$  and with normalization.
One might expect that a reasonably good optimizer should converge to global minima of smooth convex functions. However, it turns out that both showing convergence to global minima and finding a non-convergence example are quite challenging in this function class.
Indeed, we later show non-convergence examples for other relevant settings. For stochastic SAM, we provide a non-convergence example (Theorem 4.4) in the same smooth and convex function class, showing that the suboptimality gap in terms of function values cannot have an upper bound, and therefore rendering convergence to global minima impossible. For deterministic SAM in nonsmooth Lipschitz convex functions, we show an example (Theorem 3.6) where convergence to the global minimum is possible only up to a certain distance proportional to  $\rho$ .
Given these examples, we suspect that there may also exist a non-convergence example for the deterministic SAM in this smooth convex setting. We leave settling this puzzle to future work.
3.3 Smooth and Nonconvex Functions
Existing studies prove that SAM (and its variants) with decaying or sufficiently small perturbation size  $\rho$  converges to stationary points for smooth nonconvex functions [2, 18, 27, 31, 35]. Unfortunately, with constant perturbation size  $\rho$ , SAM exhibits a different convergence behavior: it does not converge all the way to stationary points.
Theorem 3.4. Consider a  $\beta$ -smooth function  $f$  satisfying  $f^{*} = \inf_{\boldsymbol{x}} f(\boldsymbol{x}) > -\infty$ . If we run deterministic SAM starting at  $\boldsymbol{x}_0$  with any perturbation size  $\rho > 0$  and step size  $\eta = \frac{1}{\beta}$  to minimize  $f$ , we have
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} \leq \mathcal {O} \left(\frac {\beta \Delta}{T}\right) + \beta^ {2} \rho^ {2}.
$$
Refer to the Appendix B.5 for the proof of Theorem 3.4. For a comparison, Theorem 9 of Andriushchenko and Flammarion [2] proves the convergence for this function class:  $\frac{1}{T}\sum_{t=0}^{T}\|\nabla f(\boldsymbol{x}_t)\|^2 = \mathcal{O}\left(\frac{1}{T}\right)$ , but again assuming SAM without gradient normalization, and boundedness of  $\rho$ . Our Theorem 3.4 guarantees  $\mathcal{O}\left(\frac{1}{T}\right)$  convergence up to an additive factor  $\beta^2\rho^2$ .
One might speculate that the undesirable additive factor  $\beta^2\rho^2$  is an artifact of our analysis. The next theorem presents an example which proves that this extra term is in fact unavoidable.
Theorem 3.5. For any  $\rho >0$  and  $\eta \leq \frac{1}{\beta}$ , there exists a  $\beta$ -smooth and  $\Theta (\beta \rho)$ -Lipschitz continuous function such that, if deterministic SAM is initialized at a point  $x_0$  sampled from a continuous probability distribution, then deterministic SAM converges to a nonstationary point, located at a distance of  $\Omega (\rho)$  from a stationary point, with probability 1.
Here we present a brief outline for the proof of Theorem 3.5. For a given  $\rho$ , consider a one-dimensional function  $f(x) = \frac{9\beta\rho^2}{25\pi^2}\sin \left(\frac{5\pi}{3\rho} x\right)$ . Figure 2(a) demonstrates the virtual loss for this example. By examining the virtual loss  $J_{f}$  and its stationary points, we observe that SAM iterates  $x_{t}$  converge to a non-stationary point, located at a distance of  $\Omega (\rho)$  from the stationary point. This means that the limit point of SAM will have gradient norm of order  $\Omega (\beta \rho)$ , thereby proving that the additive factor in Theorem 3.4 is tight in terms of  $\rho$ . A detailed analysis of Theorem 3.5 is provided in Appendix B.6.
Remark: why do we ignore  $\eta = \Omega\left(\frac{1}{\beta}\right)$ ? As the reader may have noticed, Theorem 3.5 only considers the case  $\eta = \mathcal{O}\left(\frac{1}{\beta}\right)$ , and hence does not show that the additive factor is inevitable for any choice of  $\eta > 0$ . However, one can notice that if  $\eta = \Omega\left(\frac{1}{\beta}\right)$ , then we can consider a one-dimensional  $\Omega(\beta)$ -strongly convex quadratic function and show that the SAM iterates blow up to infinity. For the same reason, in our other non-convergence results (Theorems 4.2 and 4.4), we only focus on  $\eta = \mathcal{O}\left(\frac{1}{\beta}\right)$ .
3.4 Nonsmooth Lipschitz Convex Functions
Previous theorems study convergence of SAM assuming smoothness. The next theorem shows an example where SAM on a nonsmooth convex function converges only up to  $\Omega (\rho)$  distance from the global minimum  $x^{*}$ . This means that for constant perturbation size  $\rho$ , there exist convex functions that prevent SAM from converging to global minima. In Appendix B.7, we prove the following:
Theorem 3.6. For any  $\rho > 0$  and  $\eta < \frac{7\rho}{4}$ , there exists a nonsmooth Lipschitz convex function  $f$  such that for some initialization, deterministic SAM converges to suboptimal points located at a distance of  $\Omega(\rho)$  from the global minimum.
4 Convergence Analysis Under Stochastic Settings
In this section, we present the main results on the convergence analysis of stochastic SAM, again with time-invariant (constant) perturbation size  $\rho$  and gradient normalization. We consider both types of stochastic SAM:  $n$ -SAM and  $m$ -SAM, defined in Section 2.2. We study four types of function classes: smooth strongly convex, smooth convex, smooth nonconvex, and smooth Lipschitz nonconvex functions, under the assumption that the gradient oracle has bounded variance of  $\sigma^2$  (Assumption 2.5). Our results in this section reveal that stochastic SAM exhibits different convergence properties compared to deterministic SAM.
4.1 Smooth and Strongly Convex Functions
Theorem 3.1 shows the convergence of deterministic SAM to global optima, for smooth strongly convex functions. Unlike this result, we find that under stochasticity and constant perturbation size  $\rho$ , both  $n$ -SAM and  $m$ -SAM ensure convergence only up to an additive factor  $\mathcal{O}(\rho^2)$ .
Theorem 4.1. Consider a  $\beta$ -smooth,  $\mu$ -strongly convex function  $f$ , and assume Assumption 2.5. Under  $n$ -SAM, starting at  $x_0$  with any perturbation size  $\rho > 0$  and step size  $\eta =$
(d)
Figure 2: Examples of virtual loss plot for deterministic and stochastic SAM. The graph drawn in green indicates  $f$ , and the graph drawn in blue indicates  $J_{f}$  (or  $\mathbb{E}J_{f}$ ). Red arrows indicate the (expected) directions of SAM update. The (expected) updates are directed to red stars. (a)  $f$  and  $J_{f}$  in Theorem 3.5. (b)  $f$  and its component functions  $f^{(1)}, f^{(2)}$  in Theorem 4.2. (c)  $\mathbb{E}J_{f}$  and its component functions  $J_{f^{(1)}}, J_{f^{(2)}}$  in Theorem 4.2. (d)  $f$  and its component functions  $f^{(1)}, f^{(2)}$  in Theorem 4.4. (e)  $\mathbb{E}J_{f}$  and its component functions  $J_{f^{(1)}}, J_{f^{(2)}}$  in Theorem 4.4. For the simulation results of SAM trajectories on these functions, refer to Appendix D.
$\min \left\{\frac{1}{\mu T}\cdot \max \left\{1,\log \left(\frac{\mu^2\Delta T}{\beta[\sigma^2 - \beta^2\rho^2]_+}\right)\right\} ,\frac{1}{2\beta}\right\}$  to minimize  $f$  , we have
$$
\mathbb {E} f (\pmb {x} _ {T}) - f ^ {*} \leq \tilde {\mathcal {O}} \left(\exp \left(- \frac {\mu T}{2 \beta}\right) \Delta + \frac {\beta [ \sigma^ {2} - \beta^ {2} \rho^ {2} ] _ {+}}{\mu^ {2} T}\right) + \frac {2 \beta^ {2} \rho^ {2}}{\mu}.
$$
Under  $m$ -SAM, additionally assuming  $l(\cdot, \xi)$  is  $\beta$ -smooth for any  $\xi$ , the inequality continues to hold.
For the proof, please refer to Appendix C.2. Theorem 4.1 provides a convergence rate of  $\tilde{\mathcal{O}}\left(\frac{1}{T}\right)$  to global minima, but only up to suboptimality gap  $\frac{2\beta^2\rho^2}{\mu}$ . If  $\sigma \leq \beta \rho$ , then Theorem 4.1 becomes:
$$
\mathbb {E} f (\boldsymbol {x} _ {T}) - f ^ {*} \leq \tilde {\mathcal {O}} \left(\exp \left(- \frac {\mu T}{2 \beta}\right) \Delta\right) + \frac {2 \beta^ {2} \rho^ {2}}{\mu},
$$
thereby showing a convergence rate of  $\mathcal{O}\left(\exp (-T)\right)$  modulo the additive factor.
For relevant existing studies, Theorem 2 of Andriushchenko and Flammarion [2] proves the convergence guarantee for smooth Polyak-Lojasiewicz functions:  $\mathbb{E}f(\pmb{x}_T) - f^* = \mathcal{O}\left(\frac{1}{T}\right)$ , but assuming stochastic SAM without gradient normalization, and perturbation size  $\rho$  decaying with  $t$ . In contrast, our analysis shows that the convergence property can be different with normalization and constant  $\rho$ .
The reader might be curious if the additional  $\mathcal{O}(\rho^2)$  term can be removed. Our next theorem proves that in the case of high gradient noise ( $\sigma > \beta \rho$ ),  $m$ -SAM with constant perturbation size  $\rho$  cannot converge to global optima beyond a suboptimality gap  $\Omega(\rho^2)$ , when the component function  $l(\pmb{x}; \xi)$  is smooth for any  $\xi$ . Hence, the additional term in Theorem 4.1 is unavoidable, at least for the more practical version  $m$ -SAM.
Theorem 4.2. For any  $\rho >0,\beta >0,\sigma >\beta \rho$  and  $\eta \leq \frac{3}{10\beta}$ , there exists a  $\beta$ -smooth and  $\frac{\beta}{5}$ -strongly convex function  $f$  satisfying the following. (1) The function  $f$  satisfies Assumption 2.5. (2) The component functions  $l(\cdot ;\xi)$  of  $f$  are  $\beta$ -smooth for any  $\xi$ . (3) If we run m-SAM on  $f$  initialized inside
a certain interval, then any arbitrary weighted average  $\bar{\pmb{x}}$  of the iterates  $x_0, x_1, \ldots$  must satisfy  $\mathbb{E}[f(\bar{\pmb{x}}) - f^*] \geq \Omega(\rho^2)$ .
Here we provide a brief outline for Theorem 4.2. The in-depth analysis is provided in Appendix C.3. Given  $\rho >0$ ,  $\beta >0$ ,  $\sigma >\beta \rho$ , we consider a one-dimensional quadratic function  $f(x)$  whose component function  $l(x;\xi)$  is carefully chosen to satisfy the following for  $x\in [\frac{\rho}{6},\frac{13\rho}{6} ]$
$$
f (x) = \mathbb {E} [ l (x; \xi) ] = \frac {\beta}{1 0} x ^ {2}, \quad l (x; \xi) = \left\{ \begin{array}{l l} - \frac {\beta}{1 0} \left(x - \frac {7 \rho}{6}\right) ^ {2}, & \text {w i t h p r o b a b i l i t y} \frac {2}{3} \\ \frac {3 \beta}{1 0} x ^ {2} + \frac {\beta}{5} \left(x - \frac {7 \rho}{6}\right) ^ {2}, & \text {o t h e r w i s e}. \end{array} \right.
$$
For values of  $x$  outside this interval  $\left[\frac{\rho}{6}, \frac{13\rho}{6}\right]$ , each component function  $l(x; \xi)$  takes the form of a strongly convex quadratic function. Figures 2(b) and 2(c) illustrate the original and virtual loss function plots of  $l(x; \xi)$ . The local concavity of component function plays a crucial role in making an attracting basin in the virtual loss, and this leads to an interval  $\left[\frac{\rho}{6}, \frac{13\rho}{6}\right]$  bounded away from the global minimum 0, from which  $m$ -SAM iterates cannot escape.
For this scenario, we obtain  $f(x_{t}) - f^{*} = \Omega (\rho^{2})$  and  $\| \nabla f(x_{t})\|^{2} = \Omega (\rho^{2})$  for all iterates. From this, we can realize that the additive factor in Theorem 4.1 for  $m$ -SAM is unavoidable in the  $\sigma >\beta \rho$  regime, and tight in terms of  $\rho$ . Moreover, the proof of Theorem 4.2 in Appendix C.3 reveals that even in the  $\sigma \leq \beta \rho$  case, an analogous example gives  $\| x_{t} - x^{*}\| = \Omega (\rho)$  for all iterates; hence,  $m$ -SAM fails to converge all the way to the global minimum in the small  $\sigma$  regime as well.
4.2 Smooth and Convex Functions
We now move on to smooth convex functions and investigate the convergence guarantees of stochastic SAM for this function class. As can be guessed from Theorem 3.3, our convergence analysis in this section focuses on finding stationary points. Below, we provide a bound that ensures convergence to stationary points up to an additive factor.
Theorem 4.3. Consider a  $\beta$ -smooth, convex function  $f$ , and assume Assumption 2.5. Under  $n$ -SAM, starting at  $x_0$  with any perturbation size  $\rho > 0$  and step size  $\eta = \min \left\{\frac{\sqrt{\Delta}}{\sqrt{\beta[\sigma^2 - \beta^2\rho^2] + T}}, \frac{1}{2\beta}\right\}$  to minimize  $f$ , we have
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} = \mathcal {O} \left(\frac {\beta \Delta}{T} + \frac {\sqrt {\beta [ \sigma^ {2} - \beta^ {2} \rho^ {2} ] + \Delta}}{\sqrt {T}}\right) + 4 \beta^ {2} \rho^ {2}.
$$
Under  $m$ -SAM, additionally assuming  $l(\cdot ,\xi)$  is  $\beta$ -smooth for any  $\xi$ , the inequality continues to hold.
The proof of Theorem 4.3 can be found in Appendix C.4. Theorem 4.3 obtains a bound of  $\mathcal{O}\left(\frac{1}{\sqrt{T}}\right)$  modulo an additive factor  $4\beta^{2}\rho^{2}$ . Similar to Theorem 4.1, if  $\sigma \leq \beta \rho$ , then Theorem 4.3 reads
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} = \mathcal {O} \left(\frac {\beta \Delta}{T}\right) + 4 \beta^ {2} \rho^ {2},
$$
hence showing a convergence rate of  $\mathcal{O}\left(\frac{1}{T}\right)$  modulo the additive factor. Since the non-convergence example in Theorem 4.2 provides a scenario that  $\mathbb{E}\| \nabla f(x_t)\|^2 = \Omega (\rho^2)$  for all  $t$ , we can see that the extra term is inevitable and also tight in terms of  $\rho$ .
Theorem 4.3 sounds quite weak, as it only proves convergence to a stationary point only up to an extra term. One could anticipate that stochastic SAM may actually converge to global minima of smooth convex functions modulo the unavoidable additive factor. However, as briefly mentioned in Section 3.2, the next theorem presents a counterexample illustrating that ensuring convergence to global minima, even up to an additive factor, is impossible for  $m$ -SAM.
Theorem 4.4. For any  $\rho >0$ ,  $\beta >0$ ,  $\sigma >0$ , and  $\eta \leq \frac{1}{\beta}$ , there exists a  $\beta$ -smooth and convex function  $f$  satisfying the following. (1) The function  $f$  satisfies Assumption 2.5. (2) The component functions  $l(\cdot ;\xi)$  of  $f$  are  $\beta$ -smooth for any  $\xi$ . (3) If we run m-SAM on  $f$  initialized inside a certain interval, then any arbitrary weighted average  $\bar{\pmb{x}}$  of the iterates  $\pmb {x}_0,\pmb {x}_1,\dots$  must satisfy  $\mathbb{E}[f(\bar{\pmb{x}}) - f^{*}]\geq C$  and the suboptimality gap  $C$  can be made arbitrarily large and independent of the parameter  $\rho$ .
Here, we present the intuitions of the proof for Theorem 4.4. As demonstrated in Figures 2(d) and 2(e), the local concavity of the component function significantly influences the formation of attracting
basins in its virtual loss, thereby creating a region from which the  $m$ -SAM updates get stuck inside the basin forever.
Also note that we can construct the function to form the basin at any point with arbitrary large function value (and hence large suboptimality gap). Therefore, establishing an upper bound on the convergence of function value becomes impossible in smooth convex functions. A detailed analysis for the non-convergence example is presented in Appendix C.5.
4.3 Smooth and Nonconvex Functions
We now study smooth nonconvex functions. Extending Theorem 3.4, we can show the following bound for stochastic  $n$ -SAM.
Theorem 4.5. Consider a  $\beta$ -smooth function  $f$  satisfying  $f^{*} = \inf_{x} f(\pmb{x}) > -\infty$ , and assume Assumption 2.5. Under  $n$ -SAM, starting at  $\pmb{x}_0$  with any perturbation size  $\rho > 0$  and step size  $\eta = \min \left\{\frac{1}{2\beta}, \frac{\sqrt{\Delta}}{\sqrt{\beta \sigma^2 T}}\right\}$  to minimize  $f$ , we have
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} \leq \mathcal {O} \left(\frac {\beta \Delta}{T} + \frac {\sqrt {\beta \sigma^ {2} \Delta}}{\sqrt {T}}\right) + \beta^ {2} \rho^ {2}.
$$
The proof of Theorem 4.5 is provided in Appendix C.6. Notice that the non-convergence example presented in Theorem 3.5 (already) illustrates a scenario where  $\mathbb{E}\| \nabla f(x_t)\|^2 = \Omega (\rho^2)$ , thereby confirming the tightness of additive factor in terms of  $\rho$ .
The scope of applicability for Theorem 4.5 is limited to  $n$ -SAM. Compared to  $n$ -SAM,  $m$ -SAM employs a stronger assumption where  $\xi = \tilde{\xi}$ . By imposing an additional Lipschitzness condition on the function  $f$ ,  $m$ -SAM leads to a similar but different convergence result.
Theorem 4.6. Consider a  $\beta$ -smooth,  $L$ -Lipschitz continuous function  $f$  satisfying  $f^{*} = \inf_{x} f(x) > -\infty$ , and assume Assumption 2.5. Additionally assume  $l(\cdot, \xi)$  is  $\beta$ -smooth for any  $\xi$ . Under m-SAM, starting at  $x_0$  with any perturbation size  $\rho > 0$  and step size  $\eta = \frac{\sqrt{\Delta}}{\sqrt{\beta(\sigma^2 + L^2)T}}$  to minimize  $f$ , we have
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ (\| \nabla f (\boldsymbol {x} _ {t}) \| - \beta \rho) ^ {2} \right] \leq \mathcal {O} \left(\frac {\sqrt {\beta \Delta (\sigma^ {2} + L ^ {2})}}{\sqrt {T}}\right) + 5 \beta^ {2} \rho^ {2}.
$$
Corollary 4.7. Under the setting of Theorem 4.6, we get
$$
\min  _ {t \in \{0, \dots , T \}} \left\{\mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| \right\} \leq \mathcal {O} \left(\frac {\left(\beta \Delta \left(\sigma^ {2} + L ^ {2}\right)\right) ^ {1 / 4}}{T ^ {1 / 4}}\right) + \left(1 + \sqrt {5}\right) \beta \rho .
$$
The proofs for Theorem 4.6 and Corollary 4.7 are given in Appendix C.7. Since Theorem 3.5 presents an example where  $\mathbb{E}\| \nabla f(\pmb{x}_t)\| = \Omega (\rho)$  and  $\mathbb{E}(\| \nabla f(\pmb{x}_t)\| -\beta \rho)^2 = \Omega (\rho^2)$ , we can verify that the additive factors in Theorem 4.6 and Corollary 4.7 are tight in terms of  $\rho$ .
As for previous studies, Theorems 2, 12, and 18 of Andriushchenko and Flammarion [2] prove the convergence of  $n,m$ -SAM for smooth nonconvex functions:  $\frac{1}{T}\sum_{t=0}^{T}\mathbb{E}\|\nabla f(\boldsymbol{x}_t)\|^2 = \mathcal{O}\left(\frac{1}{\sqrt{T}}\right)$ , but assuming SAM without gradient normalization, and sufficiently small  $\rho$ . For  $m$ -SAM on smooth Lipschitz nonconvex functions, Theorem 1 of Mi et al. [27] proves  $\frac{1}{T}\sum_{t=0}^{T}\mathbb{E}\|\nabla f(\boldsymbol{x}_t)\|^2 = \tilde{\mathcal{O}}\left(\frac{1}{\sqrt{T}}\right)$ , while assuming decaying  $\rho$ . From our results, we demonstrate that such full convergence results are impossible for practical versions of stochastic SAM.
This paper studies the convergence properties of SAM, under constant  $\rho$  and with gradient normalization. We establish convergence guarantees of deterministic SAM for smooth and (strongly) convex functions. To our surprise, we discover scenarios in which deterministic SAM (for smooth nonconvex functions) and stochastic  $m$ -SAM (for all function class considered) converge only up to unavoidable additive factors proportional to  $\rho^2$ . Our findings emphasize the drastically different characteristics of SAM with vs. without decaying perturbation size. Establishing tighter bounds in terms of  $\beta$  and  $\mu$ , or searching for a non-convergence example that applies to  $n$ -SAM might be interesting future research directions.
This paper was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant (No. 2019-0-00075, Artificial Intelligence Graduate School Program (KAIST)) funded by the Korea government (MSIT), two National Research Foundation of Korea (NRF) grants (No. NRF-2019R1A5A1028324, RS-2023-00211352) funded by the Korea government (MSIT), and a grant funded by Samsung Electronics Co., Ltd.