SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Furthermore, we can verify that for  $\pmb{x} \in (\mathbb{B} \cup \mathbb{C} \cup \mathbb{D})$ ,  $b^{(1)} < -\frac{7\rho}{2}$  holds. Therefore, summing up all the cases, we can conclude that if the initial iterate  $\pmb{x}_0$  is chosen from  $\pmb{x}_0 \in (\mathbb{B} \cup \mathbb{C} \cup \mathbb{D})$ , then all subsequent SAM iterates  $\pmb{x}_t$  will satisfy  $b_t^{(1)} < -\frac{7\rho}{2} + \eta$ . Since the global minimum is located at  $\pmb{x}^* = (0,0)$ , it follows that  $\| \pmb{x}_t - \pmb{x}^* \| > \frac{|7\rho / 2 - \eta|}{\sqrt{5}}$  for every  $t > 0$ . Thus, for  $\eta < \frac{7\rho}{4}$ , we can ascertain that in this particular function,  $\| \pmb{x}_t - \pmb{x}^* \| > \frac{7\rho}{4\sqrt{5}}$  for every  $t > 0$ , thereby proving that the distance from the global minimum is at least  $\Omega(\rho)$ . The trajectory plot of SAM on this example function is illustrated in Figure 3(b).
C Proofs for (Non-)Convergence of Stochastic SAM
In this section, we provide in-depth demonstrations and proofs regarding convergence of stochastic SAM with constant perturbation size  $\rho$ . The objective is defined as  $f(\pmb{x}) = \mathbb{E}_{\xi}[l(\pmb{x};\xi)]$ , where  $\xi$  represents a stochastic parameter (e.g., data sample) and  $l(\pmb{x};\xi)$  represents the loss at point  $\pmb{x}$  with a random sample  $\xi$ . The update iteration for stochastic SAM is specified as follows:
$$
\left\{ \begin{array}{l} \boldsymbol {y} _ {t} = \boldsymbol {x} _ {t} + \rho \frac {g (\boldsymbol {x} _ {t})}{\| g (\boldsymbol {x} _ {t}) \|}, \\ \boldsymbol {x} _ {t + 1} = \boldsymbol {x} _ {t} - \eta \tilde {g} (\boldsymbol {y} _ {t}). \end{array} \right.
$$
We define  $g(\pmb{x}) = \nabla_{\pmb{x}} l(\pmb{x}; \xi)$  and  $\tilde{g}(\pmb{x}) = \nabla_{\pmb{x}} l(\pmb{x}; \tilde{\xi})$ , where  $\xi$  and  $\tilde{\xi}$  are stochastic parameters. Stochastic SAM comes in two variations:  $n$ -SAM, where  $\xi$  and  $\tilde{\xi}$  are independent, and  $m$ -SAM, where  $\xi$  is equal to  $\tilde{\xi}$ .
C.1 Important Lemmas Regarding Stochastic SAM
In this section, we present a number of lemmas that are utilized in our theorem proofs regarding stochastic SAM. In order to do this, we also introduce extra notation,  $\hat{\pmb{y}}_t = \pmb{x}_t + \rho \frac{\nabla f(\pmb{x}_t)}{\|\nabla f(\pmb{x}_t)\|}$ , as a deterministically ascended parameter.
Lemma C.1. For a differentiable and  $\mu$ -strongly convex function  $f$ , we have
$$
\mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \tilde {g} (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \rangle \geq \mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \tilde {g} (\boldsymbol {y} _ {t}) - \tilde {g} (\hat {\boldsymbol {y}} _ {t}) \rangle + \mu \rho \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \|.
$$
$$
\begin{array}{l} \mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \tilde {g} (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \rangle = \mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \tilde {g} (\boldsymbol {y} _ {t}) - \tilde {g} (\hat {\boldsymbol {y}} _ {t}) \rangle + \mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \tilde {g} (\hat {\boldsymbol {y}} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \rangle \\ = \mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \tilde {g} (\boldsymbol {y} _ {t}) - \tilde {g} (\hat {\boldsymbol {y}} _ {t}) \rangle + \mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \nabla f (\hat {\boldsymbol {y}} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \rangle \\ \geq \mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \tilde {g} (\boldsymbol {y} _ {t}) - \tilde {g} (\hat {\boldsymbol {y}} _ {t}) \rangle + \mu \rho \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \|, \\ \end{array}
$$
where we use Lemma B.2 in the last inequality, thereby completing the proof.
Lemma C.2. Under  $n$ -SAM, for a  $\beta$ -smooth and  $\mu$ -strongly convex function  $f$ , we have
$$
\mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \tilde {g} (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \rangle \geq - \frac {1}{2} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} + \mu \rho \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| - 2 \beta^ {2} \rho^ {2}.
$$
Proof. First we consider  $n$ -SAM. Starting from Lemma C.1, we have
$$
\begin{array}{l} \mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \tilde {g} (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \rangle \geq \mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \tilde {g} (\boldsymbol {y} _ {t}) - \tilde {g} (\hat {\boldsymbol {y}} _ {t}) \rangle + \mu \rho \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| \\ = \mathbb {E} \left\langle \nabla f (\boldsymbol {x} _ {t}), \nabla f (\boldsymbol {y} _ {t}) - \nabla f (\hat {\boldsymbol {y}} _ {t}) \right\rangle + \mu \rho \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| \\ \geq - \frac {1}{2} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} - \frac {1}{2} \mathbb {E} \| \nabla f (\boldsymbol {y} _ {t}) - \nabla f (\hat {\boldsymbol {y}} _ {t}) \| ^ {2} + \mu \rho \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| \\ \geq - \frac {1}{2} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} - \frac {\beta^ {2}}{2} \mathbb {E} \| \boldsymbol {y} _ {t} - \hat {\boldsymbol {y}} _ {t} \| ^ {2} + \mu \rho \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| \\ = - \frac {1}{2} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} - \frac {\beta^ {2}}{2} \mathbb {E} \left\| \rho \frac {g (\boldsymbol {x} _ {t})}{\| g (\boldsymbol {x} _ {t}) \|} - \rho \frac {\nabla f (\boldsymbol {x} _ {t})}{\| \nabla f (\boldsymbol {x} _ {t}) \|} \right\| ^ {2} \\ + \mu \rho \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| \\ \geq - \frac {1}{2} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} + \mu \rho \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| - 2 \beta^ {2} \rho^ {2}. \\ \end{array}
$$
Next we consider  $m$ -SAM. additionally assuming  $l(\cdot, \xi)$  is  $\beta$ -smooth for any  $\xi$ , starting from Lemma C.1, we have
$$
\mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \tilde {g} (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \rangle \geq \mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \tilde {g} (\boldsymbol {y} _ {t}) - \tilde {g} (\hat {\boldsymbol {y}} _ {t}) \rangle + \mu \rho \| \nabla f (\boldsymbol {x} _ {t}) \|
$$
$$
\begin{array}{l} \geq - \frac {1}{2} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} - \frac {1}{2} \mathbb {E} \| \tilde {g} (\boldsymbol {y} _ {t}) - \tilde {g} (\hat {\boldsymbol {y}} _ {t}) \| ^ {2} + \mu \rho \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| \\ \geq - \frac {1}{2} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} - \frac {\beta^ {2}}{2} \mathbb {E} \| \boldsymbol {y} _ {t} - \hat {\boldsymbol {y}} _ {t} \| ^ {2} + \mu \rho \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| \\ = - \frac {1}{2} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} - \frac {\beta^ {2}}{2} \mathbb {E} \left\| \rho \frac {g (\boldsymbol {x} _ {t})}{\| g (\boldsymbol {x} _ {t}) \|} - \rho \frac {\nabla f (\boldsymbol {x} _ {t})}{\| \nabla f (\boldsymbol {x} _ {t}) \|} \right\| ^ {2} \\ + \mu \rho \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| \\ \geq - \frac {1}{2} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} + \mu \rho \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| - 2 \beta^ {2} \rho^ {2}, \\ \end{array}
$$
Lemma C.3. Consider a  $\beta$ -smooth,  $\mu$ -strongly convex function  $f$ , and assume Assumption 2.5. Under  $n$ -SAM, with step size  $\eta \leq \frac{1}{2\beta}$ , we have
$$
\mathbb {E} f (\boldsymbol {x} _ {t + 1}) \leq \mathbb {E} f (\boldsymbol {x} _ {t}) - \frac {\eta}{2} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} - \frac {\eta \mu \rho}{2} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| + 2 \eta \beta^ {2} \rho^ {2} - \eta^ {2} \beta (\beta^ {2} \rho^ {2} - \sigma^ {2}).
$$
$$
\begin{array}{l} \mathbb {E} f (\boldsymbol {x} _ {t + 1}) \leq \mathbb {E} f (\boldsymbol {x} _ {t}) - \eta \mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \tilde {g} (\boldsymbol {y} _ {t}) \rangle + \frac {\eta^ {2} \beta}{2} \mathbb {E} \| \tilde {g} (\boldsymbol {y} _ {t}) \| ^ {2} \\ = \mathbb {E} f (\boldsymbol {x} _ {t}) - \eta \mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \tilde {g} (\boldsymbol {y} _ {t}) \rangle + \frac {\eta^ {2} \beta}{2} \mathbb {E} \| \tilde {g} (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} + \frac {\eta^ {2} \beta}{2} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} \\ + \eta^ {2} \beta \mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \tilde {g} (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \rangle \\ = \mathbb {E} f (\boldsymbol {x} _ {t}) - \eta \mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \tilde {g} (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \rangle - \eta \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} \\ + \frac {\eta^ {2} \beta}{2} \mathbb {E} \| \tilde {g} (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} + \frac {\eta^ {2} \beta}{2} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} \\ + \eta^ {2} \beta \mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \tilde {g} (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \rangle \\ \leq \mathbb {E} f (\boldsymbol {x} _ {t}) - \eta (1 - \eta \beta) \mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \tilde {g} (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \rangle - \eta \left(1 - \frac {\eta \beta}{2}\right) \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} \\ + \eta^ {2} \beta \mathbb {E} \| \tilde {g} (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {y} _ {t}) \| ^ {2} + \eta^ {2} \beta \mathbb {E} \| \nabla f (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} \\ \leq \mathbb {E} f (\boldsymbol {x} _ {t}) - \eta (1 - \eta \beta) \mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \tilde {g} (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \rangle - \eta \left(1 - \frac {\eta \beta}{2}\right) \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} \\ + \eta^ {2} \beta (\sigma^ {2} + \beta^ {2} \rho^ {2}). \\ \end{array}
$$
Since we assumed  $\eta \leq \frac{1}{2\beta}$ ,  $\eta (1 - \eta \beta) \geq \frac{\eta}{2}$  hold. Applying Lemma C.2, we get
$$
\begin{array}{l} \mathbb {E} f \left(\boldsymbol {x} _ {t + 1}\right) \leq \mathbb {E} f \left(\boldsymbol {x} _ {t}\right) - \eta (1 - \eta \beta) \left(- \frac {1}{2} \mathbb {E} \| \nabla f \left(\boldsymbol {x} _ {t}\right) \| ^ {2} + \mu \rho \mathbb {E} \| \nabla f \left(\boldsymbol {x} _ {t}\right) \| - 2 \beta^ {2} \rho^ {2}\right) \\ - \eta \left(1 - \frac {\eta \beta}{2}\right) \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} + \eta^ {2} \beta (\sigma^ {2} + \beta^ {2} \rho^ {2}) \\ = \mathbb {E} f (\pmb {x} _ {t}) - \frac {\eta}{2} \mathbb {E} \| \nabla f (\pmb {x} _ {t}) \| ^ {2} - \eta (1 - \eta \beta) \mu \rho \mathbb {E} \| \nabla f (\pmb {x} _ {t}) \| + 2 \eta \beta^ {2} \rho^ {2} - \eta^ {2} \beta (\beta^ {2} \rho^ {2} - \sigma^ {2}) \\ \leq \mathbb {E} f (\boldsymbol {x} _ {t}) - \frac {\eta}{2} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} - \frac {\eta \mu \rho}{2} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| + 2 \eta \beta^ {2} \rho^ {2} - \eta^ {2} \beta (\beta^ {2} \rho^ {2} - \sigma^ {2}), \\ \end{array}
$$
C.2 Convergence Proof for Smooth and Strongly Convex Functions (Proof of Theorem 4.1)
In this section, we demonstrate the convergence result of stochastic SAM for smooth and strongly convex functions. For convenience, we restate the theorem here.
Theorem 4.1. Consider a  $\beta$ -smooth,  $\mu$ -strongly convex function  $f$ , and assume Assumption 2.5. Under  $n$ -SAM, starting at  $x_0$  with any perturbation size  $\rho > 0$  and step size  $\eta = \min \left\{\frac{1}{\mu T} \cdot \max \left\{1, \log \left(\frac{\mu^2 \Delta T}{\beta[\sigma^2 - \beta^2 \rho^2]_+}\right)\right\}, \frac{1}{2\beta}\right\}$  to minimize  $f$ , we have
Proof. We start the proof from Lemma C.3; in order to apply the lemma, additionally assuming  $\beta$ -smoothness for component functions  $l(\cdot, \xi)$  is necessary for  $m$ -SAM.
Since  $\mu$ -strongly convex functions satisfy  $\mu$ -PL inequality (B.1), we get
$$
\mathbb {E} f (\boldsymbol {x} _ {t + 1}) - f ^ {*} \leq (1 - \eta \mu) (\mathbb {E} f (\boldsymbol {x} _ {t}) - f ^ {*}) - \frac {\eta \mu \rho}{2} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| + 2 \eta \beta^ {2} \rho^ {2} - \eta^ {2} \beta (\beta^ {2} \rho^ {2} - \sigma^ {2}).
$$
Depending on the value of  $\sigma$ , there are two cases in which the convergence rate varies.
Case A:  $\sigma \leq \beta \rho$ . In this case, we have
$$
\begin{array}{l} \mathbb {E} f (\boldsymbol {x} _ {t + 1}) - f ^ {*} \leq (1 - \eta \mu) (\mathbb {E} f (\boldsymbol {x} _ {t}) - f ^ {*}) - \frac {\eta \mu \rho}{2} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| + 2 \eta \beta^ {2} \rho^ {2} \\ \leq (1 - \eta \mu) (\mathbb {E} f (\boldsymbol {x} _ {t}) - f ^ {*}) + 2 \eta \beta^ {2} \rho^ {2}, \\ \end{array}
$$
and our choice of  $\eta$  must be  $\frac{1}{2\beta}$ . Unrolling the inequality and substituting  $\eta = \frac{1}{2\beta}$  draws out
$$
\begin{array}{l} \mathbb {E} f (\boldsymbol {x} _ {T}) - f ^ {*} \leq (1 - \eta \mu) ^ {T} (\mathbb {E} f (\boldsymbol {x} _ {0}) - f ^ {*}) + \frac {2 \beta^ {2} \rho^ {2}}{\mu} \\ = \left(1 - \frac {\mu}{2 \beta}\right) ^ {T} \Delta + \frac {2 \beta^ {2} \rho^ {2}}{\mu} \\ \leq \exp \left(- \frac {\mu T}{2 \beta}\right) \Delta + \frac {2 \beta^ {2} \rho^ {2}}{\mu}. \\ \end{array}
$$
Case B:  $\sigma >\beta \rho$  .In this case, we have
$$
\begin{array}{l} \mathbb {E} f (\boldsymbol {x} _ {t + 1}) - f ^ {*} \leq (1 - \eta \mu) (\mathbb {E} f (\boldsymbol {x} _ {t}) - f ^ {*}) - \frac {\eta \mu \rho}{2} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| + 2 \eta \beta^ {2} \rho^ {2} + \eta^ {2} \beta (\sigma^ {2} - \beta^ {2} \rho^ {2}) \\ \leq (1 - \eta \mu) (\mathbb {E} f (\boldsymbol {x} _ {t}) - f ^ {*}) + 2 \eta \beta^ {2} \rho^ {2} + \eta^ {2} \beta (\sigma^ {2} - \beta^ {2} \rho^ {2}). \\ \end{array}
$$
Again, unrolling the inequality draws out
$$
\mathbb {E} f \left(\boldsymbol {x} _ {T}\right) - f ^ {*} \leq (1 - \eta \mu) ^ {T} \Delta + \frac {2 \beta^ {2} \rho^ {2}}{\mu} + \frac {\eta \beta \left(\sigma^ {2} - \beta^ {2} \rho^ {2}\right)}{\mu}. \tag {10}
$$
Similar to Section B.2, substituting  $\eta = \min \left\{\frac{1}{\mu T} \cdot \max \left\{1, \log \left(\frac{\mu^2 \Delta T}{\beta (\sigma^2 - \beta^2 \rho^2)}\right)\right\}, \frac{1}{2\beta}\right\}$  can result in four cases.
Case B-1:  $\log \left(\frac{\mu^2\Delta T}{\beta(\sigma^2 - \beta^2\rho^2)}\right) \geq 1$ , and  $\frac{1}{2\beta} \geq \frac{1}{\mu T}\log \left(\frac{\mu^2\Delta T}{\beta(\sigma^2 - \beta^2\rho^2)}\right)$ . Setting  $\eta = \frac{1}{\mu T}\log \left(\frac{\mu^2\Delta T}{\beta(\sigma^2 - \beta^2\rho^2)}\right)$ ,
$$
\mathbb {E} f (\pmb {x} _ {T}) - f ^ {*} \leq \frac {\beta (\sigma^ {2} - \beta^ {2} \rho^ {2})}{\mu^ {2} T} + \frac {\beta (\sigma^ {2} - \beta^ {2} \rho^ {2})}{\mu^ {2} T} \cdot \log \left(\frac {\mu^ {2} \Delta T}{\beta (\sigma^ {2} - \beta^ {2} \rho^ {2})}\right) + \frac {2 \beta^ {2} \rho^ {2}}{\mu}.
$$
Case B-2:  $\log \left(\frac{\mu^2\Delta T}{\beta(\sigma^2 - \beta^2\rho^2)}\right)\geq 1,$  and  $\frac{1}{2\beta}\leq \frac{1}{\mu T}\log \left(\frac{\mu^2\Delta T}{\beta(\sigma^2 - \beta^2\rho^2)}\right)$ . Setting  $\eta = \frac{1}{2\beta}$ ,
$$
\begin{array}{l} \mathbb {E} f (\boldsymbol {x} _ {T}) - f ^ {*} \leq \exp \left(- \frac {\mu T}{2 \beta}\right) \Delta + \frac {\beta \left(\sigma^ {2} - \beta^ {2} \rho^ {2}\right)}{\mu} \cdot \frac {1}{2 \beta} + \frac {2 \beta^ {2} \rho^ {2}}{\mu} \\ \leq \exp \left(- \frac {\mu T}{2 \beta}\right) \Delta + \frac {\beta (\sigma^ {2} - \beta^ {2} \rho^ {2})}{\mu^ {2} T} \cdot \log \left(\frac {\mu^ {2} \Delta T}{\beta (\sigma^ {2} - \beta^ {2} \rho^ {2})}\right) + \frac {2 \beta^ {2} \rho^ {2}}{\mu}. \\ \end{array}
$$
Case B-3:  $\log \left(\frac{\mu^2\Delta T}{\beta(\sigma^2 - \beta^2\rho^2)}\right)\leq 1,$  and  $\frac{1}{2\beta}\geq \frac{1}{\mu T}$ . Setting  $\eta = \frac{1}{\mu T}$ ,
$$
\begin{array}{l} \mathbb {E} f (\boldsymbol {x} _ {T}) - f ^ {*} \leq \left(1 - \frac {1}{T}\right) ^ {T} \Delta + \frac {\beta (\sigma^ {2} - \beta^ {2} \rho^ {2})}{\mu^ {2} T} + \frac {2 \beta^ {2} \rho^ {2}}{\mu} \\ \leq \left(1 - \frac {1}{T} \log \left(\frac {\mu^ {2} \Delta T}{\beta \left(\sigma^ {2} - \beta^ {2} \rho^ {2}\right)}\right)\right) ^ {T} \Delta + \frac {\beta \left(\sigma^ {2} - \beta^ {2} \rho^ {2}\right)}{\mu^ {2} T} + \frac {2 \beta^ {2} \rho^ {2}}{\mu} \\ \leq \frac {\beta (\sigma^ {2} - \beta^ {2} \rho^ {2})}{\mu^ {2} T} + \frac {\beta (\sigma^ {2} - \beta^ {2} \rho^ {2})}{\mu^ {2} T} + \frac {2 \beta^ {2} \rho^ {2}}{\mu}. \\ \end{array}
$$
Case B-4:  $\log \left(\frac{\mu^2\Delta T}{\beta(\sigma^2 - \beta^2\rho^2)}\right)\leq 1,$  and  $\frac{1}{2\beta}\leq \frac{1}{\mu T}$  Setting  $\eta = \frac{1}{2\beta}$
$$
\begin{array}{l} \mathbb {E} f (\boldsymbol {x} _ {T}) - f ^ {*} \leq \exp \left(- \frac {\mu T}{2 \beta}\right) \Delta + \frac {\beta \left(\sigma^ {2} - \beta^ {2} \rho^ {2}\right)}{\mu} \cdot \frac {1}{2 \beta} + \frac {2 \beta^ {2} \rho^ {2}}{\mu} \\ \leq \exp \left(- \frac {\mu T}{2 \beta}\right) \Delta + \frac {\beta \left(\sigma^ {2} - \beta^ {2} \rho^ {2}\right)}{\mu^ {2} T} + \frac {2 \beta^ {2} \rho^ {2}}{\mu}. \\ \end{array}
$$
Merging all four cases, we get
$$
\mathbb {E} f (\boldsymbol {x} _ {T}) - f ^ {*} = \tilde {\mathcal {O}} \left(\exp \left(- \frac {\mu T}{2 \beta}\right) \Delta + \frac {\beta (\sigma^ {2} - \beta^ {2} \rho^ {2})}{\mu^ {2} T}\right) + \frac {2 \beta^ {2} \rho^ {2}}{\mu},
$$
thereby finishing the proof.
C.3 Non-Convergence of  $m$ -SAM for a Smooth and Strongly Convex Function (Proof of Theorem 4.2)
In this section, we present a formal analysis of our counterexample, which shows that stochastic  $m$ -SAM with constant perturbation size  $\rho$  may fail to fully converge to a global minimum, and can get close to the global minimum by only  $\Omega(\rho)$ . This counterexample indicates that the  $\mathcal{O}(\rho^2)$  term in Theorem 4.1 is unavoidable. For readers' convenience, we restate the theorem.
Theorem 4.2. For any  $\rho >0,\beta >0,\sigma >\beta \rho$  and  $\eta \leq \frac{3}{10\beta}$ , there exists a  $\beta$ -smooth and  $\frac{\beta}{5}$ -strongly convex function  $f$  satisfying the following. (1) The function  $f$  satisfies Assumption 2.5. (2) The component functions  $l(\cdot ;\xi)$  of  $f$  are  $\beta$ -smooth for any  $\xi$ . (3) If we run m-SAM on  $f$  initialized inside a certain interval, then any arbitrary weighted average  $\bar{x}$  of the iterates  $x_0,x_1,\ldots$  must satisfy  $\mathbb{E}[f(\bar{x}) - f^* ]\geq \Omega (\rho^2)$ .
Proof. Given  $\rho > 0, \beta > 0, \sigma \geq 0$ , and  $\eta \leq \frac{3}{10\beta}$ , we choose  $p \triangleq \frac{2}{3}$ ,  $c \triangleq \left(1 + \frac{p}{4}\right)\rho$ . For  $x \in \mathbb{R}$ , and a constant  $a > 0$  which will be chosen later. Using these  $p, c$ , and  $a$ , we construct the counterexample. We consider a one-dimensional smooth strongly convex function
$$
f (x) = \frac {a}{2} x ^ {2},
$$
and the stochastic function  $l(x;\xi)$  can be given as
$$
l (x; \xi) = \left\{ \begin{array}{l l} f ^ {(1)} (x), & \text {w i t h p r o b a b i l i t y} p \\ f ^ {(2)} (x), & \text {o t h e r w i s e}, \end{array} \right.
$$
where each component functions are as described below:
$$
f ^ {(1)} (x) = \left\{ \begin{array}{l l} \frac {a}{2} (x - c + 2 \rho) ^ {2} - a \rho^ {2}, & x \leq c - \rho \\ - \frac {a}{2} (x - c) ^ {2}, & c - \rho \leq x \leq c + \rho \\ \frac {a}{2} (x - c - 2 \rho) ^ {2} - a \rho^ {2}, & c + \rho \leq x. \end{array} \right.
$$
$$
f ^ {(2)} (x) = \left\{ \begin{array}{l l} \frac {1}{1 - p} \left(\frac {a}{2} x ^ {2} - \frac {p a}{2} (x - c + 2 \rho) ^ {2} + p a \rho^ {2}\right), & x \leq c - \rho \\ \frac {1}{1 - p} \left(\frac {a}{2} x ^ {2} + \frac {p a}{2} (x - c) ^ {2}\right), & c - \rho \leq x \leq c + \rho \\ \frac {1}{1 - p} \left(\frac {a}{2} x ^ {2} - \frac {p a}{2} (x - c - 2 \rho) ^ {2} + p a \rho^ {2}\right), & c + \rho \leq x. \end{array} \right.
$$
First, it is easy to verify that  $\mathbb{E}l(x;\xi) = f(x)$ . We can also confirm that  $f$ ,  $f^{(1)}$  are  $a$ -smooth, and  $f^{(2)}$  is  $\left(\frac{1 + p}{1 - p} a\right)$ -smooth. Furthermore,  $\| \nabla f(x) - \nabla f^{(1)}(x) \|^2 \leq a^2 (2\rho + c)^2$  holds, along with  $\| \nabla f(x) - \nabla f^{(2)}(x) \|^2 \leq \left(\frac{pa}{1 - p}\right)^2 (2\rho + c)^2$ . Consequently, we have  $\mathbb{E} \| \nabla f(x) - \nabla l(x;\xi) \|^2 \leq \frac{pa^2}{1 - p} (2\rho + c)^2 = 2a^2 \cdot \left(\frac{19\rho}{6}\right)^2$ , where we used  $p = \frac{2}{3}$  and  $c = (1 + \frac{p}{4})\rho$ .
Now choose  $a = \min \left\{\frac{\beta}{5}, \frac{\sigma}{5\rho}\right\} \leq \frac{\beta}{5}$ . This choice ensures that  $f, f^{(1)}$ , and  $f^{(2)}$  are all  $\beta$ -smooth, as desired. Also notice that  $2a^2 \cdot \left(\frac{19\rho}{6}\right)^2 \leq 2 \cdot \left(\frac{19}{5\cdot 6}\right)^2\sigma^2 < \sigma^2$ , so the function satisfies Assumption 2.5.
For the remaining of the proof, we investigate the virtual gradient maps  $G_{f^{(1)}}$  and  $G_{f^{(2)}}$  within the specified region of interest:  $c - \rho \leq x \leq c + \rho$ . We will then demonstrate that if we initialize  $m$ -SAM in this interval, all subsequent iterations of  $m$ -SAM will remain inside  $[c - \rho, c + \rho]$ , as required above.
Within this interval, the perturbed iterate  $y = \rho \frac{\nabla f^{(1)}(x)}{\|\nabla f^{(1)}(x)\|}$  and the virtual gradient map  $G_{f^{(1)}}$  of  $f^{(1)}$  can be described as follows.
$$
y = \left\{ \begin{array}{l l} x + \rho , & x <   c \\ x, & x = c \\ x - \rho , & x > c, \end{array} \right. G _ {f ^ {(1)}} (x) = \left\{ \begin{array}{l l} - a (x + \rho - c), & x <   c \\ 0, & x = c \\ - a (x - \rho - c), & x > c. \end{array} \right.
$$
Additionally defining  $c' \triangleq \frac{p}{1 + p} c$ , we now calculate  $y = \rho \frac{\nabla f^{(2)}(x)}{\|\nabla f^{(2)}(x)\|}$  and the virtual gradient map  $G_{f^{(2)}}$  of  $f^{(2)}$ .
$$
y = \left\{ \begin{array}{l l} x - \rho , & x <   c ^ {\prime}, \\ x, & x = c ^ {\prime}, \\ x + \rho , & x > c ^ {\prime} \end{array} \right. G _ {f ^ {(2)}} (x) = \left\{ \begin{array}{l l} \frac {a}{1 - p} \left(x - \rho + p (x - \rho - c)\right), & x <   c ^ {\prime} \\ 0, & x = c ^ {\prime} \\ \frac {a}{1 - p} \left(x + \rho + p (x + \rho - c)\right), & x > c ^ {\prime} \end{array} \right.
$$
Here, given  $c = \left(1 + \frac{p}{4}\right)\rho$ , we can verify that
$$
0 <   c - \rho \leq c ^ {\prime} \leq c \leq c + \rho ,
$$
where  $0, c'$ , and  $c$  are the global minimum (or maximum) of  $f$ ,  $f^{(2)}$ , and  $f^{(1)}$ , respectively.
Recall that  $a \leq \frac{\beta}{5}$ . Since  $\eta \leq \frac{3}{10\beta}$ , we have  $\eta a \leq \frac{3}{50}$ , which will be useful for the rest of the proof. Now, we analyze  $x_{t+1}$ , the next iterate of  $m$ -SAM. For  $c - \rho \leq x_t \leq c + \rho$ , the next iterate  $x_{t+1}$  of  $m$ -SAM using  $G_{f^{(1)}}(x_t)$  is
$$
x _ {t + 1} = \left\{ \begin{array}{l l} x _ {t} + \eta a (x _ {t} + \rho - c), & x _ {t} <   c \\ x _ {t}, & x _ {t} = c \\ x _ {t} + \eta a (x _ {t} - \rho - c), & x _ {t} > c. \end{array} \right.
$$
We will show that  $x_{t + 1}$  must stay within the interval  $[c - \rho, c + \rho]$ . For the case  $x_t = c$ , the inclusion is trivial. To analyze the rest, we divide into two cases.
Case A-1:  $c - \rho \leq x_{t} < c$ . Since  $\eta a \leq \frac{3}{50}$ , we have
$$
\begin{array}{l} c - \rho \leq x _ {t} \leq x _ {t + 1} = x _ {t} + \eta a (x _ {t} + \rho - c) \leq c + \eta a (c + \rho - c) \\ = c + \eta a \rho \leq c + \frac {3 \rho}{5 0}. \\ \end{array}
$$
Case A-2:  $c < x_{t} \leq c + \rho$ . Since  $\eta a \leq \frac{3}{50}$ , we have
$$
\begin{array}{l} c + \rho \geq x _ {t} \geq x _ {t + 1} = x _ {t} + \eta a (x _ {t} - \rho - c) \geq c + \eta a (c - \rho - c) \\ = c - \eta a \rho \geq c - \frac {3 \rho}{5 0}. \\ \end{array}
$$