SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\min _ {t \in \{0, \dots , T \}} f (x _ {t}) - f ^ {*} = \Omega \left(\frac {\beta^ {3} \rho^ {2}}{\mu^ {2}}\right).
$$
Combining the three results, we can see that the suboptimality gap of the best iterate is at least  $\Omega\left(\frac{\beta^3\rho^2}{\mu^2T^2}\right)$ , hence proving the theorem. Below, we prove the statements for the three intervals.
Case 1:  $\eta \leq \frac{1}{2\mu T}$ . Consider a function
$$
f (x) = \frac {1}{2} \mu x ^ {2} - \mu \rho x, \quad \nabla f (x) = \mu (x - \rho).
$$
Suppose we start at initialization  $x_0 = \frac{2\beta^2\rho}{\mu^2} \geq 8\rho$ . Note from the definition of  $\nabla f$  that for any  $x_t \geq \rho$ , we have  $y_t = x_t + \rho \frac{\nabla f(x_t)}{|\nabla f(x_t)|} = x_t + \rho$  and  $\nabla f(y_t) = \mu (x_t + \rho - \rho) = \mu x_t$ . Therefore, the first SAM update can be written as
$$
x _ {1} = x _ {0} - \eta \nabla f (y _ {0}) = (1 - \eta \mu) x _ {0}.
$$
Since  $1 - \eta \mu \geq 1 - \frac{1}{2T} \geq \frac{1}{2}$ , the inequality  $x_{1} \geq \rho$  still holds and the same argument can be repeated for the second update, yielding
$$
x _ {2} = x _ {1} - \eta \nabla f (y _ {1}) = (1 - \eta \mu) x _ {1} = (1 - \eta \mu) ^ {2} x _ {0}.
$$
In fact,  $(1 - \eta \mu)^t \geq (1 - \frac{1}{2T})^T \geq \frac{1}{2}$  for all  $t \leq T$ , so all the iterates up to  $x_T$  stay above  $\rho$ . Thus, for any  $t = 0, \ldots, T$ , the iterate  $x_t$  can be written and bounded from below as
$$
x _ {t} = \left(1 - \eta \mu\right) ^ {t} x _ {0} \geq \frac {x _ {0}}{2} = \frac {\beta^ {2} \rho}{\mu^ {2}}.
$$
Therefore, in this case, the suboptimality gap of the best iterate is at least
$$
\min _ {t \in \{0, \dots , T \}} f (x _ {t}) - f ^ {*} \geq f \left(\frac {\beta^ {2} \rho}{\mu^ {2}}\right) - f (\rho) = \frac {\beta^ {4} \rho^ {2}}{2 \mu^ {3}} - \frac {\beta^ {2} \rho^ {2}}{\mu} + \frac {\mu \rho^ {2}}{2} = \Omega \left(\frac {\beta^ {4} \rho^ {2}}{\mu^ {3}}\right).
$$
Case 2:  $\frac{1}{2\mu T} \leq \eta \leq \frac{2}{\beta}$ . Consider a function
$$
f (x) = \frac {1}{4} \beta x ^ {2}, \quad \nabla f (x) = \frac {1}{2} \beta x.
$$
Suppose we start at initialization  $x_0 = \frac{\eta\beta\rho}{4 - \eta\beta}$ . We are going to show that the SAM iterates oscillate between  $\pm \frac{\eta\beta\rho}{4 - \eta\beta}$ . Indeed, if  $x_{t} = \frac{\eta\beta\rho}{4 - \eta\beta}$ , then  $\nabla f(x_{t}) > 0$  and
$$
y _ {t} = \frac {\eta \beta \rho}{4 - \eta \beta} + \rho = \frac {4 \rho}{4 - \eta \beta}, \quad \text {a n d} \quad \nabla f (y _ {t}) = \frac {2 \beta \rho}{4 - \eta \beta}.
$$
Then, after SAM update, we get
$$
x _ {t + 1} = x _ {t} - \eta \nabla f (y _ {t}) = \frac {\eta \beta \rho}{4 - \eta \beta} - \eta \frac {2 \beta \rho}{4 - \eta \beta} = - \frac {\eta \beta \rho}{4 - \eta \beta}.
$$
The same argument can be repeated to show that  $x_{t + 2} = -x_{t + 1} = x_t$ . As a result, the iterates oscillate between  $\pm \frac{\eta\beta\rho}{4 - \eta\beta}$  forever. In this case, the suboptimality gap is bounded from below by
$$
\min _ {t \in \{0, \dots , T \}} f (x _ {t}) - f ^ {*} \geq \frac {\beta}{4} \left(\frac {\eta \beta \rho}{4 - \eta \beta}\right) ^ {2} \geq \frac {\eta^ {2} \beta^ {3} \rho^ {2}}{6 4}.
$$
Applying  $\eta \geq \frac{1}{2\mu T}$  yields
$$
\min  _ {t \in \{0, \dots , T \}} f (x _ {t}) - f ^ {*} = \Omega \left(\frac {\beta^ {3} \rho^ {2}}{\mu^ {2} T ^ {2}}\right).
$$
Case 3:  $\eta \geq \frac{2}{\beta}$ . Consider a function
$$
f (x) = \frac {1}{2} \beta x ^ {2}, \quad \nabla f (x) = \beta x.
$$
Let the initialization be  $x_0 = \frac{\beta\rho}{\mu}$ . For any  $x_{t}\geq 0$ , we have  $y_{t} = x_{t} + \rho$  and  $\nabla f(y_{t}) = \beta (x_{t} + \rho)$ . Then, the resulting SAM update becomes
$$
x _ {t + 1} = x _ {t} - \eta \nabla f (y _ {t}) = (1 - \eta \beta) x _ {t} - \eta \rho \leq (1 - \eta \beta) x _ {t}.
$$
Since we have  $\eta \geq \frac{2}{\beta}$ , we have  $1 - \eta \beta \leq -1$  and  $x_{t + 1}$  has the opposite sign as  $x_{t}$  and its absolute value is at least as large as  $|x_{t}|$ . We can similarly check that any further SAM update changes the sign and does not decrease the absolute value. Therefore, for all  $t = 0,\dots ,T$ , we have  $|x_{t}|\geq |x_{0}|$ .
Consequently, the suboptimality gap of the best iterate is at least
$$
\min  _ {t \in \{0, \dots , T \}} f (x _ {t}) - f ^ {*} \geq f (x _ {0}) - f (0) = \Omega \left(\frac {\beta^ {3} \rho^ {2}}{\mu^ {2}}\right).
$$
Remarks on validity of lower bound. Lastly, we comment on why we choose different functions and initialization for different choices of  $\eta$  and why it suffices to provide a matching lower bound for Theorem 3.1. In convergence upper bounds in the form of Theorem 3.1, we aim to prove an upper bound on the following minimax risk:
$$
\inf  _ {A \in \mathcal {A}} \sup  _ {f \in \mathcal {F}} \zeta (A (f)), \tag {7}
$$
where  $\mathcal{A}$  denotes the class of algorithms,  $\mathcal{F}$  denotes the class of functions, and  $\zeta(A(f))$  is the suboptimality measure for algorithm's output  $A(f)$  for function  $f$ . In the context of Theorem 3.1, our algorithm class  $\mathcal{A}$  corresponds to the choices of the "hyperparameters"  $\eta$ ,  $\rho$ , and  $\pmb{x}_0$  of SAM, and the function class  $\mathcal{F}$  here is the class analyzed by the theorem: the collection of  $\beta$ -smooth and  $\mu$ -strongly convex functions. From this viewpoint, Theorem 3.1 can be thought of as an upper bound on (7), with the choice of  $\zeta(A(f)) \triangleq \min_{t \in \{0, \dots, T\}} f(x_t) - f^*$ .
Hence, showing a matching lower bound for Theorem 3.1 amounts to showing a matching lower bound for the minimax risk (7). For this purpose, it suffices to show that for each  $A \in \mathcal{A}$ , there exists a choice of  $f \in \mathcal{F}$  such that a certain lower bound holds. Therefore, we are allowed to choose different choices of  $f$  for each different choice of hyperparameters  $\eta$ ,  $\rho$ , and  $\pmb{x}_0$ . In fact, in our proof, we choose different choices of  $f$  and  $\pmb{x}_0$  for each different choice of  $\eta$ ; however, this is without loss of generality once we notice here that starting an algorithm at  $\pmb{x}_0$  to minimize  $f(\pmb{x})$  is equivalent to starting at  $\mathbf{0}$  to minimize  $f(\pmb{x} - \pmb{x}_0)$ . Hence, even though we choose different  $f$ 's and  $\pmb{x}_0$ 's for different choices of  $\eta$  in our proof of Theorem 3.2, this is sufficient for providing a matching upper bound for Theorem 3.1.
Admittedly, some authors show stronger versions than what we show, where a single function takes care of all possible hyperparameters. Indeed, such results provide lower bounds for  $\sup_{f\in \mathcal{F}}\inf_{A\in \mathcal{A}}\zeta (A(f))$ . Recalling that  $\sup \inf \leq \inf \sup$ , one can notice that these sup inf lower bounds are in fact much stronger than what suffices.
B.4 Convergence Proof for Smooth and Convex Functions (Proof of Theorem 3.3)
For smooth and convex function, we prove the convergence of gradient norm to zero. The theorem is restated for convenience.
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla f (\pmb {x} _ {t}) \| ^ {2} = \mathcal {O} \bigg (\frac {\beta \Delta}{T} + \frac {\sqrt {\Delta \beta^ {3} \rho^ {2}}}{\sqrt {T}} \bigg).
$$
Proof. Using Lemma B.3, for smooth and convex function  $f$ , we have
$$
f (\boldsymbol {x} _ {t + 1}) \leq f (\boldsymbol {x} _ {t}) - \frac {\eta}{2} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} + \frac {\eta^ {2} \beta^ {3} \rho^ {2}}{2},
$$
$$
\left\| \nabla f (\boldsymbol {x} _ {t}) \right\| ^ {2} \leq \frac {2}{\eta} (f (\boldsymbol {x} _ {t}) - f (\boldsymbol {x} _ {t + 1})) + \eta \beta^ {3} \rho^ {2}.
$$
Adding up the inequality for  $t = 0, \dots, T - 1$ , and the dividing both sides by  $T$ , we get
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} \leq \frac {2}{\eta T} \left(f \left(\boldsymbol {x} _ {0}\right) - f \left(\boldsymbol {x} _ {T}\right)\right) + \eta \beta^ {3} \rho^ {2} \leq \frac {2}{\eta T} \Delta + \eta \beta^ {3} \rho^ {2}. \tag {8}
$$
We now spell out how the choice of step size  $\eta = \min \left\{\sqrt{\frac{2\Delta}{\beta^3\rho^2T}},\frac{1}{2\beta}\right\}$  results in the convergence rate in the theorem statement. There are two cases to be considered, depending on the outcome of the min operation.
Case A:  $\sqrt{\frac{2\Delta}{\beta^3\rho^2T}} \leq \frac{1}{2\beta}$ . Putting  $\eta = \sqrt{\frac{2\Delta}{\beta^3\rho^2T}}$  into (8), we get
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla f (\pmb {x} _ {t}) \| ^ {2} \leq \frac {\sqrt {2 \Delta \beta^ {3} \rho^ {2}}}{\sqrt {T}} + \frac {\sqrt {2 \Delta \beta^ {3} \rho^ {2}}}{\sqrt {T}} = \frac {2 \sqrt {2 \Delta \beta^ {3} \rho^ {2}}}{\sqrt {T}}.
$$
Case B:  $\sqrt{\frac{2\Delta}{\beta^3\rho^2T}} \geq \frac{1}{2\beta}$ . Substituting  $\eta = \frac{1}{2\beta} \leq \sqrt{\frac{2\Delta}{\beta^3\rho^2T}}$  to (8) yields
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} \leq \frac {4 \beta \Delta}{T} + \beta^ {3} \rho^ {2} \cdot \frac {1}{2 \beta} \leq \frac {4 \beta \Delta}{T} + \frac {\sqrt {2 \Delta \beta^ {3} \rho^ {2}}}{\sqrt {T}}.
$$
Merging the two cases, we conclude
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} = \mathcal {O} \left(\frac {\beta \Delta}{T} + \frac {\sqrt {\Delta \beta^ {3} \rho^ {2}}}{\sqrt {T}}\right),
$$
B.5 Convergence Proof for Smooth and Nonconvex Functions (Proof of Theorem 3.4)
In this section, we prove the convergence up to an additive factor  $\beta^2\rho^2$  for smooth and nonconvex functions. The theorem is restated for convenience.
Theorem 3.4. Consider a  $\beta$ -smooth function  $f$  satisfying  $f^{*} = \inf_{\boldsymbol{x}} f(\boldsymbol{x}) > -\infty$ . If we run deterministic SAM starting at  $\boldsymbol{x}_0$  with any perturbation size  $\rho > 0$  and step size  $\eta = \frac{1}{\beta}$  to minimize  $f$ ,
$$
\begin{array}{l} f (\boldsymbol {x} _ {t + 1}) \leq f (\boldsymbol {x} _ {t}) - \eta \langle \nabla f (\boldsymbol {x} _ {t}), \nabla f (\boldsymbol {y} _ {t}) \rangle + \frac {\eta^ {2} \beta}{2} \| \nabla f (\boldsymbol {y} _ {t}) \| ^ {2} \\ = f (\boldsymbol {x} _ {t}) - \frac {\eta}{2} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} - \frac {\eta}{2} \| \nabla f (\boldsymbol {y} _ {t}) \| ^ {2} + \frac {\eta}{2} \| \nabla f (\boldsymbol {x} _ {t}) - \nabla f (\boldsymbol {y} _ {t}) \| ^ {2} + \frac {\eta^ {2} \beta}{2} \| \nabla f (\boldsymbol {y} _ {t}) \| ^ {2} \\ \leq f (\boldsymbol {x} _ {t}) - \frac {\eta}{2} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} + \frac {\eta \beta^ {2}}{2} \| \boldsymbol {x} _ {t} - \boldsymbol {y} _ {t} \| ^ {2} \\ = f (\boldsymbol {x} _ {t}) - \frac {\eta}{2} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} + \frac {\eta \beta^ {2} \rho^ {2}}{2}. \\ \end{array}
$$
The inequality can be rearranged as
$$
\left\| \nabla f (\boldsymbol {x} _ {t}) \right\| ^ {2} \leq \frac {2}{\eta} (f (\boldsymbol {x} _ {t}) - f (\boldsymbol {x} _ {t + 1})) + \beta^ {2} \rho^ {2}.
$$
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} \leq \frac {2}{\eta T} (f (\boldsymbol {x} _ {0}) - f (\boldsymbol {x} _ {T})) + \beta^ {2} \rho^ {2} \leq \frac {2 \Delta}{\eta T} + \beta^ {2} \rho^ {2}. \tag {9}
$$
Substituting  $\eta = \frac{1}{\beta}$  to (9) yields
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} \leq \frac {2 \beta \Delta}{T} + \beta^ {2} \rho^ {2}.
$$
B.6 Non-convergence for a Smooth and Nonconvex Function (Proof of Theorem 3.5)
In this section, we spell out the proof of our counterexample that SAM with constant perturbation size  $\rho$  provably fails to converge to a stationary point. Here we restate the theorem.
Proof. For  $x \in \mathbb{R}$ , consider the following one-dimensional function: given  $\rho$  and  $\beta$ ,
$$
f (x) = \frac {9 \beta \rho^ {2}}{2 5 \pi^ {2}} \sin \left(\frac {5 \pi}{3 \rho} x\right).
$$
It is easy to check that this function is  $\beta$ -smooth and  $\frac{3\beta\rho}{5\pi}$ -Lipschitz continuous. Also, let
$$
\mathbb {X} = \{x \mid x = (0. 3 + 0. 6 k) \rho , k \in \mathbb {Z} \},
$$
which is the set of points  $x$  where  $\nabla f(x) = 0$ . For SAM with perturbation size  $\rho$ , given the current iterate  $x_{t}$ , the corresponding  $y_{t} = x_{t} + \rho \frac{\nabla f(x_{t})}{|\nabla f(x_{t})|}$  is given by
$$
y _ {t} = x _ {t} + \left\{ \begin{array}{l l} \rho & \text {i f} (- 0. 3 + 1. 2 k) \rho <   x _ {t} <   (0. 3 + 1. 2 k) \rho \text {f o r s o m e} k \in \mathbb {Z}, \\ - \rho & \text {i f} (0. 3 + 1. 2 k) \rho <   x _ {t} <   (0. 9 + 1. 2 k) \rho \text {f o r s o m e} k \in \mathbb {Z}, \\ 0 & \text {i f} x _ {t} \in \mathbb {X}. \end{array} \right.
$$
This leads to the following a virtual gradient map  $G_{f}$ :
$$
G _ {f} (x _ {t}) = \nabla f (y _ {t}) = \left\{ \begin{array}{l l} \frac {3 \beta \rho}{5 \pi} \cos \left(\frac {5 \pi}{3 \rho} x _ {t} + \frac {5 \pi}{3}\right) & \text {i f} (- 0. 3 + 1. 2 k) \rho <   x _ {t} <   (0. 3 + 1. 2 k) \rho \text {f o r s o m e} k \in \mathbb {Z}, \\ \frac {3 \beta \rho}{5 \pi} \cos \left(\frac {5 \pi}{3 \rho} x _ {t} - \frac {5 \pi}{3}\right) & \text {i f} (0. 3 + 1. 2 k) \rho <   x _ {t} <   (0. 9 + 1. 2 k) \rho \text {f o r s o m e} k \in \mathbb {Z}, \\ 0 & \text {i f} x _ {t} \in \mathbb {X}. \end{array} \right.
$$
From this virtual gradient map, we can define
$$
\mathbb {Y} = \{x \mid x = (0. 7 + 1. 2 k) \rho , k \in \mathbb {Z} \} \cup \{x \mid x = (- 0. 1 + 1. 2 k) \rho , k \in \mathbb {Z} \}
$$
and  $\mathbb{Y}$  is the set of all points  $x$  where  $G_{f}(x) = 0$  and  $\nabla f(x)\neq 0$
Since  $f$  is a one-dimensional function, a virtual loss  $J_{f}$  can be obtained by integrating  $G_{f}$ . One possible example is
$$
J _ {f} (x) = \left\{ \begin{array}{l l} \frac {9 \beta \rho^ {2}}{2 5 \pi^ {2}} \sin \left(\frac {5 \pi}{3 \rho} x + \frac {5 \pi}{3}\right) & \text {i f} (- 0. 3 + 1. 2 k) \rho \leq x \leq (0. 3 + 1. 2 k) \rho \text {f o r s o m e} k \in \mathbb {Z}, \\ \frac {9 \beta \rho^ {2}}{2 5 \pi^ {2}} \sin \left(\frac {5 \pi}{3 \rho} x - \frac {5 \pi}{3}\right) & \text {i f} (0. 3 + 1. 2 k) \rho \leq x \leq (0. 9 + 1. 2 k) \rho \text {f o r s o m e} k \in \mathbb {Z}, \end{array} \right.
$$
which is a piecewise  $\beta$ -smooth function that is minimized at points in  $\mathbb{Y}$  and locally maximized at (non-differentiable) points in  $\mathbb{X}$ . Since  $J_{f}$  is well-defined, we can view SAM as GD on  $J_{f}$ .
For sufficiently small  $\eta \leq \frac{1}{\beta}$  and any initialization  $x_0 \in \mathbb{R} \setminus \mathbb{X}$ , the initialization belongs to one of the intervals  $((-0.3 + 0.6k)\rho, (0.3 + 0.6k)\rho)$ . For such small enough  $\eta$ , we can guarantee that the SAM iterates  $x_t$  will stay in the interval  $((-0.3 + 0.6k)\rho, (0.3 + 0.6k)\rho)$  for all  $t \geq 0$ . For example, suppose that  $x_t \in (-0.3\rho, 0.3\rho)$ . Then, the next iterate will stay in the same interval if and only if the image of the interval  $(-0.3\rho, 0.3\rho)$  under a map  $x \mapsto x - \frac{3\eta\beta\rho}{5\pi}\cos\left(\frac{5\pi}{3\rho}x + \frac{5\pi}{3}\right)$  is a subset of  $(-0.3\rho, 0.3\rho)$ .
$$
\begin{array}{l} \left\{x - \frac {3 \eta \beta \rho}{5 \pi} \cos \left(\frac {5 \pi}{3 \rho} x + \frac {5 \pi}{3}\right) \mid x \in (- 0. 3 \rho , 0. 3 \rho) \right\} \subset (- 0. 3 \rho , 0. 3 \rho) \\ \Longleftrightarrow \left\{z - \eta \beta \cos \left(z + \frac {5 \pi}{3}\right) \mid z \in \left(- \frac {\pi}{2}, \frac {\pi}{2}\right) \right\} \subset \left(- \frac {\pi}{2}, \frac {\pi}{2}\right), \\ \end{array}
$$
and the containment is true if  $\eta \beta \leq 1$ . By symmetry, the same argument can be applied to any intervals of the form  $((-0.3 + 0.6k)\rho, (0.3 + 0.6k)\rho)$ .
Thus, for any initialization  $x_0 \in \mathbb{R} \setminus \mathbb{X}$ , all SAM iterates stay inside the interval which  $x_0$  belongs to. Inside the interval, by  $\beta$ -smoothness of  $J_f$ , the following descent lemma always holds:
$$
\begin{array}{l} J _ {f} (x _ {t + 1}) \leq J _ {f} (x _ {t}) + \left\langle \nabla J _ {f} (x _ {t}), x _ {t + 1} - x _ {t} \right\rangle + \frac {\beta}{2} \| x _ {t + 1} - x _ {t} \| ^ {2} \\ = J _ {f} (x _ {t}) - \eta \left(1 - \frac {\eta \beta}{2}\right) \| G _ {f} (x _ {t}) \| ^ {2} \\ \leq J _ {f} (x _ {t}) - \frac {\eta}{2} \| G _ {f} (x _ {t}) \| ^ {2}. \\ \end{array}
$$
Therefore, if we add the inequalities up for  $t = 0,\dots ,T - 1$ , we get
$$
\sum_ {t = 0} ^ {T - 1} \| G _ {f} (x _ {t}) \| ^ {2} \leq \frac {2}{\eta} \left(J _ {f} (x _ {0}) - J _ {f} (x _ {T})\right) \leq \frac {2}{\eta} \left(J _ {f} (x _ {0}) - J _ {f} ^ {*}\right) <   \infty ,
$$
where  $J_{f}^{*}\triangleq \inf_{x}J_{f}(x) > -\infty$ . Since the inequality holds for all  $T$ , the series is summable, which in turn implies that  $\| G_{f}(x_{t})\| \to 0$  as  $t\to \infty$ . However, the only point in the interval  $((-0.3 + 0.6k)\rho ,(0.3 + 0.6k)\rho)$  satisfying  $\| G_{f}(x)\| = 0$  is the one in  $\mathbb{Y}\cap ((-0.3 + 0.6k)\rho ,(0.3 + 0.6k)\rho)$ . Hence, if  $x_0\in \mathbb{R}\setminus \mathbb{X}$  and  $\eta \leq \frac{1}{\beta}$ , SAM must converge to a point in  $\mathbb{Y}$ . Since  $x_0$  is drawn from a continuous probability distribution,  $x_0\in \mathbb{R}\setminus \mathbb{X}$  holds almost surely. This finishes the proof.
B.7 Non-convergence for a Nonsmooth and Convex Functions (Proof of Theorem 3.6)
For nonsmooth convex function, depending on the initialization, SAM can converge to a suboptimal point with distance  $\Omega (\rho)$  from the global minimum. The theorem is restated for convenience.
(b)
Figure 3: (a) The demonstration of four separate regions, and basis vectors of example function. (b) The trajectory of SAM for example function. As SAM enters region  $\mathbb{A}$ , the update cannot get any closer to the global minima.
Proof. For  $\pmb{x} = (x^{(1)}, x^{(2)}) \in \mathbb{R}^2$ , consider a 2-dimensional function,
$$
f (\boldsymbol {x}) = \max  \left\{\left| x ^ {(1)} \right|, \left| 2 x ^ {(1)} + x ^ {(2)} \right| \right\}.
$$
A set  $\{\pmb{v}_1, \pmb{v}_2\}$  can be established as a basis for  $\mathbb{R}^2$ , with  $\pmb{v}_1 = \pmb{e}_1$ , and  $\pmb{v}_2 = \frac{2}{\sqrt{5}} \pmb{e}_1 + \frac{1}{\sqrt{5}} \pmb{e}_2$ . Any vector  $\pmb{x} \in \mathbb{R}^2$  can be uniquely expressed by  $\pmb{v}_1$  and  $\pmb{v}_2$ . Consider the region:  $\{\pmb{x} = b^{(1)} \pmb{v}_1 + b^{(2)} \pmb{v}_2 \mid b^{(1)} < 0\}$ .
This area can be partitioned into four separate regions.
$$
\left\{ \begin{array}{r l} & {\mathbb {A}: \left\{\boldsymbol {x} = b ^ {(1)} \boldsymbol {v} _ {1} + b ^ {(2)} \boldsymbol {v} _ {2} \mid - \frac {7 \rho}{2} <   b ^ {(1)} <   0 \right\},} \\ & {\mathbb {B}: \left\{\boldsymbol {x} = b ^ {(1)} \boldsymbol {v} _ {1} + b ^ {(2)} \boldsymbol {v} _ {2} \mid b ^ {(1)} <   - \frac {7 \rho}{2} \right\} \cap \left\{\boldsymbol {x} = b ^ {(1)} \boldsymbol {v} _ {1} + b ^ {(2)} \boldsymbol {v} _ {2} \mid b ^ {(1)} + \sqrt {5} b ^ {(2)} > 0 \right\},} \\ & {\mathbb {C}: \left\{\boldsymbol {x} = b ^ {(1)} \boldsymbol {v} _ {1} + b ^ {(2)} \boldsymbol {v} _ {2} \mid b ^ {(1)} + \sqrt {5} b ^ {(2)} <   0 \right\} \cap \left\{\boldsymbol {x} = b ^ {(1)} \boldsymbol {v} _ {1} + b ^ {(2)} \boldsymbol {v} _ {2} \mid - b ^ {(1)} + \sqrt {5} b ^ {(2)} > 0 \right\}} \\ & {\qquad \cap \left\{\boldsymbol {x} = b ^ {(1)} \boldsymbol {v} _ {1} + b ^ {(2)} \boldsymbol {v} _ {2} \mid - 2 b ^ {(1)} + \sqrt {5} b ^ {(2)} > \frac {3}{2} \rho \right\},} \\ & {\mathbb {D}: \left\{\boldsymbol {x} = b ^ {(1)} \boldsymbol {v} _ {1} + b ^ {(2)} \boldsymbol {v} _ {2} \mid b ^ {(1)} <   0 \right\} - (\mathbb {A} \cup \mathbb {B} \cup \mathbb {C}).} \end{array} \right.
$$
Figure 3(a) demonstrates the regions  $\mathbb{A},\mathbb{B},\mathbb{C},\mathbb{D}$ , as well as the vectors  $\pmb{v}_1$  and  $\pmb{v}_2$ . When  $\pmb{x}$  belongs to the set  $\pmb{x} = b^{(1)}\pmb{v}_1 + b^{(2)}\pmb{v}_2 \mid b^{(1)} < 0$ , we can examine four different scenarios.
Case A:  $\mathbf{x} = b^{(1)}\mathbf{v}_1 + b^{(2)}\mathbf{v}_2 \in \mathbb{A}$ .  $\mathbf{x} \in \mathbb{A}$ , so  $\mathbf{y} = \mathbf{x} + \rho \frac{\nabla f(\mathbf{x})}{\|\nabla f(\mathbf{x})\|}$  and  $-\nabla f(\mathbf{y})$  are as follows.
$$
\boldsymbol {y} = \left\{ \begin{array}{l l} \boldsymbol {x} + \rho \boldsymbol {v} _ {2}, & - b ^ {(1)} <   \sqrt {5} b ^ {(2)} \\ \boldsymbol {x} - \rho \boldsymbol {v} _ {1}, & b ^ {(1)} <   \sqrt {5} b ^ {(2)} <   - b ^ {(1)} \\ \boldsymbol {x} - \rho \boldsymbol {v} _ {2}, & \sqrt {5} b ^ {(2)} <   b ^ {(1)}. \end{array} \right.
$$
$$
- \nabla f (\boldsymbol {y}) = \left\{ \begin{array}{l l} - \sqrt {5} \boldsymbol {v} _ {2}, & - b ^ {(1)} <   \sqrt {5} b ^ {(2)} \\ \sqrt {5} \boldsymbol {v} _ {2}, & b ^ {(1)} <   \sqrt {5} b ^ {(2)} <   - b ^ {(1)} \\ \sqrt {5} \boldsymbol {v} _ {2}, & \sqrt {5} b ^ {(2)} <   b ^ {(1)}. \end{array} \right.
$$
As a result, the SAM updates  $\nabla f(\pmb{y})$  only affect  $\pmb{v}_2$ , and does not affect on  $\pmb{v}_1$ . Thus, if  $\pmb{x}_t \in \mathbb{A}$ , the next SAM iterate  $\pmb{x}_{t+1}$  remains in  $\mathbb{A}$ .
Case B:  $\pmb{x} = b^{(1)}\pmb{v}_1 + b^{(2)}\pmb{v}_2 \in \mathbb{B}$ . We can verify that  $\pmb{y} = \pmb{x} + \rho \frac{\nabla f(\pmb{x})}{\|\nabla f(\pmb{x})\|} = \pmb{x} + \rho \pmb{v}_2$ . Therefore,  $-\nabla f(\pmb{y}) = -\sqrt{5}\pmb{v}_2$ . Therefore, the SAM update shifts in the direction of  $-\pmb{v}_2$ . Hence, if  $\pmb{x}_t \in \mathbb{B}$ , the next SAM iterate will be  $\pmb{x}_{t+1} \in (\mathbb{B} \cup \mathbb{C} \cup \mathbb{D})$ .
Case C:  $\pmb{x} = b^{(1)}\pmb{v}_1 + b^{(2)}\pmb{v}_2 \in \mathbb{C}$ .  $\pmb{y} = \pmb{x} + \rho \frac{\nabla f(\pmb{x})}{\|\nabla f(\pmb{x})\|} = \pmb{x} - \rho \pmb{v}_1$ , so
$$
- \nabla f (\boldsymbol {y} _ {t}) = \left\{ \begin{array}{l l} \sqrt {5} \boldsymbol {v} _ {2}, & b ^ {(1)} <   \sqrt {5} b ^ {(2)} <   b ^ {(1)} + \rho \\ \boldsymbol {v} _ {1}, & b ^ {(1)} + \rho <   \sqrt {5} b ^ {(2)} <   - b ^ {(1)}. \end{array} \right.
$$
For  $\pmb{x}_t = b_t^{(1)}\pmb{v}_1 + b_t^{(2)}\pmb{v}_2 \in \mathbb{C}$ , if  $b_t^{(1)} < \sqrt{5}b_t^{(2)} < b_t^{(1)} + \rho$ , then SAM update shifts in the direction of  $\pmb{v}_2$ , resulting in  $\pmb{x}_{t+1} \in (\mathbb{B} \cup \mathbb{C} \cup \mathbb{D})$ . Otherwise, the next SAM iterate  $\pmb{x}_{t+1}$  shifts in the direction of  $+\pmb{v}_1$ . Consequently,  $\pmb{x}_{t+1}$  can either remain in  $\mathbb{B} \cup \mathbb{C} \cup \mathbb{D}$ , or move to  $\mathbb{A}$ .
Assume that  $\pmb{x}_{t+1}$  moves to  $\mathbb{A}$ . Given that  $b_{t}^{(1)} < -\frac{7\rho}{2}$  for  $\pmb{x}_{t} \in \mathbb{C}$ , we can verify that  $b_{t+1}^{(1)} < -\frac{7\rho}{2} + \eta$ . Based on our previous observation in Case A where SAM updates only affect  $\pmb{v}_{2}$  when  $\pmb{x} \in \mathbb{A}$ , we can conclude that for all subsequent iterates  $\pmb{x}_{i}$  with  $i > t+1$ , the condition  $b_{i}^{(1)} < -\frac{7\rho}{2} + \eta$  continues to hold.
Case D:  $\pmb{x} = b^{(1)}\pmb{v}_1 + b^{(2)}\pmb{v}_2 \in \mathbb{D}$ .  $\pmb{y} = \pmb{x} + \rho \frac{\nabla f(\pmb{x})}{\|\nabla f(\pmb{x})\|}$  becomes
$$
\boldsymbol {y} = \left\{ \begin{array}{l l} \boldsymbol {x} - \rho \boldsymbol {v} _ {2}, & \sqrt {5} b ^ {(2)} <   b ^ {(1)} \\ \boldsymbol {x} - \rho \boldsymbol {v} _ {1}, & \text {o t h e r w i s e .} \end{array} \right.
$$
We can check that  $-\nabla f(\pmb{y}) = \sqrt{5}\pmb{v}_2$  for all cases. So the SAM update shifts in the direction of  $+v_{2}$ . As a result, if  $\pmb{x}_{t}\in \mathbb{D}$ , the next SAM iterate  $\pmb{x}_{t + 1}$  will fall into  $\pmb{x}_{t + 1}\in (\mathbb{B}\cup \mathbb{C}\cup \mathbb{D})$ .