SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} f (\pmb {x} _ {t + 1}) \leq \mathbb {E} f (\pmb {x} _ {t}) - \eta \mathbb {E} \langle \nabla f (\pmb {x} _ {t}), \tilde {g} (\pmb {y} _ {t}) \rangle + \frac {\beta \eta^ {2}}{2} \mathbb {E} \| \tilde {g} (\pmb {y} _ {t}) \| ^ {2} \\ \leq \mathbb {E} f (\boldsymbol {x} _ {t}) - \eta \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} - \eta \mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \tilde {g} (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \rangle + \frac {\beta \eta^ {2}}{2} \mathbb {E} \| \tilde {g} (\boldsymbol {y} _ {t}) \| ^ {2} \\ = \mathbb {E} f (\boldsymbol {x} _ {t}) - \eta \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} - \eta \mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \tilde {g} (\boldsymbol {y} _ {t}) - \tilde {g} (\hat {\boldsymbol {y}} _ {t}) \rangle - \eta \mathbb {E} \langle \nabla f (\boldsymbol {x} _ {t}), \nabla f (\hat {\boldsymbol {y}} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \rangle \\ + \frac {\beta \eta^ {2}}{2} \mathbb {E} \| \tilde {g} (\boldsymbol {y} _ {t}) \| ^ {2} \\ \leq \mathbb {E} f (\boldsymbol {x} _ {t}) - \eta \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} + \frac {\eta}{2} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} + \frac {\eta}{2} \mathbb {E} \| \tilde {g} (\boldsymbol {y} _ {t}) - \tilde {g} (\hat {\boldsymbol {y}} _ {t}) \| ^ {2} \\ - \eta \mathbb {E} \left\langle \frac {\| \nabla f (\boldsymbol {x} _ {t}) \|}{\rho} \cdot \left(\hat {\boldsymbol {y}} _ {t} - \boldsymbol {x} _ {t}\right), \nabla f (\hat {\boldsymbol {y}} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \right\rangle + \frac {\beta \eta^ {2}}{2} \mathbb {E} \| \tilde {g} (\boldsymbol {y} _ {t}) \| ^ {2} \\ \leq \mathbb {E} f (\boldsymbol {x} _ {t}) - \frac {\eta}{2} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} + \frac {\beta^ {2} \eta}{2} \mathbb {E} \| \boldsymbol {y} _ {t} - \hat {\boldsymbol {y}} _ {t} \| ^ {2} + \eta \mathbb {E} \left[ \frac {\beta \| \nabla f (\boldsymbol {x} _ {t}) \|}{\rho} \cdot \| \hat {\boldsymbol {y}} _ {t} - \boldsymbol {x} _ {t} \| ^ {2} \right] + \frac {\beta \eta^ {2}}{2} \mathbb {E} \| \tilde {g} (\boldsymbol {y} _ {t}) \| ^ {2} \\ \leq \mathbb {E} f (\pmb {x} _ {t}) - \frac {\eta}{2} \mathbb {E} \| \nabla f (\pmb {x} _ {t}) \| ^ {2} + \frac {\beta^ {2} \rho^ {2} \eta}{2} \mathbb {E} \left\| \frac {\tilde {g} (\pmb {x} _ {t})}{\| \tilde {g} (\pmb {x} _ {t}) \|} - \frac {\nabla f (\pmb {x} _ {t})}{\| \nabla f (\pmb {x} _ {t}) \|} \right\| ^ {2} + \beta \rho \eta \mathbb {E} \| \nabla f (\pmb {x} _ {t}) \| \\ \end{array}
$$
$$
\begin{array}{l} + \frac {\beta \eta^ {2}}{2} \mathbb {E} \| \tilde {g} (\boldsymbol {y} _ {t}) \| ^ {2} \\ \leq \mathbb {E} f (\boldsymbol {x} _ {t}) - \frac {\eta}{2} \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} + 2 \beta^ {2} \rho^ {2} \eta + \beta \rho \eta \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| + \frac {\beta \eta^ {2}}{2} \mathbb {E} \| \tilde {g} (\boldsymbol {y} _ {t}) \| ^ {2} \\ \leq \mathbb {E} f (\boldsymbol {x} _ {t}) - \frac {\eta}{2} \mathbb {E} (\| \nabla f (\boldsymbol {x} _ {t}) \| - \beta \rho) ^ {2} + \frac {5}{2} \beta^ {2} \rho^ {2} \eta + \beta \eta^ {2} (\mathbb {E} \| \nabla f (\boldsymbol {y} _ {t}) \| ^ {2} + \mathbb {E} \| \tilde {g} (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {y} _ {t}) \| ^ {2}) \\ \leq \mathbb {E} f (\boldsymbol {x} _ {t}) - \frac {\eta}{2} \mathbb {E} (\| \nabla f (\boldsymbol {x} _ {t}) \| - \beta \rho) ^ {2} + \frac {5}{2} \beta^ {2} \rho^ {2} \eta + \beta \eta^ {2} (\sigma^ {2} + L ^ {2}). \\ \end{array}
$$
$$
\mathbb {E} \left[ \left(\| \nabla f (\boldsymbol {x} _ {t}) \| - \beta \rho\right) ^ {2} \right] \leq \frac {2}{\eta} \left(\mathbb {E} f (\boldsymbol {x} _ {t}) - \mathbb {E} f (\boldsymbol {x} _ {t + 1})\right) + 5 \beta^ {2} \rho^ {2} + 2 \beta \eta \left(\sigma^ {2} + L ^ {2}\right).
$$
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \left(\| \nabla f (\boldsymbol {x} _ {t}) \| - \beta \rho\right) ^ {2} \right] \leq \frac {2}{\eta T} \left(\mathbb {E} f (\boldsymbol {x} _ {0}) - \mathbb {E} f (\boldsymbol {x} _ {T})\right) + 5 \beta^ {2} \rho^ {2} + 2 \beta \eta \left(\sigma^ {2} + L ^ {2}\right) \\ \leq \frac {2 \Delta}{\eta T} + 5 \beta^ {2} \rho^ {2} + 2 \beta \eta \left(\sigma^ {2} + L ^ {2}\right). \tag {13} \\ \end{array}
$$
Substituting  $\eta = \frac{\sqrt{\Delta}}{\sqrt{\beta(\sigma^2 + L^2)T}}$  to (13) yields
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \left(\| \nabla f (\boldsymbol {x} _ {t}) \| - \beta \rho\right) ^ {2} \right] \leq \frac {4 \sqrt {\beta \Delta \left(\sigma^ {2} + L ^ {2}\right)}}{\sqrt {T}} + 5 \beta^ {2} \rho^ {2},
$$
From the result of Theorem 4.6, Corollary 4.7 can be derived. The corollary is restated for the ease of reference.
Proof. Starting from the result of Theorem 4.6, we have
$$
\begin{array}{l} \min  _ {t \in \{0, \dots , T \}} \left\{\left(\mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| - \beta \rho\right) ^ {2} \right\} \leq \min  _ {t \in \{0, \dots , T \}} \mathbb {E} \left[ \left(\| \nabla f (\boldsymbol {x} _ {t}) \| - \beta \rho\right) ^ {2} \right] \\ \leq \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \left(\| \nabla f (\boldsymbol {x} _ {t}) \| - \beta \rho\right) ^ {2} \right] \\ \leq \mathcal {O} \left(\frac {\sqrt {\beta \Delta (\sigma^ {2} + L ^ {2})}}{\sqrt {T}}\right) + 5 \beta^ {2} \rho^ {2}. \\ \end{array}
$$
Taking the square root on both sides,
$$
\min  _ {t \in \{0, \dots , T \}} \left\{\left| \mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| - \beta \rho \right| \right\} \leq \mathcal {O} \left(\frac {\left(\beta \Delta \left(\sigma^ {2} + L ^ {2}\right)\right) ^ {1 / 4}}{T ^ {1 / 4}}\right) + \sqrt {5} \beta \rho .
$$
$$
\min  _ {t \in \{0, \dots , T \}} \left\{\mathbb {E} \| \nabla f (\boldsymbol {x} _ {t}) \| \right\} \leq \mathcal {O} \left(\frac {\left(\beta \Delta \left(\sigma^ {2} + L ^ {2}\right)\right) ^ {1 / 4}}{T ^ {1 / 4}}\right) + \left(1 + \sqrt {5}\right) \beta \rho ,
$$
(f)
Figure 6: The results of the SAM simulations on the example functions. The yellow line indicates the trajectory of SAM iterates. (a) and (d) display deterministic SAM iterates (with initialization  $x_0 = 0.4$ ) and the plot of x-coordinate values over epochs, for a smooth nonconvex function as shown in Figure 2(a) under settings in Theorem 3.5. (b) and (e) show m-SAM iterates (with initialization  $x_0 = 4$ ) and the plot of x-coordinate values over epochs, for the smooth strongly convex function as shown in Figure 2(b) under settings in Theorem 4.2. (c) and (f) demonstrate m-SAM iterates (with initialization  $x_0 = 9$ ) and the plot of x-coordinate values over epochs, for the smooth convex function as shown in Figure 2(d) under settings in Theorem 4.4. All plots empirically verify that practical SAM cannot converge all the way to optima. Instead, the iterates get trapped in certain regions.
D Simulation Results of SAM on Example Functions
Figure 6 shows the results of SAM simulations on the example functions considered in Sections 3 and 4. It demonstrates that SAM indeed does not converge in our worst-case example constructions.