SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] A. Agarwala and Y. N. Dauphin. Sam operates far from home: eigenvalue regularization as a dynamical phenomenon. arXiv preprint arXiv:2302.08692, 2023.
[2] M. Andriushchenko and N. Flammarion. Towards understanding sharpness-aware minimization. In International Conference on Machine Learning, pages 639-668. PMLR, 2022.
[3] D. Bahri, H. Mobahi, and Y. Tay. Sharpness-aware minimization improves language model generalization. arXiv preprint arXiv:2110.08529, 2021.
[4] P. L. Bartlett, P. M. Long, and O. Bousquet. The dynamics of sharpness-aware minimization: Bouncing across ravines and drifting towards wide minima. arXiv preprint arXiv:2210.01513, 2022.
[5] K. Behdin and R. Mazumder. Sharpness-aware minimization: An implicit regularization perspective. arXiv preprint arXiv:2302.11836, 2023.
[6] K. Behdin, Q. Song, A. Gupta, A. Acharya, D. Durfee, B. Ocejo, S. Keerthi, and R. Mazumdar. msam: Micro-batch-averaged sharpness-aware minimization. arXiv preprint arXiv:2302.09693, 2023.
[7] J. M. Borwein and A. S. Lewis. Convex analysis and nonlinear optimization: Theory and examples, 2010.
[8] S. Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning, 8(3-4):231-357, 2015.
[9] X. Chen, C.-J. Hsieh, and B. Gong. When vision transformers outperform resnets without pre-training or strong data augmentations. arXiv preprint arXiv:2106.01548, 2021.
[10] F. H. Clarke, Y. S. Ledyaev, R. J. Stern, and P. R. Wolenski. Nonsmooth analysis and control theory, volume 178. Springer Science & Business Media, 2008.
[11] J. M. Cohen, S. Kaur, Y. Li, J. Z. Kolter, and A. Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. arXiv preprint arXiv:2103.00065, 2021.
[12] E. M. Compagnoni, A. Orvieto, L. Biggio, H. Kersting, F. N. Proske, and A. Lucchi. An sde for modeling sam: Theory and insights. arXiv preprint arXiv:2301.08203, 2023.
[13] Y. Dai, K. Ahn, and S. Sra. The crucial role of normalization in sharpness-aware minimization. arXiv preprint arXiv:2305.15287, 2023.
[14] L. Dinh, R. Pascanu, S. Bengio, and Y. Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pages 1019-1028. PMLR, 2017.
[15] J. Du, H. Yan, J. Feng, J. T. Zhou, L. Zhen, R. S. M. Goh, and V. Y. Tan. Efficient sharpness-aware minimization for improved training of neural networks. arXiv preprint arXiv:2110.03141, 2021.
[16] P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.
[17] S. Hochreiter and J. Schmidhuber. Simplifying neural nets by discovering flat minima. Advances in neural information processing systems, 7, 1994.
[18] W. Jiang, H. Yang, Y. Zhang, and J. Kwok. An adaptive policy to employ sharpness-aware minimization. In The Eleventh International Conference on Learning Representations, 2022.
[19] Y. Jiang, B. Neyshabur, H. Mobahi, D. Krishnan, and S. Bengio. Fantastic generalization measures and where to find them. arXiv preprint arXiv:1912.02178, 2019.
[20] J. Kaddour, L. Liu, R. Silva, and M. Kusner. When do flat minima optimizers work? In Advances in Neural Information Processing Systems, 2022.
[21] N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.
[22] H. Kim, J. Park, Y. Choi, and J. Lee. Stability analysis of sharpness-aware minimization, 2023. URL https://arxiv.org/abs/2301.06308.
[23] H. Kim, J. Park, Y. Choi, W. Lee, and J. Lee. Exploring the effect of multi-step ascent in sharpness-aware minimization. arXiv preprint arXiv:2302.10181, 2023.
[24] J. Kwon, J. Kim, H. Park, and I. K. Choi. Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning, pages 5905-5914. PMLR, 2021.
[25] H. Lee, H. Cho, H. Kim, D. Gwak, J. Kim, J. Choo, S.-Y. Yun, and C. Yun. Enhancing generalization and plasticity for sample efficient reinforcement learning. arXiv preprint arXiv:2306.10711, 2023.
[26] Y. Liu, S. Mai, X. Chen, C.-J. Hsieh, and Y. You. Towards efficient and scalable sharpness-aware minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12360-12370, 2022.
[27] P. Mi, L. Shen, T. Ren, Y. Zhou, X. Sun, R. Ji, and D. Tao. Make sharpness-aware minimization stronger: A sparsified perturbation approach. arXiv preprint arXiv:2210.05177, 2022.
[28] T. Möllenhoff and M. E. Khan. Sam as an optimal relaxation of bayes. arXiv preprint arXiv:2210.01620, 2022.
[29] B. Neyshabur, S. Bhojanapalli, D. McAllester, and N. Srebro. Exploring generalization in deep learning. Advances in neural information processing systems, 30, 2017.
[30] Y. Shi, Y. Liu, K. Wei, L. Shen, X. Wang, and D. Tao. Make landscape flatter in differentially private federated learning. arXiv preprint arXiv:2303.11242, 2023.
[31] H. Sun, L. Shen, Q. Zhong, L. Ding, S. Chen, J. Sun, J. Li, G. Sun, and D. Tao. Adasam: Boosting sharpness-aware minimization with adaptive learning rate and momentum for training deep neural networks. arXiv preprint arXiv:2303.00565, 2023.
[32] P. Wang, Z. Zhang, Z. Lei, and L. Zhang. Sharpness-aware gradient matching for domain generalization. arXiv preprint arXiv:2303.10353, 2023.
[33] K. Wen, T. Ma, and Z. Li. How does sharpness-aware minimization minimize sharpness? arXiv preprint arXiv:2211.05729, 2022.
[34] X. Zhang, R. Xu, H. Yu, H. Zou, and P. Cui. Gradient norm aware minimization seeks first-order flatness and improves generalization. arXiv preprint arXiv:2303.03108, 2023.
[35] J. Zhuang, B. Gong, L. Yuan, Y. Cui, H. Adam, N. Dvornek, S. Tatikonda, J. Duncan, and T. Liu. Surrogate gap minimization improves sharpness-aware training. arXiv preprint arXiv:2203.08065, 2022.
1.1 Summary of Our Contributions 2
1.2 Related Works: Convergence Analysis of SAM 2
1.3 Notation 3
2 Sharpness-Aware Minimization: Preliminaries and Intuitions 3
2.1 Sharpness-Aware Minimization (SAM) 3
2.2 SAM under Stochastic Settings 4
2.3 Function Classes 4
2.4 SAM as GD on Virtual Loss 4
3 Convergence Analysis Under Deterministic Settings 5
3.1 Smooth and Strongly Convex Functions 5
3.2 Smooth and Convex Functions 6
3.3 Smooth and Nonconvex Functions 6
3.4 Nonsmooth Lipschitz Convex Functions 7
4 Convergence Analysis Under Stochastic Settings 7
4.1 Smooth and Strongly Convex Functions 7
4.2 Smooth and Convex Functions 9
4.3 Smooth and Nonconvex Functions 10
A More Details on the Related Works of SAM 15
B Proofs for (Non-)Convergence of full-batch SAM 16
B.1 Additional Function Class and Important Lemmas 16
B.2 Convergence Proof for Smooth and Strongly Convex Functions (Proof of Theorem 3.1) 17
B.3 Lower Bound Proof for Smooth and Strongly Convex Functions (Proof of Theorem 3.2) 18
B.4 Convergence Proof for Smooth and Convex Functions (Proof of Theorem 3.3) 21
B.5 Convergence Proof for Smooth and Nonconvex Functions (Proof of Theorem 3.4). 21
B.6 Non-convergence for a Smooth and Nonconvex Function (Proof of Theorem 3.5) 22
B.7 Non-convergence for a Nonsmooth and Convex Functions (Proof of Theorem 3.6). 23
C Proofs for (Non-)Convergence of Stochastic SAM 26
C.1 Important Lemmas Regarding Stochastic SAM 26
C.2 Convergence Proof for Smooth and Strongly Convex Functions (Proof of Theorem 4.1) 27
C.3 Non-Convergence of  $m$ -SAM for a Smooth and Strongly Convex Function (Proof of Theorem 4.2) 29
C.4 Convergence Proof for Smooth and Convex Functions (Proof of Theorem 4.3) 32
C.5 Non-Convergence of  $m$ -SAM for a Smooth and Convex Function (Proof of Theorem 4.4) 33
C.6 Convergence Proof for Smooth and Nonconvex Function under  $n$ -SAM (Proof of Theorem 4.5) 36
C.7 Convergence Proof for Smooth Lipschitz Nonconvex Function under  $m$ -SAM (Proof of Theorem 4.6 and Corollary 4.7) 37
D Simulation Results of SAM on Example Functions 39
A More Details on the Related Works of SAM
Sharpness and Generalization Hochreiter and Schmidhuber [17] propose an algorithm for finding low-complexity models with high generalization performance by finding flat minima. Keskar et al. [21] show that the poor generalization performance of large-batch training is due to the fact that large-batch training tends to converge to sharp minima. Jiang et al. [19] empirically show correlation between sharpness and generalization performance.
Motivated by prior studies, Foret et al. [16] propose Sharpness-Aware Minimization (SAM) that focuses on finding flat minima by aiming to (approximately) minimize  $f^{\mathrm{SAM}}(\pmb{x}) = \max_{\|\pmb{\epsilon}\| \leq \rho} f(\pmb{x} + \pmb{\epsilon})$  instead of  $f$ . Empirical results [3, 9, 16, 20, 25] show outstanding generalization performance of SAM on various tasks and models, including recent state-of-the-art models such as ViTs, MLP-Mixers, and T5.
Other Theoretical Properties of SAM. Bartlett et al. [4] prove that for quadratic functions, SAM dynamics oscillate and align to the eigenvector corresponding to the largest eigenvalue of Hessian, and show interpretation of SAM dynamics as GD on a "virtual loss". Wen et al. [33] prove that for sufficiently small perturbation size  $\rho$  of SAM, its trajectory follows a sharpness reduction flow which minimizes the maximum eigenvalue of Hessian. Dai et al. [13] examine the properties of SAM with and without normalization. They specifically focus on the stabilization and "drift-along-minima" effects observed in SAM with normalization, which are not present in SAM without normalization. Compagnoni et al. [12] derive a continuous time SDE model for SAM with and without normalization, and employ the model to explain why SAM has a preference for flat minima. Furthermore, they demonstrate that SAM with and without normalization exhibit distinct implicit regularization properties, especially for  $\rho = \mathcal{O}(\sqrt{\eta})$ .
Agarwala and Dauphin [1] prove that SAM produces an Edge Of Stability [11] stabilization effect, at a lower eigenvalue than gradient descent, under quadratic function settings. Kim et al. [22] show that saddle point acts as an attractor in SAM dynamics, and stochastic SAM takes more time to escape saddle point than stochastic gradient descent. Behdin and Mazumder [5] study the implicit regularization perspective of SAM, and provide a theoretical explanation of high generalization performance of SAM.
While Agarwala and Dauphin [1], Behdin and Mazumder [5], Kim et al. [22] provide useful insights on theoretical aspects of SAM, we point out that all proofs are restricted to SAM without gradient normalization in ascent steps. Considering the vastly different behaviors of SAM with and without normalization (Figure 1), it is not immediately clear whether these insights carry over to the practical version.
B Proofs for (Non-)Convergence of full-batch SAM
In this section, we show detailed proofs and explanations regarding convergence of SAM with constant  $\rho$ . The SAM algorithm we consider is defined as a two-step method:
$$
\left\{ \begin{array}{l} \boldsymbol {y} _ {t} = \boldsymbol {x} _ {t} + \rho \frac {\nabla f (\boldsymbol {x} _ {t})}{| | \nabla f (\boldsymbol {x} _ {t}) | |}, \\ \boldsymbol {x} _ {t + 1} = \boldsymbol {x} _ {t} - \eta \nabla f (\boldsymbol {y} _ {t}). \end{array} \right.
$$
B.1 Additional Function Class and Important Lemmas
In this section, we define an additional function class to use in convergence proofs. We also state and prove a few lemmas that are used in our theorem proofs.
Definition B.1 (Polyak-Lojasiewicz). A function  $f: \mathbb{R}^d \to \mathbb{R}$  satisfies  $\mu$ -PL condition if there exists  $\mu > 0$  such that,
$$
\frac {1}{2} \| \nabla f (\boldsymbol {x}) \| ^ {2} \geq \mu (f (\boldsymbol {x}) - f ^ {*}) \tag {4}
$$
for all  $\pmb {x}\in \mathbb{R}^d$
It is well-known that  $\mu$ -strongly convex functions are  $\mu$ -PL, but not vice versa.
Lemma B.2. For a differentiable and  $\mu$ -strongly convex function  $f$ , we have
$$
\langle \nabla f (\boldsymbol {x} _ {t}), \nabla f (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \rangle \geq \mu \rho \| \nabla f (\boldsymbol {x} _ {t}) \|.
$$
For a differentiable and convex function  $f$ , the inequality continues to hold with  $\mu = 0$ .
Proof. If a differentiable function  $f$  is  $\mu$ -strongly convex, then its gradient map  $\pmb{x} \mapsto \nabla f(\pmb{x})$  is  $\mu$ -strongly monotone, i.e.,
$$
\langle \boldsymbol {y} - \boldsymbol {x}, \nabla f (\boldsymbol {y}) - \nabla f (\boldsymbol {x}) \rangle \geq \mu \| \boldsymbol {y} - \boldsymbol {x} \| ^ {2}, \quad \forall \boldsymbol {x}, \boldsymbol {y} \in \mathbb {R} ^ {d}.
$$
Using this fact, we have
$$
\begin{array}{l} \langle \nabla f (\boldsymbol {x} _ {t}), \nabla f (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \rangle = \frac {\| \nabla f (\boldsymbol {x} _ {t}) \|}{\rho} \left\langle \rho \frac {\nabla f (\boldsymbol {x} _ {t})}{\| \nabla f (\boldsymbol {x} _ {t}) \|}, \nabla f (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \right\rangle \\ = \frac {\left\| \nabla f (\boldsymbol {x} _ {t}) \right\|}{\rho} \langle \boldsymbol {y} _ {t} - \boldsymbol {x} _ {t}, \nabla f (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \rangle \\ \geq \frac {\mu \| \nabla f (\boldsymbol {x} _ {t}) \|}{\rho} \| \boldsymbol {y} _ {t} - \boldsymbol {x} _ {t} \| ^ {2} = \mu \rho \| \nabla f (\boldsymbol {x} _ {t}) \|, \\ \end{array}
$$
Lemma B.3. For a  $\beta$ -smooth and  $\mu$ -strongly convex function, with step size  $\eta \leq \frac{1}{2\beta}$ , we have
$$
f (\boldsymbol {x} _ {t + 1}) \leq f (\boldsymbol {x} _ {t}) - \frac {\eta}{2} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} - \frac {\eta \mu \rho}{2} \| \nabla f (\boldsymbol {x} _ {t}) \| + \frac {\eta^ {2} \beta^ {3} \rho^ {2}}{2}.
$$
For a  $\beta$ -smooth and convex function, the inequality continues to hold with  $\mu = 0$ .
Proof. Starting from the definition of  $\beta$ -smoothness, we have
$$
\begin{array}{l} f (\boldsymbol {x} _ {t + 1}) \leq f (\boldsymbol {x} _ {t}) + \langle \nabla f (\boldsymbol {x} _ {t}), \boldsymbol {x} _ {t + 1} - \boldsymbol {x} _ {t} \rangle + \frac {\beta}{2} \| \boldsymbol {x} _ {t + 1} - \boldsymbol {x} _ {t} \| ^ {2} \\ = f (\boldsymbol {x} _ {t}) - \eta \langle \nabla f (\boldsymbol {x} _ {t}), \nabla f (\boldsymbol {y} _ {t}) \rangle + \frac {\eta^ {2} \beta}{2} \| \nabla f (\boldsymbol {y} _ {t}) \| ^ {2} \\ = f (\boldsymbol {x} _ {t}) - \eta \langle \nabla f (\boldsymbol {x} _ {t}), \nabla f (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) + \nabla f (\boldsymbol {x} _ {t}) \rangle \\ + \frac {\eta^ {2} \beta}{2} \| \nabla f (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) + \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} \\ = f (\boldsymbol {x} _ {t}) - \eta \langle \nabla f (\boldsymbol {x} _ {t}), \nabla f (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \rangle - \eta \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} + \frac {\eta^ {2} \beta}{2} \| \nabla f (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} + \frac {\eta^ {2} \beta}{2} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} + \eta^ {2} \beta \langle \nabla f (\boldsymbol {x} _ {t}), \nabla f (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \rangle \\ = f (\boldsymbol {x} _ {t}) - \eta \left(1 - \frac {\eta \beta}{2}\right) \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} - \eta (1 - \eta \beta) \langle \nabla f (\boldsymbol {x} _ {t}), \nabla f (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \rangle \\ + \frac {\eta^ {2} \beta}{2} \| \nabla f (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \| ^ {2}. \\ \end{array}
$$
Since we assumed  $\eta \leq \frac{1}{2\beta}$ , both  $\eta \left(1 - \frac{\eta\beta}{2}\right) \geq \frac{\eta}{2}$  and  $\eta (1 - \eta \beta) \geq \frac{\eta}{2}$  hold. Applying Lemma B.2 to the above, we get
$$
\begin{array}{l} f (\boldsymbol {x} _ {t + 1}) \leq f (\boldsymbol {x} _ {t}) - \frac {\eta}{2} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} - \frac {\eta \mu \rho}{2} \| \nabla f (\boldsymbol {x} _ {t}) \| + \frac {\eta^ {2} \beta}{2} \| \nabla f (\boldsymbol {y} _ {t}) - \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} \\ \leq f (\boldsymbol {x} _ {t}) - \frac {\eta}{2} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} - \frac {\eta \mu \rho}{2} \| \nabla f (\boldsymbol {x} _ {t}) \| + \frac {\eta^ {2} \beta}{2} (\beta \| \boldsymbol {y} _ {t} - \boldsymbol {x} _ {t} \|) ^ {2} \\ = f (\boldsymbol {x} _ {t}) - \frac {\eta}{2} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} - \frac {\eta \mu \rho}{2} \| \nabla f (\boldsymbol {x} _ {t}) \| + \frac {\eta^ {2} \beta^ {3} \rho^ {2}}{2}, \\ \end{array}
$$
B.2 Convergence Proof for Smooth and Strongly Convex Functions (Proof of Theorem 3.1)
In this section, we prove Theorem 3.1, restated below for the sake of convenience.
Theorem 3.1. Consider a  $\beta$ -smooth and  $\mu$ -strongly convex function  $f$ . If we run deterministic SAM starting at  $\mathbf{x}_0$  with any perturbation size  $\rho > 0$  and step size  $\eta = \min \left\{\frac{1}{\mu T} \max \left\{1, \log \left(\frac{\mu^5 \Delta T^2}{\beta^6 \rho^2}\right)\right\}, \frac{1}{2\beta}\right\}$  to minimize  $f$ , we have
$$
\min _ {t \in \{0, \dots , T \}} f (\boldsymbol {x} _ {t}) - f ^ {*} = \tilde {\mathcal {O}} \left(\exp \left(- \frac {\mu T}{2 \beta}\right) \Delta + \frac {\beta^ {6} \rho^ {2}}{\mu^ {5} T ^ {2}}\right).
$$
Proof. The proof is divided into two cases, based on the  $\| \nabla f(\pmb{x}_t)\|$  observed throughout the entire optimization process. In the first case, the gradient norm  $\| \nabla f(\pmb{x}_t)\|$  at  $\pmb{x}_t$  is sufficiently large for all  $t = 0,\dots ,T - 1$ ; in this case, we show that the SAM algorithm converges linearly. The other scenario corresponds to the case where there exists an iteration index  $t\in \{0,\ldots ,T - 1\}$  such that  $\| \nabla f(\pmb{x}_t)\|$  is smaller than a certain threshold. In this case, we can show from the PL inequality (4) that we are already close to global optimality.
From Lemma B.3, we have
Now, recall from Definition B.1 that  $\mu$ -strongly convex functions satisfy  $\mu$ -PL inequality (4). From this, we get
$$
f \left(\boldsymbol {x} _ {t + 1}\right) - f ^ {*} \leq (1 - \eta \mu) \left(f \left(\boldsymbol {x} _ {t}\right) - f ^ {*}\right) - \frac {\eta \mu \rho}{2} \| \nabla f \left(\boldsymbol {x} _ {t}\right) \| + \frac {\eta^ {2} \beta^ {3} \rho^ {2}}{2}. \tag {5}
$$
Case 1: when  $\| \nabla f(\pmb{x}_t)\|$  remains large. Let us now consider the first case, where  $\| \nabla f(\pmb{x}_t)\| \geq \frac{\eta\beta^3\rho}{\mu}$  holds for all  $t = 0,\dots ,T - 1$ . In this case, (5) becomes
$$
f \left(\boldsymbol {x} _ {t + 1}\right) - f ^ {*} \leq (1 - \eta \mu) \left(f \left(\boldsymbol {x} _ {t}\right) - f ^ {*}\right),
$$
which holds for all  $t = 0,\dots ,T - 1$ . Unrolling the inequality, we obtain
$$
f \left(\boldsymbol {x} _ {T}\right) - f ^ {*} \leq (1 - \eta \mu) ^ {T} \left(f \left(\boldsymbol {x} _ {0}\right) - f ^ {*}\right).
$$
Case 2: when some  $\| \nabla f(\pmb{x}_t)\|$  is small. In the other case where there exists  $t$  such that  $\| \nabla f(\pmb{x}_t)\| \leq \frac{\eta\beta^3\rho}{\mu}$ , notice from  $\mu$ -PL inequality (4) that
$$
f (\boldsymbol {x} _ {t}) - f ^ {*} \leq \frac {1}{2 \mu} \| \nabla f (\boldsymbol {x} _ {t}) \| ^ {2} \leq \frac {\eta^ {2} \beta^ {6} \rho^ {2}}{2 \mu^ {3}}.
$$
Therefore, combining the two cases, it is guaranteed that
$$
\min  _ {t \in \{0, \dots , T \}} f \left(\boldsymbol {x} _ {t}\right) - f ^ {*} \leq (1 - \eta \mu) ^ {T} \Delta + \frac {\eta^ {2} \beta^ {6} \rho^ {2}}{2 \mu^ {3}}. \tag {6}
$$
We now elaborate how the choice of step size  $\eta = \min \left\{\frac{1}{\mu T}\max \left\{1,\log \left(\frac{\mu^5\Delta T^2}{\beta^6\rho^2}\right)\right\} ,\frac{1}{2\beta}\right\}$  results in the convergence rate in the theorem statement. Naturally, there are four cases, depending on the outcomes of the min and max operations.
Case A:  $\log \left(\frac{\mu^5\Delta T^2}{\beta^6\rho^2}\right)\geq 1$  and  $\frac{1}{\mu T}\log \left(\frac{\mu^5\Delta T^2}{\beta^6\rho^2}\right)\leq \frac{1}{2\beta}$ . Putting  $\eta = \frac{1}{\mu T}\log \left(\frac{\mu^5\Delta T^2}{\beta^6\rho^2}\right)$  into (6),
$$
\min _ {t \in \{0, \dots , T \}} f (\boldsymbol {x} _ {t}) - f ^ {*} \leq \frac {\beta^ {6} \rho^ {2}}{\mu^ {5} T ^ {2}} + \frac {\beta^ {6} \rho^ {2}}{2 \mu^ {5} T ^ {2}} \log^ {2} \left(\frac {\mu^ {5} \Delta T ^ {2}}{\beta^ {6} \rho^ {2}}\right).
$$
Case B:  $\log \left(\frac{\mu^5\Delta T^2}{\beta^6\rho^2}\right)\geq 1$  and  $\frac{1}{\mu T}\log \left(\frac{\mu^5\Delta T^2}{\beta^6\rho^2}\right)\geq \frac{1}{2\beta}$ . Substituting  $\eta = \frac{1}{2\beta}\leq \frac{1}{\mu T}\log \left(\frac{\mu^5\Delta T^2}{\beta^6\rho^2}\right)$  to (6),
$$
\begin{array}{l} \min  _ {t \in \{0, \dots , T \}} f (\boldsymbol {x} _ {t}) - f ^ {*} \leq \left(1 - \frac {\mu}{2 \beta}\right) ^ {T} \Delta + \frac {\beta^ {6} \rho^ {2}}{2 \mu^ {3}} \cdot \left(\frac {1}{2 \beta}\right) ^ {2} \\ \leq \exp \left(- \frac {\mu T}{2 \beta}\right) \Delta + \frac {\beta^ {6} \rho^ {2}}{2 \mu^ {5} T ^ {2}} \log^ {2} \left(\frac {\mu^ {5} \Delta T ^ {2}}{\beta^ {6} \rho^ {2}}\right). \\ \end{array}
$$
Case C:  $\log \left(\frac{\mu^5\Delta T^2}{\beta^6\rho^2}\right)\leq 1$  and  $\frac{1}{\mu T}\leq \frac{1}{2\beta}$ . Putting  $\eta = \frac{1}{\mu T}\geq \frac{1}{\mu T}\log \left(\frac{\mu^5\Delta T^2}{\beta^6\rho^2}\right)$  into (6),
$$
\begin{array}{l} \min  _ {t \in \{0, \dots , T \}} f (\boldsymbol {x} _ {t}) - f ^ {*} \leq \left(1 - \frac {1}{T}\right) ^ {T} \Delta + \frac {\beta^ {6} \rho^ {2}}{2 \mu^ {5} T ^ {2}} \\ \leq \left(1 - \frac {1}{T} \log \left(\frac {\mu^ {5} \Delta T ^ {2}}{\beta^ {6} \rho^ {2}}\right)\right) ^ {T} \Delta + \frac {\beta^ {6} \rho^ {2}}{2 \mu^ {5} T ^ {2}} \leq \frac {\beta^ {6} \rho^ {2}}{\mu^ {5} T ^ {2}} + \frac {\beta^ {6} \rho^ {2}}{2 \mu^ {5} T ^ {2}}. \\ \end{array}
$$
Case D:  $\log \left(\frac{\mu^5\Delta T^2}{\beta^6\rho^2}\right)\leq 1$  and  $\frac{1}{\mu T}\geq \frac{1}{2\beta}$ . By substituting  $\eta = \frac{1}{2\beta}\leq \frac{1}{\mu T}$  to (6) we obtain
$$
\begin{array}{l} \min _ {t \in \{0, \dots , T \}} f (\boldsymbol {x} _ {t}) - f ^ {*} \leq \left(1 - \frac {\mu}{2 \beta}\right) ^ {T} \Delta + \frac {\beta^ {6} \rho^ {2}}{2 \mu^ {3}} \cdot \left(\frac {1}{2 \beta}\right) ^ {2} \\ \leq \exp \left(- \frac {\mu T}{2 \beta}\right) \Delta + \frac {\beta^ {6} \rho^ {2}}{2 \mu^ {5} T ^ {2}}. \\ \end{array}
$$
Combining the four cases, we conclude
$$
\min _ {t \in \{0, \dots , T \}} f (\pmb {x} _ {t}) - f ^ {*} = \tilde {\mathcal {O}} \left(\exp \left(- \frac {\mu T}{2 \beta}\right) \Delta + \frac {\beta^ {6} \rho^ {2}}{\mu^ {5} T ^ {2}}\right),
$$
B.3 Lower Bound Proof for Smooth and Strongly Convex Functions (Proof of Theorem 3.2)
In this section, we prove Theorem 3.2, restated below for the sake of convenience.
Theorem 3.2. Suppose  $\frac{\beta}{\mu} \geq 2$ . For any choice of perturbation size  $\rho$ , step size  $\eta$ , and initialization  $x_0$ , there exists a differentiable,  $\beta$ -smooth, and  $\mu$ -strongly convex function  $f$  such that
$$
\min _ {t \in \{0, \dots , T \}} f (\boldsymbol {x} _ {t}) - f ^ {*} = \Omega \left(\frac {\beta^ {3} \rho^ {2}}{\mu^ {2} T ^ {2}}\right)
$$
The proof is divided into three cases, depending on the step size. For each case, we will define and analyze a one-dimensional function to show the lower bound. In doing so, without loss of generality we will fix an initialization  $x_0$  because we can appropriately shift the function for different choices of  $x_0$ .
1. For  $\eta \leq \frac{1}{2\mu T}$ , we show that there exists a function  $f$  such that
$$
\min  _ {t \in \{0, \dots , T \}} f (x _ {t}) - f ^ {*} = \Omega \left(\frac {\beta^ {4} \rho^ {2}}{\mu^ {3}}\right).
$$
2. For  $\frac{1}{2\mu T} \leq \eta \leq \frac{2}{\beta}$ , we show existence of a function  $f$  that satisfies
$$
\min _ {t \in \{0, \dots , T \}} f (x _ {t}) - f ^ {*} = \Omega \left(\frac {\beta^ {3} \rho^ {2}}{\mu^ {2} T ^ {2}}\right).
$$
3. For  $\eta \geq \frac{2}{\beta}$ , we show that there exists a function  $f$  such that