Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise is not the main factor behind the gap between SGD and Adam on transformers, but sign descent might be. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=a65YK0cqH8g. (Cited on page 4)
Puya Latafat, Andreas Themelis, Lorenzo Stella, and Panagiotis Patrinos. Adaptive proximal algorithms for convex optimization under local Lipschitz continuity of the gradient. arXiv preprint arXiv:2301.04431, 2023. (Cited on page 3)
Kfir Y. Levy. Online to offline conversions, universality and adaptive minibatch sizes. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 1613-1622, 2017. URL https://proceedings.neurips.cc/paper/2017/ hash/ce5140df15d046a66883807d18d0264b-Abstract.html. (Cited on pages 3, 5, and 15)
Kfir Yehuda Levy, Alp Yurtsever, and Volkan Cevher. Online adaptive methods, universality and acceleration. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 6501-6510, 2018. URL https://proceedings.neurips.cc/paper/2018混沌/b0169350cd35566c47ba83c6ec1d6f82-AAbstract.html. (Cited on pages 1, 2, 4, 7, 15, and 18)
Haochuan Li, Ali Jadbabaie, and Alexander Rakhlin. Convergence of adam under relaxed assumptions. arXiv preprint arXiv:2304.13972, abs/2304.13972, 2023. URL https://arXiv.org/abs/2304.13972. (Cited on page 2)
Xiaoyu Li and Francesco Orabona. On the convergence of stochastic gradient descent with adaptive stepsizes. In Kamalika Chaudhuri and Masashi Sugiyama, editors, The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Learning Research, pages 983-992. PMLR, 2019. URL http://proceedings.mlr.press/v89/1i19c.html. (Cited on page 4)
Zijian Liu, Ta Duy Nguyen, Alina Ene, and Huy L. Nguyen. On the convergence of AdaGrad(norm) on  $\mathbb{R}^d$ : Beyond convexity, non-asymptotic rate and acceleration. arXiv preprint arXiv:2209.14827, abs/2209.14827, 2022. URL https://arXiv.org/abs/2209.14827. (Cited on page 4)
Nicolas Loizou, Sharan Vaswani, Issam Hadj Laradji, and Simon Lacoste-Julien. Stochastic polyak step-size for SGD: an adaptive learning rate for fast convergence. In Arindam Banerjee and Kenji Fukumizu, editors, The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event, volume 130 of Proceedings of Machine Learning Research, pages 1306-1314. PMLR, 2021. URL http://proceedings.mlr.press/v130/loizou21a.html. (Cited on page 3)
Yura Malitsky and Konstantin Mishchenko. Adaptive gradient descent without descent. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 6702-6712. PMLR, 2020. URL http://proceedings.mlr.press/v119/malitsky20a.html. (Cited on pages 3 and 4)
H. Brendan McMahan and Francesco Orabona. Unconstrained online linear learning in hilbert spaces: Minimax algorithms and normal approximations. In Maria-Florina Balcan, Vitaly Feldman, and Csaba Szepesvári, editors, Proceedings of The 27th Conference on Learning Theory, COLT 2014, Barcelona, Spain, June 13-15, 2014, volume 35 of JMLR Workshop and Conference Proceedings, pages 1020-1039. JMLR.org, 2014. URL http://proceedings.mlr.press/v35/mcmahan14.html. (Cited on page 3)
Zakaria Mhammedi and Wouter M Koolen. Lipschitz and comparator-norm adaptivity in online learning. In Conference on Learning Theory, pages 2858-2887. PMLR, 2020. (Cited on page 3)