Zakaria Mhammedi, Wouter M Koolen, and Tim Van Erven. Lipschitz adaptivity with multiple learning rates in online learning. In Conference on Learning Theory, pages 2490-2511. PMLR, 2019. (Cited on page 3)
Ryan Murray, Brian Swenson, and Soummya Kar. Revisiting normalized gradient descent: Fast evasion of saddle points. IEEE Transactions on Automatic Control, 64(11):4818-4824, 2019. doi: 10.1109/TAC.2019.2914998. (Cited on page 5)
Yuri Nesterov. Universal gradient methods for convex optimization problems. Mathematical Programming, 152(1-2):381-404, 2014. doi: 10.1007/s10107-014-0790-0. URL https://doi.org/10.1007/s10107-014-0790-0. (Cited on pages 1, 2, and 4)
Yurii Nesterov. Lectures on Convex Optimization. Springer Publishing Company, Incorporated, 2nd edition, 2018. ISBN 3319915770. (Cited on pages 4 and 5)
Francesco Orabona. Dimension-free exponentiated gradient. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL https://proceedings.neurips.cc/paper_files/paper/2013/file/7634ea65a4e6d9041cdf3f7de18e334a-Paper.pdf. (Cited on page 3)
Francesco Orabona. Normalized gradients for all. arXiv preprint arXiv:2308.05621, abs/2308.05621, 2023. URL https://arXiv.org/abs/2308.05621. (Cited on pages 2 and 3)
Francesco Orabona and Ashok Cutkosky. ICML 2020 tutorial on parameter-free online optimization. ICML Tutorials, 2020. URL https://parameterfree.com/icml-tutorial/. (Cited on pages 1, 2, 3, 15, and 18)
Francesco Orabona and David Pal. Coin betting and parameter-free online learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, page 577-585, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819. (Cited on pages 2 and 3)
Francesco Orabona and David Pal. Parameter-free stochastic optimization of variationally coherent functions. arXiv preprint arXiv:2102.00236, abs/2102.00236, 2021. URL https://arXiv.org/abs/2102.00236. (Cited on page 3)
Francesco Orabona and Tatiana Tommasi. Training deep networks without learning rates through coin betting. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 2160-2170, 2017. URL https://proceedings.neurips.cc/paper/2017混沌/7c82fab8c8f89124e2ce92984e04fb40-AAbstract.html. (Cited on page 3)
Antonio Orvieto, Simon Lacoste-Julien, and Nicolas Loizou. Dynamics of sgd with stochastic polyak stepsizes: Truly adaptive variants and convergence to exact solution. arXiv preprint arXiv:2205.04583, abs/2205.04583, 2022. URL https://arXiv.org/abs/2205.04583. (Cited on page 3)
Yan Pan and Yuanzhi Li. Toward understanding why Adam converges faster than SGD for transformers. OPT2023: 14th Annual Workshop on Optimization for Machine Learning, 2022. URL https://openreview.net/pdf?id=Sf1N1V2r6P0. (Cited on page 9)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. URL http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf. (Cited on page 9)