[1] Kwangjun Ahn, Jingzhao Zhang, and Suvrit Sra. Understanding the unstable convergence of gradient descent. arXiv preprint arXiv:2204.01050, 2022. (Cited on page 6.)
[2] Ahmad Ajalloeian and Sebastian U. Stich. On the convergence of sgd with biased gradients. arXiv preprint arXiv:2008.00051, 2021. (Cited on page 4.)
[3] Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming, pages 1–50, 2022. (Cited on pages 2, 4, 5, 7, 10, and 16.)
[4] Larry Armijo. Minimization of functions having lipschitz continuous first partial derivatives. Pacific Journal of mathematics, 16(1):1-3, 1966. (Cited on page 4.)
[5] Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients. In International Conference on Machine Learning, pages 404-413. PMLR, 2018. (Cited on page 5.)
[6] Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157-166, 1994. (Cited on page 2.)
[7] Yoshua Bengio. Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2(1):1-127, 2009. (Cited on page 1.)
[8] Julius R Blum. Multidimensional stochastic approximation methods. The Annals of Mathematical Statistics, pages 737-744, 1954. (Cited on page 4.)
[9] Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223-311, 2018. (Cited on pages 4 and 5.)
[10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. (Cited on page 4.)
[11] Yair Carmon, John C Duchi, Oliver Hinder, and Aaron Sidford. Lower bounds for finding stationary points i. Mathematical Programming, 184(1-2):71-120, 2020. (Cited on pages 2 and 16.)
[12] Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of a class of Adam-type algorithms for non-convex optimization. In International Conference on Learning Representations. International Conference on Learning Representations, ICLR, 2019. (Cited on pages 4 and 9.)
[13] Kai Lai Chung. On a stochastic approximation method. The Annals of Mathematical Statistics, pages 463-483, 1954. (Cited on page 4.)
[14] Ashok Cutkosky and Harsh Mehta. Momentum improves normalized sgd. In International conference on machine learning, pages 2260–2268. PMLR, 2020. (Cited on pages 2, 4, 8, 16, 23, and 24.)
[15] Felipe RS De Gusmao, Edwin MM Ortega, and Gauss M Cordeiro. The generalized inverse weibull distribution. Statistical Papers, 52:591-619, 2011. (Cited on page 30.)
[16] Alexandre Défossez, Léon Bottou, Francis Bach, and Nicolas Usunier. A simple convergence proof of adam and adagrad. arXiv preprint arXiv:2003.02395, 2020. (Cited on page 4.)
[17] Yoel Drori and Ohad Shamir. The complexity of finding stationary points with stochastic gradient descent. In International Conference on Machine Learning, pages 2658-2667. PMLR, 2020. (Cited on pages 2, 4, 6, 7, 19, 21, and 28.)