David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350, abs/2104.10350, 2021. URL https://arXiv.org/abs/2104.10350. (Cited on page 1)
Boris Polyak. Introduction to optimization. Optimization Software, 1987. (Cited on pages 2 and 3)
Boris T. Polyak. Some methods of speeding up the convergence of iteration methods. *USSR Computational Mathematics and Mathematical Physics*, 4(5):1-17, 1964. ISSN 0041-5553. doi: https://doi.org/10.1016/0041-5553(64)90137-5. URL https://www.sciencedirect.com/science/article/pii/0041555364901375. (Cited on page 1)
Or Sharir, Barak Peleg, and Yoav Shoham. The cost of training NLP models: a concise overview. arXiv preprint arXiv:2004.08900, abs/2004.08900, 2020. URL https://arXiv.org/abs/2004.08900. (Cited on page 1)
Naum Zuselevich Shor. Minimization methods for non-differentiable functions, volume 3. Springer Science & Business Media, 2012. (Cited on page 5)
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In *Joshua Bengio and Yann LeCun*, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1409.1556. (Cited on page 9)
Matthew Streeter and H. Brendan McMahan. No-regret algorithms for unconstrained online convex optimization. In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 2, NIPS'12, page 2402-2410, Red Hook, NY, USA, 2012. Curran Associates Inc. (Cited on page 3)
Cheik Traore and Edouard Pauwels. Sequential convergence of AdaGrad algorithm for smooth convex optimization. Operations Research Letters, 49(4):452-458, 2021. (Cited on page 4)
Rachel Ward, Xiaoxia Wu, and Léon Bottou. AdaGrad stepsizes: sharp convergence over nonconvex landscapes. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 6677-6686. PMLR, 2019. URL http://proceedings.mlr.press/v97/ward19a.html. (Cited on page 4)
Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4148-4158, 2017. URL https://proceedings.neurips.cc/paper/2017/bit/81b3833e2504647f9d794f7d7b9bf341-AAbstract.html. (Cited on page 1)
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jabbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020a. URL https://openreview.net/forum?id=BJgnXpVYwS. (Cited on page 4)
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank J. Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and HsuanTien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020b. URL https://proceedings.neurips.cc/paper/2020/bit/ b05b57f6add810d3b7490866d74c0053-AAbstract.html. (Cited on page 4)