Alina Ene, Huy L. Nguyen, and Adrian Vladu. Adaptive gradient methods for constrained convex optimization and variational inequalities. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7314-7321, 2021. (Cited on pages 2 and 4)
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Practical Methodology, chapter 11. MIT Press, 2016. URL http://www.deeplearningbook.org. (Cited on page 1)
Benjamin Grimmer. Convergence rates for deterministic and stochastic subgradient methods without lipschitz continuity. SIAM Journal on Optimization, 29(2):1350-1365, 2019. doi: 10.1137/18M117306X. URL https://doi.org/10.1137/18M117306X. (Cited on page 5)
Benjamin Grimmer. On optimal universal first-order methods for minimizing heterogeneous sums. arXiv preprint arXiv:2208.08549, abs/2208.08549, 2022. URL https://arXiv.org/abs/2208.08549. (Cited on pages 1 and 2)
Vineet Gupta, Tomer Koren, and Yoram Singer. A unified approach to adaptive regularization in online and stochastic optimization. arXiv preprint arXiv:1706.06569, abs/1706.06569, 2017. URL https://arXiv.org/abs/1706.06569. (Cited on page 17)
Elad Hazan and Sham Kakade. Revisiting the Polyak step size. arXiv preprint arXiv:1905.00313, abs/1905.00313, 2019. URL https://arXiv.org/abs/1905.00313. (Cited on pages 2 and 3)
Elad Hazan and Nimrod Megiddo. Online learning with prior knowledge. In Nader H. Bshouty and Claudio Gentile, editors, Learning Theory, pages 499-513, Berlin, Heidelberg, 2007. Springer Berlin Heidelberg. ISBN 978-3-540-72927-3. (Cited on page 3)
Elad Hazan, Kfir Y. Levy, and Shai Shalev-Shwartz. Beyond convexity: Stochastic quasi-convex optimization. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 1594-1602, 2015. URL https://proceedings.neurips.cc/paper/2015/ hash/934815ad542a4a7c5e8a2dfa04fea9f5-AAbstract.html. (Cited on page 5)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. doi: 10.1109/CVPR.2016.90. (Cited on page 9)
Maor Ivgi, Oliver Hinder, and Yair Carmon. DoG is SGD's best friend: A parameter-free dynamic step size schedule. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 14465-14499. PMLR, 23-29 Jul 2023. (Cited on pages 2, 3, 7, 8, 9, 15, 16, 17, and 22)
Ali Kavis, Kfir Y. Levy, Francis R. Bach, and Volkan Cevher. UniXGrad: A universal, adaptive algorithm with optimal guarantees for constrained optimization. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 6257-6266, 2019. URL https://proceedings.neurips.cc/paper/2019/ hash/88855547570f7ff053fff7c54e5148cc-Abstract.html. (Cited on page 2)
Ahmed Khaled and Peter Richtárik. Better theory for SGD in the nonconvex world. arXiv preprint arXiv:2002.03329, abs/2002.03329, 2020. URL https://arXiv.org/abs/2002.03329. (Cited on page 15)
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arXiv.org/abs/1412.6980. (Cited on pages 1, 2, 7, and 9)
Alex Krizhevsky. Learning multiple layers of features from tiny images. pages 32-33, 2009. URL https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf. (Cited on page 9)