Shuang Li, William J Swartworth, Martin Takáč, Deanna Needell, and Robert M Gower. SP2: A second order stochastic Polyak method. arXiv preprint arXiv:2207.08171, 2022. Cited on 2
Jinxiu Liang, Yong Xu, Chenglong Bao, Yuhui Quan, and Hui Ji. Barzilai-Borwein-based adaptive learning rate for deep learning. Pattern Recognition Letters, 128:197-203, 2019. Cited on 5, 10
Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over-parameterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis, 59: 85-116, 2022. Cited on 2, 3, 6, 10
Nicolas Loizou, Sharan Vaswani, Issam Hadj Laradji, and Simon Lacoste-Julien. Stochastic polyak step-size for sgd: An adaptive learning rate for fast convergence. In International Conference on Artificial Intelligence and Statistics, pages 1306-1314. PMLR, 2021. Cited on 2, 3, 4, 5, 6, 8
L. Luo, Y. Xiong, Y. Liu, and X. Sun. Adaptive gradient methods with dynamic bound of learning rate. In International Conference on Learning Representations, 2019. Cited on 1, 6
Maren Mahsereci and Philipp Hennig. Probabilistic line searches for stochastic optimization. The Journal of Machine Learning Research, 18(1):4262-4320, 2017. Cited on 1, 3
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The Penn Treebank. Computational Linguistics 19.2, pages 313-330, 1993. Cited on 9
Si Yi Meng, Sharan Vaswani, Issam Hadj Laradji, Mark Schmidt, and Simon Lacoste-Julien. Fast and furious convergence: Stochastic second order methods under interpolation. In International Conference on Artificial Intelligence and Statistics, pages 1375-1386. PMLR, 2020. Cited on 3
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. Cited on 9
Maximus Mutschler and Andreas Zell. Parabolic approximation line search for DNNs. Advances in Neural Information Processing Systems, 33:5405-5416, 2020. Cited on 1
Mor Shpigel Nacson, Kavya Ravichandran, Nathan Srebro, and Daniel Soudry. Implicit bias of the step size in linear diagonal neural networks. In International Conference on Machine Learning, pages 16270-16295. PMLR, 2022. Cited on 1, 6
Jorge Nocedal and Stephen Wright. Numerical optimization. Springer Science & Business Media, 2006. Cited on 1, 5, 7
Yan Pan and Yuanzhi Li. Toward understanding why Adam converges faster than SGD for transformers. In Optimization for Machine Learning (NeurIPS Workshop), 2022. Cited on 9
Courtney Paquette and Katya Scheinberg. A stochastic line search method with convergence rate analysis. arXiv preprint arXiv:1807.07994, 2018. Cited on 1, 3
Vassilis P Plagianakos, George D Magoulas, and Michael N Vrahatis. Deterministic nonmonotone strategies for effective training of multilayer perceptrons. IEEE Transactions on Neural Networks, 13(6):1268-1284, 2002. Cited on 2
Boris Teodorovich Polyak. Minimization of unsmooth functionals. USSR Computational Mathematics and Mathematical Physics, 9(3):14-29, 1969. Cited on 2
M. Raydan. The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem. SIAM Journal on Optimization, 7(1):26-33, 1997. Cited on 2
Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathematical Statistics, pages 400-407, 1951. Cited on 1, 6
Mark Schmidt and Nicolas Le Roux. Fast convergence of stochastic gradient descent under a strong growth condition. arXiv preprint arXiv:1308.6370, 2013. Cited on 2