Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, 2019. Cited on 9
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(7), 2011. Cited on 1
Chen Fan, Christos Thrampoulidis, and Mark Schmidt. Fast convergence of random reshuffling under over-parameterization and the polyak- $\backslash 1$  ojasiewicz condition. arXiv preprint arXiv:2304.00459, 2023. Cited on 4
MC Ferris and S Lucidi. Nonmonotone stabilization methods for nonlinear equations. Journal of Optimization Theory and Applications, 81(1):53-71, 1994. Cited on 1
Robert M Gower, Aaron Defazio, and Michael Rabbat. Stochastic Polyak stepsize with a moving target. arXiv preprint arXiv:2106.11851, 2021. Cited on 2
Robert M Gower, Mathieu Blondel, Nidham Gazagnadou, and Fabian Pedregosa. Cutting some slack for SGD with adaptive polyak step sizes. arXiv preprint arXiv:2202.12328, 2022. Cited on 2
Geovani N, Grapiglia and Ekkehard W Sachs. A generalized worst-case complexity analysis for non-monotone line searches. Numerical Algorithms, 87:779-796, 2021. Cited on 3
Luigi Grippo, Francesco Lampariello, and Stefano Lucidi. A nonmonotone line search technique for Newton's method. SIAM Journal on Numerical Analysis, 23(4):707-716, 1986. Cited on 1, 2, 3, 6
Sajad Fathi Hafshejani, Daya Gaur, Shahadat Hossain, and Robert Benkoczi. Fast armijo line search for stochastic gradient descent. Research Square preprint https://doi.org/10.21203/rs.3.rs-2285238/v1, 2023. Cited on 2, 3
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016. Cited on 3, 6
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017. Cited on 3, 6
Nitish Shirish Keskar and George Saon. A nonmonotone learning rate strategy for sgd training of deep neural networks. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4974-4978. IEEE, 2015. Cited on 2
D.P. Kingma and J.L. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. Cited on 6
Nataša Krejić and Nataša Krklec Jerinkić. Nonmonotone line search methods with variable sample size. Numerical Algorithms, 68(4):711-739, 2015. Cited on 2, 3
Nataša Krejic and Nataša Krklec Jerinkić. Spectral projected gradient method for stochastic optimization. Journal of Global Optimization, 73(1):59-81, 2019. Cited on 2, 3
Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be. In The Eleventh International Conference on Learning Representations, 2023. Cited on 8, 9
Nicolas Le Roux, Mark Schmidt, and Francis Bach. A stochastic gradient method with an exponential convergence_rate for finite training sets. Advances in neural information processing systems, 25, 2012. Cited on 2
Donghui Li and Masao Fukushima. A globally and superlinearly convergent Gauss-Newton-based BFGS method for symmetric nonlinear equations. SIAM Journal on Numerical Analysis, 37(1): 152-172, 1999. Cited on 2