Si Yi Meng, Sharan Vaswani, Issam Hadj Laradji, Mark Schmidt, and Simon Lacoste-Julien (2020). "Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation". In: AIS-TATS, pp. 1375-1386.
Jorge J. Moré and David J. Thuente (1994). "Line search algorithms with guaranteed sufficient decrease". In: ACM Transactions on Mathematical Software (TOMS) 20.3, pp. 286-307.
Ted Moskovitz, Rui Wang, Janice Lan, Sanyam Kapoor, Thomas Miconi, Jason Yosinski, and Aditya Rawal (2019). First-Order Preconditioning via Hypergradient Descent. arXiv/1910.08461.
Yurii E. Nesterov and Boris T. Polyak (2006). "Cubic regularization of Newton method and its global performance". In: Mathematical Programming 108.1, pp. 177-205.
Jorge Nocedal and Stephen J. Wright (1999). Numerical Optimization. Springer.
Francesco Orabona and David Pál (2016). "Coin Betting and Parameter-Free Online Learning". In: NeurIPS, pp. 577-585.
Francesco Orabona and Tatiana Tommasi (2017). "Training Deep Networks without Learning Rates Through Coin Betting". In: NeurIPS, pp. 2160-2170.
Youngsuk Park, Sauptik Dhar, Stephen P. Boyd, and Mohak Shah (2020). "Variable Metric Proximal Gradient Method with Diagonal Barzilai-Borwein Stepsize". In: ICASSP, pp. 3597-3601.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala (2019). "PyTorch: An Imperative Style, High-Performance Deep Learning Library". In: NeurIPS, pp. 8024–8035.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay (2011). "Scikit-learn: Machine Learning in Python". In: Journal of Machine Learning Research 12, pp. 2825-2830.
Vassilis P. Plagianakos, George D. Magoulas, and Michael N. Vrahatis (2001). "Learning rate adaptation in stochastic gradient descent". In: Advances in Convex Analysis and Global Optimization: Honoring the Memory of C. Caratheodory (1873-1950), pp. 433-444.
Boris T. Polyak (1963). "Gradient methods for minimizing functionals". In: Ž. Vyčisl. Mat i Mat. Fiz. 3, pp. 643-653.
Zhaonan Qu, Wenzhi Gao, Oliver Hinder, Yinyu Ye, and Zhengyuan Zhou (2022). Optimal Diagonal Preconditioning: Theory and Practice. arXiv/2209.00809.
Zheng Qu, Peter Richtárik, Martin Takác, and Olivier Fercoq (2016). "SDNA: Stochastic Dual Newton Ascent for Empirical Risk Minimization". In: ICML, pp. 1823-1832.
Martin A. Riedmiller and Heinrich Braun (1993). "A direct adaptive method for faster backpropagation learning: the RPROP algorithm". In: ICNN, pp. 586-591.
Mher Safaryan, Filip Hanzely, and Peter Richtárik (2021). "Smoothness Matrices Beat Smoothness Constants: Better Communication Compression Techniques for Distributed Optimization". In: NeurIPS, pp. 25688-25702.
George N. Saridis (1970). "Learning Applied to Successive Approximation Algorithms". In: IEEE Transactions on Systems Science and Cybernetics 6.2, pp. 97-103.
Nicol N. Schraudolph (1999). "Local gain adaptation in stochastic gradient descent". In: ICANN, 569-574 vol.2.
Nicol N. Schraudolph, Douglas Aberdeen, and Jin Yu (2005). "Fast Online Policy Gradient Learning with SMD Gain Vector Adaptation". In: NeurIPS, pp. 1185-1192.
Naum Z. Shor (1977). "Cut-off method with space extension in convex programming problems". In: Cybernetics 13.1, pp. 94-96.
Fernando M. Silva and Luís B. Almeida (1990). "Acceleration techniques for the backpropagation algorithm". In: Neural Networks, pp. 110-119.