[25] B. Kaltenbacher, A. Neubauer, and O. Scherzer. "Iterative regularization methods for nonlinear ill-posed problems." In: Iterative Regularization Methods for Nonlinear Ill-Posed Problems. de Gruyter, 2008.
[26] R. Karakida and K. Osawa. "Understanding approximate fisher information for fast convergence of natural gradient descent in wide neural networks." In: Advances in neural information processing systems 33 (2020), pp. 10891-10901.
[27] A. Kerekes, A. Mészáros, and F. Huszár. "Depth Without the Magic: Inductive Bias of Natural Gradient Descent." In: arXiv preprint arXiv:2111.11542 (2021).
[28] C. Liu, L. Zhu, and M. Belkin. "Loss landscapes and optimization in over-parameterized nonlinear systems and neural networks." In: Applied and Computational Harmonic Analysis 59 (2022), pp. 85-116.
[29] J. Martens. "Deep Learning via Hessian-free Optimization." In: Proceedings of the 27th International Conference on International Conference on Machine Learning. ICML'10. event-place: Haifa, Israel. USA: Omnipress, 2010, pp. 735-742. URL: http://dl.acm.org/citation.cfm?id=3104322.3104416.
[30] J. Martens. “New insights and perspectives on the natural gradient method.” In: The Journal of Machine Learning Research 21.1 (2020), pp. 5776–5851.
[31] J. Martens and R. Grosse. "Optimizing Neural Networks with Kronecker-factored Approximate Curvature." In: arXiv:1503.05671 [cs, stat] (Mar. 2015). arXiv: 1503.05671. URL: http://arxiv.org/abs/1503.05671.
[32] J. Martens and I. Sutskever. "Training Deep and Recurrent Networks with Hessian-Free Optimization." en. In: Neural Networks: Tricks of the Trade: Second Edition. Ed. by G. Montavon, G. B. Orr, and K.-R. Müller. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, pp. 479-535. URL: https://doi.org/10.1007/978-3-642-35289-8_27.
[33] H. Min, S. Tarmoun, R. Vidal, and E. Mallada. "On the explicit role of initialization on the convergence and implicit bias of overparametrized linear networks." In: International Conference on Machine Learning. PMLR. 2021, pp. 7760-7768.
[34] K. Mishchenko. "Regularized Newton Method with Global  $O(1/k * *2)$  Convergence." In: arXiv preprint arXiv:2112.02089 (2021).
[35] J. J. Moré. “The Levenberg-Marquardt algorithm: implementation and theory.” In: Numerical Analysis: Proceedings of the Biennial Conference Held at Dundee, June 28-July 1, 1977. Springer. 2006, pp. 105-116.
[36] R. Mulayoff, T. Michaeli, and D. Soudry. "The implicit bias of minima stability: A view from function space." In: Advances in Neural Information Processing Systems 34 (2021), pp. 17749-17761.
[37] V. Nair and G. E. Hinton. "Rectified linear units improve restricted boltzmann machines." In: Proceedings of the 27th international conference on machine learning (ICML-10). 2010, pp. 807-814.
[38] Y. Nesterov. "Modified Gauss-Newton scheme with worst case guarantees for global performance." In: Optimisation methods and software 22.3 (2007), pp. 469-483.
[39] B. Neyshabur, R. Tomioka, and N. Srebro. "In search of the real inductive bias: On the role of implicit regularization in deep learning." In: arXiv preprint arXiv:1412.6614 (2014).
[40] S. Pesme, L. Pillaud-Vivien, and N. Flammarion. "Implicit bias of sgd for diagonal linear networks: a provable benefit of stochasticity." In: Advances in Neural Information Processing Systems 34 (2021), pp. 29218-29230.
[41] A. G. Ramm. "Dynamical systems method for solving operator equations." In: Communications in Nonlinear Science and Numerical Simulation 9.4 (2004), pp. 383-402.
[42] Y. Ren and D. Goldfarb. "Efficient subsampled gauss-newton and natural gradient methods for training neural networks." In: arXiv preprint arXiv:1906.02353 (2019).
[43] D. A. R. Robin, K. Scaman, and M. Lelarge. "Convergence beyond the over-parameterized regime using Rayleigh quotients." In: Advances in Neural Information Processing Systems. Ed. by A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho. 2022. URL: https://openreview.net/forum?id=pl279ju4G0u.
[44] N. N. Schraudolph. "Fast curvature matrix-vector products for second-order gradient descent." In: Neural computation 14.7 (2002), pp. 1723-1738.