[35] Mahmut Kaya and Hasan Şakir Bilge. Deep metric learning: A survey. Symmetry, 11(9), 2019.
[36] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In ICLR, 2017.
[37] Minyoung Kim, Da Li, Shell X Hu, and Timothy Hospedales. Fisher SAM: Information geometry and sharpness aware minimisation. In ICML, 2022.
[38] SungYub Kim, Sihwan Park, Kyung-Su Kim, and Eunho Yang. Scale-invariant Bayesian neural networks with connectivity tangent kernel. In ICLR, 2023.
[39] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[40] Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In ICLR, 2014.
[41] Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. An infinite-feature extension for Bayesian ReLU nets that fixes their asymptotic overconfidence. In NeurIPS, 2021.
[42] Věra Kúrková and Paul C Kainen. Functionally equivalent feedforward neural networks. Neural Computation, 6(3), 1994.
[43] Daniel Kunin, Javier Sagastuy-Brena, Surya Ganguli, Daniel LK Yamins, and Hidenori Tanaka. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics. In ICLR, 2021.
[44] Frederik Kunstner, Lukas Balles, and Philipp Hennig. Limitations of the empirical Fisher approximation for natural gradient descent. In NeurIPS, 2019.
[45] Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as Gaussian processes. In ICLR, 2018.
[46] John M Lee. Introduction to smooth manifolds. Springer, 2013.
[47] John M Lee. Introduction to Riemannian manifolds. Springer, 2018.
[48] Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after SGD reaches zero loss? a mathematical framework. In NeurIPS, 2021.
[49] Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-Rao metric, geometry, and complexity of neural networks. In AISTATS, 2019.
[50] Wu Lin, Frank Nielsen, Mohammad Emtiyaz Khan, and Mark Schmidt. Structured second-order methods via natural gradient descent. In ICML Workshop on beyond first-order methods in machine learning systems, 2021.
[51] Wu Lin, Frank Nielsen, Mohammad Emtiyaz Khan, and Mark Schmidt. Tractable structured natural gradient descent using local parameterizations. In ICML, 2021.
[52] Christos Louizos and Max Welling. Structured and efficient variational deep learning with matrix Gaussian posteriors. In ICML, 2016.
[53] Kaifeng Lyu, Zhiyuan Li, and Sanjeev Arora. Understanding the generalization benefit of normalization layers: Sharpness reduction. arXiv preprint arXiv:2206.07085, 2022.
[54] David JC MacKay. The evidence framework applied to classification networks. *Neural Computation*, 4(5), 1992.
[55] James Martens. New insights and perspectives on the natural gradient method. JMLR, 21(146), 2020.
[56] James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In International conference on machine learning, pages 2408-2417, 2015.
[57] Alexander G de G Matthews, Mark Rowland, Jiri Hron, Richard E Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. In ICLR, 2018.
[58] Ted Moskovitz, Michael Arbel, Ferenc Huszar, and Arthur Gretton. Efficient Wasserstein natural gradients for reinforcement learning. In ICLR, 2021.
[59] Kevin P. Murphy. Machine Learning: A Probabilistic Perspective. The MIT Press, 2012.
[60] Radford M Neal. Bayesian Learning for Neural Networks. PhD thesis, University of Toronto, 1995.