[24] B. Duplantier, Comment on Parisi's equation for the SK model for spin glasses, J. Phys. A, Math. Gen. 14 (1) (1981) 283.

[25] Amir Dembo, Ofer Zeitouni, Large Deviations Techniques and Applications, Springer, 1998.

[26] Ahmed El Alaoui, Andrea Montanari, Mark Sellke, Algorithmic thresholds in mean field spin glasses, Ann. Probab. (2021).

[27] Ahmed El Alaoui, Andrea Montanari, Mark Sellke, Optimization of mean-field spin glasses, Ann. Probab. 49 (6) (2021) 2922-2960.

[28] Richard S. Ellis, Entropy, Large Deviations, and Statistical Mechanics, vol. 1431, Taylor & Francis, 2006.

[29] Andreas Engel, Christian Van den Broeck, Statistical Mechanics of Learning, Cambridge University Press, 2001.

[30] Silvio Franz, Giorgio Parisi, Maxime Sevelev, Pierfrancesco Urbani, Francesco Zamponi, Universality of the SAT-UNSAT (jamming) threshold in non-convex continuous constraint satisfaction problems, SciPost Phys. 2 (3) (2017) 019.

[31] Silvio Franz, Giorgio Parisi, Pierfrancesco Urbani, Francesco Zamponi, Universal spectrum of normal modes in low-temperature glasses, Proc. Natl. Acad. Sci. 112 (47) (2015) 14539-14544.

[32] Mahyar Fazlyab, Alexander Robey, Hamed Hassani, Manfred Morari, George Pappas, Efficient and accurate estimation of Lipschitz constants for deep neural networks, Adv. Neural Inf. Process. Syst. 32 (2019).

[33] Elizabeth Gardner, The space of interactions in neural network models, J. Phys. A, Math. Gen. 21 (1) (1988) 257.

[34] Elizabeth Gardner, Bernard Derrida, Optimal storage properties of neural network models, J. Phys. A, Math. Gen. 21 (1) (1988) 271.

[35] Pascal Getreuer, A survey of Gaussian convolution algorithms, Image Process. Line 2013 (2013) 286-310.

[36] Henry Gouk, Eibe Frank, Bernhard Pfahringer, Michael J. Cree, Regularisation of neural networks by enforcing Lipschitz continuity, Mach. Learn. 110 (2) (2021) 393-416.

[37] Federica Gerace, Florent Krzakala, Bruno Loureiro, Ludovic Stephan, Lenka Zdeborová, Gaussian universality of perceptrons with random labels, Phys. Rev. E 109 (3) (2024) 034305.

[38] Yehoram Gordon, Some inequalities for Gaussian processes and applications, Isr. J. Math. 50 (4) (1985) 265-289.

[39] Yehoram Gordon, On Milman's inequality and random subspaces which escape through a mesh in $\mathbb{R}^n$, in: Geometric Aspects of Functional Analysis, Springer, 1988, pp. 84-106.

[40] Francesco Guerra, Fabio Lucio Toninelli, The thermodynamic limit in mean field spin glass models, Commun. Math. Phys. 230 (1) (2002) 71-79.

[41] Francesco Guerra, Broken replica symmetry bounds in the mean field spin glass model, Commun. Math. Phys. 233 (1) (2003) 1-12.

[42] Boris Hanin, Mihai Nica, Products of many large random matrices and gradients in deep neural networks, Commun. Math. Phys. 376 (1) (2020) 287-322.

[43] Matt Jordan, Alexandros G. Dimakis, Exactly computing the local Lipschitz constant of relu networks, Adv. Neural Inf. Process. Syst. 33 (2020) 7344-7353.

[44] Konik Kothari, AmirEhsan Khorashadizadeh, Maarten de Hoop, Ivan Dokmanić, Trumpets: injective flows for inference and inverse problems, in: Uncertainty in Artificial Intelligence, PMLR, 2021, pp. 1269-1278.

[45] Cosme Louart, Zhenyu Liao, Romain Couillet, A random matrix approach to neural networks, Ann. Appl. Probab. 28 (2) (2018) 1190-1248.

[46] Antoine Maillard, *Numerical code used to produce the figures*, https://github.com/AnMaillard/Injectivity_ReLu layer, 2023.

[47] Andrea Montanari, Optimization of the Sherrington-Kirkpatrick Hamiltonian, SIAM J. Comput. (2021) FOCS19-1.

[48] Marc Mézard, Giorgio Parisi, Replica field theory for random manifolds, J. Phys. I 1 (6) (1991) 809-836.

[49] Marc Mézard, Giorgio Parisi, Nicolas Sourlas, G. Toulouse, Miguel Virasoro, Nature of the spin-glass phase, Phys. Rev. Lett. 52 (13) (1984) 1156.

[50] Marc Mézard, Giorgio Parisi, Miguel Virasoro, Spin Glass Theory and Beyond: An Introduction to the Replica Method and Its Applications, vol. 9, World Scientific Publishing Company, 1987.

[51] Andrea Montanari, Basil N. Saeed, Universality of empirical risk minimization, in: Conference on Learning Theory, PMLR, 2022, pp. 4310-4312.

[52] Andrea Montanari, Subhabrata Sen, A friendly tutorial on mean-field spin glass techniques for non-physicists, Found. Trends Mach. Learn. 17 (1) (2024) 1-173.

[53] Andrea Montanari, Yiqiao Zhong, Kangjie Zhou, Tractability from overparametrization: the example of the negative perceptron, Probab. Theory Relat. Fields 188 (3) (2024) 805-910.

[54] Daniel Paleka, Injectivity of ReLU neural networks at initialization, Master's thesis, ETH Zurich, 2021.

[55] Dmitry Panchenko, The Parisi ultrametricity conjecture, Ann. Math. (2013) 383-393.

[56] Dmitry Panchenko, The Parisi formula for mixed $p$-spin models, Ann. Probab. 42 (3) (2014) 946-958.

[57] Dmitry Panchenko, The free energy in a multi-species Sherrington-Kirkpatrick model, Ann. Probab. 43 (6) (2015) 3494-3513.

[58] Giorgio Parisi, Infinite number of order parameters for spin-glasses, Phys. Rev. Lett. 43 (23) (1979) 1754.

[59] Giorgio Parisi, The order parameter for spin glasses: a function on the interval 0-1, J. Phys. A, Math. Gen. 13 (3) (1980) 1101.

[60] Giorgio Parisi, A sequence of approximated solutions to the sk model for spin glasses, J. Phys. A, Math. Gen. 13 (4) (1980) L115.

[61] Michael Puthawala, Konik Kothari, Matti Lassas, Ivan Dokmanic, Maarten de Hoop, Globally injective relu networks, J. Mach. Learn. Res. 23 (105) (2022) 1-55.

[62] Michael Puthawala, Matti Lassas, Ivan Dokmanic, Maarten de Hoop, Universal joint approximation of manifolds and densities by simple injective flows, in: International Conference on Machine Learning, PMLR, 2022, pp. 17959-17983.

[63] Jeffrey Pennington, Pratik Worah, Nonlinear random matrix theory for deep learning, Adv. Neural Inf. Process. Syst. 30 (2017).

[64] Brendan Ross, Jesse Cresswell, Tractable density estimation on learned manifolds with conformal embedding flows, Adv. Neural Inf. Process. Syst. 34 (2021) 26635-26648.

[65] Corrado Rainone, Pierfrancesco Urbani, Hajime Yoshino, Francesco Zamponi, Following the evolution of hard sphere glasses in infinite dimensions under external perturbations: compression and shear strain, Phys. Rev. Lett. 114 (1) (2015) 015701.

[66] David Sherrington, Scott Kirkpatrick, Solvable model of a spin-glass, Phys. Rev. Lett. 35 (26) (1975) 1792.

[67] Maria Shcherbina, Brunello Tirozzi, On the volume of the intersection of a sphere with random half spaces, C. R. Math. 334 (9) (2002) 803-806.

[68] Marija Shcherbina, Brunello Tirozzi, Rigorous solution of the Gardner problem, Commun. Math. Phys. 234 (3) (2003) 383-422.

[69] Mihailo Stojnic, Another look at the Gardner problem, arXiv preprint, arXiv:1306.3979, 2013.

[70] Mihailo Stojnic, Negative spherical perceptron, arXiv preprint, arXiv:1306.3980, 2013.

[71] Plamen Stefanov, Gunther Uhlmann, Linearizing non-linear inverse problems and an application to inverse backscattering, J. Funct. Anal. 256 (9) (2009) 2842-2866.

[72] Eliran Subag, Following the ground states of Full-RSB spherical spin glasses, Commun. Pure Appl. Math. 74 (5) (2021) 1021-1044.

[73] Eliran Subag, Convergence of the free energy for spherical spin glasses, J. Stat. Phys. 189 (2) (2022) 29.

[74] Rolf Schneider, Wolfgang Weil, Stochastic and Integral Geometry, vol. 1, Springer, 2008.

[75] Christos Thrampoulidis, Ehsan Abbasi, Babak Hassibi, Precise error analysis of regularized M-estimators in high dimensions, IEEE Trans. Inf. Theory 64 (8) (2018) 5592-5628.

[76] Michel Talagrand, Free energy of the spherical mean field model, Probab. Theory Relat. Fields 134 (3) (2006) 339.

[77] Michel Talagrand, The Parisi formula, Ann. Math. (2006) 221-263.

[78] Michel Talagrand, Mean Field Models for Spin Glasses: Volume I: Basic Examples, vol. 54, Springer Science & Business Media, 2010.

[79] Michel Talagrand, Mean Field Models for Spin Glasses: Volume II: Advanced Replica-Symmetry and Low Temperature, vol. 55, Springer Science & Business Media, 2011.

[80] Christos Thrampoulidis, Samet Oymak, Babak Hassibi, Regularized linear regression: a precise analysis of the estimation error, in: Conference on Learning Theory, PMLR, 2015, pp. 1683-1709.

[81] Pierfrancesco Urbani, Statistical physics of glassy systems: tools and applications, 2018.
[82] Roman Vershynin, High-Dimensional Probability: An Introduction with Applications in Data Science, vol. 47, Cambridge University Press, 2018.

[83] Ramon van Handel, Probability in high dimension, Technical report, Princeton Univ., NJ, 2014.

[84] Keyulu Xu, Weihua Hu, Jure Leskovec, Stefanie Jegelka, How powerful are graph neural networks?, in: International Conference on Learning Representations, 2018.