SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Definition 3 (Isometry strength). Given activation  $\sigma$  with Hermite expansion  $\{c_k\}_{k\geq 0}$ , define its isometry strength  $\beta_{\sigma}$  as:
$$
\beta_ {\sigma} := 2 - \frac {c _ {1} ^ {2}}{\sum_ {k = 1} ^ {\infty} c _ {k} ^ {2}} \tag {11}
$$
We can readily check from the definition that isometry strength  $\beta_{\sigma}$  has the following basic properties: (i) it ranges between 1 and 2, and (ii) it is 1 if and only if the activation is a linear function. Table 2 presents the isometry strength of certain activations in closed form. With this definition, we can finally analyze Gram matrix mean-field dynamics. Interestingly, the negative log of isometry -  $\log \mathcal{I}(G_{*}^{\ell})$  can serve as a Lyapunov function for the above dynamics. The following theorem proves non-linear activations also impose isometry similar to normalization layers.
Theorem 4. Let  $\sigma$  be an activation function with a Hermite expansion and a non-linearity strength  $\beta_{\sigma}$  (see equation (11)). Given non-degenerate input Gram matrix  $G_{*}^{0}$ , then for sufficiently large layer  $\ell \gtrsim \beta_{\sigma}^{-1}(-n\log \mathcal{I}(G_{*}^{0}) + \log (4n))$ , we have
$$
- \log \mathcal {I} \left(G _ {*} ^ {\ell}\right) \leq \exp (- \ell \log \beta_ {\sigma} - n \log \mathcal {I} \left(G _ {*} ^ {0}\right) + \log (4 n)). \tag {12}
$$
Note that the condition on input being non-degenerate is essential to reach isometry through depth. For example, if the input batch contains a duplicated sample, their corresponding representations across all layers will remain duplicated, implying that all  $G_{*}^{\ell}$ 's will be degenerate.
Theorem 4 reveals the importance of non-linear Hermite coefficients  $(c_k, k \geq 2)$  in activation function to ensure  $\beta_{\sigma} > 1$  and obtain isometry in depth. This connection between  $\beta_{\sigma}$  and isometry is the rationale for referring to  $\beta_{\sigma}$  as the isometry strength. This constant can be computed in closed form for various activations, as shown in Table 2, and for all other activations, it can be computed numerically by sampling.
He1(x)	He2(x)	Sine	Exponential	Step	ReLU
σ	x	x2-1	sin(x)	exp(x-2)	1[x>0]	max(x,0)
βσ	1	2	2-2e/e2-1	2-1/e-1	2-2/π	3π-4/2π-2
Table 2: Isometry strength  ${\beta }_{\sigma }$  (see Definition 3) for various activation functions.
Figure 3 compares the established bound on the isometry gap with those observed in practice, i.e.  $G^{\ell}$ , for three activations. We observe  $\beta_{\sigma}$  predicts the decay rate in isometry of Gram matrices  $G^{\ell}$ . While so far, we have only discussed the direct results of our theory for isometry of Gram matrices, in the next section, we will discuss other insights from the above analysis.
Figure 3: Validation of Theorem 4. Solid blue traces show the isometry of MLP with  $n = 100$ ,  $d = 5000$ , and various activations  $\sigma$ . Solid lines show the average of #10 independent runs. The dashed traces are theoretical upper bounds given in Theorem 4, with constant  $C = 1$ .
6 Implications of our theory
In this section, we elucidate the implications of our theory, beginning with insights into layer normalization through the Hermit expansion of activation functions, followed by an examination of its impact on training.
Layer normalization primarily involves two steps: (i) centering and (ii) normalizing the norms. Through the Hermit expansion of activation functions, we unravel the underlying intricacies of these components and propose alternatives based on insights from the Hermit expansion.
**Experimental Setup.** Our experiments utilize MLPs with layer normalization and various activation functions for the task of image classification on the CIFAR10 dataset [Krizhevsky et al.]. For training, we use stochastic gradient descent (SGD) with a fixed step size of 0.01. Unless stated otherwise, the MLP has constant width of  $d = 1000$  is maintained across hidden layers, and a batch size of  $n = 10$ . Throughout this section,  $a^\ell$  and  $x^\ell$  respectively denote the post-activation and post-normalization vector for layer  $\ell$ .
6.1 Centering and Hermit expansion
One of the key insights of our mean-field theory is that the centering step in layer normalization is crucial in obtaining isometry. In the mean-field regime, pre-activations follow standard Gaussian distributions, and thus the average post-activation  $\overline{a^{\ell}} = \frac{1}{d}\sum_{i}^{d}a_{i}^{\ell}$  will converge to their expectation  $\overline{a^{\ell}} = \mathbb{E}_{z\sim \mathcal{N}(0,1)}[\sigma (X)] = c_{0}$ . This insight suggests an alternative way of obtaining isometry by explicitly removing the offset term from activation, i.e., replacing activation  $\sigma (x)$  by  $\sigma (x) - c_0$ . Strikingly, our experiments presented in Figure 4 indicate that such replacement can also impose isometry. This result provides novel insights into the role of centering in layer normalization.
Figure 4: Layer vs. mean-field centering in obtaining isometry. Layer centering (orange):  $x^{\ell +1} = \mathrm{LN}(a^{\ell} - \overline{a^{\ell}})$ . Mean-field centering (blue):  $x^{\ell +1} = \mathrm{LN}(a^{\ell} - c_{0})$ .
6.2 Normalization and Hermit expansion
Theorem 4 further reveals the importance of normalization of norms in addition to centering to achieve isometry. Figure 5 underlines the importance of the normalization for different activations, where we observe the isometry gap may increase without normalization. Similar to our mean-field analysis of centering, the factor  $\frac{1}{d}\sum_{i=1}^{d}(a_i^\ell - \overline{a^\ell})^2$  in layer normalization converges to variance
$\operatorname{var}_{z \sim N(0,1)}(\sigma(z)) = \sum_{k=1}^{\infty} c_k^2 =: \bar{\sigma}(1)$ . Thus, as the width increases, the layer normalization operator  $\mathrm{LN}(a^\ell - a^\ell)$  will converge to  $(a^\ell - c_0) / \sqrt{\bar{\sigma}(1)}$ . Figure 6 demonstrates that the constant scaling  $1 / \sqrt{\bar{\sigma}(1)}$ , achieves comparable isometry to layer normalization for hyperbolic tangent and sigmoid and ReLU, while it is not effective for exp function. This observation calls for future research on the link between normalization and activation in deep neural networks.
Figure 5: Comparing no normalization (blue)  $x^{\ell +1} = (a^{\ell} - c_{0})$  with layer normalization (orange)  $x^{\ell +1} = \mathrm{LN}(a^{\ell} - c_{0})$  in obtaining isometry.
Figure 6: Comparing layer normalization (blue)  $x^{\ell +1} = \mathrm{LN}(a^{\ell} - c_{0})$  vs. mean-field normalization (orange)  $x^{\ell +1} = \frac{a^{\ell} - c_{0}}{\sqrt{\bar{\sigma}(1)}}$  in obtaining isometry.
6.3 Isometry strength correlates with SGD convergence rate in shallow MLPs.
Besides the direct consequences of our theory, we observe a striking correlation between the convergence of SGD and isometry strength in a specific range of neural network hyper-parameters. Figure 7 shows the convergence of SGD is faster for activations with a significantly larger isometry strength  $\beta_{\sigma}$  (see Definition 3) for shallow MLPs, e.g., with 10 layers or less. We can speculate that this correlation reflects the input-output sensitivity of the networks with higher non-linearity. Surprisingly, this correlation does not extend to deeper networks. This discrepancy between shallow and deep networks regarding SGD convergence may be due to the issue of gradient explosion studied by Meterez et al. [2023]. This finding suggests multiple avenues for future research.
Figure 7:  $\beta_{\sigma}$  correlates with SGD training speed. Training loss (y-axis) vs epoch (x-axis) on a training dataset of CIFAR10. Each curve shows the MLP with corresponding activation, with color codes denoting isometry strength  $\beta_{\sigma}$ . Hyper parameters: depth: 10, width: 1000, batch size: 512, Average of #5 independent runs.
In this study, we explored the influence of layer normalization and nonlinear activation functions on the isometry of MLP representations. Our findings open up several avenues for future research.
Self normalized activations. It is worth investigating whether we can impose isometry without layer normalization. Our empirical observations suggest that certain activations, such as ReLU, require layer normalization to attain isometry. In contrast, other activations, which can be considered as "self-normalizing" (e.g., SeLU [Klambauer et al., 2017] and hyperbolic tangent), can achieve isometry with only offset and scale adjustments (see Figure 8). We experimentally show how we can replace centering and normalization by leveraging Hermit expansion of activation. Thus, we believe Hermit expansion provides a theoretical grounding to analyze the isometry of SeLU.
(b)
Figure 8: (a) Exploring the potential of achieving isometry with self-normalized activations. Batch size  $n = 10$ , width  $d = 1000$ . (b) Impact of the order of activation and normalization layers. Batch size  $n = 10$ , width  $d = 1000$ .
Impact of the ordering of normalization and activation layers on isometry. Theorem 4 highlights that the ordering of activation and normalization layers has a critical impact on the isometry. Figure 8 demonstrates that a different ordering can lead to a non-isotropic Gram matrix. Remarkably, the structure analyzed in this paper is used in transformers [Vaswani et al., 2017].
Normalization's role in stabilizing mean-field accuracy through depth. Numerous theoretical studies conjecture that mean-field predictions may not be reliable for considerably deep neural networks [Li et al., 2021, Joudaki et al., 2023]. Mean-field analysis incurs a  $O(1 / \sqrt{\mathrm{width}})$  error per layer when the network width is finite. This error may accumulate with depth, making mean-field predictions increasingly inaccurate with an increase in depth. However, Figure 9 illustrates that layer normalization controls this error accumulation through depth. This might be attributable to the isometry bias induced by normalization, as proven in Theorem 1. Similarly, batch normalization also prevents error propagation with depth by imposing the same isometry [Joudaki et al., 2023]. This observation calls for future research on the essential role normalization plays in ensuring the accuracy of mean-field predictions.
Amir Joudaki is funded through Swiss National Science Foundation Project Grant #200550 to Andre Kahles, and partially funded by ETH Core funding award to Gunnar Ratsch. Hadi Daneshmand acknowledges support from the NSF TRIPODS program (DMS-2022448).
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
Figure 9: Role of normalization in stabilizing mean-field accuracy for ReLU, batch size  $n = 10$ .
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448-456. pmlr, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
Hadi Daneshmand, Amir Joudaki, and Francis Bach. Batch normalization orthogonalizes representations in deep random networks. Advances in Neural Information Processing Systems, 34: 4896-4906, 2021.
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel S Schoenholz. A mean field theory of batch normalization. arXiv preprint arXiv:1902.08129, 2019.
Jonas Kohler, Hadi Daneshmand, Aurelien Lucchi, Thomas Hofmann, Ming Zhou, and Klaus Neymeyr. Exponential convergence rates for batch normalization: The power of length-direction decoupling in non-convex optimization. In *The 22nd International Conference on Artificial Intelligence and Statistics*, pages 806–815. PMLR, 2019.
Hadi Daneshmand, Jason D Lee, and Chi Jin. Efficient displacement convex optimization with particle gradient descent. International Conference on Machine Learning, 2023.
Amir Joudaki, Hadi Daneshmand, and Francis Bach. On bridging the gap between mean field and finite width in deep random neural networks with batch normalization. International Conference on Machine Learning, 2023.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. The emergence of spectral universality in deep networks. In International Conference on Artificial Intelligence and Statistics, pages 1924-1932, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurelien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems, 35:27198-27211, 2022a.
Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations, 2018.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). URL http://www.cs.toronto.edu/~kriz/cifar.html.
Alexandru Meterez, Amir Joudaki, Francesco Orabona, Alexander Immer, Gunnar Ratsch, and Hadi Daneshmand. Towards training without depth limits: Batch normalization without gradient explosion. arXiv preprint arXiv:2310.02012, 2023.
Mufan Li, Mihai Nica, and Dan Roy. The future is log-gaussian: Resnets and their infinite-depth-and-width limit at initialization. Advances in Neural Information Processing Systems, 34:7852-7864, 2021.
Semyon Aranovich Gershgorin. Über die abgrenzung der eigenwerte einer matrix. News of the Russian Academy of Sciences. Mathematical series, (6):749-754, 1931.
F Gustav Mehler. Ueber die Entwicklung einer function von beliebig vielen variablen nach lapliceschen functionen higherer ordnung. Journal für die Reine und Angewandte Mathematik (in German), 1866.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierrick Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, Jamie Brew, and Guillaume Dulac-Arnold. Transformers: State-of-the-art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, October 2020. URL https://www.aclweb.org/anthology/2020.emnlp-demos.6.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
Appendix outline
The appendix is partitioned into four main components, each serving its purpose as described:
1. Section A details the all the proofs, with most of it dedicated to proof of Theorem 4 alongside numerical confirmation of significant steps
- Section A.1 elaborates on the basic properties of isometry.
- Section A.2 provides an elaborate review of the mean-field Gram dynamics.
- Section A.3 presents the Lyapunov function  $\gamma$ , and establishes that this function provides both upper and lower bounds for the isometry, thereby implying that geometric contraction of  $\gamma(G^{\ell})$  indicates a geometric contraction of isometry gap -  $\log \mathcal{I}(G^{\ell})$ .
- Section A.4 proves that  $\gamma (G^{\ell})$  exhibits an exponential contraction in depth with rate  $\beta_{\sigma}$ .
2. Section B outlines our rebuttal responses to the reviews that we chose to leave out of the main text.
- Section B.1 gives additional details concerning the experiments reported in the main text and appendix.
- Section B.2 explores the effect of gain on isometry, and links the rate to the associated isometry strength.
- Section B.3 explores the effect of varying widths of hidden layers on the isometry.
- Section B.4 explores the notion of isometry for representations in language models.
A.1 Basic properties of isometry
Basic properties of isometry It is straightforward to check isometry obeys the following basic isometry-preserving properties:
Lemma A.1. For PSD matrix  $M$ , the isometry defined in (3) obeys the following properties: 1) scale-invariance  $\mathcal{I}(cM) = \mathcal{I}(M)$ , 2) only takes value in the unit range  $\mathcal{I}(M) \in [0,1]$  3) it takes its maximum value if and only if  $M$  is identity  $\mathcal{I}(M) = 1 \iff M = I_n$ , and 3) takes minimum value if and only if  $M$  is degenerate  $\mathcal{I}(M) = 0$ .
Proof of Lemma A.1. The scale-invariance is trivially true as scaling  $M$  by any constant will scale  $\operatorname{det}(M)^{1 / n}$  and  $\operatorname{tr}(M)$  by the same amount. The proof of other properties is a straightforward consequence of writing the isometry in terms of the eigenvalues  $\mathcal{I}(M) = (\prod_{i}\lambda_{i})^{1 / n} / (\frac{1}{n}\sum_{i}\lambda_{i})$ , where  $\lambda_{i}$ 's are eigenvalues of  $M$ . By arithmetic vs geometric mean inequality over the eigenvalues we have  $(\prod_{i}\lambda_{i})^{1 / n}\leq \frac{1}{n}\sum_{i}\lambda_{i})$ , which proves that  $\mathcal{I}(M)\in [0,1]$ . Furthermore, the inequality is tight iff the values are all equal  $\lambda_1 = \dots = \lambda_n$ , which holds only for identity  $M = I_{n}$ . Finally, isometry is zero iff at least one eigenvalue is zero, which is the case for degenerate matrix  $M$ .
A.2 Mean-field Gram Dynamics
Recall the mean-field Gram dynamics stated in equation (10):
$$
G _ {*} ^ {\ell + 1} = \mathbb {E} _ {h \sim N \left(0, G _ {*} ^ {\ell}\right)} \left[ \phi (\sigma (h)) \phi (\sigma (h)) ^ {\top} \right], \quad \text {w h e r e} [ \phi (a) ] _ {i} := \left(a _ {i} - \mathbb {E} a _ {i}\right) / \sqrt {\operatorname {V a r} a _ {i}}, \tag {13}
$$
Assuming that inputs are encoded as columns of  $X$ , we can restate the MLP dynamics as follows
$$
X ^ {0} := \frac {1}{\sqrt {d}} \mathrm {L N} (X - \bar {X}), \quad \text {i n p u t s} \tag {14}
$$
$$
H ^ {\ell} := W ^ {\ell} X ^ {\ell}, W ^ {\ell} \sim N (0, 1) ^ {d \times d}, \quad \text {p r e a c t i v a t i o n} \tag {15}
$$
$$
A ^ {\ell} := \sigma \left(H ^ {\ell}\right), \quad \text {a c t i v a t i o n s} \tag {16}
$$
$$
X ^ {\ell + 1} = \frac {1}{\sqrt {d}} \mathrm {L N} \left(A ^ {\ell} - \overline {{A ^ {\ell}}}\right) \quad \text {c e n t e r i n g} \tag {17}
$$
where centering and layer normalization are applied column-wise, as defined in the main text. Observe that Gram matrix of representations can be written as
$$
\begin{array}{l} G _ {d} ^ {\ell + 1} = X ^ {\ell + 1} ^ {\top} X ^ {\ell + 1} (18) \\ = \frac {1}{d} \sum_ {k = 1} ^ {d} \left(\frac {A _ {k 1} ^ {\ell} - \mu_ {1}}{s _ {1}}, \dots , \frac {A _ {k n} ^ {\ell} - \mu_ {n}}{s _ {n}}\right) ^ {\otimes 2}, \quad \mu_ {i} := \frac {1}{d} \sum_ {k = 1} ^ {d} \left(A _ {k i} ^ {\ell}\right), s _ {i} := \sqrt {\frac {1}{d} \sum_ {k = 1} ^ {d} \left(A _ {k i} ^ {\ell} - \mu_ {i}\right) ^ {2}}, (19) \\ \end{array}
$$
where  $\otimes$  denotes Hadamard product, and subscript  $^d$  emphasises the dependence of Gram on width  $d$ . Note that conditioned on the previous layer, rows of  $H^{\ell}$  and  $A^\ell$  are i.i.d., because of independence of rows of  $W^{\ell}$ . Thus, by law of large numbers, in the infinitely wide network regime,  $\mu_{i}$  and  $s_i$  will converge to the expected mean and variance respectively  $\lim_{d\to \infty}\mu_i = \mathbb{E}A_{1i}^\ell$ , and  $\lim_{d\to \infty}s_i = \sqrt{\mathrm{Var}(A_{1i}^\ell)}$ , for all  $i = 1,\dots ,n$ . By construction of  $\phi$ , in the infinitely wide regime, we can rewrite Gram dynamics as  $\lim_{d\to \infty}G_d^\ell = \frac{1}{d}\sum_{k = 1}^{d}\phi (A_i^\ell)^\top \phi (A_i^\ell)$ . We can invoke the fact that rows of  $A^\ell$  are i.i.d. to conclude that  $G_{d}$  is the sample Gram matrix that converges to its expectation
$$
\lim  _ {d \rightarrow \infty} G _ {d} ^ {\ell} = \mathbb {E} \phi \left(A _ {1.} ^ {\ell}\right) \phi \left(A _ {1.} ^ {\ell}\right) ^ {\top} = \mathbb {E} _ {h \sim N \left(0, G ^ {\ell}\right)} \phi (\sigma (h)) \phi (\sigma (h)) ^ {\top} =: G _ {*} ^ {\ell}, \tag {20}
$$
where  $*$  denotes the mean-field regime  $d\to \infty$ . This concludes the connection between the mean-field Gram dynamics and infinitely wide Gram dynamics.
A.3 Introducing a potential
Here we will introduce a Lyapunov function that enables us to precisely quantify the isometry of activations in deep networks:
Definition 4. Given a positive semidefinite matrix  $G \in \mathbb{R}^{n \times n}$ , we define  $\gamma: \mathbb{R}^{n \times n} \to \mathbb{R}^{\geq 0}$  as:
$$
\gamma (G) := \max  _ {i \neq j} \frac {\left| \widetilde {G} _ {i j} \right|}{1 - \left| \widetilde {G} _ {i j} \right|}, \quad \widetilde {G} _ {i j} := G _ {i j} / \sqrt {G _ {i i} G _ {j j}}. \tag {21}
$$
Remarkably,  $\gamma$  obey exhibits an geometric contraction under one MLP layer update, which is stated in the following theorem:
Theorem A.2. Let  $G$  be PSD matrix with unit diagonals  $G_{ii} = 1, i = 1, \ldots, n$ . It holds:
$$
\gamma \left(\mathbb {E} _ {h \sim N (0, G)} \left[ \phi (\sigma (h)) \phi (\sigma (h)) ^ {\top} \right]\right) \leq \gamma (G) / \beta_ {\sigma}, \quad \text {w h e r e} [ \phi (a) ] _ {i} := \left(a _ {i} - \mathbb {E} a _ {i}\right) / \sqrt {\operatorname {V a r} a _ {i}}. \tag {22}
$$
Thus, we may apply Theorem A.2 iteratively to prove that in the mean-field, the Lyapunov function  $\gamma (G^{\ell})$  decays at an exponential rate  $\beta_{\sigma}$ . A straightforward induction over layers leads to a decay rate in  $\gamma$ , which is presented in the next corollary.
Corollary A.3. If activation  $\sigma$  has Hermite expansion with non-linearity strength  $\beta_{\sigma}$ , the mean-field Gram matrices  $G_{*}^{\ell}$ , obey Lyapunov of these Gram matrices decays at an exponential rate  $\beta_{\sigma}$ :
$$
\gamma \left(G _ {*} ^ {\ell}\right) \leq \gamma \left(G _ {*} ^ {0}\right) \beta_ {\sigma} ^ {- \ell}, \tag {23}
$$
where  $G_{*}^{0}$  denotes the input Gram matrix.
In Figure A.1, we experimentally validated the above equation for MLPs with a finite width and activations  $\{\tanh ,\mathrm{relu}$  sigmoid} where we observe that the mean-field analysis well predicates the decay in  $\gamma$
Figure A.1:  $\gamma (G^{\ell})$  vs depth  $\ell$ , for MLP with  $n = 10$ ,  $d = 10000$ , various activations  $\sigma$ . Solid lines show average of 10 independent runs. The dashed traces the theoretical upper bounds given in Corollary A.3.
Interestingly, we can connect the Lyapunov  $\gamma$  to the isometry, by proving an upper and lower based on the determinant  $G$  based on  $\gamma (G)$ , when  $G$  is PSD and has unit diagonals:
Lemma A.4. For PSD matrix  $G$  with unit diagonals holds:
$$
\left(1 - (n - 1) \max  _ {i \neq j} \left| G _ {i j} \right|\right) ^ {n} \leq \det  (G) \leq 1 - \max  _ {i \neq j} G _ {i j} ^ {2}, \tag {24}
$$
where the lower bound holds if  $(n - 1)|G_{ij}|\leq 1$
Now, we are ready to prove the main theorem.
Theorem A.5 (Restated Theorem 4). Let  $\sigma$  be an activation function with a Hermite expansion and a nonlinearity strength  $\beta_{\sigma}$ , (see equation (11)). Given non-degenerate input Gram matrix  $G_{*}^{0}$ , then for sufficiently large layer  $\ell \gtrsim \beta_{\sigma}^{-1}(-n\log \mathcal{I}(G_{*}^{0}) + \log (4n))$ , we have
$$
- \log \mathcal {I} \left(G _ {*} ^ {\ell}\right) \leq \exp (- \ell \log \beta_ {\sigma} - n \log \mathcal {I} \left(G _ {*} ^ {0}\right) + \log (4 n)). \tag {25}
$$
Proof of Theorem 4. First, note that we have the following transformation