SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
t = \frac {x}{1 - x} \Rightarrow x = \frac {t}{t + 1} \Rightarrow \max  _ {i \neq j} | G _ {i j} | = \frac {\gamma (G)}{1 + \gamma (G)}, \tag {26}
$$
$$
\det  \left(G _ {*} ^ {\ell}\right) \geq (1 - (n - 1) \max  _ {i \neq j} \left| G _ {i j _ {*}} ^ {\ell} \right|\left. \right) ^ {n} \tag {27}
$$
$$
\det  \left(G _ {*} ^ {\ell}\right) = \left(1 - (n - 1) \left(\gamma \left(G _ {*} ^ {\ell}\right) / \left(1 + \gamma \left(G _ {*} ^ {\ell}\right)\right)\right)\right) ^ {n} \quad \text {I n v o k e} \tag {28}
$$
$$
\geq \left(1 - (n - 1) \gamma \left(G _ {*} ^ {\ell}\right)\right) ^ {n} \tag {29}
$$
$$
\Longrightarrow - \frac {1}{n} \log \det  \left(G _ {*} ^ {\ell}\right) \leq - \log (1 - (n - 1) \gamma \left(G _ {*} ^ {\ell}\right)) \tag {30}
$$
$$
\leq 2 n \gamma \left(G _ {*} ^ {\ell}\right) \quad \text {f o r} \gamma \left(G _ {*} ^ {\ell}\right) <   1 / (2 n - 2) \tag {31}
$$
$$
\leq 2 n \gamma \left(G _ {0}\right) \beta_ {\sigma} ^ {- \ell}. \tag {32}
$$
By upper bound of Lemma A.4 we have  $\max_{i\neq j}|G_{ij}|\leq \sqrt{1 - \operatorname*{det}(G)}\leq 1 - \operatorname*{det}(G) / 2$ , implying  $\gamma (G_0)\leq 2\operatorname *{det}(G_0)^{-1}$ , which allows us to
$$
- \log \mathcal {I} \left(G _ {*} ^ {\ell}\right) \leq 4 n \det  \left(G _ {0}\right) ^ {- 1} \beta_ {\sigma} ^ {- \ell} \tag {33}
$$
$$
\leq \exp (- \ell \log \beta_ {\sigma} - n \log \mathcal {I} (G _ {0}) + \log (4 n)) \tag {34}
$$
Proof of Lemma A.4. Lower bound. The lower bound is a result of Gershgorin circle theorem [Gershgorin, 1931], which implies that every eigenvalue must be within the disc  $[1 - (n - 1)\max_{i\neq j}|G_{ij}|,1 + (n - 1)\max_{i\neq j}|G_{ij}|]$ . Thus, the determinant is lower-bounded by  $(1 - (n - 1)\max_{i\neq j}|G_{ij}|)^{n}$ .
Upper bound. Since  $G$  is PSD, we can write  $G = X^{\top}X$ , which implies that columns of  $X$  are unit norm  $\| x_{i}\| = 1$ , and  $G$  encodes the angles between them  $\cos \angle (x_i,x_j) = \langle x_i,x_j\rangle = G_{ij}$ . Furthermore, we have  $\operatorname*{det}(G) = \operatorname {vol}(X)^2$ , where the volume refers to the parallelepiped spanned by the columns of  $X$ . With this formulation, we write volume recursively as volume spanned by columns 1 up to  $n - 1$ , times the projection distance of  $x_{n}$  from their span
$$
\operatorname {v o l} \left(x _ {1}, \dots , x _ {n}\right) = \operatorname {d i s t} \left(x _ {n}, \operatorname {s p a n} \left(x _ {1}, \dots , x _ {n - 1}\right)\right) \operatorname {v o l} \left(x _ {1}, \dots , x _ {n - 1}\right),
$$
where span refers to the space of all linear combinations of these vectors. Note that projection distance of  $x_{i}$  onto  $x_{j}$  can be written as  $\sin \angle (x_i,x_j) = \sqrt{1 - \langle x_i,x_j\rangle^2} = \sqrt{1 - G_{ij}^2}$ . Since the projection distance onto the linear span cannot be greater than projection distance onto a single vector, which is bounded by  $\sqrt{1 - \max_{i\neq j}G_{ij}^2}$ . This concludes the upper bound that  $\operatorname*{det}(G)\leq 1 - \max_{i\neq j}G_{ij}^2$ .
A.4 Dual activation and proof of Thm. A.2
According to Daniely et al. [2016], we leverage the notion dual activation associated with the activation function  $\sigma$ , which is defined in following.
Definition 5. Given activation  $\sigma : \mathbb{R} \to \mathbb{R}$  that is square integrable with respect to the Gaussian kernel, define its dual activation  $\hat{\sigma} : \mathbb{R} \to \mathbb{R}$ , and its mean reduced dual  $\bar{\sigma} : \mathbb{R} \to \mathbb{R}$  as:
$$
\hat {\sigma} (\rho) := \mathbb {E} \sigma (x) \sigma (y) \quad \bar {\sigma} (\rho) := \mathbb {E} \left[ (\sigma (x) - \mathbb {E} \sigma (x)) (\sigma (y) - \mathbb {E} \sigma (y)) \right] \tag {35}
$$
The following lemma connects the dual activation and its mean-reduced version with the Hermite expansion of  $\sigma$ :
Lemma A.6. Given two standard Gaussian variables  $X, Y \sim N(0,1)$  with covariance  $\mathbb{E}XY = \rho$ , and activation  $\sigma$  with normalized Hermite coefficients  $c_{kk}$ , we have
$$
\hat {\sigma} (\rho) = \sum_ {k = 0} ^ {\infty} c _ {k} ^ {2} \rho^ {k}, \quad \bar {\sigma} (\rho) = \sum_ {k = 1} ^ {\infty} c _ {k} ^ {2} \rho^ {k}. \tag {36}
$$
Finally, we have the tools to prove the first main theorem.
Proof of Theorem A.2. Let  $a = \sigma(h)$  for  $h \sim N(0, G)$ . Note that by definition of the dual-activation we have  $\mathbb{E}aa^{\top} = [\hat{\sigma}(G_{ij})]_{i,j \leq n}$ . Since we assumed  $G$  has unit diagonals, we have  $h_i \sim N(0, 1)$ , which implies that  $\mathbb{E}a_i = \mathbb{E}_{h_i \sim N(0, 1)}\sigma(h_i) = c_0$ , implying that  $\mathbb{E}(a_i - \mathbb{E}a_i)(a_j - \mathbb{E}a_j)^{\top} = \hat{\sigma}(G_{ij}) - c_0^2 = \bar{\sigma}(G_{ij})$ . Furthermore, the variance can be driven as  $\mathrm{Var}(a_i) = \mathbb{E}a_i^2 - (\mathbb{E}a_i)^2 = \sum_{k=0}^{\infty}c_k^2 = \sum_{k=1}^{\infty}c_k^2 = \bar{\sigma}(1)$  for all  $i = 1, \ldots, n$ . Thus, we have  $\mathbb{E}\phi(a_i)\phi(a_j) = \bar{\sigma}(G_{ij}) / \bar{\sigma}(1)$ . In the matrix form we have
$$
\gamma \left(\mathbb {E} _ {h \sim N (0, G)} \left[ \phi (\sigma (h)) \phi (\sigma (h)) ^ {\top} \right]\right) = \gamma \left([ \bar {\sigma} (G _ {i j}) / \bar {\sigma} (1) ] _ {i, j \leq n}\right) \tag {37}
$$
The remainder proof relies on the following contractive property of Gram matrix potential:
Lemma A.7. Consider activation  $\sigma$ , with normalized Hermite coefficients  $\{c_k\}_{k\geq 0}$ . For all  $\rho \in (0,1)$ , the mean-reduced dual activation  $\bar{\sigma}$  obeys
$$
\frac {\left| \bar {\sigma} (\rho) \right| / \bar {\sigma} (1)}{1 - \left| \bar {\sigma} (\rho) \right| / \bar {\sigma} (1)} \leq \beta_ {\sigma} ^ {- 1} \frac {| \rho |}{1 - | \rho |}, \tag {38}
$$
which the right hand-side is strictly larger if some nonlinear coefficient is nonzero  $c_k \neq 0$  for some  $k \geq 2$ .
Thus we can apply Lemma A.7 on each element  $i \neq j$  to conclude that
$$
\begin{array}{l} \frac {\left| \bar {\sigma} \left(G _ {i j}\right) \right| / \bar {\sigma} (1)}{1 - \left| \bar {\sigma} \left(G _ {i j}\right) \right| / \bar {\sigma} (1)} \leq \frac {\left| G _ {i j} \right|}{1 - \left| G _ {i j} \right|} \beta_ {\sigma} ^ {- 1} (39) \\ \leq \beta_ {\sigma} ^ {- 1} \max  _ {i \neq j} \frac {\left| G _ {i j} \right|}{1 - \left| G _ {i j} \right|} (40) \\ = \beta_ {\sigma} ^ {- 1} \gamma (G). (42) \\ \end{array}
$$
since the inequality holds for any value of  $i \neq j$ , we can take the maximum over  $i \neq jj$  to write:
$$
\gamma (\bar {\sigma} (G) / \bar {\sigma} (1)) = \max  _ {i \neq j} \frac {| \bar {\sigma} (G _ {i j}) | / \bar {\sigma} (1)}{1 - | \bar {\sigma} (G _ {i j}) | / \bar {\sigma} (1)} \leq \beta_ {\sigma} ^ {- 1} \gamma (G), \tag {43}
$$
Proof of Lemma A.7. Note the ratio is invariant to scaling of  $\sum_{k=1}^{\infty} c_k^2$ . Hence, we assume  $\sum_{k=1}^{\infty} c_k^2 = 1$  without loss of generality. With this simplification, we have  $\bar{\sigma}(1) = 1$ . For the positive range  $\rho \in [0,1]$  we have
$$
\begin{array}{l} \left(\frac {\bar {\sigma} (\rho)}{1 - \bar {\sigma} (\rho)}\right) \left(\frac {\rho}{1 - \rho}\right) ^ {- 1} = \frac {\rho^ {- 1} \sum_ {k = 1} ^ {\infty} c _ {k} ^ {2} \rho^ {k}}{(1 - \rho) ^ {- 1} \sum_ {k = 1} ^ {\infty} c _ {k} ^ {2} (1 - \rho^ {k})} (44) \\ = \frac {\sum_ {k = 1} ^ {\infty} c _ {k} ^ {2} \rho^ {k - 1}}{\sum_ {k = 1} ^ {\infty} c _ {k} ^ {2} \left(\frac {1 - \rho^ {k}}{1 - \rho}\right)} (45) \\ = \frac {\sum_ {k = 1} ^ {\infty} c _ {k} ^ {2} \rho^ {k - 1}}{\sum_ {k = 1} ^ {\infty} c _ {k} ^ {2} \left(\sum_ {i = 0} ^ {k - 1} \rho^ {i}\right)} (46) \\ \leq \frac {\sum_ {k = 1} ^ {\infty} c _ {k} ^ {2} \rho^ {k - 1}}{\left(\sum_ {k = 1} ^ {\infty} c _ {k} ^ {2} \rho^ {k - 1}\right) + \sum_ {k = 2} ^ {\infty} c _ {k} ^ {2}} (47) \\ \leq \max  _ {\rho \in [ 0, 1 ]} \frac {\sum_ {k = 1} ^ {\infty} c _ {k} ^ {2} \rho^ {k - 1}}{\left(\sum_ {k = 1} ^ {\infty} c _ {k} ^ {2} \rho^ {k - 1}\right) + \sum_ {k = 2} ^ {\infty} c _ {k} ^ {2}} (48) \\ = \frac {\sum_ {k = 1} ^ {\infty} c _ {k} ^ {2}}{\sum_ {k = 1} ^ {\infty} c _ {k} ^ {2} + \sum_ {k = 1} ^ {\infty} c _ {k} ^ {2} - c _ {1} ^ {2}} (49) \\ = \frac {1}{\beta_ {\sigma}}. (50) \\ \end{array}
$$
Thus for  $\rho \in [0,1]$  we have
$$
\frac {\bar {\sigma} (\rho)}{1 - \bar {\sigma} (\rho)} \leq \frac {\rho}{1 - \rho} \beta_ {\sigma} ^ {- 1}. \tag {51}
$$
By Jensen inequality for convex function  $x\mapsto |x|$  we have
$$
| \bar {\sigma} (\rho) | = \left| \sum_ {k = 1} ^ {\infty} c _ {k} ^ {2} \rho^ {k} \right| \leq \sum_ {k = 1} ^ {\infty} c _ {k} ^ {2} | \rho | ^ {k} = \bar {\sigma} (| \rho |) \tag {52}
$$
Because  $x \mapsto x / (1 - x)$  is monotonically increasing for  $x \in [0, 1]$ , we have
$$
\frac {\left| \bar {\sigma} (\rho) \right|}{1 - \left| \bar {\sigma} (\rho) \right|} \leq \frac {\bar {\sigma} \left(\left| \rho \right|\right)}{1 - \bar {\sigma} \left(\left| \rho \right|\right)} \leq \frac {\left| \rho \right|}{1 - \left| \rho \right|} \beta_ {\sigma} ^ {- 1}. \tag {53}
$$
Where we invoked the inequality that was proven for  $\rho \in [0,1]$ , because  $|\rho |\in [0,1]$ .
The following lemma, which is a consequence of Mehler's formula, is at the hart of proof of Lemma A.6:
Lemma A.8 (Consequence of Mehler's kernel). If  $X, Y \sim N(0,1)$  with covariance  $EXY = \rho$  we have
$$
E _ {X, Y} H e _ {n} (X) H e _ {k} (Y) = \rho^ {n} \delta_ {n k}
$$
where  $\delta_{nk}$  is the Dirac delta.
Proof of Lemma A.6. Let  $\sigma(x) = \sum_{k=0} c_k H e_k(x)$ , denote the Hermite expansion of  $\sigma$ . Thus, we have
$$
\begin{array}{l} \hat {\sigma} (\rho) := \mathbb {E} _ {X, Y} \sigma (X) \sigma (Y) (54) \\ = \sum_ {n, k = 0} ^ {\infty} c _ {n} c _ {k} \mathbb {E} _ {X, Y} H e _ {n} (X) H e _ {k} (Y) (55) \\ = \sum_ {k = 0} ^ {\infty} c _ {k} ^ {2} \rho^ {k}, (56) \\ \end{array}
$$
where in the last line we applied result of Lemma A.8. For the mean-reduced dual kernel  $\bar{\sigma}$ , observe that  $\mathbb{E}_{X\sim N(0,1)}\sigma (X) = c_0$ . Thus, the reduction of mean will cancel the  $k = 0$  term, which concludes the proof that  $\bar{\sigma} (\rho) = \sum_{k = 1}c_k^2\rho^k$ .
Proof of Lemma A.8. The property can be deduced from Mehler's formula Mehler [1866]. The formula states that
$$
\begin{array}{l} \frac {1}{\sqrt {1 - \rho^ {2}}} \exp \left(- \frac {\rho^ {2} \left(x ^ {2} + y ^ {2}\right) - 2 x y \rho}{2 \left(1 - \rho^ {2}\right)}\right) (57) \\ = \sum_ {m = 0} ^ {\infty} H e _ {m} (x) H e _ {m} (y), (58) \\ \end{array}
$$
where the  $m!$  factor difference is due to the definition of Hermite polynomials with an additional  $1 / \sqrt{m!}$  compared to the one used in Mehler's kernel. Observe that the left hand side is equal to  $p(x,y) / p(x)p(y)$ , where  $p(x,y)$  is the joint PDF of  $(X,Y)$ , and  $p(x),p(y)$  are PDF of  $X$  and  $Y$  respectively. Therefore, we can take the expectation using the expansion
$$
\begin{array}{l} \mathbb {E} _ {X, Y} \left[ H e _ {n} (X) H e _ {k} (Y) \right] = \int H e _ {n} (x) H e _ {k} (y) p (x, y) d x d y (59) \\ = \sum_ {m = 0} ^ {\infty} \rho^ {m} \int H e _ {n} (x) H e _ {k} (y) H e _ {m} (x) H e _ {m} (y) d p (x) d p (y) (60) \\ = \sum_ {m = 0} ^ {\infty} \rho^ {m} \mathbb {E} _ {X \sim N (0, 1)} [ H e _ {n} (X) H e _ {m} (X) ] \mathbb {E} _ {Y \sim N (0, 1)} [ H e _ {k} (Y) H e _ {m} (Y) ] (61) \\ = \rho^ {n} \delta_ {n k} (62) \\ \end{array}
$$
where in the last line we used the orthogonality property  $E_{X\sim N(0,1)}H_k(x)H_n(X) = \delta_{nk}$ .
B.1 Details about empirical validations
Hardware. Experiments that did not require training a model were run on a AMD Ryzen 9 3900X 12-Core CPU, which takes about 5 minutes overall. All experiments that required training a model were trained a single NVIDIA GeForce RTX 3090 GPU, which takes about 1 minutes per each training task.
Figures. The solid lines in all plots represent the average performance over multiple independent runs, and the shaded regions indicate the confidence intervals. Unless stated otherwise, each average is computed over #10 independent runs.
Codes and reproducibility. We implemented our experiments in Python using the PyTorch framework Paszke et al. [2019]. All the figures are reproducible with the code attached in the supplementary.
Training procedure For all training-related experiments, the isometry or isometry gap are computed per each batch by sampling a few of the batches randomly, and then averaged over. The epoch  $i$  corresponds to the network at after  $i$  steps of training on the training set of CIFAR10 (epoch 0 means network is at initialization).
Pre-trained large language models The pre-trained language models and their default configuration was downloaded from Huggingface Wolf et al. [2020] library.
B.2 Quantifying the influence of gain on isometry through non-linearity strength
The concept of gain in neural networks is vital and closely connected with the weights initialization. A neural network with properly initialized weights can learn faster, have a lesser chance of getting stuck at sub-optimal solutions, and provide better generalization. The impact of gain can be visualized through the lens of weight initialization strategies such as Xavier normalization [Glorot and Bengio, 2010], which has shown significant effectiveness in optimizing neural networks. These initialization strategies apply a gain value to the weights, which is a scaling factor, to ensure a good signal flow through many layers during the forward and backward passes. The gain value essentially determines the variance of the weights in the initialization stage.
As an extension to our prior investigations, we delve into understanding the influence of gain on isometry, predominantly through our calculated metric, the non-linearity strength, denoted as  $\beta_{\sigma}$ , as a function of gain  $\alpha$ . For certain instances, such as ReLU, sine, and exponential activations, we are capable of deriving  $\beta_{\sigma}(\alpha)$  in a closed form. Table B.1 presents a few of these cases.
σ	exp(αx)	sin(αx)	max(αx,0)
βσ	2-2eα2+α2/1-eα2	2(-1+e2α2-eα2α2)/-1+e2α2	4-3π/2-2π
Table B.1: Relationship between non-linearity strength and gain.
For a more extensive selection of activation functions, we have numerically computed the non-linearity strength as a function of gain  $\alpha$ , as visualized in Figure B.1. Leveraging Theorem 4, these computed values provide an estimation for the isometry strength for various activations. This correlation proves to be remarkably predictive, as shown in Figure B.2. Remarkably, in the case of ReLU activation, its unique characteristics lead to both our closed form  $\beta_{\sigma}$  (refer Table B.1) and rate towards isometry (refer Figure B.2) remaining consistent across different values of  $\alpha$ .
Figure B.1: Analysis of gain through the lens of isometry strength.
Comparison to Xavier gain for initialization Inspired by the results so far, we can compare mean-field centering and normalization to Xavier gain for activations. Figure B.3 demonstrates that all mean-field based gains improve the isometry when compared with Xavier initialization. However, this is markedly stronger for ReLU and leaky ReLU. We can explain this starker contrast by the fact that both activations have a significant offset term  $c_{0}$ , which is not corrected by the Xavier initialization.
B.3 Varying width of hidden layers
While in our theoretical setup, we assume the network width is constant across the layers, this is only a choice to streamline our proof and notation. Since our primary result is derived from the mean-field regime, the
Figure B.2: Influence of gain on the attainment of isometry.
Figure B.3: Isometry vs depth for mean-field centering and normalization to Xavier initialization.
only criterion for it to hold is for the width to be sufficiently large to approximate the mean-field regime. Our experiments in Figure B.4 substantiate this claim that the specific sizes of hidden layers, as long as they are large, will not impact our main results on the isometry. We empirically validate this for four different configurations and show that the decay of the isometry gap remains largely consistent across these configurations.
Figure B.4: Theorem 4 holds for variable width MLP. Isometry gap of various layers for MLP with ReLU activation and fixed width (blue), growing width (orange), narrowing width (green), and interchanging width (red) on MNIST (left) and CIFAR10 (right) training data. The weights of MLP are random. These plots show that our central theory holds regardless of the dataset or shape of MLP widths.
B.4 Isometry in pre-trained large language models
Since our theory for normalization is not limited to initialization, we can expand our search for isometry to other architectures. Figure B.5 shows the important role of normalization in the pre-trained GPT2 network. However, we need to adjust the notion of isometry with the architecture of layer norm in a transformer in mind. In fact, the mean and standard deviation are computed over features separately for each token. Thus, to adapt the notion of isometry, we can view each token as a sample and define Gram over different tokens. Thus, isometry here quantifies the similarity between various tokens within one sample. As can be seen in the figure below, LayerNorm layers (shaded in red) in the last six layers of the pre-trained GPT2 increase the isometry between tokens, which is consistent with our theory of layer normalization. It is crucial that our theory holds deterministically, which extends to the pre-trained model.
One caveat in interpreting our results is that, in practice, LayerNorm layers have learnable parameters that make them deviate from our theory. It would be fruitful to study the effects of learned parameters to discern it from the role of centering and normalization for a future study.
Figure B.5: Validation of corollary 2 in GPT2. The last six transformer layers of pre-trained GPT2, LayerNorm layers (shaded; red) decrease the isometry gap.
B.5 Tracking isometry at initialization and optimization for more activations
Here there are more numerical experiments related to to tracking isometry before and after training.
Figure B.6: Validation of Corollary 2 and Theorem 4 for multiple activations.
The following plot shows that isometry gap remains relatively stable.
Figure B.7: Isometry gap change during training is relatively small.