SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Assumption B.1 (Gradient independence assumption [105]). (a) We use a different set of weights for backpropagation than those used to compute the network outputs, but sampled i.i.d. from the same distributions. (b) For any loss  $\mathcal{L}$ , the gradient at layer  $l$ ,  $\partial \mathcal{L} / \partial \pmb{x}^{(l)}$ , is independent of  $\pmb{h}^{(l)}$  and  $\pmb{x}^{(l-1)}$ .
Although not strictly accurate, this assumption has been empirically found to hold well [81, 105]. It has been rigorously justified for specific architectures, including vanilla, residual, and convolutional networks [103]. In this study, we applied the assumption to the standard loss function but not to the adversarial loss function. In addition, we regard a network output as a loss and apply Asm B.1 to the network output in Sec. 5.4.
C Setting of residual networks
The mean field theory for residual networks is studied in [105]. Although they employed trainable weights in shortcuts, we employ the untrainable matrix. Formally, the pre- and post-activations in the  $l$ -th layer of a residual network are defined as follows:
$$
\boldsymbol {h} ^ {(l)} := \boldsymbol {W} ^ {(l)} \boldsymbol {x} ^ {(l - 1)} + \boldsymbol {b} ^ {(l)}, \quad \quad \quad \boldsymbol {x} ^ {(l)} := \boldsymbol {x} ^ {(l - 1)} + \boldsymbol {P} ^ {(l)} \phi (\boldsymbol {h} ^ {(l)}), \tag {A16}
$$
where  $P^{(l)} \in \mathbb{R}^{N \times N}$  is an untrained random matrix. Each entry of  $P^{(l)}$  is i.i.d. sampled from a Gaussian  $\mathcal{N}(0,1 / N)$  at initialization. The random matrix  $P^{(l)}$  is introduced for simplifying probabilistic calculations and is applied in accordance with Asm B.1(a). The definition of  $x^{(l)}$ , given in Eq. (A16), is slightly different from the original definition in [105]. Therefore, we will derive the basic probabilistic properties of pre- and post-activation again, based on Eq. (A16). Following a similar approach to [71, 105], the mean squared pre- and post-activation can be calculated as follows:
$$
\mathbb {E} \left[ \left(h _ {i} ^ {(l)}\right) ^ {2} \right] = \sigma_ {w} ^ {2} \mathbb {E} \left[ \left(x _ {i} ^ {(l - 1)}\right) ^ {2} \right] + \sigma_ {b} ^ {2}, \quad \mathbb {E} \left[ \left(x _ {i} ^ {(l)}\right) ^ {2} \right] = \mathbb {E} \left[ \left(x _ {i} ^ {(l - 1)}\right) ^ {2} \right] + \mathbb {E} \left[ \phi \left(h _ {i} ^ {(l)}\right) ^ {2} \right]. \tag {A17}
$$
Additionally, following [81, 105], the mean squared gradient with respect to pre- and post-activation can be calculated as follows:
$$
\mathbb {E} \left[ \left(\frac {\partial \mathcal {L}}{\partial h _ {i} ^ {(l)}}\right) ^ {2} \right] = \sigma_ {w} ^ {2} \mathbb {E} \left[ \phi^ {\prime} \left(h _ {i} ^ {(l)}\right) ^ {2} \right] \mathbb {E} \left[ \left(\frac {\partial \mathcal {L}}{\partial h _ {i} ^ {(l + 1)}}\right) ^ {2} \right], \tag {A18}
$$
$$
\chi^ {(l)} = \left(1 + \sigma_ {w} ^ {2} \mathbb {E} \left[ \phi^ {\prime} \left(h _ {i} ^ {(l + 1)}\right) ^ {2} \right]\right) \chi^ {(l + 1)}. \tag {A19}
$$
We can derive Eq. (A18) under Asm B.1 as follows:
$$
\mathbb {E} \left[ \left(\frac {\partial \mathcal {L}}{\partial h _ {i} ^ {(l)}}\right) ^ {2} \right] = \mathbb {E} \left[ \left(\frac {\partial \mathcal {L}}{\partial \boldsymbol {h} ^ {(l + 1)}} \frac {\partial \boldsymbol {h} ^ {(l + 1)}}{\partial x _ {i} ^ {(l)}} \frac {\partial x _ {i} ^ {(l)}}{\partial h _ {i} ^ {(l)}}\right) ^ {2} \right] \tag {A20}
$$
$$
= \mathbb {E} \left[ \left(\sum_ {j = 1} ^ {N} \frac {\partial \mathcal {L}}{\partial h _ {j} ^ {(l + 1)}} W _ {j i} ^ {(l + 1)}\right) ^ {2} \phi^ {\prime} \left(h _ {i} ^ {(l)}\right) ^ {2} \right] \tag {A21}
$$
$$
= \sigma_ {w} ^ {2} \mathbb {E} \left[ \phi^ {\prime} \left(h _ {i} ^ {(l)}\right) ^ {2} \right] \mathbb {E} \left[ \left(\frac {\partial \mathcal {L}}{\partial h _ {j} ^ {(l + 1)}}\right) ^ {2} \right]. \tag {A22}
$$
We can derive Eq. (A19) under Asm B.1 as follows.
$$
\begin{array}{l} \chi^ {(l)} := \mathbb {E} \left[ \left(\frac {\partial \mathcal {L}}{\partial x _ {i} ^ {(l)}}\right) ^ {2} \right] (A23) \\ = \mathbb {E} \left[ \left(\frac {\partial \mathcal {L}}{\partial x _ {i} ^ {(l + 1)}}\right) ^ {2} \right] + 2 \mathbb {E} \left[ \sum_ {j = 1} ^ {N} \sum_ {k = 1} ^ {N} \frac {\partial \mathcal {L}}{\partial x _ {i} ^ {(l + 1)}} \frac {\partial \mathcal {L}}{\partial x _ {j} ^ {(l + 1)}} P _ {j k} ^ {(l + 1)} \phi^ {\prime} (h _ {k} ^ {(l + 1)}) W _ {k i} ^ {(l + 1)} \right] \\ + \mathbb {E} \left[ \sum_ {j = 1} ^ {N} \sum_ {k = 1} ^ {N} \sum_ {j ^ {\prime} = 1} ^ {N} \sum_ {k ^ {\prime} = 1} ^ {N} \frac {\partial \mathcal {L}}{\partial x _ {j} ^ {(l + 1)}} \frac {\partial \mathcal {L}}{\partial x _ {j ^ {\prime}} ^ {(l + 1)}} P _ {j k} ^ {(l + 1)} P _ {j ^ {\prime} k ^ {\prime}} ^ {(l + 1)} \phi^ {\prime} \left(h _ {k} ^ {(l + 1)}\right) \phi^ {\prime} \left(h _ {k ^ {\prime}} ^ {(l + 1)}\right) W _ {k i} ^ {(l + 1)} W _ {k ^ {\prime} i} ^ {(l + 1)} \right] (A24) \\ = \left(1 + \sigma_ {w} ^ {2} \mathbb {E} \left[ \phi^ {\prime} \left(h _ {k} ^ {(l + 1)}\right) ^ {2} \right]\right) \chi^ {(l + 1)}. (A25) \\ \end{array}
$$
From Eq. (A23) to Eq. (A24), we used the following equation:
$$
\begin{array}{l} \frac {\partial \mathcal {L}}{\partial x _ {i} ^ {(l)}} = \sum_ {j = 1} ^ {N} \frac {\partial \mathcal {L}}{\partial x _ {j} ^ {(l + 1)}} \frac {\partial x _ {j} ^ {(l + 1)}}{\partial x _ {i} ^ {(l)}} (A26) \\ = \sum_ {j = 1} ^ {N} \frac {\partial \mathcal {L}}{\partial x _ {j} ^ {(l + 1)}} \left(\delta_ {i j} + \sum_ {k = 1} ^ {N} P _ {j k} ^ {(l + 1)} \frac {\partial \phi \left(h _ {k} ^ {(l + 1)}\right)}{\partial x _ {i} ^ {(l)}}\right) (A27) \\ = \frac {\partial \mathcal {L}}{\partial x _ {i} ^ {(l + 1)}} + \sum_ {j = 1} ^ {N} \sum_ {k = 1} ^ {N} \frac {\partial \mathcal {L}}{\partial x _ {j} ^ {(l + 1)}} P _ {j k} ^ {(l + 1)} \phi^ {\prime} \left(h _ {k} ^ {(l + 1)}\right) W _ {k i} ^ {(l + 1)}. (A28) \\ \end{array}
$$
The second term of Eq. (A24) is rearranged as follows:
$$
\begin{array}{l} 2 \mathbb {E} \left[ \sum_ {j = 1} ^ {N} \sum_ {k = 1} ^ {N} \frac {\partial \mathcal {L}}{\partial x _ {i} ^ {(l + 1)}} \frac {\partial \mathcal {L}}{\partial x _ {j} ^ {(l + 1)}} P _ {j k} ^ {(l + 1)} \phi^ {\prime} \left(h _ {k} ^ {(l + 1)}\right) W _ {k i} ^ {(l + 1)} \right] \\ = 2 \sum_ {j = 1} ^ {N} \sum_ {k = 1} ^ {N} \mathbb {E} \left[ P _ {j k} ^ {(l + 1)} \right] \mathbb {E} \left[ \frac {\partial \mathcal {L}}{\partial x _ {i} ^ {(l + 1)}} \frac {\partial \mathcal {L}}{\partial x _ {j} ^ {(l + 1)}} \phi^ {\prime} \left(h _ {k} ^ {(l + 1)}\right) W _ {k i} ^ {(l + 1)} \right] (A29) \\ = 0. (A30) \\ \end{array}
$$
The third term of Eq. (A24) is rearranged as follows:
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {j = 1} ^ {N} \sum_ {k = 1} ^ {N} \sum_ {j ^ {\prime} = 1} ^ {N} \sum_ {k ^ {\prime} = 1} ^ {N} \frac {\partial \mathcal {L}}{\partial x _ {j} ^ {(l + 1)}} \frac {\partial \mathcal {L}}{\partial x _ {j ^ {\prime}} ^ {(l + 1)}} P _ {j k} ^ {(l + 1)} P _ {j ^ {\prime} k ^ {\prime}} ^ {(l + 1)} \phi^ {\prime} \left(h _ {k} ^ {(l + 1)}\right) \phi^ {\prime} \left(h _ {k ^ {\prime}} ^ {(l + 1)}\right) W _ {k i} ^ {(l + 1)} W _ {k ^ {\prime} i} ^ {(l + 1)} \right] \\ = \sum_ {j = 1} ^ {N} \sum_ {k = 1} ^ {N} \mathbb {E} \left[ \left(\frac {\partial \mathcal {L}}{\partial x _ {j} ^ {(l + 1)}}\right) ^ {2} \left(P _ {j k} ^ {(l + 1)}\right) ^ {2} \phi^ {\prime} \left(h _ {k} ^ {(l + 1)}\right) ^ {2} \left(W _ {k i} ^ {(l + 1)}\right) ^ {2} \right] (A31) \\ = \sum_ {j = 1} ^ {N} \sum_ {k = 1} ^ {N} \mathbb {E} \left[ \left(\frac {\partial \mathcal {L}}{\partial x _ {j} ^ {(l + 1)}}\right) ^ {2} \right] \mathbb {E} \left[ \left(P _ {j k} ^ {(l + 1)}\right) ^ {2} \right] \mathbb {E} \left[ \phi^ {\prime} \left(h _ {k} ^ {(l + 1)}\right) ^ {2} \right] \mathbb {E} \left[ \left(W _ {k i} ^ {(l + 1)}\right) ^ {2} \right] (A32) \\ = \sigma_ {w} ^ {2} \mathbb {E} \left[ \phi^ {\prime} \left(h _ {k} ^ {(l + 1)}\right) ^ {2} \right] \chi^ {(l + 1)}. (A33) \\ \end{array}
$$
D Sketch of proof for Thm 4.1
In this section, we provide a plain but informal proof of Thm 4.1 for the stepping stone to the formal proof. We present two levels of explanation: the most straightforward one and the other that is closer to a formal proof.
First, we introduce the simplest proof of counterintuitive independence of the distribution of  $J(\pmb{x}^{\mathrm{in}})$  from  $\pmb{x}^{\mathrm{in}}$ . As indicated in Eq. (6), the definition of  $J(\pmb{x}^{\mathrm{in}})$  includes  $\phi'(h^{(l)}(\pmb{x}^{\mathrm{in}}))$ ; thus, the distribution appears to depend  $\pmb{x}^{\mathrm{in}}$ . Here, we consider the distribution of  $\phi'(h(x))$ , where  $h(x) := wx$  with  $w \sim \mathcal{N}(0, \sigma_w^2)$  and  $x \in \mathbb{R}$ . From the following proposition, although  $\phi'(h(x))$  is defined as a function of  $x$ , its distribution is independent of  $x$ . This characteristic holds even for  $J(\pmb{x}^{\mathrm{in}})$ , which encompasses multiple  $\phi'(h^{(l)}(\pmb{x}^{\mathrm{in}}))$ .
Proposition D.1. Let  $\phi(z) \coloneqq uz$  ( $z \geq 0$ );  $vz$  ( $z < 0$ ) be a ReLU-like function,  $w \sim \mathcal{N}(0, \sigma_w^2)$  be a Gaussian variable, and  $x \in \mathbb{R}$  be a fixed real number. Then, the distribution of  $\phi'(wx)$  is independent of  $x$ .
Proof. Consider the derivative of  $\phi$ , given by  $\phi'(z) \coloneqq u (z \geq 0); v (z < 0)$ . The input to  $\phi'$ , i.e.,  $wx$ , follows a Gaussian  $\mathcal{N}(0, x^2 \sigma_w^2)$ . The probability of a zero-mean Gaussian being greater than or equal to zero is the same as it being less than zero, regardless of the value of  $x$ . Therefore, for any given  $x$ , the probabilities of  $\phi'(wx) = u$  and  $\phi'(wx) = v$  are invariant to  $x$ . Consequently, the claim is established.
Then, let us consider Thm 4.1 in a neural network with one activation and one weight layer, respectively, as  $f(\pmb{x}) \coloneqq P^{\mathrm{out}}\phi(W^{(1)}P^{\mathrm{in}}\pmb{x}^{\mathrm{in}})$ . In this setting, the network Jacobian is represented by  $J(x^{\mathrm{in}}) \coloneqq P^{\mathrm{out}}D(\phi'(W^{(1)}P^{\mathrm{in}}x^{\mathrm{in}}))W^{(1)}P^{\mathrm{in}}$ . Moreover, we assume that the uncorrelated Gaussian variables are independent. This assumption is incorrect. Uncorrelated Gaussian variables are not necessarily independent (cf. Remark E.3). However, for the intuition about the formal proof, we assume this. The simplified Thm 4.1 and its proof are as follows:
Proposition D.2. Consider a neural network  $f(\pmb{x}) \coloneqq P^{\mathrm{out}}\phi(W^{(1)}P^{\mathrm{in}}\pmb{x}^{\mathrm{in}})$ . Suppose that the width  $N$  is sufficiently large. Assume that uncorrelated Gaussian variables are independent. Then, for any  $\pmb{x}^{\mathrm{in}} \in \mathbb{R}^d$ , each entry of  $J(\pmb{x}^{\mathrm{in}}) \coloneqq P^{\mathrm{out}}D(\phi'(W^{(1)}P^{\mathrm{in}}\pmb{x}^{\mathrm{in}}))W^{(1)}P^{\mathrm{in}}$  is i.i.d. and follows the Gaussian  $\mathcal{N}(0,\alpha\sigma_w^2/d)$ .
Proof. First, we prove that each entry of  $J(\pmb{x}^{\mathrm{in}})$  is i.i.d. and follows a Gaussian. To this end, we consider the probabilistic properties in the order of  $P^{\mathrm{in}}\pmb{x}^{\mathrm{in}}$ ,  $D(\phi'(W^{(1)}P^{\mathrm{in}}\pmb{x}^{\mathrm{in}}))$ ,  $W^{(1)}P^{\mathrm{in}}$ , and  $P^{\mathrm{out}}D(\phi'(W^{(1)}P^{\mathrm{in}}\pmb{x}^{\mathrm{in}}))W^{(1)}P^{\mathrm{in}}$ . Then, we derive the concrete distribution, i.e., its mean and variance. Comparing the distributions between  $f(\pmb{x}^{\mathrm{in}})^2$  and  $(J(\pmb{x}^{\mathrm{in}})\pmb{x}^{\mathrm{in}})^2$ , we achieve this.
We first consider  $P^{\mathrm{in}}x^{\mathrm{in}} \eqqcolon x^{(0)}$ . Recall that  $x^{\mathrm{in}}$  is a deterministic vector and  $P^{\mathrm{in}}$  is a random matrix where each entry is i.i.d. and follows a Gaussian. Since the weighted sum of independent Gaussian variables follows a Gaussian, each entry of  $x^{(0)}$  is i.i.d. and follows a Gaussian.
Then, let us consider  $W^{(1)}\pmb{x}^{(0)}$ . The  $i$ -th entry of  $W^{(1)}\pmb{x}^{(0)}$  is expressed as  $\sum_{j=1}^{N}W_{ij}^{(1)}x_j^{(0)}$ . Since  $W^{(1)}$  is a random matrix with i.i.d. entries,  $W_{ij}^{(1)}x_j^{(0)}$  is i.i.d. with respect to  $j$ . By the central limit theorem with infinite  $N$ , the  $i$ -th entry of  $W^{(1)}\pmb{x}^{(0)}$  follows a Gaussian. In addition, the entries of  $W^{(1)}\pmb{x}^{(0)}$  follow the same Gaussian. Moreover, since  $\mathbb{E}[(\sum_{j=1}^{N}W_{ij}^{(1)}x_j^{(0)})(\sum_{j=1}^{N}W_{i'j}^{(1)}x_j^{(0)})] = 0$  holds for  $i \neq i'$ , different entries of  $W^{(1)}\pmb{x}^{(0)}$  are uncorrelated and thus independent (this is originally incorrect but guaranteed by the assumption here). In conclusion,  $W^{(1)}\pmb{x}^{(0)}$  is a random vector with i.i.d. Gaussian entries. Naturally,  $D(\phi'(W^{(1)}P^{\mathrm{in}}\pmb{x}^{\mathrm{in}}))$  is a diagonal matrix with i.i.d. entries.
Similar to the discussion above,  $\pmb{W}^{(1)}\pmb{P}^{\mathrm{in}} \eqqcolon \pmb{A}$  is a random matrix with i.i.d. entries.
Finally, consider  $P^{\mathrm{out}}D(\phi'(W^{(1)}P^{\mathrm{in}}x^{\mathrm{in}}))W^{(1)}P^{\mathrm{in}}$ . For notational simplicity, we denote  $D := D(\phi'(W^{(1)}P^{\mathrm{in}}x^{\mathrm{in}}))$ . Thus,  $P^{\mathrm{out}}D(\phi'(W^{(1)}P^{\mathrm{in}}x^{\mathrm{in}}))W^{(1)}P^{\mathrm{in}} = P^{\mathrm{out}}DA$ . The entry of  $i$ -th row and  $j$ -th column of this matrix is expressed as  $\sum_{k=1}^{N}P_{ik}^{\mathrm{out}}D_kA_{kj}$ . Note that  $P^{\mathrm{out}}$  is independent of  $DA$  and its entries are i.i.d. Moreover, based on network symmetry, the dependence between  $D_k$  and  $A_{kj}$  is invariant to  $k$ . Thus,  $P_{ik}^{\mathrm{out}}D_kA_{kj}$  is i.i.d. with respect to  $k$  and each entry of this matrix follows the same Gaussian by the central limit theorem. Moreover, since  $\mathbb{E}[(\sum_{k=1}^{N}P_{ik}^{\mathrm{out}}D_kA_{kj})(\sum_{k=1}^{N}P_{i'k}^{\mathrm{out}}D_kA_{kj'})] = 0$  holds for  $(i,j) \neq (i',j')$ , different entries of  $P^{\mathrm{out}}DA$  are uncorrelated and thus independent. Therefore, each entry of  $J(x^{\mathrm{in}})$  is i.i.d. and follows the Gaussian.
Next, we derive the mean and variance of  $J(\pmb{x}^{\mathrm{in}})$ . Trivially,  $\mathbb{E}[J(\pmb{x}^{\mathrm{in}})_i] = 0$  since the mean of an entry of  $P^{\mathrm{out}}$  is zero. We derive the variance by comparing  $\mathbb{E}[f(\pmb{x})_i^2]$  and  $\mathbb{E}[(\sum_{j=1}^{d} J(\pmb{x}^{\mathrm{in}})_{ij}x_j^{\mathrm{in}})^2]$ . Recall  $f(\pmb{x})_i \coloneqq \sum_{j=1}^{d} J(\pmb{x}^{\mathrm{in}})_{ij}x_j^{\mathrm{in}}$ . As a preliminary, with a Gaussian variable  $z \sim \mathcal{N}(0,\sigma^2)$  and its probabilistic density function  $g(z)$ , we derive the following equation:
$$
\begin{array}{l} \mathbb {E} _ {z \sim \mathcal {N} (0, \sigma^ {2})} [ \phi (z) ^ {2} ] = \int_ {- \infty} ^ {\infty} \phi (z) ^ {2} g (z; \sigma^ {2}) d z (A34) \\ = \int_ {0} ^ {\infty} u ^ {2} z ^ {2} g (z; \sigma^ {2}) d z + \int_ {- \infty} ^ {0} v ^ {2} z ^ {2} g (z; \sigma^ {2}) d z (A35) \\ = \frac {u ^ {2}}{2} \sigma^ {2} + \frac {v ^ {2}}{2} \sigma^ {2} (A36) \\ = \alpha \sigma^ {2}. (A37) \\ \end{array}
$$
In addition, if  $P^{\mathrm{out}}$  and  $z$  are independent, then
$$
\mathbb {E} _ {z \sim \mathcal {N} (0, \sigma^ {2})} \left[ \left(\sum_ {j = 1} ^ {N} P _ {i j} ^ {\text {o u t}} \phi (z) _ {j}\right) ^ {2} \right] = \sum_ {j = 1} ^ {N} \mathbb {E} [ (P _ {i j} ^ {\text {o u t}}) ^ {2} ] \mathbb {E} [ \phi (z) _ {j} ^ {2} ] = \alpha \sigma^ {2}. \tag {A38}
$$
As shown in the discussion above, each entry of  $\mathbf{W}^{(1)}\mathbf{x}^{(0)}$  follows the Gaussian. Moreover, we can simply derive its variance as  $\sigma_w^2\| \pmb{x}^{\mathrm{in}}\| ^2 /d$ . Thus,
$$
\mathbb {E} \left[ f (\boldsymbol {x}) _ {i} ^ {2} \right] = \mathbb {E} \left[ \left(\sum_ {j = 1} ^ {N} P _ {i j} ^ {\text {o u t}} \phi \left(\boldsymbol {W} ^ {(1)} \boldsymbol {x} ^ {(0)}\right) _ {j}\right) ^ {2} \right] = \alpha \sigma_ {w} ^ {2} \| \boldsymbol {x} ^ {\text {i n}} \| ^ {2} / d. \tag {A39}
$$
Since  $f(\pmb{x})_i \coloneqq \sum_{j=1}^d J(\pmb{x}^{\mathrm{in}})_{ij} x_j^{\mathrm{in}}, \mathbb{E}[f(\pmb{x})_i^2]$  is also expanded as:
$$
\mathbb {E} [ f (\boldsymbol {x}) _ {i} ^ {2} ] = \mathbb {E} \left[ \left(\sum_ {j = 1} ^ {d} J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i j} x _ {j} ^ {\text {i n}}\right) ^ {2} \right] = \sum_ {j = 1} ^ {d} \mathbb {E} [ J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i j} ^ {2} ] \left(x _ {j} ^ {\text {i n}}\right) ^ {2} = \mathbb {E} [ J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i j} ^ {2} ] \| \boldsymbol {x} ^ {\text {i n}} \| ^ {2}. \tag {A40}
$$
Comparing the two equations above, we obtain  $\mathbb{E}[J(\pmb{x}^{\mathrm{in}})_{ij}^2] = \alpha \sigma_w^2 /d$ . Therefore, the claim is established.
E Derivation of Thm 4.1 for vanilla networks
E.1 Preliminary
First, we introduce some lemmas. These results are more general and not restricted to the context of neural networks. The notation in this section is independent of that in other sections.
We refer to a random matrix where the mean of an entry is zero as a zero-mean matrix. Formally, this is defined as follows:
Definition E.1 (zero-mean matrix/vector). A random matrix/vector  $\mathbf{A}$  is called a zero-mean matrix/vector if  $\mathbb{E}[A_{ij}] = 0$  for any  $i$  and  $j$ .
For zero-mean matrices, the following properties hold:
Remark E.2 (Properties of zero-mean matrix). Let  $\mathbf{A}$  be a zero-mean matrix.
- Let  $B$  be a zero-mean matrix. Then, their sum,  $A + B$ , is a zero-mean matrix.
- Let  $B$  be a random matrix. If  $A$  and  $B$  are independent, then their products,  $AB$  and  $BA$ , are zero-mean matrices.
Then, we review several properties of Gaussian variables.
Remark E.3 (Properties of Gaussian variables). The following statements hold:
- Any linear combination of multiple Gaussian variables follows a Gaussian if and only if they are jointly distributed Gaussian.
- A weighted sum of independent Gaussian variables is Gaussian distributed.
- Gaussian variables are jointly distributed and uncorrelated if and only if they are independent.
Next, we examine the properties of random matrices/vectors with i.i.d. Gaussian entries, called a Gaussian matrix/vector. Their formal definitions are as follows:
Definition E.4 (Gaussian matrix/vector). A matrix/vector is called a Gaussian matrix/vector if all the entries are i.i.d. and follow a Gaussian.
Gaussian matrices and vectors satisfy the following lemmas.
Lemma E.5. If  $A$  and  $B$  are independent Gaussian matrices, then  $A + B$  is a Gaussian matrix. In particular, when  $A$  and  $B$  are zero-mean,  $A + B$  is a zero-mean Gaussian matrix.
Proof. By Remarks E.2 and E.3, this is trivial.
The following lemmas are derived for the proof of Thm 4.1.
Lemma E.6. Suppose that random matrices  $\mathbf{A} \in \mathbb{R}^{m \times n}$  and  $\mathbf{B} \in \mathbb{R}^{n \times l}$  satisfy the following properties: (a)  $\mathbf{A}$  and  $\mathbf{B}$  are independent. (b)  $\mathbf{A}$  is a zero-mean matrix. (c) All the entries of  $\mathbf{A}$  are i.i.d. (d) All the entries of  $\mathbf{B}$  are identically distributed. (e) Any two entries of  $\mathbf{B}$  from different rows are independent. (f) For any  $i \in [n]$ ,  $j, k \in [l]$ ,  $j \neq k$ , the dependence between  $B_{ij}$  and  $B_{ik}$  is invariant to  $i, j, k$ . (g) For any  $i \in [n]$ ,  $j, k \in [l]$ ,  $j \neq k$ ,  $\mathbb{E}[B_{ij}B_{ik}] = 0$  holds. Then, for a sufficiently large  $n$ ,  $AB$  is a zero-mean Gaussian matrix.
Proof. By (a), (b), and Remark E.2,  $AB$  is a zero-mean matrix. The linear combination of all the entries of  $AB$  is expressed as
$$
c := \sum_ {i = 1} ^ {m} \sum_ {j = 1} ^ {l} s _ {i j} \boldsymbol {A} _ {i}. \boldsymbol {B}. _ {j} = \sum_ {i = 1} ^ {m} \sum_ {j = 1} ^ {l} s _ {i j} \sum_ {k = 1} ^ {n} A _ {i k} B _ {k j} = \sum_ {k = 1} ^ {n} \left(\sum_ {i = 1} ^ {m} \sum_ {j = 1} ^ {l} s _ {i j} A _ {i k} B _ {k j}\right), \tag {A41}
$$
where  $s_{ij} \in \mathbb{R}$  is a constant. By (a), (c), (d), (e), and (f),  $\sum_{i=1}^{m} \sum_{j=1}^{l} s_{ij} A_{ik} B_{kj}$  is i.i.d. with respect to  $k$ . By the central limit theorem,  $c$  follows a Gaussian. By Remark E.3, all the entries of  $AB$  are jointly Gaussian distributed. Moreover, by (c) and (d), the entries of  $AB$  follow the same Gaussian. By (a), (b), (c), (e), and (g), the covariance between any two entries of  $AB$  is zero. Thus, by Remark E.3, the claim is established.
Lemma E.7. Suppose that random matrices  $\mathbf{A} \in \mathbb{R}^{m \times n}$ ,  $\mathbf{B} \in \mathbb{R}^{n \times l_1}$ , and  $\mathbf{C} \in \mathbb{R}^{n \times l_2}$  satisfy the following properties: (a)  $\mathbf{A}$ ,  $\mathbf{B}$ , and  $\mathbf{C}$  are independent. (b)  $\mathbf{A}$  is a zero-mean matrix. (c) All the entries of  $\mathbf{A}$  are i.i.d. (d) All the entries of  $\mathbf{B}$  are identically distributed. (e) All the entries of  $\mathbf{C}$  are identically distributed. (f) Any two entries of  $\mathbf{B}$  from different rows are independent. (g) Any two entries of  $\mathbf{C}$  from different rows are independent. (h) For any  $i \in [n], j, k \in [l_1], j \neq k$ , the dependence between  $B_{ij}$  and  $B_{ik}$  is invariant to  $i, j, k$ . (i) For any  $i \in [n], j, k \in [l_2], j \neq k$ , the dependence between  $C_{ij}$  and  $C_{ik}$  is invariant to  $i, j, k$ . (j) For any  $i \in [n], j \in [l_1], k \in [l_2]$ ,  $\mathbb{E}[B_{ij}C_{ik}] = 0$  holds. Then, for a sufficiently large  $n$ ,  $AB$  and  $AC$  are independent.
Proof. By (a), (c), (d), (e), (f), (g), (h), and (i), similar to Lemma E.6, all the entries of  $AB$  and  $AC$  are jointly Gaussian distributed. By (a), (b), (c), and (j), the covariance between any two entries from  $AB$  and  $AC$  is zero. Thus, by Remark E.3, the claim is established.
Lemma E.8. Let  $\pmb{w},\pmb{x}\in \mathbb{R}^{m}$  be Gaussian vectors. Let  $\phi^{\prime}(z)\coloneqq u$ $(z\geq 0)$ $v(z < 0)$ . Then, for a sufficiently large  $m$ ,  $\phi^{\prime}(\pmb{w}^{\top}\pmb {x})\pmb{w}$  and  $\pmb{x}$  are independent.
Proof. We prove  $\mathbb{P}[\phi'(\boldsymbol{w}^\top \boldsymbol{x})\boldsymbol{w} = \boldsymbol{a} \mid \boldsymbol{x} = \boldsymbol{b}] = \mathbb{P}[\phi'(\boldsymbol{w}^\top \boldsymbol{x})\boldsymbol{w} = \boldsymbol{a}]$ . For any  $\boldsymbol{x}$ ,  $\boldsymbol{w} = \boldsymbol{a} / u + \boldsymbol{x} / \sqrt{m}$  or  $\boldsymbol{w} = \boldsymbol{a} / v - \boldsymbol{x} / \sqrt{m}$  satisfy  $\phi'(\boldsymbol{w}^\top \boldsymbol{x})\boldsymbol{w} = \boldsymbol{a}$ . Since  $m$  is sufficiently large,  $\boldsymbol{w} = \boldsymbol{a} / u + \boldsymbol{x} / \sqrt{m}$  and  $\boldsymbol{w} = \boldsymbol{a} / v - \boldsymbol{x} / \sqrt{m}$  asymptotically approach  $\boldsymbol{w} = \boldsymbol{a} / u$  and  $\boldsymbol{w} = \boldsymbol{a} / v$ , respectively. Thus, the claim is established.
E.2 Definitions of  $J(x^{\mathrm{in}})$  and  $a(x^{\mathrm{in}})$  for vanilla networks
Here, we define (derive)  $J(x^{\mathrm{in}})$  and  $a(x^{\mathrm{in}})$  for vanilla networks. The  $l$ -th pre- and post-activation are defined as follows:
$$
\boldsymbol {h} ^ {(l)} \left(\boldsymbol {x} ^ {\text {i n}}\right) := \boldsymbol {W} ^ {(l)} \boldsymbol {x} ^ {(l - 1)} \left(\boldsymbol {x} ^ {\text {i n}}\right) + \boldsymbol {b} ^ {(l)}, \quad \boldsymbol {x} ^ {(l)} \left(\boldsymbol {x} ^ {\text {i n}}\right) := \phi \left(\boldsymbol {h} ^ {(l)} \left(\boldsymbol {x} ^ {\text {i n}}\right)\right). \tag {A42}
$$
Using  $\phi (\pmb{h}^{(l)}(\pmb{x}^{\mathrm{in}})) = D(\phi^{\prime}(\pmb{h}^{(l)}(\pmb{x}^{\mathrm{in}})))\pmb{h}^{(l)}(\pmb{x}^{\mathrm{in}})$ , the  $l$ -th pre-activation can be rearranged as follows:
$$
\boldsymbol {h} ^ {(l)} \left(\boldsymbol {x} ^ {\text {i n}}\right) = \boldsymbol {W} ^ {(l)} \boldsymbol {D} \left(\phi^ {\prime} \left(\boldsymbol {h} ^ {(l - 1)} \left(\boldsymbol {x} ^ {\text {i n}}\right)\right)\right) \boldsymbol {h} ^ {(l - 1)} \left(\boldsymbol {x} ^ {\text {i n}}\right) + \boldsymbol {b} ^ {(l)}. \tag {A43}
$$
Because ReLU-like networks are piecewise linear, the  $l$ -th pre-activation is also represented as follows: