SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\boldsymbol {h} ^ {(l)} (\boldsymbol {x} ^ {\mathrm {i n}}) = \boldsymbol {J} ^ {(l)} (\boldsymbol {x} ^ {\mathrm {i n}}) \boldsymbol {x} ^ {(0)} (\boldsymbol {x} ^ {\mathrm {i n}}) + \boldsymbol {a} ^ {(l)} (\boldsymbol {x} ^ {\mathrm {i n}}). \tag {A44}
$$
Substituting Eq. (A44) for Eq. (A43), the equation can be rearranged as follows:
$$
\begin{array}{l} \boldsymbol {h} ^ {(l)} (\boldsymbol {x} ^ {\mathrm {i n}}) = \boldsymbol {W} ^ {(l)} \boldsymbol {D} \left(\phi^ {\prime} \left(\boldsymbol {h} ^ {(l - 1)} (\boldsymbol {x} ^ {\mathrm {i n}})\right)\right) \boldsymbol {J} ^ {(l - 1)} (\boldsymbol {x} ^ {\mathrm {i n}}) \boldsymbol {x} ^ {(0)} (\boldsymbol {x} ^ {\mathrm {i n}}) \\ + \boldsymbol {W} ^ {(l)} \boldsymbol {D} \left(\phi^ {\prime} \left(\boldsymbol {h} ^ {(l - 1)} \left(\boldsymbol {x} ^ {\text {i n}}\right)\right)\right) \boldsymbol {a} ^ {(l - 1)} \left(\boldsymbol {x} ^ {\text {i n}}\right) + \boldsymbol {b} ^ {(l)}. \tag {A45} \\ \end{array}
$$
Comparing Eq. (A44) and Eq. (A45), the following equations can be derived:
$$
\boldsymbol {J} ^ {(l)} \left(\boldsymbol {x} ^ {\text {i n}}\right) := \boldsymbol {W} ^ {(l)} \boldsymbol {D} \left(\phi^ {\prime} \left(\boldsymbol {h} ^ {(l - 1)} \left(\boldsymbol {x} ^ {\text {i n}}\right)\right)\right) \boldsymbol {J} ^ {(l - 1)} \left(\boldsymbol {x} ^ {\text {i n}}\right), \tag {A46}
$$
$$
\boldsymbol {a} ^ {(l)} \left(\boldsymbol {x} ^ {\text {i n}}\right) := \boldsymbol {W} ^ {(l)} \boldsymbol {D} \left(\phi^ {\prime} \left(\boldsymbol {h} ^ {(l - 1)} \left(\boldsymbol {x} ^ {\text {i n}}\right)\right)\right) \boldsymbol {a} ^ {(l - 1)} \left(\boldsymbol {x} ^ {\text {i n}}\right) + \boldsymbol {b} ^ {(l)}, \tag {A47}
$$
where  $J^{(1)}(\pmb{x}^{\mathrm{in}}) \coloneqq \pmb{W}^{(1)}$  and  $\pmb{a}^{(1)}(\pmb{x}^{\mathrm{in}}) \coloneqq \pmb{b}^{(1)}$ . Finally,  $J(\pmb{x}^{\mathrm{in}})$  and  $\pmb{a}(\pmb{x}^{\mathrm{in}})$  are defined as follows:
$$
\boldsymbol {J} \left(\boldsymbol {x} ^ {\text {i n}}\right) := \boldsymbol {P} ^ {\text {o u t}} \boldsymbol {D} \left(\phi^ {\prime} \left(\boldsymbol {h} ^ {(L)} \left(\boldsymbol {x} ^ {\text {i n}}\right)\right)\right) \boldsymbol {J} ^ {(L)} \left(\boldsymbol {x} ^ {\text {i n}}\right) \boldsymbol {P} ^ {\text {i n}}, \tag {6}
$$
$$
\boldsymbol {a} \left(\boldsymbol {x} ^ {\text {i n}}\right) := \boldsymbol {P} ^ {\text {o u t}} \boldsymbol {D} \left(\phi^ {\prime} \left(\boldsymbol {h} ^ {(L)} \left(\boldsymbol {x} ^ {\text {i n}}\right)\right)\right) \boldsymbol {a} ^ {(L)} \left(\boldsymbol {x} ^ {\text {i n}}\right). \tag {A48}
$$
E.3 Main derivation
First, we remark several fundamental properties of weights, biases, random projections, and preactivations in a network.
Remark E.9. By definition in Sec. 3.1, the following statements hold for any  $l \in [L]$ :
- (Gaussian matrix/vector)  $\mathbf{W}^{(l)}$ ,  $\mathbf{P}^{\mathrm{in}}$ , and  $\mathbf{P}^{\mathrm{out}}$  are zero-mean Gaussian matrices and  $\mathbf{b}^{(l)}$  is a zero-mean Gaussian vector.
(Variance) The variance of  $\pmb{W}^{(l)}$ ,  $\pmb{P}^{\mathrm{in}}$ , and  $\pmb{P}^{\mathrm{out}}$  is  $\sigma_w^2 / N$ ,  $1 / d$ , and  $1 / N$ , respectively.
- (Independence)  $\pmb{W}^{(l)}$  and  $\pmb{b}^{(l)}$  are independent of each other, as is any random variable in the layer before the  $l$ -th. In addition,  $\pmb{P}^{\mathrm{out}}$  is independent of any random variable without a network output.
- (Dependence between pre-activation and bias) For  $i = j$ , the dependence between  $h_i^{(l)}$  and  $b_i^{(l)}$  is invariant to  $i$ . For  $i \neq j$ ,  $h_i^{(l)}$  and  $b_j^{(l)}$  are independent.
Lemma E.10. Consider a vanilla network. For any  $l \in [L]$ , the following hold:
(a)  $\pmb{J}^{(l)}(\pmb{x}^{\mathrm{in}})$  is a zero-mean Gaussian matrix.
(b)  $\pmb{a}^{(l)}(\pmb{x}^{\mathrm{in}})$  is a zero-mean Gaussian vector.
(c)  $\mathbf{J}^{(l)}(\mathbf{x}^{\mathrm{in}})$  and  $\mathbf{a}^{(l)}(\mathbf{x}^{\mathrm{in}})$  are independent.
Proof. We prove the claim by induction. As a preliminary, see Remark E.9. The case with  $l = 1$  is trivial. Suppose that all the claims hold for  $l - 1 \in [L - 1]$ .
(a) By Lemma E.6, the claim is established. Note that  $\mathbb{E}[\phi^{\prime}(\pmb{h}_i^{(l)})^2\pmb{J}^{(l)}(\pmb{x}^{\mathrm{in}})_{ij}\pmb{J}^{(l)}(\pmb{x}^{\mathrm{in}})_{ik}] = \mathbb{E}[\phi^{\prime}(\pmb{h}_i^{(l)})^2]\mathbb{E}[\pmb{J}^{(l)}(\pmb{x}^{\mathrm{in}})_{ij}]\mathbb{E}[\pmb{J}^{(l)}(\pmb{x}^{\mathrm{in}})_{ik}] = 0$  since  $N$  is sufficiently large.
(b) By Lemma E.6, the claim is established.
(c) By Lemma E.7, the claim is established.
Lemma E.11. The mean squared network output is given by:
$$
\mathbb {E} \left[ f \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i} ^ {2} \right] = \frac {\omega^ {L}}{d} \left\| \boldsymbol {x} ^ {\text {i n}} \right\| _ {2} ^ {2} + \alpha \sigma_ {b} ^ {2} \sum_ {i = 1} ^ {L} \omega^ {i - 1}. \tag {A49}
$$
Proof. By Remark E.9, the mean squared pre- and post- activation are calculated as follows (cf. Eq. (A34)):
$$
\mathbb {E} \left[ \left(h _ {i} ^ {(l)}\right) ^ {2} \right] = \sigma_ {w} ^ {2} \mathbb {E} \left[ \left(x _ {i} ^ {(l - 1)}\right) ^ {2} \right] + \sigma_ {b} ^ {2}, \tag {A50}
$$
$$
\mathbb {E} \left[ \left(x _ {i} ^ {(l)}\right) ^ {2} \right] = \left\{ \begin{array}{l l} \alpha \mathbb {E} \left[ \left(h _ {i} ^ {(l)}\right) ^ {2} \right] & (\text {v a n i l l a}) \\ \mathbb {E} \left[ \left(x _ {i} ^ {(l - 1)}\right) ^ {2} \right] + \alpha \mathbb {E} \left[ \left(h _ {i} ^ {(l)}\right) ^ {2} \right] & (\text {r e s i d u a l}) \end{array} . \right. \tag {A51}
$$
Recursively calculating Eqs. (A50) and (A51), the mean squared post-activation in a vanilla network is calculated as follows:
$$
\mathbb {E} [ (x _ {i} ^ {(l)}) ^ {2} ] = \alpha \mathbb {E} [ (h _ {i} ^ {(l)}) ^ {2} ] = \omega_ {\mathrm {v}} \mathbb {E} [ (x _ {i} ^ {(l - 1)}) ^ {2} ] + \alpha \sigma_ {b} ^ {2} = \omega_ {\mathrm {v}} ^ {l} \mathbb {E} [ (x _ {i} ^ {(0)}) ^ {2} ] + \alpha \sigma_ {b} ^ {2} \sum_ {i = 1} ^ {l} \omega_ {\mathrm {v}} ^ {i - 1}. \tag {A52}
$$
The mean squared post-activation in a residual network is calculated as follows:
$$
\begin{array}{l} \mathbb {E} \left[ \left(x _ {i} ^ {(l)}\right) ^ {2} \right] = \mathbb {E} \left[ \left(x _ {i} ^ {(l - 1)}\right) ^ {2} \right] + \alpha \left(\sigma_ {w} ^ {2} \mathbb {E} \left[ \left(x _ {i} ^ {(l - 1)}\right) ^ {2} \right] + \sigma_ {b} ^ {2}\right) (A53) \\ = \omega_ {\mathrm {r}} \mathbb {E} \left[ \left(x _ {i} ^ {(l - 1)}\right) ^ {2} \right] + \alpha \sigma_ {b} ^ {2} (A54) \\ = \omega_ {\mathrm {r}} ^ {l} \mathbb {E} \left[ \left(x _ {i} ^ {(0)}\right) ^ {2} \right] + \alpha \sigma_ {b} ^ {2} \sum_ {i = 1} ^ {l} \omega_ {\mathrm {r}} ^ {i - 1}. (A55) \\ \end{array}
$$
Using  $x^{(0)} \coloneqq P^{\mathrm{in}}x^{\mathrm{in}}$ , the above results are expanded as follows:
$$
\mathbb {E} \left[ \left(x _ {i} ^ {(l)}\right) ^ {2} \right] = \omega^ {l} \sum_ {j = 1} ^ {d} \mathbb {E} \left[ \left(P _ {i j} ^ {\text {i n}}\right) ^ {2} \right] \left(x _ {j} ^ {\text {i n}}\right) ^ {2} + \alpha \sigma_ {b} ^ {2} \sum_ {i = 1} ^ {l} \omega^ {i - 1} = \frac {\omega^ {l}}{d} \left\| \boldsymbol {x} ^ {\text {i n}} \right\| _ {2} ^ {2} + \alpha \sigma_ {b} ^ {2} \sum_ {i = 1} ^ {l} \omega^ {i - 1}. \tag {A56}
$$
The mean squared network output is rearranged to the mean squared  $L$ -th post-activation as follows:
$$
\mathbb {E} \left[ f \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i} ^ {2} \right] = \sum_ {j = 1} ^ {N} \mathbb {E} \left[ \left(P _ {i j} ^ {\text {o u t}}\right) ^ {2} \right] \mathbb {E} \left[ \left(x _ {j} ^ {(L)}\right) ^ {2} \right] = \mathbb {E} \left[ \left(x _ {i} ^ {(L)}\right) ^ {2} \right]. \tag {A57}
$$
By Eqs. (A56) and (A57), the claim is established.
Proof. Similar to Lemma E.8,  $P^{\mathrm{in}}$  and  $a(x^{\mathrm{in}})$  are independent. Similar to Lemmas E.10 and F.2,  $J(x^{\mathrm{in}})$  and  $a(x^{\mathrm{in}})$  are a zero-mean Gaussian matrix and vector, respectively, and  $J(x^{\mathrm{in}})$  and  $a(x^{\mathrm{in}})$  are independent. The mean squared network output can be expanded as follows:
$$
\begin{array}{l} \mathbb {E} [ f (\boldsymbol {x} ^ {\mathrm {i n}}) _ {i} ^ {2} ] \\ = \mathbb {E} \left[ \left(\sum_ {j = 1} ^ {d} J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i j} x _ {j} ^ {\text {i n}} + a \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i}\right) ^ {2} \right] (A58) \\ = \sum_ {j = 1} ^ {d} \sum_ {k = 1} ^ {d} \mathbb {E} [ J (\boldsymbol {x} ^ {\mathrm {i n}}) _ {i j} J (\boldsymbol {x} ^ {\mathrm {i n}}) _ {i k} ] x _ {j} ^ {\mathrm {i n}} x _ {k} ^ {\mathrm {i n}} + 2 \sum_ {j = 1} ^ {d} \mathbb {E} [ J (\boldsymbol {x} ^ {\mathrm {i n}}) _ {i j} a (\boldsymbol {x} ^ {\mathrm {i n}}) _ {i} ] x _ {j} ^ {\mathrm {i n}} + \mathbb {E} [ a (\boldsymbol {x} ^ {\mathrm {i n}}) _ {i} ^ {2} ] (A59) \\ = \sum_ {j = 1} ^ {d} \mathbb {E} \left[ J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i j} ^ {2} \right] \left(x _ {j} ^ {\text {i n}}\right) ^ {2} + \mathbb {E} \left[ a \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i} ^ {2} \right]. (A60) \\ \end{array}
$$
Using the symmetry of entries of  $J(\pmb{x}^{\mathrm{in}})$  and  $a(x^{\mathrm{in}})$ , the equation can be simplified as follows:
$$
\mathbb {E} \left[ f \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i} ^ {2} \right] = \mathbb {E} \left[ J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i j} ^ {2} \right] \left\| \boldsymbol {x} ^ {\text {i n}} \right\| _ {2} ^ {2} + \mathbb {E} \left[ a \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i} ^ {2} \right]. \tag {A61}
$$
Comparing Lemma E.11, Eq. (7) is obtained. Thus, the claim is established.
F Derivation of Thm 4.1 for residual networks
F.1 Definitions of  $J(x^{\mathrm{in}})$  and  $a(x^{\mathrm{in}})$  for residual networks
Similar to Appx. E.2, we define (derive)  $J(\pmb{x}^{\mathrm{in}})$  and  $a(\pmb{x}^{\mathrm{in}})$  for residual networks. For notational simplicity, we omit the argument  $\pmb{x}^{\mathrm{in}}$ . First, we represent  $\pmb{x}^{(l)}$  as follows:
$$
\boldsymbol {x} ^ {(l)} = \left(\boldsymbol {I} + \boldsymbol {V} ^ {(l)}\right) \boldsymbol {x} ^ {(0)} + \boldsymbol {c} ^ {(l)}, \tag {A62}
$$
where  $\pmb{V}^{(0)}\coloneqq \mathbf{0}$  and  $c^{(0)}\coloneqq 0$  . With  $\pmb{V}^{(l)}$  and  $c^{(l)}$  , we can represent  $\pmb{h}^{(l)}$  as follows:
$$
\begin{array}{l} \boldsymbol {h} ^ {(l)} = \boldsymbol {W} ^ {(l)} \boldsymbol {x} ^ {(l - 1)} + \boldsymbol {b} ^ {(l)} (A63) \\ = \boldsymbol {W} ^ {(l)} \left(\left(\boldsymbol {I} + \boldsymbol {V} ^ {(l - 1)}\right) \boldsymbol {x} ^ {(0)} + \boldsymbol {c} ^ {(l - 1)}\right) + \boldsymbol {b} ^ {(l)} (A64) \\ = \boldsymbol {W} ^ {(l)} (\boldsymbol {I} + \boldsymbol {V} ^ {(l - 1)}) \boldsymbol {x} ^ {(0)} + \boldsymbol {W} ^ {(l)} \boldsymbol {c} ^ {(l - 1)} + \boldsymbol {b} ^ {(l)}. (A65) \\ \end{array}
$$
Then, we derive recurrence representations of  $\mathbf{V}^{(l)}$  and  $\pmb{c}^{(l)}$  as follows:
$$
\begin{array}{l} \boldsymbol {x} ^ {(l)} = \boldsymbol {x} ^ {(l - 1)} + \boldsymbol {P} ^ {(l)} \phi (\boldsymbol {h} ^ {(l)}) (A66) \\ = (\boldsymbol {I} + \boldsymbol {V} ^ {(l - 1)}) \boldsymbol {x} ^ {(0)} + \boldsymbol {c} ^ {(l - 1)} \\ + \boldsymbol {P} ^ {(l)} \boldsymbol {D} \left(\phi^ {\prime} \left(\boldsymbol {h} ^ {(l)}\right)\right) \left(\boldsymbol {W} ^ {(l)} \left(\boldsymbol {I} + \boldsymbol {V} ^ {(l - 1)}\right) \boldsymbol {x} ^ {(0)} + \boldsymbol {W} ^ {(l)} \boldsymbol {c} ^ {(l - 1)} + \boldsymbol {b} ^ {(l)}\right). (A67) \\ \end{array}
$$
For reference, we denote
$$
\boldsymbol {U} ^ {(l)} := \boldsymbol {P} ^ {(l)} \boldsymbol {D} \left(\phi^ {\prime} \left(\boldsymbol {h} ^ {(l)}\right)\right) \boldsymbol {W} ^ {(l)}, \quad \boldsymbol {d} ^ {(l)} := \boldsymbol {P} ^ {(l)} \boldsymbol {D} \left(\phi^ {\prime} \left(\boldsymbol {h} ^ {(l)}\right)\right) \boldsymbol {b} ^ {(l)}. \tag {A68}
$$
With the notations above,
$$
\begin{array}{l} \boldsymbol {x} ^ {(l)} = (\boldsymbol {I} + \boldsymbol {V} ^ {(l - 1)}) \boldsymbol {x} ^ {(0)} + \boldsymbol {c} ^ {(l - 1)} + \boldsymbol {U} ^ {(l)} (\boldsymbol {I} + \boldsymbol {V} ^ {(l - 1)}) \boldsymbol {x} ^ {(0)} + \boldsymbol {U} ^ {(l)} \boldsymbol {c} ^ {(l - 1)} + \boldsymbol {d} ^ {(l)} (A69) \\ = (\boldsymbol {I} + \boldsymbol {U} ^ {(l)}) (\boldsymbol {I} + \boldsymbol {V} ^ {(l - 1)}) \boldsymbol {x} ^ {(0)} + (\boldsymbol {I} + \boldsymbol {U} ^ {(l)}) \boldsymbol {c} ^ {(l - 1)} + \boldsymbol {d} ^ {(l)} (A70) \\ = \left(\boldsymbol {I} + \boldsymbol {U} ^ {(l)} + \boldsymbol {V} ^ {(l - 1)} + \boldsymbol {U} ^ {(l)} \boldsymbol {V} ^ {(l - 1)}\right) \boldsymbol {x} ^ {(0)} + \boldsymbol {c} ^ {(l - 1)} + \boldsymbol {U} ^ {(l)} \boldsymbol {c} ^ {(l - 1)} + \boldsymbol {d} ^ {(l)}. (A71) \\ \end{array}
$$
$$
\boldsymbol {V} ^ {(l)} := \boldsymbol {U} ^ {(l)} + \boldsymbol {V} ^ {(l - 1)} + \boldsymbol {U} ^ {(l)} \boldsymbol {V} ^ {(l - 1)}, \quad \boldsymbol {c} ^ {(l)} := \boldsymbol {c} ^ {(l - 1)} + \boldsymbol {U} ^ {(l)} \boldsymbol {c} ^ {(l - 1)} + \boldsymbol {d} ^ {(l)}. \tag {A72}
$$
Finally,  $J(\pmb{x}^{\mathrm{in}})$  and  $a(\pmb{x}^{\mathrm{in}})$  are defined as follows:
$$
\boldsymbol {J} \left(\boldsymbol {x} ^ {\text {i n}}\right) := \boldsymbol {P} ^ {\text {o u t}} \left(\boldsymbol {I} + \boldsymbol {V} ^ {(L)}\right) \boldsymbol {P} ^ {\text {i n}}, \quad \boldsymbol {a} \left(\boldsymbol {x} ^ {\text {i n}}\right) := \boldsymbol {P} ^ {\text {o u t}} \boldsymbol {c} ^ {(L)}. \tag {A73}
$$
F.2 Main derivation
First, we note the properties of random projections in shortcuts.
Remark F.1. By definition (cf. Appx. C), the following statements hold for any  $l \in [L]$ :
(Gaussian matrix/vector)  $P^{(l)}$  is a zero-mean Gaussian matrix.
(Variance) The variance of  $P^{(l)}$  is  $1 / N$ .
- (Independence)  $\mathbf{P}^{(l)}$  is independent of  $\mathbf{W}^{(l)}, \mathbf{b}^{(l)}$ , and  $\mathbf{h}^{(l)}$ , as is any random variable in the layer before the  $l$ -th.
Finally, we introduce a lemma similar to Lemma E.10. A subsequent discussion to derive Thm 4.1 is the same as the proof in Appx. E.3.
Lemma F.2. Consider a residual network. For any  $l \in [L]$ , the following statements hold:
(a)  $\pmb{J}^{(l)}(\pmb{x}^{\mathrm{in}})$  is a zero-mean Gaussian matrix.
(b)  $\pmb{a}^{(l)}(\pmb{x}^{\mathrm{in}})$  is a zero-mean Gaussian vector.
(c)  $\pmb{J}^{(l)}(\pmb{x}^{\mathrm{in}})$  and  $\pmb{a}^{(l)}(\pmb{x}^{\mathrm{in}})$  are independent.
Proof. As a preliminary, please refer to Remarks E.9 and F.1 and Lemma E.10.
Note that effects of all the random variables in the layer before the  $l$ -th, such as  $\pmb{P}^{(l-1)}$ ,  $\pmb{h}^{(l-1)}$ ,  $\pmb{W}^{(l-1)}$ , and  $\pmb{b}^{(l-1)}$ , are aggregated to  $\pmb{x}^{(l-1)}$ . By Lemma E.8,  $\pmb{U}^{(l)}$  and  $\pmb{d}^{(l)}$  are independent of  $\pmb{U}^{(l')}$  and  $\pmb{d}^{(l')} \neq l$ . Similarly, as  $\pmb{V}^{(l)}$  and  $\pmb{c}^{(l)}$  consist of  $\pmb{U}^{(1)}, \dots, \pmb{U}^{(l)}, \pmb{d}^{(1)}, \dots, \pmb{d}^{(l)}, \pmb{U}^{(l+1)}$  and  $\pmb{d}^{(l+1)}$  are independent of  $\pmb{V}^{(l)}$  and  $\pmb{c}^{(l)}$ . In addition, similar to Lemma E.10,  $\pmb{U}^{(l)}$  and  $\pmb{d}^{(l)}$  are independent by Lemma E.7.
We prove the claim by induction. Assume that  $\mathbf{V}^{(l-1)}$  is a Gaussian matrix. This holds for  $l' = 1$  because of  $\mathbf{V}^{(1)} = \mathbf{U}^{(1)}$ . Similar to Lemma E.8,  $U_{ij}$  is independent of  $U_{i}.V_{j}$ . Since  $\mathbf{U}^{(l)}$  and  $\mathbf{V}^{(l-1)}$  are independent Gaussian matrices and  $N$  is sufficiently large,  $\mathbf{V}^{(l)}$  is a Gaussian matrices. Similarly,  $\mathbf{c}^{(l)}$  is a Gaussian vector. By Lemma E.7,  $\mathbf{V}^{(l)}$  and  $\mathbf{c}^{(l)}$  are independent. Thus, similar to Lemma E.10,  $\mathbf{J}(\mathbf{x}^{\mathrm{in}})$  and  $\mathbf{a}(\mathbf{x}^{\mathrm{in}})$  are independent Gaussian matrices.
G Derivation of the theorems in Sec. 5
G.1 Derivation of the theorems in Sec. 5.1
First, we introduce the following lemma, which is required to derive the maximum  $\ell_{\infty}$  norm of a column of  $J(x^{\mathrm{in}})$ .
Lemma G.1. The maximum absolute value of  $n \in \mathbb{N}$  i.i.d. Gaussian variables with zero mean and variance  $\sigma^2 > 0$  is smaller than  $\sqrt{2\sigma^2 \ln n}$ .
Table A3: Computable combination of a  $(p,q)$ -operator norm [91].
q = 1	q = 2	q = ∞
p = 1	max. ℓ1 norm of a column	max. ℓ2 norm of a column	max. ℓ∞ norm of a column
p = 2	NP-hard	max. singular value	max. ℓ2 norm of a row
p = ∞	NP-hard	NP-hard	max. ℓ1 norm of a row
Proof. Let  $z \in \mathbb{R}$  be a Gaussian variable with zero mean and variance  $\sigma^2$  and  $a > 0$  be a positive value. First, we compute the upper bound of probability of  $z > a$  as follows:
$$
\begin{array}{l} \mathbb {P} [ z > a ] = \int_ {a} ^ {\infty} g (z; \sigma^ {2}) d z (A74) \\ \leq \int_ {a} ^ {\infty} \frac {z}{a} g \left(z; \sigma^ {2}\right) d z (A75) \\ = \frac {1}{a \sqrt {2 \pi \sigma^ {2}}} \left[ - \sigma^ {2} \exp \left(- \frac {z ^ {2}}{2 \sigma^ {2}}\right) \right] _ {a} ^ {\infty} (A76) \\ = \frac {1}{a} \sqrt {\frac {\sigma^ {2}}{2 \pi}} \exp \left(- \frac {a ^ {2}}{2 \sigma^ {2}}\right). (A77) \\ \end{array}
$$
Then, we compute the lower bound of probability of  $|z| < a$  as follows:
$$
\begin{array}{l} \mathbb {P} [ | z | <   a ] = \int_ {- a} ^ {a} g (z; \sigma^ {2}) d z (A78) \\ = 2 \int_ {0} ^ {a} g (z; \sigma^ {2}) d z + 2 \int_ {- \infty} ^ {0} g (z; \sigma^ {2}) d z - 1 (A79) \\ = 2 \int_ {- \infty} ^ {a} g (z; \sigma^ {2}) d z - 1 (A80) \\ = 2 \left(1 - \int_ {a} ^ {\infty} g (z; \sigma^ {2}) d z\right) - 1 (A81) \\ = 1 - 2 \int_ {a} ^ {\infty} g (z; \sigma^ {2}) d z (A82) \\ \geq 1 - \frac {1}{a} \sqrt {\frac {2 \sigma^ {2}}{\pi}} \exp \left(- \frac {a ^ {2}}{2 \sigma^ {2}}\right). (A83) \\ \end{array}
$$
Let  $\{z_i\}_{i=1}^n$  be  $n$  i.i.d. Gaussian variables. Finally, we compute the upper bound of probability of  $\max_{i \in [n]} |z_i| > a$  as follows:
$$
\mathbb {P} \left[ \max  _ {i \in [ n ]} | z _ {i} | > a \right] = (1 - \mathbb {P} [ | z | <   a ]) ^ {n} \leq \left(\frac {1}{a} \sqrt {\frac {2 \sigma^ {2}}{\pi}} \exp \left(- \frac {a ^ {2}}{2 \sigma^ {2}}\right)\right) ^ {n}. \tag {A84}
$$
In particular, when  $a = \sqrt{2\sigma^2\ln n}$
$$
\mathbb {P} \left[ \max  _ {i \in [ n ]} | z _ {i} | > \sqrt {2 \sigma^ {2} \ln n} \right] \leq \left(\frac {1}{n \sqrt {\pi \ln n}}\right) ^ {n} \xrightarrow {n \to \infty} 0. \tag {A85}
$$
Thus, the claim is established.
Proof. We note the following:
- By Thm 4.1, each entry of  $J(\pmb{x}^{\mathrm{in}})$  is i.i.d. and follows a Gaussian with zero mean and variance  $\mathbb{V}[J(\pmb{x}^{\mathrm{in}})_{ij}] = \omega^{L} / d$ .
- The input and output dimensions,  $d$  and  $K$ , are sufficiently large (cf. Sec. 3.1).
- For some combinations of  $(p,q)$ ,  $(p,q)$ -operator norms are computable (cf. Tab. A3) [91].
In addition, we note the following upper bound (cf. Sec. 5.1):
$$
\mathcal {L} _ {\mathrm {a d v}} (\boldsymbol {x} ^ {\text {i n}}) \leq \epsilon \max  _ {\boldsymbol {x} \in \mathbb {R} ^ {d}} \| \boldsymbol {J} (\boldsymbol {x}) \| _ {p, q}. \tag {Ineq. (8)}
$$
As a preliminary, we compute the mean absolute value of  $J(\pmb{x}^{\mathrm{in}})_{ij}$  as follows:
$$
\mathbb {E} \left[ | J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i j} | \right] = \sqrt {\frac {2 \omega^ {L}}{\pi d}}. \tag {A86}
$$
In the above derivation, we use the following equation
$$
\mathbb {E} _ {z \sim \mathcal {N} (0, \sigma^ {2})} [ | z | ] = \sqrt {\frac {2 \sigma^ {2}}{\pi}}. \tag {A87}
$$
Based on Tab. A3, we compute  $\| J(\pmb{x}^{\mathrm{in}})\|_{p,q}$  as follows:
Maximum  $\ell_{1}$  norm of a column,  $\| J(x^{\mathrm{in}})\|_{1,1}$
$$
\begin{array}{l} \max  _ {j \in [ d ]} \sum_ {i = 1} ^ {K} | J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i j} | = \max  _ {j \in [ d ]} K \frac {1}{K} \sum_ {i = 1} ^ {K} | J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i j} | (A88) \\ = \max  _ {j \in [ d ]} K \mathbb {E} \left[ \left| J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i j} \right| \right] (A89) \\ = \max  _ {j \in [ d ]} K \sqrt {\frac {2 \omega^ {L}}{\pi d}} (A90) \\ = \sqrt {\frac {2}{\pi d}} K \omega^ {L / 2}. (A91) \\ \end{array}
$$
Maximum  $\ell_2$  norm of a column,  $\| J(x^{\mathrm{in}})\|_{1,2}$
$$
\begin{array}{l} \max  _ {j \in [ d ]} \sqrt {\sum_ {i = 1} ^ {K} J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i j} ^ {2}} = \max  _ {j \in [ d ]} \sqrt {K \frac {1}{K} \sum_ {i = 1} ^ {K} J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i j} ^ {2}} (A92) \\ = \max  _ {j \in [ d ]} \sqrt {K \mathbb {E} \left[ J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i j} ^ {2} \right]} (A93) \\ = \max  _ {j \in [ d ]} \sqrt {K \frac {\omega^ {L}}{d}} (A94) \\ = \sqrt {\frac {K}{d}} \omega^ {L / 2}. (A95) \\ \end{array}
$$
Maximum  $\ell_{\infty}$  norm of a column,  $\| J(\pmb{x}^{\mathrm{in}})\|_{1,\infty}$ . By Lemma G.1,
$$
\max  _ {i \in [ K ]} | J (\boldsymbol {x} ^ {\text {i n}}) _ {i j} | \leq \sqrt {2 \mathbb {V} [ J (\boldsymbol {x} ^ {\text {i n}}) _ {i j} ] \ln K} = \sqrt {\frac {2 \ln K}{d}} \omega^ {L / 2}. \tag {A96}
$$
Maximum singular value,  $\| J(\pmb{x}^{\mathrm{in}})\|_{2,2}$ . By the Marchenko-Pastur law,
$$
\left\| \boldsymbol {J} \left(\boldsymbol {x} ^ {\text {i n}}\right) \right\| _ {2} \leq \left(1 + \sqrt {\frac {K}{d}}\right) \omega^ {L / 2}, \tag {A97}
$$
where  $\| \cdot \|_2$  denotes the spectral norm (largest singular value).
Maximum  $\ell_2$  norm of a row,  $\| J(x^{\mathrm{in}})\|_{2,\infty}$
$$
\begin{array}{l} \max  _ {i \in [ K ]} \sqrt {\sum_ {j = 1} ^ {d} J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i j} ^ {2}} = \max  _ {i \in [ K ]} \sqrt {d \frac {1}{d} \sum_ {j = 1} ^ {d} J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i j} ^ {2}} (A98) \\ = \max  _ {i \in [ K ]} \sqrt {d \mathbb {E} \left[ J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i j} ^ {2} \right]} (A99) \\ = \max  _ {i \in [ K ]} \sqrt {d \frac {\omega^ {L}}{d}} (A100) \\ = \omega^ {L / 2}. (A101) \\ \end{array}
$$
Maximum  $\ell_{1}$  norm of a row,  $\| J(\pmb{x}^{\mathrm{in}})\|_{\infty ,\infty}$
$$
\begin{array}{l} \max  _ {i \in [ K ]} \sum_ {j = 1} ^ {d} | J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i j} | = \max  _ {i \in [ K ]} d \frac {1}{d} \sum_ {j = 1} ^ {d} | J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i j} | (A102) \\ = \max  _ {i \in [ K ]} d \mathbb {E} \left[ \left| J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {i j} \right| \right] (A103) \\ = \max  _ {i \in [ K ]} d \sqrt {\frac {2 \omega^ {L}}{\pi d}} (A104) \\ = \sqrt {\frac {2 d}{\pi}} \omega^ {L / 2}. (A105) \\ \end{array}
$$
In addition, we try to derive equalities rather than inequalities (upper bounds). First, we note the following equation:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {a d v}} \left(\boldsymbol {x} ^ {\mathrm {i n}}\right) := \max  _ {\| \boldsymbol {\eta} \| _ {p} \leq \epsilon} \left\| \boldsymbol {f} \left(\boldsymbol {x} ^ {\mathrm {i n}} + \boldsymbol {\eta}\right) - \boldsymbol {f} \left(\boldsymbol {x} ^ {\mathrm {i n}}\right) \right\| _ {q} (Eq. (3)) \\ = \max  _ {\| \boldsymbol {\eta} \| _ {p} \leq \epsilon} \left\| \boldsymbol {J} (\boldsymbol {x} ^ {\text {i n}} + \boldsymbol {\eta}) (\boldsymbol {x} ^ {\text {i n}} + \boldsymbol {\eta}) + \boldsymbol {a} (\boldsymbol {x} ^ {\text {i n}} + \boldsymbol {\eta}) - \boldsymbol {J} (\boldsymbol {x} ^ {\text {i n}}) \boldsymbol {x} ^ {\text {i n}} - \boldsymbol {a} (\boldsymbol {x} ^ {\text {i n}}) \right\|. (A106) \\ \end{array}
$$
Because  $J(\pmb{x}^{\mathrm{in}})$  and  $\pmb{a}(\pmb{x}^{\mathrm{in}})$  are the slope and bias of a piecewise linear region, respectively, for sufficiently small  $\eta$ ,  $J(\pmb{x}^{\mathrm{in}} + \pmb{\eta})$  and  $\pmb{a}(\pmb{x}^{\mathrm{in}} + \pmb{\eta})$  are identical to  $J(\pmb{x}^{\mathrm{in}})$  and  $\pmb{a}(\pmb{x}^{\mathrm{in}})$ , respectively. Therefore, for sufficiently small  $\eta$ , we can derive the following equation: