SlowGuess/NeurIPS_2023_OCR huggingface dataset:
By choosing small  $\sigma_w$ , it follows from Theorem A.2 that the transition 2 becomes a contraction mapping so that we obtain the following results.
Lemma F.1. Choose  $\sigma_w > 0$  small for which  $\gamma \coloneqq 2\sqrt{2}\sigma_w < 1$ . Then for every  $x \in \mathbb{S}^{n_{in} - 1}$  and for any  $k \geq \ell$ , we have
$$
\left\| h ^ {\ell} (x) - h ^ {k} (x) \right\| \leq \frac {\gamma^ {\ell} \left(1 - \gamma^ {k - \ell}\right)}{1 - \gamma} \| h ^ {1} (x) \|, \quad a. s. \tag {55}
$$
Consequently, the equilibrium point  $h^*(x)$  is uniquely determined a.s. Additionally, we have  $\| h^\ell(x) \| \leq \frac{1 - \gamma^\ell}{1 - \gamma} \| h^1(x) \|$  a.s.
To simplify the notation, let us denote
$$
A _ {n, \ell} := \frac {1}{n} \left\langle h ^ {\ell} (x), h ^ {\ell} \left(x ^ {\prime}\right) \right\rangle \tag {56}
$$
By using Lemma F.1, we can show  $\lim_{\ell \to \infty}A_{n,\ell} = B_n$  in uniformly  $n$ . Then the desired result is obtained by using similar arguments from Moore-Osgood theorem for interchanging limits.
Lemma F.2. For any  $n\in \mathbb{N}$  and  $k\geq \ell$  , we have  $|A_{n,\ell} - A_{n,k}|\leq 2(1 - \gamma)^{-2}\gamma^{\ell}$
By using Lemma F.2, we can show the two iterated limits exist and equal to the double limit.
Let  $\varepsilon > 0$ , then there exists  $L(\epsilon) \in \mathbb{N}$  such that  $2(1 - \gamma)^{-2}\gamma^{L} \leq \epsilon$ . Thus,  $k \geq \ell \geq L(\epsilon)$  implies
$$
\left| A _ {n, \ell} - A _ {n, k} \right| \leq \epsilon .
$$
Then let  $n\to \infty$  , it follows from Theorem 4.1 that  $A_{n,\ell}\stackrel {a.s.}{\to}\Sigma^{\ell}(x,x^{\prime})\coloneqq \Sigma^{\ell}$  and so
$$
\left| \Sigma^ {\ell} - \Sigma^ {k} \right| \leq \epsilon .
$$
Therefore,  $\{\Sigma^{\ell}\}$  is a Cauchy sequence that must converge to a unique limit  $\Sigma^{*} := \Sigma^{*}(x,x^{\prime})$  because of the completeness of  $\mathbb{R}$ . Furthermore, let  $k\to \infty$ . we obtain
$$
\left| \Sigma^ {\ell} - \Sigma^ {*} \right| \leq \epsilon .
$$
Since have  $h^\ell(x) \to h^*(x)$  as  $\ell \to \infty$ , there exists  $N(\epsilon, L) \in \mathbb{N}$  such that  $n \geq N_2(\epsilon, L)$  implies
$$
\left| A _ {n, L} - H _ {n} \right| \leq \epsilon ,
$$
where  $H_{n}\coloneqq \frac{1}{n}\left\langle h^{*}(x),h^{*}(x^{\prime})\right\rangle$  . Combines everything together,  $n\geq N$  implies
$$
\left| H _ {n} - \Sigma^ {*} \right| \leq \left| H _ {n} - A _ {n, L} \right| + \left| A _ {n, L} - \Sigma^ {L} \right| + \left| \Sigma^ {L} - \Sigma^ {*} \right| \leq 3 \varepsilon .
$$
Therefore,  $\lim_{n\to \infty}H_n\stackrel {a.s.}{=}\Sigma^* = \lim_{\ell \to \infty}\Sigma^*$ . Additionally, by taking  $M\coloneqq \max \{L,N\}$ , we can see that the limit  $\Sigma^{*}$  equal to the double limit  $\lim_{\ell ,n\to \infty}A_{n,\ell}$ .
F.1 Proof of Lemma F.1
It follows from Theorem A.2 that  $\frac{1}{\sqrt{n}} \| W \| \leq 2\sqrt{2} \sigma_w$  a.s. Then we can choose a small  $\sigma_w$  for which  $\gamma := 2\sqrt{2} \sigma_w < 1$ . Then for any  $\ell \geq 0$ , the Lipschitz continuity of  $\phi$  implies
$$
\begin{array}{l} \| h ^ {\ell + 1} - h ^ {\ell} \| = \frac {1}{\sqrt {n}} \| \phi (W h ^ {\ell} + g ^ {1}) - \phi (W h ^ {\ell - 1} + g ^ {1}) \| \\ \leq \frac {1}{\sqrt {n}} \| W h ^ {\ell} - W h ^ {\ell - 1} \| \\ \leq \frac {1}{\sqrt {n}} \| W \| \| h ^ {\ell} - h ^ {\ell - 1} \| \\ \leq \gamma \| h ^ {\ell} - h ^ {\ell - 1} \|. \\ \end{array}
$$
Thus, we repeat this argument  $\ell$  times and obtain
$$
\left\| h ^ {\ell + 1} - h ^ {\ell} \right\| \leq \gamma^ {\ell} \left\| h ^ {1} - h ^ {0} \right\| = \gamma^ {\ell} \left\| h ^ {1} \right\|
$$
From here, for any  $k\geq \ell \geq 0$  , we have
$$
\left\| h ^ {\ell} - h ^ {k} \right\| \leq \sum_ {s = \ell} ^ {k - 1} \left\| h ^ {s} - h ^ {s + 1} \right\| \leq \sum_ {s = \ell} ^ {k - 1} \gamma^ {s} \left\| h ^ {1} \right\| \leq \frac {\gamma^ {\ell} \left(1 - \gamma^ {k - \ell}\right)}{1 - \gamma} \left\| h ^ {1} \right\|. \tag {57}
$$
Thus, it follows from the completeness of  $\mathbb{R}^m$  that the unique  $h^* (x)$  exists. Additionally, let  $k\to \infty$  then we have
$$
\left\| h ^ {\ell} - h ^ {*} \right\| \leq \frac {\gamma^ {\ell}}{1 - \gamma} \left\| h ^ {1} \right\|.
$$
Let  $\ell = 0$ , then we obtain
$$
\| h ^ {k} \| \leq \frac {1 - \gamma^ {k}}{1 - \gamma} \| h ^ {1} \|.
$$
F.2 Proof of Lemma F.2
WLOG, we can assume  $k \geq \ell$ . To simplify the notation, we further denote  $h_i^\ell = h^\ell(x)$  and  $h_j^\ell = h^\ell(x')$ . Then we have
$$
\begin{array}{l} \left| A _ {n, \ell} - A _ {n, k} \right| = \left| \frac {1}{n} \left\langle h _ {i} ^ {\ell}, h _ {j} ^ {\ell} \right\rangle - \frac {1}{n} \left\langle h _ {i} ^ {k}, h _ {j} ^ {k} \right\rangle \right| \\ \leq \frac {1}{n} \left| \left\langle h _ {i} ^ {\ell}, h _ {j} ^ {\ell} \right\rangle - \left\langle h _ {i} ^ {\ell}, h _ {j} ^ {k} \right\rangle \right| + \frac {1}{n} \left| \left\langle h _ {i} ^ {\ell}, h _ {j} ^ {k} \right\rangle \left\langle h _ {i} ^ {k}, h _ {j} ^ {k} \right\rangle \right| \\ \leq \frac {1}{n} \| h _ {i} ^ {\ell} \| \cdot \| h _ {j} ^ {\ell} - h _ {j} ^ {k} \| + \frac {1}{n} \| h _ {i} ^ {\ell} - h _ {i} ^ {k} \| \cdot \| h _ {j} ^ {k} \| \\ \end{array}
$$
It follows from Lemma F.1 that  $\| h_i^\ell \| \leq \frac{1 - \gamma^\ell}{1 - \gamma} \| h_i^1\|$  and  $\| h_j^\ell -h_j^k\| \leq \frac{\gamma^\ell}{1 - \gamma}\| h_j^1\|$ . Based on Theorem A.2, we have
$$
\| h _ {i} ^ {1} \| = \| \phi (U x _ {i}) \| \leq \| U x _ {i} \| \leq C \frac {\sqrt {n}}{\sqrt {d}} \sigma_ {u} \| x _ {i} \|,
$$
where  $C$  is some absolute fixed constant. WLOG, we can assume we choose  $\sigma_u = \sqrt{d} / C$ . As  $\| x_i \| = 1$ , we obtain
$$
\left\| h _ {i} ^ {1} \right\| \leq \sqrt {n}, \quad \text {a . s .} \tag {58}
$$
$$
\left| A _ {n, \ell} - A _ {n, k} \right| \leq \frac {2}{n} \cdot \frac {1 - \gamma^ {\ell}}{1 - \gamma} \sqrt {n} \cdot \frac {\gamma^ {\ell}}{1 - \gamma} \sqrt {n} = \frac {2}{(1 - \gamma) ^ {2}} \gamma^ {\ell}
$$
By condition on the values of  $h^*$ , the outputs
$$
f _ {\theta , k} (x) = \left\langle v _ {k}, h ^ {*} \right\rangle
$$
are i.i.d. centered Gaussian random variables with covariance
$$
\hat {\Sigma} (x, x ^ {\prime}) = \left\langle h ^ {*} (x), h ^ {*} \left(x ^ {\prime}\right) \right\rangle / n.
$$
It follows from Lemma 4.1 that
$$
\hat {\Sigma} (x, x ^ {\prime}) \stackrel {{a. s.}} {{\to}} \Sigma^ {*} (x, x ^ {\prime}).
$$
Specifically, the covariance  $\Sigma^{*}$  is deterministic and hence independent to  $h^*$ . Consequently, the conditioned and unconditioned distributions of  $f_{\theta ,k}$  are equal in the limit of  $n\to \infty$ : they are i.i.d. centered Gaussian random variables with covariance  $\Sigma^{*}$ .
H Proof of Theorem 4.5
Equipped with the notion of dual activation and Theorem D.1, we are ready to prove Theorem 4.5, i.e.,  $\Sigma^{*}$  is strictly positive definite.
By Lemma E.1, we have  $\Sigma^{*}(x,x) = \Sigma^{*}(x^{\prime},x^{\prime})\coloneqq c$  and  $0 <   c <   \infty$  for all  $x,x^{\prime}$  . Then we have
$$
\Sigma^ {*} (x, x ^ {\prime}) = \mathbb {E} _ {u (x), u (x ^ {\prime}) \sim \mathcal {N} (0, A ^ {*})} [ \phi (u (x)) \phi (u (x ^ {\prime})) ]
$$
$$
A ^ {*} = \left[ \begin{array}{c c} \Sigma^ {*} (x, x) + \Sigma^ {1} (x, x) & \Sigma^ {*} (x, x ^ {\prime}) + \Sigma^ {1} (x, x ^ {\prime}) \\ \Sigma^ {*} (x ^ {\prime}, x) + \Sigma^ {1} (x ^ {\prime}, x) & \Sigma^ {*} (x, x) + \Sigma^ {1} (x ^ {\prime}, x ^ {\prime}) \end{array} \right] = \left[ \begin{array}{c c} c + 1 & \Sigma^ {*} (x, x ^ {\prime}) + \langle x, x ^ {\prime} \rangle \\ \Sigma^ {*} (x, x ^ {\prime}) + \langle x, x ^ {\prime} \rangle & c + 1 \end{array} \right].
$$
By changing variable with  $u(x) = \sqrt{c + 1} z(x)$ , we obtain
$$
\Sigma^ {*} (x, x ^ {\prime}) = \mathbb {E} \left[ \mu (z (x)) \mu (z (x ^ {\prime})) \right] = \hat {\mu} \left(\frac {\Sigma^ {*} (x , x ^ {\prime}) + \langle x , x ^ {\prime} \rangle}{c + 1}\right),
$$
where  $\hat{\mu}:[-1,1]\to \mathbb{R}$  is dual activation of activation function  $\mu (z)\coloneqq \phi (\sqrt{c + 1} z)$ .
Let  $\mu = \sum_{n=0}^{\infty} a_n h_n$  be the Hermite expansion, then we obtain  $\hat{\mu}$  as
$$
\hat {\mu} (\rho) = \sum_ {n = 0} ^ {\infty} a _ {n} ^ {2} \rho^ {n}.
$$
Therefore,  $\Sigma^{*}$  has the expression
$$
\Sigma^ {*} (x, x ^ {\prime}) = \sum_ {n = 0} ^ {\infty} a _ {n} ^ {2} \left(\frac {\Sigma^ {*} (x , x ^ {\prime}) + \langle x , x ^ {\prime} \rangle}{c + 1}\right) ^ {n}.
$$
Since  $\phi$  is non-polynomial, so is  $\mu$ , and hence, there is an infinite number of nonzero  $a_{n}$ 's. By Theorem D.1, we can conclude that  $\Sigma^{*}$  is strictly positive definite and complete the proof.
Code is made at https://github.com/deqg/deq.git.
Distribution of the output distribution
Figure 5: Histplot of the distribution of  $h^L$  for five neural networks with widths 10, 50, 100, 500, 1000 (left to right); KS statistics: 0.0154, 0.0080, 0.0054, 0.0065, 0.0068, pvalue: 0.0173, 0.5395, 0.9331, 0.7924, 0.7446.
Results for different activation function relu
Figure 6: Plot of Figure 1 with activation function relu
Figure 7: Plot of Figure 2 with activation function relu
Figure 8: Plot of Figure 3 with activation function relu
Figure 9: Plot of Figure 4 with activation function relu
Figure 10: Plot of Figure 5 with activation function relu; KS statistics: 0.1217, 0.0137, 0.0127, 0.0070, 0.0056, pvalue:  $1.88 \times 10^{-129}$ , 0.0456, 0.0774, 0.7138, 0.9057.