SlowGuess/NeurIPS_2023_OCR huggingface dataset:
with  $w_{i} = s_{1}\frac{x_{i}^{(2)} - x_{i}^{(1)}}{\|x^{(2)} - x^{(1)}\|^{2}}$ $x_{i}^{(2)} = x_{i}^{(1)} + \epsilon_{i}^{\prime}\hat{w}_{1}$ , and for all  $x\in \mathcal{X}$ . It is clear that  $x_{i}^{(1)},x_{i}^{(2)}\in \mathcal{X}$ . We may now rewrite the sum of tanh functions as
$$
\begin{array}{l} g (x) = \sum_ {i = 1} ^ {M} \tilde {w} _ {i} \psi \left(\langle w _ {i}, x - x _ {i} ^ {(1)} \rangle - s _ {2}\right) + \tilde {w} _ {i} \\ = \sum_ {i = 1} ^ {M} \tilde {w} _ {i} \psi \left(\frac {s _ {1}}{\| \epsilon_ {i} ^ {\prime} \hat {w} _ {1} \|} \left\langle \frac {\epsilon_ {i} ^ {\prime} \hat {w} _ {1}}{\| \epsilon_ {i} ^ {\prime} \hat {w} _ {1} \|}, x - x _ {i} ^ {(1)} \right\rangle - s _ {2}\right) + \tilde {w} _ {i} \\ = \sum_ {i = 1} ^ {M} \tilde {w} _ {i} \psi \left(\frac {s _ {1}}{\epsilon_ {i} ^ {\prime}} \langle \hat {w} _ {1}, x \rangle - \tilde {c} _ {i} - s _ {2}\right) + \tilde {w} _ {i} \\ = \sum_ {i = 1} ^ {M} \frac {a _ {i}}{2} \left[ \psi \left(\tilde {\omega} _ {i} \langle \hat {w} _ {1}, x \rangle - \tilde {c} _ {i} - s _ {2}\right) + 1 \right], \\ \end{array}
$$
where  $\tilde{c}_i = \langle \hat{w}_1,x_i^{(1)}\rangle$  . As  $\omega_{i}\leq \tilde{\omega}_{i}$  for all  $i = 1,2,\ldots ,M$  , it follows from Lemma 6 and the construction above that
$$
\begin{array}{l} | g (x) - \hat {f} (x) | \leq | g (x) - \sum_ {i = 1} ^ {M} \frac {a _ {i}}{2} [ \psi (\tilde {\omega} _ {i} \langle \hat {w} _ {1}, x \rangle - c _ {i} - s _ {2}) + 1 ] | \\ + \left| \sum_ {i = 1} ^ {M} \frac {a _ {i}}{2} \left[ \psi \left(\tilde {\omega} _ {i} \langle \hat {w} _ {1}, x \rangle - c _ {i} - s _ {2}\right) + 1 \right] - f (x) \right| \\ + | f (x) - \hat {f} (x) | \\ <   \frac {\epsilon}{4} + \frac {\epsilon}{2} + \frac {\epsilon}{4} = \epsilon . \\ \end{array}
$$
We are now ready to prove that sampled networks with one hidden layer with tanh as activation function are universal approximators.
Theorem 6. Let  $\mathcal{F}_{1,\infty}^{S}$  be the set of all sampled networks with one hidden layer of arbitrary width and activation function  $\psi$ .  $F_{1,\infty}^{S}$  is dense in  $C(\mathcal{X},\mathbb{R}^{N_2})$ , with respect to the uniform norm.
Proof. Let  $g \in C(X, \mathbb{R}^{N_2})$ ,  $\epsilon > 0$ , and w.l.o.g.,  $N_2 = 1$ . By Theorem 4, we know there exists a network  $\Phi$ , with  $\hat{N}_1$  neurons and parameters  $\{\hat{w}_{1,i}, \hat{w}_{2,i}, \hat{b}_{1,i}, \hat{b}_{2,i}\}_{i=1}^{\hat{N}_1}$ , and ReLU as activation function, such that  $\| \Phi - g \|_{\infty} < \frac{\epsilon}{2}$ . We can then construct a new network  $\Psi$ , with  $\psi$  as activation function, where for each neuron  $\Phi^{(1,n)}$ , we construct  $M_i$  neurons in  $\Psi$ , according to Lemma 7, with  $\frac{\epsilon}{2\hat{N}_1}$ . Setting the biases in last layer of  $\Psi$  based on  $\Phi$ , i.e., for every  $i = 1, 2, \ldots, \hat{N}_1$ ,  $b_{2,j} = \frac{\hat{b}_{2,i}}{M_i}$ ,
where  $j = 1,2,\ldots ,M_{i}$ . We then have, letting  $N_{1}$  be the number of neurons in  $\Psi$ ,
$$
\begin{array}{l} | \Psi (x) - \Phi (x) | = \left| \left(\sum_ {i = 1} ^ {N _ {1}} w _ {2, i} \Psi^ {(1, i)} (x) - b _ {2, i}\right) - \left(\sum_ {i = 1} ^ {\hat {N} _ {1}} \hat {w} _ {2, i} \Phi^ {(1, i)} (x) - \hat {b} _ {2, i}\right) \right| \\ = \left| \sum_ {i = 1} ^ {\hat {N} _ {1}} \left(\sum_ {j = 1} ^ {M _ {i}} w _ {2, j} \psi (\langle w _ {1, j}, x \rangle - b _ {1, j})\right) - \hat {w} _ {2, i} \phi (\langle \hat {w} _ {1, i}, x \rangle - \hat {b} _ {1, i}) \right| \\ \leq \sum_ {i = 1} ^ {\hat {N} _ {1}} \left| \left(\sum_ {j = 1} ^ {M _ {i}} w _ {2, j} \psi (\langle w _ {1, j}, x \rangle - b _ {1, j})\right) - \hat {w} _ {2, i} \phi (\langle \hat {w} _ {1, i}, x \rangle - \hat {b} _ {1, i}) \right| \\ <   \sum_ {i = 1} ^ {\hat {N} _ {1}} \frac {\epsilon}{2 \hat {N} _ {1}} = \frac {\epsilon}{2}, \\ \end{array}
$$
for all  $x \in \mathcal{X}$ . The last inequality follows from Lemma 7. This implies that
$$
\| \Psi - g \| _ {\infty} \leq \| \Psi - \Phi \| _ {\infty} + \| \Phi - g \| _ {\infty} <   \frac {\epsilon}{2} + \frac {\epsilon}{2} = \epsilon ,
$$
and  $F_{1,\infty}^{S}$  is dense in  $C(\mathcal{X},\mathbb{R}^{N_2})$
A.2 Barron spaces
Working with neural networks and sampling makes it very natural to connect our theory to Barron spaces [2, 20]. This space of functions can be considered a continuum analog of neural networks with one hidden layer of arbitrary width. We start by considering all functions  $f\colon \mathcal{X}\to \mathbb{R}$  that can be written as
$$
f (x) = \int_ {\Omega} w _ {2} \phi (\langle w _ {1}, x \rangle - b) d \mu (b, w _ {1}, w _ {2}),
$$
where  $\mu$  is a probability measure over  $(\Omega, \Sigma_{\Omega})$ , with  $\Omega = \mathbb{R} \times \mathbb{R}^D \times \mathbb{R}$ . A Barron space  $\mathcal{B}_p$  is equipped with a norm of the form,
$$
\| f \| _ {\mathcal {B} _ {p}} = \inf  _ {\mu} \left\{\mathbb {E} _ {\mu} \left[ | w _ {2} | ^ {p} \left(\| w _ {1} \| _ {1} + | b |\right) ^ {p} \right] ^ {1 / p} \right\}, \quad 1 \leq p \leq \infty ,
$$
taken over the space of probability measure  $\mu$  over  $(\Omega, \Sigma_{\Omega})$ . When  $p = \infty$ , we have
$$
\| f\|_{\mathcal{B}_{\infty}} = \inf_{\mu}\max_{(b,w_{1},w_{2})\in \operatorname {supp}(\mu)}\{|w_{2}|(\| w_{1}\|_{1} + |b|)\} .
$$
The Barron space can then be defined as
$$
\mathcal {B} _ {p} = \{f \colon f (x) = \int_ {\Omega} w _ {2} \phi (\langle w _ {1}, x \rangle - b) d \mu (b, w _ {1}, w _ {2}) \text {a n d} \| f \| _ {\mathcal {B} _ {p}} <   \infty \}.
$$
As for any  $1 \leq p \leq \infty$ , we have  $\mathcal{B}_p = \mathcal{B}_{\infty}$ , and so we may drop the subscript  $p$  [20]. Given our previous results, we can easily show approximation bounds between our sampled networks and Barron functions.
Theorem 7. Let  $f \in \mathcal{B}$  and  $\mathcal{X} = [0,1]^D$ . For any  $N_1 \in \mathbb{N}_{>0}$ ,  $\epsilon > 0$ , and an arbitrary probability measure  $\pi$ , there exist sampled networks  $\Phi$  with one hidden layer,  $N_1$  neurons, and ReLU activation function, such that
Proof. Let  $N_{1} \in \mathbb{N}_{>0}$  and  $\epsilon > 0$ . By E et al. [20], we know there exists a network  $\hat{\Phi} \in \mathcal{F}_{1,N_1}$ , where  $\hat{\Phi}(\cdot) = \sum_{i=1}^{N_1} \hat{w}_{2,i} \phi(\langle \hat{w}_{1,i}, \cdot \rangle - \hat{b}_{1,i})$ , such that  $\|f - \hat{\Phi}\|_2^2 \leq \frac{3 \|f\|_B^2}{N_1}$ . By Theorem 4, letting
$\mathcal{X}' = [\epsilon_I, 1 - \epsilon_I]^D$ , we know there exists a network  $\Phi \in \mathcal{F}_{1,N_1}^S$ , such that  $\| \Phi - \hat{\Phi} \|_{\infty} < \sqrt{\frac{\epsilon \| f \|_B^2}{N_1}}$ . We do not need constant blocks, because  $\mathcal{X}$  is connected. We then have
$$
\begin{array}{l} \| \Phi - f \| _ {2} ^ {2} \leq \int_ {\mathcal {X}} | \Phi (x) - \hat {\Phi} (x) | ^ {2} d \pi (x) + \int_ {\mathcal {X}} | \hat {\Phi} (x) - f (x) | ^ {2} d \pi (x) \\ <   \frac {\epsilon \| f \| _ {\mathcal {B}} ^ {2}}{N _ {1}} \int_ {\mathcal {X}} d \pi (x) + \frac {3 \| f \| _ {\mathcal {B}} ^ {2}}{N _ {1}} = \frac {(3 + \epsilon) \| f \| _ {\mathcal {B}} ^ {2}}{N _ {1}}. \\ \end{array}
$$
A.3 Distribution of sampled networks
In this section we prove certain invariance properties for sampled networks and our proposed distribution. First, we define the distribution to sample the pair of points, or equivalently, the parameters. Note that  $\mathcal{X}$  is now any compact subset of  $\mathbb{R}^D$ , as long as  $\lambda_D(\mathcal{X}) > 0$ , where  $\lambda_D$  the  $D$ -dimensional Lebesgue measure.
As we are in a supervised setting, we assume access to values of the true function  $f$ , and define  $\mathcal{Y} = f(\mathcal{X})$ . We also choose a norm  $\| \cdot \|_{\mathcal{Y}}$  over  $\mathbb{R}^{N_{L + 1}}$ , and norms  $\| \cdot \|_{\mathcal{X}_{l - 1}}$  over  $\mathbb{R}^{N_{l - 1}}$ , for each  $l = 1,2,\ldots ,L$ . For the experimental part of the paper, we choose the  $L^{\infty}$  norm for  $\| \cdot \|_{\mathcal{Y}}$  and for  $l = 1,2,\ldots ,L$ , we choose the  $L^2$  norm for  $\| \cdot \|_{\mathcal{X}_{l - 1}}$ . We also denote  $\bar{N} = \sum_{l = 1}^{L}N_{l}$  as the total number of neurons in a given network, and  $\bar{N}_l = \sum_{k = 1}^l N_k$ . Due to the nature of sampled networks and because we sample each layer sequentially, we start by giving a more precise definition of the conditional density given in Definition 2. As a pair of points from  $\mathcal{X}$  identifies a weight and bias, we need a distribution over  $\mathcal{X}^{2\bar{N}}$ , and for each layer  $l$  condition on sets of  $2\bar{N}_{l - 1}$  points, which then parameterize the network  $\Phi^{(l - 1)}$  by constructing weights and biases according to Definition 4.
Definition 7. Let  $\mathcal{X}$  be compact,  $\lambda_D(\mathcal{X}) > 0$ , and  $f\colon \mathbb{R}^D\to \mathbb{R}^{N_{L + 1}}$  be Lipschitz-continuous w.r.t. the metric spaces induced by  $\| \cdot \|_{\mathcal{Y}}$  and  $\| \cdot \|_{\mathcal{X}}$ . For any  $l\in \{1,2,\dots ,L\}$ , setting  $\epsilon = 0$  when  $l = 1$  and otherwise  $\epsilon >0$ , we define
$$
q _ {l} ^ {\epsilon} \left(x _ {0} ^ {(1)}, x _ {0} ^ {(2)} \mid X _ {l - 1}\right) = \left\{ \begin{array}{l l} \frac {\| f (x _ {0} ^ {(2)}) - f (x _ {0} ^ {(1)}) \| _ {\mathcal {Y}}}{\max  \{\| x _ {l - 1} ^ {(2)} - x _ {l - 1} ^ {(1)} \| _ {\mathcal {X} _ {l - 1}}, \epsilon \}}, & x _ {l - 1} ^ {(1)} \neq x _ {l - 1} ^ {(2)} \\ 0, & o t h e r w i s e, \end{array} \right.
$$
where  $x_0^{(1)}, x_0^{(2)} \in \mathcal{X}$ ,  $x_{l-1}^{(1)} = \Phi^{(l-1)}(x_0^{(1)})$ , and  $x_{l-1}^{(2)} = \Phi^{(l-1)}(x_0^{(2)})$ , with the network  $\Phi^{(l-1)}$  parameterized by pairs of points in  $\mathcal{X}^{\bar{N}_{l-1}}$ . Then, we define the integration constant  $C_l = \int_{\mathcal{X} \times \mathcal{X}} q_l^\epsilon d\lambda$ . The  $l$ -layered density  $p_l^\epsilon$  is defined as
$$
p _ {l} ^ {\epsilon} = \left\{ \begin{array}{l l} \frac {q _ {l} ^ {\epsilon}}{C _ {l}}, & \text {i f} C _ {l} > 0 \\ \frac {1}{\lambda_ {2 D} (\mathcal {X} \times \mathcal {X})}, & \text {o t h e r w i s e .} \end{array} \right. \tag {6}
$$
Remark 6. The added  $\epsilon$  is there to ensure the density is bounded, but is not needed when considering the first layer, due to the Lipschitz assumption. Adding  $\epsilon > 0$  for  $l = 1$  is both unnecessary and affects equivariant/invariant results in Theorem 8. We drop the  $\epsilon$  superscript wherever it is unambiguously included.
We can now use this definition to define the probability of the whole parameter space  $\mathcal{X}^{2\bar{N}}$ , i.e., given an architecture provide a distribution over all weights and biases the network require. Let  $\bar{\mathcal{X}} = \mathcal{X}^{2\bar{N}}$ , i.e., the sampling space of  $P$  with the product topology, and  $\bar{D} = 2\bar{N}D$  as the dimension of the space. Since all the parameters is defined through points of  $\mathcal{X}\times \mathcal{X}$ , we may choose an arbitrary ordering of the points, which means one set of weights and biases for the whole network can be written as  $\{x_{i}^{(1)},x_{i}^{(2)}\}_{i = 1}^{\bar{N}}$ .
Definition 8. The probability distribution  $\rho_f$  over  $\bar{\mathcal{X}}$  have density  $p$ ,
$$
p \left(x _ {1} ^ {(1)}, x _ {1} ^ {(2)}, x _ {2} ^ {(1)}, x _ {2} ^ {(2)}, \dots , x _ {\bar {N}} ^ {(1)}, x _ {\bar {N}} ^ {(2)}\right) = \prod_ {l = 1} ^ {L} \prod_ {i = 1} ^ {N _ {l}} p _ {l} \left(x _ {\bar {N} _ {l - 1} + i} ^ {(1)}, x _ {\bar {N} _ {l - 1} + i} ^ {(2)} \mid X _ {l - 1}\right),
$$
with  $X_{l - 1} = \bigcup_{k = 1}^{l - 1}\bigcup_{j = 1}^{N_k}\{x_{\bar{N}_k + j}^{(1)},x_{\bar{N}_k + j}^{(2)}\}$
It is not immediately clear from above that  $P$  is valid distribution, and in particular, that the density is integrable. This is what we show next.
Proposition 1. Let  $\mathcal{X} \subseteq \mathbb{R}^D$  be compact,  $\lambda_D(\mathcal{X}) > 0$ , and  $f$  be Lipschitz continuous w.r.t. the metric spaces induced by  $\| \cdot \|_{\mathcal{Y}}$  and  $\| \cdot \|_{\mathcal{X}}$ . For fixed architecture  $\{N_l\}_{l=1}^L$ , the proposed function  $p$  is integrable and
$$
\int_ {\bar {\mathcal {X}}} p d \lambda_ {\bar {D}} = 1.
$$
It therefore follows that  $P$  is a valid probability distribution.
Proof. We will show for each  $l = 1,2,\dots ,L$  that  $p_l$  is bounded a.e. and is nonzero for at least one subset with nonzero measure. There exist  $A\subseteq \mathcal{X}\times \mathcal{X}$  such that  $p_{l}(A)\neq 0$  and  $\lambda_{2D}(A) > 0$ , as either  $C_l > 0$  or  $p_l$  is the uniform density by Definition 7 and  $\lambda_{2D}(\mathcal{X}\times \mathcal{X}) > 0$  by assumption and the product topology.
For  $l = 1$ , let  $K_{l} > 0$  be the Lipschitz constant. Then  $q_{l}(x^{(1)}, x^{(2)})$  by assumption of  $f$  for all  $x^{(1)}, x^{(2)} \in \mathcal{X}$ . When  $l > 1$ , for all  $X_{l-1} \in \mathcal{X}^{2\bar{N}_{l-1}}$ , we have that there exist a constant  $K_{l} > 0$  due to continuity and compactness, such that
$$
p _ {l} \left(x ^ {(1)}, x ^ {(2)} \mid X _ {l}\right) \leq \max  \left\{\frac {K _ {l}}{\epsilon}, \frac {1}{\lambda_ {2 D} (\mathcal {X} \times \mathcal {X})} \right\}.
$$
As  $p$  is a multiplication of a finite set of elements, we end up with
$$
\begin{array}{l} 0 <   \int_ {\bar {\mathcal {X}}} p d \lambda <   \int_ {\bar {\mathcal {X}}} \prod_ {l = 1} ^ {L} N _ {l} \left(\frac {K _ {l}}{\epsilon} + K _ {l} + \frac {1}{\lambda_ {2 D} (\mathcal {X} \times \mathcal {X})}\right) d \lambda_ {\bar {D}} \\ <   L \max \left\{N _ {l} \left(\frac {K _ {l}}{\epsilon} + K _ {l} + \frac {1}{\lambda_ {2 D} (\mathcal {X} \times \mathcal {X})}\right) \right\} _ {l = 1} ^ {L} \lambda_ {\bar {D}} (\bar {\mathcal {X}}) <   \infty . \\ \end{array}
$$
using the fact that  $0 < \lambda_{D}(\mathcal{X}) < \infty$  and  $\lambda_{\bar{D}}$  being the product measure, implies  $0 < \lambda_{\bar{D}}(\bar{\mathcal{X}}) < \infty$ . Due to the normalization constants added to  $q_{l}$ , we see  $p_l$  integrates to one. This means  $P$  is a valid distribution of  $\bar{\mathcal{X}}$ , with implied independence between the neurons in the same layer.
One special property of sampled networks, and in particular of the distribution  $P$ , is their invariance under both linear isometries and translation (together forming the Euclidean group, i.e., rigid body transformations), as well as scaling. We denote the set of possible transformations as  $\mathcal{H}(D) = \mathbb{R} \setminus \{0\} \times O(D) \times \mathbb{R}$ , with  $O(D)$  being the orthogonal group of dimension  $D$ . We then denote  $(a, A, c) = H \in \mathcal{H}(D)$  as  $H(x) = aAx + c$ , where  $x \in \mathcal{X}$ . The space  $\mathcal{H}_f(D) \subseteq \mathcal{H}$  are all transformations such that  $f: \mathcal{X} \rightarrow \mathbb{R}^{N_{L+1}}$  is equivariant with respect to the transformations, with the underlying metric space given by  $\|\cdot\|_{\mathcal{Y}}$ . That is, for any  $H \in \mathcal{H}_f(D)$ , there exists a  $H' \in \mathcal{H}(N_{L+1})$  such that  $f(H(x)) = H'(f(x))$ , where  $x \in \mathcal{X}$ , and the orthogonal matrix part of  $H'$  is isometric w.r.t.  $\|\cdot\|_{\mathcal{Y}}$ . Note that often the  $H'$  will be the identity transformation, for example by having the same labels for the transformed data. When  $H'$  is the identity function, we say  $f$  is invariant with respect to  $H$ . In the next result, we assume we choose norms  $\|\cdot\|_{\mathcal{X}_0}$  and  $\|\cdot\|_{\mathcal{Y}}$ , such that the orthogonal matrix part of  $H$  is isometric w.r.t. those norms and the canonical norm  $\|\cdot\|$ , as well as continue the assumption of Lipschitz-continuous  $f$ .
Theorem 8. Let  $H \in \mathcal{H}_f(D)$ ,  $\Phi, \hat{\Phi}$  be two sampled networks with the same number of layers  $L$  and neurons  $N_1, \ldots, N_L$ , where  $\Phi: \mathcal{X} \to \mathbb{R}^{N_{L+1}}$  and  $\hat{\Phi}: H(\mathcal{X}) \to \mathbb{R}^{N_{L+1}}$ , and  $f: \mathcal{X} \to \mathbb{R}^{N_{L+1}}$  is the true function. Then the following statements hold:
$$
\Phi^ {(1)} (x) = \hat {\Phi} ^ {(1)} (H (x)), f o r a l l x \in \mathcal {X}.
$$
(1) If  $\hat{x}_{0,i}^{(1)} = H(x_{0,i}^{(1)})$  and  $\hat{x}_{0,i}^{(2)} = H(x_{0,i}^{(2)})$ , for all  $i = 1,2,\ldots,N_1$ , then
(2) If  $f$  is invariant w.r.t.  $H$ :  $\Phi \in \mathcal{F}_{L,[N_1,\ldots N_L]}^S(\mathcal{X})$  if and only if  $\hat{\Phi} \in \mathcal{F}_{L,[N_1,\ldots N_L]}^S(H(\mathcal{X}))$ , such that  $\Phi(x) = \hat{\Phi}(x)$  for all  $x \in \mathcal{X}$ .
(3) The probability measure  $P$  over the parameters is invariant under  $H$ .
Proof. Let  $H = (a, A, c)$ . Assume we have sampled  $\hat{x}_{0,i}^{(1)} = H(x_{0,i}^{(1)})$  and  $\hat{x}_{0,i}^{(2)} = H(x_{0,i}^{(2)})$ , for all  $i = 1, 2, \ldots, N_1$ . The points sampled determine the weights and biases in the usual way, giving
$$
\begin{array}{l} \phi \left(\left\langle w _ {1, i}, x - x _ {0, i} ^ {(1)} \right\rangle\right) = \phi \left(\left\langle \frac {x _ {0 , i} ^ {(2)} - x _ {0 , i} ^ {(1)}}{\| x _ {0 , i} ^ {(2)} - x _ {0 , i} ^ {(1)} \| ^ {2}}, x - x _ {0, i} ^ {(1)} \right\rangle\right) \\ = \phi \left(\left\langle A \frac {x _ {0 , i} ^ {(2)} - x _ {0 , i} ^ {(1)}}{\| A \left(x _ {0 , i} ^ {(2)} - x _ {0 , i} ^ {(1)}\right) \| ^ {2}}, A \left(x - x _ {0, i} ^ {(1)}\right) \right\rangle\right) \\ = \frac {a \cdot a}{a ^ {2}} \phi \left(\left\langle A \frac {x _ {0 , i} ^ {(2)} - x _ {0 , i} ^ {(1)}}{\| A (x _ {0 , i} ^ {(2)} - x _ {0 , i} ^ {(1)}) \| ^ {2}}, A (x - x _ {0, i} ^ {(1)}) \right\rangle\right) \\ = \phi \left(\left\langle a A \frac {x _ {0 , i} ^ {(2)} - x _ {0 , i} ^ {(1)}}{\| a A (x _ {0 , i} ^ {(2)} - x _ {0 , i} ^ {(1)}) \| ^ {2}}, a A (x - x _ {0, i} ^ {(1)}) \right\rangle\right) \\ = \phi \left(\left\langle a A \frac {x _ {0 , i} ^ {(2)} + c - x _ {0 , i} ^ {(1)} - c}{\| a A (x _ {0 , i} ^ {(2)} + c - x _ {0 , i} ^ {(1)} - c) \| ^ {2}}, a A (x + c - x _ {0, i} ^ {(1)} - c) \right\rangle\right) \\ = \phi \left(\left\langle \frac {H \left(x _ {0 , i} ^ {(2)}\right) - H \left(x _ {0 , i} ^ {(1)}\right)}{\| H \left(x _ {0 , i} ^ {(2)}\right) - H \left(x _ {0 , i} ^ {(1)}\right) \| ^ {2}}, H (x) - H \left(x _ {0, i} ^ {(1)}\right) \right\rangle\right) \\ = \phi \left(\left\langle \hat {w} _ {1, i}, \hat {x} - \hat {x} _ {0, i} ^ {(1)} \right\rangle\right) \\ \end{array}
$$
for all  $x\in \mathcal{X},\hat{x}\in H(\mathcal{X})$  , and  $i = 1,2,\dots ,N_{1}$  . Which implies that (1) holds.
Assuming  $f$  is invariant w.r.t.  $H$ , then for any  $\Phi \in \mathcal{F}_{L,[N_1,\ldots N_L]}^S(\mathcal{X})$ , let  $X = \{x_{1,i}^{(1)}, x_{1,i}^{(2)}\}_{i=1}^{N_1}$ , we can then choose  $H(X)$  as points to construct weights and biases in the first layer of  $\hat{\Phi}$ , and by (1), we have  $\mathcal{X}_1 = \Phi^{(1)}(\mathcal{X}) = \hat{\Phi}^{(1)}(H(\mathcal{X})) = \hat{\mathcal{X}}_1$ . As the sampling space is the same for the next layer, we see that we can choose the points for the weights and biases of  $\hat{\Phi}$ , such that  $\mathcal{X}_l = \hat{\mathcal{X}}_l$ , where  $l = 1,2,\dots,L$ . As the input after the final hidden layer is also the same, by the same argument, means the weights in the last layer must be the same, due to the loss function  $\mathcal{L}$  in Definition 4 is the same due to the invariance assumption. Thus,  $\Phi(x) = \hat{\Phi}(H(x))$  for all  $x \in \mathcal{X}$ . As  $H$  is bijective, means the same must hold true starting with  $\hat{\Phi}$ , and constructing  $\Phi$ , and we conclude that (2) holds.
For any distinct points  $x^{(1)}, x^{(2)} \in \mathcal{X}$ , letting  $(a', A', c')$  be the set such that  $g(aAx + c) = a'A'g(x) + c'$ , and  $\hat{C}_l, C_l$  be the normalization constants over the conditional density  $p_l$ , for  $H(\mathcal{X})$  and  $\mathcal{X}$  resp. We have for the conditional density when  $l = 1$  is,
$$
\begin{array}{l} p _ {1} \left(H \left(x ^ {(1)}\right), H \left(x ^ {(2)}\right)\right) = \frac {1}{\hat {C} _ {1}} \frac {\| f \left(a A x ^ {(2)} + c\right) - f \left(a A x ^ {(1)} + c\right) \| _ {\mathcal {Y}}}{\| a A x ^ {(2)} + c - a A x ^ {(1)} - c \| _ {\mathcal {X}}} \\ = \frac {1}{\hat {C} _ {1}} \frac {\| a ^ {\prime} A ^ {\prime} f (x ^ {(2)}) + c ^ {\prime} - a ^ {\prime} A ^ {\prime} f (x ^ {(1)}) - c ^ {\prime} \| _ {\mathcal {Y}}}{\| a A x ^ {(2)} - a A x ^ {(1)} \| _ {\mathcal {X}}} \\ = \frac {\left| a ^ {\prime} \right|}{\left| a \right| \hat {C} _ {1}} \frac {\left\| f (x ^ {(2)}) - f (x ^ {(1)}) \right\| \mathcal {Y}}{\left\| x ^ {(2)} - x ^ {(1)} \right\| \mathcal {X}} = \frac {\left| a ^ {\prime} \right|}{\left| a \right| \hat {C} _ {1}} p _ {1} (x ^ {(1)}, x ^ {(2)}). \\ \end{array}
$$
With similar calculations, we have
$$
\frac {| a |}{| a ^ {\prime} |} \hat {C} _ {1} = \frac {| a |}{| a ^ {\prime} |} \int_ {\mathcal {X} \times \mathcal {X}} p _ {1} (H (x), H (z)) d x d z = \frac {| a |}{| a ^ {\prime} |} \frac {| a ^ {\prime} |}{| a |} \int_ {\mathcal {X} \times \mathcal {X}} p _ {1} (x, z) d x d z = C _ {1}.
$$
Hence, the conditional probability distribution for the first layer is invariant under  $H$ , and then by (1) and (2), the possible sampling spaces are equal for the following layers, and therefore the conditional distributions for each layer is the same, and therefore  $P$  is invariant under  $H$ .
Remark 7. Point (2) in the theorem requires  $f$  to be invariant w.r.t.  $H$ . This is due to the fact that the parameters in the last layer minimize the implicitly given loss function  $\mathcal{L}$ , seen in Definition 4. We have rarely discussed the loss function, as it depends on what function we are approximating.
Technically, (2) also holds as long as the loss function is not altered by  $H$  in a way that the final layer alters, but we simplified it to be invariant, as this is also the most likely case to stumble upon in practice. The loss function also appears in the proofs given in Appendix A.1 and Appendix A.2. Here both uses, implicitly, the loss function based on the uniform norm.
B Illustrative example: approximating a Barron function
All computations for this experiment were performed on the Intel Core i7-7700 CPU @ 3.60GHz  $\times$  8 with 32GB RAM.
We compare random Fourier features and our sampling procedure on a test function for neural networks [64]:  $f(x) = \sqrt{3/2}(\|x - a\|_{\ell^2} - \|x + a\|_{\ell^2})$ , with the constant vector  $a \in \mathbb{R}^d$  defined by  $a_j = 2j / d - 1$ . For all dimensions  $d$ , we sample 10,000 points uniformly at random in the cube  $[-1,1]^d$  for training (i.e., sampling and solving the last, linear layer problem), and another 10,000 points for evaluating the error. We re-run the same experiment with five different random seeds, but with the same train/test datasets. This means the random seed only influences the weight distributions in sampled networks and random feature models, not the data. Figure 8 shows the relative  $L^2$  error in the full hyperparameter study, i.e. in dimensions  $d \in \{1,2,3,4,5,10\}$  (rows) and for tanh (left column) and sine activation functions (right column). For the random feature method, we always use sine activation, because we did not find any data-agnostic probability distributions for tanh. Figure 9 shows the fit times for the same experiments, demonstrating that sampling networks are not slower than random feature models. This is obvious from the complexity analysis in Appendix F, and confirmed experimentally here. Interesting observations regarding the full hyperparameter study are that for one dimension,  $d = 1$ , random feature models outperform sampled networks and can even use up to two hidden layers. In higher dimensions, and with larger widths / more layers, sampled networks consistently outperform random features. In  $d = 10$ , a sampled network with even a single hidden layer is about one order of magnitude more accurate than the same network with normally distributed weights. The convergence rate of the error of sampled networks with respect to increasing layer widths is consistent over all dimensions and layers, even sometimes outperforming the theoretical bound of  $O(N^{-1/2})$ .
C Classification benchmark from OpenML