SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} w _ {2, i} \phi (\langle w _ {1, i}, x - x _ {0, i} ^ {(1)} \rangle) - b _ {2, i} = \| \hat {w} _ {1, i} \| \hat {w} _ {2, i} \phi \left(\left\langle \frac {\hat {w} _ {1 , i}}{\| \hat {w} _ {1 , i} \|}, x - x _ {i} \right\rangle\right) - \hat {b} _ {2, i} \\ = \hat {w} _ {2, i} \phi (\left\langle \hat {w} _ {i}, x - x _ {i} \right\rangle) - \hat {b} _ {2, i}, \\ \end{array}
$$
where last equality follows by  $\phi$  being positive homogeneous.
If  $x_{i} \in \partial \mathcal{X}$ , by continuity we find  $\delta > 0$  such that for all  $i = 1,2,\dots,N_1$  and  $x',x \in \mathcal{X}$ , where  $\| x' - x_i \| < \delta$ , we have
$$
| \phi (\langle \hat {w} _ {1, i}, x - x ^ {\prime} \rangle) - \phi (\langle \hat {w} _ {1, i}, x - x _ {i} \rangle) | <   \frac {\epsilon}{N _ {1} w _ {2}},
$$
where  $w_{2} = \max \{| \hat{w}_{2,i} | \}_{i=1}^{N_{1}}$ . We set  $x_{0,i}^{(1)} = \eta(x_{i}, \min \{\delta, 1\})$ , with  $x_{0,i}^{(1)} \in \operatorname{int} \mathcal{X}$  and  $\| x_{i} - x_{0,i}^{(1)} \| < \delta$ , due to  $\delta > 0$  and Lemma 1. We may now proceed by constructing  $x_{0,i}^{(2)}$  as above, and similarly setting  $w_{2,i} = \| \hat{w}_{1,i} \| \hat{w}_{2,i}$ ,  $b_{2,i} = \hat{b}_{2,i}$ , we have
$$
\begin{array}{l} \left| \left(\sum_ {i = 1} ^ {N _ {1}} w _ {2, i} \phi (\langle w _ {1, i}, x - x _ {0, i} ^ {(1)} \rangle) - b _ {2, i}\right) - \left(\sum_ {i = 1} ^ {N _ {1}} \hat {w} _ {2, i} \phi (\langle \hat {w} _ {1, i}, x - x _ {i} \rangle) - \hat {b} _ {2, i}\right) \right| \\ = \left| \sum_ {i = 1} ^ {N _ {1}} \hat {w} _ {2, i} \left(\phi \left(\left\langle \hat {w} _ {1, i}, x - x _ {0, i} ^ {(1)} \right\rangle\right) - \phi \left(\left\langle \hat {w} _ {1, i}, x - x _ {i} \right\rangle\right)\right) \right| \\ \leq \sum_ {i = 1} ^ {N _ {1}} \left| w _ {2} \left(\phi (\langle \hat {w} _ {1, i}, x - x _ {0, i} ^ {(1)} \rangle) - \phi (\langle \hat {w} _ {1, i}, x - x _ {i} \rangle)\right) \right| \\ <   \left| \sum_ {i = 1} ^ {N _ {1}} w _ {2} \frac {\epsilon}{N _ {1} w _ {2}} \right| = \epsilon , \\ \end{array}
$$
and thus  $\| \Phi -\hat{\Phi}\|_{\infty} <   \epsilon$
Until now, we have worked with weights of the form  $w = \frac{x^{(2)} - x^{(1)}}{\|x^{(2)} - x^{(1)}\|}$ , however, the weights in a sampled network are divided by the norm squared, not just the norm. We now show that for all the results so far, and also for any other results later on, differing by a positive scalar (such as this norm) is irrelevant when ReLU is the activation function.
Lemma 4. Let  $\Phi$  be a network with one hidden layer, with weights and biases of the form
$$
w _ {1, i} = \frac {x _ {0 , i} ^ {(2)} - x _ {0 , i} ^ {(1)}}{\| x _ {0 , i} ^ {(2)} - x _ {0 , i} ^ {(1)} \|}, \quad b _ {1, i} = \langle w _ {1, i}, x _ {0, i} ^ {(1)} \rangle ,
$$
for  $i = 1,2,\ldots ,N_{1}$ . For any weights and biases in the last layer,  $\{w_{2,i},b_{2,i}\}_{i = 1}^{N_2}$ , and set of strictly positive scalars  $\{\omega_i\}_{i = 1}^{N_1}$ , there exist sampled networks  $\Phi_{\omega}$  where weights and biases in the hidden layer  $\{w_{1,i}',b_{1,i}'\}_{i = 1}^{N_1}$  are of the form
$$
w _ {1, i} ^ {\prime} = \omega_ {i} w _ {1, i}, \quad b _ {1, i} ^ {\prime} = \langle w _ {1, i} ^ {\prime}, x _ {0, i} ^ {(1)} \rangle ,
$$
such that  $\Phi_{\omega}(x) = \Phi (x)$  for all  $x\in \mathcal{X}$
Proof. We set  $w_{2,i}' = \frac{w_{2,i}}{\omega_i}$  and  $b_{2,i}' = b_{2,i}$ . As ReLU is a positive homogeneous function, we have for all  $x \in \mathcal{X}$ ,
$$
\begin{array}{l} w _ {2, i} ^ {\prime} \phi (\langle w _ {1, i} ^ {\prime}, x \rangle - b _ {1, i} ^ {\prime}) - b _ {1, i} ^ {\prime} = \frac {\omega_ {i} w _ {2 , i}}{\omega_ {i}} \phi (\langle w _ {1, i}, x \rangle - b _ {1, i}) \\ w _ {2, i} \phi (\langle w _ {1, i}, x \rangle - b _ {1, i}). \\ \end{array}
$$
The result itself is not too exciting, but it allows us to use the results proven earlier, applying them to sampled networks by setting the scalar to be  $\omega_{i} = \frac{1}{\|x_{0,i}^{(2)} - x_{0,i}^{(1)}\|}$ .
We are now ready to show the universal approximation property for sampled networks with one hidden layer. We let  $\mathcal{F}_{1,\infty}^{S}$  be defined similarly to  $\mathcal{F}_{1,\infty}$ , with every  $\Phi \in \mathcal{F}_{1,\infty}^{S}$  being a sampled neural network.
Theorem 4. Let  $g \in C(\mathcal{X},\mathbb{R}^{N_L + 1})$ . Then, for any  $\epsilon >0$ , there exist  $\Phi \in \mathcal{F}_{1,\infty}^S$  with ReLU activation function, such that
$$
\| g - \Phi \| _ {\infty} <   \epsilon .
$$
That is,  $\mathcal{F}_{1,\infty}^{S}$  is dense in  $C(\mathcal{X},\mathbb{R}^{N_{L + 1}})$ .
Proof. W.l.o.g., let  $N_{L + 1} = 1$ , and in addition, let  $\epsilon > 0$  and  $g \in C(\mathcal{X},\mathbb{R}^{N_{L + 1}})$ . Using the universal approximation theorem of Pinkus [52], we have that for  $\epsilon > 0$ , there exist a network  $\hat{\Phi} \in \mathcal{F}_{1,\infty}$  such that  $\| g - \hat{\Phi}\|_{\infty} < \frac{\epsilon}{2}$ . Let  $K$  be the number of neurons in  $\hat{\Phi}$ . We then create a new network  $\Phi$  by first keeping the weights fixed to the original ones,  $\{\hat{w}_{1,i},\hat{w}_{2,i}\}_{i = 1}^{K}$ , and setting the biases of  $\Phi$  according to Lemma 2 using  $\{\hat{b}_{1,i},\hat{b}_{2,i}\}_{i = 1}^{K}$ , adding constant blocks if necessary. We then change the weights of our  $\Phi$  with the respect to the new biases, according to Lemma 3 (with the epsilon set to  $\epsilon /2$ ). It follows from the two lemmas that  $\| \Phi -\hat{\Phi}\|_{\infty} < \frac{\epsilon}{2}$ . That means
$$
\left\| g - \Phi \right\| _ {\infty} = \left\| g - \hat {\Phi} + \hat {\Phi} - \Phi \right\| _ {\infty} \leq \left\| g - \hat {\Phi} \right\| _ {\infty} + \left\| \hat {\Phi} - \Phi \right\| _ {\infty} <   \epsilon .
$$
As the weights of the first layer of  $\Phi$  can be written as  $w_{1,i} = \frac{x_{0,i}^{(2)} - x_{0,i}^{(1)}}{\|x_{0,i}^{(2)} - x_{0,i}^{(1)}\|^2}$  and bias  $b_{1,i} = \langle w_{1,i},x_{0,i}^{(1)}\rangle$  both guaranteed by Lemma 4, means  $\Phi \in \mathcal{F}_{1,\infty}^S$ . Thus,  $\mathcal{F}_{1,\infty}^S$  is dense in  $C(\mathcal{X},\mathbb{R}^{N_L + 1})$ .
Remark 4. By the same two lemmas, Lemma 2 and Lemma 3, one can show that other results regarding networks with one hidden layer with at most  $K$  neurons, also hold for sampled networks, but with  $6 \cdot K$  neurons, due to the possibility of one constant block for each neuron. When  $\mathcal{X}$  is a connected set, we only need  $K$  neurons, as no additional constant blocks must be added; see proof of Lemma 2 for details.
A.1.3 Deep networks
The extension of sampled networks into several layers is not obvious, as the choice of weights in the first layer affects the sampling space for the weights in the next layer. This additional complexity raises the question, letting  $N_{L} = [N_{1},N_{2},\ldots ,N_{L - 1},N_{L}]$ , is the space  $\bigcup_{N_L = 1}^{\infty}\mathcal{F}_{L,N_L}^{S}$  dense in  $C(\mathcal{X},\mathbb{R}^{N_L + 1})$ , when  $L > 1$ ? We aim to answer this question in this section. With dimensions in each layer being  $N_{L} = [D,D,\dots ,D,N_{L}]$ , we start by showing it holds for  $\bigcup_{N_L = 1}^{\infty}\mathcal{F}_{L,N_L}^{S}$ , i.e., the space of sampled networks with  $L$  hidden layers, and  $D$  neurons in the first  $L - 1$  layers and arbitrary width in the last layer.
Lemma 5. Let  $N_L = [D, D, \ldots, D, N_L]$  and  $L > 1$ . Then  $\bigcup_{N_L = 1}^{\infty} \mathcal{F}_{L,N_L}^S$  is dense in  $C(\mathcal{X},\mathbb{R}^{N_{L + 1}})$ .
Proof. Let  $\epsilon > 0$ , and  $g \in C(\mathcal{X}, \mathbb{R}^{N_{L+1}})$ . Basic linear algebra implies there exists a set of  $D$  linearly independent unit vectors  $v = \{v_j\}_{j=1}^D$ , such that  $\mathcal{X} \subseteq \operatorname{span}\{v_j\}_{j=1}^D$ , with  $v_j \in \mathcal{X}$  for all  $v_j \in v$ . For  $l = 1$  and  $i \in \{1,2,\ldots,N_l\}$ , the bias is set to  $b_{l,i} = \inf \{\langle v_i, x \rangle : x \in \mathcal{X}\}$ . Due to continuity and compactness, we can set  $x_{l-1,i}^{(1)}$  to correspond to an  $x \in \mathcal{X}$  such that  $\langle v_i, x_{l-1,i}^{(1)} \rangle = b_{l,i}$ . We must have  $x_{l-1,i}^{(1)} \in \partial \mathcal{X}$  — otherwise the inner product between some points in  $\mathcal{X}$  and  $v_i$  is smaller than the bias, which contradicts the construction of  $b_{l,i}$ . We now need to show that  $\zeta(x_{l-1,i}^{(1)}) - x_{l-1,i}^{(1)} = c v_i + x_{l-1,i}^{(1)}$  for  $c \in \mathbb{R}_{>0}$ , to proceed constructing  $x_{l-1,i}^{(2)}$  in similar fashion as in Lemma 3.
Let  $U = \overline{\mathbb{B}}_{\epsilon_I}(\zeta(x_{l-1,i}^{(1)}))$ , with  $x_{l-1,i}^{(1)} \in U$ . Also,  $U$  is the only closed ball with center  $\zeta(x_{l-1,i}^{(1)})$  and  $\{x_{l-1,i}^{(1)}\} = U \cap \partial \mathcal{X}$  — otherwise  $\zeta$  would not give a unique element, which is guaranteed by Definition 5. We define the hyperplane  $\mathrm{Ker} = \{x \in \mathbb{R}^D : \langle v_i, x \rangle - \langle v_i, x_{l-1,i}^{(1)} \rangle = 0\}$ , and the projection matrix  $P$  from  $\mathbb{R}^D$  onto  $\mathrm{Ker}$ . Let  $x' = P\zeta(x_{l-1,i}^{(1)})$ . If  $x' \neq x_{l-1,i}^{(1)}$ , then  $x' \in \mathrm{int}U$ , as  $x_{l-1,i}^{(1)} \in \partial U$  and  $P$  is a unique projection minimizing the distance. As  $x' \in \mathrm{int}U$ , means there is an open ball around  $x'$ , where there are points in  $\overline{\mathbb{B}}_{\epsilon_I}(\zeta(x_{l-1,i}^{(1)}))$  separated by  $\mathrm{Ker}$ , which implies  $\langle v_i, x_{l-1,i}^{(1)} \rangle$  is not the minimum. This is a contradiction, and hence  $x' = x_{l-1,i}^{(1)}$ . This means  $\mathrm{Ker}$  is a tangent hyperplane, and as any vectors along the hyperplane is orthogonal to  $\zeta(x_{l,i}^{(1)}) - x_{l,i}^{(1)}$ , implies the angle between  $v_i$  and  $\zeta(x_{l-1,i}^{(1)}) - x_{l-1,i}^{(1)}$  is 0 or  $\pi$ . As  $\langle v_i, x_{l-1,i}^{(1)} \rangle$  is the minimum, means there exist a  $c \in \mathbb{R}_{>0}$ , such that  $\zeta(x_{l-1,i}^{(1)}) - x_{l-1,i}^{(1)} = cv_i + x_{l-1,i}^{(1)}$ .
We may now construct  $x_{l-1,i}^{(2)} = x_{l-1,i}^{(1)} + cv_i$ , assured that  $x_{l-1,i}^{(2)} \in \mathcal{X}$  due to the last paragraph. We then have  $w_{l,i} = \frac{v_i}{\|v_i\|}$  and  $b_{l,i} = \langle v_i, x_{l-1,i}^{(1)} \rangle$  for all neurons in the first hidden layer. The image
after this first hidden layer  $\Psi^{(l)}(\mathcal{X})$  is injective, and therefore bijective. To show this, let  $u_{1}, u_{2} \in \mathcal{X}$  such that  $u_{1} \neq u_{2}$ . As the vectors in  $v$  spans  $\mathcal{X}$ , means there exists two unique sets of coefficients,  $c_{1}, c_{2} \in \mathbb{R}^{D}$ , such that  $u_{1} = \sum c_{1}^{(i)} v_{i}$  and  $u_{2} = \sum c_{2}^{(i)} v_{i}$ . We then have
$$
\Phi^ {(l)} \left(u _ {1}\right) = V c _ {1} - b _ {l} \neq V c _ {2} - b _ {l} = \Phi^ {(l)} \left(u _ {2}\right),
$$
where  $V$  is the Gram matrix of  $v$ . As vectors in  $v$  are linearly independent,  $V$  is positive definite, and combined with  $c_{1} \neq c_{2}$ , this implies the inequality. That means  $\Phi^{(l)}$  is a bijective mapping of  $\mathcal{X}$ . As the mapping is bijective and continuous, we have that for any  $x \in \mathcal{X}'$ , there is an  $\epsilon_{I}^{(l)} > 0$ , such that  $\overline{\mathbb{B}}_{\epsilon_{I}^{\prime}}(\Phi^{(l)}(x)) \subseteq \mathcal{X}$ .
For  $1 < l < L$ , we repeat the procedure, but swap  $\mathcal{X}$  with  $\mathcal{X}_{l-1}$ . As  $\Phi^{(l-1)}$  is a bijective mapping, we may find similar linear independent vectors and construct similar points  $x_{l-1,i}^{(1)}, x_{l-1,i}^{(2)}$ , but now with noise level  $\epsilon_I^{(l)}$ . For  $l = L$ , as we have a subset of  $\mathcal{X}$  that is a closed ball around each point in  $\Phi^{l-1}(x)$ , for every  $x \in \mathcal{X}'$ , means we can proceed by constructing the last hidden layer and the last layer in the same way as explained when proving Theorem 4. The only caveat is that we are approximating a network with one hidden layer with the domain  $\mathcal{X}_{l-1}$ , and the function we approximate is  $\tilde{g} = g \circ [\Phi^{(L-1)}]^{-1}$ . Given this, denoting  $\Phi^{(L:L+1)}$  as the function of last hidden layer and the last layer, there exists a number of nodes, weights, and biases in the last hidden layer and the last layer, such that
$$
\left\| g - \Phi \right\| _ {\infty} = \left\| \tilde {g} - \Phi^ {(L: L + 1)} \right\| _ {\infty} <   \epsilon ,
$$
due to construction above and Theorem 4. As  $\Phi$  is a unit sampled network, it follows by Lemma 4 that  $\bigcup_{N_L = 1}^{\infty}\mathcal{F}_{L,N_L}^S$  is dense in  $C(\mathcal{X},\mathbb{R}^{N_{L + 1}})$ .
We can now prove that sampled networks with  $L$  layers, and different dimensions in all neurons, with arbitrary width in the last hidden layer, are universal approximators, with the obvious caveat that each hidden layer  $l = 1,2,\dots ,L - 1$  needs at least  $D$  neurons, otherwise we will lose some information regardless of how we construct the network.
Theorem 5. Let  $\mathbf{N}_{\mathbf{L}} = [N_{1}, N_{2}, \ldots, N_{L - 1}, N_{L}]$ , where  $\min \{N_{l} \colon l = 1, 2, \ldots, L - 1\} \geq D$ , and  $L \geq 1$ . Then  $\bigcup_{N_{L} = 1}^{\infty} \mathcal{F}_{L, \mathbf{N}_{L}}^{S}$  is dense in  $C(\mathcal{X}, \mathbb{R}^{N_{L + 1}})$ .
Proof. Let  $\epsilon > 0$  and  $g \in C(\mathcal{X}, \mathbb{R}^{N_{L+1}})$ . For  $L = 1$ , Theorem 4 is enough, and we therefore assume  $L > 1$ . We start by constructing a network  $\tilde{\Phi} \in \bigcup_{N_L=1}^{\infty} \mathcal{F}_{L,\tilde{N}_L}^S$ , where  $\tilde{N}_L = [D,D,\ldots,D,N_L]$  according to Lemma 5, such that  $\| \tilde{\Phi} - g \|_{\infty} < \epsilon$ . To construct  $\Phi \in \bigcup_{N_L=1}^{\infty} \mathcal{F}_{L,N_L}^S$ , let  $l = 1$ , and start by constructing weights/biases for the first  $D$  nodes according to  $\tilde{\Phi}$ . For the additional nodes, in the first hidden layer, select an arbitrary direction  $w$ . Let  $X_1 = \{\tilde{x}_{1,i}^{(j)}: i = 1,2,\ldots,D$  and  $j = 1,2\}$  be the set of all points needed to construct the  $D$  neurons in the next layer of  $\tilde{\Phi}$ . Then for each additional node  $i = D + 1,\ldots,N_1$ , we set
$$
x _ {0, i} ^ {(1)} = \arg \max  \left\{\langle w, x \rangle : x \in \mathcal {X} \text {a n d} \tilde {\Phi} ^ {(1)} (x) \in X _ {1} \right\}.
$$
and choose  $x_{0,i}^{(2)} \in \mathcal{X}$  such that  $x_{0,i}^{(2)} - x_{0,i}^{(1)} = aw$ , where  $a \in \mathbb{R}_{>0}$ , similar to what is done in the proof for Lemma 3. Using these points to define the weights and biases of the last  $N_{1} - D$  nodes, the following space  $\mathcal{X}_1$  now contains points  $[\tilde{x}_{1,i}^{(j)}, 0, 0, \ldots, 0]$ , for  $j = 1, 2$  and  $i = 1, 2, \ldots, D$ . For  $1 < l < L$  repeat the process above, setting  $x_{l,i}^{(j)} = [\tilde{x}_{l,i}^{(j)}, 0, 0, \ldots, 0]$  for the first  $D$  nodes, and construct the weights of the additional nodes as described above, but with sampling space being  $\mathcal{X}_{l-1}$ . When  $l = L$ , set number of nodes to the same as in  $\tilde{\Phi}$ , and choose the points to construct the weights and biases as  $x_{L-1,i}^{(j)} = [\tilde{x}_{L-1,i}^{(j)}, 0, 0, \ldots, 0]$ , for  $j = 1, 2$  and  $i = 1, 2, \ldots, N_L$ . The weights and biases in the last layer are the same as in  $\tilde{\Phi}$ . This implies,
$$
\| \Phi - g \| _ {\infty} = \| \tilde {\Phi} - g \| _ {\infty} <   \epsilon ,
$$
and thus  $\bigcup_{N_L = 1}^{\infty}\mathcal{F}_{L,N_L}^S$  is dense in  $C(\mathcal{X},\mathbb{R}^{N_{L + 1}})$
Remark 5. Important to note that the proof is only showing existence, and that we expect networks to have a more interesting representation after the first  $L - 1$  layers. With this theorem, we can conclude that stacking layers is not necessarily detrimental for the expressiveness of the networks, even though it may alter the sampling space in non-trivial ways. Empirically, we also confirm this, with several cases performing better under deep networks — very similar to iteratively trained neural networks.
Corollary 1.  $\mathcal{F}_{\infty ,\infty}^{S}$  is dense in  $C(\mathcal{X},\mathbb{R}^{N_{L + 1}})$
A.1.4 Networks with a single hidden layer, tanh activation
We now turn to using tanh as activation function, which we find useful for both prediction tasks, and if we need the activation function to be smooth. We will use the results for sampled networks with ReLU as activation function, and show we can arbitrarily well approximate these. The reason for this, instead of using arbitrary network with ReLU as activation function, is that we are using weights and biases of the correct form in the former, such that the tanh networks we construct will more easily have the correct form. We set  $s_2 = \frac{\ln(3)}{2}$ ,  $s_1 = 2 \cdot s_2$  — as already discussed — and let  $\psi$  be the tanh function, with  $\Psi$  being neural networks with  $\psi$  as activation function — simply to separate from the ReLU  $\phi$ , as we are using both in this section. Note that  $\Phi$  is still network with ReLU as activation function and  $s_1 = 1$ ,  $s_2 = 0$ .
We start by showing how a sum of tanh functions can approximate a set of particular functions.
Lemma 6. Let  $f\colon [c_0,c_{M + 1}]\to \mathbb{R}^+$ , defined as  $f(x) = \sum_{i = 1}^{M}a_{i}\mathbf{1}_{[c_{i},c_{M + 1}]}(x)$ , with  $\mathbf{1}$  being the indicator function,  $c_{0} < c_{1} < \dots < c_{M} < c_{M + 1}$ , and for all  $i = 0,1,\ldots ,M + 1$ ,  $c_{i}\in \mathbb{R}$  and  $a_i\in \mathbb{R}_{>0}$ . Then there exists strictly positive scalars  $\omega = \{\omega_{i}\}_{i = 1}^{M}$  such that
$$
g (x) = \sum_ {i = 1} ^ {M} g _ {i} (x) = \sum_ {i = 1} ^ {M} \frac {a _ {i}}{2} [ \psi (\omega_ {i} (x - c _ {i}) - s _ {2}) + 1 ],
$$
fulfills  $f(c_{i-1}) < g(x) < f(c_{i+1})$  whenever  $x \in [c_{i-1}, c_{i+1}]$ , for all  $i = 1, 2, \ldots, M$ .
Proof. We start by observing that both functions,  $f$  and  $g$ , are increasing, with the latter strictly increasing. We also have that  $f(c_0) = 0 < g(x) < \sum_{i=1}^{M} a_i = f(c_M)$ , for all  $x \in [c_0, c_{M+1}]$ , regardless of  $\omega$ . We then fix constants
$$
0 <   \delta_ {i} <   \frac {\frac {3}{4} a _ {i}}{M - i} \quad 0 <   \epsilon_ {i + 1} <   \frac {\frac {a _ {i}}{4}}{i},
$$
for  $i = 1,2,\ldots ,M - 1$ . We have, due to  $s_2$ , that  $g_{i}(c_{i}) = \frac{a_{i}}{4}$ , for all  $i = 1,2,\dots ,M$ . In addition, we can always increase  $\omega_{i}$  to make sure  $g_{i}(c_{i + 1})$  is large enough, and  $g_{i}(c_{i - 1})$  small enough for our purposes, as  $\psi$  is bijective and strictly increasing. We set  $\omega_{1}$  large enough such that  $g_{1}(c_{j}) > a_{1} - \epsilon_{j}$  for all  $j = 2,3,\ldots ,M - 1$ . For  $i = 2,3,\ldots ,M - 1$ , we set  $\omega_{i}$  large enough such that  $g_{i}(c_{j}) < \delta_{j}$  where  $j = 1,2,\ldots ,i - 1$ , and  $g_{i}(c_{j}) > a_{i} - \epsilon_{j}$ , where  $j = i + 1,i + 2,\ldots ,M - 1$ . Finally, let  $\omega_{M}$  be large enough such that  $g_{M}(c_{j}) < \delta_{j}$ , for  $j = 1,2,\ldots ,M - 1$ . With the strictly increasing property of every  $g_{i}$ , we see that
$$
\begin{array}{l} g \left(c _ {i}\right) = \sum_ {j = 1} ^ {i - 1} g _ {j} \left(c _ {i}\right) + \frac {a _ {i}}{4} + \sum_ {j = i + 1} ^ {M} g _ {j} \left(c _ {i}\right) \\ <   \sum_ {j = 1} ^ {i - 1} a _ {j} + \frac {a _ {i}}{4} + \sum_ {j = i + 1} ^ {M} \delta_ {j} \\ = \sum_ {j = 1} ^ {i - 1} a _ {j} + \frac {a _ {i}}{4} + \frac {3 a _ {i}}{4} = f (c _ {i}), \\ \end{array}
$$
$$
\begin{array}{l} g \left(c _ {i}\right) = \sum_ {j = 1} ^ {i - 1} g _ {j} \left(c _ {i}\right) + \frac {a _ {i}}{4} + \sum_ {j = i + 1} ^ {M} g _ {j} \left(c _ {i}\right) \\ > \sum_ {j = 1} ^ {i - 1} \left(a _ {j} - \epsilon_ {i}\right) + \frac {a _ {i}}{4} \\ = \sum_ {j = 1} ^ {i - 1} a _ {j} - \frac {a _ {i}}{4} + \frac {a _ {i}}{4} = f (c _ {i - 1}). \\ \end{array}
$$
Combining the observations at the start with  $f(c_{i-1}) < g(c_i) < f(c_i)$ , for  $i = 1, 2, \ldots, M$ , and the property sought after follows quickly.
We can now show that we can approximate a neuron with ReLU  $\phi$  activation function and with unit sampled weights arbitrarily well.
Lemma 7. Let  $\hat{x}^{(1)},\hat{x}^{(2)}\in \mathcal{X}$  and  $\hat{w}_2\in \mathbb{R}$ . For any  $\epsilon >0$ , there exist a  $M\in \mathbb{N}_{>0}$ , and  $M$  pairs of distinct points  $\{(x_i^{(2)},x_i^{(1)})\in \mathcal{X}\times \mathcal{X}\}_{i = 1}^M$ , such that
$$
\left| \hat {w} _ {2} \phi (\langle \hat {w} _ {1}, x - \hat {x} ^ {(1)} \rangle) - \sum_ {i = 1} ^ {M} \tilde {w} _ {i} \left[ \psi \left(\left\langle w _ {i}, x - x _ {i} ^ {(1)} \right\rangle - s _ {2}\right) + 1 \right] \right| <   \epsilon ,
$$
where  $\hat{w}_1 = \frac{\hat{x}^{(2)} - \hat{x}^{(1)}}{\|\hat{x}^{(2)} - \hat{x}^{(1)}\|}$ ,  $\tilde{w}_i \in \mathbb{R}$ , and  $w_i = s_1 \frac{x_i^{(2)} - x_i^{(1)}}{\|x_i^{(2)} - x_i^{(1)}\|^2}$ .
Proof. Let  $\epsilon > 0$  and, w.l.o.g.,  $\hat{w}_2 > 0$ . Let  $B = \{\langle \hat{w}_1, x \rangle : x \in \mathcal{X}\}$ , as well as  $\hat{f}(x) = \hat{w}_2 \phi(\langle \hat{w}_1, x - \hat{x}^{(1)} \rangle)$ . We start by partitioning  $\hat{f}$  into  $\epsilon/4$ -chunks. More specifically, let  $c = \max B$ , and  $M' = \left\lceil \frac{4\hat{w}_2|c|}{\epsilon} \right\rceil$ . Set  $d_k = \frac{(k-1)c}{M'}$ , for  $k = 1, 2, \ldots, M', M' + 1$ . We will now define points  $c_j$  with the goal of constructing a function  $f$  as in Lemma 6. Still, because we require  $c_j \in B$  to define biases in our tanh functions later, we must define the  $c_j$ s iteratively, by setting  $c_1 = 0$ ,  $k = j = 2$ , and define every other  $c_j$  as follows:
1. If  $d_{k} = c$ , we are done, otherwise proceed to 2.
2. Set
$$
d _ {k} ^ {\prime} = \sup  B \cap \left[ c _ {j - 1}, d _ {k} \right] \quad d _ {k} ^ {\prime \prime} = \inf  B \cap \left[ d _ {k}, c \right].
$$
3. If  $d_k' = d_k''$ , set  $c_j = d_k$ , and increase  $j$  and  $k$ , and go to 1.
4. If  $d_k' > c_{j-1}$ , set  $c_j = d_k'$ , and increase  $j$ , otherwise discard the point  $d_k'$ .
5. Set  $c_{j} = d_{k}^{\prime \prime}$ , and increase  $j$ . Set  $k = \arg \min \{d_{k} - c_{j} \colon d_{k} - c_{j} > 0\}$  and go to 1.
We have  $M < 2 \cdot M'$  points, and can now construct the  $a_j$ s of  $f$ . For  $j = 1, 2, \ldots, M$ , with  $c_{M+1} = c$ , let
$$
a _ {j} = \left\{ \begin{array}{l l} \hat {f} (c _ {j + 1}) - \hat {f} (c _ {j}), & c _ {j + 1} - c _ {j} \leq \frac {c}{M ^ {\prime}} \\ \hat {f} (\rho (c _ {j})) - \hat {f} (c _ {j}), & \text {o t h e r w i s e}, \end{array} \right.
$$
where  $\rho(c_{j}) = \arg \min \{d_{k} - c_{j} : d_{k} - c_{j} \geq 0 \text{ and } k = 1,2,\ldots,M' + 1\}$ . Note that  $0 < c - c_{M} \leq \frac{c}{M'}$ , by Definition 5 and continuity of the inner product  $\langle \hat{w}_{1},\cdot -x^{(1)}\rangle$ . We then construct  $f$  as in Lemma 6, namely,
$$
f (x) = \sum_ {i = 1} ^ {M} a _ {i} \mathbf {1} _ {[ c _ {i}, c _ {M + 1} ]} (x).
$$
Letting  $C = \{[c_j, c_{j+1}]: j = 0, 1, \ldots, M$  and  $c_{j+1} - c_j \leq \frac{c}{M'}\}$ , it is easy to see
$$
| f (x) - \hat {f} (x) | <   \frac {\epsilon}{4},
$$
for all  $x \in \bigcup_{c' \in C} c'$ . For any  $x$  outside said set, we are not concerned with, as it is not part of  $B$ , and hence nothing from  $\mathcal{X}$  is mapped to said points.
Construct  $\omega = \{\omega_i\}_{i=1}^M$  according to Lemma 6. We will now construct a sum of tanh functions, using only weights/biases allowed in sampled networks. For all  $i = 1,2,\ldots,M$ , define  $\tilde{w}_i = \frac{a_i}{2}$  and set  $x_i^{(1)} = \eta(x,\delta_i)$ , with  $\delta_i \geq 0$  and  $x \in \mathcal{X}$ , such that  $\langle \hat{w}_1,x\rangle = c_i$  — where  $\delta_i = 0$  iff  $x \notin \partial \mathcal{X}$ . We specify  $\delta_i$  and  $\epsilon_i' > 0$  such that  $\frac{s_1}{\epsilon_i'} \geq \omega_i$ ,  $\mathbb{B}_{2\epsilon_i'}(x_i^{(1)}) \subseteq \mathcal{X}$ , and
$$
| \psi (\langle w _ {i}, x - x _ {i} ^ {(1)} \rangle - s _ {2}) - \psi (\langle w _ {i}, x \rangle - c _ {i} - s _ {2}) | <   \frac {\epsilon}{4 | \tilde {w} _ {i} | M},
$$