SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We start by defining neural networks and sampled neural networks, and show more in depth the choices of the scalars in sampled networks for ReLU and tanh. Let the input space  $\mathcal{X}$  be a subset of  $\mathbb{R}^D$ . We now define a fully connected, feed forward neural network; mostly to introduce some notation.
Definition 3. Let  $\phi \colon \mathbb{R} \to \mathbb{R}$  be non-polynomial,  $L \geq 1$ ,  $N_0 = D$ , and  $N_1, \ldots, N_{L+1} \in \mathbb{N}_{>0}$ . A (fully connected) neural network with  $L$  hidden layers, activation function  $\phi$ , and weight space  $\Theta = \{W_l \in \mathbb{R}^{N_l, N_{l-1}}, b_l \in \mathbb{R}^{N_l}\}_{l=1}^{L+1}$ , is a function  $\Phi \colon \mathbb{R}^D \to \mathbb{R}^{N_{L+1}}$ , of the form
$$
\Phi (x) = W _ {L + 1} x _ {L} - b _ {L + 1}, \tag {3}
$$
where  $x_{l}$  is defined recursively as  $x_0 = x\in \mathcal{X}$  and
$$
x _ {l} = \phi \left(W _ {l} x _ {l - 1} - b _ {l}\right), \quad l = 1, 2, \dots , L. \tag {4}
$$
The image after  $l$  layers is denoted as  $\mathcal{X}_l = \Phi^{(l)}(\mathcal{X})$ .
As we study the space of these networks, we find it useful to introduce the following notation. The space of all neural networks with  $L$  hidden layers, with  $N = [N_{1},\dots,N_{L}]$  neurons in the separate layers, is denoted as  $\mathcal{F}_{L,N}$ , assuming the input dimension and the output dimension are implicitly given. If each hidden layer has  $N$  neurons, and we let the input dimension and the output dimension be fixed, we write the space of all such neural networks as  $\mathcal{F}_{L,N}$ . We then let
$$
\mathcal {F} _ {\infty , N} = \bigcup_ {L = 1} ^ {\infty} \mathcal {F} _ {L, N},
$$
meaning the set of neural networks with  $N$  width, and arbitrary depth. Similarly,
$$
\mathcal {F} _ {L, \infty} = \bigcup_ {N = 1} ^ {\infty} \mathcal {F} _ {L, N},
$$
for arbitrary width at fixed depth. Finally,
$$
\mathcal {F} _ {\infty , \infty} = \bigcup_ {L = 1} ^ {\infty} \bigcup_ {N = 1} ^ {\infty} \mathcal {F} _ {L, N}.
$$
We will now introduce sampled networks again, and go into depth the choice of constants. The input space  $\mathcal{X}$  is a subset of Euclidean space, and we will work with canonical inner products  $\langle \cdot ,\cdot \rangle$  and their induced norms  $\| \cdot \|$  if we do not explicitly specify the norm.
Definition 4. Let  $\Phi$  be an neural network with  $L$  hidden layers. For  $l = 1, \ldots, L$ , let  $(x_{0,i}^{(1)}, x_{0,i}^{(2)})_{i=1}^{N_l}$  be pairs of points sampled over  $\mathcal{X} \times \mathcal{X}$ . We say  $\Phi$  is a sampled network if the weights and biases of every layer  $l = 1, 2, \ldots, L$  and neurons  $i = 1, 2, \ldots, N_l$ , are of the form
$$
w _ {l, i} = s _ {1} \frac {x _ {l - 1 , i} ^ {(2)} - x _ {l - 1 , i} ^ {(1)}}{\left\| x _ {l - 1 , i} ^ {(2)} - x _ {l - 1 , i} ^ {(1)} \right\| ^ {2}}, \quad b _ {l, i} = \langle w _ {l, i}, x _ {l - 1, i} ^ {(1)} \rangle + s _ {2} \tag {5}
$$
where  $s_1, s_2 \in \mathbb{R}$  are constants related to the activation function, and  $x_{l-1,i}^{(j)} = \Phi^{(l-1)}(x_{0,i}^{(j)})$  for  $j = 1,2$ , assuming  $x_{l-1,i}^{(1)} \neq x_{l-1,i}^{(2)}$ . The last set of weights and biases are  $W_{L+1}, b_{L+1} = \arg \min \mathcal{L}(W_{L+1}\Phi^{(L)}(\cdot) - b_{L+1})$ , where  $\mathcal{L}$  is a suitable loss.
Remark 1. The constants  $s_1, s_2$  can be fixed such that for a given activation function, we can specify values at specific points, e.g., we can specify what value to map the two points  $x^{(1)}$  and  $x^{(2)}$  to; cf. Figure 7. Also note that the points we sampled from  $\mathcal{X} \times \mathcal{X}$  are sampled such that we have unique points after mapping the two sampled points through the first  $l - 1$  layers. This is enforced by constructing the density of the distribution we use to sample the points so that zero density is assigned to points that map to the same output of  $\Phi^{(l-1)}$  (see Definition 2).
Figure 7: Illustration of the placement of point pairs  $x^{(1)}, x^{(2)}$  for activation functions ReLU (left) and tanh (right).
Example A.1. We consider the construction of the constants  $s_1, s_2$  for ReLU and tanh, as they are the two activation functions used for both the proofs and the experiments. We start by considering the activation function tanh, i.e.,
$$
\phi (x) = \frac {\exp^ {2 x} - 1}{\exp^ {2 x} + 1}.
$$
Consider an arbitrary weight  $w$ , where we drop the subscripts for simplicity. Setting  $s_1 = 2 \cdot s_2$ , the input of the corresponding activation function, at the mid-point  $x = x^{(1)} + \frac{x^{(2)} - x^{(1)}}{2} = \frac{x^{(2)} + x^{(1)}}{2}$ , is
$$
\begin{array}{l} \langle w, x \rangle - b = \left\langle w, \frac {x ^ {(2)} + x ^ {(1)}}{2} \right\rangle - \langle w, x ^ {(1)} \rangle - s _ {2} \\ = \left\langle w, \frac {x ^ {(2)} - x ^ {(1)}}{2} \right\rangle - s _ {2} \\ = \frac {2 s _ {2}}{\left\| x ^ {(2)} - x ^ {(1)} \right\| ^ {2}} \cdot \frac {1}{2} \left\langle x ^ {(2)} - x ^ {(1)}, x ^ {(2)} - x ^ {(1)} \right\rangle - s _ {2} \\ = s _ {2} - s _ {2} = 0. \\ \end{array}
$$
The output of the neuron corresponding to the input above is then zero, regardless of what the constant  $s_2$  is. Another aspect of the constants are that we can decide activation values for the two sampled points. This can be seen with a similar calculation as above,
$$
\langle w, x ^ {(1)} \rangle - b = - s _ {2}.
$$
Letting  $s_2 = \frac{\ln 3}{2}$ , we see that the output of the corresponding neuron at point  $x^{(1)}$  is
$$
\phi (- s _ {2}) = \frac {\exp^ {\ln (3)} - 1}{\exp^ {\ln (3)} + 1} = - \frac {1}{2},
$$
and for  $x^{(2)}$  we get  $\frac{1}{2}$ . We can also consider the ReLU activation function where we choose to set  $s_1 = 1$  and  $s_2 = 0$ . The center of the real line is then placed at  $x^{(1)}$ , and  $x^{(2)}$  is mapped to one.
A.1 Universality of sampled networks
In this section we show that universal approximation results also hold for our sampled networks. We start by considering the ReLU function. To separate it from tanh, which is the second activation function we consider, we denote  $\phi$  to be ReLU and  $\psi$  to be tanh. Networks with ReLU as activation function are then denoted by  $\Phi$ , and  $\Psi$  are networks with tanh activation function. When we use ReLU, we set  $s_1 = 1$  and  $s_2 = 0$ , as already discussed. We provide the notation that is used throughout Appendix A in Table 2.
The rest of this section is structured as follows:
1. We introduce the type of input spaces we are working with.
2. We then aim to rewrite an arbitrary fully connected neural network with one hidden layer into a sampled network.
3. This is done by first constructing neurons that add a constant to parts of the input space while leaving the rest untouched.
4. We then show that we can translate between an arbitrary neural network and a sampled network, if the weights of the former are given. This gives us the first universal approximator result, by combining this step with the former.
5. We go on to construct deep sampled networks with arbitrary width, showing that we can contain the information needed through the first  $L - 1$  layers, and then apply the result for sampled networks with one hidden layer, to the last hidden layer.
6. We conclude the section by showing that sampled networks with tanh activation functions are also universal approximators. Concretely, we show that it is dense in the space of sampled networks with ReLU activation function. The reason we need a different proof for the tanh case is that it is not positive homogeneous, a property of ReLU that we heavily depend upon for the proof of sampled networks with ReLU as activation function.
Before we can start with the proof that  $\mathcal{F}_{1,\infty}$  is dense in the space of continuous functions, we start by specifying the domain/input space for the functions.
Table 2: Notation used through the theory section of this appendix.
X'	Pre-noise input space. Subset of R^D and compact for Appendix A.1 and Appendix A.2.
X	Input space. Subset of R^D, X' ⊂ X, and compact for Appendix A.1 and Appendix A.2.
εI	Noise level for X'. Definition 5.
φ	Activation function ReLU, φ(x) = max{x,0}.
Φ	Neural network with activation function ReLU.
ψ	Activation function tanh.
Ψ	Neural network with activation function tanh.
Φc	Constant block. 5 neurons that combined add a constant value c to parts of the input space X, while leaving the rest untouched. Definition 6.
Nl	Number of neurons in layer l of a neural network. N0 = D and NL+1 is the output dimension of network.
wl,i	The weight at the ith neuron, of the lth layer.
bl,i	The bias at the ith neuron, of the lth layer.
Φ(l)	Function that maps x ∈ X to the output of the lth hidden layer.
Φ(l,i)	The ith neuron for the lth layer.
Xl	The image of X after l layers, i.e., Xl = Φ(X). We set X0 = X.
xl	x_l = Φ(l)(x).
F_L,[N_1,N_2,...,N_L]	Space of neural networks with L hidden layers and N_l neurons in layer l.
F_{1,\infty}	Space of all neural networks with one hidden layer, i.e., networks with arbitrary width.
F_{1,\infty}^S	Space of all sampled networks with one hidden layer and arbitrary width.
ζ	Function that maps points from X to X', mapping to the closest point in X', which is unique.
η	Maps x ∈ X × [0,1] to X, by η(x,δ) = x + δ(ζ(x) - x).
||·||	Canonical norm of R^D, with inner product ⟨·⟩.
B_ε(x)	The ball B_ε(x) = {x' ∈ R^D: ||x-x'|| < ε}.
B_ε(x)	The closed ball B_ε(x) = {x' ∈ R^D: ||x-x'|| ≤ ε}.
τ	Reach of the set X'.
λ	Lebesgue measure.
A.1.1 Input space
We again assume that the input space  $\mathcal{X}$  is a subset of  $\mathbb{R}^D$ . We also assume that  $\mathcal{X}$  contains some additional additive noise. Before we can define this noisy input space, we recall two concepts.
Let  $(\mathcal{Z},d)$  be a metric space, and the distance between a subset  $A\subseteq \mathcal{Z}$  and a point  $z\in \mathcal{Z}$  be defined as
$$
d _ {\mathcal {Z}} (z, A) = \inf  \{d (z, a): a \in A \}.
$$
If  $\mathcal{Z}$  is also a normed space, the medial axis is then defined as
$$
\operatorname {M e d} (A) = \{z \in \mathcal {Z}: \exists p \neq q \in A, \| p - z \| = \| q - z \| = d _ {\mathcal {Z}} (z, A) \}
$$
The reach of  $A$  is defined as
$$
\tau_ {A} = \inf  _ {a \in A} d _ {\mathcal {Z}} (a, \operatorname {M e d} (A)).
$$
Informally, the reach is the smallest distance from a point in the subset  $A$  to a non-unique projection of it in the complement  $A^c$ . This means that the reach of convex subsets is infinite (all projections are unique), while other sets can have zero reach, which means  $0 \leq \tau_A \leq \infty$ .
Let  $\mathcal{Z} = \mathbb{R}^D$ , and  $d$  be the canonical Euclidean distance.
Definition 5. Let  $\mathcal{X}'$  be a nonempty compact subset of  $\mathbb{R}^D$  with reach  $\tau_{\mathcal{X}'} > 0$ . The input space  $\mathcal{X}$  is defined as
$$
\mathcal {X} = \{x \in \mathbb {R} ^ {D}: d _ {\mathcal {Z}} (x, \mathcal {X} ^ {\prime}) \leq \epsilon_ {I} \},
$$
where  $0 < \epsilon_I < \min\{\tau_{\mathcal{X}'}', 1\}$ . We refer to  $\mathcal{X}'$  as the pre-noise input space.
Remark 2. As  $d_{\mathcal{Z}}(x, \mathcal{X}') = 0$  for all  $x \in \mathcal{X}'$ , we have that  $\mathcal{X}' \subset \mathcal{X}$ . Due to  $d_{\mathcal{Z}}(x, X') \leq \epsilon_I$  we preserve that  $\mathcal{X}$  is closed, and as  $\epsilon_I < \infty$  means it is bounded, and hence compact. Informally, we have enlarged the input space with new points at most  $\epsilon_I$  distance away from  $\mathcal{X}'$ . This preserves many properties of the pre-noise input space  $\mathcal{X}'$ , while also being helpful in the proofs to come. We also argue that in practice, we are often faced with "noisy" datasets  $\mathcal{X}$  anyway, rather than non-perturbed data  $\mathcal{X}'$ .
We also define a function that maps elements in  $\mathcal{X} \setminus \mathcal{X}'$  down to  $\mathcal{X}'$ , using the uniqueness property given by the medial axis. That is,  $\zeta \colon \mathcal{X} \to \mathcal{X}'$  is the mapping defined as  $\zeta(x) = x'$ , for  $x \in \mathcal{X}$  and  $x' \in \mathcal{X}'$ , such that  $d(x, x') = d_{\mathcal{Z}}(x, \mathcal{X}')$ . As we also want to work along the line between these two points, we set  $\eta \colon \mathcal{X} \times [0,1] \to \mathcal{X}$ , where  $\eta(x, \delta) = x + \delta(\zeta(x) - x)$ . We conclude this part with an elementary result.
Lemma 1. Let  $x \in \mathcal{X}$ ,  $\delta \in [0,1]$ , and  $x' = \eta(x, \delta) \in \mathcal{X}$ , then  $\|x' - x\| \leq \delta$ , with strict inequality when  $\delta > 0$ . Furthermore, when  $\delta > 0$ , we have  $x' \notin \partial \mathcal{X}$ .
Proof. We start by noticing
$$
d (\zeta (x), x ^ {\prime}) = \| \zeta (x) - x ^ {\prime} \| = (1 - \delta) \| \zeta (x) - x \| \leq (1 - \delta) \epsilon_ {I} \leq \epsilon_ {I},
$$
which means  $x' \in \mathcal{X}$ . When  $\delta > 0$ , we have strict inequality, which implies  $x' \in \operatorname{int} \mathcal{X} = \mathcal{X} \setminus \partial \mathcal{X}$ . Furthermore,
$$
d (x, x ^ {\prime}) = \| x - x ^ {\prime} \| = \delta \| \zeta (x) - x \| \leq \delta \cdot \epsilon_ {I} \leq \delta ,
$$
where the last inequality holds due to  $\epsilon_I < 1$ , and is equal only when  $\delta = 0$ .
A.1.2 Networks with a single hidden layer
We can now proceed to the proof that sampled networks with one hidden layer are indeed universal approximators. The main idea is to start off with an arbitrary network with a single hidden layer, and show that we can approximate this network arbitrarily well. Then we can rely on previous universal approximation theorems [13, 33, 52] to finalize the proof. We start by showing some results for a different type of neural networks, but very similar in form. We consider networks where the bias is of the same form as a sampled network. However, the weight is normalized to be a unit weight, that is, divided by the norm, and not the square of the norm. We show in Lemma 4 that the results also hold for any positive scalar multiple of the unit weight, and therefore our sampled network, where we divide by the norm of the weight squared. To be more precise, the weights are of the form
$$
w _ {l, i} = \frac {x _ {l - 1 , i} ^ {(2)} - x _ {l - 1 , i} ^ {(1)}}{\| x _ {l - 1 , i} ^ {(2)} - x _ {l - 1 , i} ^ {(1)} \|},
$$
and biases  $b_{l,i} = \langle w_{l,i},x_{l - 1,i}^{(1)}\rangle$ . Networks with weights/biases in the hidden layers of this form is referred to as unit sampled network. In addition, when the output dimension is  $N_{L + 1} = 1$ , we split the bias of the last layer,  $b_{L + 1}$  into  $N_{L}$  parts, to make the proof easier to follow. This is of no consequence for the final result, as we can always sum up the parts to form the original bias. We write the different parts of the split as  $b_{L + 1,i}$  for  $i = 1,2,\ldots ,N_L$ .
We start by defining a constant block. This is crucial for handling the bias, as we can add constants to the output of certain parts of the input space, while leaving the rest untouched. This is important when proving Lemma 2.
Definition 6. Let  $c_{1} < c_{2} < c_{3}$ , and  $c \in \mathbb{R}^{+}$ . A constant block  $\Phi_c$  is defined as five neurons summed together as follows. For  $x \in \mathbb{R}$
$$
\Phi_ {c} (x) = \sum_ {i = 1} ^ {5} f _ {i} (x),
$$
$$
f _ {1} (x) = a _ {1} \phi (x - c _ {2}), \quad f _ {2} (x) = a _ {1} \phi (- (x - c _ {3})),
$$
$$
f _ {3} (x) = - a _ {1} \phi (x - c _ {3}), \quad f _ {4} (x) = - a _ {2} \phi (- (x - c _ {1}))
$$
$$
f _ {5} (x) = a _ {3} \phi \left(x - c _ {1}\right),
$$
and  $a_1 = \frac{c}{c_3 - c_2}, a_2 = a_1 \frac{c_1 - c_3}{c_1 - c_2}$ , and  $a_3 = a_2 - a_1$ .
Remark 3. The function  $\Phi_c$  are constructed using neurons, but can also be written as the continuous function,
$$
\Phi_ {c} (x) = \left\{ \begin{array}{l l} 0, & x \leq c _ {1} \\ a _ {3} \cdot x + d, & c _ {1} <   x \leq c _ {2} \\ c, & x > c _ {2}, \end{array} \right.
$$
where  $d = a_{1}c_{3} + a_{2}c_{2}$ . Obviously, if  $c$  needs to be negative, one can simply swap the sign on each of the three parameters,  $a_{i}$ .
We can see by the construction of  $\Phi_c$  that we might need the negative of some original weight. That is, the input to  $\Phi_c$  is of the form  $\langle w_{1,i},x\rangle$ , and for  $f_{2}$  and  $f_{4}$ , we require  $\langle -w_{1,i},x\rangle$ . In Lemma 3, we shall see that this is not an issue and that we can construct neurons such that they approximate constant blocks arbitrarily well, as long as  $c_{1},c_{2},c_{3}$  can be produced by the inner product between a weight and points in  $\mathcal{X}$ , that is, if we can produce the biases equal to the constants  $c_{1},c_{2},c_{3}$ .
Let  $\hat{\Phi} \in \mathcal{F}_{1,K}$ , with parameters  $\{\hat{w}_{l,i},\hat{b}_{l,i}\}_{l = 1,i = 1}^{2,K}$ , be an arbitrary neural network. Unless otherwise stated, the weights in this arbitrary network  $\hat{\Phi}$  are always nonzero. Sampled networks cannot have zero weights, as the point pairs used to construct weights, both for unit and regular sampled networks, are distinct. However, one can always construct the same output as neurons with zero weights by setting certain weights in  $W_{L + 1}$  to zero.
We start by showing that, given all weights of a network, we can construct all biases in a unit sampled network so that the function values agree. More precisely, we want to construct a network with weights  $\hat{w}_{l,i}$ , and show that we can find points in  $\mathcal{X}$  to construct the biases of the form in unit sampled networks, such that the resulting neural network output on  $\mathcal{X}$  equals exactly the values of the arbitrary network  $\hat{\Phi}$ .
Lemma 2. Let  $\hat{\Phi} \in \mathcal{F}_{1,K} \colon \mathcal{X} \to \mathbb{R}^{N_2}$ . There exists a set of at most  $6 \cdot N_1$  points  $x_i \in \mathcal{X}$  and biases  $b_{2,i} \in \mathbb{R}$ , such that a network  $\Phi$  with weights  $w_{1,i} = \hat{w}_{1,i}$ ,  $w_{2,i} = \hat{w}_{2,i}$ , and biases  $b_{2,i} \in \mathbb{R}$ ,
$$
b _ {1, i} = \langle w _ {1, i}, x _ {i} \rangle ,
$$
for  $i = 1,2,\ldots ,N_{1}$  satisfies  $\Phi (x) - \hat{\Phi} (x) = 0$  for all  $x\in \mathcal{X}$
Proof. W.l.o.g., we assume  $N_{2} = 1$ . For any weight/bias pair  $\hat{w}_{1,i},\hat{b}_{1,i}$ , we let  $w_{1,i} = \hat{w}_{1,i}$ ,  $w_{2,i} = \hat{w}_{2,i}$ ,  $B_{i} = \{\langle w_{1,i},x\rangle : x\in \mathcal{X}\}$ ,  $b_{\bigwedge}^{(i)} = \inf B_i$ , and  $b_{\bigvee}^{(i)} = \sup B_i$ . As  $\langle w_{1,i},\cdot \rangle$  is continuous, means  $B_{i}$  is compact and  $b_{\bigwedge}^{(i)}, b_{\bigvee}^{(i)}\in B_i$ .
We have four different cases, depending on  $\hat{b}_{1,\iota}$ .
(1) If  $\hat{b}_{1,i} \in B_i$ , then we simply choose a corresponding  $x_i \in \mathcal{X}$  such that  $b_{1,i} = \langle w_{1,i}, x_i \rangle = \hat{b}_{1,i}$ . Letting  $b_{2,i} = \hat{b}_{2,i}$ , we have
$$
w _ {2, i} \phi (\langle w _ {1, i}, x \rangle - b _ {1, i}) - b _ {2, i} = \hat {w} _ {2, i} \phi (\langle \hat {w} _ {1, i}, x \rangle - \hat {b} _ {1, i}) - \hat {b} _ {2, i}.
$$
(2) If  $\hat{b}_{1,i} > b_{\vee}^{(i)}$ , we choose  $x_{i}$  such that  $b_{1,i} = \langle w_{1,i}, x_{i} \rangle = b_{\vee}^{(i)}$  and  $b_{2,i} = \hat{b}_{2,i}$ . As  $\phi(\langle w_{1,i}, x \rangle - b_{1,i}) = \phi(\langle \hat{w}_{1,i}, x \rangle - \hat{b}_{1,i}) = 0$ , for all  $x \in \mathcal{X}$ , we are done.
(3) If  $\hat{b}_{1,i} < b_{\wedge}^{(i)}$ , we choose corresponding  $x_{i}$  such that  $b_{1,i} = \langle w_{1,i}, x_{i} \rangle = b_{\wedge}^{(i)}$ , and set  $b_{2,i} = \hat{b}_{2,i} + w_{2,i} \left(\hat{b}_{1,i} - b_{\wedge}^{(i)}\right)$ . We then have
$$
\begin{array}{l} w _ {2, i} \phi (\langle w _ {1, i}, x \rangle - b _ {1, i}) - b _ {2, i} = w _ {2, i} \langle w _ {1, i}, x \rangle - w _ {2, i} b _ {1, i} - b _ {2, i} \\ = w _ {2, i} \langle w _ {1, i}, x \rangle - \hat {b} _ {2, i} - w _ {2, i} \hat {b} _ {1, i} \pm w _ {2, i} b _ {\wedge} ^ {(i)} \\ = \hat {w} _ {2, i} \langle \hat {w} _ {1, i}, x \rangle - \hat {b} _ {2, i} - \hat {w} _ {2, i} \hat {b} _ {1, i} \\ = \hat {w} _ {2, i} \phi (\langle \hat {w} _ {1, i}, x \rangle - \hat {b} _ {1, i}) - \hat {b} _ {2, i}, \\ \end{array}
$$
where first and last equality holds due to  $\langle w_{1,i},x\rangle >b_{1,i} > \hat{b}_{1,i}$  for all  $x\in \mathcal{X}$ .
(4) If  $b_{\wedge}^{(i)} < \hat{b}_{1,i} < b_{\vee}^{(i)}$ , and  $\hat{b}_{1,i} \notin B_i$ , things are a bit more involved. First notice that  $B_i^{(1)} = B_i \cap [b_{\wedge}^{(i)}, \hat{b}_{1,i}]$  and  $B_i^{(2)} = B_i \cap [\hat{b}_{1,i}, b_{\vee}^{(i)}]$  are both non-empty compact sets. We therefore have that supremum and infimum of both sets are members of their respective set, and thus also part of  $B_i$ . We therefore choose  $x_i$  such that  $b_{1,i} = \langle w_{1,i}, x_i \rangle = \inf B_i^{(2)}$ . To make up for the difference between  $\hat{b}_{1,i} < b_{1,i}$ , we add a constant to all  $x \in \mathcal{X}$  where  $\langle w_{1,i}, x \rangle > b_{1,i}$ . To do this we add some additional neurons, using our constant block  $\Phi_c^{(i)}(\langle w_{1,i}, \cdot \rangle)$ . Letting  $c = w_{2,i}\left(b_{1,i} - \hat{b}_{1,i}\right)$ ,  $c_1 = \sup B_i^{(1)}$ ,  $c_2 = b_{1,i}$ , and  $c_3 = b_{\vee}^{(i)}$ . We have now added five more neurons, and the weights and bias in second layer corresponding to the neurons is set to be  $\pm a$  and 0, respectively, where both  $a$  and the sign of  $a$  depends on Definition 6. In case we require a negative sign in front of  $\langle w_{1,i}, \cdot \rangle$ , we simply set it as we are only concerned with finding biases given weights. We then have that for all  $x \in \mathcal{X}$ ,
$$
\Phi_ {c} ^ {(i)} (\langle w _ {1, i}, x \rangle) = \left\{ \begin{array}{l l} c, & \langle w _ {1, i}, x \rangle > b _ {1, i} \\ 0, & \text {o t h e r w i s e .} \end{array} \right.
$$
Finally, by letting  $b_{2,i} = \hat{b}_{2,i}$ , we have
$$
\begin{array}{l} w _ {2, i} \phi (\left\langle w _ {1, i}, x \right\rangle - b _ {1, i}) - b _ {2, i} + \Phi_ {c} ^ {(i)} (\left\langle w _ {1, i}, x \right\rangle) \\ = w _ {2, i} \langle w _ {1, i}, x \rangle - w _ {2, i} b _ {1, i} - \hat {b} _ {2, i} + w _ {2, i} b _ {1, i} - \hat {w} _ {2, i} \hat {b} _ {1, i} \\ = \left\langle \hat {w} _ {1, i}, x \right\rangle - \hat {w} _ {2, i} \hat {b} _ {1, i} - \hat {b} _ {2, i} \\ = \hat {w} _ {2, i} \phi \left(\left\langle \hat {w} _ {1, i}, x \right\rangle \hat {b} _ {1, i}\right) - \hat {b} _ {2, i}, \\ \end{array}
$$
when  $\langle w_{1,i},x\rangle >b_{1,i}$  , and
$$
\begin{array}{l} w _ {2, i} \phi (\langle w _ {1, i}, x \rangle - b _ {1, i}) - b _ {2, i} + \Phi_ {c} ^ {(i)} (\langle w _ {1, i}, x \rangle) = b _ {2, i} = \hat {b} _ {2, i} \\ = \hat {w} _ {2, i} \phi (\langle w _ {1, i}, x \rangle \hat {b} _ {1, i}) - \hat {b} _ {2, i}, \\ \end{array}
$$
otherwise. And thus,  $\Phi(x) = \hat{\Phi}(x)$  for all  $x \in \mathcal{X}$ . As we add five additional neurons for each constant block, and we may need one for each neuron, means we need to construct our network with at most  $6 \cdot N_1$  neurons.
Now that we know we can construct suitable biases in our unit sampled networks for all types of biases  $\hat{b}_{l,i}$  in the arbitrary network  $\hat{\Phi}$ , we show how to construct the weights.
Lemma 3. Let  $\hat{\Phi} \in \mathcal{F}_{1,K}$ :  $\mathcal{X} \to \mathbb{R}^{N_2}$ , with biases of the form  $\hat{b}_{1,i} = \langle \hat{w}_{1,i}, x_i \rangle$ , where  $x_i \in \mathcal{X}$  for  $i = 1, 2, \ldots, N_1$ . For any  $\epsilon > 0$ , there exist unit sampled network  $\Phi$  such that  $\| \Phi - \hat{\Phi} \|_{\infty} < \epsilon$ .
Proof. W.l.o.g., we assume  $N_{2} = 1$ . For all  $i = 1,2,\ldots ,N_{1}$ , we construct the weights and biases as follows: If  $x_{i}\notin \partial \mathcal{X}$ , then we set  $\epsilon^{\prime} > 0$  such that  $\mathbb{B}_{\epsilon '}(x_i)\subset \mathcal{X}$ . Let  $x_{0,i}^{(1)} = x_{i}$  and  $x_{0,i}^{(2)} = x_{0,i}^{(1)} + \frac{\epsilon'}{2} w_{1,i}\in \mathbb{B}_{\epsilon '}(\boldsymbol {x}_i)\subset \mathcal{X}$ . Setting  $w_{2,i} = \| \hat{w}_{1,i}\| \hat{w}_{2,i}$ ,  $b_{2,i} = \hat{b}_{2,i}$ , and  $w_{1,i} = \frac{x_{0,i}^{(2)} - x_{0,i}^{(1)}}{\|x_{0,i}^{(2)} - x_{0,i}^{(1)}\|} = \frac{\hat{w}_{1,i}}{\|\hat{w}_{1,i}\|}$ , implies