SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(a)  $\mathcal{B}(x', r) \subseteq [-(B - \delta), B - \delta]^d$ .
(b) For each  $p \neq \ell$ , the ball  $\mathcal{B}(x', r)$  is either included in the half-space  $H_p^{(\epsilon, +)} := \{x \in \mathbb{R}^d \mid \alpha_p^\top x + \beta_p > \epsilon\}$  or in the half-space  $H_p^{(\epsilon, -)} := \{x \in \mathbb{R}^d \mid \alpha_p^\top x + \beta_p < -\epsilon\}$ .
(c) The intersection of  $\mathcal{B}(x', r)$  with  $H_{\ell}^{(\epsilon, +)}$  and  $H_{\ell}^{(\epsilon, -)}$  are not empty.
For the remaining of the proof, we will use Lemma C.5, another result taken from [26]. We only state the lemma. Its formal proof can be found in the proof of [26, Theorem 3.8, Steps 4-5].
Lemma C.5 (Affine linear area [26]). Given a sequence  $\{\theta^k\}_{k\in \mathbb{N}}$  satisfying the second condition of Lemma C.2, we have:
(a) For any  $0 < \delta < B$ , there exists a constant  $\kappa_{\delta}$  such that  $\forall i \in \bar{J}$ ,  $h_i^k$  are affine linear on  $[-B - \delta, B - \delta]^{N_0}$  for all  $k \geq \kappa_{\delta}$ .
(b) For any  $\epsilon > 0$ , there exists a constant  $\kappa_{\epsilon}$  such that for each  $1 \leq \ell \leq r$  and each  $i \in J_{\ell}$  the function  $h_i^k$  is affine linear on  $H_{\ell}^{(\epsilon, +)} \cup H_{\ell}^{(\epsilon, -)}$  for all  $k \geq \kappa_{\epsilon}$ .
The lemma implies the existence of  $K = \max (\kappa_{\delta},\kappa_{\epsilon})$  such that for all  $k\geq K$  , we have:
$$
\sum_ {p \neq \ell} g _ {p} ^ {k} (x) = \mathbf {B} ^ {k} x + \nu^ {k}, \quad \forall x \in \mathcal {B} (x ^ {\prime}, r),
$$
for some  $\mathbf{B}^k\in \mathbb{R}^{N_2\times N_0},\nu^k\in \mathbb{R}^{N_2}$ . Therefore, for  $k\geq K$ , we have:
$$
\begin{array}{l} \mathcal {R} _ {\theta^ {k}} (x) = \mathbf {B} ^ {k} x + \nu^ {k} + \sum_ {i \in J _ {\ell} ^ {+}} \mathbf {W} _ {2} ^ {k} [:, i ] (\mathbf {W} _ {1} ^ {k} [ i,: ] x + \mathbf {b} _ {1} ^ {k} [ i]), \forall x \in \mathcal {B} (x ^ {\prime}, r) \cap H _ {\ell} ^ {(\epsilon , +)} \\ \mathcal {R} _ {\theta^ {k}} (x) = \mathbf {B} ^ {k} x + \nu^ {k} + \sum_ {i \in J _ {\ell} ^ {-}} \mathbf {W} _ {2} ^ {k} [:, i ] (\mathbf {W} _ {1} ^ {k} [ i,: ] x + \mathbf {b} _ {1} ^ {k} [ i]), \forall x \in \mathcal {B} (x ^ {\prime}, r) \cap H _ {\ell} ^ {(\epsilon , -)}. \\ \end{array}
$$
Since we proved that  $f$  has the form Equation (22), there exist  $\mathbf{C} \in \mathbb{R}^{N_2 \times N_0}$ ,  $\mu \in \mathbb{R}^{N_2}$  such that
$$
\begin{array}{l} f (x) = \left(\mathbf {C} + \gamma_ {\ell} \alpha_ {\ell}\right) x + (\mu + \gamma_ {\ell} \beta_ {\ell}), \quad \forall x \in \mathcal {B} \left(x ^ {\prime}, r\right) \cap H _ {\ell} ^ {(\epsilon , +)} \\ f (x) = \mathbf {C} x + \mu , \quad \forall x \in \mathcal {B} \left(x ^ {\prime}, r\right) \cap H _ {\ell} ^ {(\epsilon , -)} \\ \end{array}
$$
As both  $\mathcal{B}(x', r) \cap H_{\ell}^{(\epsilon, +)}$  and  $\mathcal{B}(x', r) \cap H_{\ell}^{(\epsilon, -)}$  are open sets, and given our hypothesis of uniform convergence of  $\mathcal{R}_{\theta^k} \to f$ , we obtain,
$$
\begin{array}{l} \lim _ {k \to \infty} \mathbf {B} ^ {k} + \sum_ {i \in J _ {\ell} ^ {+}} \mathbf {W} _ {2} ^ {k} [:, i ] \mathbf {W} _ {1} ^ {k} [ i,: ] = \mathbf {C} + \gamma_ {\ell} \alpha_ {\ell} \\ \lim  _ {k \rightarrow \infty} \mathbf {B} ^ {k} + \sum_ {i \in J _ {\ell} ^ {-}} \mathbf {W} _ {2} ^ {k} [:, i ] \mathbf {W} _ {1} ^ {k} [ i,: ] = \mathbf {C} \tag {27} \\ \lim  _ {k \rightarrow \infty} \nu^ {k} + \sum_ {i \in J _ {\ell} ^ {+}} \mathbf {b} _ {1} ^ {k} [ i ] \mathbf {W} _ {2} ^ {k} [:, i ] = \mu + \gamma_ {\ell} \beta_ {\ell} \\ \lim  _ {k \to \infty} \nu^ {k} + \sum_ {i \in J _ {\ell} ^ {-}} \mathbf {b} _ {1} ^ {k} [ i ] \mathbf {W} _ {2} ^ {k} [:, i ] = \mu . \\ \end{array}
$$
Proof for Equation (27) can be found in Appendix C.4. Equations (23) and (24) follow directly from Equation (27).
3. Proof of Equation (25): Since  $\alpha_{\ell} \neq 0$  (remember that  $\| \alpha_{\ell} \| = 1$ ), this is an immediate consequence of Equation (23) as each vector  $\mathbf{W}_2^k [:, j]$ ,  $j \in J_{\ell}$  is supported in  $I_2 [:, j] \subseteq \cup_{i \in J_{\ell}} I_2 [:, i]$ .
813   
814   
815   
816   
817   
818   
819   
820   
821   
822   
823   
824   
825   
826   
827   
828   
829   
830   
831   
832   
833   
834
We state an immediate corollary of Lemma C.3, which characterizes the limit of the sequence of contributions  $\{g_{\ell}^{k}\}_{k\in \mathbb{N}}$  of the  $\ell$ th equivalence class with  $|J_{\ell}| = 1$ .
Corollary C.1. Consider  $f \in \overline{\mathcal{F}_{\mathbf{I}}([-B,B]^d)}$  that admits the analytical form in Equation (22), a sequence  $\{\theta^k\}_{k \in \mathbb{N}}$  as given by Lemma C.2, and Definition C.1. For all singleton equivalence classes  $J_{\ell} = \{i\}$ ,  $1 \leq \ell \leq r$ , we have  $\lim_{k \to \infty} \mathbf{W}_2^k[:, i] = \gamma_\ell$  and  $\lim_{k \to \infty} \| h_\ell^k - \gamma_\ell \sigma(\alpha_\ell^\top x + \beta_\ell) \|_\infty = 0$ .
Proof. We first prove that  $\mathbf{W}_2^k [(:,i]$  has a finite limit. In fact, applying the second point of Lemma C.3 for  $J_{\ell} = \{i\}$ , we have:
$$
\lim _ {k \rightarrow \infty} \mathbf {W} _ {2} ^ {k} [:, i ] \mathbf {W} _ {1} ^ {k} [ i,: ] = \gamma_ {\ell} \alpha_ {\ell}
$$
where  $\gamma_{\ell},\alpha_{\ell}$  are defined in Lemma C.3. Because  $\lim_{k\to \infty}\mathbf{W}_1^k [i,:] = \alpha_\ell$  and  $\| \alpha_{\ell}\|_{2} = 1$ , it follows that  $\gamma_{\ell} = \lim_{k\to \infty}\mathbf{W}_2^k [:,i]$ . To conclude, since we also have  $\beta_{\ell} = \lim_{k\to \infty}\mathbf{b}_1^k [i]$ , we obtain  $h_\ell^k (\cdot) = \mathbf{W}_2^k [\ell ,:]\sigma (\mathbf{W}_1^k [\ell ,:]\cdot +\mathbf{b}_1^k [\ell])\to \gamma_\ell \sigma (\alpha_\ell x + \beta_\ell)$  as claimed.
The nice thing about Corollary C.1 is that the contribution  $g_{\ell}^{k} = h_{\ell}^{k}$  admits a (uniform) limit if  $J_{\ell} = \{i\}$ . Moreover, this limit is even implementable by using only the  $i$ th neuron because  $\operatorname{supp}(\alpha_{\ell}) \subseteq I_1[i,:]$  and  $\operatorname{supp}(\gamma_{\ell}) \subseteq I_2[:,i]$ .
It would be tempting to believe that, for each  $P \in \{\bar{J}\} \cup \{J_{\ell} \mid \ell = 1, \dots, r\}$  the sequence of functions  $\sum_{i \in P} g_i^k(x)$  must admit a limit (when  $k$  tends to  $\infty$ ) and that this limit is implementable using only neurons in  $P$ . This would obviously imply that  $\mathcal{F}_{\mathbf{I}}(\Omega)$  is closed. This intuition is however wrong. For non-singleton equivalence class (i.e., for cases not covered by Corollary C.1), the limit function does not necessarily exist as we show in the following example.
Example C.1. Consider the case where  $\mathbf{N} = (1,3,1)$  and no support constraint,  $\Omega = [-1,1]$ , take the sequence  $\{\theta^k\}_{k\in \mathbb{N}}$  which satisfies:
$$
\mathbf {W} _ {1} ^ {k} = \left( \begin{array}{c} 1 \\ - 1 \\ 1 \end{array} \right), \mathbf {b} _ {1} ^ {k} = \left( \begin{array}{c} 0 \\ 0 \\ 1 \end{array} \right), \mathbf {W} _ {2} ^ {k} = (k \quad - k \quad - k), \mathbf {b} _ {2} ^ {k} = k
$$
Then for  $x \in \Omega$ , it is easy to verify that  $\mathcal{R}_{\theta^k} = 0$ . Indeed,
$$
\begin{array}{l} \mathcal {R} _ {\theta^ {k}} (x) = \sum_ {i = 1} ^ {3} \mathbf {W} _ {2} ^ {k} [:, i ] \sigma \left(\mathbf {W} _ {1} ^ {k} [ i,: ] + \mathbf {b} _ {1} ^ {k} [ i ]\right) + \mathbf {b} _ {2} ^ {k} \\ = k \sigma (x) - k \sigma (- x) - k \sigma (x + 1) + k \\ = k (\sigma (x) - \sigma (- x)) - k (x + 1) + k \quad (s i n c e x + 1 \geq 0, \forall x \in \Omega) \\ = k x - k (x + 1) + k = 0 \\ \end{array}
$$
Thus, this sequence converges (uniformly) to  $f = 0$ . Moreover, this sequence also satisfies the assumptions of Lemma C.2. Using the classification in Definition C.1, we have one class equivalence  $J_{1} = \{1,2\}$  and  $\bar{J} = \{3\}$ . The function  $g_{1}^{k}(x) = k\sigma(x) - k\sigma(-x) = kx$ , however, does not have any limit.
C.2.2 Actual proof of Theorem 4.2
Therefore, our analysis cannot treat each equivalence class entirely separately. The last result in this section is about a property of the matrix  $\mathbf{A}$  in Equation (22). This is one of our key technical contributions in this work.
Lemma C.6. Consider  $\Omega = [-B, B]^d$ ,  $f \in \overline{\mathcal{F}_{\mathbf{I}}(\Omega)}$  that admits the analytical form in Equation (22), a sequence  $\{\theta^k\}_{k \in \mathbb{N}}$  as given by Lemma C.2, then the matrix  $\mathbf{A} \in \overline{\mathcal{L}_{\mathbf{I}'}}$  where  $\mathbf{I}' = (I_2[:, S], I_1[S,:])$ ,  $S = \bar{J} \cup (\cup_{1 \leq \ell \leq r} J_\ell^-)$ ,  $\bar{J}, J_\ell^\pm$  are defined as in Definition C.1).
Combining Lemma C.6 and the assumptions of Theorem 4.2, we can prove Theorem 4.2 immediately as follow:
Proof of Theorem 4.2. Consider  $f \in \overline{\mathcal{F}_{\mathbf{I}}(\Omega)}$ , we deduce that there exists a sequence of  $\{\theta^k\}_{k \in \mathbb{N}}$  that satisfies the properties of Lemma C.2. This allows us to define  $\bar{J}$  and equivalence classes
835   
836   
837   
838   
839   
840   
841   
842   
843   
844   
845   
846   
847   
848   
849   
850   
851   
852   
853   
854   
855   
856   
857   
858   
859   
860   
861   
862   
863
$J_{\ell}, 1 \leq \ell \leq r$  as well as  $(\alpha_{\ell}, \beta_{\ell})$  as in Definition C.1. Using Lemma C.3, we can also deduce an analytical formula for  $f$  as in Equation (22):
$$
f (x) = \sum_ {i = 1} ^ {r} \gamma_ {\ell} \sigma (\alpha_ {\ell} x + \beta_ {\ell}) + \mathbf {A} x + \mathbf {b}, \quad \forall x \in \Omega .
$$
Finally, Lemma C.6 states that matrix  $\mathbf{A}$  in Equation (22) satisfies:  $\mathbf{A} \in \overline{\mathcal{L}_{\mathbf{I}^{\prime}}}$  with  $\mathbf{I}^{\prime} = (I_{2}[:,S], I_{1}[S,:])$ , where  $S = \bar{J} \cup (\cup_{\ell=1}^{r} J_{\ell}^{-})$ . To prove that  $f \in \mathcal{F}_{\mathbf{I}}$ , we construct the parameters  $\theta = \left\{(\mathbf{W}_i, \mathbf{b}_i)_{i=1}^2\right\}$  of the limit network as follows:
1. For each  $1 \leq \ell \leq r$ , choose one index  $j \in J_{\ell}^{+}$  (which is possible since  $J_{\ell}^{+}$  is non-empty). We set:
$$
\left(\mathbf {W} _ {1} [ i,: ], \mathbf {W} _ {2} [:, i ], \mathbf {b} _ {1} [ i ]\right) = \left\{ \begin{array}{l l} \left(\alpha_ {\ell}, \gamma_ {\ell}, \beta_ {\ell}\right) & \text {i f} i = j \\ \left(\alpha_ {\ell}, \mathbf {0}, \beta_ {\ell}\right) & \text {o t h e r w i s e} \end{array} \right. \tag {28}
$$
This satisfies the support constraint because  $\mathsf{supp}(\alpha_{\ell}) \subseteq I_1[j,:]$  (by (21))  $\alpha_{\ell} = \lim_{k \to \infty} \mathbf{W}_1^k[j,:]$  and  $I_2 = \mathbf{1}_{N_2 \times N_1}$ . This is where we use the first assumption of Theorem 4.2. Without it,  $\mathsf{supp}(\gamma_{\ell})$  might not be a subset of  $I_2[:, j]$ .
2. For  $i \in S$ : Since  $\mathbf{A} \in \overline{\mathcal{L}_{\mathbf{I}^{\prime}}}$  (cf Lemma C.6) and  $\mathcal{L}_{\mathbf{I}^{\prime}}$  is closed (second assumptions of Theorem 4.2), there exist two matrices  $\hat{\mathbf{W}}_1, \hat{\mathbf{W}}_2$  such that:  $\mathrm{supp}(\hat{\mathbf{W}}_1) \subseteq I_1[;,S]$ ,  $\mathrm{supp}(\hat{\mathbf{W}}_2) \subseteq I_2[S,:]$ , and  $\mathbf{A} = \hat{\mathbf{W}}_2\hat{\mathbf{W}}_1$ . We set:
$$
\left(\mathbf {W} _ {1} [ i,: ], \mathbf {W} _ {2} [:,: i ], \mathbf {b} _ {1} [ i ]\right) = \left(\tilde {\mathbf {W}} _ {1} [ i,: ], \tilde {\mathbf {W}} _ {2} [:,: i ], C\right) \tag {29}
$$
where  $C = \sup_{x\in \Omega}\| \hat{\mathbf{W}}_1x\|_\infty$ . This satisfies the support constraints  $\mathbf{I}$  due to our choice of  $\hat{\mathbf{W}}_1, \hat{\mathbf{W}}_2$ . The choice of  $C$  ensures that the function  $h_i(x) \coloneqq \mathbf{W}_2[i,:]\sigma(\mathbf{W}_1[i,:]x + \mathbf{b}_1[i])$  is linear on  $\Omega$ .
3. For  $\mathbf{b}_2$ : Let  $\mathbf{b}_2 = \mathbf{b} - C\left(\sum_{i\in S}\hat{\mathbf{W}}_2[;,i]\right)$  ( $\mathbf{b}$  is the bias in Equation (22)).
Verifying  $\mathcal{R}_{\theta} = f$  on  $\Omega$  is thus trivial since:
$$
\begin{array}{l} \mathcal {R} _ {\theta} (x) = \sum_ {i = 1} ^ {N _ {1}} \mathbf {W} _ {2} [:, i ] \sigma (\mathbf {W} _ {1} [ i,: ] x + \mathbf {b} _ {1} [ i ]) + \mathbf {b} _ {2} \\ = \sum_ {i \notin S} \mathbf {W} _ {2} [:, i ] \sigma \left(\mathbf {W} _ {1} [ i,: ] x + \mathbf {b} _ {1} [ i ]\right) + \sum_ {i \in S} \mathbf {W} _ {2} [:, i ] \sigma \left(\mathbf {W} _ {1} [ i,: ] x + \mathbf {b} _ {1} [ i ]\right) + \mathbf {b} _ {2} \\ = \sum_ {\ell = 1} ^ {r} \gamma_ {\ell} \sigma \left(\alpha_ {\ell} x + \beta_ {\ell}\right) + \sum_ {j \in S} \hat {\mathbf {W}} _ {2} [:, i ] \left(\hat {\mathbf {W}} _ {1} [ i,: ] x + C\right) + \mathbf {b} - C \left(\sum_ {i \in S} \hat {\mathbf {W}} _ {2} [:, i ]\right) \\ = \sum_ {\ell = 1} ^ {r} \gamma_ {\ell} \sigma \left(\alpha_ {\ell} x + \beta_ {\ell}\right) + \hat {\mathbf {W}} _ {2} \hat {\mathbf {W}} _ {1} x + \mathbf {b} = \sum_ {\ell = 1} ^ {r} \gamma_ {\ell} \sigma \left(\alpha_ {\ell} x + \beta_ {\ell}\right) + \mathbf {A} x + \mathbf {b} = f. \\ \end{array}
$$
Proof of Lemma C.6. In this proof, we define  $\Omega_{\delta}^{\circ} = (-B + \delta, B - \delta)^{N_0}$ ,  $0 < \delta < B$ . The choice of  $\delta$  is not important in this proof (any  $0 < \delta < B$  will do).
The proof of this lemma revolves around the following idea: We will construct a sequence of functions  $\{f^k\}_{k\in \mathbb{N}}$  such that, for  $k$  large enough,  $f^{k}$  has the following analytical form:
$$
f ^ {k} (x) = \sum_ {\ell = 1} ^ {r} \gamma_ {\ell} \sigma \left(\alpha_ {\ell} x + \beta_ {\ell}\right) + \mathbf {A} ^ {k} x + \mathbf {b} ^ {k}, \forall x \in \Omega_ {\delta} ^ {\circ} \tag {30}
$$
and  $\lim_{k\to \infty}f^{k}(x) = f(x)\forall x\in \Omega \setminus (\cup_{\ell = 1}^{r}H_{\ell})$  (or equivalently  $f^{k}$  converges pointwise to  $f$  on  $\Omega \setminus (\cup_{\ell = 1}^{r}H_{\ell}))$  and  $\mathbf{A}^k$  admits a factorization into two factors  $\mathbf{A}^k = \mathbf{X}^k\mathbf{Y}^k$  satisfying  $\mathrm{supp}(\mathbf{X}^k)\subseteq I_2[:,S]$ ,  $\mathrm{supp}(\mathbf{Y}^k)\subseteq I_1[S,:]$ , so that  $\mathbf{A}^k\in \mathcal{L}_{\mathbf{I}'}$ . Comparing Equation (22) and Equation (30), we deduce that the sequence of affine functions  $\mathbf{A}^k x + \mathbf{b}^k$  converges pointwise to the affine function  $\mathbf{A}x + \mathbf{b}$  on the open set  $\Omega_{\delta}^{\circ}\setminus (\cup_{\ell = 1}^{r}H_{\ell})$ . Therefore,  $\lim_{k\to \infty}\bar{\mathbf{A}}^k = \bar{\mathbf{A}}$  by Lemma C.7, hence the conclusion.
864   
865   
866   
867   
868   
869   
870   
871   
872   
873   
874   
875   
876   
877   
878   
879   
880   
881   
882   
883   
884   
885   
886   
887
The rest of this proof is devoted to the construction of  $f^{k} = \mathcal{R}_{\tilde{\theta}^{k}}$  where  $\tilde{\theta}^k \in \mathcal{N}_{\mathbf{N}}$  are parameters of a neural network of the same dimension as those in  $\mathcal{N}_{\mathbf{I}}$  but only partially satisfying the support constraint  $\mathbf{I}$ . To guarantee that  $f^{k}$  converges pointwise to  $f$ , we construct  $\tilde{\theta}^k$  based on  $\theta^k$  and harness their relation.
Choice of parameters. We set  $\tilde{\theta}^k = \{(\tilde{\mathbf{W}}_i^k,\tilde{\mathbf{b}}_i^k)_{i = 1}^2\} \in \mathcal{N}_{\mathbf{N}}$  where  $\tilde{\mathbf{W}}_2^k\in \mathbb{R}^{N_2\times N_1}$ ,  $\tilde{\mathbf{W}}_1^k\in \mathbb{R}^{N_1\times N_0}$  are defined as follows, where we use  $C^k\coloneqq \sup_{x\in \Omega}\| \mathbf{W}_1^k x\|_\infty$ :
- For inactive neurons  $i \in \bar{J}$ , we simply set  $(\tilde{\mathbf{W}}_1^k[:,i],\tilde{\mathbf{W}}_2^k[:,i],\tilde{\mathbf{b}}_1^k[:,i]) = (\mathbf{W}_1^k[:,i],\mathbf{W}_2^k[:,i],\mathbf{b}_1^k[:,i])$ .
- For each equivalence class of active neurons  $1 \leq \ell \leq r$ , we choose some  $j_{\ell} \in J_{\ell}^{+}$  (note that  $J_{\ell}^{+}$  is non-empty due to Definition C.1) and set the parameters  $(\tilde{\mathbf{W}}_2^k[:,i], \tilde{\mathbf{W}}_1^k[i,:], \mathbf{b}_1[i])$ ,  $i \in J_{\ell}$  as:
$$
\left(\tilde {\mathbf {W}} _ {1} ^ {k} [ i, : ], \tilde {\mathbf {W}} _ {2} ^ {k} [:, i ], \tilde {\mathbf {b}} _ {1} ^ {k} [ i ]\right) = \left\{ \begin{array}{l l} \left(\mathbf {W} _ {1} ^ {k} [ i,: ], \mathbf {W} _ {2} ^ {k} [:, i ], C ^ {k}\right), & \forall j \in J _ {\ell} ^ {-} \\ \left(\mathbf {W} _ {1} ^ {k} [ i,: ], \mathbf {0}, C ^ {k}\right), & \forall i \in J _ {\ell} ^ {+} \backslash \left\{j _ {\ell} \right\} \\ \left(\alpha_ {\ell}, \gamma_ {\ell}, \beta_ {\ell}\right), & i = j _ {\ell} \end{array} \right. \tag {31}
$$
For  $i \in J_{\ell} \setminus \{j_{\ell}\}$ , we clearly have:  $\operatorname{supp}(\tilde{\mathbf{W}}_1^k[i,:]) \subseteq I_1[i,:]$  and  $\operatorname{supp}(\tilde{\mathbf{W}}_2^k(:,i]) \subseteq I_2(:,i]$ . The  $j_{\ell}$ -th column of  $\tilde{\mathbf{W}}_2^k$  is the only one that does not necessarily satisfy the support constraint, as  $\operatorname{supp}(\gamma_{\ell}) \not\subset I_2(:,j_{\ell}]$  in general.
- Finally, the output bias  $\mathbf{b}_2^k$  is set as:
$$
\tilde {\mathbf {b}} _ {2} ^ {k} := \mathbf {b} _ {2} ^ {k} + \sum_ {\ell = 1} ^ {r} \underbrace {\sum_ {i \in J _ {\ell} ^ {-}} \left(\mathbf {b} _ {1} ^ {k} [ i ] - C ^ {k}\right) \mathbf {W} _ {2} ^ {k} [: , i ]} _ {=: \xi_ {\ell} ^ {k}} \tag {32}
$$
Proof that  $f^k \coloneqq R_{\tilde{\theta}^k}$  converges pointwise to  $f$  on  $\Omega \setminus (\cup_{\ell=1}^r H_\ell)$ . We introduce notations analogous to Definition C.2: for every  $x \in \mathbb{R}^{N_0}$  we define:
$$
\tilde {h} _ {i} ^ {k} (x) = \tilde {\mathbf {W}} _ {2} ^ {k} [:, i ] \sigma (\tilde {\mathbf {W}} _ {1} ^ {k} [ i,: ] x + \tilde {\mathbf {b}} _ {1} ^ {k} [ i]), \quad i = 1, \ldots , N _ {1}; \qquad \tilde {g} _ {\ell} ^ {k} (x) = \sum_ {i \in J _ {\ell}} \tilde {h} _ {i} ^ {k} (x), \quad \ell = 1, \ldots , r
$$
By construction
$$
\tilde {h} _ {i} ^ {k} = h _ {i} ^ {k}, \quad \forall i \in \bar {J}, \forall k, \tag {33}
$$
and we further explicit the form of  $\tilde{h}_i^k$ ,  $i \in J_\ell$  for  $x \in \Omega$  (but not on  $\mathbb{R}^{N_0}$ ) as:
$$
\tilde {h} _ {i} ^ {k} (x) = \left\{ \begin{array}{l l} \mathbf {W} _ {2} ^ {k} [:, i ] \left(\mathbf {W} _ {1} ^ {k} [ i,: ] x + C ^ {k}\right), & \forall i \in J _ {\ell} ^ {-} \\ 0, & \forall i \in J _ {\ell} ^ {+} \backslash \left\{j _ {\ell} \right\}, \\ \gamma_ {\ell} \sigma \left(\alpha_ {\ell} x + \beta_ {\ell}\right), & i = j _ {\ell} \end{array} \right. \tag {34}
$$
We justify our formula in Equation (34) as follow:
1. For  $i \in J_{\ell}^{-}$ : since  $C^k = \sup_{x \in \Omega} \| \mathbf{W}_1^k x \|_{\infty}$  by construction,  $\tilde{\mathbf{W}}_1^k[i,:] x + \mathbf{b}_1^k[i] = \mathbf{W}_1^k[i,:] x + \mathbf{b}_1^k[i] \geq 0$ . The activation  $\sigma$  acts simply as an identity function.
2. For  $i\in J_{\ell}^{+}$  : Because we choose  $\tilde{\mathbf{W}}_2^k [(:,i] = \mathbf{0}$
3. For  $i = j_{\ell}$ : Obvious due to the construction in Equation (31).
Given  $x\in \Omega \setminus (\cup_{\ell = 1}^{r}H_{\ell})$  , we now prove that this construction ensures that for each  $\ell \in \{1,\dots ,r\}$
$$
\lim  _ {k \rightarrow \infty} \left(\tilde {g} _ {\ell} ^ {k} (x) - g _ {\ell} ^ {k} (x) + \xi_ {\ell} ^ {k}\right) = 0. \tag {35}
$$
889   
890   
891   
892   
893   
894   
895   
896   
897   
898   
899   
900   
901   
902   
903   
904   
905
This will imply the claimed poinwise convergence since
$$
\begin{array}{l} \lim  _ {k \rightarrow \infty} f ^ {k} (x) = \lim  _ {k \rightarrow \infty} R _ {\tilde {\theta} ^ {k}} (x) = \lim  _ {k \rightarrow \infty} \left(\sum_ {i \in \bar {J}} \tilde {h} _ {i} ^ {k} (x) + \sum_ {\ell = 1} ^ {r} \tilde {g} _ {\ell} ^ {k} (x) + \tilde {\mathbf {b}} _ {2} ^ {k}\right) \\ \stackrel {(3 3) \&(3 5)} {=} \lim  _ {k \rightarrow \infty} \left(\sum_ {i \in \bar {J}} h _ {i} ^ {k} (x) + \sum_ {\ell = 1} ^ {r} g _ {\ell} ^ {k} (x) - \sum_ {\ell = 1} ^ {r} \xi_ {\ell} ^ {k} + \tilde {\mathbf {b}} _ {2} ^ {k}\right) \\ \stackrel {(3 2)} {=} \lim  _ {k \to \infty} \left(\sum_ {i \in \bar {J}} h _ {i} ^ {k} (x) + \sum_ {\ell = 1} ^ {r} g _ {\ell} ^ {k} (x) + \mathbf {b} _ {2} ^ {k}\right) = \lim  _ {k \to \infty} R _ {\theta^ {k}} (x) = f (x). \\ \end{array}
$$
To establish (35), observe that as  $x \in \Omega \setminus (\cup_{\ell=1}^{r} H_{\ell})$  we have  $x \notin H_{\ell}$ . We thus distinguish two cases: Case  $x \in H_{\ell}^{-}$ .
908 Using (31) we show below that for  $k$  large enough and  $x \in H_{\ell}^{-}$ , we have
$$
\tilde {h} _ {i} ^ {k} (x) - h _ {i} ^ {k} (x) = \left\{ \begin{array}{l l} \left(C ^ {k} - \mathbf {b} _ {1} ^ {k} [ i ]\right) \mathbf {W} _ {2} ^ {k} [:, i ], & i \in J _ {\ell} ^ {-} \\ 0, & i \in J _ {\ell} ^ {+} \end{array} \right. \tag {36}
$$
$$
\tilde {g} _ {\ell} ^ {k} (x) - g _ {\ell} ^ {k} (x) + \xi_ {\ell} ^ {k} = \sum_ {i \in J _ {\ell}} \left(\tilde {h} _ {i} ^ {k} (x) - h _ {i} ^ {k} (x)\right) + \xi_ {\ell} ^ {k} = \sum_ {i \in J _ {\ell} ^ {-}} \left(C ^ {k} - \mathbf {b} _ {1} ^ {k} [ i ]\right) \mathbf {W} _ {2} ^ {k} [:, i ] + \xi_ {\ell} ^ {k} = 0.
$$
We indeed obtain (36) as follows. Since  $x \in H_{\ell}^{-}$ ,  $\alpha_{\ell}x + \beta_{\ell} < 0$ , i.e.,  $-\alpha_{\ell}x - \beta_{\ell} > 0$ . Therefore, given the definitions of  $J_{\ell}^{\pm}$  (cf Definition C.1) we have:
- For  $i \in J_{\ell}^{-}$ :  $\lim_{k \to \infty} (\mathbf{W}_1^k[i,:], \mathbf{b}_1^k[i]) = -(\alpha_\ell, \beta_\ell)$ , hence for  $k$  large enough, we have  $\mathbf{W}_1^k[i,:]x + \mathbf{b}_1^k[i] > 0$  so that  $\sigma(\mathbf{W}_1^k[i,:]x + \mathbf{b}_1^k[i]) = \mathbf{W}_1^k[i,:]x + \mathbf{b}_1^k[i]$  and, as expressed in (36):
$$
\tilde {h} _ {i} ^ {k} (x) - h _ {i} ^ {k} (x) \stackrel {(3 4)} {=} \mathbf {W} _ {2} ^ {k} [:, i ] (\mathbf {W} _ {1} ^ {k} [ i,: ] x + C ^ {k}) - \mathbf {W} _ {2} ^ {k} [:, i ] (\mathbf {W} _ {1} ^ {k} [ i,: ] x + \mathbf {b} _ {1} ^ {k} [ i ]) = (C ^ {k} - \mathbf {b} _ {1} ^ {k} [ i ]) \mathbf {W} _ {2} ^ {k} [:, i ].
$$
- For  $i \in J_{\ell}^{+}$ : similarly, we have  $\mathbf{W}_1^k[i,:]x + \mathbf{b}_1^k[i] < 0$  for  $k$  large enough. Therefore,  $h_i^k(x) = 0$  for  $k$  large enough. The fact that we also have  $\tilde{h}_i^k(x) = 0$  is immediate from Equation (34) if  $i \neq j_{\ell}$ , and for  $i = j_{\ell}$  we also get from Equation (34) that  $\tilde{h}_i^k(x) = \gamma_{\ell}\sigma(\alpha_{\ell}x + \beta_{\ell}) = 0$  since  $\alpha_{\ell}x + \beta_{\ell} < 0$ .
Case  $x\in H_{\ell}^{+}$  . An analog to Equation (36) for  $x\in H_{\ell}^{+}$  is
$$
\tilde {h} _ {i} ^ {k} (x) - h _ {i} ^ {k} (x) = \left\{ \begin{array}{l l} \mathbf {W} _ {2} ^ {k} [:, i ] \left(\mathbf {W} _ {1} ^ {k} [ i,: ] x + C ^ {k}\right), & i \in J _ {\ell} ^ {-} \\ - \mathbf {W} _ {2} ^ {k} [:, i ] \left(\mathbf {W} _ {1} ^ {k} [ i,: ] x + \mathbf {b} _ {1} ^ {k} [ i]\right), & i \in J _ {\ell} ^ {+} \backslash \{j \}. \\ \gamma_ {\ell} \left(\alpha_ {\ell} x + \beta_ {\ell}\right) - \mathbf {W} _ {2} ^ {k} [:, i ] \left(\mathbf {W} _ {1} ^ {k} [ i,: ] x + \mathbf {b} _ {1} ^ {k} [ i]\right), & i = j \end{array} \right. \tag {37}
$$
We establish it before concluding for this case.
- For  $i \in J_{\ell}^{-}$ : by a reasoning analog to the case  $x \in H_{\ell}^{-}$ , we deduce that for  $k$  large enough
$$
\tilde {h} _ {i} ^ {k} (x) - h _ {i} ^ {k} (x) \stackrel {(3 4)} {=} \mathbf {W} _ {2} ^ {k} [:, i ] (\mathbf {W} _ {1} ^ {k} [ i,: ] x + C ^ {k}).
$$
- For  $i \in J_{\ell}^{+}$ : a similar reasoning yields  $h_i^k(x) = \mathbf{W}_2^k[;, i] (\mathbf{W}_1^k[i, :]x + \mathbf{b}_1^k[i])$  for  $k$  large enough,
- while Equation (34) yields  $\tilde{h}_{j_\ell}^k(x) = \gamma_\ell \sigma(\alpha_\ell x + \beta_\ell) = \gamma_\ell(\alpha_\ell x + \beta_\ell)$  (since  $\alpha_\ell x + \beta_\ell > 0$  as  $x \in H_\ell^+$ )
- and  $\tilde{h}_i^k(x) = 0$  if  $i \neq j_\ell$ .
Using (37) we obtain for  $k$  large enough