SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \tilde {g} _ {\ell} ^ {k} (x) - g _ {\ell} ^ {k} (x) + \xi_ {\ell} ^ {k} = \sum_ {i \in J _ {\ell}} \left(\tilde {h} _ {i} ^ {k} (x) - h _ {i} ^ {k} (x)\right) + \xi_ {\ell} ^ {k} \\ = \sum_ {i \in J _ {\ell} ^ {-}} \mathbf {W} _ {2} ^ {k} [:, i ] (\mathbf {W} _ {1} ^ {k} [ i,: ] x + C ^ {k}) - \sum_ {i \in J _ {\ell} ^ {+}} \mathbf {W} _ {2} ^ {k} [:, i ] (\mathbf {W} _ {1} ^ {k} [ i,: ] x + \mathbf {b} _ {1} ^ {k} [ i ]) + \gamma_ {\ell} (\alpha_ {\ell} x + \beta_ {\ell}) + \xi_ {\ell} ^ {k} \\ \stackrel {(3 2)} {=} \left(\sum_ {i \in J _ {\ell} ^ {-}} \mathbf {W} _ {2} ^ {k} [:, i ] \mathbf {W} _ {1} ^ {k} [ i,: ] - \sum_ {j \in J _ {\ell} ^ {+}} \mathbf {W} _ {2} ^ {k} [:, i ] \mathbf {W} _ {1} ^ {k} [ i,: ] + \gamma_ {\ell} \alpha_ {\ell}\right) x \\ + \Big (\underbrace {\xi_ {\ell} ^ {k} + \sum_ {i \in J _ {\ell} ^ {-}} \mathbf {W} _ {2} ^ {k} [: , i ] C ^ {k}} _ {\sum_ {i \in J _ {\ell} ^ {-}} \mathbf {W} _ {2} ^ {k} [: , i ] \mathbf {b} _ {1} ^ {k} [ i ]} - \sum_ {i \in J _ {\ell} ^ {+}} \mathbf {W} _ {2} ^ {k} [:, i ] \mathbf {b} _ {1} ^ {k} [ i ] + \gamma_ {\ell} \beta_ {\ell} \Big) \\ \end{array}
$$
where in the last line we used the expression of  $\xi_{\ell}^{k}$  from (32). Due to Equations (23) and (24) it follows that  $\lim_{k\to \infty}\tilde{g}_{\ell}^{k}(x) - g_{\ell}^{k}(x) + \xi_{\ell}^{k} = 0, \forall x\in H_{\ell}^{+}$ .
Thus combining both cases, we conclude that  $\lim_{k\to \infty}\tilde{g}_\ell^k (x) - g_\ell^k (x) + \xi_\ell^k = 0,\forall x\notin H_\ell$  , as desired.
Proof of the expression (30) with  $\mathbf{A}^k\in \mathcal{L}_{\mathbf{I}}$  for large enough  $k$ . From (34), we first deduce that
$$
f ^ {k} (x) = \sum_ {i = 1} ^ {N _ {1}} \tilde {h} _ {i} ^ {k} (x) + \tilde {\mathbf {b}} _ {2} ^ {k} = \sum_ {\ell = 1} ^ {r} \gamma_ {\ell} \sigma (\alpha_ {\ell} x + \beta_ {\ell}) + \sum_ {i \in S} \tilde {h} _ {i} ^ {k} (x) + \tilde {\mathbf {b}} _ {2} ^ {k}, \quad \forall x \in \mathbb {R} ^ {N _ {0}}.
$$
where we recall that  $S \coloneqq \bar{J} \cup (\cup_{1 \leq \ell \leq r} J_{\ell}^{-})$ . There only remains to show that, for  $k$  large enough, we have  $\sum_{i \in S} \tilde{h}_i^k(x) = \mathbf{A}^k x + \mathbf{b}^k$  for every  $x$  in the restricted domain  $\Omega_{\delta}^{\circ}$ , where  $\mathbf{A}^k \in \mathcal{L}_{\mathbf{I}'}$  and  $\mathbf{b}^k \in \mathbb{R}^{N_2}$ . Note that for  $i \in J_{\ell}$ , our construction assures that  $\tilde{h}_i^k$  is affine on  $\Omega$ . Moreover, in the restricted domain  $\Omega_{\delta}^{\circ}$ , for  $k \geq \kappa_{\delta}$  large enough,  $\tilde{h}_i^k$ ,  $i \in \bar{J}$  also behave like affine functions (cf Lemma C.5). Therefore,
$$
\sum_ {i \in S} \tilde {h} _ {i} ^ {k} (x) = \left(\sum_ {i \in S} \delta_ {i} ^ {k} \tilde {\mathbf {W}} _ {2} ^ {k} [:, i ] \tilde {\mathbf {W}} _ {1} ^ {k} [ i,: ]\right) x + \mathbf {c} ^ {k}, \quad \forall x \in \Omega_ {\delta} ^ {\circ}, k \geq \kappa_ {\delta}
$$
for some vector  $\mathbf{c}^k$  and binary scalars  $\delta_i^k$ . In fact,  $\delta_i^k = 0$  if  $i \in \bar{J}^- := \{j \in \bar{J} | \mathbf{W}_1^* [j,:]x + \mathbf{b}_1^* [j] \leq 0, \forall x \in \Omega\}$  and  $\delta_k^i = 1$  otherwise. Thus, one chooses  $\mathbf{A}^k = \sum_{i \in S} \delta_i^k \tilde{\mathbf{W}}_2^k[:, i] \tilde{\mathbf{W}}_1^k[i,:]$ ,  $\mathbf{b}^k = \mathbf{c}^k$  and the construction is complete. This construction allows us to write  $\mathbf{A}^k = \hat{\mathbf{W}}_2^k \hat{\mathbf{W}}_1^k$  with:
$$
\begin{array}{l} \tilde {\mathbf {W}} _ {1} ^ {k} = \tilde {\mathbf {W}} _ {1} ^ {k} [ S, : ] \\ \hat {\mathbf {W}} _ {2} ^ {k} = \tilde {\mathbf {W}} _ {2} ^ {k} [:, S ] \operatorname {d i a g} \left(\left\{\nu_ {i} ^ {k} \mid i = 1, \dots , N _ {1} \right\}\right) \\ \end{array}
$$
where  $\mathrm{diag}(\{\nu_i^k\mid i = 1,\dots ,N_1\})\in \mathbb{R}^{N_1\times N_1}$  is a diagonal matrix,  $\nu_{i}^{k} = \delta_{i}^{k}$  for  $i\in S$  and 0 otherwise. It is also evident that  $\mathrm{supp}(\hat{\mathbf{W}}_2^k [.;S])\subseteq I_2[.;,S],\mathrm{supp}(\hat{\mathbf{W}}_1^k [S,:])\subseteq I_1[S,:]$ . (since the multiplication with a diagonal matrix does not increase the support of a matrix). This concludes the proof.
C.3 Proof for Corollary 4.2
Proof. The proof is inductive on the number of hidden neurons  $N_{1}$ :
1. Basic case  $N_{1} = 1$ : Consider  $\theta \coloneqq \{(\mathbf{W}_i,\mathbf{b}_i)_{i = 1}^2\} \in \mathcal{N}_{\mathbf{I}}$ , the function  $\mathcal{R}_{\theta}$  has the form:
$$
\mathcal {R} _ {\theta} (x) = \mathbf {w} _ {2} \sigma \left(\mathbf {w} _ {1} ^ {\top} x + \mathbf {b} _ {1}\right) + \mathbf {b} _ {2}
$$
where  $\mathbf{w}_1 = \mathbf{W}_1[1, :] \in \mathbb{R}^{N_0}$ ,  $\mathbf{w}_2 = \mathbf{W}_2[1, 1] \in \mathbb{R}$ . There are two possibilities:
(a)  $I_{2} = \emptyset$ : then  $\mathbf{w}_2 = 0$ ,  $\mathcal{F}_I$  is simply a set of constant functions on  $\Omega$ , which is closed.
919   
920   
921   
922   
923   
924   
925   
926   
927   
928   
929
(b)  $I_{2} = \{(1,1)\}$ : We have  $I_{2} = \mathbf{1}_{1\times N_{1}}$ , which makes the first assumption of Theorem 4.2 satisfied. To check that the second assumption of Theorem 4.2 also holds, we consider all the possible non-empty subsets  $S$  of  $\llbracket 1\rrbracket$ : there is only one non-empty subset of  $I_{2}$ , which is  $S = \llbracket 1\rrbracket$ . In that case,  $\mathcal{L}_{\mathbf{I}_S} = \{\mathbf{W}\in \mathbb{R}^{1\times N_0}\mid \operatorname {supp}(\mathbf{W})\subseteq I_1\}$ , which is closed (since  $\mathcal{L}_{\mathbf{I}_S}$  is isomorphic to  $\mathbb{R}^{|I_1|}$ ). The result thus follows using Theorem 4.2.
2. Assume the conclusion of the theorem holds for all  $1 \leq N_{1} \leq k$  (and any  $N_{0} \geq 1$ ). We need to prove the result for  $N_{1} = k + 1$ . Define  $H = \{i \mid I_{2}[1, i] = 1\}$  the set of hidden neurons that are allowed to be connected to the output via a nonzero weight. Consider two cases:
(a) If  $|H| \leq k$ , we have  $\mathcal{F}_{\mathbf{I}} = \mathcal{F}_{\mathbf{I}_H}$ , which is closed due to the induction hypothesis.
(b) If  $H = \llbracket k + 1\rrbracket$ , we can apply Theorem 4.2. Indeed, since  $I_{2} = \mathbf{1}_{1\times N_{1}}$ , the first condition of Theorem 4.2 is satisfied. In addition, for any non-empty  $S\subseteq [[N_1]]$ , define  $\mathcal{H}:= \cup_{i\in S}I[i,:]\subseteq[[N_0]]$  the union of row supports of  $I_{1}[S,:]$ . It is easy to verify that  $\mathcal{L}_{\mathbf{I}_S}$  is isomorphic to  $\mathbb{R}^{|\mathcal{H}|}$  which is closed. As such, Theorem 4.2 can be applied.
C.4 Other technical lemmas
Lemma C.7 (Convergence of affine function). Let  $\Omega$  be a non-empty interior subset  $\mathbb{R}^n$ . If the sequence  $\{f^k\}_{k\in \mathbb{N}}, f^k: \mathbb{R}^n \mapsto \mathbb{R}^m: x \mapsto \mathbf{A}^k x + \mathbf{b}^k$  where  $\mathbf{A}^k \in \mathbb{R}^{m\times n}$ ,  $\mathbf{b}^k \in \mathbb{R}^m$  converges pointwise to a function  $f$  on  $\Omega$ , then  $f$  is affine (i.e.,  $f = \mathbf{A}x + \mathbf{b}$  for some  $\mathbf{A} \in \mathbb{R}^{m\times n}$ ,  $\mathbf{b} \in \mathbb{R}^m$ ). Moreover,  $\lim_{k\to \infty}\mathbf{A}^k = \mathbf{A}$  and  $\lim_{k\to \infty}\mathbf{b}_k = \mathbf{b}$ .
Proof. Consider  $x_0 \in \Omega'$ , an open subset of  $\Omega$  ( $\Omega'$  exists since  $\Omega$  is a non-empty interior subset of  $\mathbb{R}^n$ ). Define  $g^k(x) = f^k(x) - f^k(x_0)$  and  $g(x) = f(x) - g(x_0)$ . The function  $g^k$  is linear and  $g^k$  converges pointwise to  $g$  on  $\Omega$  (and thus, on  $\Omega'$ ). We first prove that  $g$  is linear. Indeed, for any  $x, y \in \Omega, \alpha, \beta \in \mathbb{R}$  such that  $\alpha x + \beta y \in \Omega$ , we have:
$$
\begin{array}{l} g (\alpha x + \beta y) = \lim  _ {k \rightarrow \infty} g ^ {k} (\alpha x + \beta y) \\ = \lim  _ {k \rightarrow \infty} \alpha g ^ {k} (x) + \beta g ^ {k} (y) \\ = \alpha \lim  _ {k \rightarrow \infty} g ^ {k} (x) + \beta \lim  _ {k \rightarrow \infty} g ^ {k} (y) \\ = \alpha g (x) + \beta g (y) \\ \end{array}
$$
Therefore, there must exist  $\mathbf{A} \in \mathbb{R}^{m \times n}$  such that  $g(x) = \mathbf{A}x$ . Choosing  $\mathbf{b} := g(x_0)$ , we have  $f(x) = g(x) + g(x_0) = \mathbf{A}x + \mathbf{b}$ .
Moreover, since  $\Omega'$  is open, there exists a positive  $r$  such that the ball  $\mathcal{B}(x,r)\subseteq \Omega'$ . Choosing  $x_{i} = x_{0} + (r / 2)\mathbf{e}_{i}$  with  $\mathbf{e}_i$  the  $i$ th canonical vector, we have:
$$
\lim _ {k \to \infty} g ^ {k} (x _ {i}) = \lim _ {k \to \infty} (r / 2) \mathbf {A} ^ {k} \mathbf {e} _ {i} = (r / 2) \mathbf {A} e _ {i},
$$
or, equivalently, the  $i$ th column of  $\mathbf{A}$  is the limit of the sequence generated by the  $i$ th column of  $\mathbf{A}^k$ . Repeating this argument for all  $1 \leq i \leq n$ , we have  $\lim_{k \to \infty} \mathbf{A}^k = \mathbf{A}$ . This also implies  $\lim_{k \to \infty} \mathbf{b}^k = \mathbf{b}$  immediately.
D Closedness does not imply the best approximation property
Since we couldn't find any source discussing the fact that closedness does not imply the BAP, we provide an example to show this fact.
Consider  $C^0([-1,1])$  the set of continuous functions on the interval  $[-1,1]$ , equipped with the norm  $\sup \|f\|_{\infty} = \max_{x \in [-1,1]} |f(x)|$ , and define  $S$ , the subset of all functions  $f \in C^0([-1,1])$  such that:
$$
\int_ {0} ^ {1} f d x - \int_ {- 1} ^ {0} f d x = 1
$$
It is easy to verify that  $S$  is closed. We show that the constant function  $f = 0$  does not have a projection in  $S$  (i.e., a function  $g \in S$  such that  $\| f - g \|_{\infty} = \inf_{h \in S} \| f - h \|_{\infty}$ ).
930   
931   
932   
933   
934   
935   
936   
937   
938   
939   
940   
941   
942   
943   
944   
945   
946   
947   
948   
949   
950   
951   
952   
953   
954   
955   
956   
957
First we observe that since  $f = 0$ , we have  $\| f - h \|_{\infty} = \| h \|_{\infty}$  for each  $h \in S$ , and we show that  $\inf_{h \in S} \| f - h \|_{\infty} \geq 1/2$ . Indeed, for  $h \in S$  we have:
$$
1 = \int_ {0} ^ {1} h d x - \int_ {- 1} ^ {0} h d x \leq \left| \int_ {0} ^ {1} h d x \right| + \left| \int_ {- 1} ^ {0} h d x \right| \leq 2 \| h \| _ {\infty} = 2 \| f - h \| _ {\infty}. \tag {38}
$$
Secondly, we show a sequence of  $\{h_n\}_{k\in \mathbb{N}}$  such that  $h_n\in S$  and  $\lim_{n\to \infty}\| h_n\|_{\infty} = 1 / 2$ . Consider the odd function  $h_n$  (i.e.  $h_n(x) = -h_n(-x)$ ) such that:
$$
h _ {n} (x) = \left\{ \begin{array}{l l} c _ {n}, & x \in [ 1 / n, 1 ] \\ n c _ {n} x & x \in [ 0, 1 / n) \end{array} \right.
$$
where  $c_{n} = n / (2n - 1)$ . It is evident that  $h_n \in S$  because:
$$
\begin{array}{l} \int_ {0} ^ {1} h _ {n} d x - \int_ {- 1} ^ {0} h _ {n} d x = 2 \int_ {0} ^ {1} h _ {n} d x = 2 \left(\int_ {0} ^ {1 / n} h _ {n} d x + \int_ {1 / n} ^ {1} h _ {n} d x\right) \\ = 2 \left(\frac {c _ {n}}{2 n} + \frac {c _ {n} (n - 1)}{n}\right) = \frac {c _ {n} (2 n - 1)}{n} = 1 \\ \end{array}
$$
Moreover, we also have  $\lim_{n\to \infty}\| h_n\|_{\infty} = \lim_{n\to \infty}c_n = 1 / 2$
Finally, we show that  $1/2$  cannot be attained. By contradiction, assume that there exists  $g \in S$  such that  $\| f - g \|_{\infty} = 1/2$ , i.e., as we have seen,  $\| g \|_{\infty} = 1/2$ . Using Equation (38), the equality will only hold if  $g(x) = 1/2$  in  $[0,1]$  and  $g(x) = -1/2$  in  $[-1,0]$ . However,  $g$  is not continuous, a contradiction.
958 959   
960   
961   
962   
963   
964