SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sketch of the proof. Since  $\mathcal{L}_{\mathbf{I}}$  is not closed, there exists  $\mathbf{A} \in \overline{\mathcal{L}_{\mathbf{I}}} \setminus \mathcal{L}_{\mathbf{I}}$  ( $\overline{\mathcal{L}}$  is the closure of the set  $\mathcal{L}$ ). Considering  $f(x) \coloneqq \mathbf{A}x$ , we construct a set  $\Omega = \{x_{i}\}_{i=1}^{P}$  such that  $[f(x_{1}), \ldots, f(x_{P})] \in \overline{\mathcal{F}_{\mathbf{I}}(\Omega)} \setminus \mathcal{F}_{\mathbf{I}}(\Omega)$ . Therefore,  $\mathcal{F}_{\mathbf{I}}(\Omega)$  is not closed.
The proof is in Appendix B.2. Besides showing a topological connection between  $\mathcal{F}_{\mathbf{I}}$  (NNs with ReLU activation) and  $\mathcal{L}_{\mathbf{I}}$  (linear NNs), Theorem 3.1 leads to a simple example where  $\mathcal{F}_{\mathbf{I}}$  is not closed.
Example 3.1 (LU architecture). Consider  $\mathbf{I} = (I_2, I_1) \in \{0, 1\}^{d \times d} \times \{0, 1\}^{d \times d}$  where  $I_1 = \{(i, j) \mid 1 \leq i \leq j \leq d\}$  and  $I_2 = \{(i, j) \mid 1 \leq j \leq i \leq d\}$ . Any pair of matrices  $\mathbf{X}_2, \mathbf{X}_1 \in \mathbb{R}^{d \times d}$  such that  $\mathrm{supp}(\mathbf{X}_i) \subseteq I_i$ ,  $i = 1, 2$  are respectively lower and upper triangular matrices. Therefore,  $\mathcal{L}_{\mathbf{I}}$  is the set of matrices that admit an exact lower-upper (LU) factorization/decomposition. That explains its name: LU architecture. This set is well known to a) contain an open and dense subset of  $\mathbb{R}^{d \times d}$ ; b) be strictly contained in  $\mathbb{R}^{d \times d}$  [13, Theorem 3.2.1] [25, Theorem 1]. Therefore,  $\mathcal{L}_{\mathbf{I}}$  is not closed and by the contraposition of Theorem 3.1 we conclude that there exists a finite set  $\Omega$  such that  $\mathcal{F}_{\mathbf{I}}(\Omega)$  is not closed.
Figure 1: Training a one-hidden-layer fixed support (LU architecture) neural network with different regularization hyperparameters  $\lambda$  (we use weight decay, i.e., an  $L^2$  regularizer). Subfigures a)-b) show the relative loss (the lower, the better) for training (empirical loss) and testing (validation loss) respectively. Subfigure c) shows the norm of two weight matrices. The experiments are conducted 10 times to produce the error bars in all figures (almost invisible due to a small variability).
Let us illustrate the impact of the non-closedness in Example 3.1 via the behavior during the training of a fixed support one-hidden-layer neural network with the LU support constraint  $\mathbf{I}$ . This network is trained to learn the linear function  $f(x) \coloneqq \mathbf{A}x$  where  $\mathbf{A} \in \mathbb{R}^{d \times d}$  is an anti-diagonal identity matrix. Using the necessary and sufficient condition of LU decomposition existence [25, Theorem 1], we have that  $\mathbf{A} \in \overline{\mathcal{L}_{\mathbf{I}}} \backslash \mathcal{L}_{\mathbf{I}}$  as in the sketch proof of Lemma 3.2. Given network parameters  $\theta$  and a training set, approximation quality can be measured by the relative loss:  $\frac{1}{P} \left( \sum_{i=1}^{P} \| \mathcal{R}_{\theta}(x_i) - y_i \|_2^2 / \| y_i \|_2^2 \right)$ .
Figure 1 illustrates the behavior of the relative errors of the training set, validation set and the sum of weight matrices norm along epochs, using Stochastic Gradient Descent (SGD) with batch size 3000, learning rate 0.1, momentum 0.9 and four different weight decays (the hyperparameter controlling the  $L^2$  regularizer)  $\lambda \in \times \{0,10^{-4},5\times 10^{-4},10^{-3}\}$ . The case  $\lambda = 0$  corresponds to the unregularized case. Our training and testing sets contain each  $P = 10^{5}$  samples generated independently as  $x_{i}\sim \mathcal{U}([-1,1]^{d})$  ( $d = 100$ ) and  $y_{i}\coloneqq \mathbf{A}x_{i}$ .
Example 3.1 and Figure 1 also lead to two interesting remarks: while the  $L^2$  regularizer (weight decay) does prevent the parameter divergence phenomenon, the empirical loss is improved when using the non-regularized version. This is the situation where adding a regularization term might be detrimental, as stated earlier. More interestingly, the size of the dataset is  $10^5$ , which is much smaller than the theoretical  $P$  in Lemma 3.2. It is thus interesting to see if we can reduce the theoretical value of  $P$ , which is currently exponential w.r.t. to the input dimension.
197   
198   
199   
200   
201   
202   
203   
204   
205   
206   
207   
208   
209   
210   
211   
212   
213   
214   
215   
216   
217   
218   
219   
220   
221   
222   
223   
224   
225   
226   
227   
228   
229   
230   
231
3.3 The closedness of  $\mathcal{L}_{\mathrm{I}}$  is algorithmically decidable
Theorem 3.1 leads to a natural question: given  $\mathbf{I}$ , how to check the closedness of  $\mathcal{L}_{\mathbf{I}}$ , a subset of  $\mathbb{R}^{N_L\times N_0}$ . To the best of our knowledge, there is not any study on the closedness of  $\mathcal{L}_{\mathbf{I}}$  in the literature. It is, thus, not known whether deciding on the closedness of  $\mathcal{L}_{\mathbf{I}}$  for a given  $\mathbf{I}$  is polynomially tractable. In this work, we show it is at least decidable with a doubly-exponential algorithm. This algorithm is an application of quantifier elimination, an algorithm from real algebraic geometry [2].
Lemma 3.3. Given  $\mathbf{I} = (I_L, \ldots, I_1)$ , the closedness of  $\mathcal{L}_{\mathbf{I}}$  is decidable with an algorithm of complexity  $O((4L)^{C^{k-1}})$  where  $k = N_L N_0 + 1 + 2 \sum_{i=1}^L |L_i|$  and  $C$  is a universal constant.
We prove Lemma 3.3 in Appendix B.4. Since the knowledge of  $\mathbf{I}$  is usually available (either fixed before training [19, 31, 4, 21, 3] or discovered by a procedure before re-training [10, 14, 32]), the algorithm in Lemma 3.3 is able to verify whether the training problem might not admit an optimum. While such a doubly exponential algorithm in Lemma 3.3 is seemingly impractical in practice, small toy examples (for example, Example 3.1 with  $d = 2$ ) can be verified using Z3Prover<sup>1</sup>, a software implementing exactly the algorithm in Lemma 3.3. However, Z3Prover is already unable to terminate when run on the LU architecture of Example 3.1 with  $d = 3$ . This calls for more efficient algorithms to determine the closedness of  $\mathcal{L}_{\mathbf{I}}$  given  $\mathbf{I}$ . The same algorithmic question can be also asked for  $\mathcal{F}_{\mathbf{I}}$ . We leave these problems (in this general form) as open questions.
In fact, if such a polynomial algorithm (to decide the closedness of  $\mathcal{L}_{\mathbf{I}}$ ) exists, it can be used to answer the following interesting question:
Question 3.2. If the supports of the weight matrices are randomly sampled from a distribution, what is the probability that the corresponding training problem potentially admits no optimal solutions?
While simple, this setting does happen in practice since random supports/binary masks are considered a strong and common baseline for sparse DNNs training [22]. Thanks to Theorem 3.1, if  $\mathcal{L}_{\mathbf{I}}$  is not closed then the support is "bad". Thus, to have an estimation of a lower bound on the probability of "bad" supports, we could sample the supports from the given distribution and use the polynomial algorithm in question to decide if  $\mathcal{L}_{\mathbf{I}}$  is closed. Unfortunately, the algorithm in Lemma 3.3 has doubly exponential complexity, thus hindering its practical use. However, for one-hidden-layer NNs, there is a polynomial algorithm to detect non-closedness: intuitively, if the support constraint is "locally similar" to the LU structure, then  $\mathcal{L}_{\mathbf{I}}$  is not closed. This result is elaborated in Appendix B.5 and Lemma B.8. The resulting detection algorithm can have false negatives (i.e., it can fail to detect more complex configurations where  $\mathcal{L}_{\mathbf{I}}$  is not closed) but no false positive.
Figure 2: Probability of detectable "bad" support constraints sampled from uniform distribution over 100 samples.
232   
233   
234   
235   
236   
237   
238   
239   
240   
241   
242   
243   
244   
245   
246   
247   
248   
249   
250   
251   
252   
253   
254   
255   
256   
257   
258   
259   
260   
261   
262
<sup>1</sup>The package is developed by Microsoft research and it can be found at https://github.com/Z3Prover/z3
We test this algorithm on a one-hidden layer ReLU network with two  $100 \times 100$  weight matrices. We randomly choose their supports whose cardinality are  $p_1 \cdot 100^2$  and  $p_2 \cdot 100^2$  respectively, with  $(p_1, p_2) \in \{0.1, 0.2, \ldots, 1.0\}^2$ . For each pair  $(p_1, p_2)$ , we sample 100 instances. Using the detection algorithm, we obtain Figure 2. The numbers in Figure 2 indicate the probability that a random support constraint  $(I, J)$  has  $\mathcal{E}_{I,J}$  non-closed (as detected by the algorithm). This figure shows two things: 1) "Bad" architectures such as LU are not rare and one can (randomly) generate plenty of them. 2) At a sparse regime  $(a_1, a_2 \leq 0.2)$ , most of the random supports might lead to training problems without optimal solutions. We remind that the detection algorithm may give some false negatives. Thus, for less sparse regimes, it is possible that our heuristic fails to detect the non-closedness. The algorithm indeed gives a lower bound on the probability of finding non-closed instances. The code for Example 3.1, Question 3.2 and the algorithm in Lemma 3.3 is provided in [17].
3.4 Best approximation property of scalar-valued one-hidden-layer sparse networks
So far, we introduced a necessary condition for the closedness (and thus, by Proposition 3.1, the best approximation property) of sparse ReLU networks, and we provided an example of an architecture  $\mathbf{I}$  whose training problem might not admit any optimal solution. One might wonder if there are architectures  $\mathbf{I}$  that avoid the issue caused by the non-closedness of  $\mathcal{F}_{\mathbf{I}}$ . Indeed, we show that for one-hidden-layer sparse ReLU neural networks with scalar output dimension (i.e.,  $L = 2$ ,  $N_{2} = 1$ ), the existence of optimal solutions is guaranteed, regardless of the sparsity pattern.
Theorem 3.4. Consider scalar-valued, one-hidden-layer ReLU neural networks (i.e.,  $L = 2$ ,  $N_{2} = 1$ ). For any support pairs  $\mathbf{I} = (I_2, I_1)$  and any finite set  $\Omega := \{x_1, \ldots, x_P\}$ ,  $\mathcal{F}_{\mathbf{I}}(\Omega)$  is closed.
The proof of Theorem 3.4 is deferred to Appendix B.3. As a sanity check, observe that when  $L = 2$ ,  $N_{2} = 1$ , the necessary condition in Theorem 3.1 is satisfied. Indeed, since  $N_{2} = 1$ ,  $\mathcal{L}_{\mathbf{I}} \subseteq \mathbb{R}^{1 \times N_{0}}$  can be thought as a subset of  $\mathbb{R}^{N_{0}}$ . Any  $\mathbf{X} \in \mathcal{L}_{\mathbf{I}}$  can be written as a sum:  $\mathbf{X} = \sum_{i \in I_{2}} \mathbf{W}_{2}[i] \mathbf{W}_{1}[i,:]$ , a decomposition of the product  $\mathbf{W}_{2} \mathbf{W}_{1}$ , where  $\mathbf{W}_{2}[i] \in \mathbb{R}$ ,  $\mathbf{W}_{1}[i,:] \in \mathbb{R}^{N_{0}}$ ,  $\text{supp}(\mathbf{W}_{1}[i,:]) \subseteq I_{1}[i,:]$ . Define  $\mathcal{H} := \cup_{i \in I_{2}} I_{1}[i,:] \subseteq [[N_{0}]]$  the union of row supports of the first weight matrix. It is easy to verify that  $\mathcal{L}_{\mathbf{I}}$  is isomorphic to  $\mathbb{R}^{|H|}$ , which is closed. In fact, this argument only works for scalar-valued output,  $N_{2} = 1$ . Thus, there is no conflict between Theorem 3.1 and Theorem 3.4.
In practice, many approaches search for the best support I among a collection of possible supports, for example, the approach of pruning and training [14, 32] or the lottery ticket hypothesis [10]. Our result for fixed support in Theorem 3.4 can be also applied in this case and is stated in Corollary 3.1. In particular, we consider a set of supports such that the support sizes (or sparsity ratios) of the layers are kept below a certain threshold  $K_{i}, i = 1,\dots ,L$ . This constraint on the sparsity level of each layer is widely used in many works on sparse neural networks [14, 32, 10].
Corollary 3.1. Consider scalar-valued, one-hidden-layer ReLU neural networks. For any finite data set  $\mathcal{D} = (x_i, y_i)_{i=1}^P$ , problem (3) under the constraints  $\| \mathbf{W}_i \|_0 \leq K_i$ ,  $i = 1, 2$  has a minimizer.
Proof. Denote  $\mathcal{I}$  the collection of sparsity patterns satisfying  $\| I_i\| _0\leq K_i,i = 1,2$  , so that a set of parameters satisfies the sparsity constraints  $\| \mathbf{W}_i\| _0\leq K_i,i = 1,2$  if and only if the supports of the weight matrices belong to  $\mathcal{I}$  . Therefore, to solve the optimization problem under sparsity constraints  $\| \mathbf{W}_i\| _0\leq K_i,i = 1,2$  , it is sufficient to solve the same problem for every sparsity pattern in  $\mathcal{I}$
For each  $\mathbf{I} \in \mathcal{I}$ , we solve a training problem with architecture  $\mathbf{I}$  on a given finite dataset  $\mathcal{D}$ . Thanks to Theorem 3.4 and Proposition 3.1, the infimum is attained. We take the optimal solution corresponding to  $\mathbf{I}$  that yields the smallest value of the loss function  $\mathcal{L}$ . This is possible because the set  $\mathcal{I}$  has a finite number of elements (the total number of possible sparsity patterns is finite).
4 Analysis of fixed support ReLU networks on continuous domains
We now investigate closedness properties when the domain  $\Omega \subseteq \mathbb{R}^d$  is no longer finite. Denoting  $\mathcal{F}_{\mathbf{I}} = \{\mathcal{R}_{\theta}:\mathbb{R}^{\widetilde{N}_0}\mapsto \mathbb{R}^{N_L}\mid \theta \in \mathcal{N}_{\mathbf{I}}\}$  (with  $N_0 = d$ ) the functions that can be implemented on a given ReLU network architecture  $\mathbf{I}$ , we are interested in  $\mathcal{F}_{\mathbf{I}}(\Omega) = \{f_{|\Omega}:f\in \mathcal{F}_{\mathbf{I}}\}$ , the restriction of elements of  $\mathcal{F}_{\mathbf{I}}$  to  $\Omega$ . This is a natural extension of the set  $\mathcal{F}_{\mathbf{I}}(\Omega)$  studied in the case of finite  $\Omega$ .
263   
264   
265   
266   
267   
268   
269   
270   
271   
272   
273   
274   
275   
276   
277   
278   
279   
280   
281   
282   
283   
284   
285   
286   
287   
288   
289   
290   
291   
292   
293   
294   
295   
296   
297   
298   
299   
300   
301   
302   
303   
304   
305   
306   
307   
308   
309   
310
2Notice that  $\mathcal{D}$  contains both input vectors  $x_{i}$  and targets  $y_{i}$ , unlike  $\Omega$  which only contains the inputs.
Specifically, we investigate the closedness of  $\mathcal{F}_{\mathbf{I}}(\Omega)$  in  $(C^0 (\Omega),\| \cdot \|_\infty)$  (the set of continuous functions on  $\Omega$  equipped with the supremum norm  $\| f\|_{\infty}\coloneqq \sup_{x\in \Omega}\| f(x)\| _2$ ). Contrary to the previous section, we can no longer exploit Proposition 3.1 to deduce that the closedness property and the BAP are equivalent. The results in this section can be seen as a continuation (and also generalization) of the line of research on the topological property of function space of neural networks [26, 12, 16, 15, 23]. In Section 4.1 and Section 4.2, we provide a necessary and a sufficient condition on  $\mathbf{I}$  for the closedness of  $\mathcal{F}_{\mathbf{I}}(\Omega)$  in  $(C^0 (\Omega ,\| \cdot \|_\infty))$  respectively. The condition of the former is valid for any depth, while that of the latter is applicable for one-hidden-layer networks  $(L = 2)$ . These results are established under various assumptions on  $\Omega$  (such as  $\Omega = [-B,B]^d$ , or  $\Omega$  being bounded with non-empty interior) that will be specified in each result.
4.1 A necessary condition for closedness of fixed support ReLU network
Theorem 4.1 states our result on the necessary condition for the closedness. Interestingly, observe that this result (which is proved in Appendix C.1) naturally generalizes Theorem 3.1. Again, closedness of  $\mathcal{L}_{\mathbf{I}}$  in  $\mathbb{R}^{N_L\times N_0}$  is with respect to the usual topology defined by any norm.
Theorem 4.1. Consider  $\Omega \subset \mathbb{R}^d$  a bounded set with non-empty interior, and  $\mathbf{I}$  a sparse architecture with input dimension  $N_0 = d$ . If  $\mathcal{F}_{\mathbf{I}}(\Omega)$  is closed in  $(C^0 (\Omega),\| \cdot \|_{\infty})$  then  $\mathcal{L}_{\mathbf{I}}$  is closed in  $\mathbb{R}^{N_L\times N_0}$ .
Theorem 4.1 applies for any  $\Omega$  which is bounded and has non-empty interior. Thus, it encompasses not only the hypercubes  $[-B,B]^d$ ,  $B > 0$  but also many other domains such as closed or open  $\mathbb{R}^d$  balls. Similar to Theorem 3.1, Theorem 4.1 is interesting in the sense that it allows us to check the non-closedness of the function space  $\mathcal{F}_{\mathbf{I}}$  (a subset of the infinite-dimensional space  $C^0 (\Omega)$ ) by checking that of  $\mathcal{L}_{\mathbf{I}}\subseteq \mathbb{R}^{N_L\times N_0}$  (a finite-dimensional space). The latter can be checked using the algorithm presented in Lemma 3.3. Moreover, the LU architecture presented in Example 3.1 is also an example of  $\mathbf{I}$  whose function space is not closed in  $(C^0 (\Omega),\| \cdot \|_\infty)$ .
4.2 A sufficient condition for closedness of fixed support ReLU network
The following theorem is the main result of this section. It provides a sufficient condition to verify the closedness of  $\mathcal{F}_{\mathbf{I}}(\Omega)$  for  $\Omega = [-B,B]^d$ ,  $B > 0$  with one-hidden-layer sparse ReLU neural networks.
Theorem 4.2. Consider  $\Omega = [-B, B]^d$ ,  $N_0 = d$  and a sparsity pattern  $\mathbf{I} = (I_2, I_1)$  such that:
1. There is no support constraint for the weight matrix of the second layer,  $\mathbf{W}_2$ :  $I_2 = \mathbf{1}_{N_2\times N_1}$
2. For each non-empty set of hidden neurons,  $S \subseteq [[N_1]]$ ,  $\mathcal{L}_{\mathbf{I}_S}$  is closed in  $\mathbb{R}^{N_2 \times N_1}$ , where  $\mathbf{I}_S := (I_2(:, S], I_1[S,:])$  is the support constraint restricted to the sub-network with hidden neurons in  $S$ .
Then the set  $\mathcal{F}_{\mathbf{I}}(\Omega)$  is closed in  $(C^0 (\Omega),\| \cdot \|_{\infty})$
Both conditions in Theorem 4.2 can be verified algorithmically: while the first one is trivial to check, the second one requires us to check the closedness of at most  $2^{N_1}$  sets  $\mathcal{L}_{\mathbf{I}_S}$  (because there are at most  $2^{N_1}$  subsets of  $[[N_1]]$ ), which is still algorithmically possible (although perhaps practically intractable) with the algorithm of Lemma 3.3. Apart from its algorithmic aspect, we present two interesting corollaries of Theorem 4.2. The first one, Corollary 4.1, is about the closedness of the function space of fully connected (i.e., with no sparsity constraint) one-hidden-layer neural networks.
Corollary 4.1 (Closedness of fully connected one-hidden-layer ReLU networks of any output dimension). Given  $\mathbf{I} = (\mathbf{1}_{N_2\times N_1},\mathbf{1}_{N_1\times N_0})$ , the set  $\mathcal{F}_{\mathbf{I}}$  is closed in  $(C^0 ([-B,B]^d),\| \cdot \|_{\infty})$  where  $d = N_0$ .
Proof. The result follows from Theorem 4.2 once we check if its assumptions hold. The first one is trivial. To check the second, observe that for every non-empty set of hidden neurons  $S \subseteq [[N_1]]$ , the set  $\mathcal{L}_{\mathbf{I}_S} \subseteq \mathbb{R}^{N_2 \times N_0}$  is simply the set of matrices of rank at most  $|S|$ , which is closed for any  $S$ .
Corollary 4.2 states the closedness of scalar-valued, one-hidden-layer sparse ReLU NNs. In a way, it can be seen as the analog of Theorem 3.4 for  $\Omega = [-B,B]^d$ .
Corollary 4.2 (Closedness of fixed support one-hidden-layer ReLU networks with scalar output). Given any input dimension  $d = N_0 \geq 1$ , any number of hidden neurons  $N_1 \geq 1$ , scalar output dimension  $N_2 = 1$ , and any prescribed supports  $\mathbf{I} = (I_2, I_1)$ , the set  $\mathcal{F}_{\mathbf{I}}$  is closed in  $(C^0([-B, B]^d), \| \cdot \|_{\infty})$ .
311   
312   
313   
314   
315   
316   
317   
318   
319   
320   
321   
322   
323   
324   
325   
326   
327   
328   
329   
330   
331   
332   
333   
334   
335   
336   
337   
338   
339   
340   
341   
342   
343   
344   
345   
346   
347   
348   
349   
350   
351   
352   
353   
354   
355   
356   
357   
358
Sketch of the proof. If there exists a hidden neuron  $i \in [[N_1]]$  such that  $I_2[i] = 0$  (i.e.,  $i \notin I_2$ :  $i$  is not connected to the only output of the network), we have:  $\mathcal{F}_{\mathbf{I}} = \mathcal{F}_{\mathbf{I}'}$  where  $\mathbf{I}' = \mathbf{I}_S$ ,  $S = [[N_1]] \setminus \{i\}$ . By repeating this process, we can assume without loss of generality that  $I_2[i] = \mathbf{1}_{1 \times N_1}$ . That is the first condition of Theorem 4.2.
Therefore, it is sufficient to verify the second condition of Theorem 4.2. Consider any non-empty set of hidden neurons  $S \subseteq [[N_1]]$ , and define  $\mathcal{H} \coloneqq \cup_{i \in S} I[i,:] \subseteq [[N_0]]$  the union of row supports of  $I_1[S,:]$ . It is easy to verify that  $\mathcal{L}_{\mathbf{I}_S}$  is isomorphic to  $\mathbb{R}^{| \mathcal{H}|}$ , which is closed. The result follows by Theorem 4.2. For a more formal proof, readers can find an inductive one in Appendix C.3.
In fact, both Corollary 4.1 and Corollary 4.2 generalize [26, Theorem 3.8], which proves the closedness of  $\mathcal{F}_{\mathbf{I}}([-B,B]^d)$  when  $I_{2} = \mathbf{1}_{1\times N_{1}},I_{1} = \mathbf{1}_{N_{1}\times N_{0}}$  (classical fully connected one-hiddenlayer ReLU networks with output dimension equal to one).
To conclude, let us consider the analog to Corollary 3.1: we study the function space implementable as a sparse one-hidden-layer network with constraints on the sparsity level of each layer (i.e.,  $\| \mathbf{W}_i\|_0 \leq K_i$ ,  $i = 1,2$ ).
Corollary 4.3. Consider scalar-valued, one-hidden-layer ReLU networks  $(L = 2, N_2 = 1, N_1, N_0)$  with  $\ell^0$  constraints  $\| \mathbf{W}_1\| _0\leq K_1$ ,  $\| \mathbf{W}_2\| _0\leq K_2$  for some constants  $K_{1},K_{2}\in \mathbb{N}$ . The function space  $\mathcal{F}([-B,B]^d)$  associated with this architecture is closed in  $(C^0 ([-B,B]^{N_0}),\| \cdot \|_\infty)$ .
Proof. Denote  $\mathcal{I} \coloneqq \{(I_2, I_1) \mid I_2 \subseteq [[1]] \times [[N_1]], I_1 \subseteq [[N_1]] \times [[N_0]], |I_1| \leq K_1, |I_2| \leq K_2\}$  the set of sparsity patterns respecting the  $\ell^0$  constraints, so that  $\mathcal{F}([-B, B]^d) = \bigcup_{\mathbf{I} \in \mathcal{I}} \mathcal{F}_{\mathbf{I}}([-B, B]^d)$ . Since  $\mathcal{I}$  is finite and  $\forall \mathbf{I} \in \mathcal{I}, \mathcal{F}_{\mathbf{I}}([-B, B]^d)$  is closed (Corollary 4.2), the result is proved.
In this paper, we study the somewhat overlooked question of the existence of an optimal solution to sparse neural network training problems. The study is accomplished by adopting the point of view of topological properties of the function spaces of such networks on two types of domains: a finite domain  $\Omega$ , or (typically) a hypercube. On the one hand, our investigation of the BAP and the closedness of these function spaces reveals the existence of pathological sparsity patterns that fail to have optimal solutions on some instances (cf Theorem 3.1 and Theorem 4.1) and thus possibly cause instabilities in optimization algorithms (see Example 3.1 and Figure 1). On the other hand, we also prove several positive results on the BAP and closedness, notably for sparse one-hidden-layer ReLU neural networks (cf. Theorem 3.4 and Theorem 4.2). These results provide new instances of network architectures where the BAP is proved (cf Theorem 3.4) and substantially generalize existing ones (cf. Theorem 4.2).