Title: Does a sparse ReLU network training problem always admit an optimum ?
Venue: NeurIPS 2023
Accepted: yes
Keywords: Topology

best approximation property

closedness

function space

sparse neural networks
Abstract: Given a training set, a loss function, and a neural network architecture, it is often taken for granted that optimal network parameters exist, and a common practice is to apply available optimization algorithms to search for them. In this work, we show that the existence of an optimal solution is not always guaranteed, especially in the context of sparse ReLU neural networks.
In particular, we first show that optimization problems involving deep networks with certain sparsity patterns do not always have optimal parameters, and that optimization algorithms may then diverge. Via a new topological relation between sparse ReLU neural networks and their linear counterparts, we derive --using existing tools from real algebraic geometry-- an algorithm to verify that a given sparsity pattern suffers from this issue. Then, the existence of a global optimum is proved for every concrete optimization problem involving 
a shallow sparse ReLU neural network of output dimension one. Overall, the analysis is based on the investigation of two topological properties of the space of functions implementable as sparse ReLU neural networks: a best approximation property, and a closedness property, both in the uniform norm. This is studied both for (finite) domains corresponding to practical training on finite training sets, and for more general domains such as the unit cube. This allows us to provide conditions for the guaranteed existence of an optimum given a sparsity pattern. The results apply not only to several sparsity patterns proposed in recent works on network pruning/sparsification, but also to classical dense neural networks, including architectures not covered by existing results.

Abstract

Given a training set, a loss function, and a neural network architecture, it is often taken for granted that optimal network parameters exist, and a common practice is to apply available optimization algorithms to search for them. In this work, we show that the existence of an optimal solution is not always guaranteed, especially in the context of sparse ReLU neural networks. In particular, we first show that optimization problems involving deep networks with certain sparsity patterns do not always have optimal parameters, and that optimization algorithms may then diverge. Via a new topological relation between sparse ReLU neural networks and their linear counterparts, we derive -using existing tools from real algebraic geometry- an algorithm to verify that a given sparsity pattern suffers from this issue. Then, the existence of a global optimum is proved for every concrete optimization problem involving a one-hidden-layer sparse ReLU neural network of output dimension one. Overall, the analysis is based on the investigation of two topological properties of the space of functions implementable as sparse ReLU neural networks: a best approximation property, and a closedness property, both in the uniform norm. This is studied both for (finite) domains corresponding to practical training on finite training sets, and for more general domains such as the unit cube. This allows us to provide conditions for the guaranteed existence of an optimum given a sparsity pattern. The results apply not only to several sparsity patterns proposed in recent works on network pruning/sparsification, but also to classical dense neural networks, including architectures not covered by existing results.

1 Introduction

The optimization phase in deep learning consists in minimizing an objective function w.r.t. the set of parameters θ of a neural network (NN). While it is arguably sufficient for optimization algorithms to find local minima in practice, training is also expected to achieve the infimum in many situations (for example, in overparameterized regimes networks are trained to zero learning error).

In this work, we take a step back and study a rather fundamental question: Given a deep learning architecture possibly with sparsity constraints, does its corresponding optimization problem actually admit an optimal θ ∗ ? The question is important for at least two reasons:

Practical viewpoint: If the problem does not admit an optimal solution, optimized parameters necessarily diverge to infinity to approximate the infimum (which always exists). This phenomenon has been studied thoroughly in previous works in other contexts such as tensor decomposition [7], robust principal component analysis [30], sparse matrix factorization [18] and also deep learning itself [26, 23, 12] . It causes inherent numerical instability for optimization algorithms. Moreover,

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

the answer to this question depends on the architecture of the neural networks (specified by the number of layers, layers width, activation function, and so forth). A response to this question might suggest a guideline for model and architecture selection.

Theoretical viewpoint: the existence of optimal solutions is crucial for the analysis of algorithms and their properties (for example, the properties of convergence, or the characterization of properties of the optimum, related to the notion of implicit bias).

One usual practical (and also theoretical) trick to bypass the question of the existence of optimal solutions is to add a regularization term, which is usually coercive, e.g., the L 2 norm of the parameters. The existence of optimal solutions then follows by a classical argument on the extrema of a continuous function in a compact domain. Nevertheless, there are many settings where minimizing the regularized version might result in a high value of the loss since the algorithm has to make a trade-off between the loss and the regularizer. Such a scenario is discussed in Example 3.1. Therefore, studying the existence of optimal solutions without (explicit) regularization is also a question of interest.

Given a training set { ( x i , y i ) } P i =1 , the problem of the existence of optimal solutions can be studied from the point of view of the set of functions implementable by the considered network architecture on the finite input domain Ω = { x i } P i =1 . This is the case since the loss is usually of the form glyph[lscript] ( f θ ( x i ) , y i ) where f θ is the realization of the neural network with parameters θ . Therefore, the loss involves directly the image of { x i } P i =1 under the function f θ . For theoretical purposes, we also study the function space on the domain Ω = [ -B,B ] d , B > 0 . In particular, we investigate two topological properties of these function spaces, both w.r.t. the infinity norm ‖ · ‖ ∞ : the best approximation property (BAP), i.e., the guaranteed existence of an optimal solution θ ∗ , and the closedness, a necessary property for the BAP. These properties are studied in Section 3 and Section 4, respectively. Most of our analysis is dedicated to the case of regression problems . We do make some links to the case of classification problems in Section 3.

We particularly focus on analyzing the function space associated with (structured) sparse ReLU neural networks , which is motivated by recent advances in machine learning witnessing a compelling empirical success of sparsity based methods in NNs and deep learning techniques, such as pruning [32, 14], sparse designed NN [4, 5], or the lottery ticket hypothesis [10] to name a few. Our approach exploits the notion of networks either with fixed sparsity level or with fixed sparsity pattern (or support) . This allows us to establish results covering both classical NNs (whose weights are not contrained to be sparse) and sparse NNs architectures. Our main contributions are:

To study the BAP (i.e., the existence of optimal solutions) in practical problems (finite Ω ) : we provide a necessary condition and a sufficient one on the architecture (embodied by a sparsity pattern) to guarantee such existence. As a particular consequence of our results, we show that: a) for one-hidden-layer NNs with a fixed sparsity level , the training problem on a finite data set always admits an optimal solution (cf. Theorem 3.4 and Corollary 3.1); b) however, practitioners should be cautious since there also exist fixed sparsity patterns that do not guarantee the existence of optimal solutions (cf. Theorem 3.1 and Example 3.1). In the context of an emerging emphasis on structured sparsity (e.g. for GPU-friendliness), this highlights the importance of choosing adequate sparsity patterns.

To study the closedness of the function space on Ω = [ -B,B ] d . As in the finite case, we provide a necessary condition and a sufficient one for the closedness of the function space of ReLU NNs with a fixed sparsity pattern. In particular, our sufficient condition on one-hidden-layer networks generalizes the closedness results of [26, Theorem 3.8] on 'dense' one-hidden-layer ReLU NNs to the case of sparse ones, either with fixed sparsity pattern (cf. Theorem 4.2, Corollary 4.1 and Corollary 4.2) or fixed sparsity level (Corollary 4.3). Moreover, our necessary condition (Theorem 4.1), which is also applicable to deep architectures, exhibits sparsity structures failing the closedness property.

Table 1 and Table 2 summarize our results and their positioning with respect to existing ones. Somewhat surprisingly, the necessary conditions in both domains ( Ω finite and Ω = [ -B,B ] d ) are identical. Our necessary/sufficient conditions also suggest a relation between sparse ReLU neural networks and their linear counterparts.

The rest of this paper is organized as follows: Section 2 discusses related works and introduces notations; the two technical sections, Section 3 and Section 4, presents the results for the case Ω finite set and Ω = [ -B,B ] d respectively.

91

92

2 Related works

The fact that optimal solutions may not exist in tensor decomposition problems is well-documented

[7]. The cause of this phenomenon (also referred to as ill-posedness [7]) is the non-closedness of 93

the set of tensors of order at least three and of rank at least two. Similar phenomena were shown to 94

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

[12, 26, 23], showing that the set of functions implemented as conventional multilayer perceptrons with various activation functions such as Inverse Square Root Linear Unit (ISRLU), Inverse Square Root Unit (ISRU), parametric ReLU (pReLU), Exponential Linear Unit (ELU) [26, Table 1] is not a closed subset of classical function spaces (e.g., the Lebesgue spaces L p , the set of continuous functions C 0 equipped with the sup-norm, or Sobolev spaces W k,p ). In a more practical setting, [20] hand-crafts a dataset of six points which makes the training problem of a dense one-hidden-layer neural network not admit any solution. Positive results are proved in [26, 16, 15], which establish both the closedness and/or the BAP. The BAP implies closedness [12, Proposition 3.1][26, Section 3] (but the converse is not true, see Appendix D) hence the BAP can be more difficult to prove than closedness. So far, the only architecture proved to admit the best approximation property (and thus, also closedness) is one-hidden-layer neural networks with heavyside activation function and scalar-valued output (i.e., output dimension equal to one ) [15] in L p (Ω) , ∀ p ∈ [1 , ∞ ] . If one allows additional assumptions such as the target function f being continuous, then BAP is also established for one-hidden layer and residual one-hidden-layer NNs with ReLU activation function [9, 8]. In all other settings, to the best of our knowledge, the only property proved in the literature is closedness, but the BAP remains elusive. We compare our results with existing works in Tables 1 and 2.

In machine learning, there is an ongoing endeavour to explore sparse deep neural networks, as a prominent approach to reduce memory and computation overheads inherent in deep learning. One of its most well-known methods is Iterative Magnitude Pruning (IMP), which iteratively trains and prunes connections/neurons to achieve a certain level of sparsity. This method is employed in various works [14, 32], and is related to the so-called Lottery Ticket Hypothesis (LTH) [10]. The main issue of IMP is its running time: one typically needs to perform many steps of pruning and retraining to achieve a good trade-off between sparsity and performance. To address this issue, many works attempt to identify the sparsity patterns of the network before training. Once they are found, it is sufficient to train the sparse neural networks once. These pre-trained sparsity patterns can be found through algorithms [29, 31, 19] or leveraging the sparse structure of well-known fast linear operators such as the Discrete Fourier Transform [5, 4, 21, 6, 3]. Regardless of the approaches, these methods are bound to train a neural network with fixed sparsity pattern at some points. This is a particular motivation for our work and our study on the best approximation property of sparse ReLU neural networks with fixed sparsity pattern.

glyph[negationslash]

Notations In this work, J n K := { 1 , . . . , n } . For a matrix A ∈ R m × n , A [ i, j ] denotes the coefficient at the index ( i, j ) ; for subsets S r ⊆ J m K , S c ⊆ J n K , A [ S r , :] (resp. A [: , S c ] ) is a matrix of the same size as A and agrees with A on rows in S r (resp. columns in S c ) of A while its remaining coefficients are zero. The operator supp ( A ) := { ( glyph[lscript],k ) | A [ glyph[lscript],k ] = 0 } returns the support of the matrix A . We denote 1 m × n (resp. 0 m × n ) an all-one (resp. all-zero) matrix of size m × n .

An architecture with fixed sparsity pattern is specified via I = ( I L , . . . , I 1 ) , a collection of binary masks I i ∈ { 0 , 1 } N i × N i -1 , 1 ≤ i ≤ L , where the tuple ( N L , . . . , N 0 ) denotes the dimensions of the input layer N 0 = d , hidden layers ( N L -1 , . . . , N 1 ) and output layer ( N L ), respectively. The binary mask I i encodes the support constraints on the i th weight matrix W i , i.e., I i [ glyph[lscript],k ] = 0 implies W i [ glyph[lscript],k ] = 0 . It is also convenient to think of I i as the set { ( glyph[lscript],k ) | I i [ glyph[lscript],k ] = 1 } , a subset of J N i K × J N i -1 K . We will use these two interpretations (binary mask and subset) interchangeably and the meaning should be clear from context. We will even abuse notations by denoting I l ⊆ 1 N l × N l -1 . Because the support constraint I can be thought as a binary matrix, the notation I [ S r , :] (resp. I [: , S c ] ) represents the support constraint of I ∩ S r × J n K (resp. I ∩ J n K × S c ).

The space of parameters on the sparse architecture I is denoted N I , and for each θ ∈ N I , R θ : R N 0 ↦→ R N L is the function implemented by the ReLU network with parameter θ :

where σ ( x ) = max(0 , x ) is the ReLU activation.

Finally, for a given architecture I , we define

the set of matrices factorized into L factors respecting the support constraints I i , i ∈ J L K . In fact, L I is the set of linear operators implementable as linear neural networks (i.e., with σ = id instead of the ReLU in (1), and no biases) with parameters θ ∈ N I .

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

3 Analysis of fixed support ReLU neural networks for finite Ω

The setting of a finite set Ω = { x i } P i =1 is common in many practical machine learning tasks: models such as (sparse) neural networks are trained on often large (but finite) annotated dataset D = { ( x i , y i ) } P i =1 . The optimization/training problem usually takes the form:

where glyph[lscript] is a loss function measuring the similarity between R θ ( x i ) and y i . A natural question that we would like to address for this task is:

Question 3.1. Under which conditions on I , the prescribed sparsity pattern for θ , does the training problem of sparse neural networks admit an optimal solution for any finite data set D ?

We investigate this question both for parameters θ constrained to satisfy a fixed sparsity pattern I , and in the case of a fixed sparsity level, see e.g. Corollary 4.3.

After showing in Section 3.1 that the answer to Question 3.1 is intimately connected with the closedness of the function space of neural networks with architecture I , we establish in Section 3.2 that this closedness implies the closedness of the matrix set L I (a property that can be checked using algorithms from real algebraic geometry, see Section 3.3). We also provide concrete examples of support patterns I where closedness provably fails, and neural network training can diverge. Section 3.4 presents sufficient conditions for closedness that enable us to show that an optimal solution always exists on scalar-valued one-hidden-layer networks under a constraint on the sparsity level of each layer.

3.1 Equivalence between closedness and best approximation property

To answer Question 3.1, it is convenient to view Ω as the matrix [ x 1 , . . . , x P ] ∈ R d × P and to consider the function space implemented by neural networks with the given architecture I on the input domain Ω in dimension d = N 0 , with output dimension N L , defined as the set

[

where the matrix

R

θ

(Ω) :=

R

θ

(

x

1

)

, . . . ,

R

θ

(

x

P

)

]

∈

R

N

L

×

P

is the image under

R

θ

of

Ω

.

We study the closedness of F I (Ω) under the usual topology induced by any norm ‖ · ‖ of R N L × P . This property is interesting because if F I (Ω) is closed for any Ω = { x i } P i =1 , then an optimal solution is guaranteed to exist for any D under classical assumptions of glyph[lscript] ( · , · ) . The following result is not difficult to prove, we nevertheless provide a proof in Appendix B.1 for completeness.

Proposition 3.1. Assume that, for any fixed y ∈ R N L , glyph[lscript] ( · , y ) : R N L ↦→ R is continuous, coercive and that y = arg min y ′ glyph[lscript] ( y ′ , y ) . For any sparsity pattern I with input dimension N 0 = d the following properties are equivalent:

irrespective of the training set, problem (3) under the constraint θ ∈ N I has an optimal solution;

for every P and every Ω ∈ R d × P , the function space F I (Ω) is a closed subspace of R N L × P .

The assumption on glyph[lscript] is natural and realistic in regression problems: any loss function based on any norm on R d (e.g. glyph[lscript] ( y ′ , y ) = ‖ y ′ -y ‖ ), such as the quadratic loss, satisfies this assumption. In the classification case, using the soft-max after the last layer together with the cross-entropy loss function indeed leads to an optimization problem with no optimum (regardless of the architecture) when given a single training pair. This is due to the fact that changing either the bias or the scales of the last layer can lead the output of the soft-max arbitrarily close to an ideal Dirac mass. It is an interesting challenge to identify whether sufficiently many and diverse training samples (as in concrete learning scenarios) make the problem better posed, and amenable to a relevant closedness analysis.

In light of Proposition 3.1 we investigate next the closedness of F I (Ω) for finite Ω .

3.2 A necessary closedness condition for fixed support ReLU networks

Our next result reveals connections between the closedness of F I (Ω) for finite Ω and the closedness of L I , the space of sparse matrix products with sparsity pattern I .

Theorem 3.1. If F I (Ω) is closed for every finite Ω then L I is closed. 197

198

199

200

Theorem 3.1 is a direct consequence of (and in fact logically equivalent to) the following lemma: Lemma 3.2. If L I is not closed then there exists a set Ω ⊂ R d , d = N 0 , of cardinality at most P := (3 N 0 4 ∑ L -1 i =1 N i +1) N 0 such that F I (Ω) is not closed.

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

Sketch of the proof. Since L I is not closed, there exists A ∈ L I \ L I ( L is the closure of the set L ). Considering f ( x ) := A x , we construct a set Ω = { x i } P i =1 such that [ f ( x 1 ) , . . . , f ( x P )] ∈ F I (Ω) \ F I (Ω) . Therefore, F I (Ω) is not closed.

The proof is in Appendix B.2. Besides showing a topological connection between F I (NNs with ReLU activation) and L I (linear NNs), Theorem 3.1 leads to a simple example where F I is not closed. Example 3.1 ( LU architecture) . Consider I = ( I 2 , I 1 ) ∈ { 0 , 1 } d × d ×{ 0 , 1 } d × d where I 1 = { ( i, j ) | 1 ≤ i ≤ j ≤ d } and I 2 = { ( i, j ) | 1 ≤ j ≤ i ≤ d } . Any pair of matrices X 2 , X 1 ∈ R d × d such that supp ( X i ) ⊆ I i , i = 1 , 2 are respectively lower and upper triangular matrices. Therefore, L I is the set of matrices that admit an exact lower - upper (LU) factorization/decomposition. That explains its name: LU architecture. This set is well known to a) contain an open and dense subset of R d × d ; b) be strictly contained in R d × d [13, Theorem 3 . 2 . 1 ] [25, Theorem 1 ]. Therefore, L I is not closed and by the contraposition of Theorem 3.1 we conclude that there exists a finite set Ω such that F I (Ω) is not closed.

(e

b)

c)

入=0

入=0

入=0

1.0

入=0.0001

入=0.0005

1.0

入=0.0001

入=0.0005

matrices

30

入=0.0001

入=0.0005

入=0.001

入=0.001

A=0.001

1loss

SSOI

0.8

25

0.8

Emperical l

Validation

Norm of weight r

0.6

0.6

20

15

0.4

0.4

10

0.2

0.2

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

epochs

epochs

epochs

Let us illustrate the impact of the non-closedness in Example 3.1 via the behavior during the training of a fixed support one-hidden-layer neural network with the LU support constraint I . This network is trained to learn the linear function f ( x ) := A x where A ∈ R d × d is an anti-diagonal identity matrix. Using the necessary and sufficient condition of LU decomposition existence [25, Theorem 1 ], we have that A ∈ L I \L I as in the sketch proof of Lemma 3.2. Given network parameters θ and a training set, approximation quality can be measured by the relative loss: 1 P ( ∑ P i =1 ‖R θ ( x i ) -y i ‖ 2 2 / ‖ y i ‖ 2 2 ) .

Figure 1 illustrates the behavior of the relative errors of the training set, validation set and the sum of weight matrices norm along epochs, using Stochastic Gradient Descent (SGD) with batch size 3000 , learning rate 0 . 1 , momentum 0 . 9 and four different weight decays (the hyperparameter controlling the L 2 regularizer) λ ∈ ×{ 0 , 10 -4 , 5 × 10 -4 , 10 -3 } . The case λ = 0 corresponds to the unregularized case. Our training and testing sets contain each P = 10 5 samples generated independently as x i ∼ U ([ -1 , 1] d ) ( d = 100 ) and y i := A x i .

Example 3.1 and Figure 1 also lead to two interesting remarks: while the L 2 regularizer (weight decay) does prevent the parameter divergence phenomenon, the empirical loss is improved when using the non-regularized version. This is the situation where adding a regularization term might be detrimental, as stated earlier. More interestingly, the size of the dataset is 10 5 , which is much smaller than the theoretical P in Lemma 3.2. It is thus interesting to see if we can reduce the theoretical value of P , which is currently exponential w.r.t. to the input dimension.

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

3.3 The closedness of L I is algorithmically decidable

Theorem 3.1 leads to a natural question: given I , how to check the closedness of L I , a subset of R N L × N 0 . To the best of our knowledge, there is not any study on the closedness of L I in the literature. It is, thus, not known whether deciding on the closedness of L I for a given I is polynomially tractable. In this work, we show it is at least decidable with a doubly-exponential algorithm. This algorithm is an application of quantifier elimination , an algorithm from real algebraic geometry [2].

Lemma 3.3. Given I = ( I L , . . . , I 1 ) , the closedness of L I is decidable with an algorithm of complexity O ((4 L ) C k -1 ) where k = N L N 0 +1+2 ∑ L i =1 | L i | and C is a universal constant.

We prove Lemma 3.3 in Appendix B.4. Since the knowledge of I is usually available (either fixed before training [19, 31, 4, 21, 3] or discovered by a procedure before re-training [10, 14, 32]), the algorithm in Lemma 3.3 is able to verify whether the training problem might not admit an optimum. While such a doubly exponential algorithm in Lemma 3.3 is seemingly impractical in practice, small toy examples (for example, Example 3.1 with d = 2 ) can be verified using Z3Prover 1 , a software implementing exactly the algorithm in Lemma 3.3. However, Z3Prover is already unable to terminate when run on the LU architecture of Example 3.1 with d = 3 . This calls for more efficient algorithms to determine the closedness of L I given I . The same algorithmic question can be also asked for F I . We leave these problems (in this general form) as open questions.

In fact, if such a polynomial algorithm (to decide the closedness of L I ) exists, it can be used to answer the following interesting question:

Question 3.2. If the supports of the weight matrices are randomly sampled from a distribution, what is the probability that the corresponding training problem potentially admits no optimal solutions?

While simple, this setting does happen in practice since random supports/binary masks are considered a strong and common baseline for sparse DNNs training [22]. Thanks to Theorem 3.1, if L I is not closed then the support is 'bad'. Thus, to have an estimation of a lower bound on the probability of 'bad' supports, we could sample the supports from the given distribution and use the polynomial algorithm in question to decide if L I is closed. Unfortunately, the algorithm in Lemma 3.3 has doubly exponential complexity, thus hindering its practical use. However, for one-hidden-layer NNs, there is a polynomial algorithm to detect non-closedness: intuitively, if the support constraint is 'locally similar' to the LU structure, then L I is not closed. This result is elaborated in Appendix B.5 and Lemma B.8. The resulting detection algorithm can have false negatives (i.e., it can fail to detect more complex configurations where L I is not closed) but no false positive.

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

- 1.0

0

1

1

1

1

0.99

0.67

0.17

0

0.01

0

2

1

1

0.75

0.01

0

0

0

0

0

0

-0.8

1

0.85

0

0

0

0

0

0

0

0

1

0.03

0

0

0

0

0

0

0

0

0.6

5.

1

0

0

0

0

0

0

0

0

0

0

0.71

0

0

0

0

0

0

0

0

0

0.4

一

0.14

0

0

0

0

0

0

0

0

0

8.

一

0.01

0

0

0

0

0

0

0

0

0

0.2

°

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.0

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

We test this algorithm on a one-hidden layer ReLU network with two 100 × 100 weight matrices. We randomly choose their supports whose cardinality are p 1 · 100 2 and p 2 · 100 2 respectively, with ( p 1 , p 2 ) ∈ { 0 . 1 , 0 . 2 , . . . , 1 . 0 } 2 . For each pair ( p 1 , p 2 ) , we sample 100 instances. Using the detection algorithm, we obtain Figure 2. The numbers in Figure 2 indicate the probability that a random support constraint ( I, J ) has E I,J non-closed (as detected by the algorithm). This figure shows two things: 1) 'Bad' architectures such as LU are not rare and one can (randomly) generate plenty of them. 2) At a sparse regime ( a 1 , a 2 ≤ 0 . 2 ), most of the random supports might lead to training problems without optimal solutions. We remind that the detection algorithm may give some false negatives. Thus, for less sparse regimes, it is possible that our heuristic fails to detect the non-closedness. The algorithm indeed gives a lower bound on the probability of finding non-closed instances. The code for Example 3.1, Question 3.2 and the algorithm in Lemma 3.3 is provided in [17].

3.4 Best approximation property of scalar-valued one-hidden-layer sparse networks

So far, we introduced a necessary condition for the closedness (and thus, by Proposition 3.1, the best approximation property) of sparse ReLU networks, and we provided an example of an architecture I whose training problem might not admit any optimal solution. One might wonder if there are architectures I that avoid the issue caused by the non-closedness of F I . Indeed, we show that for one-hidden-layer sparse ReLU neural networks with scalar output dimension (i.e., L = 2 , N 2 = 1 ), the existence of optimal solutions is guaranteed, regardless of the sparsity pattern .

Theorem 3.4. Consider scalar-valued, one-hidden-layer ReLU neural networks (i.e., L = 2 , N 2 = 1 ). For any support pairs I = ( I 2 , I 1 ) and any finite set Ω := { x 1 , . . . , x P } , F I (Ω) is closed.

The proof of Theorem 3.4 is deferred to Appendix B.3. As a sanity check, observe that when L = 2 , N 2 = 1 , the necessary condition in Theorem 3.1 is satisfied. Indeed, since N 2 = 1 , L I ⊆ R 1 × N 0 can be thought as a subset of R N 0 . Any X ∈ L I can be written as a sum: X = ∑ i ∈ I 2 W 2 [ i ] W 1 [ i, :] , a decomposition of the product W 2 W 1 , where W 2 [ i ] ∈ R , W 1 [ i, :] ∈ R N 0 , supp ( W 1 [ i, :]) ⊆ I 1 [ i, :] . Define H := ∪ i ∈ I 2 I 1 [ i, :] ⊆ J N 0 K the union of row supports of the first weight matrix. It is easy to verify that L I is isomorphic to R |H| , which is closed. In fact, this argument only works for scalar-valued output, N 2 = 1 . Thus, there is no conflict between Theorem 3.1 and Theorem 3.4.

In practice, many approaches search for the best support I among a collection of possible supports , for example, the approach of pruning and training [14, 32] or the lottery ticket hypothesis [10]. Our result for fixed support in Theorem 3.4 can be also applied in this case and is stated in Corollary 3.1. In particular, we consider a set of supports such that the support sizes (or sparsity ratios) of the layers are kept below a certain threshold K i , i = 1 , . . . , L . This constraint on the sparsity level of each layer is widely used in many works on sparse neural networks [14, 32, 10].

Corollary 3.1. Consider scalar-valued, one-hidden-layer ReLU neural networks. For any finite data set 2 D = ( x i , y i ) P i =1 , problem (3) under the constraints ‖ W i ‖ 0 ≤ K i , i = 1 , 2 has a minimizer.

Proof. Denote I the collection of sparsity patterns satisfying ‖ I i ‖ 0 ≤ K i , i = 1 , 2 , so that a set of parameters satisfies the sparsity constraints ‖ W i ‖ 0 ≤ K i , i = 1 , 2 if and only if the supports of the weight matrices belong to I . Therefore, to solve the optimization problem under sparsity constraints ‖ W i ‖ 0 ≤ K i , i = 1 , 2 , it is sufficient to solve the same problem for every sparsity pattern in I .

For each I ∈ I , we solve a training problem with architecture I on a given finite dataset D . Thanks to Theorem 3.4 and Proposition 3.1, the infimum is attained. We take the optimal solution corresponding to I that yields the smallest value of the loss function L . This is possible because the set I has a finite number of elements (the total number of possible sparsity patterns is finite).

4 Analysis of fixed support ReLU networks on continuous domains

We now investigate closedness properties when the domain Ω ⊆ R d is no longer finite. Denoting F I = {R θ : R N 0 ↦→ R N L | θ ∈ N I } (with N 0 = d ) the functions that can be implemented on a given ReLU network architecture I , we are interested in F I (Ω) = { f | Ω : f ∈ F I } , the restriction of elements of F I to Ω . This is a natural extension of the set F I (Ω) studied in the case of finite Ω .

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

Specifically, we investigate the closedness of F I (Ω) in ( C 0 (Ω) , ‖ · ‖ ∞ ) (the set of continuous functions on Ω equipped with the supremum norm ‖ f ‖ ∞ := sup x ∈ Ω ‖ f ( x ) ‖ 2 ). Contrary to the previous section, we can no longer exploit Proposition 3.1 to deduce that the closedness property and the BAP are equivalent. The results in this section can be seen as a continuation (and also generalization) of the line of research on the topological property of function space of neural networks [26, 12, 16, 15, 23]. In Section 4.1 and Section 4.2, we provide a necessary and a sufficient condition on I for the closedness of F I (Ω) in ( C 0 (Ω , ‖ · ‖ ∞ ) respectively. The condition of the former is valid for any depth, while that of the latter is applicable for one-hidden-layer networks ( L = 2 ). These results are established under various assumptions on Ω (such as Ω = [ -B,B ] d , or Ω being bounded with non-empty interior) that will be specified in each result.

4.1 A necessary condition for closedness of fixed support ReLU network

Theorem 4.1 states our result on the necessary condition for the closedness. Interestingly, observe that this result (which is proved in Appendix C.1) naturally generalizes Theorem 3.1. Again, closedness of L I in R N L × N 0 is with respect to the usual topology defined by any norm.

Theorem 4.1. Consider Ω ⊂ R d a bounded set with non-empty interior, and I a sparse architecture with input dimension N 0 = d . If F I (Ω) is closed in ( C 0 (Ω) , ‖ · ‖ ∞ ) then L I is closed in R N L × N 0 .

Theorem 4.1 applies for any Ω which is bounded and has non-empty interior. Thus, it encompasses not only the hypercubes [ -B,B ] d , B > 0 but also many other domains such as closed or open R d balls. Similar to Theorem 3.1, Theorem 4.1 is interesting in the sense that it allows us to check the non-closedness of the function space F I (a subset of the infinite-dimensional space C 0 (Ω) ) by checking that of L I ⊆ R N L × N 0 (a finite-dimensional space). The latter can be checked using the algorithm presented in Lemma 3.3. Moreover, the LU architecture presented in Example 3.1 is also an example of I whose function space is not closed in ( C 0 (Ω) , ‖ · ‖ ∞ ) .

4.2 A sufficient condition for closedness of fixed support ReLU network

The following theorem is the main result of this section. It provides a sufficient condition to verify the closedness of F I (Ω) for Ω = [ -B,B ] d , B > 0 with one-hidden-layer sparse ReLU neural networks.

Theorem 4.2. Consider Ω = [ -B,B ] d , N 0 = d and a sparsity pattern I = ( I 2 , I 1 ) such that:

There is no support constraint for the weight matrix of the second layer, W 2 : I 2 = 1 N 2 × N 1 ;

For each non-empty set of hidden neurons, S ⊆ J N 1 K , L I S is closed in R N 2 × N 1 , where I S := ( I 2 [: , S ] , I 1 [ S, :]) is the support constraint restricted to the sub-network with hidden neurons in S .

Then the set F I (Ω) is closed in ( C 0 (Ω) , ‖ · ‖ ∞ ) .

Both conditions in Theorem 4.2 can be verified algorithmically: while the first one is trivial to check, the second one requires us to check the closedness of at most 2 N 1 sets L I S (because there are at most 2 N 1 subsets of J N 1 K ), which is still algorithmically possible (although perhaps practically intractable) with the algorithm of Lemma 3.3. Apart from its algorithmic aspect, we present two interesting corollaries of Theorem 4.2. The first one, Corollary 4.1, is about the closedness of the function space of fully connected (i.e., with no sparsity constraint) one-hidden-layer neural networks.

Corollary 4.1 (Closedness of fully connected one-hidden-layer ReLU networks of any output dimension ) . Given I = ( 1 N 2 × N 1 , 1 N 1 × N 0 ) , the set F I is closed in ( C 0 ([ -B,B ] d ) , ‖ · ‖ ∞ ) where d = N 0 .

Proof. The result follows from Theorem 4.2 once we check if its assumptions hold. The first one is trivial. To check the second, observe that for every non-empty set of hidden neurons S ⊆ J N 1 K , the set L I S ⊆ R N 2 × N 0 is simply the set of matrices of rank at most | S | , which is closed for any S .

Corollary 4.2 states the closedness of scalar-valued, one-hidden-layer sparse ReLU NNs. In a way, it can be seen as the analog of Theorem 3.4 for Ω = [ -B,B ] d .

Corollary 4.2 (Closedness of fixed support one-hidden-layer ReLU networks with scalar output) . Given any input dimension d = N 0 ≥ 1 , any number of hidden neurons N 1 ≥ 1 , scalar output dimension N 2 = 1 , and any prescribed supports I = ( I 2 , I 1 ) , the set F I is closed in ( C 0 ([ -B,B ] d ) , ‖·‖ ∞ ) .

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

Sketch of the proof. If there exists a hidden neuron i ∈ J N 1 K such that I 2 [ i ] = 0 (i.e., i / ∈ I 2 : i is not connected to the only output of the network), we have: F I = F I ′ where I ′ = I S , S = J N 1 K \ { i } . By repeating this process, we can assume without loss of generality that I 2 [ i ] = 1 1 × N 1 . That is the first condition of Theorem 4.2.

Therefore, it is sufficient to verify the second condition of Theorem 4.2. Consider any non-empty set of hidden neurons S ⊆ J N 1 K , and define H := ∪ i ∈ S I [ i, :] ⊆ J N 0 K the union of row supports of I 1 [ S, :] . It is easy to verify that L I S is isomorphic to R |H| , which is closed. The result follows by Theorem 4.2. For a more formal proof, readers can find an inductive one in Appendix C.3.

In fact, both Corollary 4.1 and Corollary 4.2 generalize [26, Theorem 3.8], which proves the closedness of F I ([ -B,B ] d ) when I 2 = 1 1 × N 1 , I 1 = 1 N 1 × N 0 (classical fully connected one-hiddenlayer ReLU networks with output dimension equal to one).

To conclude, let us consider the analog to Corollary 3.1: we study the function space implementable as a sparse one-hidden-layer network with constraints on the sparsity level of each layer (i.e., ‖ W i ‖ 0 ≤ K i , i = 1 , 2 .

Corollary 4.3. Consider scalar-valued, one-hidden-layer ReLU networks ( L = 2 , N 2 = 1 , N 1 , N 0 ) with glyph[lscript] 0 constraints ‖ W 1 ‖ 0 ≤ K 1 , ‖ W 2 ‖ 0 ≤ K 2 for some constants K 1 , K 2 ∈ N . The function space F ([ -B,B ] d ) associated with this architecture is closed in ( C 0 ([ -B,B ] N 0 ) , ‖ · ‖ ∞ ) .

Proof. Denote I := { ( I 2 , I 1 ) | I 2 ⊆ J 1 K × J N 1 K , I 1 ⊆ J N 1 K × J N 0 K , | I 1 | ≤ K 1 , | I 2 | ≤ K 2 } the set of sparsity patterns respecting the glyph[lscript] 0 constraints, so that F ([ -B,B ] d ) = ⋃ I ∈I F I ([ -B,B ] d ) . Since I is finite and ∀ I ∈ I , F I ([ -B,B ] d ) is closed (Corollary 4.2), the result is proved.

5 Conclusion

In this paper, we study the somewhat overlooked question of the existence of an optimal solution to sparse neural network training problems. The study is accomplished by adopting the point of view of topological properties of the function spaces of such networks on two types of domains: a finite domain Ω , or (typically) a hypercube. On the one hand, our investigation of the BAP and the closedness of these function spaces reveals the existence of pathological sparsity patterns that fail to have optimal solutions on some instances (cf Theorem 3.1 and Theorem 4.1) and thus possibly cause instabilities in optimization algorithms (see Example 3.1 and Figure 1). On the other hand, we also prove several positive results on the BAP and closedness, notably for sparse one-hidden-layer ReLU neural networks (cf. Theorem 3.4 and Theorem 4.2). These results provide new instances of network architectures where the BAP is proved (cf Theorem 3.4) and substantially generalize existing ones (cf. Theorem 4.2).

In the future, a particular theoretical challenge is to propose necessary and sufficient conditions for the BAP and closedness of F I (Ω) , if possible covering in a single framework both types of domains Ω considered here. The fact that the conditions established on these two types of domains are very similar (cf. the similarity between Theorem 3.1 and Theorem 4.1, as well as between Theorem 3.4 and Corollary 4.2) is encouraging. Another interesting algorithmic challenge is to substantially reduce the complexity of the algorithm to decide the closedness of L I in Lemma 3.3, which is currently doubly exponential. It calls for a more efficient algorithm to make this check more practical. Achieving a practically tractable algorithm would for instance allow to check if a support selected e.g. by IMP is pathological or not. This would certainly consolidate the algorithmic robustness and theoretical foundations of pruning techniques to sparsity deep neural networks. From a more theoretical perspective, the existence of an optimum solution in the context of classical linear inverse problems has been widely used to analyze the desirable properties of certain cost functions, e.g. glyph[lscript] 1 minimization for sparse recovery. Knowing that an optimal solution exists for a given sparse neural network training problem is thus likely to open the door to further fruitful insights.

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

References

Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee. Understanding deep neural networks with rectified linear units. In International Conference on Learning Representations , 2018.

Saugata Basu, Richard Pollack, and Marie-Françoise Roy. Algorithms in Real Algebraic Geometry (Algorithms and Computation in Mathematics) . Springer-Verlag, Berlin, Heidelberg, 2006.

Beidi Chen, Tri Dao, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, and Christopher Ré. Pixelated butterfly: Simple and efficient sparse training for neural network models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

Tri Dao, Beidi Chen, Nimit Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher Ré. Monarch: Expressive structured matrices for efficient and accurate training. In 39th International Conference on Machine Learning , 2022.

Tri Dao, Albert Gu, Matthew Eichhorn, Atri Rudra, and Christopher Ré. Learning fast algorithms for linear transforms using butterfly factorizations. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 1517-1527. PMLR, 09-15 Jun 2019.

Tri Dao, Nimit Sharad Sohoni, Albert Gu, Matthew Eichhorn, Amit Blonder, Megan Leszczynski, Atri Rudra, and Christopher Ré. Kaleidoscope: An efficient, learnable representation for all structured linear maps. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020.

Vin de Silva and Lek-Heng Lim. Tensor rank and the ill-posedness of the best low-rank approximation problem. SIAM Journal on Matrix Analysis and Applications , 30:1084-1127, 08 2006.

Steffen Dereich, Arnulf Jentzen, and Sebastian Kassing. On the existence of minimizers in shallow residual relu neural network optimization landscapes, 2023.

Steffen Dereich and Sebastian Kassing. On the existence of optimal shallow feedforward networks with relu activation, 2023.

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representation , 2019.

Nicolas Gillis and François Glineur. Low-rank matrix approximation with weights or missing data is np-hard. SIAM Journal on Matrix Analysis and Applications , 32, 12 2010.

Federico Girosi and Tomaso Poggio. Networks and the best approximation property. Biological Cybernetics , 63, 08 2002.

Gene H. Golub and Charles F. Van Loan. Matrix Computations . The Johns Hopkins University Press, third edition, 1996.

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015.

P. Kainen, V. K˚ urková, and A. Vogt. Best approximation by heaviside perceptron networks. Neural Networks , 13(7):695-697, 2000.

Vˇ era K˚ urková. Approximation of functions by perceptron networks with bounded number of hidden units. Neural Networks , 8(5):745-750, 1995.

Quoc-Tung Le, Rémi Gribonval, and Elisa Riccietti. Code for reproducible research: Does a sparse ReLU network training problemalways admit an optimum? Code repository available at https://hal.science/hal-04233925 , October 2023.

Quoc-Tung Le, Elisa Riccietti, and Rémi Gribonval. Spurious Valleys, NP-hardness, and Tractability of Sparse Matrix Factorization With Fixed Support. SIAM Journal on Matrix Analysis and Applications , 2022.

Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. SNIP: single-shot network pruning based on connection sensitivity. ICLR , 2019.

Lek-Heng Lim, Mateusz Michalek, and Yang Qi. Best k -layer neural network approximations. Constructive Approximation , June 2021.

Rui Lin, Jie Ran, King Hung Chiu, Graziano Chesi, and Ngai Wong. Deformable butterfly: A highly structured and sparse linear transform. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 16145-16157. Curran Associates, Inc., 2021.

Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. arXiv preprint arXiv:2202.02643 , 2022.

Scott Mahan, Emily J. King, and Alex Cloninger. Nonclosedness of sets of neural networks in sobolev spaces. Neural Networks , 137:85-96, may 2021.

Guido Montúfar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 , NIPS'14, page 2924-2932, Cambridge, MA, USA, 2014. MIT Press.

Pavel Okunev and Charles R. Johnson. Necessary and sufficient conditions for existence of the lu factorization of an arbitrary matrix, 2005.

Philipp Petersen, Mones Raslan, and Felix Voigtlaender. Topological properties of the set of functions generated by neural networks of fixed size. Found. Comput. Math. , 21(2):375-444, apr 2021.

E. Quemener and M. Corvellec. SIDUS-the Solution for Extreme Deduplication of an Operating System. Linux Journal , 2013.

Pierre Stock and Rémi Gribonval. An Embedding of ReLU Networks and an Analysis of their Identifiability. Constructive Approximation , 2022.

Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc.

Jared Tanner, Andrew Thompson, and Simon Vary. Matrix rigidity and the ill-posedness of robust pca and matrix completion. SIAM Journal on Mathematics of Data Science , 1(3):537-554, 2019.

Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by 496 preserving gradient flow. In International Conference on Learning Representations , 2020. 497

Michael Zhu and Suyog Gupta. To prune, or not to prune: Exploring the efficacy of pruning for 498 model compression. In 6th International Conference on Learning Representations, ICLR 2018, 499 Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track Proceedings . OpenReview.net, 500 2018. 501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

A Additional notations

In this work, matrices are written in bold uppercase letters. Vectors are written in bold lowercase letters only if they indicate network parameters (such as bias). For a matrix A ∈ R m × n , we use A [ i, :] ∈ R 1 × n (resp. A [: , i ] ∈ R m × 1 ) to denote the row (resp. column) vector corresponding the i th row (resp. column) of A . To ease the notation, we write A [ i, :] v to denote the scalar product between A [ i, :] and the vector v ∈ R n . This notation will be used regularly when we decompose the functions of one-hidden-neural networks into sum of functions corresponding to hidden neurons.

For a vector v ∈ R d , v [ I ] ∈ R | I | is the vector v restricted to coefficients in I ⊆ J d K . If I = { i } a singleton, v [ i ] ∈ R is the i th coefficient of v . We also use 1 m and 0 m to denote an all-one (resp. all-zero) vector of size m .

For a dense (fully connected) feedforward architecture, we denote N = ( N L , . . . , N 0 ) the dimensions of the input layer N 0 = d , hidden layers ( N L -1 , . . . , N 1 ) and output layer ( N L ), respectively. The parameters space of the dense architecture N is denoted by N N : it is the set of all coefficients of the weight matrices W i ∈ R N i × N i -1 and bias vectors b i ∈ R N i , i = 1 , . . . , L . It is easy to verify that

N

N

is isomorphic to

R

N

of the architecture.

Clearly, N I ⊆ N N since:

A special subset of N I is the set of network parameters with zero biases,

Given an activation function ν , the realization R ν θ of a neural network θ ∈ N N is the function

We denote R ν : θ ↦→R ν θ the functional mapping from a set of parameters θ to its realization. The function space associated to a sparse architecture I and activation function ν is the image of N I under R ν :

When ν = σ the ReLU activation function, we recover the definition of realization in Equation (1). We use the shorthands

as in the main text. This allows us to define L I (cf. Equation (2)) as L I := R Id ( N 0 I ) where ν = Id is the identity map, which is a subset of linear maps R N 0 ↦→ R N L .

B Proofs for results in Section 3

B.1 Proof of Proposition 3.1

Proof. First, we remind the problem of the training of a sparse neural network on a finite data set D = { ( x i , y i ) } P i =1 :

which shares the same optimal value as the following optimization problem: 532

where Ω = { x i } P i =1 . This is simply a change of variables: from R θ ( x i ) to the i th column of 533 D = R θ (Ω) . We prove two implications as follows: 534

where

N

=

∑

L

i

=1

N

N

+

L

i

=1

∑

N

is the total number of parameters

-

i

1

i

i

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

Assume the closedness of F I (Ω) for every finite Ω . Then an optimal solution of the optimization problem (10) exists for every finite data set { ( x i , y i ) } P i =1 . Consider a training set { ( x i , y i ) } P i =1 and Ω := { x i } P i =1 . Since D := 0 P × N L ∈ F I (Ω) (by setting all parameters in θ equal to zero), the set F I (Ω) is non-empty. The optimal value of (11) is thus upper bounded by L ( 0 ) . Since the function glyph[lscript] ( · , y i ) is coercive for every y i in the training set, there exists a constant C (dependent on the training set and the loss) such that minimizing (11) on F I (Ω) or on F I (Ω) ∩ B (0 , C ) (with B ( 0 , C ) the L 2 ball of radius C centered at zero) yields the same infimum. The function L is continuous, since each glyph[lscript] ( · , y i ) is continuous by assumption, and the set F I (Ω) ∩B (0 , C ) is compact, since it is closed (as an intersection of two closed sets) and bounded (since B (0 , C ) is bounded). As a result there exists a matrix D ∈ F I (Ω) ∩ B (0 , C ) yielding the optimal value for (11). Thus, the parameters θ such that R θ (Ω) = D is an optimal solution of (10).

Assume that an optimal solution of problem 10 exists for every finite data set { ( x i , y i ) } P i =1 . Then F I (Ω) is closed for every Ω finite. We prove the contraposition of this claim. Assume there exists a finite set Ω = { x i } P i =1 such that F I (Ω) is not closed. Then, there exists a matrix D ∈ R N L × P such that D ∈ F I (Ω) \ F I (Ω) . Consider the dataset { ( x i , y i ) } P i =1 where y i ∈ R N L is the i th column of D . We prove that the infimum value of (10) is V := ∑ P i =1 glyph[lscript] ( y i , y i ) . Indeed, since D ∈ F I (Ω) , there exists a sequence { θ k } k ∈ N such that lim k →∞ R θ k (Ω) = D . Therefore, by continuity of glyph[lscript] ( · , y i ) , we have:

Moreover, the infimum cannot be smaller than V because the i th summand is at least glyph[lscript] ( y i , y i ) (due to the assumption on glyph[lscript] in Proposition 3.1). Therefore, the infimum value is indeed V . Since we assume that y is the only minimizer of y ′ ↦→ glyph[lscript] ( y ′ , y ) , this value can be achieved only if there exists a parameter θ ∈ I such that R θ (Ω) = D . This is impossible due to our choice of D which does not belong to F I (Ω) . We conclude that with our constructed data set D , an optimal solution does not exist for (10).

B.2 Proof of Lemma 3.2

The proof of Lemma 3.2 (and thus, as discussed in the main text, of Theorem 3.1) use four technical lemmas. Lemma B.1 is proved in Appendix C.1 since it involves Theorem 4.1. The other lemmas are proved right after the proof of Lemma 3.2.

Lemma B.1. If A ∈ L I \L I ⊆ R N L × N 0 then the function f : x ↦→ f ( x ) := A x satisfies f ∈ F I (Ω) \ F I (Ω) for every subset Ω of R N 0 that is bounded with non-empty interior.

Lemma B.2. Consider Ω = { x i } P i =1 a finite subset of R d and Ω ′ = [ -B,B ] d such that Ω ⊆ Ω ′ . If f ∈ F I (Ω ′ ) (under the topology induced by ‖ · ‖ ∞ ), then D := [ f ( x 1 ) . . . f ( x P ) ] ∈ F I (Ω) .

Lemma B.3. Consider R θ , the realization of a ReLU neural network with parameter θ ∈ I . This function is continuous and piecewise linear. On the interior of each piece, its Jacobian matrix is constant and satisfies J ∈ L I .

Lemma B.4. For p, N ∈ N , consider the following set of points (a discretized grid for [0 , 1] N ):

If H ∈ N satisfies p ≥ 3 NH , then for any collection of H hyperplanes, there exists x ∈ Ω N p such that the elementary hypercube whose vertices are of the form

lies entirely inside a polytope delimited by these hyperplanes. 563

We are now ready to prove Lemma 3.2. 564

565

566

567

568

569

570

571

Proof of Lemma 3.2. Since L I is not closed, there exists a matrix A ∈ L I \ L I , and we consider f ( x ) := A x . Setting p := 3 N 0 4 ∑ L -1 i =1 N i we construct Ω as the grid:

so that the cardinality of Ω = { x i } P i =1 is P := ( p + 1) N 0 . Similar to the sketch proof, consider D := [ f ( x 1 ) , f ( x 2 ) , . . . , f ( x P ) ] . Our goal is to prove that D ∈ F I (Ω) \ F I (Ω) .

First, notice that D ∈ F I (Ω) as an immediate consequence of Lemma B.2 and Lemma B.1.

It remains to show that D / ∈ F I (Ω) . We proceed by contradiction, assuming that there exists θ ∈ N I such that R θ (Ω) = D .

To show the contradiction, we start by showing that, as a consequence of Lemma B.4 there exists x ∈ Ω such that the hypercube whose vertices are the 2 N 0 points

lies entirely inside a linear region P of the continuous piecewise linear function R θ [1]. Denote 572 K = 2 ∑ L i =1 N i a bound on the number of such linear regions, see e.g. [24]. Each frontier between a 573 pair of linear regions can be completed into a hyperplane, leading to at most H = K 2 hyperplanes. 574 Since p = 3 N 0 K 2 ≥ 3 N 0 H , by Lemma B.4 there exists x ∈ Ω such that the claimed hypercube lies 575 entirely inside a polytope delimited by these hyperplanes. As this polytope is itself included in some 576 linear region P of R θ , this establishes our intermediate claim. 577

Now, define v 0 := x and v i := x +(1 /p ) e i , i ∈ J N 0 K where e i is the i th canonical vector. Denote P ∈ R N L × N 0 the matrix such that the restriction of R θ to the piece P is f P ( x ) = P x + b . Since P is the Jacobian matrix of R θ in the linear region P , we deduce from Lemma B.3 that P ∈ L I . Since the points v i belong to the hypercube which is both included in P and in Ω we have for each i :

where the third equality follows from the definition of D and the fact that we assume R θ (Ω) = D . 578 Since v 0 -v i = e i /p, i = 1 , . . . , n are linearly independent, we conclude that P = A . This implies 579 A ∈ L I , hence the contradiction. This concludes the proof. 580

We now prove the intermediate technical lemmas. 581

Proof of Lemma B.2. Since f ∈ F I (Ω ′ ) , there exists a sequence { θ k } k ∈ N such that:

Denoting D k := [ R θ k ( x 1 ) . . . R θ k ( x r ) ] , since x i ∈ Ω ⊆ Ω ′ , i = 1 , . . . , P , it follows that D k converges to D . Since D k ∈ F I (Ω) by construction, we get that D ∈ F I (Ω) .

Proof of Lemma B.3. For any θ ∈ I , R θ is a continuous piecewise linear function since it is the realization of a ReLU neural network [1]. Consider P a linear region of R θ with non-empty interior. The Jacobian matrix of P has the following form [28, Lemma 9]:

where D i is a binary diagonal matrix (diagonal matrix whose coefficients are either one or zero). 584 Since supp ( D i W i ) ⊆ supp ( W i ) ⊆ I i , we have: J = W L ∏ L -1 i =1 ( D i W i ) ∈ L I . 585

582

583

598

The rest of the proof is devoted to showing that this number is strictly smaller than p N if p ≥ 3 NH , 600

Proof of Lemma B.4. Every edge of an elementary hypercube can be written as:

where e i is the i th canonical vector, 1 ≤ i ≤ N . The points x and x +(1 /p ) e i are two endpoints . 586 Note that in this proof we use the notation ( a, b ) to denote the line segment whose endpoints are 587 a and b . By construction, Ω N p contains p N such elementary hypercubes. Given a collection of H 588 hyperplanes, we say that an elementary hypercube is an intersecting hypercube if it does not lie 589 entirely inside a polytope generated by the hyperplanes, meaning that there exists a hyperplane that 590 intersects at least one of its edges. More specifically, an edge and a hyperplane intersect if they have 591 exactly one common point. We exclude the case where there are more than two common points since 592 that implies that the edge lies completely in the hyperplane. The edges that are intersected by at least 593 one hyperplane are called intersecting edges . Note that a hypercube can have intersecting edges, but 594 it may not be an intersecting one. A visual illustration of this idea is presented in Figure 3. 595

intersecting edge

hyperplane

non-intersecting edge

a)

b)

c)

Formally, a hyperplane { w glyph[latticetop] x + b = 0 } for w ∈ R N and b ∈ R intersects an edge ( x, x + 1 p e i ) if: 596

glyph[negationslash]

glyph[negationslash]

We further illustrate these notions in Figure 4. We emphasize that according to Equation (13), glyph[lscript] 3 in 597 Figure 4 does not intersect any edge along its direction .

glyph[lscript]

1

glyph[lscript]

2

glyph[lscript]

3

intersecting hypercube

non-intersecting hypercube

hyperplane

intersecting edge

602

603

604

605

606

607

608

609

610

611

612

613

614

To estimate the maximum number of intersecting edges, we analyze the maximum number of edges that a given hyperplane can intersect. For a fixed index 1 ≤ i ≤ N , we count the number of edges of the form ( x, x + 1 p e i ) intersected by a single hyperplane. The key observation is: if we fix all the coordinates of x except the i th one, then the edges ( x, x + 1 p e i ) form a line in the ambient space. Among those edges, there are at most two intersecting edges with respect to the given hyperplane. This happens only when the hyperplane intersects an edge at one of its endpoints (e.g., the hyperplane glyph[lscript] 2 and the second vertical line in Figure 4). In total, for each 1 ≤ i ≤ N and each given hyperplane, there are at most 2( p +1) N -1 intersecting edges of the form ( x, x + 1 p e i ) . For a given hyperplane, there are thus at most 2 N ( p +1) N -1 intersecting edges in total (since i ∈ J N K ). Since the number of hyperplanes is at most H , there are at most 2 NH ( p +1) N -1 intersecting edges, and this quantity also bounds the number of intersecting cubes as we have seen. With the assumption p ≥ 3 NH , we conclude by proving that p N > 2 NH ( p +1) N -1 . Indeed, we have:

where we used that (1 + 1 /n ) n ≤ e ≈ 2 . 71828 , the Euler number.

B.3 Proof of Theorem 3.4

Proof. We denote X = [ x 1 , . . . , x P ] ∈ R N 0 × P , the matrix representation of Ω . Our proof has three main steps:

Step 1: We show that we can reduce the study of the closedness of F I (Ω) to that of the closedness of a union of subsets of R P , associated to the vectors W 2 . To do this, we prove that for any element f ∈ F I (Ω) , there exists a set of parameters θ ∈ N I such that the matrix of the second layer W 2 belongs to {-1 , 0 , 1 } 1 × N 1 (since we assume N 2 = 1 ). This idea is reused from the proof of [1, Theorem 4.1].

For θ := { ( W i , b i ) 2 i =1 } ∈ N I , the function R ( θ ) has the form:

where w 1 ,i = W 1 [ i, :] ∈ R 1 × N 0 , w 2 ,i = W 2 [ i ] ∈ R , b 1 ,i = b [ i ] ∈ R . Moreover, if w 2 ,i is different from zero, we have:

In that case, one can assume that w 2 ,i can be equal to either -1 or 1 . Thus, we can assume w 2 ,i ∈ {± 1 , 0 } . For a vector v ∈ {-1 , 0 , 1 } 1 × N 1 , we define:

where N I , v ⊆ N I is the set of θ = { ( W i , b i ) 2 i =1 } with W 2 = v ∈ { 0 , 1 } 1 × N 1 , i.e., in words, F v is the image of Ω through the function R θ , θ ∈ N I , v .

Define V := { v | supp ( v ) ⊆ I 2 } ∩ { 0 , ± 1 } 1 × N 1 . Clearly, for v ∈ V , F v ⊆ F I (Ω) . Therefore,

Moreover, by our previous argument, we also have:

Therefore,

Step 2: Using the first step, to prove that F I (Ω) is closed, it is sufficient to prove that F v is closed, 615 ∀ v ∈ V . This can be accomplished by further decomposing F v into smaller closed sets. We denote 616 θ ′ the set of parameters W 1 , b 1 and b 2 . In the following, only the parameters of θ ′ are varied since 617 W 2 is now fixed to v . 618

Due to the activation function σ , for a given data point x j ∈ Ω , we have: 619

where D j ∈ D , the set of binary diagonal matrices, and its diagonal coefficients D j [ i, i ] are 620 determined by: 621

Note that D j [ i, i ] can take both values 0 or 1 if W [ i, :] x j + b 1 [ i ] = 0 . We call the matrix D j the activation matrix of x j . Therefore, for (15) to hold, the N 1 constraints of the form (16) must hold simultaneously. It is important to notice that all these constraints are linear w.r.t. θ ′ . We denote z a vectorized version of θ ′ (i.e., we concatenate all coefficients whose indices are in I 1 of W and b 1 , b 2 into a long vector), and we write all the constraints in (15) in a compact form:

where A ( D j , x j ) is a constant matrix that depend on D j and x j . 622

Set θ = ( v , z ) . Given that (15) holds, we deduce that:

where V ( D j , x j , v ) is a constant matrix that depends on D j , v , x j . In particular, R θ ( x j ) is also a linear function w.r.t the parameters z . Assume that the activation matrices of ( x 1 , . . . , x P ) are ( D 1 , . . . , D P ) , then we have:

To emphasize that R θ (Ω) depends linearly on z , for the rest of the proof, we will write R θ (Ω) as a vector of size P (instead of a row matrix 1 × P ) as follows:

Moreover, to have ( D 1 , . . . , D P ) activation matrices, the parameters z need to satisfy:

Thus, the set of R θ (Ω) given the activation matrices ( D 1 , . . . , D P ) has the following compact form:

Clearly, F ( D 1 ,..., D P ) v ⊆ F v since each element is equal to R θ (Ω) with θ = ( v , z ) for some z . On the other hand, each element of F v is an element of F ( D 1 ,..., D P ) v for some ( D 1 , . . . , D P ) ∈ D P since the set of activation matrices corresponding to any θ is in D P . Therefore,

where Q = PN 1 and

Step 3: Using the previous step, it is sufficient to prove that F ( D 1 ,..., D P ) v is closed, for any 623 v , ( D 1 , . . . , D P ) ∈ D P . To do so, we write F ( D 1 ,..., D P ) v in a more general form: 624

Therefore, it is sufficient to prove that a set as in Equation (17) is closed. These sets are linear 625 transformations of an intersection of a finite number of half-spaces. Since the intersection of a 626 finite number of halfspaces is stable under linear transformations (cf. Lemma B.5 below), and the 627 intersection of a finite number of half-spaces is a closed set itself, the proof can be concluded. 628

Lemma B.5 (Closure of intersection of half-spaces under linear transformations) . For any A ∈ R m × n , C ∈ R glyph[lscript] × n , y ∈ R glyph[lscript] , there exists C ′ ∈ R k × m , b ′ ∈ R k such that:

Proof. The proof uses Fourier-Motzkin elimination 4 . This method is a quantifier elimination algorithm for linear functions 5 . In fact, the LHS can be written as: { t | t = Ax , Cx ≤ y } , or more generally,

where ( x t ) is the concatenation of two vectors ( x , t ) and the linear constraints imposed by B ( x t ) ≤ v 629 replace the two linear constraints Cx ≤ y and t = Ax . The idea is to show that: 630

631

632

633

634

635

636

637

638

for some matrix B ′ and vector v ′ . By doing so, we reduce the dimension of the quantified parameter x by one. By repeating this procedure until there is no more quantifier, we prove the lemma. The rest of this proof is thus devoted to show that B ′ , v ′ as in (18) do exist.

We will show how to eliminate the first coordinate of x [1] . First, we partition the set of linear constraints of LHS of (18) into three groups:

S 0 := { j | B [ j, 1] = 0 } : In this case, x [1] does not appear in this constraint, there is nothing to do.

S + := { j | B [ j, 1] > 0 } , for j ∈ S + , we can rewrite the constraints B [ j, :] ( x t ) ≤ v [ j ] as:

for some suitable γ [ j ] , α [ i ] , β [ i ] where x ′ is the last ( n -1) coordinate of the vector x .

S -:= { j | B [ j, 1] < 0 } : for j ∈ S -, we can rewrite the constraints B [ j, :] ( x t ) ≤ v j as:

For the existence of such x [1] , it is necessary and sufficient that: 639

Thus, we form the matrix B ′ and the vector v ′ such that the linear constraints written in the following form:

represent all the linear constraints in the set S 0 and those in the form of (19). Using this procedure 640 recursively, one can eliminate all quantifiers and prove the lemma. 641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

B.4 Proofs for Lemma 3.3

Since we use tools of real algebraic geometry, this section provides basic notions of real algebraic geometry for readers who are not familiar with this domain. It is organized and presented as in the textbook [2] (with slight modifications to better suit our needs). For a more complete presentation, we refer readers to [2, Chapter 2].

Definition B.1 (Semi-algebraic sets) . A semi-algebraic set of R n has the form:

where P i , Q i,j : R n ↦→ R are polynomials and ∧ is the 'and' logic.

The following theorem is known as the projection theorem of semi-algebraic sets. In words, the theorem states that: The projection of a semi-algebraic set to a lower dimension is still a semi-algebraic set (of lower dimension).

Theorem B.6 (Projection theorem of semi-algebraic sets [2, Theorem 2.92]) . Let A be a semialgebraic set of R n and define:

then B is a semi-algebraic set of R n -1 .

Theorem B.6 is a powerful result. Its proof [2, Section 2 . 4 ] (which is constructive) shows a way to express B (in Theorem B.6) by using only the first n -1 variables ( x 1 , . . . , x n -1 ) .

Next, we introduce the language of an ordered field and sentence. Readers which are not familiar to the notion of ordered field can simply think of it as R and its subring as Q . Example for fields that is not ordered is C (we cannot compare two arbitrary complex number). Therefore, the notion of semi-algebraic set in Definition B.1 (which contains Q i,j ( x ) > 0 ) does not make sense when the underlying field is not ordered.

The central definition of the language of R is formula , an abstraction of semi-algebraic sets. In particular, the definition of formula is recursive: formula is built from atoms - equalities and inequalities of polynomials whose coefficients are in a subring Q of R . It can be also formed by combining with logical connectives 'and', 'or', and 'negation' ( ∧ , ∨ , ¬ ) and existential/universal quantifiers ( ∃ , ∀ ). Formula has variables, which are those of atoms in the formula itself. Free variables of a formula are those which are not preceded by a quantifier ( ∃ , ∀ ). The definitions of a formula and its free variables are given recursively as follow:

Definition B.2 (Language of the ordered field with coefficients in a ring) . Consider R an ordered field and Q ⊆ R a subring, a formula Φ and its set of free variables Free ( X ) are defined recursively as:

An atom: if P ∈ Q [ X ] (where Q [ X ] is the set of polynomials with coefficients in Q ) then Φ := ( P = 0) (resp. Φ := ( P > 0) ) is a formula and its set of free variables is Free (Φ) := { X 1 , . . . , X n } where n is the number of variables.

If Φ 1 and Φ 2 are formulas, then so are Φ 1 ∨ Φ 2 , Φ 1 ∧ Φ 2 and ¬ Φ 1 . The set of free variables is defined as:

(a) Free (Φ 1 ∨ Φ 2 ) := Free (Φ 1 ) ∪ Free (Φ 2 )

(b) Free (Φ 1 ∧ Φ 2 ) := Free (Φ 1 ) ∪ Free (Φ 2 )

(c) Free ( ¬ Φ 1 ) = Free (Φ 1 )

If Φ is a formula and X ∈ Free (Φ) , then Φ ′ = ( ∃ X )Φ and Φ ′′ = ( ∀ X )Φ are also formulas and Free (Φ ′ ) := Free (Φ) \ { X } , and Free (Φ ′′ ) := Free (Φ) \ { X } .

Definition B.3 (Sentence) . A sentence is a formula of an ordered field with no free variable. Example B.1. Consider two formulas:

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

While Φ 1 is a normal formula, Φ 2 is a sentence and given an underlying field ( R , for instance), Φ 2 is either correct or not. Here, Φ 2 is correct (since X 2 1 + X 2 2 = 0 has a root (0 , 0) ). Nevertheless, if one consider Φ ′ 2 = {∃ X 1 , ∃ X 2 , X 2 1 + X 2 2 = -1 } , then Φ ′ 2 is not correct.

An algorithm deciding whether a sentence is correct or not is very tempting since formula and sentence can be used to express many theorems in the language of an ordered field. The proof or disproof will be then given by an algorithm. Such an algorithm does exist, as follow:

Theorem B.7 (Decision problem [2, Algorithm 11.36]) . There exists an algorithm to decide whether a given sentence is correct is not with complexity O ( sd ) O (1) k -1 where s is the bound on the number of polynomials in Φ , d is the bound on the degrees of the polynomials in Φ and k is the number of variables.

A full description of [2, Algorithm 11.36] (quantifier elimination algorithm) is totally out of the scope of this paper. Nevertheless, we will try to explain it in a concise way. The key observation is Theorem B.6, the central result of real algebraic geometry. As discussed right after Theorem B.6, its proof implies that one can replace a sentence by another whose number of quantifiers is reduced by one such that both sentences agree (both are true or false). Applying this procedure iteratively will result into a sentence without any variable (and the remain are only coefficients in the subring ). We check the correctness of this final sentence by trivially verifying all the equalities/inequalities and obtain the answer for the original one.

Proof of Lemma 3.3. To decide whether L I is closed or not, it is equivalent to decide if the following sentence (see Definition B.3) is true or false:

This sentence basically asks whether there exists a matrix A ∈ F I \ F I or not. It can be proved that this sentence can be decided to be true or false using real algebraic geometry tools (see Theorem B.7), with a complexity O ( ( sd ) C k -1 ) where C is a universal constant and s, d, k are the number of polynomials, the maximum degree of the polynomials and the number of variables in the sentence, respectively. Applying this to our case, we have s = 2 , d = 2 L, k = N L N 0 + 1 + 2 ∑ L glyph[lscript] =1 | I glyph[lscript] | (remind that | I glyph[lscript] | is the total number of unmasked coefficients of X glyph[lscript] ).

B.5 Polynomial algorithm to detect support constraints I = ( I, J ) with non-closed L I .

The following sufficient condition for non-closedness is based on the existence in the support constraint of 2 × 2 blocks sharing the essential properties of a 2 × 2 LU support constraint.

glyph[negationslash]

Lemma B.8. Consider a pair I = ( I, J ) ∈ { 0 , 1 } m × r ×{ 0 , 1 } r × n of support constraints for the weight matrices of one-hidden-layer neural network. If there exists four indices 1 ≤ i 1 , i 2 ≤ m, 1 ≤ j 1 , j 2 ≤ n and two indices k = l, 1 ≤ k, l ≤ r such that:

For each pair ( i, j ) ∈ { ( i 1 , j 1 ) , ( i 1 , j 2 ) , ( i 2 , j 1 ) } we have:

glyph[negationslash]

2. The pair ( i 2 , j 2 ) belongs to supp ( I [: , k ] J [ k, :]) and to supp ( I [: , l ] J [ l, :]) .

then L I is non-closed. 712

Proof. First, it is easy to see that the assumptions of this lemma are equivalent to those of [18, Theorem 4.20] since supp ( I [: , k ] J [ k, :]) is precisely the k th rank-one support of the pair ( I, J ) [18, Definition 3.1]. Without loss of generality, one can assume that i 1 , j 1 = 1 , i 2 , j 2 = 2 and k = 1 , l = 2 . We will prove that A ∈ L I \ L I where

This can be shown in two steps: 713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

Proof that A ∈ L I : For any glyph[epsilon1] > 0 , consider two factors:

where X ′ glyph[epsilon1] , Y ′ glyph[epsilon1] ∈ R 2 × 2 respect the support constraints corresponding to the LU architecture. It is not hard to see that such a construction of ( X glyph[epsilon1] , Y glyph[epsilon1] ) satisfies the support constraints ( I, J ) (due to the assumption of the lemma and the value of indices). Moreover, we also have:

Thus, to have ‖ A -X glyph[epsilon1] Y glyph[epsilon1] ‖ F ≤ glyph[epsilon1] , it is sufficient to choose a pair of factors ( X ′ glyph[epsilon1] , Y ′ glyph[epsilon1] ) respecting the LU architecture of size 2 × 2 such that ‖ A ′ -X ′ glyph[epsilon1] Y ′ glyph[epsilon1] ‖ F ≤ glyph[epsilon1] . Such a pair exists, since the set of matrices admitting the exact LU decomposition is dense in R 2 × 2 . This holds for any glyph[epsilon1] > 0 . Therefore, A ∈ L I .

Proof that A / ∈ L I : Assume there exist a pair of factors ( X , Y ) whose product XY = A and supports are included in ( I, J ) . Due to the assumptions on the pairs ( i 1 , j 1 ) , ( i 1 , j 2 ) , ( i 2 , j 1 ) , we must have:

It is easy to see that it is impossible. Therefore, A / ∈ L I .

Given a pair of support constraints I , it is possible to check in time polynomial in m,r, n whether the conditions of Lemma B.8 hold. A brute force algorithm has complexity O ( m 2 n 2 r ) . A more clever implementation with careful book-marking can reduce this complexity to O (min( m,n ) mnr ) .

C Proofs for results in Section 4

C.1 Proof of Theorem 4.1

In fact, Theorem 4.1 is a corollary of Lemma B.1. Thus, we will give a proof for Lemma B.1 in the following.

Proof of Lemma B.1. Since A ∈ L I \L I ⊆ R N L × N 0 , we have:

There exists a sequence { ( X k i ) L i =1 } k ∈ N such that lim k →∞ ‖ X k L . . . X k 1 -A ‖ = 0 for any norm defined on R N 0 .

We will prove that the linear function: f ( x ) := A x satisfies f ∈ F I \ F I (where F I is the closure of F I in ( C 0 (Ω) , ‖·‖ ∞ ) , that is to say f is not the realization of any neural network but it is the uniform limit of the realizations of a sequence of neural networks). Firstly, we prove that f / ∈ F I . For the sake of contradiction, assume there exists θ = ( W i , b i ) L i =1 ∈ N I such that R θ = f . Since R θ is the realization of a ReLU neural network, it is a continuous piecewise linear function. Therefore, since Ω has non-empty interior, there exist a non-empty open subset Ω ′ of R d such that Ω ′ ⊆ Ω and R θ is linear on Ω ′ , i.e., there are A ′ ∈ R N L × N 0 , b ∈ R N L such that R θ ( x ) = A ′ x + b ′ , ∀ x ∈ Ω ′ . Since f = R θ , we have: A ′ = A and also equal to the Jacobian matrix of R θ on Ω ′ . Using Lemma B.3 and the fact that A / ∈ L I , we conclude that f / ∈ F I .

There remains to construct a sequence { θ k } k ∈ N , θ k = ( W k i , b k i ) L i =1 ∈ N I such that lim k →∞ ‖R θ k -f ‖ ∞ = 0 . We will rely on the sequence { ( X k i ) L i =1 } k ∈ N for our construction. Given k ∈ N we simply define the weight matrices as W k i = X k i , 1 ≤ i ≤ L . The biases are built recursively. Starting from c k 1 := sup x ∈ Ω ‖ W k 1 x ‖ ∞ and b k 1 := c k 1 1 N 1 , we iteratively define for 2 ≤ i ≤ L -1 :

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

The boundedness of Ω ensures that c k i is well-defined with a finite supremum. For i = L we define:

We will prove that R θ k ( x ) = ( X k L . . . X k 1 ) x, ∀ x ∈ Ω . As a consequence, it is immediate that:

where we used that all matrix norms are equivalent and denoted ‖·‖ 2 → 2 the operator norm associated to Euclidean vector norms. Back to the proof that R θ k ( x ) = ( X k L . . . X k 1 ) x, ∀ x ∈ Ω , due to our choice of c k i , we have for 2 ≤ i ≤ L -1 :

where ≥ is taken in coordinate-wise manner. Therefore, an easy induction yields:

C.2 Proof of Theorem 4.2

Given the involvement of Theorem 4.2, we decompose its proof and present it in two subsections: the first one establishes general results that do not use the assumption of Theorem 4.2. The second one combines the established results with the assumption of Theorem 4.2 to provide a full proof.

C.2.1 Properties of the limit function of fixed support one-hidden-layer NNs

The main results of this parts are summarized in Lemma C.2 and Lemma C.3. It is important to emphasize that all results in this section do not make any assumption on I .

We first introduce the following technical results.

Lemma C.1 (Normalization of the rows of the first layer [26]) . Consider Ω a bounded subset of R N 0 . Given any θ = { ( W i , b i ) 2 i =1 } ∈ N I and any norm ‖ · ‖ on R N 0 , there exists ˜ θ := { ( ˜ W i , ˜ b i ) 2 i =1 } ∈ N I such that the matrix ˜ W 1 has unit norm rows, ‖ ˜ b 1 ‖ ∞ ≤ C := sup x ∈ Ω sup ‖ u ‖≤ 1 〈 u, x 〉 and R θ ( x ) = R ˜ θ ( x ) , ∀ x ∈ Ω .

Proof. We report this proof for self-completeness of this work. It is not a contribution, as it merely combines ideas from the proof of [26, Lemma D.2] and [26, Theorem 3.8, Steps 1-2].

glyph[negationslash]

We first show that for each set of weights θ ∈ N I we can find another set of weights θ ′ = { ( W ′ i , b ′ i ) 2 i =1 } ∈ N I such that R θ = R θ ′ on R N 0 and W ′ 1 has unit norm rows. Note that ‖ b ′ 1 ‖ ∞ can be larger than C . Indeed, given θ ∈ N I , the function R θ can be written as: R θ : x ∈ R N 0 ↦→ ∑ N 1 j =1 h j ( x ) + b 2 where h j ( x ) = W 2 [: , j ] σ ( W 1 [ j, :] x + b 1 [ j ]) denotes the contribution of the j th hidden neuron. For hidden neurons corresponding to nonzero rows of W k 1 , we can rescale the rows of W k 1 , the columns of W k 2 and b k 1 such that the realization of h j is invariant. This is due to the fact that w 2 σ ( w glyph[latticetop] 1 x + b ) = ‖ w 1 ‖ w 2 σ (( w 1 / ‖ w 1 ‖ ) glyph[latticetop] x +( b/ ‖ w 1 ‖ )) for any w 1 = 0 ∈ R N 0 , w 2 ∈ R N 2 , b ∈ R . Neurons corresponding to null rows of W k 1 are handled similarly,

in an iterative manner, by setting them to an arbitrary normalized row, setting the corresponding 762 column of W k 2 to zero, and changing the bias b k 2 to keep the function R θ unchanged on R N 0 , using 763 that w 2 σ ( 0 glyph[latticetop] x + b ) + b 2 = 0 σ ( v glyph[latticetop] x + b ) + ( b 2 + w 2 σ ( b )) for any normalized vector v ∈ R N 0 . 764 Thus, we obtain θ ′ whose matrix of the first layer, W ′ 1 , has normalized rows and R θ = R θ ′ on R N 0 . 765

To construct ˜ θ with ‖ ˜ b 1 ‖ ∞ ≤ C we see that, by definition of C , if ‖ w 1 ‖ = 1 and b ≥ C then 766

Thus, the function w 2 σ ( w glyph[latticetop] 1 x + b ) = w 2 ( w glyph[latticetop] 1 x + b ) is linear on Ω and

Thus, for any hidden neuron with a bias exceeding C , the bias can be saturated to C by changing accordingly the output bias b 2 , keeping the function R θ unchanged on the bounded domain Ω (but not on the whole space R N 0 ). Hidden neurons with a bias b ≤ -C can be similarly modified. Sequentially saturating each hidden neuron yields ˜ θ which satisfies all conditions of Lemma C.1.

Lemma C.2. Consider Ω a bounded subset of R N 0 , for any I = ( I 2 , I 1 ) , given a continuous function f ∈ F I (Ω) , there exists a sequence { θ k } k ∈ N , θ k = ( W k i , b k i ) 2 i =1 ∈ N I such that:

The sequence R θ k admits f as its uniform limit, i.e., lim k →∞ ‖R θ k -f ‖ ∞ = 0 .

The sequence { ( W k 1 , b k 1 ) } k ∈ N has a finite limit ( W glyph[star] 1 , b glyph[star] 1 ) where W glyph[star] 1 has unit norm rows and supp ( W glyph[star] 1 ) ⊆ I 1 .

Proof. Given a function f ∈ F I (Ω) , by definition, there exists a sequence { θ k } k ∈ N , θ k ∈ N I ∀ k ∈ N such that lim k →∞ ‖R θ k -f ‖ ∞ = 0 . We can assume that W k 1 has normalized rows and b k 1 is bounded using Lemma C.1. We can also assume WLOG that the parameters of the first layer (i.e W k 1 , b k 1 ) have finite limits W glyph[star] 1 and b glyph[star] 1 . Indeed, since both W k 1 and b k 1 are bounded (by construction from Lemma C.1), there exists a subsequence { ϕ k } k ∈ N such that W ϕ k 1 and b ϕ k 1 have finite limits and R θ ϕ k → f as R θ k → f . Replacing the sequence { θ k } k ∈ N by { θ ϕ k } k ∈ N yields the desired sequence. Finally, since W glyph[star] 1 = lim k →∞ W k 1 , W glyph[star] 1 obviously has normalized rows and supp ( W glyph[star] 1 ) ⊆ I 1 .

Definition C.1. Consider Ω bounded subset of R d , a function f ∈ F I (Ω) and a sequence { θ k } k ∈ N as given by Lemma C.2. We define ( a i , b i ) = ( W glyph[star] 1 [ i, :] , b glyph[star] 1 [ i ]) the limit parameters of the first layer corresponding to the i th neuron. We partition the set of neurons into two subsets (one of them may be empty):

Set of active neurons: J := { i | ( ∃ x ∈ Ω , a i x + b i > 0) ∧ ( ∃ x ∈ Ω , a i x + b i < 0) } .

Set of non-active neurons: ¯ J = J N 1 K \ J .

For i, j ∈ J , we write i glyph[similarequal] j if ( W glyph[star] 1 [ j, :] , b glyph[star] 1 [ j ]) = ± ( W glyph[star] 1 [ i, :] , b glyph[star] 1 [ i ]) . The relation glyph[similarequal] is an equivalence relation.

We define ( J glyph[lscript] ) glyph[lscript] =1 ,...,r the equivalence classes induced by glyph[similarequal] and we use ( α glyph[lscript] , β glyph[lscript] ) := ( a i , b i ) for some i ∈ J glyph[lscript] as the representative limit of the glyph[lscript] th equivalence class. For i ∈ J glyph[lscript] , we have: ( a i , b i ) = glyph[epsilon1] i ( α glyph[lscript] , β glyph[lscript] ) , glyph[epsilon1] i ∈ {± 1 } . We define J + glyph[lscript] = { i ∈ J glyph[lscript] | glyph[epsilon1] i = 1 } glyph[negationslash] = ∅ and J -glyph[lscript] = J glyph[lscript] \ J + glyph[lscript] .

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

For each equivalence class J glyph[lscript] , define H glyph[lscript] := { x ∈ Ω | α glyph[lscript] x + β glyph[lscript] = 0 } the boundary generated by neurons in J glyph[lscript] and the positive (resp. negative) half-space partitioned by H glyph[lscript] , H + glyph[lscript] := { x ∈ Ω | α glyph[lscript] x + β glyph[lscript] > 0 } (resp. H -glyph[lscript] := { x ∈ Ω | α glyph[lscript] x + β glyph[lscript] < 0 } ). For any glyph[epsilon1] > 0 we also define the open half-spaces H ( glyph[epsilon1], +) glyph[lscript] := { x ∈ R d | α glyph[latticetop] glyph[lscript] x + β glyph[lscript] > glyph[epsilon1] } and H ( glyph[epsilon1], -) glyph[lscript] := { x ∈ R d | α glyph[latticetop] glyph[lscript] x + β glyph[lscript] < -glyph[epsilon1] } .

Definition C.1 groups neurons sharing the same 'linear boundary' (or 'singular hyperplane' as in 798 [26]). This concept is related to 'twin neurons' [28], which also groups neurons with the same active 799 zone. This partition somehow allows us to treat classes independently. Observe also that 800

801

802

803

804

805

806

807

808

809

810

811

812

Definition C.2 (Contribution of an equivalence class) . In the setting of Definition C.1, we define the contribution of the i th neuron 1 ≤ i ≤ N 1 (resp. of the glyph[lscript] th ( 1 ≤ glyph[lscript] ≤ r ) equivalence class) of θ k as:

Lemma C.3. Consider Ω = [ -B,B ] d , f ∈ F I (Ω) and a sequence { θ k } k ∈ N as given by Lemma C.2, and α glyph[lscript] , β glyph[lscript] , 1 ≤ glyph[lscript] ≤ r, glyph[epsilon1] i , i ∈ J as given by Definition C.1. There exist some γ glyph[lscript] , b ∈ R N 2 , A ∈ R N 2 × N 0 such that:

Proof. The proof is divided into three parts: We first show that there exist γ glyph[lscript] , b ∈ R N 2 and A ∈ R N 2 × N 0 such that Equation (22) holds. The last two parts will be devoted to prove that equations (23) - (25) hold.

Proof of Equation (22): Our proof is based on a result of [26], which deals with the case of a scalar output (i.e, N 2 = 1 ). It is proved in [26, Theorem 3.8, Steps 3 , 6 , 7 ] and states the following:

LemmaC.4 (Analytical form of a limit function with scalar output [26]) . In case N 2 = 1 (i.e., output dimension equal to one), consider Ω = [ -B,B ] d , a scalar-valued function f : Ω ↦→ R , f ∈ F I (Ω) and a sequence as given by Lemma C.2, there exist µ ∈ R N 0 , γ glyph[lscript] , ν ∈ R such that:

Back to our proof, one can write f = ( f 1 , . . . , f N 2 ) where f j : Ω ⊆ R N 0 ↦→ R is the function f restricted to the j th coordinate. Clearly, f j is also a uniform limit on Ω of {R ˜ θ k } k ∈ N for a sequence { ˜ θ k } k ∈ N which shares the same W k 1 with { θ k } k ∈ N but ˜ W k 2 is the j th row of W k 2 . Therefore, { ˜ θ k } k ∈ N also satisfies the assumptions of Lemma C.4, which gives us:

for some µ j ∈ R N 0 , γ i,j , ν j ∈ R . Note that α glyph[lscript] , β glyph[lscript] and r are not dependent on the index j since they are defined directly from the considered sequence. Therefore, the function f (which is the concatenation of f j coordinate by coordinate) is:

2. Proof for Equations (23) -(24): With the construction of γ , we will prove Equation (23) and Equation (24). We consider an arbitrary 1 ≤ glyph[lscript] ≤ r . Denoting Ω ◦ the interior of Ω and H glyph[lscript] := { x ∈ Ω | α glyph[lscript] x + β glyph[lscript] = 0 } the hyperplane defined by the input weights and bias of the glyph[lscript] -th class of

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

glyph[negationslash]

neurons, we take a point x ′ ∈ (Ω ◦ ∩ H glyph[lscript] ) \ ⋃ p = glyph[lscript] H p and a fixed scalar r > 0 such that the open ball B ( x ′ , r ) ⊆ Ω ◦ \ ⋃ p = glyph[lscript] H p . Notice that x ′ is well-defined due to the definition of J (Definition C.1). In addition, r also exists because Ω ◦ \ ⋃ p = glyph[lscript] H p is an open set. Thus, there exists two constants 0 < δ < B and glyph[epsilon1] > 0 such that:

glyph[negationslash]

(b) For each p = glyph[lscript] , the ball B ( x ′ , r ) is either included in the half-space H ( glyph[epsilon1], +) p := { x ∈ R d | α glyph[latticetop] p x + β p > glyph[epsilon1] } or in the half-space H ( glyph[epsilon1], -) p := { x ∈ R d | α glyph[latticetop] p x + β p < -glyph[epsilon1] } .

(c) The intersection of B ( x ′ , r ) with H ( glyph[epsilon1], +) glyph[lscript] and H ( glyph[epsilon1], -) glyph[lscript] are not empty.

For the remaining of the proof, we will use Lemma C.5, another result taken from [26]. We only state the lemma. Its formal proof can be found in the proof of [26, Theorem 3 . 8 , Steps 4-5].

Lemma C.5 (Affine linear area [26]) . Given a sequence { θ k } k ∈ N satisfying the second condition of Lemma C.2, we have:

(a) For any 0 < δ < B , there exists a constant κ δ such that ∀ i ∈ ¯ J , h k i are affine linear on [ -( B -δ ) , B -δ ] N 0 for all k ≥ κ δ .

(b) For any glyph[epsilon1] > 0 , there exists a constant κ glyph[epsilon1] such that for each 1 ≤ glyph[lscript] ≤ r and each i ∈ J glyph[lscript] the function h k i is affine linear on H ( glyph[epsilon1], +) glyph[lscript] ∪ H ( glyph[epsilon1], -) glyph[lscript] for all k ≥ κ glyph[epsilon1] .

The lemma implies the existence of K = max( κ δ , κ glyph[epsilon1] ) such that for all k ≥ K , we have:

glyph[negationslash]

for some B k ∈ R N 2 × N 0 , ν k ∈ R N 2 . Therefore, for k ≥ K , we have:

Since we proved that f has the form Equation (22), there exist C ∈ R N 2 × N 0 , µ ∈ R N 2 such that

As both B ( x ′ , r ) ∩ H ( glyph[epsilon1], +) glyph[lscript] and B ( x ′ , r ) ∩ H ( glyph[epsilon1], -) glyph[lscript] are open sets, and given our hypothesis of uniform 829 convergence of R θ k → f , we obtain, 830

Proof for Equation (27) can be found in Appendix C.4. Equations (23) and (24) follow directly from 831 Equation (27). 832

glyph[negationslash]

3. Proof of Equation (25) : Since α glyph[lscript] = 0 (remember that ‖ α glyph[lscript] ‖ = 1 ), this is an immediate conse833 quence of Equation (23) as each vector W k 2 [: , j ] , j ∈ J glyph[lscript] is supported in I 2 [: , j ] ⊆ ∪ i ∈ J glyph[lscript] I 2 [: , i ] . 834

glyph[negationslash]

glyph[negationslash]

835

836

837

838

839

840

841

842

843

844

845

846

847

848

849

850

851

852

853

854

855

856

857

858

859

860

861

862

863

We state an immediate corollary of Lemma C.3, which characterizes the limit of the sequence of contributions { g k glyph[lscript] } k ∈ N of the glyph[lscript] th equivalence class with | J glyph[lscript] | = 1 .

Corollary C.1. Consider f ∈ F I ([ -B,B ] d ) that admits the analytical form in Equation (22) , a sequence { θ k } k ∈ N as given by Lemma C.2, and Definition C.1. For all singleton equivalence classes J glyph[lscript] = { i } , 1 ≤ glyph[lscript] ≤ r , we have lim k →∞ W k 2 [: , i ] = γ glyph[lscript] and lim k →∞ ‖ h k glyph[lscript] -γ glyph[lscript] σ ( α glyph[latticetop] glyph[lscript] x + β glyph[lscript] ) ‖ ∞ = 0 .

Proof. We first prove that W k 2 [: , i ] has a finite limit. In fact, applying the second point of Lemma C.3 for J glyph[lscript] = { i } , we have:

where γ glyph[lscript] , α glyph[lscript] are defined in Lemma C.3. Because lim k →∞ W k 1 [ i, :] = α glyph[lscript] and ‖ α glyph[lscript] ‖ 2 = 1 , it follows that γ glyph[lscript] = lim k →∞ W k 2 [: , i ] . To conclude, since we also have β glyph[lscript] = lim k →∞ b k 1 [ i ] , we obtain h k glyph[lscript] ( · ) = W k 2 [ glyph[lscript], :] σ ( W k 1 [ glyph[lscript], :] · + b k 1 [ glyph[lscript] ]) → γ glyph[lscript] σ ( α glyph[lscript] x + β glyph[lscript] ) as claimed.

The nice thing about Corollary C.1 is that the contribution g k glyph[lscript] = h k glyph[lscript] admits a (uniform) limit if J glyph[lscript] = { i } . Moreover, this limit is even implementable by using only the i th neuron because supp ( α glyph[lscript] ) ⊆ I 1 [ i, :] and supp ( γ glyph[lscript] ) ⊆ I 2 [: , i ] .

It would be tempting to believe that, for each P ∈ { ¯ J } ∪ { J glyph[lscript] | glyph[lscript] = 1 , . . . , r } the sequence of functions ∑ i ∈ P g k i ( x ) must admit a limit (when k tends to ∞ ) and that this limit is implementable using only neurons in P . This would obviously imply that F I (Ω) is closed. This intuition is however wrong. For non-singleton equivalence class (i.e., for cases not covered by Corollary C.1), the limit function does not necessarily exist as we show in the following example.

Example C.1. Consider the case where N = (1 , 3 , 1) and no support constraint, Ω = [ -1 , 1] , take the sequence { θ k } k ∈ N which satisfies:

Then for x ∈ Ω , it is easy to verify that R θ k = 0 . Indeed,

Thus, this sequence converges (uniformly) to f = 0 . Moreover, this sequence also satisfies the assumptions of Lemma C.2. Using the classification in Definition C.1, we have one class equivalence J 1 = { 1 , 2 } and ¯ J = { 3 } . The function g k 1 ( x ) = kσ ( x ) -kσ ( -x ) = kx , however, does not have any limit.

C.2.2 Actual proof of Theorem 4.2

Therefore, our analysis cannot treat each equivalence class entirely separately. The last result in this section is about a property of the matrix A in Equation (22). This is one of our key technical contributions in this work.

Lemma C.6. Consider Ω = [ -B,B ] d , f ∈ F I (Ω) that admits the analytical form in Equation (22) , a sequence { θ k } k ∈ N as given by Lemma C.2, then the matrix A ∈ L I ′ where I ′ = ( I 2 [: , S ] , I 1 [ S, :]) , S = ¯ J ∪ ( ∪ 1 ≤ glyph[lscript] ≤ r J -glyph[lscript] ) , ¯ J, J ± glyph[lscript] are defined as in Definition C.1).

Combining Lemma C.6 and the assumptions of Theorem 4.2, we can prove Theorem 4.2 immediately as follow:

Proof of Theorem 4.2. Consider f ∈ F I (Ω) , we deduce that there exists a sequence of { θ k } k ∈ N that satisfies the properties of Lemma C.2. This allows us to define ¯ J and equivalence classes

864

865

866

867

868

869

870

871

872

873

874

875

876

877

J glyph[lscript] , 1 ≤ glyph[lscript] ≤ r as well as ( α glyph[lscript] , β glyph[lscript] ) as in Definition C.1. Using Lemma C.3, we can also deduce an analytical formula for f as in Equation (22):

Finally, Lemma C.6 states that matrix A in Equation (22) satisfies: A ∈ L I ′ with I ′ = ( I 2 [: , S ] , I 1 [ S, :]]) , where S = ¯ J ∪ ( ∪ r glyph[lscript] =1 J -glyph[lscript] ) . To prove that f ∈ F I , we construct the parameters θ = { ( W i , b i ) 2 i =1 } of the limit network as follows:

1. For each 1 ≤ glyph[lscript] ≤ r , choose one index j ∈ J + glyph[lscript] (which is possible since J + glyph[lscript] is non-empty). We set:

This satisfies the support constraint because supp ( α glyph[lscript] ) ⊆ I 1 [ j, :] (by (21)) α glyph[lscript] = lim k →∞ W k 1 [ j, :] ) and I 2 = 1 N 2 × N 1 . This is where we use the first assumption of Theorem 4.2. Without it, supp ( γ glyph[lscript] ) might not be a subset of I 2 [: , j ] .

2. For i ∈ S : Since A ∈ L I ′ (cf Lemma C.6) and L I ′ is closed (second assumptions of Theorem 4.2), there exist two matrices ˆ W 1 , ˆ W 2 such that: supp ( ˆ W 1 ) ⊆ I 1 [: , S ] , supp ( ˆ W 2 ) ⊆ I 2 [ S, :] , and A = ˆ W 2 ˆ W 1 . We set:

where C = sup x ∈ Ω ‖ ˆ W 1 x ‖ ∞ . This satisfies the support constraints I due to our choice of ˆ W 1 , ˆ W 2 . The choice of C ensures that the function h i ( x ) := W 2 [ i, :] σ ( W 1 [ i, :] x + b 1 [ i ]) is linear on Ω .

3. For b 2 : Let b 2 = b -C ( ∑ i ∈ S ˆ W 2 [: , i ] ) ( b is the bias in Equation (22)).

Verifying R θ = f on Ω is thus trivial since:

Proof of Lemma C.6. In this proof, we define Ω ◦ δ = ( -B + δ, B -δ ) N 0 , 0 < δ < B . The choice of 878 δ is not important in this proof (any 0 < δ < B will do). 879

880

881

The proof of this lemma revolves around the following idea: We will construct a sequence of functions { f k } k ∈ N such that, for k large enough, f k has the following analytical form:

and lim k →∞ f k ( x ) = f ( x ) ∀ x ∈ Ω \ ( ∪ r glyph[lscript] =1 H glyph[lscript] ) (or equivalently f k converges pointwise to f on 882 Ω \ ( ∪ r glyph[lscript] =1 H glyph[lscript] ) ) and A k admits a factorization into two factors A k = X k Y k satisfying supp ( X k ) ⊆ 883 I 2 [: , S ] , supp ( Y k ) ⊆ I 1 [ S, :] , so that A k ∈ L I ′ . Comparing Equation (22) and Equation (30), we 884 deduce that the sequence of affine functions A k x + b k converges pointwise to the affine function 885 A x + b on the open set Ω ◦ δ \ ( ∪ r glyph[lscript] =1 H glyph[lscript] ) . Therefore, lim k →∞ A k = A by Lemma C.7, hence the 886 conclusion. 887

The rest of this proof is devoted to the construction of f k = R ˜ θ k where ˜ θ k ∈ N N are parameters 888 of a neural network of the same dimension as those in N I but only partially satisfying the support 889 constraint I . To guarantee that f k converges pointwise to f , we construct ˜ θ k based on θ k and harness 890 their relation. 891

892

893

Choice of parameters. We set ˜ θ k = { ( ˜ W k i , ˜ b k i ) 2 i =1 } ∈ N N where ˜ W k 2 ∈ R N 2 × N 1 , ˜ W k 1 ∈ R N 1 × N 0 are defined as follows, where we use C k := sup x ∈ Ω ‖ W k 1 x ‖ ∞ :

894

For inactive neurons i ∈ ¯ J , we simply set ( ˜ W k 1 [: , i ] , ˜ W k 2 [ i, :] , ˜ b k 1 [ i ]) = ( W k 1 [: , i ] , W k 2 [ i, :] , b k 1 [ i ]) .

For each equivalence class of active neurons 1 ≤ glyph[lscript] ≤ r , we choose some j glyph[lscript] ∈ J + glyph[lscript] (note that J + glyph[lscript] is non-empty due to Definition C.1) and set the parameters ( ˜ W k 2 [: , i ] , ˜ W k 1 [ i, :] , b 1 [ i ]) , i ∈ J glyph[lscript] as:

For i ∈ J glyph[lscript] \ { j glyph[lscript] } , we clearly have: supp ( ˜ W k 1 [ i, :]) ⊆ I 1 [ i, :] and supp ( ˜ W k 2 [: , i ]) ⊆ I 2 [: , i ] . The 895 j glyph[lscript] -th column of ˜ W k 2 is the only one that does not necessarily satisfy the support constraint, as 896 supp ( γ glyph[lscript] ) glyph[notsubseteql] I 2 [: , j glyph[lscript] ] in general. 897

Finally, the output bias b k 2 is set as:

Proof that f k := R ˜ θ k converges pointwise to f on Ω \ ( ∪ r glyph[lscript] =1 H glyph[lscript] ) . We introduce notations analog to Definition C.2: for every x ∈ R N 0 we define:

By construction 898

and we further explicit the form of ˜ h k i , i ∈ J glyph[lscript] for x ∈ Ω (but not on R N 0 ) as: 899

We justify our formula in Equation (34) as follow: 900

1. For i ∈ J -glyph[lscript] : since C k = sup x ∈ Ω ‖ W k 1 x ‖ ∞ by construction, ˜ W k 1 [ i, :] x + b k 1 [ i ] = W k 1 [ i, :] x + 901 b k 1 [ i ] ≥ 0 . The activation σ acts simply as an identity function. 902

For i ∈ J + glyph[lscript] : Because we choose ˜ W k 2 [: , i ] = 0 . 903

For i = j glyph[lscript] : Obvious due to the construction in Equation (31). 904

Given x ∈ Ω \ ( ∪ r glyph[lscript] =1 H glyph[lscript] ) , we now prove that this construction ensures that for each glyph[lscript] ∈ { 1 , . . . , r } 905

911

912

913

and thus

We indeed obtain (36) as follows. Since x ∈ H -glyph[lscript] , α glyph[lscript] x + β glyph[lscript] < 0 , i.e., -α glyph[lscript] x -β glyph[lscript] > 0 . Therefore, 909 given the definitions of J ± glyph[lscript] (cf Definition C.1) we have: 910

For i ∈ J -glyph[lscript] : lim k →∞ ( W k 1 [ i, :] , b k 1 [ i ]) = -( α glyph[lscript] , β glyph[lscript] ) , hence for k large enough, we have W k 1 [ i, :] x + b k 1 [ i ] > 0 so that σ ( W k 1 [ i, :] x + b k 1 [ i ]) = W k 1 [ i, :] x + b k 1 [ i ] and, as expressed in (36):

glyph[negationslash]

For i ∈ J + glyph[lscript] : similarly, we have W k 1 [ i, :] x + b k 1 [ i ] < 0 for k large enough. Therefore, h k i ( x ) = 0 for k large enough. The fact that we also have ˜ h k i ( x ) = 0 is immediate from Equation (34) if i = j glyph[lscript] , and for i = j glyph[lscript] we also get from Equation (34) that ˜ h k i ( x ) = γ glyph[lscript] σ ( α glyph[lscript] x + β glyph[lscript] ) = 0 since α glyph[lscript] x + β glyph[lscript] < 0 .

Case x ∈ H + glyph[lscript] . An analog to Equation (36) for x ∈ H + glyph[lscript] is 914

We establish it before concluding for this case. 915

For i ∈ J -glyph[lscript] : by a reasoning analog to the case x ∈ H -glyph[lscript] , we deduce that for k large enough

· For i ∈ J + glyph[lscript] : a similar reasoning yields h k i ( x ) = W k 2 [: , i ]( W k 1 [ i, :] x + b k 1 [ i ]) for k large enough, 916 while Equation (34) yields ˜ h k j glyph[lscript] ( x ) = γ glyph[lscript] σ ( α glyph[lscript] x + β glyph[lscript] ) = γ glyph[lscript] ( α glyph[lscript] x + β glyph[lscript] ) (since α glyph[lscript] x + β glyph[lscript] > 0 as x ∈ H + glyph[lscript] ) 917 and ˜ h k i ( x ) = 0 if i = j glyph[lscript] . 918

This will imply the claimed poinwise convergence since

To establish (35), observe that as x ∈ Ω \ ( ∪ r glyph[lscript] =1 H glyph[lscript] ) we have x / ∈ H glyph[lscript] . We thus distinguish two cases: 906 Case x ∈ H -glyph[lscript] . 907

Using (31) we show below that for k large enough and x ∈ H -glyph[lscript] , we have 908

glyph[negationslash]

922

923

924

925

926

927

Using (37) we obtain for k large enough

where in the last line we used the expression of ξ k glyph[lscript] from (32). Due to Equations (23) and (24) it 919 follows that lim k →∞ ˜ g k glyph[lscript] ( x ) -g k glyph[lscript] ( x ) + ξ k glyph[lscript] = 0 , ∀ x ∈ H + glyph[lscript] . 920

Thus combining both cases, we conclude that lim k →∞ ˜ g k glyph[lscript] ( x ) -g k glyph[lscript] ( x ) + ξ k glyph[lscript] = 0 , ∀ x / ∈ H glyph[lscript] , as desired. 921

Proof of the expression (30) with A k ∈ L I ′ for large enough k . From (34), we first deduce that

where we recall that S := ¯ J ∪ ( ∪ 1 ≤ glyph[lscript] ≤ r J -glyph[lscript] ) . There only remains to show that, for k large enough, we have ∑ i ∈ S ˜ h k i ( x ) = A k x + b k for every x in the restricted domain Ω ◦ δ , where A k ∈ L I ′ and b k ∈ R N 2 . Note that for i ∈ J glyph[lscript] , our construction assures that ˜ h k i is affine on Ω . Moreover, in the restricted domain Ω ◦ δ , for k ≥ κ δ large enough, ˜ h k i , i ∈ ¯ J also behave like affine functions (cf Lemma C.5). Therefore,

for some vector c k and binary scalars δ k i . In fact, δ k i = 0 if i ∈ ¯ J -:= { j ∈ ¯ J | W glyph[star] 1 [ j, :] x + b glyph[star] 1 [ j ] ≤ 0 , ∀ x ∈ Ω } and δ i k = 1 otherwise. Thus, one chooses A k = ∑ i ∈ S δ k i ˜ W k 2 [: , i ] ˜ W k 1 [ i, :] , b k = c k and the construction is complete. This construction allows us to write A k = ˆ W k 2 ˆ W k 1 with:

where diag ( { ν k i | i = 1 , . . . , N 1 } ) ∈ R N 1 × N 1 is a diagonal matrix, ν k i = δ k i for i ∈ S and 0 otherwise. It is also evident that supp ( ˆ W k 2 [: , S ]) ⊆ I 2 [: , S ] , supp ( ˆ W k 1 [ S, :]) ⊆ I 1 [ S, :] . (since the multiplication with a diagonal matrix does not increase the support of a matrix). This concludes the proof.

C.3 Proof for Corollary 4.2

Proof. The proof is inductive on the number of hidden neurons N 1 :

Basic case N 1 = 1 : Consider θ := { ( W i , b i ) 2 i =1 } ∈ N I , the function R θ has the form:

where w 1 = W 1 [1 , :] ∈ R N 0 , w 2 = W 2 [1 , 1] ∈ R . There are two possibilities: 928

(a) I 2 = ∅ : then w 2 = 0 , F I is simply a set of constant functions on Ω , which is closed. 929

930

931

932

933

934

935

936

937

938

939

940

941

942

943

944

945

946

947

950

951

952

953

954

955

(b) I 2 = { (1 , 1) } : We have I 2 = 1 1 × N 1 , which makes the first assumption of Theorem 4.2 satisfied. To check that the second assumption of Theorem 4.2 also holds, we consider all the possible non-empty subsets S of J 1 K : there is only one non-empty subset of I 2 , which is S = J 1 K . In that case, L I S = { W ∈ R 1 × N 0 | supp ( W ) ⊆ I 1 } , which is closed (since L I S is isomorphic to R | I 1 | ). The result thus follows using Theorem 4.2.

Assume the conclusion of the theorem holds for all 1 ≤ N 1 ≤ k (and any N 0 ≥ 1 ). We need to prove the result for N 1 = k +1 . Define H = { i | I 2 [1 , i ] = 1 } the set of hidden neurons that are allowed to be connected to the output via a nonzero weight. Consider two cases:

(a) If | H | ≤ k , we have F I = F I H , which is closed due to the induction hypothesis.

(b) If H = J k +1 K , we can apply Theorem 4.2. Indeed, since I 2 = 1 1 × N 1 , the first condition of Theorem 4.2 is satisfied. In addition, for any non-empty S ⊆ J N 1 K , define H := ∪ i ∈ S I [ i, :] ⊆ J N 0 K the union of row supports of I 1 [ S, :] . It is easy to verify that L I S is isomorphic to R |H| , which is closed. As such, Theorem 4.2 can be applied.

C.4 Other technical lemmas

Lemma C.7 (Convergence of affine function) . Let Ω be a non-empty interior subset R n . If the sequence { f k } k ∈ N , f k : R n ↦→ R m : x ↦→ A k x + b k where A k ∈ R m × n , b k ∈ R m converges pointwise to a function f on Ω , then f is affine (i.e., f = A x + b for some A ∈ R m × n , b ∈ R m ). Moreover, lim k →∞ A k = A and lim k →∞ b k = b .

Proof. Consider x 0 ∈ Ω ′ , an open subset of Ω ( Ω ′ exists since Ω is a non-empty interior subset of R n ). Define g k ( x ) = f k ( x ) -f k ( x 0 ) and g ( x ) = f ( x ) -g ( x 0 ) . The function g k is linear and g k converges pointwise to g on Ω (and thus, on Ω ′ ). We first prove that g is linear. Indeed, for any x, y ∈ Ω , α, β ∈ R such that αx + βy ∈ Ω , we have:

Therefore, there must exist A ∈ R m × n such that g ( x ) = A x . Choosing b := g ( x 0 ) , we have 948 f ( x ) = g ( x ) + g ( x 0 ) = A x + b . 949

Moreover, since Ω ′ is open, there exists a positive r such that the ball B ( x, r ) ⊆ Ω ′ . Choosing x i = x 0 +( r/ 2) e i with e i the i th canonical vector, we have:

or, equivalently, the i th column of A is the limit of the sequence generated by the i th column of A k . Repeating this argument for all 1 ≤ i ≤ n , we have lim k →∞ A k = A . This also implies lim k →∞ b k = b immediately.

D Closedness does not imply the best approximation property

Since we couldn't find any source discussing the fact that closedness does not imply the BAP, we provide an example to show this fact.

Consider C 0 ([ -1 , 1]) the set of continuous functions on the interval [ -1 , 1] , equipped with the norm sup ‖ f ‖ ∞ = max x ∈ [ -1 , 1] | f ( x ) | , and define S , the subset of all functions f ∈ C 0 ([ -1 , 1]) such that:

It is easy to verify that S is closed. We show that the constant function f = 0 does not have a 956 projection in S (i.e., a function g ∈ S such that ‖ f -g ‖ ∞ = inf h ∈ S ‖ f -h ‖ ∞ ). 957

First we observe that since f = 0 , we have ‖ f -h ‖ ∞ = ‖ h ‖ ∞ for each h ∈ S , and we show that 958 inf h ∈ S ‖ f -h ‖ ∞ ≥ 1 / 2 . Indeed, for h ∈ S we have: 959

Secondly, we show a sequence of { h n } k ∈ N such that h n ∈ S and lim n →∞ ‖ h n ‖ ∞ = 1 / 2 . Consider the odd function h n (i.e. h n ( x ) = -h n ( -x ) ) such that:

where c n = n/ (2 n -1) . It is evident that h n ∈ S because:

Moreover, we also have lim n →∞ ‖ h n ‖ ∞ = lim n →∞ c n = 1 / 2 .

960

Finally, we show that 1 / 2 cannot be attained. By contradiction, assume that there exists g ∈ S such 961 that ‖ f -g ‖ ∞ = 1 / 2 , i.e., as we have seen, ‖ g ‖ ∞ = 1 / 2 . Using Equation (38), the equality will 962 only hold if g ( x ) = 1 / 2 in [0 , 1] and g ( x ) = -1 / 2 in [ -1 , 0] . However, g is not continuous, a 963 contradiction. 964

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors are interested in the following question : « Given a deep learning architecture possibly with sparsity constraints, does its corresponding optimisation problem actually admit an optimal $\\theta^*$? » While it is mostly taken for granted that the answer is « yes », the author proves it to be true in some cases, and false in others. They derive an algorithm in order to verify a sufficient condition in order for there to exist. More then a single way the sparseness of the considered network in ensured is considered in the article.\n\nSTRENGTHS:\n1. The relevance of the work is well defended in the introduction.\n\n2. The results concern a large pool of predictors, considering a regression task.\n\n3. The results concern more than one way of ensuring networks to be sparse.\n\n4. The results seem to be a significant add-on to the literature concerning closeness and non-closeness of problems involving the training of (sometimes sparse) neural networks.\n\n5. Example 3.1., though having some flaws (see Weaknesses - Major - 1.2.), is really important for illustrating what might otherwise be a bit hard to grasp for the reader, and proving the issues that might be encountered when not considering the initial question : « Given a deep learning architecture possibly with sparsity constraints, does its corresponding optimisation problem actually admit an optimal $\\theta^*$? ».\n\nWEAKNESSES:\n_Major_\n\n1.1. Regularisation is indeed (lines 41-42) a common way to bypass this question, but the notion of regularisation in itself is, especially when it comes to neural networks, mostly important for limiting over-fitting. I didn’t find the arguments regarding the discarding of regularisation for neural networks training really compelling (lines 44-47), especially since it wasn’t supported by any work from the literature.\n\n1.2. Concerning Example 3.1 and the discussion that follows (lines 216-221), the discussion is based on what's displayed on Figure 1 only, forgetting about validation / test loss; saying that $L^2$ regularisation « might be detrimental » thus forget about validation / test loss. And if the goal is simply to over-fit a problem, then even smallest regularisation would make the parameter converge to certain values, practically not affecting the obtained error in train, which probably wasn't the case, since a single (and arbitrary) regularisation parameter was applied for each training.\n\n2. The assumptions required for having Proposition 3.1. to hold (continuous, coercive) are limiting, making the use of 0-1 loss impossible (and cross-entropy as well, if I’m correct). The way the networks are defined too, since no soft-max activation function can be used on the output layer. Those loss functions (and the soft-max) being crucial when it comes to classifications, the results are mostly relevant only for regression, which is really not the most popular type of task when it comes to neural networks. Though it has been briefly mentioned on lines 176-177, I feel like the authors were not honest enough regarding how limiting the assumptions and the way the networks are defined are. For example, a sentence such as « Though our study theoretically applies to classification schemes, it more naturally suites regression schemes, considering the assumptions made in [...] » in the abstract, or at least in the Introduction section would be sufficient.\n\n3. The motivations of the work are partially theoretical and mostly practical, but the results, though interesting, for it proves not every network has an optimal set of parameters in different settings, are mostly inapplicable in practice, limiting the impact of the work.\n\n4. The terminology in Table 1 might be misleading, since the term « shallow » is vague and that when it comes to Theorem 3.4 and Corollaries 3.1, 4.1, 4.2 and 4.3, « shallow » stands for « 1-hidden-layer ». What are the limitations for the works from the literature with regards to the depth of the networks? If it is 1-hidden-layer, then the whole table should refer to « 1-hidden-layers architecture », and not « shallow architecture »; if it is deeper than 1-hidden-layer in at least one case, than the underlying maximum depth should be displayed in the table.\n\n\n\n_Typos / Minor_\n\n1. Line 126 : « agrees with $\\mathbf{A}$ on rows in $S_{\\underline{r}}$ (resp. columns in in $S_{\\underline{c}}$) »\n\n2. I expect Lemma 3.2. of the main paper to be coherent with what is stated on lines 505-506 for a camera-ready version of the article.\n\n3. Lines 259 and 260 : The statement « This constraint on the sparsity level of each 260 layer is widely used in many works on sparse neural networks » is not supported by any citation.\n\n4. Line 261 : « Consider scalar-val\\underline{u}ed ».\n\n5. Line 359 : « cf\\underline{.} »\n\n6. Using $\\Omega$ for both the finite and the non-close cases is a bit confusing; it might be more convenient to differentiate these two cases with a separate notation.\n\nQUESTIONS:\n1. If the practical interest in studying problems with finite $\\Omega$ is clear, I have a hard time understanding the interest of studying non-closed $\\Omega$ and how it applies to real-world situations.\n\nLIMITATIONS:\nI feel like the authors downplayed the importance of some limitations of their work (see Weaknesses – Major).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-22"}

comments: SUMMARY:
This work studies the existence of global minimum in a sparse deep learning setting

STRENGTHS:
I think this work touches on a very important problem that is understudied in the field: the existence of the global minimum for deep nonlinear networks. It is easy to image toy examples for which the global minimum does not exist, but until this work, there does not exist a formal study of this problem

The problem is important and novel. Linking this problem to training a sparse network is novel, I thus support its publication, though with some reservation

WEAKNESSES:
To me, I think the main problem is that the results are quite weak, and that the main example is not quite convincing or relevant. The example of a LU network is quite artificial and it is hard to imagine that this situation arises in deep learning. I think this work can benefit greatly if it identifies a more relevant and convincing example (and it should not be difficult to achieve). This is the main reason I give this work a weak accept

There is one minor problem that I do not find serious but is worth some attention. While the paper mostly motivates from the viewpoint of the popular pruning literature, this work can benefit from discussing more about regularization-based methods for compressing neural networks. For example, see the recent work in https://arxiv.org/abs/2210.01212, which also discusses the existence of the global minimum. At its face value, the results in this work seems to motivate and advocate the use of regularization-based compression methods in deep learning; is this interpretation correct? If not, why? I think discussing this point can better clarify the implication of this work

QUESTIONS:
See weakness

LIMITATIONS:
I think the work discussed the limitations well
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides necessary and sufficient conditions for whether a neural network training problem admits an optimal solution, that is, whether weights and biases exist such that the infimum of the loss function is actually attained. This is done for the classical case of empirical risk minimization on a finite training set as well as for function approximation on a continuous input domain.\n\nSTRENGTHS:\n- I think the investigated question is highly interesting already for purely theoretical reasons: the existence of an optimal solution is such a fundamental property of an optimization problem. We should be able to answer this question for our \"favorite\" optimization problem, namely training a neural network. This work seems to play a crucial role in (i) posing this question and (ii) providing results for some cases.\n- Apparently the non-existence of an optimal solution can also explain divergence in practical settings, which I find compelling. Still, I would like to emphasize that the paper is a theory paper and should be judged as such.\n- The paper involves quite some amount of non-trivial mathematics. Although the tight reviewing schedule does not allow me to verify all details, the parts I read seem to be mathematically sound.\n\nWEAKNESSES:\n- Many cases for the posed question remain open. In particular, most of the results are concerned with 2-layer NNs only.\n- The presentation of the results in the intro could be improved. Some definitions could be made earlier (or at all). See my detailed comments below. Also I find it quite difficult to read the Tables 1 and 2. One basically has to read the rest of the paper in order to truely understand what these tables are about.\n\nComments for the authors to improve the paper (not meant as true weaknesses):\n\n- title: remove the space between \"optimum\" and the question mark\n- line 54: \"the best approximation property (BAP), which guarantees the existence of an optimal solution\" -> Is this the definition of BAP or just a consequence? Please make this clear. You might want to define BAP in a proper definition environment.\n- lines 57-79: for readers unfamiliar with sparsity in neural networks, it is quite hard to understand your contributions without reading the \"notations\" part later. You should spend some efforts in explaining terms like \"(structured) sparse networks\", \"fixed sparsity level\" vs. \"fixed sparsity pattern\", etc. already in the introduction. I think \"sparsity level\" is only implicitly defined very late in the paper.\n- line 67: Here (and at some other places in the paper) you use the term \"learning problem\" as a synonym to \"training problem\". Some people understand \"learning problem\" as the problem to minimize the generalization error, as opposed to the \"training problem\", which only aims to minimize the training error. At least in the finite domain case you are definitely in the latter regime, so I suggest to use \"training problem\" consistently.\n- Tables 1 and 2: the relation to the sparsity constraints is not really clear in the tables. In particular, what does the \"sparse\" adjective in brackets in the architecture column mean? Why is it in brackets?\n- caption of table 2: there is an extra space after the opening bracket\n- line 124: it is a bit weird that the \"notations\" section is part of the \"related work\" section.\n- line 195: contains -> contain\n- line 261: valed -> valued\n- line 293: a bit redundant \"many other domains such as [...] and much more\".\n- line 301: \"(in the whole paper we naturally assume B > 0)\" -> state this earlier in the paper when you use B for the first time.\n- line 303: provide a reference to the proof of this theorem in the appendix.\n- line 312: what you call \"plain\" here is what I would call \"fully connected\". You should maybe use this term here and elsewhere in the paper.\n\nQUESTIONS:\n- Is there any chance to obtain a hardness result for the computational problem considered in Section 3.3? NP-hardness? Hardness for the existential theory of the reals (which is always a good candidate if the best known solution uses quantifier elimination)? See related results for NN training: https://arxiv.org/abs/2102.09798; https://arxiv.org/abs/2204.01368.\n- Pushing the relation to the existential theory of the reals even further, I find it very interesting that you obtain such a sharp difference between the scalar-valued output case and the multi-dimensional output case (Section 3.4). The same seems to be true for the training complexity of such shallow networks: while scalar-output networks can be trained with a combinatorial search algorithm, certifying that the problem is in NP, it turns out that training a shallow network with multi-dimensional output is ER-complete and therefore much harder (https://arxiv.org/abs/2204.01368). Again I wonder: is there any relation to your results?\n\nLIMITATIONS:\nThe paper properly states under which circumstances (mathematical assumptions) the results are valid.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the existence of an optimal solution to the objective of training a ReLU neural network under certain sparsity patterns. They consider two topological properties of the input function space (the best approximation property, and closedness). They provided series of results on scalar-valued neural networks, shallow neural networks, neural network with one hidden layer. Specifically, they provide necessary and sufficient conditions on the sparsity (e.g., fixed sparsity level or pattern) of the neural network to guarantee existence of an optimal solution to the training loss function.\n\nSTRENGTHS:\nThe paper studies an interesting problem which relates to the stability of the neural network training due to the non-existence of an optimal solution given a sparsity pattern. The paper covers a good literature review and stand themselves very well compared to prior works. Their contribution is clear.\n\nThe title of the paper is very related to the goal of the paper. The paper is coherent in studying the problem of interest.\n\nThe paper is written clearly. Each theorem/proposition is followed by lemma/corollary and informal proof sketch which makes it very easy to follow. For examples, the paper contains a numerical example very early on in the paper to further give intuition on the importance of the problem and a plausible scenario where an optimal solution does not exist.\n\nThe results provide new insights on if a neural network will have an optimal solution. Table 1 and 2 clearly state how their work differs from prior work.\n\nWEAKNESSES:\nIt would be nice in the main paper to provide more tangible examples on what sparsity examples on what architectures may/may not result in existence of an optimal solution. NeurIPS has a broad range of audiences from theory and application (addition of this can attract more application-based deep learning researchers).\n\n\n- minor: corollary 3.1, valed --> valued\n\nQUESTIONS:\n- For corollary 3.1, what is the dimension of the hidden layer?\n\n- Could you clarify in simple words the term number 2 on line 305 (addition of this will improve clarity).\n\nLIMITATIONS:\nAddressed properly.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}