SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Quoc-Tung Le
Elisa Riccietti
Univ. Lyon, Inria, CNRS, ENS de Lyon, UCB Lyon 1, LIP UMR 5668, F-69342 Lyon, France
Given a training set, a loss function, and a neural network architecture, it is often taken for granted that optimal network parameters exist, and a common practice is to apply available optimization algorithms to search for them. In this work, we show that the existence of an optimal solution is not always guaranteed, especially in the context of sparse ReLU neural networks. In particular, we first show that optimization problems involving deep networks with certain sparsity patterns do not always have optimal parameters, and that optimization algorithms may then diverge. Via a new topological relation between sparse ReLU neural networks and their linear counterparts, we derive –using existing tools from real algebraic geometry– an algorithm to verify that a given sparsity pattern suffers from this issue. Then, the existence of a global optimum is proved for every concrete optimization problem involving a one-hidden-layer sparse ReLU neural network of output dimension one. Overall, the analysis is based on the investigation of two topological properties of the space of functions implementable as sparse ReLU neural networks: a best approximation property, and a closedness property, both in the uniform norm. This is studied both for (finite) domains corresponding to practical training on finite training sets, and for more general domains such as the unit cube. This allows us to provide conditions for the guaranteed existence of an optimum given a sparsity pattern. The results apply not only to several sparsity patterns proposed in recent works on network pruning/sparsification, but also to classical dense neural networks, including architectures not covered by existing results.
The optimization phase in deep learning consists in minimizing an objective function w.r.t. the set of parameters  $\theta$  of a neural network (NN). While it is arguably sufficient for optimization algorithms to find local minima in practice, training is also expected to achieve the infimum in many situations (for example, in overparameterized regimes networks are trained to zero learning error).
In this work, we take a step back and study a rather fundamental question: Given a deep learning architecture possibly with sparsity constraints, does its corresponding optimization problem actually admit an optimal  $\theta^{*}$ ? The question is important for at least two reasons:
1. Practical viewpoint: If the problem does not admit an optimal solution, optimized parameters necessarily diverge to infinity to approximate the infimum (which always exists). This phenomenon has been studied thoroughly in previous works in other contexts such as tensor decomposition [7], robust principal component analysis [30], sparse matrix factorization [18] and also deep learning itself [26, 23, 12]. It causes inherent numerical instability for optimization algorithms. Moreover,
the answer to this question depends on the architecture of the neural networks (specified by the number of layers, layers width, activation function, and so forth). A response to this question might suggest a guideline for model and architecture selection.
2. Theoretical viewpoint: the existence of optimal solutions is crucial for the analysis of algorithms and their properties (for example, the properties of convergence, or the characterization of properties of the optimum, related to the notion of implicit bias).
One usual practical (and also theoretical) trick to bypass the question of the existence of optimal solutions is to add a regularization term, which is usually coercive, e.g., the  $L^2$  norm of the parameters. The existence of optimal solutions then follows by a classical argument on the extrema of a continuous function in a compact domain. Nevertheless, there are many settings where minimizing the regularized version might result in a high value of the loss since the algorithm has to make a trade-off between the loss and the regularizer. Such a scenario is discussed in Example 3.1. Therefore, studying the existence of optimal solutions without (explicit) regularization is also a question of interest.
Given a training set  $\{(x_i, y_i)\}_{i=1}^P$ , the problem of the existence of optimal solutions can be studied from the point of view of the set of functions implementable by the considered network architecture on the finite input domain  $\Omega = \{x_i\}_{i=1}^P$ . This is the case since the loss is usually of the form  $\ell(f_{\theta}(x_i), y_i)$  where  $f_{\theta}$  is the realization of the neural network with parameters  $\theta$ . Therefore, the loss involves directly the image of  $\{x_i\}_{i=1}^P$  under the function  $f_{\theta}$ . For theoretical purposes, we also study the function space on the domain  $\Omega = [-B, B]^d$ ,  $B > 0$ . In particular, we investigate two topological properties of these function spaces, both w.r.t. the infinity norm  $\|\cdot\|_{\infty}$ : the best approximation property (BAP), i.e., the guaranteed existence of an optimal solution  $\theta^*$ , and the closedness, a necessary property for the BAP. These properties are studied in Section 3 and Section 4, respectively. Most of our analysis is dedicated to the case of regression problems. We do make some links to the case of classification problems in Section 3.
We particularly focus on analyzing the function space associated with (structured) sparse ReLU neural networks, which is motivated by recent advances in machine learning witnessing a compelling empirical success of sparsity based methods in NNs and deep learning techniques, such as pruning [32, 14], sparse designed NN [4, 5], or the lottery ticket hypothesis [10] to name a few. Our approach exploits the notion of networks either with fixed sparsity level or with fixed sparsity pattern (or support). This allows us to establish results covering both classical NNs (whose weights are not constrained to be sparse) and sparse NNs architectures. Our main contributions are:
1. To study the BAP (i.e., the existence of optimal solutions) in practical problems (finite  $\Omega$ ): we provide a necessary condition and a sufficient one on the architecture (embodied by a sparsity pattern) to guarantee such existence. As a particular consequence of our results, we show that: a) for one-hidden-layer NNs with a fixed sparsity level, the training problem on a finite data set always admits an optimal solution (cf. Theorem 3.4 and Corollary 3.1); b) however, practitioners should be cautious since there also exist fixed sparsity patterns that do not guarantee the existence of optimal solutions (cf. Theorem 3.1 and Example 3.1). In the context of an emerging emphasis on structured sparsity (e.g. for GPU-friendliness), this highlights the importance of choosing adequate sparsity patterns.
2. To study the closedness of the function space on  $\Omega = [-B, B]^d$ . As in the finite case, we provide a necessary condition and a sufficient one for the closedness of the function space of ReLU NNs with a fixed sparsity pattern. In particular, our sufficient condition on one-hidden-layer networks generalizes the closedness results of [26, Theorem 3.8] on "dense" one-hidden-layer ReLU NNs to the case of sparse ones, either with fixed sparsity pattern (cf. Theorem 4.2, Corollary 4.1 and Corollary 4.2) or fixed sparsity level (Corollary 4.3). Moreover, our necessary condition (Theorem 4.1), which is also applicable to deep architectures, exhibits sparsity structures failing the closedness property.
Table 1 and Table 2 summarize our results and their positioning with respect to existing ones. Somewhat surprisingly, the necessary conditions in both domains  $(\Omega$  finite and  $\Omega = [-B,B]^d)$  are identical. Our necessary/sufficient conditions also suggest a relation between sparse ReLU neural networks and their linear counterparts.
The rest of this paper is organized as follows: Section 2 discusses related works and introduces notations; the two technical sections, Section 3 and Section 4, presents the results for the case  $\Omega$  finite set and  $\Omega = [-B,B]^d$  respectively.
36   
37   
38   
39   
40   
41   
42   
43   
44   
45   
46   
47   
48   
49   
50   
51   
52   
53   
54   
55   
56   
57   
58   
59   
60   
61   
62   
63   
64   
65   
66   
67   
68   
69   
70   
71   
72   
73   
74   
75   
76   
77   
78   
79   
80   
81   
82   
83   
84   
85   
86   
87   
88   
89   
90
Works	Architecture	Activation functions	Ω	Function space	BAP
Theorem 3.4 Corollary 3.1	Sparse feed-forward network	ReLU	finite set	(RP×do, ||·||), arbitrary ||·||	✓
[16][15]	Feed-forward network	Heaviside	[0,1]d	(Lp(Ω), ||·||LP), ∀p ∈ [1,∞)	✓
[9][8]°	Feed-forward network, Residual feed-forward network	ReLU	Rd	(Lp(Ω), ||·||LP), p = 2, μ is a measure with compact support and is continuous w.r.t Lebesgue measure	✓ (if the target function is continuous)
[26]	Feedforward network	ReLU, pReLU	[-B, B]d	(C0(Ω), ||·||∞)	×
Corollary 4.1†	Feed-forward network	ReLU	[-B, B]d	(C0(Ω), ||·||∞)	×
Corollary 4.2 Corollary 4.3	Sparse feed-forward network	ReLU	[-B, B]d	(C0(Ω), ||·||∞)	×
Table 1: Closedness results. All results are established for one-hidden-layer architectures with scalar-valued output, except  $\dagger$  (which is valid for one-hidden-layer architectures with vector-valued output). In  $\diamond$ , if the architecture is simply a feed-forward network, then the result is valid for any  $p > 1$ .
Works	Architecture	Activation functions	Function space	Assumptions are valid for any ...
L	NL-1	NL
[12]	Feedforward network	Sigmoid	(C0(Ω), ||·||∞)	× (L=2)	× (NL-1≥2)	× (NL=1)
[20]°	Feedforward network	ReLU	(RNL×P, ||·||), P=6	× (L=2)	× (NL-1=2)	× (NL=2)
[26]	Feedforward network	sigmoid, tanh, arctan, ISRLU, ISRU	(C0(Ω), ||·||∞)	✓	× (NL-1≥2)	× (NL=1)
sigmoid, tanh, arctan, ISRLU, ISRU, ReLU, pReLU	(Lp(Ω), ||·||LP)
[23]	Feedforward network	ELU, softsign	(W1,p(Ω), ||·||LP) ∀p∈[1,∞]
ISRU	(W2,p(Ω), ||·||LP) ∀p∈[1,∞]
ISRU, sigmoid, tanh, arctan	(WK,p(Ω), ||·||LP) ∀k, ∀p∈[1,∞]
Theorem 4.1‡	Sparse feedforward network	ReLU	(C0(Ω), ||·||∞)	✓	✓	✓
Theorem 3.1°	Sparse feedforward network	ReLU	(RNL×P, ||·||)	✓	✓	✓
Table 2: Non-closedness results (notations in Section 2). Previous results consider  $\Omega = [-B,B]^d$ ; ours cover:  $\diamond$  a finite  $\Omega$  with  $P$  points;  $\ddagger$  a bounded  $\Omega$  with non-empty interior (this includes  $\Omega = [-B,B]^d$ ).
The fact that optimal solutions may not exist in tensor decomposition problems is well-documented [7]. The cause of this phenomenon (also referred to as ill-posedness [7]) is the non-closedness of the set of tensors of order at least three and of rank at least two. Similar phenomena were shown to happen in various settings such as matrix completion [11, Example 2], robust principal component analysis [30] and sparse matrix factorization [18, Remark A.1]. Our work indeed establishes bridges between the phenomenon on sparse matrix factorization [18] and on sparse ReLU NNs.
There is also an active line of research on the best approximation property and closedness of function spaces of neural networks. Existing results can be classified into two categories: negative results, which demonstrate the non-closedness and positive results for those showing the closedness or best approximation property of function spaces of NNs. Negative results can notably be found in
91   
92   
93   
94   
95   
96   
97   
98   
99   
100   
101
[12, 26, 23], showing that the set of functions implemented as conventional multilayer perceptrons with various activation functions such as Inverse Square Root Linear Unit (ISRLU), Inverse Square Root Unit (ISRU), parametric ReLU (pReLU), Exponential Linear Unit (ELU) [26, Table 1] is not a closed subset of classical function spaces (e.g., the Lebesgue spaces  $L^p$ , the set of continuous functions  $C^0$  equipped with the sup-norm, or Sobolev spaces  $W^{k,p}$ ). In a more practical setting, [20] hand-crafts a dataset of six points which makes the training problem of a dense one-hidden-layer neural network not admit any solution. Positive results are proved in [26, 16, 15], which establish both the closedness and/or the BAP. The BAP implies closedness [12, Proposition 3.1][26, Section 3] (but the converse is not true, see Appendix D) hence the BAP can be more difficult to prove than closedness. So far, the only architecture proved to admit the best approximation property (and thus, also closedness) is one-hidden-layer neural networks with heavyside activation function and scalar-valued output (i.e., output dimension equal to one) [15] in  $L^p(\Omega), \forall p \in [1,\infty]$ . If one allows additional assumptions such as the target function  $f$  being continuous, then BAP is also established for one-hidden layer and residual one-hidden-layer NNs with ReLU activation function [9, 8]. In all other settings, to the best of our knowledge, the only property proved in the literature is closedness, but the BAP remains elusive. We compare our results with existing works in Tables 1 and 2.
In machine learning, there is an ongoing endeavour to explore sparse deep neural networks, as a prominent approach to reduce memory and computation overheads inherent in deep learning. One of its most well-known methods is Iterative Magnitude Pruning (IMP), which iteratively trains and prunes connections/neurons to achieve a certain level of sparsity. This method is employed in various works [14, 32], and is related to the so-called Lottery Ticket Hypothesis (LTH) [10]. The main issue of IMP is its running time: one typically needs to perform many steps of pruning and retraining to achieve a good trade-off between sparsity and performance. To address this issue, many works attempt to identify the sparsity patterns of the network before training. Once they are found, it is sufficient to train the sparse neural networks once. These pre-trained sparsity patterns can be found through algorithms [29, 31, 19] or leveraging the sparse structure of well-known fast linear operators such as the Discrete Fourier Transform [5, 4, 21, 6, 3]. Regardless of the approaches, these methods are bound to train a neural network with fixed sparsity pattern at some points. This is a particular motivation for our work and our study on the best approximation property of sparse ReLU neural networks with fixed sparsity pattern.
Notations In this work,  $\llbracket n\rrbracket \coloneqq \{1,\dots ,n\}$ . For a matrix  $\mathbf{A}\in \mathbb{R}^{m\times n}$ ,  $\mathbf{A}[i,j]$  denotes the coefficient at the index  $(i,j)$ ; for subsets  $S_{r}\subseteq \llbracket m\rrbracket$ ,  $S_{c}\subseteq \llbracket n\rrbracket$ ,  $\mathbf{A}[S_r,:]$  (resp.  $\mathbf{A}[:,S_c]$ ) is a matrix of the same size as  $\mathbf{A}$  and agrees with  $\mathbf{A}$  on rows in  $S_{r}$  (resp. columns in  $S_{c}$ ) of  $\mathbf{A}$  while its remaining coefficients are zero. The operator  $\mathrm{supp}(\mathbf{A})\coloneqq \{(\ell ,k)\mid \mathbf{A}[\ell ,k]\neq 0\}$  returns the support of the matrix  $\mathbf{A}$ . We denote  $\mathbf{1}_{m\times n}$  (resp.  $\mathbf{0}_{m\times n}$ ) an all-one (resp. all-zero) matrix of size  $m\times n$ .
An architecture with fixed sparsity pattern is specified via  $\mathbf{I} = (I_L, \ldots, I_1)$ , a collection of binary masks  $I_i \in \{0, 1\}^{N_i \times N_{i-1}}$ ,  $1 \leq i \leq L$ , where the tuple  $(N_L, \ldots, N_0)$  denotes the dimensions of the input layer  $N_0 = d$ , hidden layers  $(N_{L-1}, \ldots, N_1)$  and output layer  $(N_L)$ , respectively. The binary mask  $I_i$  encodes the support constraints on the  $i$ th weight matrix  $\mathbf{W}_i$ , i.e.,  $I_i[\ell, k] = 0$  implies  $\mathbf{W}_i[\ell, k] = 0$ . It is also convenient to think of  $I_i$  as the set  $\{(\ell, k) \mid I_i[\ell, k] = 1\}$ , a subset of  $\llbracket N_i\rrbracket \times \llbracket N_{i-1}\rrbracket$ . We will use these two interpretations (binary mask and subset) interchangeably and the meaning should be clear from context. We will even abuse notations by denoting  $I_l \subseteq \mathbf{1}_{N_l \times N_{l-1}}$ . Because the support constraint  $I$  can be thought as a binary matrix, the notation  $I[S_r, :]$  (resp.  $I[:, S_c]$ ) represents the support constraint of  $I \cap S_r \times [n]$  (resp.  $I \cap [n] \times S_c$ ).
The space of parameters on the sparse architecture  $\mathbf{I}$  is denoted  $\mathcal{N}_{\mathbf{I}}$ , and for each  $\theta \in \mathcal{N}_{\mathbf{I}}$ ,  $\mathcal{R}_{\theta} : \mathbb{R}^{N_0} \mapsto \mathbb{R}^{N_L}$  is the function implemented by the ReLU network with parameter  $\theta$ :
$$
\mathcal {R} _ {\theta}: x \in \mathbb {R} ^ {N _ {0}} \mapsto \mathcal {R} _ {\theta} (x) := \mathbf {W} _ {L} \sigma (\dots \sigma (\mathbf {W} _ {1} x + \mathbf {b} _ {1}) \dots + \mathbf {b} _ {L - 1}) + \mathbf {b} _ {L} \in \mathbb {R} ^ {N _ {L}} \tag {1}
$$
where  $\sigma (x) = \max (0,x)$  is the ReLU activation.
Finally, for a given architecture  $\mathbf{I}$ , we define
$$
\mathcal {L} _ {\mathbf {I}} = \left\{\mathbf {X} _ {L} \dots \mathbf {X} _ {1} \mid \operatorname {s u p p} \left(\mathbf {X} _ {i}\right) \subseteq I _ {i}, i \in [   [ L ]   ] \right\} \subseteq \mathbb {R} ^ {N _ {L} \times N _ {0}} \tag {2}
$$
the set of matrices factorized into  $L$  factors respecting the support constraints  $I_{i}, i \in [[L]]$ . In fact,  $\mathcal{L}_{\mathbf{I}}$  is the set of linear operators implementable as linear neural networks (i.e., with  $\sigma = \mathrm{id}$  instead of the ReLU in (1), and no biases) with parameters  $\theta \in \mathcal{N}_{\mathbf{I}}$ .
102   
103   
104   
105   
106   
107   
108   
109   
110   
111   
112   
113   
114   
115   
116   
117   
118   
119   
120   
121   
122   
123   
124   
125   
126   
127   
128   
129   
130   
131   
132   
133   
134   
135   
136   
137   
138   
139   
140   
141   
142   
143   
144   
145   
146   
147
3 Analysis of fixed support ReLU neural networks for finite  $\Omega$
The setting of a finite set  $\Omega = \{x_{i}\}_{i = 1}^{P}$  is common in many practical machine learning tasks: models such as (sparse) neural networks are trained on often large (but finite) annotated dataset  $\mathcal{D} = \{(x_i,y_i)\}_{i = 1}^P$ . The optimization/training problem usually takes the form:
$$
\underset {\theta} {\text {M i n i m i z e}} \quad \mathcal {L} (\theta) = \sum_ {i = 1} ^ {P} \ell \left(\mathcal {R} _ {\theta} \left(x _ {i}\right), y _ {i}\right), \quad \text {u n d e r s p a r s i t y c o n s t r a i n t s o n} \theta \tag {3}
$$
where  $\ell$  is a loss function measuring the similarity between  $\mathcal{R}_{\theta}(x_i)$  and  $y_{i}$ . A natural question that we would like to address for this task is:
Question 3.1. Under which conditions on  $\mathbf{I}$ , the prescribed sparsity pattern for  $\theta$ , does the training problem of sparse neural networks admit an optimal solution for any finite data set  $\mathcal{D}$ ?
We investigate this question both for parameters  $\theta$  constrained to satisfy a fixed sparsity pattern  $\mathbf{I}$ , and in the case of a fixed sparsity level, see e.g. Corollary 4.3.
After showing in Section 3.1 that the answer to Question 3.1 is intimately connected with the closedness of the function space of neural networks with architecture  $\mathbf{I}$ , we establish in Section 3.2 that this closedness implies the closedness of the matrix set  $\mathcal{L}_{\mathbf{I}}$  (a property that can be checked using algorithms from real algebraic geometry, see Section 3.3). We also provide concrete examples of support patterns  $\mathbf{I}$  where closedness provably fails, and neural network training can diverge. Section 3.4 presents sufficient conditions for closedness that enable us to show that an optimal solution always exists on scalar-valued one-hidden-layer networks under a constraint on the sparsity level of each layer.
3.1 Equivalence between closedness and best approximation property
To answer Question 3.1, it is convenient to view  $\Omega$  as the matrix  $[x_1,\dots ,x_P]\in \mathbb{R}^{d\times P}$  and to consider the function space implemented by neural networks with the given architecture  $\mathbf{I}$  on the input domain  $\Omega$  in dimension  $d = N_{0}$ , with output dimension  $N_{L}$ , defined as the set
$$
\mathcal {F} _ {\mathbf {I}} (\Omega) := \left\{\mathcal {R} _ {\theta} (\Omega) \mid \theta \in \mathcal {N} _ {\mathbf {I}} \right\} \subseteq \mathbb {R} ^ {N _ {L} \times P} \tag {4}
$$
where the matrix  $\mathcal{R}_{\theta}(\Omega) \coloneqq \left[\mathcal{R}_{\theta}(x_1), \ldots, \mathcal{R}_{\theta}(x_P)\right] \in \mathbb{R}^{N_L \times P}$  is the image under  $\mathcal{R}_{\theta}$  of  $\Omega$ .
We study the closedness of  $\mathcal{F}_{\mathbf{I}}(\Omega)$  under the usual topology induced by any norm  $\| \cdot \|$  of  $\mathbb{R}^{N_L\times P}$ . This property is interesting because if  $\mathcal{F}_{\mathbf{I}}(\Omega)$  is closed for any  $\Omega = \{x_{i}\}_{i = 1}^{P}$ , then an optimal solution is guaranteed to exist for any  $\mathcal{D}$  under classical assumptions of  $\ell (\cdot ,\cdot)$ . The following result is not difficult to prove, we nevertheless provide a proof in Appendix B.1 for completeness.
Proposition 3.1. Assume that, for any fixed  $y \in \mathbb{R}^{N_L}$ ,  $\ell(\cdot, y): \mathbb{R}^{N_L} \mapsto \mathbb{R}$  is continuous, coercive and that  $y = \arg \min_{y'} \ell(y', y)$ . For any sparsity pattern  $\mathbf{I}$  with input dimension  $N_0 = d$  the following properties are equivalent:
1. irrespective of the training set, problem (3) under the constraint  $\theta \in \mathcal{N}_{\mathbf{I}}$  has an optimal solution;  
2. for every  $P$  and every  $\Omega \in \mathbb{R}^{d\times P}$ , the function space  $\mathcal{F}_{\mathbf{I}}(\Omega)$  is a closed subspace of  $\mathbb{R}^{N_L\times P}$ .
The assumption on  $\ell$  is natural and realistic in regression problems: any loss function based on any norm on  $\mathbb{R}^d$  (e.g.  $\ell(y', y) = \|y' - y\|$ ), such as the quadratic loss, satisfies this assumption. In the classification case, using the soft-max after the last layer together with the cross-entropy loss function indeed leads to an optimization problem with no optimum (regardless of the architecture) when given a single training pair. This is due to the fact that changing either the bias or the scales of the last layer can lead the output of the soft-max arbitrarily close to an ideal Dirac mass. It is an interesting challenge to identify whether sufficiently many and diverse training samples (as in concrete learning scenarios) make the problem better posed, and amenable to a relevant closedness analysis.
In light of Proposition 3.1 we investigate next the closedness of  $\mathcal{F}_{\mathbf{I}}(\Omega)$  for finite  $\Omega$ .
3.2 A necessary closedness condition for fixed support ReLU networks
Our next result reveals connections between the closedness of  $\mathcal{F}_{\mathbf{I}}(\Omega)$  for finite  $\Omega$  and the closedness of  $\mathcal{L}_{\mathbf{I}}$ , the space of sparse matrix products with sparsity pattern  $\mathbf{I}$ .
153   
154   
155   
156   
157   
158   
159   
160   
161   
162   
163   
164   
165   
166   
167   
168   
169   
170   
171   
172   
173   
174   
175   
176   
177   
178   
179   
180   
181   
182   
183   
184   
185   
186   
187   
188   
189   
190   
191   
192   
193   
194   
195   
196
Theorem 3.1. If  $\mathcal{F}_{\mathbf{I}}(\Omega)$  is closed for every finite  $\Omega$  then  $\mathcal{L}_{\mathbf{I}}$  is closed.
Theorem 3.1 is a direct consequence of (and in fact logically equivalent to) the following lemma:
Lemma 3.2. If  $\mathcal{L}_{\mathbf{I}}$  is not closed then there exists a set  $\Omega \subset \mathbb{R}^d$ ,  $d = N_0$ , of cardinality at most  $P \coloneqq (3N_04^{\sum_{i=1}^{L-1}N_i} + 1)^{N_0}$  such that  $\mathcal{F}_{\mathbf{I}}(\Omega)$  is not closed.