SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Etienne Boursier  
INRIA CELESTE, LMO, Orsay, France  
etienne.boursier@inria.fr
Nicolas Flammarion  
TML Lab, EPFL, Switzerland  
nicolas.flammarion@epfl.ch
Controlling the parameters' norm often yields good generalisation when training neural networks. Beyond simple intuitions, the relation between regularising parameters' norm and obtained estimators remains theoretically misunderstood. For one hidden ReLU layer networks with unidimensional data, this work shows the parameters' norm required to represent a function is given by the total variation of its second derivative, weighted by a  $\sqrt{1 + x^2}$  factor. Notably, this weighting factor disappears when the norm of bias terms is not regularised. The presence of this additional weighting factor is of utmost significance as it is shown to enforce the uniqueness and sparsity (in the number of kinks) of the minimal norm interpolator. Conversely, omitting the bias' norm allows for non-sparse solutions. Penalising the bias terms in the regularisation, either explicitly or implicitly, thus leads to sparse estimators.
Although modern neural networks are not particularly limited in terms of their number of parameters, they still demonstrate remarkable generalisation capabilities when applied to real-world data [Belkin et al., 2019, Zhang et al., 2021]. Intriguingly, both theoretical and empirical studies have indicated that the crucial factor determining the network's generalisation properties is not the sheer number of parameters, but rather the norm of these parameters [Bartlett, 1996, Neyshabur et al., 2014]. This norm is typically controlled through a combination of explicit regularisation techniques, such as weight decay [Krogh and Hertz, 1991], and some form of implicit regularisation resulting from the training algorithm employed [Soudry et al., 2018, Lyu and Li, 2019, Ji and Telgarsky, 2019, Chizat and Bach, 2020].
Neural networks with a large number of parameters can approximate any continuous function on a compact set [Barron, 1993]. Thus, without norm control, the space of estimated functions encompasses all continuous functions. In the parameter space, this implies considering neural networks with infinite width and unbounded weights [Neyshabur et al., 2014]. Yet, when weight control is enforced, the exact correspondence between the parameter space (i.e., the parameters  $\theta$  of the network) and the function space (i.e., the estimated function  $f_{\theta}$  produced by the network's output) becomes unclear. Establishing this correspondence is pivotal for comprehending the generalisation properties of overparameterised neural networks. Two fundamental questions arise.
Question 1. What quantity in the function space, does the parameters' norm of a neural network correspond to?
Question 2. What functions are learnt when fitting training data with minimal parameters' norm?
We study these questions in the context of a one-hidden ReLU layer network with a skip connection. Previous research [Kurkova and Sanguineti, 2001, Bach, 2017] has examined generalisation guarantees for small representational cost functions, where the representational cost refers to the
norm required to parameterise the function. However, it remains challenging to interpret this representational cost using classical analysis tools and identify the corresponding function space. To address this issue, Question 1 seeks to determine whether this representational cost can be translated into a more interpretable functional (pseudo) norm. Note that Question 1 studies the parameters' norm required to fit a function on an entire domain. In contrast, when training a neural network for a regression task, we only fit a finite number of points given by the training data. Question 2 arises to investigate the properties of the learned functions when minimising some empirical loss with a regularisation of the parameters' norm regardless of whether it is done explicitly or implicitly.
In relation to our work, Savarese et al. [2019], Ongie et al. [2019] address Question 1 for one-hidden layer ReLU neural networks, focusing on univariate and multivariate functions, respectively. For a comprehensive review of this line of work, we recommend consulting the survey of Parhi and Nowak [2023]. On the other hand, Parhi and Nowak [2021], Debarre et al. [2022], Stewart et al. [2022] investigate Question 2 specifically in the univariate case. Additionally, Sanford et al. [2022] examine a particular multidimensional case. However, all of these existing studies overlook the bias parameters of the neural network when considering the  $\ell_2$  regularisation term. By omitting the biases, the analysis and solutions to these questions become simpler.
In sharp contrast, our work addresses both Questions 1 and 2 for univariate functions while also incorporating regularisation of the bias parameters. It may appear as a minor detail—it is commonly believed that similar estimators are obtained whether or not the biases' norm<sup>1</sup> is penalised [see e.g. Ng, 2011]. Nonetheless, our research demonstrates that penalising the bias terms enforce sparsity and uniqueness of the estimated function, which is not achieved without including the bias regularisation. The practical similarity between these two explicit regularisations can be attributed to the presence of implicit regularisation, which considers the bias terms as well. The updates performed by first-order optimisation methods do not distinguish between bias and weight parameters, suggesting that they are subject to the same implicit regularisation. Consequently, while both regularisation approaches may yield similar estimators in practical settings, we contend that the theoretical estimators obtained with bias term regularisation capture the observed implicit regularisation effect. Hence, it is essential to investigate the implications of penalising the bias terms when addressing Questions 1 and 2, as the answers obtained in this scenario significantly differ from those without bias penalisation.
It is also worth mentioning that Shevchenko et al. [2022], Safran et al. [2022] prove that gradient flow learns sparse estimators for networks with ReLU activations. These sparsity guarantees are yet much weaker (larger number of activated directions) as they additionally deal with optimisation considerations (in opposition to directly considering the minimiser of the optimisation problem in both our work and the line of works mentioned above).
Contributions. After introducing the setting in Section 2, we address Question 1 in Section 3 using a similar analysis approach as Savarese et al. [2019]. The key result, Theorem 1, establishes that the representational cost of a function, when allowed a free skip connection, is given by the weighted total variation of its second derivative, incorporating a  $\sqrt{1 + x^2}$  term. Notably, penalising the bias terms introduces a  $\sqrt{1 + x^2}$  multiplicative weight in the total variation, contrasting with the absence of bias penalisation.
This weighting fundamentally impacts the answer to Question 2. In particular, it breaks the shift invariance property of the function's representational cost, rendering the analysis technique proposed by Debarre et al. [2022] inadequate<sup>2</sup>. To address this issue, we delve in Sections 4 and 5 into the computation and properties of solutions to the optimisation problem:
$$
\inf  _ {f} \left\| \sqrt {1 + x ^ {2}} f ^ {\prime \prime} \right\| _ {\mathrm {T V}} \quad \text {s u b j e c t} \forall i \in [ n ], f (x _ {i}) = y _ {i}.
$$
In Section 4, we reformulate this problem as a continuous dynamic program, enabling a simpler analysis of the minimisation problem. Leveraging this dynamic program reformulation, Section 5 establishes the uniqueness of the solution. Additionally, under certain data assumptions, we demonstrate that the minimiser is among the sparest interpolators in terms of the number of kinks. It is
<sup>1</sup>Even though Goodfellow et al. [2016, Chapter 7] claim that penalising the biases might lead to underfitting, our work does not focus on the optimisation aspect and assumes interpolation occurs.
2Although shift invariance is useful for analytical purposes, it is not necessarily desirable in practice. Notably, Nacson et al. [2022] show that dealing with uncentered data might be beneficial for learning relevant features, relying on the lack of shift invariance.
worth noting that similar results have been studied in the context of sparse spikes deconvolution [Candès and Fernandez-Granda, 2014, Fernandez-Granda, 2016, Poon et al., 2019], and our problem can be seen as a generalisation of basis pursuit [Chen et al., 2001] to infinite-dimensional parameter spaces. However, classical techniques for sparse spikes deconvolution are ill-suited for addressing Question 2, as the set of sparsest interpolators is infinite in our setting.
Finally, the significance of bias term regularisation in achieving sparser estimators during neural network training is illustrated on toy examples in Section 6. To ensure conciseness, only proof sketches are presented in the main paper, while the complete proofs can be found in the Appendix.
2 Infinite width networks
This section introduces the considered setting, representing unidimensional functions as infinite width networks. Some precise mathematical arguments are omitted here, since this construction follows directly the lines of Savarese et al. [2019], Ongie et al. [2019]. This work considers unidimensional functions  $f_{\theta}:\mathbb{R}\to \mathbb{R}$  parameterised by a one hidden layer neural networks with ReLU activation as
$$
f _ {\theta} (x) = \sum_ {j = 1} ^ {m} a _ {j} \sigma \left(w _ {j} x + b _ {j}\right),
$$
where  $\sigma (z) = \max (0,z)$  is the ReLU activation and  $\theta = (a_j,w_j,b_j)_{j\in [m]}\in \mathbb{R}^{3m}$  are the parameters defining the neural network. The vector  $\mathbf{a} = (a_j)_{j\in [m]}$  stands for the weights of the last layer, while  $\mathbf{w}$  and  $\mathbf{b}$  respectively stand for the weights and biases of the hidden layer. For any width  $m$  and parameters  $\theta$ , the quantity of importance is the squared Euclidean norm of the parameters:  $\| \theta \| _2^2 = \sum_{j = 1}^m a_j^2 +w_j^2 +b_j^2$
We recall that contrary to Savarese et al. [2019], Ongie et al. [2019], the bias terms are included in the considered norm here. We now define the representational cost of a function  $f: \mathbb{R} \to \mathbb{R}$  as
$$
R(f) = \inf_{\substack{m\in \mathbb{N}\\ \theta \in \mathbb{R}^{3m}}}\frac{1}{2}\| \theta \|_{2}^{2}\quad \text{such that}\quad f_{\theta} = f.
$$
By homogeneity of the parameterisation, a typical rescaling trick [see e.g. Neyshabur et al., 2014, Theorem 1] allows to rewrite
$$
R (f) = \inf  _ {m, \theta \in \mathbb {R} ^ {3 m}} \| \mathbf {a} \| _ {1} \quad \text {s u c h t h a t} \quad f _ {\theta} = f \text {a n d} w _ {j} ^ {2} + b _ {j} ^ {2} = 1 \text {f o r a n y} j \in [ m ].
$$
Note that  $R(f)$  is only finite when the function  $f$  is exactly described as a finite width neural network. We aim at extending this definition to a much larger functional space, i.e. to any function that can be arbitrarily well approximated by finite width networks, while keeping a (uniformly) bounded norm of the parameters. Despite approximating the function with finite width networks, the width necessarily grows to infinity when the approximation error goes to 0. Similarly to Ongie et al. [2019], define
$$
\overline {{R}} (f) = \lim  _ {\varepsilon \rightarrow 0 ^ {+}} \left(\inf  _ {m, \theta \in \mathbb {R} ^ {3 m}} \frac {1}{2} \| \theta \| _ {2} ^ {2} \quad \text {s u c h t h a t} \quad | f _ {\theta} (x) - f (x) | \leq \varepsilon \text {f o r a n y} x \in [ - 1 / \varepsilon , 1 / \varepsilon ]\right).
$$
Note that the approximation has to be restricted to the compact set  $[-1/\varepsilon, 1/\varepsilon]$  to avoid problematic degenerate situations. The functional space for which  $\overline{R}(f)$  is finite is much larger than for  $R$ , and includes every compactly supported Lipschitz function, while coinciding with  $R$  when the latter is finite.
By rescaling argument again, we can assume the hidden layer parameters  $(w_{j},b_{j})$  are in  $\mathbb{S}_1$  and instead consider the  $\ell_1$  norm of the output layer weights. The parameters of a network can then be seen as a discrete signed measure on the unit sphere  $\mathbb{S}_1$ . When the width goes to infinity, a limit is then properly defined and corresponds to a possibly continuous signed measure. Mathematically, define  $\mathcal{M}(\mathbb{S}_1)$  the space of signed measures  $\mu$  on  $\mathbb{S}_1$  with finite total variation  $\| \mu \|_{\mathrm{TV}}$ . Following the typical construction of Bengio et al. [2005], Bach [2017], an infinite width network is parameterised by a measure  $\mu \in \mathcal{M}(\mathbb{S}_1)$  as<sup>3</sup>
$$
f _ {\mu}: x \mapsto \int_ {\mathbb {S} _ {1}} \sigma (w x + b) \mathrm {d} \mu (w, b). \tag {1}
$$
Similarly to Ongie et al. [2019],  $\overline{R}(f)$  verifies the equality
$$
\overline {{R}} (f) = \inf  _ {\mu \in \mathcal {M} (\mathbb {S} _ {1})} \| \mu \| _ {\mathrm {T V}} \text {s u c h t h a t} f = f _ {\mu}.
$$
<sup>3</sup>By abuse of notation, we write both  $f_{\theta}$  and  $f_{\mu}$ , as it is clear from context whether the subscript is a vector or a measure.
The right term defines the  $\mathcal{F}_1$  norm [Kurkova and Sanguineti, 2001], i.e.  $\bar{R}(f) = \| f\|_{\mathcal{F}_1}$ . The  $\mathcal{F}_1$  norm is intuited to be of major significance for the empirical success of neural networks. In particular, generalisation properties of small  $\mathcal{F}_1$  norm estimators are derived by Kurkova and Sanguineti [2001], Bach [2017], while many theoretical results support that training one hidden layer neural networks with gradient descent often yields an implicit regularisation on the  $\mathcal{F}_1$  norm of the estimator [Lyu and Li, 2019, Ji and Telgarsky, 2019, Chizat and Bach, 2020, Boursier et al., 2022]. However, this implicit regularisation of the  $\mathcal{F}_1$  norm is not systematic and some works support that a different quantity can be implicitly regularised on specific examples [Razin and Cohen, 2020, Vardi and Shamir, 2021, Chistikov et al., 2023]. Still,  $\mathcal{F}_1$  norm seems to be closely connected to the implicit bias and its significance is the main motivation of this paper. While previous works also studied the representational costs of functions by neural networks [Savarese et al., 2019, Ongie et al., 2019], they did not penalise the bias term in the parameters' norm, studying a functional norm slightly differing from the  $\mathcal{F}_1$  norm. This subtlety is at the origin of different levels of sparsity between the obtained estimators with or without penalising the bias terms, as discussed in Sections 5 and 6; where sparsity of an estimator here refers to the minimal width required for a network to represent the function (or similarly to the cardinality of the support of  $\mu$  in Equation (1)). This notion of sparsity is more meaningful than the sparsity of the parameters  $\theta$  here, since different  $\theta$  (with different levels of sparsity) can represent the exact same estimated function.
2.1 Unpenalised skip connection
Our objective is now to characterise the  $\mathcal{F}_1$  norm of unidimensional functions and minimal norm interpolators, which can be approximately obtained when training a neural network with norm regularisation. The analysis and result yet remain complex despite the unidimensional setting. Allowing for an unpenalised affine term in the neural network representation leads to a cleaner characterisation of the norm and description of minimal norm interpolators. As a consequence, we parameterise in the remaining of this work finite and infinite width networks as follows:
$$
f _ {\theta , a _ {0}, b _ {0}}: x \mapsto a _ {0} x + b _ {0} + f _ {\theta} (x), \quad \text {a n d} \quad f _ {\mu , a _ {0}, b _ {0}}: x \mapsto a _ {0} x + b _ {0} + f _ {\mu} (x),
$$
where  $(a_0, b_0) \in \mathbb{R}^2$ . The affine part  $a_0x + b_0$  actually corresponds to a free skip connection in the neural network architecture [He et al., 2016] and allows to ignore the affine part in the representational cost of the function  $f$ , which we now define as
$$
\overline{R}_{1}(f) = \lim_{\varepsilon \to 0^{+}}\Bigg(\inf_{\substack{m,\theta \in \mathbb{R}^{3m}\\ (a_{0},b_{0})\in \mathbb{R}^{2}}}\frac{1}{2}\| \theta \|_{2}^{2}\quad \text{such that}\quad |f_{\theta ,a_{0},b_{0}}(x) - f(x)|\leq \varepsilon \text{for any} x\in [-1 / \varepsilon ,1 / \varepsilon ]\Bigg).
$$
The representational cost  $\overline{R}_1(f)$  is similar to  $\overline{R}(f)$ , but allows for a free affine term in the network architecture. Similarly to  $\overline{R}(f)$ , it can be proven, following the lines of Savarese et al. [2019], Ongie et al. [2019], that  $\overline{R}_1(f)$  verifies
$$
\overline{R}_{1}(f) = \inf_{\substack{\mu \in \mathcal{M}(\mathbb{S}_{1})\\ a_{0},b_{0}\in \mathbb{R}}}\| \mu \|_{\mathrm{TV}}\text{such that} f = f_{\mu ,a_{0},b_{0}}.
$$
The remaining of this work studies more closely the cost  $\bar{R}_1(f)$ . Theorem 1 in Section 3 can be directly extended to the cost  $\bar{R}(f)$ , i.e. without unpenalised skip connection. Its adapted version is given by Theorem 4 in Appendix C for completeness.
Multiple works also consider free skip connections as it allows for a simpler analysis [e.g. Savarese et al., 2019, Ongie et al., 2019, Debarre et al., 2022, Sanford et al., 2022]. Since a skip connection can be represented by two ReLU neurons  $(z = \sigma(z) - \sigma(-z))$ , it is commonly believed that considering a free skip connection does not alter the nature of the obtained results. This belief is further supported by empirical evidence in Section 6 and Appendix B, where our findings hold true both with and without free skip connections.
3 Representational cost
Theorem 1 below characterises the representational cost  $\overline{R}_1(f)$  of any univariate function.
Theorem 1. For any Lipschitz function  $f: \mathbb{R} \to \mathbb{R}$ ,
$$
\overline {{R}} _ {1} (f) = \left\| \sqrt {1 + x ^ {2}} f ^ {\prime \prime} \right\| _ {\mathrm {T V}} = \int_ {\mathbb {R}} \sqrt {1 + x ^ {2}} \mathrm {d} | f ^ {\prime \prime} | (x).
$$
For any non-Lipschitz function,  $\overline{R}_1(f) = \infty$
In Theorem 1,  $f''$  is the distributional second derivative of  $f$ , which is well defined for Lipschitz functions. Without penalisation of the bias terms, the representational cost is given by the total variation of  $f''$  [Savarese et al., 2019]. Theorem 1 states that penalising the biases adds a weight  $\sqrt{1 + x^2}$  to  $f''$ . This weighting favors sparser estimators when training neural networks, as shown in Section 5. Also, the space of functions that can be represented by infinite width neural networks with finite parameters' norm, when the bias terms are ignored, corresponds to functions with bounded total variation of their second derivative. When including these bias terms in the representational cost, second derivatives additionally require a light tail. Without a free affine term, Theorem 4 in Appendix C characterises  $\overline{R}(f)$ , which yields an additional term accounting for the affine part of  $f$ .
We note that Remark 4.2 of E and Wojtowytsch [2021] and Theorem 1 by Li et al. [2020b] are closely related to Theorems 1 and 4. However, these results only establish an equivalence between the norm  $\overline{R}(f)$  and another norm that quantifies the total variation of  $\sqrt{1 + x^2} f''$ , aside from the affine term. Our result, on the other hand, provides an exact equality between both norms, which proves to be particularly useful in the analysis of minimal norm interpolators.
Example 1. If the function  $f$  is given by a finite width network  $f(x) = \sum_{i=1}^{n} a_i \sigma(w_i x + b_i)$  with  $a_i, w_i \neq 0$  and pairwise different  $\frac{b_i}{w_i}$ ; Theorem 1 yields  $\overline{R}_1(f) = \sum_{i=1}^{n} |a_i| \sqrt{w_i^2 + b_i^2}$ . This exactly corresponds to half of the squared  $\ell_2$  norm of the vector  $(c_i a_i, \frac{w_i}{c_i}, \frac{b_i}{c_i})_{i=1,\dots,n}$  when  $c_i = \sqrt{\frac{\sqrt{w_i + b_i^2}}{|a_i|}}$ . This vector is thus a minimal representation of the function  $f$ .
According to Theorem 1, the minimisation problem considered when training one hidden ReLU layer infinite width neural network with  $\ell_2$  regularisation is equivalent to the minimisation problem
$$
\inf  _ {f} \sum_ {i = 1} ^ {n} (f (x _ {i}) - y _ {i}) ^ {2} + \lambda \left\| \sqrt {1 + x ^ {2}} f ^ {\prime \prime} \right\| _ {\mathrm {T V}}. \tag {2}
$$
What types of functions do minimise this problem? Which solutions does the  $\left\| \sqrt{1 + x^2} f''\right\|_{\mathrm{TV}}$  regularisation term favor? These fundamental questions are studied in the following sections. We show that this regularisation favors functions that can be represented by small (finite) width neural networks. On the contrary, when the weight decay term does not penalise the biases of the neural network, such a sparsity is not particularly preferred as highlighted by Section 6.
4 Computing minimal norm interpolator
To study the properties of solutions obtained by training data with either an implicit or explicit weight decay regularisation, we consider the minimal norm interpolator problem
$$
\inf  _ {\theta , a _ {0}, b _ {0}} \frac {1}{2} \| \theta \| _ {2} ^ {2} \quad \text {s u c h t h a t} \forall i \in [ n ], f _ {\theta , a _ {0}, b _ {0}} \left(x _ {i}\right) = y _ {i}, \tag {3}
$$
where  $(x_{i},y_{i})_{i\in [n]}\in \mathbb{R}^{2n}$  is a training set. Without loss of generality, we assume in the following that the observations  $x_{i}$  are ordered, i.e.,  $x_{1} < x_{2} < \ldots < x_{n}$ . Thanks to Theorem 1, this problem is equivalent, when allowing infinite width networks, to
$$
\inf  _ {f} \left\| \sqrt {1 + x ^ {2}} f ^ {\prime \prime} \right\| _ {\mathrm {T V}} \quad \text {s u c h} \forall i \in [ n ], f (x _ {i}) = y _ {i}. \tag {4}
$$
Lemma 1 below actually makes these problems equivalent as soon as the width is larger than some threshold smaller than  $n - 1$ . Equation (4) then corresponds to Equation (2) when the regularisation parameter  $\lambda$  is infinitely small.
Lemma 1. The problem in Equation (4) admits a minimiser. Moreover, with  $i_0 \coloneqq \min \{i \in [n] | x_i \geq 0\}$ , any minimiser is of the form
$$
f (x) = a x + b + \sum_ {i = 1} ^ {n - 1} a _ {i} (x - \tau_ {i}) _ {+}
$$
where  $\tau_{i}\in (x_{i},x_{i + 1}]$  for any  $i\in \{1,\ldots ,i_0 - 2\}$ ,  $\tau_{i_0 - 1}\in (x_{i_0 - 1},x_{i_0})$  and  $\tau_{i}\in [x_{i},x_{i + 1})$  for any  $i\in \{i_0,\dots ,n - 1\}$ .
Lemma 1 already provides a first guarantee on the sparsity of any minimiser of Equation (4). It indeed includes at most  $n - 1$  kinks. In contrast, minimal norm interpolators with an infinite number
of kinks exist when the bias terms are not regularised [Debarre et al., 2022]. An even stronger sparse recovery result is given in Section 5. Lemma 1 can be seen as a particular case of Theorem 1 of Wang et al. [2021]. In the multivariate case and without a free skip connection, the latter states that the minimal norm interpolator has at most one kink (i.e. neuron) per activation cone of the weights<sup>4</sup> and has no more than  $n + 1$  kinks in total. The idea of our proof is that several kinks among a single activation cone could be merged into a single kink in the same cone. The resulting function then still interpolates, but has a smaller representational cost.
Lemma 1 allows to only consider 2 parameters for each interval  $(x_{i}, x_{i+1})$  (potentially closed at one end). Actually, the degree of freedom is only 1 on such intervals: choosing  $a_{i}$  fixes  $\tau_{i}$  (or inversely) because of the interpolation constraint. Lemma 2 below uses this idea to recast the minimisation Problem (4) as a dynamic program with unidimensional state variables  $s_{i} \in \mathbb{R}$  for any  $i \in [n]$ .
Lemma 2. If  $x_{1} < 0$  and  $x_{n} \geq 0$ , then we have for  $i_{0} = \min \{i \in [n] | x_{i} \geq 0\}$  the following equivalence of optimisation problems