Title: Penalising the biases in norm regularisation enforces sparsity
Venue: NeurIPS 2023
Accepted: yes
Keywords: Neural networks

Min norm interpolators

Sparsity

Representational cost
Abstract: Controlling the parameters' norm often yields good generalisation when training neural networks. Beyond simple intuitions, the relation between regularising parameters' norm and obtained estimators remains theoretically misunderstood. 
For one hidden ReLU layer networks with unidimensional data, this work shows the parameters' norm required to represent a function is given by the total variation of its second derivative, weighted by a $\sqrt{1+x^2}$ factor. Notably, this weighting factor disappears when the norm of bias terms is not regularised. The presence of this additional weighting factor is of utmost significance as it is shown to enforce the uniqueness and sparsity (in the number of kinks) of the minimal norm interpolator. Conversely, omitting the bias' norm  allows for non-sparse solutions.
Penalising the bias terms in the regularisation, either explicitly or implicitly, thus leads to sparse estimators.

Penalising the biases in norm regularisation enforces sparsity

Abstract

Controlling the parameters' norm often yields good generalisation when training neural networks. Beyond simple intuitions, the relation between regularising parameters' norm and obtained estimators remains theoretically misunderstood. For one hidden ReLU layer networks with unidimensional data, this work shows the parameters' norm required to represent a function is given by the total variation of its second derivative, weighted by a √ 1 + x 2 factor. Notably, this weighting factor disappears when the norm of bias terms is not regularised. The presence of this additional weighting factor is of utmost significance as it is shown to enforce the uniqueness and sparsity (in the number of kinks) of the minimal norm interpolator. Conversely, omitting the bias' norm allows for non-sparse solutions. Penalising the bias terms in the regularisation, either explicitly or implicitly, thus leads to sparse estimators.

1 Introduction

Although modern neural networks are not particularly limited in terms of their number of parameters, they still demonstrate remarkable generalisation capabilities when applied to real-world data [Belkin et al., 2019, Zhang et al., 2021]. Intriguingly, both theoretical and empirical studies have indicated that the crucial factor determining the network's generalisation properties is not the sheer number of parameters, but rather the norm of these parameters [Bartlett, 1996, Neyshabur et al., 2014]. This norm is typically controlled through a combination of explicit regularisation techniques, such as weight decay [Krogh and Hertz, 1991], and some form of implicit regularisation resulting from the training algorithm employed [Soudry et al., 2018, Lyu and Li, 2019, Ji and Telgarsky, 2019, Chizat and Bach, 2020].

Neural networks with a large number of parameters can approximate any continuous function on a compact set [Barron, 1993]. Thus, without norm control, the space of estimated functions encompasses all continuous functions. In the parameter space, this implies considering neural networks with infinite width and unbounded weights [Neyshabur et al., 2014]. Yet, when weight control is enforced, the exact correspondence between the parameter space (i.e., the parameters θ of the network) and the function space (i.e., the estimated function f θ produced by the network's output) becomes unclear. Establishing this correspondence is pivotal for comprehending the generalisation properties of overparameterised neural networks. Two fundamental questions arise.

Question 1. What quantity in the function space, does the parameters' norm of a neural network correspond to?

Question 2. What functions are learnt when fitting training data with minimal parameters' norm?

We study these questions in the context of a one-hidden ReLU layer network with a skip connection. Previous research [Kurková and Sanguineti, 2001, Bach, 2017] has examined generalisation guarantees for small representational cost functions, where the representational cost refers to the 37th Conference on Neural Information Processing Systems (NeurIPS 2023).

Nicolas Flammarion

norm required to parameterise the function. However, it remains challenging to interpret this representational cost using classical analysis tools and identify the corresponding function space. To address this issue, Question 1 seeks to determine whether this representational cost can be translated into a more interpretable functional (pseudo) norm. Note that Question 1 studies the parameters' norm required to fit a function on an entire domain. In contrast, when training a neural network for a regression task, we only fit a finite number of points given by the training data. Question 2 arises to investigate the properties of the learned functions when minimising some empirical loss with a regularisation of the parameters' norm regardless of whether it is done explicitly or implicitly.

In relation to our work, Savarese et al. [2019], Ongie et al. [2019] address Question 1 for one-hidden layer ReLU neural networks, focusing on univariate and multivariate functions, respectively. For a comprehensive review of this line of work, we recommend consulting the survey of Parhi and Nowak [2023]. On the other hand, Parhi and Nowak [2021], Debarre et al. [2022], Stewart et al. [2022] investigate Question 2 specifically in the univariate case. Additionally, Sanford et al. [2022] examine a particular multidimensional case. However, all of these existing studies overlook the bias parameters of the neural network when considering the ℓ 2 regularisation term. By omitting the biases, the analysis and solutions to these questions become simpler.

In sharp contrast, our work addresses both Questions 1 and 2 for univariate functions while also incorporating regularisation of the bias parameters . It may appear as a minor detail-it is commonly believed that similar estimators are obtained whether or not the biases' norm 1 is penalised [see e.g. Ng, 2011]. Nonetheless, our research demonstrates that penalising the bias terms enforce sparsity and uniqueness of the estimated function, which is not achieved without including the bias regularisation. The practical similarity between these two explicit regularisations can be attributed to the presence of implicit regularisation, which considers the bias terms as well. The updates performed by first-order optimisation methods do not distinguish between bias and weight parameters, suggesting that they are subject to the same implicit regularisation. Consequently, while both regularisation approaches may yield similar estimators in practical settings, we contend that the theoretical estimators obtained with bias term regularisation capture the observed implicit regularisation effect. Hence, it is essential to investigate the implications of penalising the bias terms when addressing Questions 1 and 2, as the answers obtained in this scenario significantly differ from those without bias penalisation.

It is also worth mentioning that Shevchenko et al. [2022], Safran et al. [2022] prove that gradient flow learns sparse estimators for networks with ReLU activations. These sparsity guarantees are yet much weaker (larger number of activated directions) as they additionally deal with optimisation considerations (in opposition to directly considering the minimiser of the optimisation problem in both our work and the line of works mentioned above).

Contributions. After introducing the setting in Section 2, we address Question 1 in Section 3 using a similar analysis approach as Savarese et al. [2019]. The key result, Theorem 1, establishes that the representational cost of a function, when allowed a free skip connection, is given by the weighted total variation of its second derivative, incorporating a √ 1 + x 2 term. Notably, penalising the bias terms introduces a √ 1 + x 2 multiplicative weight in the total variation, contrasting with the absence of bias penalisation.

This weighting fundamentally impacts the answer to Question 2. In particular, it breaks the shift invariance property of the function's representational cost, rendering the analysis technique proposed by Debarre et al. [2022] inadequate 2 . To address this issue, we delve in Sections 4 and 5 into the computation and properties of solutions to the optimisation problem:

In Section 4, we reformulate this problem as a continuous dynamic program, enabling a simpler analysis of the minimisation problem. Leveraging this dynamic program reformulation, Section 5 establishes the uniqueness of the solution. Additionally, under certain data assumptions, we demonstrate that the minimiser is among the sparest interpolators in terms of the number of kinks. It is

worth noting that similar results have been studied in the context of sparse spikes deconvolution [Candès and Fernandez-Granda, 2014, Fernandez-Granda, 2016, Poon et al., 2019], and our problem can be seen as a generalisation of basis pursuit [Chen et al., 2001] to infinite-dimensional parameter spaces. However, classical techniques for sparse spikes deconvolution are ill-suited for addressing Question 2, as the set of sparsest interpolators is infinite in our setting.

Finally, the significance of bias term regularisation in achieving sparser estimators during neural network training is illustrated on toy examples in Section 6. To ensure conciseness, only proof sketches are presented in the main paper, while the complete proofs can be found in the Appendix.

2 Infinite width networks

This section introduces the considered setting, representing unidimensional functions as infinite width networks. Some precise mathematical arguments are omitted here, since this construction follows directly the lines of Savarese et al. [2019], Ongie et al. [2019]. This work considers unidimensional functions f θ : R → R parameterised by a one hidden layer neural networks with ReLU activation as

where σ ( z ) = max(0 , z ) is the ReLU activation and θ = ( a j , w j , b j ) j ∈ [ m ] ∈ R 3 m are the parameters defining the neural network. The vector a = ( a j ) j ∈ [ m ] stands for the weights of the last layer, while w and b respectively stand for the weights and biases of the hidden layer. For any width m and parameters θ , the quantity of importance is the squared Euclidean norm of the parameters: ∥ θ ∥ 2 2 = ∑ m j =1 a 2 j + w 2 j + b 2 j .

We recall that contrary to Savarese et al. [2019], Ongie et al. [2019], the bias terms are included in the considered norm here. We now define the representational cost of a function f : R → R as

By homogeneity of the parameterisation, a typical rescaling trick [see e.g. Neyshabur et al., 2014, Theorem 1] allows to rewrite

Note that R ( f ) is only finite when the function f is exactly described as a finite width neural network. We aim at extending this definition to a much larger functional space, i.e. to any function that can be arbitrarily well approximated by finite width networks, while keeping a (uniformly) bounded norm of the parameters. Despite approximating the function with finite width networks, the width necessarily grows to infinity when the approximation error goes to 0 . Similarly to Ongie et al. [2019], define

Note that the approximation has to be restricted to the compact set [ -1 / ε , 1 / ε ] to avoid problematic degenerate situations. The functional space for which R ( f ) is finite is much larger than for R , and includes every compactly supported Lipschitz function, while coinciding with R when the latter is finite.

By rescaling argument again, we can assume the hidden layer parameters ( w j , b j ) are in S 1 and instead consider the ℓ 1 norm of the output layer weights. The parameters of a network can then be seen as a discrete signed measure on the unit sphere S 1 . When the width goes to infinity, a limit is then properly defined and corresponds to a possibly continuous signed measure. Mathematically, define M ( S 1 ) the space of signed measures µ on S 1 with finite total variation ∥ µ ∥ TV . Following the typical construction of Bengio et al. [2005], Bach [2017], an infinite width network is parameterised by a measure µ ∈ M ( S 1 ) as 3

Similarly to Ongie et al. [2019], R ( f ) verifies the equality

The right term defines the F 1 norm [Kurková and Sanguineti, 2001], i.e. R ( f ) = ∥ f ∥ F 1 . The F 1 norm is intuited to be of major significance for the empirical success of neural networks. In particular, generalisation properties of small F 1 norm estimators are derived by Kurková and Sanguineti [2001], Bach [2017], while many theoretical results support that training one hidden layer neural networks with gradient descent often yields an implicit regularisation on the F 1 norm of the estimator [Lyu and Li, 2019, Ji and Telgarsky, 2019, Chizat and Bach, 2020, Boursier et al., 2022]. However, this implicit regularisation of the F 1 norm is not systematic and some works support that a different quantity can be implicitly regularised on specific examples [Razin and Cohen, 2020, Vardi and Shamir, 2021, Chistikov et al., 2023]. Still, F 1 norm seems to be closely connected to the implicit bias and its significance is the main motivation of this paper. While previous works also studied the representational costs of functions by neural networks [Savarese et al., 2019, Ongie et al., 2019], they did not penalise the bias term in the parameters' norm, studying a functional norm slightly differing from the F 1 norm. This subtlety is at the origin of different levels of sparsity between the obtained estimators with or without penalising the bias terms, as discussed in Sections 5 and 6; where sparsity of an estimator here refers to the minimal width required for a network to represent the function (or similarly to the cardinality of the support of µ in Equation (1)). This notion of sparsity is more meaningful than the sparsity of the parameters θ here, since different θ (with different levels of sparsity) can represent the exact same estimated function.

2.1 Unpenalised skip connection

Our objective is now to characterise the F 1 norm of unidimensional functions and minimal norm interpolators, which can be approximately obtained when training a neural network with norm regularisation. The analysis and result yet remain complex despite the unidimensional setting. Allowing for an unpenalised affine term in the neural network representation leads to a cleaner characterisation of the norm and description of minimal norm interpolators. As a consequence, we parameterise in the remaining of this work finite and infinite width networks as follows:

where ( a 0 , b 0 ) ∈ R 2 . The affine part a 0 x + b 0 actually corresponds to a free skip connection in the neural network architecture [He et al., 2016] and allows to ignore the affine part in the representational cost of the function f , which we now define as

The representational cost R 1 ( f ) is similar to R ( f ) , but allows for a free affine term in the network architecture. Similarly to R ( f ) , it can be proven, following the lines of Savarese et al. [2019], Ongie et al. [2019], that R 1 ( f ) verifies

The remaining of this work studies more closely the cost R 1 ( f ) . Theorem 1 in Section 3 can be directly extended to the cost R ( f ) , i.e. without unpenalised skip connection. Its adapted version is given by Theorem 4 in Appendix C for completeness.

Multiple works also consider free skip connections as it allows for a simpler analysis [e.g. Savarese et al., 2019, Ongie et al., 2019, Debarre et al., 2022, Sanford et al., 2022]. Since a skip connection can be represented by two ReLU neurons ( z = σ ( z ) -σ ( -z ) ), it is commonly believed that considering a free skip connection does not alter the nature of the obtained results. This belief is further supported by empirical evidence in Section 6 and Appendix B, where our findings hold true both with and without free skip connections.

3 Representational cost

Theorem 1 below characterises the representational cost R 1 ( f ) of any univariate function.

Theorem 1. For any Lipschitz function f : R → R ,

For any non-Lipschitz function, R 1 ( f ) = ∞ .

In Theorem 1, f ′′ is the distributional second derivative of f , which is well defined for Lipschitz functions. Without penalisation of the bias terms, the representational cost is given by the total variation of f ′′ [Savarese et al., 2019]. Theorem 1 states that penalising the biases adds a weight √ 1 + x 2 to f ′′ . This weighting favors sparser estimators when training neural networks, as shown in Section 5. Also, the space of functions that can be represented by infinite width neural networks with finite parameters' norm, when the bias terms are ignored, corresponds to functions with bounded total variation of their second derivative. When including these bias terms in the representational cost, second derivatives additionally require a light tail . Without a free affine term, Theorem 4 in Appendix C characterises R ( f ) , which yields an additional term accounting for the affine part of f .

We note that Remark 4.2 of E and Wojtowytsch [2021] and Theorem 1 by Li et al. [2020b] are closely related to Theorems 1 and 4. However, these results only establish an equivalence between the norm R ( f ) and another norm that quantifies the total variation of √ 1 + x 2 f ′′ , aside from the affine term. Our result, on the other hand, provides an exact equality between both norms, which proves to be particularly useful in the analysis of minimal norm interpolators.

n

Example 1. If the function f is given by a finite width network f ( x ) = ∑ i =1 a i σ ( w i x + b i ) with a i , w i = 0 and pairwise different b i w i ; Theorem 1 yields R 1 ( f ) = ∑ n i =1 | a i | √ w 2 i + b 2 i . This exactly corresponds to half of the squared ℓ 2 norm of the vector ( c i a i , w i c i , b i c i ) i =1 ,...,n when c i = √ √ w i + b 2 i | a i | . This vector is thus a minimal representation of the function f .

̸

According to Theorem 1, the minimisation problem considered when training one hidden ReLU layer infinite width neural network with ℓ 2 regularisation is equivalent to the minimisation problem

What types of functions do minimise this problem? Which solutions does the ∥ ∥ √ 1 + x 2 f ′′ ∥ ∥ TV regularisation term favor? These fundamental questions are studied in the following sections. We show that this regularisation favors functions that can be represented by small (finite) width neural networks. On the contrary, when the weight decay term does not penalise the biases of the neural network, such a sparsity is not particularly preferred as highlighted by Section 6.

4 Computing minimal norm interpolator

To study the properties of solutions obtained by training data with either an implicit or explicit weight decay regularisation, we consider the minimal norm interpolator problem

where ( x i , y i ) i ∈ [ n ] ∈ R 2 n is a training set. Without loss of generality, we assume in the following that the observations x i are ordered, i.e., x 1 < x 2 < . . . < x n . Thanks to Theorem 1, this problem is equivalent, when allowing infinite width networks, to

Lemma 1 below actually makes these problems equivalent as soon as the width is larger than some threshold smaller than n -1 . Equation (4) then corresponds to Equation (2) when the regularisation parameter λ is infinitely small.

Lemma 1. The problem in Equation (4) admits a minimiser. Moreover, with i 0 := min { i ∈ [ n ] | x i ≥ 0 } , any minimiser is of the form

Lemma 1 already provides a first guarantee on the sparsity of any minimiser of Equation (4). It indeed includes at most n -1 kinks. In contrast, minimal norm interpolators with an infinite number

of kinks exist when the bias terms are not regularised [Debarre et al., 2022]. An even stronger sparse recovery result is given in Section 5. Lemma 1 can be seen as a particular case of Theorem 1 of Wang et al. [2021]. In the multivariate case and without a free skip connection, the latter states that the minimal norm interpolator has at most one kink (i.e. neuron) per activation cone of the weights 4 and has no more than n +1 kinks in total. The idea of our proof is that several kinks among a single activation cone could be merged into a single kink in the same cone. The resulting function then still interpolates, but has a smaller representational cost.

Lemma 1 allows to only consider 2 parameters for each interval ( x i , x i +1 ) (potentially closed at one end). Actually, the degree of freedom is only 1 on such intervals: choosing a i fixes τ i (or inversely) because of the interpolation constraint. Lemma 2 below uses this idea to recast the minimisation Problem (4) as a dynamic program with unidimensional state variables s i ∈ R for any i ∈ [ n ] .

Lemma 2. If x 1 < 0 and x n ≥ 0 , then we have for i 0 = min { i ∈ [ n ] | x i ≥ 0 } the following equivalence of optimisation problems

where the set Λ and the functions g i and c i are defined in Equations (6) to (8) below.

Let us describe the dynamic program defining the functions c i , which characterises the minimal norm interpolator thanks to Lemma 2. First define for any i ∈ [ n -1] , the slope δ i := y i +1 -y i x i +1 -x i ; the function

The set Λ is then the union of three product spaces given by

Finally, we define the functions c i : R → R + recursively as c 1 = c n ≡ 0 and

Equation (8) defines a dynamic program with a continuous state space. Intuitively for i ≥ i 0 , the variable s i accounts for the left derivative at the point x i . The term g i +1 ( s i +1 , s i ) is the minimal cost (in neuron norm) for reaching the point ( x i +1 , y i +1 ) with a slope s i +1 , knowing that the left slope is s i at the point ( x i , y i ) . Similarly, the interval S i ( s i ) gives the reachable slopes 5 at x i +1 , knowing the slope in x i is s i . Finally, c i ( s i ) holds for the minimal cost of fitting all the points ( x i +1 , y i +1 ) , . . . , ( x n , y n ) when the left derivative in ( x i , y i ) is given by s i . It is defined recursively by minimising the sum of the cost for reaching the next point ( x i +1 , y i +1 ) with a slope s i +1 , given by g i +1 ( s i +1 , s i ) ; and the cost of fitting all the points after x i +1 , given by c i +1 . This recursive definition is illustrated in Figure 1 below. A symmetric definition holds for i < i 0 .

The idea to derive Equation (6) is to first use Lemma 1 to get a finite representation of a minimal function f ∗ . From there, the minimal cost for connecting the point ( x i +1 , y i +1 ) with ( x i , y i ) is done by using a single kink in between. The restrictions given by s i and s i +1 then yield a unique possible kink. Minimizing its neuron norm then yields Equation (6)

Remark 1. Equation (5) actually considers the junction of two dynamic programs: a first one corresponding to the points with negative x values and a second one for positive values. This separation around x = 0 is not needed for Lemma 2, but allows for a cleaner analysis in Section 5. Lemmas 1 and 2 also hold for any arbitrary choice of i 0 . In particular for i 0 = 1 , Equation (5) would not consider the junction of two dynamic programs anymore, but a single one.

y

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

x

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

kink of cost

g

i

+1

(

s

i

+1

, s

i

)

xi

s

i

xi

+1

s

i

+1

total cost

c

i

(

s

i

)

total cost

c

i

+1

(

s

i

+1

)

f

Remark 2. The assumption x 1 < 0 and x n ≥ 0 is not fundamental, but is only required to properly define the junction mentioned in Remark 1. If all the x values are positive (or negative by symmetry), the analysis of the right term in Equation (5) is simplified, since there is no junction to consider. In particular, all the results from Section 5 hold without this assumption. These results are proven in the hardest case x 1 < 0 and x n ≥ 0 in Appendix E, from which other cases can be directly deferred.

Lemma 2 formulates the minimisation of the representational cost among the interpolating functions as a simpler dynamic program on the sequence of slopes at each x i . This equivalence is the key technical result of this work, from which Section 5 defers many properties on the minimiser(s) of Equation (4).

5 Properties of minimal norm interpolator

Thanks to the dynamic program formulation given by Lemma 2, this section derives key properties on the interpolating functions of minimal representational cost. In particular, it shows that Equation (4) always admits a unique minimum. Moreover, under some condition on the training set, this minimising function has the smallest number of kinks among the set of interpolators.

Theorem 2. The following optimisation problem admits a unique minimiser:

The proof of Theorem 2 uses the correspondence between interpolating functions and sequences of slopes ( s i ) i ∈ [ n ] ∈ S , where the set S is defined by Equation (23) in Appendix D.2. In particular, we show that the following problem admits a unique minimiser:

We note in the following s ∗ ∈ S the unique minimiser of the problem in Equation (9). From this sequence of slopes s ∗ , the unique minimising function of Equation (4) can be recovered. Moreover, s ∗ minimises the dynamic program given by the functions c i as follows:

Using simple properties of the functions c i given by Lemma 7 in Appendix E, properties on s ∗ can be derived besides the uniqueness of the minimal norm interpolator. Lemma 3 below gives a first intuitive property of this minimiser, which proves helpful in showing the main result of the section.

Lemma 3. For any i ∈ [ n ] , s ∗ i ∈ [min( δ i -1 , δ i ) , max( δ i -1 , δ i )] , where δ 0 := δ 1 and δ n := δ n -1 by convention.

A geometric interpretation of Lemma 3 is that the optimal (left or right) slope in x i is between the line joining ( x i -1 , y i -1 ) with ( x i , y i ) and the line joining ( x i , y i ) with ( x i +1 , y i +1 ) .

5.1 Recovering a sparsest interpolator

We now aim at characterising when the minimiser of Equation (4) is among the set of sparsest interpolators, in terms of number of kinks. Before describing the minimal number of kinks required

to fit the data in Lemma 4, we partition [ x 1 , x n ) into intervals of the form [ x n k , x n k +1 ) where

and sign(0) := 0 by convention. If we note f lin the canonical piecewise linear interpolator, it is either convex, concave or affine on every interval [ x n k -1 , x n k +1 ] . This partitioning thus splits the space into convex, concave and affine parts of f lin , as illustrated by Figure 2 on a toy example. This partition is crucial in describing the sparsest interpolators, thanks to Lemma 4.

̸

3

2

1

0

1

2

3

4

5

-

1

0

1

2

3

4

5

6

y

xn

0

xn

1

xn

2

convex

xn

3

affine

xn

4

convex

xn

5

concave

f

-

-

-

x

Lemma 4. If we denote by ∥ f ′′ ∥ 0 the cardinality of the support of the measure f ′′ ,

̸

Lemma 4's proof idea is that for any interval [ x k -1 , x k +1 ) where f lin is convex (resp. concave) non affine, any function requires at least one positive (resp. negative) kink to fit the three data points in this interval. The result then comes from counting the number of such disjoint intervals and showing that a specific interpolator exactly reaches this number.

The minimal number of kinks required to interpolate the data is given by Lemma 4. Before giving the main result of this section, we introduce the following assumption on the data ( x k , y k ) k ∈ [ n ] .

Assumption 1. For the sequence ( n k ) k defined in Equation (10) :

Assumption 1 exactly means there are no 6 (or more) consecutive points x k , . . . , x k +5 such that f lin is convex (without 3 aligned points) or concave on [ x k , x k +5 ] . This assumption depends a lot on the structure of the true model function (if there is any). For example, it holds if the truth is given by a piecewise linear function, while it may not if the truth is given by a quadratic function. Theorem 3 below shows that under Assumption 1, the minimal cost interpolator is amongst the sparsest interpolators, in number of its kinks.

Theorem 3. If Assumption 1 holds, then

Theorem 3 states conditions under which the interpolating function f with the smallest representational cost R 1 ( f ) also has the minimal number of kinks, i.e. ReLU hidden neurons, among the set of interpolators. It illustrates how norm regularisation, and in particular adding the biases' norm to the weight decay, favors estimators with a small number of neurons. While training neural networks with norm regularisation, the final estimator can actually have many non-zero neurons, but they all align towards a few key directions. As a consequence, the obtained estimator is actually equivalent to a small width network, meaning they have the same output for every input x ∈ R .

Recall that such a sparsity does not hold when the bias terms are not regularised. More precisely, some sparsest interpolators have a minimal representational cost in that case, but there are also minimal cost interpolators with an arbitrarily large (even infinite) number of kinks [Debarre et al., 2022]. There is thus no particular reason that the obtained estimator is sparse when minimising

the representational cost without penalising the bias terms. Section 6 empirically illustrates this difference of sparsity in the recovered estimators, depending on whether or not the bias parameters are penalised in the norm regularisation.

The generalisation benefit of this sparsity remains unclear. Indeed, generalisation bounds in the literature often rely on the parameters' norm rather than the network width (i.e., sparsity level). The relation between sparsest and min norm interpolators is important to understand in the particular context of implicit regularisation. In particular, while Boursier et al. [2022] conjectured that the implicit bias for regression problem was towards min norm interpolators, Chistikov et al. [2023] recently proved that the implicit bias could sometimes instead lead to sparsest interpolators. Our result suggests that both min norm and sparsest interpolators often coincide, which could explain the prior belief of convergence towards min norm interpolators. Yet, Theorem 3 and Chistikov et al. [2023] instead suggest that, at least in some situations, implicit bias favors sparsest interpolators, yielding different estimators 6 .

Remark 3. Theorem 3 states that sparse recovery, given by Equation (11) , occurs if Assumption 1 holds. When n k +1 -n k ≥ 4 , i.e. there are convex regions of f lin with at least 6 points, Appendix A gives a counterexample where Equation (11) does not hold. However, Equation (11) can still hold under weaker data assumptions than Assumption 1. In particular, Appendix A gives a necessary and sufficient condition for sparse recovery when there are convex regions with exactly 6 points. When we allow for convex regions with at least 7 points, it however becomes much harder to derive conditions where sparse recovery still occurs.

Remark 4. The counterexample presented in Appendix A reveals an unexpected outcome: minimal representational cost interpolators may not necessarily belong to the sparest interpolators. This finding is closely related to the idea that it may not be generally feasible to characterize the implicit regularisation of gradient descent as minimising parameters norm [Vardi and Shamir, 2021, Chistikov et al., 2023]. In particular, Vardi and Shamir [2021], Chistikov et al. [2023] rely on examples where minimal norm interpolators are not the sparsest ones; and the implicit regularisation instead favors the latter. We believe that this inherent limitation is one of the underlying reason for the different implicit regularization effects observed in other settings such as matrix factorization [Gunasekar et al., 2017, Razin and Cohen, 2020, Li et al., 2020a].

5.2 Application to classification

In the binary classification setting, max-margin classifiers, defined as the minimiser of the problem

are known to be the estimators of interest. Indeed, gradient descent on the cross entropy loss l (ˆ y, y ) = log(1 + e -ˆ yy ) converges in direction to such estimators [Lyu and Li, 2019, Chizat and Bach, 2020]. Theorem 3 can be used to characterise max- margin classifiers, leading to Corollary 1.

Corollary 1.

Theorem 3 yields that the max-margin classifier is among the sparsest margin classifiers, when a free skip connection is allowed. We believe that the left minimisation problem admits a unique solution. However, uniqueness cannot be directly derived from Theorem 3, but would instead require another thorough analysis, using an adapted dynamic programming reformulation. Since the uniqueness property is of minor interest, we here prefer to focus on a direct corollary of Theorem 3. We emphasise that no data assumptions are required for classification tasks, apart from being univariate.

6 Experiments

This section compares, through Figure 3, the estimators that are obtained with and without counting the bias terms in the regularisation, when training a one-hidden ReLU layer neural network. The code is made available at github.com/eboursier/penalising_biases .

For this experiment, we train neural networks by minimising the empirical loss, regularised with the ℓ 2 norm of the parameters (either with or without the bias terms) with a regularisation factor λ = 10 -3 . Each neural network has m = 200 hidden neurons and all parameters are initialised i.i.d. as centered Gaussian variables of variance 1 / √ m (similar results are observed for larger initialisation scales). 7 There is no free skip connection here, which illustrates its benignity: the results that are expected by the above theory also happen without free skip connection. Experiments with a free skip connection are given in Appendix B and yield similar observations.

‖

‖

-

2

-

1

0

1

2

3

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

fθ

(

x

)

x

wj,

2

/wj,

1

-

0

.

6

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

s

j

wj

(a) Penalising bias terms in the ℓ 2 regularisation.

-

-

-

2

-

1

0

1

2

3

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

fθ

(

x

)

x

wj,

2

/wj,

1

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

6

s

j

wj

‖

‖

As predicted by our theoretical study, penalising the bias terms in the ℓ 2 regularisation enforces the sparsity of the final estimator. The estimator of Figure 3a indeed counts 2 kinks (the smallest number required to fit the data), while in Figure 3b, the directions of the neurons are scattered. More precisely, the estimator is almost smooth near x = -0 . 5 , while the sparse estimator of Figure 3a is clearly not differentiable at this point. Also, the estimator of Figure 3b includes a clear additional kink at x = 0 . Figure 3 thus illustrates that counting the bias terms in regularisation can lead to sparser estimators.

7 Conclusion

This work studies the importance of parameters' norm for one hidden ReLU layer neural networks in the univariate case. In particular, the parameters' norm required to represent a function is given by ∥ ∥ √ 1 + x 2 f ′′ ∥ ∥ TV when allowing for a free skip connection. In comparison to weight decay, which omits the bias parameters in the norm, an additional √ 1 + x 2 weighting term appears in the representational cost. This weighting is of crucial importance since it implies uniqueness of the minimal norm interpolator. Moreover, it favors sparsity of this interpolator in number of kinks. Minimising the parameters' norm (with the biases), which can be either obtained by explicit or implicit regularisation when training neural networks, thus leads to sparse interpolators. We believe this sparsity is a reason for the good generalisation properties of neural networks observed in practice.

Although these results provide some understanding of minimal norm interpolators, extending them to more general and difficult settings remains open. Even if the representational cost might be described in the multivariate case [as done by Ongie et al., 2019, without bias penalisation], characterising minimal norm interpolators seems very challenging in that case. Characterising minimal norm interpolators, with no free skip connection, also presents a major challenge for future work.

Acknowledgments and Disclosure of Funding

The authors thank Gal Vardi, for suggesting the proof of Corollary 1, through a direct use of Theorem 3. The authors also thank Claire Boyer, Julien Fageot and Loucas Pillaud-Vivien for very helpful discussions and feedback.

References

Francis Bach. Breaking the curse of dimensionality with convex neural networks. The Journal of Machine Learning Research , 18(1):629-681, 2017.

Andrew R Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory , 39(3):930-945, 1993.

Peter Bartlett. For valid generalization the size of the weights is more important than the size of the network. Advances in neural information processing systems , 9, 1996.

Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences , 116(32):15849-15854, 2019.

Yoshua Bengio, Nicolas Roux, Pascal Vincent, Olivier Delalleau, and Patrice Marcotte. Convex neural networks. Advances in neural information processing systems , 18, 2005.

Etienne Boursier, Loucas Pillaud-Vivien, and Nicolas Flammarion. Gradient flow dynamics of shallow relu networks for square loss and orthogonal inputs. arXiv preprint arXiv:2206.00939 , 2022.

Emmanuel J Candès and Carlos Fernandez-Granda. Towards a mathematical theory of superresolution. Communications on pure and applied Mathematics , 67(6):906-956, 2014.

Scott Shaobing Chen, David L Donoho, and Michael A Saunders. Atomic decomposition by basis pursuit. SIAM review , 43(1):129-159, 2001.

Dmitry Chistikov, Matthias Englert, and Ranko Lazic. Learning a neuron by a shallow relu network: Dynamics and implicit bias for correlated inputs. arXiv preprint arXiv:2306.06479 , 2023.

Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory , pages 1305-1338. PMLR, 2020.

Thomas Debarre, Quentin Denoyelle, Michael Unser, and Julien Fageot. Sparsest piecewise-linear regression of one-dimensional data. Journal of Computational and Applied Mathematics , 406: 114044, 2022.

Weinan E and Stephan Wojtowytsch. Representation formulas and pointwise properties for barron functions, 2021.

Carlos Fernandez-Granda. Super-resolution of point sources via convex programming. Information and Inference: A Journal of the IMA , 5(3):251-303, 2016.

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning . MIT Press, 2016. http: //www.deeplearningbook.org .

Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in Neural Information Processing Systems , 30, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Ziwei Ji and Matus Telgarsky. The implicit bias of gradient descent on nonseparable data. In Conference on Learning Theory , pages 1772-1798. PMLR, 2019.

Anders Krogh and John Hertz. A simple weight decay can improve generalization. Advances in neural information processing systems , 4, 1991.

Vera Kurková and Marcello Sanguineti. Bounds on rates of variable-basis and neural-network approximation. IEEE Transactions on Information Theory , 47(6):2659-2665, 2001.

Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. arXiv preprint arXiv:2012.09839 , 2020a.

Zhong Li, Chao Ma, and Lei Wu. Complexity measures for neural networks with general activation functions using path-based norms. arXiv preprint arXiv:2009.06132 , 2020b.

Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations , 2019.

Mor Shpigel Nacson, Kavya Ravichandran, Nathan Srebro, and Daniel Soudry. Implicit bias of the step size in linear diagonal neural networks. In International Conference on Machine Learning , pages 16270-16295. PMLR, 2022.

Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.6614 , 2014.

Andrew Ng. Sparse autoencoder. CS294A Lecture notes , 72(2011):1-19, 2011.

Greg Ongie, Rebecca Willett, Daniel Soudry, and Nathan Srebro. A function space view of bounded norm infinite width relu nets: The multivariate case. In International Conference on Learning Representations (ICLR 2020) , 2019.

Rahul Parhi and Robert D Nowak. Banach space representer theorems for neural networks and ridge splines. Journal of Machine Learning Research , 22(43):1-40, 2021.

Rahul Parhi and Robert D Nowak. Deep learning meets sparse regularization: A signal processing perspective. arXiv preprint arXiv:2301.09554 , 2023.

Clarice Poon, Nicolas Keriven, and Gabriel Peyré. Support localization and the fisher metric for off-the-grid sparse regularization. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1341-1350. PMLR, 2019.

Noam Razin and Nadav Cohen. Implicit regularization in deep learning may not be explainable by norms. Advances in neural information processing systems , 33:21174-21187, 2020.

Itay Safran, Gal Vardi, and Jason D Lee. On the effective number of linear regions in shallow univariate relu networks: Convergence guarantees and implicit bias. Advances in Neural Information Processing Systems , 35:32667-32679, 2022.

Clayton Sanford, Navid Ardeshir, and Daniel Hsu. Intrinsic dimensionality and generalization properties of the R -norm inductive bias. arXiv preprint arXiv:2206.05317 , 2022.

Pedro Savarese, Itay Evron, Daniel Soudry, and Nathan Srebro. How do infinite width bounded norm networks look in function space? In Conference on Learning Theory , pages 2667-2690. PMLR, 2019.

Alexander Shevchenko, Vyacheslav Kungurtsev, and Marco Mondelli. Mean-field analysis of piecewise linear solutions for wide relu networks. The Journal of Machine Learning Research , 23 (1):5660-5714, 2022.

Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research , 19(1): 2822-2878, 2018.

Lawrence Stewart, Francis Bach, Quentin Berthet, and Jean-Philippe Vert. Regression as classification: Influence of task formulation on neural network features. arXiv preprint arXiv:2211.05641 , 2022.

Gal Vardi and Ohad Shamir. Implicit regularization in relu networks with the square loss. In Conference on Learning Theory , pages 4224-4258. PMLR, 2021.

Yifei Wang, Jonathan Lacotte, and Mert Pilanci. The hidden convex optimization landscape of regularized two-layer relu networks: an exact characterization of optimal solutions. In International Conference on Learning Representations , 2021.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107-115, 2021.

A Discussing Assumption 1

Theorem 3 requires Assumption 1, which assumes that there are no convex (or concave) regions of f lin with at least 6 data points. Actually, when there is a convex (or concave) region with exactly 6 data points, i.e. n k +1 = n k +4 , Theorem 3 holds (for this region) if and only if for i = n k +1 :

The proof of this result (omitted here) shows that the problem

is minimised for ( s i , s i +1 ) = ( δ i , δ i ) if and only if Equation (13) holds, which corresponds to the (unique) sparsest way to interpolate the data on this convex region. To show that the minimum is reached at that point, we can first notice that s ∗ i -1 = δ i -2 and s ∗ i +2 = δ i +2 . Then, it requires a meticulous study of the directional derivatives of the (convex but non-differentiable) objective function at the point ( δ i , δ i ) .

Figure 4 below illustrates a case of 6 data points, where the condition of Equation (13) does not hold. Clearly, the minimal norm interpolator differs from the (unique) sparsest interpolator in that case.

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

y

min norm

sparsest

x

When considering more than 6 points, studying the minimisation problem becomes cumbersome and no simple condition of sparse recovery can be derived. When generating random data with large convex regions, e.g. 35 points, the minimal norm interpolator is rarely among the sparsest interpolators. Moreover, it seems that its number of kinks could be arbitrarily close to 34 , which is the trivial upper bound of the number of kinks given by Lemma 3; while the sparsest interpolators only have 17 kinks.

B Additional experiments

Figure 5 shows the minimiser of Equation (4) on the toy example of Figure 2. The minimising function is computed thanks to the dynamic program given by Lemma 2. Although the variables of this dynamic program are continuous, we can efficiently solve it by approximating the constraint space of each slope s i as a discrete grid of [ δ i -1 , δ i ] thanks to Lemma 3. For the data used in Figure 5, Assumption 1 holds. It is clear that the minimiser is very sparse, counting only 4 kinks. The partition given by Figure 2 then shows that this is indeed the smaller possible number of kinks, thanks to Lemma 4. On the other hand, the canonical piecewise linear interpolator f lin is is not as sparse and counts 7 kinks here.

3

2

1

0

1

2

3

4

5

-

1

0

1

2

3

4

5

6

y

Min norm interpolator

-

-

-

x

Figure 6 considers the exact same setting as Figure 3 in Section 6. The only difference is that we here allow a free skip connection in the neural network architecture, which represents the setting exactly described by Theorem 3.

-

2

-

1

0

1

2

3

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

fθ

(

x

)

x

wj,

2

/wj,

1

-

0

.

6

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

6

s

j

wj

‖

‖

(a) Final estimator when penalising bias terms in the ℓ 2 regularisation.

-

-

2

-

1

0

1

2

3

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

fθ

(

x

)

x

wj,

2

/wj,

1

-

0

.

6

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

6

s

j

wj

‖

‖

(b) Final estimators when training one-hidden ReLU neural networks with ℓ 2 regularisation and a free skip connection .

Figure 6: Final estimators when training one-hidden ReLU neural networks with ℓ 2 regularisation. The green dots correspond to the data, while the green line is the estimated function. Each blue star represents a hidden neuron ( w j , b j ) of the network: its x -axis value is given by -b j /w j , which coincides with the position of the kink of its associated ReLU; its y -axis value is given by the output layer weight a j .

Similar observations can be made: the obtained estimator when counting the bias terms in regularisation only has 2 kinks, while the estimator obtained by omitting the biases in the regularisation is much smoother (and thus much less sparse in the number of kinks). The only difference is that the latter estimator here does not have a clear kink at x = 0 , but is instead even smoother on the interval [ -0 . 5 , 0] . This is explained by the presence of more scattered kinks in this interval. Despite this slight difference, the main observation remains unchanged: the estimator is a sparsest one when counting the bias terms, while it counts a lot of kinks (and is even smooth) when omitting the biases.

C Proofs of Section 3

Theorem 4 below extends the characterisation of the representational cost R 1 ( f ) of Theorem 1.

Theorem 4. For any Lipschitz function f : R → R ,

-

where

For any non-Lipschitz function, R 1 ( f ) = R ( f ) = + ∞ .

Proof. We only prove the equality on R ( f ) here. The other part of Theorem 4 can be directly deduced from this proof. First assume that R ( f ) is finite. We can then consider some µ ∈ M ( S 1 ) such that for any x ∈ R , f ( x ) = ∫ S 1 σ ( wx + b )d µ ( w,b ) . Note that f is necessarily ∥ µ ∥ TV -Lipschitz, which proves the second part of Theorem 4. Without loss of generality, we can parameterise S 1 on θ ∈ [ -π 2 , 3 π 2 ) with T -1 ( θ ) = (cos θ, sin θ ) . If we note ν = T # µ the pushforward measure of µ by T , we then have

Since the total variation of µ and thus ν is bounded, we can derive under the integral sign:

Nownote that x cos θ +sin θ ≥ 0 if and only if θ ∈ [ -arctan( x ) , π -arctan( x )] since θ ∈ [ -π 2 , 3 π 2 ) , i.e.

When deriving this expression over x , we finally get for the distribution f ′′

This equality is straightforward for continuous distributions ν . Extending it to any distribution ν requires some extra work but can be obtained following the typical definition of distributional derivative.

Defining ν + ( θ ) = ν ( θ ) + ν ( π + θ ) for any θ ∈ [ -π 2 , π 2 ) and noting that cos(arctan x ) = 1 √ 1+ x 2 ,

Or equivalently, for any θ ( π , π

Similarly to the proof of Savarese et al. [2019], f ′′ fixes ν + and the only degree of freedom is on ν ⊥ ( θ ) := ν ( θ ) -ν ( π + θ ) . The proof now determines which valid ν ⊥ minimises ∥ µ ∥ TV = ∥ ν ∥ TV . Equation (14) implies the following condition on ν ⊥

While ν + is given by f ′′ , ν ⊥ holds for affine part of f . The above equality directly leads to the following condition on ν ⊥

Now note that 2 ∥ µ ∥ TV = ∥ ν + + ν ⊥ ∥ TV + ∥ ν + -ν ⊥ ∥ TV , so that R ( f ) is given by

Lemma 5 below then implies 8

where x f and C f are defined in Theorem 4. Equation (15) leads with a simple change of variable when R ( f ) is finite to

Reciprocally, when ∥ ∥ √ 1 + x 2 f ′′ ∥ ∥ TV is finite, we can define ν + as in Equation (15) and ν ⊥ as a sum of Diracs in -π 2 and 0 verifying Equation (16). The corresponding µ is then of finite total variation, implying that R ( f ) is finite. This ends the proof of the first part of Theorem 4:

For R 1 ( f ) , the analysis is simpler since there is no constraint on ν ⊥ , whose optimal choice is then given by ν ⊥ = 0 .

Lemma 5. The minimisation program

is equivalent to

Proof. For any ν ⊥ in the constraint set of Equation (17), we can use a decomposition ν ⊥ = φν + + µ 2 where ∥ φ ∥ ∞ ≤ 1 . It then comes pointwise

As a consequence, if we note v the infimum given by Equation (17):

where

Moreover for a fixed ν ⊥ , we can choose ( φ, µ 2 ) as:

where d ν ⊥ d ν + denotes by abuse of notation the Radon-Nikodym derivative d ν a d ν + , with the Lebesgue decomposition ν ⊥ = ν a + ν s with ν a ≪ ν + and ν s ⊥ ν + . For this choice, Equation (19) becomes an equality, which directly implies that

It now remains to prove that min u ∈C ∥ ( a, b ) -u ∥ 2 = min ( φ,µ 2 ) ∈ Γ ∫ π 2 -π 2 d | µ 2 | . Fix in the following φ such that ∥ φ ∥ ∞ ≤ 1 and note

It now suffices to show that for any fixed φ :

The constraint set is actually { µ 2 | ∫ π/ 2 -π/ 2 (cos θ, sin θ ) d µ 2 ( θ ) = ( x, y ) } . Now define

̸

where δ θ ∗ is the Dirac distribution located at θ ∗ . This definition is only valid if x = 0 , otherwise we choose µ ∗ 2 = -yδ -π 2 .

Note that µ ∗ 2 is in the constraint set and ∫ π 2 -π 2 d | µ 2 | = ∥ ( x, y ) ∥ , i.e.

Now consider any µ 2 in the constraint set and decompose µ 2 = µ + 2 -µ -2 with ( µ + 2 , µ -2 ) ∈ M + ([ -π 2 , π 2 )) 2 . Define

By Cauchy-Schwarz inequality,

Summing these two inequalities yields

Recall that ∫ d | µ 2 | = ∫ d µ + 2 + ∫ d µ -2 . By triangle inequality, this yields:

As a consequence:

We finally showed that

This leads to Lemma 5 when taking the infimum over φ .

Similarly, we have

D Proof of Section 4

D.1 Proof of Lemma 1

We first need to show the existence of a minimum. Using the definition of R 1 ( f ) and Theorem 1, Equation (4) is equivalent to

Consider a sequence ( µ j , a j , b j ) j such that f µ j ,a j ,b j ( x i ) = y i for any i and j and ∥ µ j ∥ TV converges to the infimum of Equation (20). The sequence ∥ µ j ∥ TV is necessarily bounded. This also implies that both ( a j ) and ( b j ) are bounded 9 . Since the space of finite signed measures on S 1 is a Banach space, there is a subsequence converging weakly towards some ( µ, a, b ) . By weak convergence, ( µ, a, b ) is in the constraints set of Equation (20) and ∥ µ ∥ TV = lim j ∥ µ j ∥ TV . ( µ, a, b ) is thus a minimiser of Equation (20). We thus proved the existence of a minimum for Equation (4), which is reached for f µ,a,b .

Define for the sake of the proof the activation cones C i as

As the x i are ordered, note that ( C 0 , C 1 , -C 1 , . . . , C n -1 , -C n -1 , C n ) forms a partition of R 2 . To prove Lemma 1, it remains to show that any minimiser ( µ a , b ) of Equation (20) has a function f µ,a,b of the form

Let f be a minimiser of Equation (4). Let µ, a, b be a minimiser of Equation (20) such that f µ,a,b = f . Define ˜ µ, ˜ a, ˜ b as

Thanks to the identity σ ( u ) -u = σ ( -u ) , f µ,a,b = f ˜ µ, ˜ a, ˜ b . Moreover, ∥ ˜ µ ∥ TV ≤ ∥ µ ∥ TV , so we can assume w.l.o.g. that the support of µ is included 10 in ⋃ n i =0 C i . In that case, for any i = , 1 . . . , n

First, the reduction

does not increase the total variation of µ and still interpolates the data. As a consequence, the support of µ is included in ⋃ n -1 i =1 C i . Now let µ = µ + -µ -be the Jordan decomposition of µ and define for any i ∈ [ n -1]

Note that α i and β i are both in the positive convex cone C i . For θ i := α i -β i , Equation (22) rewrites

If θ i ∈ C i ∪ -C i , we can then define

Thanks to Equation (22), the function f ˜ µ,a,b still interpolates the data and

By minimisation of ∥ µ ∥ TV , this is an equality. Moreover as µ is a measure on the sphere,

By minimisation, all inequalities are equalities. Jensen's case of equality implies for the first inequality that both µ + | C i and µ -| C i are Diracs, while the second inequality implies that either α i = 0 or β i = 0 . Overall, µ | C i is at most a single Dirac.

Now if θ i ̸∈ C i ∪ -C i , assume first that ⟨ θ i , ( x i +1 , 1) ⟩ > 0 . This implies ⟨ θ i , ( x i , 1) ⟩ > 0 since θ i ̸∈ C i ∪ -C i . This then implies that either α i ∈ ◦ C i or β i ∈ ◦ C i , depending on whether i ≥ i 0 or i < i 0 . Assume first that β i ∈ ◦ C i ( i ≥ i 0 ) and define

By continuity, tα i -β i ∈ -C i . Moreover 0 < t < 1 , since β i ∈ ◦ C i and θ i ̸∈ -C i . We now define ˜ µ = µ -µ | C i +(1 -t ) ∥ α i ∥ δ α i ∥ α i ∥ + ∥ tα i -β i ∥ δ tα i -β i ∥ tα i -β i ∥ .

The function f ˜ µ,a,b still interpolates the data. Similarly to the case θ i ∈ C i ∪ -C i , the minimisation of ∥ µ ∥ TV implies that µ | C i is at most a single Dirac.

If α i ∈ ◦ C i instead, similar arguments follow defining

Symmetric arguments also hold if ⟨ θ i , ( x i +1 , 1) ⟩ < 0 . In any case, µ | C i is at most a single Dirac. This holds for any i = 1 , . . . , n -1 , which finally leads to Lemma 1.

D.2 Proof of Lemma 2

Before proving Lemma 2, let us show a one to one mapping from the parameterisation given by Lemma 1 to a parameterisation given by the sequences of slopes in the points x i . Let us define the

sets

and

The condition τ i = x i + x i +1 2 if a i = 0 in the definition of I is just to avoid redundancy, as any arbitrary value of τ i would yield the same interpolating function. Lemma 6 below gives a one to one mapping between these two sets.

Lemma 6. The function

is a one to one mapping. Its inverse is given by where

̸

Proof. For ( a 0 , b 0 , ( a i , τ i ) i =1 ,...,n -1 ) ∈ I , let f be the associated interpolator:

and let ( s i ) i ∈ [ n ] = ψ ( a 0 , b 0 , ( a i , τ i ) i ) . Given the definition of ψ , it is straightforward to check that s i corresponds to the left (resp. right) derivative of f at x i ≥ 0 (resp. x i < 0 ). We actually have the two following inequalities linking the parameters ( a 0 , b 0 , ( a i , τ i ) i ) and ( s i ) i for any i ∈ [ n -1] :

The first equality comes from the (left or right) derivatives of f in x i , while the second equality is due to the interpolation of the data by f . These two equalities imply that an interpolator with ReLU parameters in I (i.e., f ) can be equivalently described by its (left or right) derivatives in each x i . A straightforward computation then allows to show that ψ and ψ -1 are well defined and indeed verify ψ ◦ ψ -1 = I S and ψ -1 ◦ ψ = I I .

Using this bijection from I to S , we can now prove Lemma 2. Note for the remaining of the proof α = min f ∀ i ∈ [ n ] ,f ( x i )= y i ∫ R √ 1 + x 2 d | f ′′ ( x ) | . Thanks to Lemma 1, we have the first equivalence:

For any ( a 0 , b 0 , ( a i , τ i ) i =1 ,...,n -1 ) ∈ I , we can define thanks to Lemma 6 ( s i ) i = ψ ( a 0 , b 0 , ( a i , τ i ) i ) ∈ S . We then have ( a 0 , b 0 , ( a i , τ i ) i ) = ψ -1 (( s i ) i ) . Moreover, by definition

of ψ -1 , we can easily check that

As ψ is a one to one mapping, we have for any function h the equivalence min u ∈ ψ -1 ( S ) h ( u ) = min s ∈S h ( ψ -1 ( s )) . In particular, thanks to Equation (24):

From there, define for any i ≥ i 0 ,

and for any i < i 0

Obviously, we have from Equation (25) and the definition of S that

It now remains to show by induction that for any i that c i = d i . This is obviously the case for i = n . Let us now consider i ∈ { i 0 , . . . , n -1 } . The definition of d i leads to

Now note that for any s i +1 ∈ S i ( s i ) , we have the equality of the sets

{ (˜ s j ) j ≥ i +1 | (˜ s ) j ∈ [ n -1] ∈ S s.t. ˜ s i = s i and ˜ s i +1 = s i +1 } = { (˜ s j ) j ≥ i +1 | (˜ s ) j ∈ [ n -1] ∈ S s.t. ˜ s i +1 = s i +1 } Since the last term in Equation (27) only depends on (˜ s j ) j ≥ i +1 , this implies that

By induction, it naturally comes from the definition of c i that c i = d i for any i ≥ i 0 . Symmetric arguments hold for any i < i 0 , which finally gives c i = d i for any i ∈ [ n ] . Equation (26) then yields Lemma 2.

E Proof of Section 5

The proofs of this section are shown in the case where x 1 < 0 and x n ≥ 0 . When all the x are positive, i.e., x 1 ≥ 0 , the adapted version of Lemma 2 would yield for i 0 = 1 the equivalence 11

The proofs of Appendix E can then be easily adapted to this case (and similarly if x n < 0 ). Appendix E.5 at the end of the section more precisely states how to adapt them to this case.

E.1 Proof of Theorem 2

Before proving Theorem 2, Lemma 7 below provides important properties verified by the functions c i defined in Equation (8).

Lemma 7. For each i ∈ { i 0 , . . . , n -1 } , the function c i is convex, √ 1 + x 2 i -Lipschitz on R and minimised for s i = δ i . :

Moreover, on both intervals ( -∞ , δ i ] and [ δ i , + ∞ )

either c i ( s i ) = √ 1 + x 2 i | s i -δ i | + c i +1 ( δ i ) for all s i in the considered interval, or c i is strictly convex on the considered interval;

Similarly, for each i ∈ { 1 , . . . , i 0 -2 } , the function c i +1 is convex, √ 1 + x 2 i +1 -Lipschitz on R and and minimised for s i +1 = δ i . Moreover, on both intervals ( -∞ , δ i ] and [ δ i , + ∞ ) :

either c i +1 ( s i +1 ) = √ 1 + x 2 i +1 | s i +1 -δ i | + c i ( δ i ) for all s i +1 in the considered interval, or c i +1 is strictly convex on the considered interval;

| c i +1 ( s i +1 ) -c i +1 ( s ′ i +1 ) | ≥ 1+ x i x i +1 √ 1+ x 2 i | s i +1 -s ′ i +1 | for all s i +1 , s ′ i +1 in the considered interval.

Proof. For any i ∈ { 1 , . . . , i 0 -2 } , we prove the result by (backward) induction. Since c n = 0 , a straightforward calculation gives 12

which gives the wanted properties for i = n -1 .

Now consider i ∈ { i 0 , . . . , n -2 } such that c i +1 verifies all the properties in the first part of Lemma 7. We first show the Lipschitz property of c i . Let s i , s ′ i < δ i first. By inductive assumption, the function s i +1 ↦→ g i +1 ( s i +1 , s i ) + c i +1 ( s i +1 ) reaches a minimum on [ δ i , + ∞ ) . Consider s i +1 ≥ δ i such that c i ( s i ) = g i +1 ( s i +1 , s i ) + c i +1 ( s i +1 ) .

Also by minimisation, c i ( s ′ i ) ≤ g i +1 ( s i +1 , s ′ i ) + c i +1 ( s i +1 ) . For the vectors u = ( x i +1 , 1) and v = ( x i , 1) , it then holds:

The first equality comes from the definition of g i +1 as a norm and the second inequality comes from the triangle inequality. By symmetry, we showed | c i ( s ′ i ) -c i ( s i ) | ≤ √ 1 + x 2 i | s i -s ′ i | for s i , s ′ i < δ i .

Note that if s i = δ i , then s i +1 = δ i and we show similarly that c i ( s ′ i ) -c i ( δ i ) ≤ √ 1 + x 2 i | δ i -s ′ i | . Moreover,

The first inequality comes from the Lipschitz property of c i +1 . The second from the fact that ( s ′ i +1 -δ i ) u and ( s ′ i -δ i ) v are negatively correlated, since x i and x i +1 are both positive. As a consequence, c i is √ 1 + x 2 i -Lipschitz on ( -∞ , δ i ] . Symmetrically, it is also √ 1 + x 2 i -Lipschitz on [ δ i , + ∞ ) , which finally implies it is √ 1 + x 2 i -Lipschitz on R . Moreover, the last calculation also shows that c i is minimised for s i = δ i .

Let us now show that c i verifies the first point on ( -∞ , δ i ] . By continuity, we only have to show it on ( -∞ , δ i ) . Let s i ∈ ( -∞ , δ i ) , we then have by definition

If δ i +1 ≤ δ i , note that both functions g i +1 ( · , s i ) and c i +1 are increasing on [ δ i , + ∞ ) 13 . The minimum is thus reached for s i +1 = δ i and

If δ i +1 > δ i , both functions g i +1 ( · , s i ) and c i +1 are increasing on [ δ i +1 , + ∞ ) . As a consequence, we can then rewrite

Assume first that c i +1 ( s i +1 ) = √ 1 + x 2 i +1 | s i +1 -δ i +1 | + c i +2 ( δ i +1 ) on [ δ i , δ i +1 ] . By triangle inequality, we actually have

This leads for s i +1 ∈ [ δ i , δ i +1 ] to

The minimum in Equation (28) is thus reached for , which finally gives for any

s i +1 = δ i +1 s i ≤ δ i

Since δ i +1 > δ i , it is easy to check that g i +1 ( δ i +1 , · ) is strictly convex on ( -∞ , δ i ) and so is c i .

Let us now assume the last case, where c i +1 is strictly convex on [ δ i , δ i +1 ] . By contradiction, assume that the first point on ( -∞ , δ i ] does not hold. Note in the following h ( s i +1 , s i ) = g i +1 ( s i +1 , s i ) + c i +1 ( s i +1 ) . For s i , s ′ i < δ i , by continuity of h , let s i +1 , s ′ i +1 ∈ [ δ i , δ i +1 ] be such that

For any t ∈ (0 , 1) , by convexity of h :

c i is thus convex on ( -∞ , δ i ] . Moreover, the case of equality corresponds to the case of equality for both g i +1 and c i +1 :

The former leads to the colinearity of the vectors ( s i +1 -δ i , s i -δ i ) and ( s ′ i +1 -δ i , s ′ i -δ i ) ; the latter gives s i +1 = s ′ i +1 by strict convexity of c i +1 . Two cases are then possible

The former case then implies that c i ( s i ) = √ 1 + x 2 i | s i -δ i | + c i +1 ( δ i ) . Since c i ( δ i ) = c i +1 ( δ i ) , c i is √ 1 + x 2 i -Lipschitz and convex on ( -∞ , δ i ] , this leads to c i ( s ) = √ 1 + x 2 i | s -δ i | + c i +1 ( δ i ) for any s ∈ ( -∞ , δ i ] . This contradicts the assumption that the first point does not hold on ( -∞ , δ i ] . Necessarily, we have s i = s ′ i . So c i is strictly convex on ( -∞ , δ i ] , which leads to another contradiction: the first point does hold on ( -∞ , δ i ] .

Finally, we just showed that in any case, c i is either strictly convex or equal to s i ↦→ √ 1 + x 2 i | s i -δ i | on ( -∞ , δ i ] . Symmetric arguments yield the same on [ δ i , + ∞ ) . c i is thus minimised in δ i , √ 1 + x 2 i -Lipschitz and verifies the first point on both intervals ( -∞ , δ i ] and [ δ i , + ∞ ) . This directly implies that c i is convex on R .

It now remains to show the second point on the two intervals. Let us show it on ( -∞ , δ i ] : on ( -∞ , δ i ) is actually sufficient by continuity. Consider s i < s ′ i < δ i and s i +1 ∈ [ δ i , + ∞ ) such that

By definition of c i ,

Straightforward computations yield that the function

is convex and h ′ 2 ( δ i ) = -1+ x i x i +1 √ 1+ x 2 i +1 . Thus, h ′ 2 ≤ -1+ x i x i +1 √ 1+ x 2 i +1 , which finally implies

The second point is thus verified on ( -∞ , δ i ) and on ( -∞ , δ i ] by continuity. Symmetric arguments lead to the same property on [ δ i , + ∞ ) .

By induction, this implies the first part of Lemma 7. Symmetric arguments lead to the second part of Lemma 7 for i ≤ i 0 -2 .

We can now prove Theorem 2. Following the proof of Lemma 2, there is a unique minimiser of Equation (4) if and only if the following problem admits a unique minimiser:

We already know that the minimum is attained thanks to Lemma 1. By construction of the functions c i , any minimum ˜ s of Equation (29) verifies

It now remains to show that all these problems admit unique minimisers. First assume Equation (31) admits different minimisers ( s i 0 -1 , s i 0 ) and ( s ′ i 0 -1 , s ′ i 0 ) . Note in the following h i 0 -1 : ( s, s ′ ) ↦→ g i 0 ( s, s ′ ) + c i 0 -1 ( s ′ ) + c i 0 ( s ) . By minimisation and convexity of the three functions g i 0 , c i 0 -1 , c i 0 , for any t ∈ (0 , 1) :

The whole segment joining ( s i 0 -1 , s i 0 ) and ( s ′ i 0 -1 , s ′ i 0 ) is then a minimiser. Without loss of generality, we can thus assume that both s i 0 and s ′ i 0 -1 are on the same side of δ i 0 -1 (e.g. smaller than δ i 0 -1 ) and both s i 0 and s ′ i 0 are on the same side of δ i 0 .

Moreover, Equation (33) implies an equality on g i 0 that leads to the colinearity of the vectors ( s i 0 -1 -δ i 0 -1 , s i 0 -δ i 0 -1 ) and ( s ′ i 0 -1 -δ i 0 -1 , s ′ i 0 -δ i 0 -1 ) . In particular, both s i 0 = s ′ i 0 and

̸

̸

s i 0 -1 = s ′ i 0 -1 . Moreover, we have equality cases on both c i 0 -1 and c i 0 implying, thanks to the first point of Lemma 7

For u = ( x i 0 -1 , 1) and v = ( x i 0 , 1) , we have by (positive) colinearity of ( s i 0 -1 -δ i 0 -1 , s i 0 -δ i 0 -1 ) and ( s ′ i 0 -1 -δ i 0 -1 , s ′ i 0 -δ i 0 -1 ) :

Since s i 0 = s ′ i 0 and s i 0 -1 = s ′ i 0 -1 , the triangle inequality gives both strict inequalities

̸

̸

Using this with Equation (34), this yields

This contradicts the fact that ( s i 0 -1 , s i 0 ) and ( s ′ i 0 -1 , s ′ i 0 ) both minimise Equation (31). Hence, Equation (31) admits a unique minimiser.

Also the minimisation problem

admits a unique minimiser for any i ∈ { i 0 , . . . , n -1 } . Indeed, either ˜ s i = δ i in which case the constraint set is a singleton, or the function s i +1 ↦→ g i +1 ( s i +1 , ˜ s i ) is strictly convex for ˜ s i = δ i . A symmetric argument exists for the minimisation problem of Equation (28). It thus concludes the proof of Theorem 2.

E.2 Proof of Lemma 3

Let i ∈ { i 0 , . . . , n -1 } . Recall that

If i = n -1 , the objective is obviously minimised for s ∗ n = δ n -1 as both x n -1 and x n are positive. Otherwise, assume for example that s ∗ i > δ i . Thanks to Lemma 7, the objective is decreasing on ( -∞ , min( δ i , δ i +1 )] and S i ( s ∗ i ) = [ δ i , + ∞ ) which yields that s ∗ i +1 ∈ [min( δ i , δ i +1 ) , δ i ] ⊂ [min( δ i , δ i +1 ) , max( δ i , δ i +1 )] . The case s ∗ i = δ i is trivial and similar arguments hold for s ∗ i < δ i .

Now consider i = i 0 -1 . Assume first that s ∗ i 0 -1 > δ i 0 -1 , then

Thanks to the last point of Lemma 7

Note that the function is convex and verifies h ′ ( δ i 0 -1 ) = -1+ x i 0 -1 x i 0 √ 1+ x 2 i 0 -1 . Since x i 0 -1 < 0 ≤ x i 0 +1 , it comes

Thanks to Equation (35), the function s i 0 ↦→ g i 0 ( s i 0 , s ∗ i 0 -1 ) + c i 0 ( s i 0 ) is thus decreasing on ( -∞ , min( δ i 0 -1 , δ i 0 )] . As above, this implies that s ∗ i 0 ∈ [min( δ i 0 -1 , δ i 0 ) , max( δ i 0 -1 , δ i 0 )] . The case s ∗ i 0 -1 = δ i 0 -1 is trivial and similar arguments hold if s ∗ i 0 -1 < δ i 0 -1 .

̸

̸

We showed s ∗ i +1 ∈ [min( δ i , δ i +1 ) , max( δ i , δ i +1 )] for any i ∈ { i 0 , . . . , n } . Symmetric arguments hold for i ∈ [ i 0 -1] . This concludes the proof of Lemma 3.

E.3 Proof of Lemma 4

Let us first prove that any sparsest interpolator f has at least a number of kinks given by the right sum. For that, we actually show that for k 1 , on any interval ( x , x ) with δ = δ

̸

≥ n k -1 n k +1 n k -1 n k , f has at least ⌈ n k +1 -n k 2 ⌉ kinks, whose signs are given by sign( δ n k -δ n k -1 ) . Consider any k ≥ 1 such that δ n k -1 = δ n k . Assume w.l.o.g. that δ n k -1 < δ n k . By the definition of Equation (10): .

Obviously, f must count at least one positive kink on each interval of the form 14 ( x j -1 , x j +1 ) for any n k ≤ j ≤ n k +1 -1 . Note that we can build ⌈ n k +1 -n k 2 ⌉ disjoint such intervals. Thus, f has at least ⌈ n k +1 -n k 2 ⌉ positive kinks on ( x n k -1 , x n k +1 ) .

The intervals of the form ( x n k -1 , x n k +1 ) with δ n k -1 < δ n k are disjoint by definition. As a consequence, f has a total number of positive kinks at least

Similarly, f has a total number of negative kinks at least

which leads to the first part of Lemma 4

̸

We now construct an interpolating function that has exactly the desired number of kinks. Note that the problem considered in Lemma 4 is shift invariant (which is not the case of Equation (4)). As a consequence, we can assume without loss of generality that x 1 ≥ 0 . This simplifies the definition of the following sequence of slopes s ∈ S :

̸

It is easy to check that s ∈ S . We now consider the function f associated to the sequence of slopes by the mapping of Lemma 6 and an interval [ x n k -1 , x n k +1 ) with δ n k -1 = δ n k . By definition, s n k +1+2 p = δ n k +1+2 p for any p such that n k +1 ≤ n k +1+2 p < n k +1 . This implies that f has no kink in the interval [ x n k +1+2 p , x n k +2+2 p ) . From there, a simple calculation shows that f has at most

⌈ n k +1 -n k 2 ⌉ kinks on [ x n k , x n k +1 ) . Moreover, as s i = δ i -1 if i = n k , f has no kink on intervals [ x n k , x n k +1 ) when δ n k -1 = δ n k . f is thus an interpolating function with at most

̸

kinks, which concludes the proof of Lemma 4.

E.4 Proof of Theorem 3

̸

Let f be the minimiser of Equation (4). The proof of Theorem 3 separately shows that f has exactly ⌈ n k +1 -n k 2 ⌉ 1 δ n k -1 = δ n k kinks on each ( x n k -1 , x n k +1 ) . Fix in the following k ≥ 0 .

̸

Assume first that δ n k -1 = δ n k . Then Lemma 3 along with the definitions of n k and n k +1 directly imply that s ∗ i = δ n k -1 for any i = n k , . . . , n k +1 -1 . This then implies that the associated interpolator, i.e. f has no kink on ( x n k -1 , x n k +1 ) .

̸

Now assume that δ n k -1 = δ n k . Without loss of generality, assume δ n k -1 < δ n k . By the definition of Equation (10):

Moreover, by definition of n k , we have

Since n k +1 ≤ n k +3 by Assumption 1, Lemma 8 below states that for all the cases, f has exactly ⌈ n k +1 -n k 2 ⌉ kinks on ( x n k -1 , x n k +1 ) .

̸

Symmetric arguments hold if δ n k -1 > δ n k . In conclusion, f has exactly ⌈ n k +1 -n k 2 ⌉ 1 δ n k -1 = δ n k kinks on each ( x n k -1 , x n k +1 ) . This implies that f has at most

̸

kinks in total. This concludes the proof of Theorem 3, thanks to Lemma 4.

Lemma 8. For any k ≥ 0 , if δ n k -1 < δ n k , then the minimiser of Equation (4) f has

1 kink on ( x n k -1 , x n k +1 ) if n k +1 = n k +1 ;

1 kink on ( x n k -1 , x n k +1 ) if n k +1 = n k +2 ;

2 kinks on ( x n k -1 , x n k +1 ) if n k +1 = n k +3 .

Lemma 8 is written in this non-compact way since its proof shows separately (with similar arguments) the three cases.

Proof. 1) Consider n k +1 = n k + 1 . First assume that x n k ≥ 0 . Lemma 3 implies that s ∗ n k +1 ∈ [ δ n k +1 , δ n k ] and s ∗ n k ∈ [ δ n k -1 , δ n k ] . In particular, s ∗ n k ≤ δ n k , which implies that s ∗ n k +1 = δ n k . Similarly, s ∗ n k -1 ≥ δ n k -1 , which implies that s ∗ n k = δ n k -1 . Using the mapping from Lemma 6, both values s ∗ n k and s ∗ n k +1 yield that the associated function f has exactly one kink on ( x n k -1 , x n k +1 ) , which is located at x n k . Similar arguments hold if x n k < 0 .

2) Consider now n k +1 = n k +2 .

First assume that x n k +2 < 0 . Thanks to Lemma 3, we can show similarly to the case 1) that s ∗ n k +1 = δ n k +1 .

Now assume that x n k +2 ≥ 0 . Similarly to the case 1), s ∗ n k +2 = δ n k +1 . The minimisation problem of the slopes becomes on s ∗ i +1 for i = n k :

where ˜ S = S i ( s ∗ i ) if x i +1 ≥ 0 , and ˜ S = { δ i +1 } otherwise. Note that g i +1 ( s, s ∗ i ) is √ 1 + x 2 i +1 -Lipschitz in its first argument, while g i +2 ( δ i +1 , s ) = √ 1 + x 2 i +1 | s -δ i +1 | . Moreover, s ∗ n k ∈ [ δ n k -1 , δ n k ] . As a consequence, either x n k +1 ≥ 0 and s ∗ n k = δ n k = s ∗ n k +1 ; or s ∗ n k +1 = δ n k +1 . Symmetrically, when reasoning on the points x n k -1 , x n k :

either s ∗ n k = δ n k -1 ;

or ( x n k < 0 and s ∗ n k = δ n k = s ∗ n k +1 ).

There are thus two possible cases in the end:

either ( s ∗ n k = δ n k -1 and s ∗ n k +1 = δ n k +1 );

or ( s ∗ n k = δ n k = s ∗ n k +1 and x n k < 0 ≤ x n k +1 ).

In the case where x n k < 0 ≤ x n k +1 , we also have s ∗ n k -1 = δ n k -1 and s ∗ n k +2 = δ n k +1 . A straightforward computation then yields a smaller cost on the functions g i for the choice of slopes s ∗ n k = δ n k -1 and s ∗ n k +1 = δ n k +1 .

As a consequence, s ∗ n k = δ n k -1 and s ∗ n k +1 = δ n k +1 in any case. The mapping of Lemma 6 then yields that f has exactly one kink on ( x n k -1 , x n k +2 ) , which is located in ( x n k , x n k +1 ) . Indeed, we either have a n k -1 = 0 or τ n k -1 = x n k -1 ; similarly either a n k +1 = 0 or τ n k +1 = x n k +2 .

3) Consider now n k +1 = n k +3 . Similarly to the case 2), we have both

When considering all the possible cases, the mapping of Lemma 6 implies that f has exactly two kinks on ( x n k -1 , x n k +3 ) , which are located in [ x n k , x n k +2 ] .

E.5 Adapted analysis for the case x 1 ≥ 0

This section explains how to adapt the analysis of this section to the easier case where all x are positive. Lemma 7 holds under the exact same terms (but its second part is useless) in that case. From there, the proof of Theorem 2 consists in just showing the uniqueness of the minimisation problems for any ˜ s ∈ S :

The unique solution of the first problem is δ i 0 thanks to Lemma 7, while same arguments as in Appendix E.1 hold for the second problem.

For the proof of Lemma 3, the exact same arguments as in Appendix E.2 hold for any i ≥ i 0 +1 . For i = i 0 = 1 , it is obvious in that case that s ∗ 1 = δ 1 , leading to Lemma 3.

Finally, the proof of Theorem 3 follows the same lines when x 1 ≥ 0 .

E.6 Proof of Corollary 1

Proof of Corollary 1. For classification, the natural partition to define is the following:

̸

This partition splits the data so that for any k , y i has a the same value for i ∈ { n k , n k +1 -1 } . Denote K the number of n k defined in Equation (36), i.e., n K = n +1 . From there, by simply noting that any margin classifier has at least a kink in [ x n k , x n k +1 ) for k ∈ [ K -2] :

Similarly to the proof of Lemma 1, we can first show the existence of a minimum. 15 Let us now consider f a minimiser of

Define the set

By continuity of f , we can choose an alternative training set (˜ x i , ˜ y i ) satisfying:

Then, a direct application of Theorem 2 yields that the minimisation problem

admits a unique minimiser, that we denote f reg . But also note that this unique minimiser is also in the constraint set of Equation (37) thanks to Lemma 3, so that

However, since f is in the constraint set of Equation (38), we actually have an equality, and by unicity of the minimiser of Equation (38),

Moreover, it is easy to check that Assumption 1 holds for the data (˜ x i , y i ) i ∈ S , with n k +1 = n k +2 . As a consequence, Theorem 3 implies that the minimiser of Equation (38) is among the sparsest interpolators for the set (˜ x i , y i ) i ∈ S , i.e. it exactly counts K -2 kinks. This then implies that ∥ f ′′ ∥ 0 = K -2 , so that

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers a single hidden layer ReLU network $f_\\theta: \\mathbb{R} \\rightarrow \\mathbb{R}$ with biases in the hidden layer as well as skip connections. In this context the representational cost $R(f)$ required to interpolate a function $f: \\mathbb{R} \\rightarrow \\mathbb{R}$ is studied, which is defined as the infinimum of the 2-norm of the network parameters such that $f_\\theta = f$. Following prior work and in order to consider the larger space of functions that can be approximated to an arbitrary degree by such finite width networks, (potentially) infinite width networks can be parameterized by a measure on the unit sphere, allowing the representation cost to be recast as the total variation of this measure such that $f_\\mu = f$. Unlike prior work, the the bias terms of the hidden layer are included in the norm penalty. As a result of this, for a univariate Lipschitz function $f$ the representation cost is the total variation of $\\sqrt{1-x^2}f''$ instead of just $f''$ when the bias terms are not included. Given a training sample $(x_i, y_i)_{i=1}^n$ the minimum-norm (with respect to the the total variation of $\\sqrt{1-x^2}f''$) of interpolators $f$ such that $f(x_i) = y_i$ are studied. In Theorem 2 it is shown that the minimum-norm interpolator in this context is unique, which is not necessarily the case when the bias terms are not included. Furthermore, assuming the data is such that the linear interpolation between any consecutive set of 6 points is neither convex or concave, then in Theorem 3 it is shown that the minimum norm interpolator has a minimal number of kinks.\n\nSTRENGTHS:\nThis work builds on prior work by studying the impact of including the bias parameters in the 2-norm parameter regularization term. In particular, the min-norm interpolator in this context is unique and the results also suggest including the bias terms perhaps encourages the network towards finding sparser network solutions in terms of the number of kinks or bumps. The work appears technically sound however I did not thoroughly read the appendices. The paper is also well written and structured. The contribution appears tangible if somewhat incremental.\n\nWEAKNESSES:\n1. One of the key findings over prior works is that regularizing the bias terms encourages a simpler interpolating function with only a few kinks. However, this relationship is only proved in a restrictive data setting where the linear interpolation between a small number of consecutive points is neither convex or concave. This feels like quite a restriction, furthermore, from the discussion lines 427-432 of the supplementary material, it seems at least experimentally that when the data does contain convex or concave regions then the minimum norm interpolator and sparsest interpolators are not really related. Therefore I wonder whether this takeaway is really a general principle or more a quirk only exhibited in a highly specific setting. \n\n2. The setting, i.e., univariate data, the necessity for skip connections whose parameters are not included in the penalty is also restrictive, albeit  in line with other works. The focus on studying the min-norm interpolators also does not capture learning.\n\n3. Another minor comment I would have is that the statement to the effect of ``the most important factor behind generalization is the norm of the parameters\" in the introduction is a bit strong in my opinion. I would argue the jury is still out on this, in practice one can fairly often train a network with no explicit regularization and it can do fine and there is also a line of work suggesting implicit regularization cannot be explained in terms of norm minimization, see e.g., https://arxiv.org/pdf/2005.06398.pdf, https://arxiv.org/abs/2012.05156.pdf.\n\nQUESTIONS:\n1. Does one observe learning a smoother function, i.e., one with less kinks or bumps experimentally when the bias term is included versus not included and how does this vary with say the regularization parameter as well as the smoothness of the target function?\n\n2. If one considers deeper networks or multivariate inputs or a different loss function etc. is there still evidence of a tendency towards learning a function with fewer bumps when bias terms are included in the penalty?\n\n3. Empirically simulating across a number of trials what is the distribution of the learned function look like? Does it concentrate around the minimum norm interpolator?\n\n3. In general how sensitive are the findings of this work to say to changes in network architecture (e.g., different activation function), moving to higher dimensional inputs, changing the loss function from squared error to cross-entropy say?\n\nLIMITATIONS:\nA brief discussion of limitations is provided in the conclusion avenues for future work are identified. I cannot foresee any negative societal outcomes from this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper studies the representational cost of a toy ReLU MLP with a single hidden layer, and a scalar input. The main result of the paper is that, when regularizing both the weights and the biases, the minimum cost functions are min norm interpolators which are also sparse (defined in the paper as having the minimal number of kinks). When the bias is not regularized (as in the standard weight decay), the minimum cost functions are not necessarily sparse.

STRENGTHS:
Overall i enjoyed reading this paper. Its main strengths are:
1. Very well written, and positioned relative to prior work.
2. A rigorous, clean result, albeit on a very toyish model
2. A novel result that is somewhat surprising and non-trivial.

WEAKNESSES:
The main drawbacks of the paper are:
1. The studied model is very simple, even for a theory paper (1D input)
2. Its not clear whether the main conclusion of the paper has any bearing in practice. Therefore, its impact might be restricted to the theoretically inclined.

These drawbacks are somewhat nitpicky though, and i do not see a reason not to accept it.

QUESTIONS:
Although the sparsest interpolator is not the only solution when the bias is not regularized, we do see (figure 3b) that standard weight decay does produce sparse (almost sparsest) interpolators. Is it possible that in practice the effect of regularizing the bias is not that pronounced?

LIMITATIONS:
Limitations are adequately addressed
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses two main questions: (1) Can the parameters' norm be understood in terms of a quantity that is more insightful from a functional analysis point of view? (2) What kind of functions are learned when minimizing the empirical loss, either with an explicit or implicit regularization of the parameters' norm? The paper provides an answer to these questions for univariate functions when the bias parameters are regularized. It shows that penalizing the bias terms enforces sparsity and uniqueness of the estimated function, which does not hold without penalization of the biases. The paper provides a theorem (see Theorem 1) that states the representational cost of a function is given by the total variation of its second derivative, weighted by an sqrt(1 + x^2) factor. This weighting does favor sparse estimators when training neural networks.\n\nSTRENGTHS:\n- It is well-known that omitting the bias parameters of the NN makes the analysis easier. So, the fact that the bias terms are included is highly welcomed. There are interesting conclusions garnished from including the bias terms, such as the enforcement of sparsity. \n\n- The paper shows that the representational cost of a function is given by the total variation of its second derivative, weighted by an sqrt(1 + x^2) factor.\n\nWEAKNESSES:\n- The work is only for univariate functions, mainly for ReLU NNs with skip connections, penalization terms (that are not common in practice), and continuous dynamics for training. Therefore, the theoretical study here may say very little about the answers to the manuscript's two main questions for practical NNs.  However, I acknowledge that it may be extremely hard to do analysis when the setup is modified and the authors provide empirical evidence. \n\n- A skip connection is a useful architecture to make the analysis easier; however, whether it says anything about the NNs that are used in practice is unclear to me. The author says: \"Since a skip connection can be represented by two ReLU neurons, it is commonly believed that considering a free skip connection does not alter the nature of the obtained results.\"; however, I am not aware of a theoretical statement that makes it precise in this setting.\n\nQUESTIONS:\n1. I wonder if there is anything special to the weight sqrt(1 + x^2) from a higher level and how particular this result is to the setup considered in the paper, e.g., Relu, Skip connection, 1D setup, and one hidden neural network. \n\n2. The definition of the parameter norm is the squared $\\ell_2$ norm of all the {a_j}, {b_j}, and {w_j}. While in 1D, we can consider them as 3 different vectors, but in higher dimensions, I am not sure about viewing them as vectors is the correct way of defining the parameter norm. Also, when it comes to sparsity, one would compute the number of its nonzero elements, an alternative of which is the \\ell_1 norm. Using the squared $\\ell_2$ norm as a sparsity measure is an uncommon choice since a neural network can be non-sparse at all but with a very small parameter squared $\\ell_2$ norm.\n\n3. It is not immediately clear from the paper what type of function f being considered in the main result. It was said that the second derivative of f, i.e., f'' should be understood in terms of distribution, which I assume that non-smooth 1D functions are considered. Based on the 2nd definition of \\bar{R}_1(f), f is restricted to those represented by a simple Relu NN.\n\n4. It is written that the two \\bar{R}_1(f) in Section 2.1 can be proven to be the same. Can the authors provide proof or pointing some references to it?\n\nLIMITATIONS:\nThere are some comments on the future directions of this work in the conclusion, but I am not sure they are classified as limitations of this work.\n\nHere is the main limitation in my perspective of the paper:\n\n1. The analysis is restricted to a 1D setting with a very particular structure for the neural network. The weight sqrt(1 + x^2) is too particular, and I am not confident that one can return the same result with a slight change in the problem setup.\n\n2. The implication of minimizing the TV norm of sqrt(1 + x^2) f''(x) is not well understood, and its link to sparsity promoting is not so clear.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper theoretically investigates the effect of regularizing bias parameters’ norm, mostly ignored or omitted in previous analyses, for univariate one hidden layer ReLU networks. It turns out that minimizing $L_2$ norms for both weights and biases corresponds to regularizing a functional norm defined as the total variation of the second derivative of the function modeled by the neural network with a weight of $\\sqrt{1+x^2}$ factor, where the weight factor does not exist when regularizing only the weight norm. The minimal functional norm interpolator can be obtained from a proposed dynamic programming algorithm, and the uniqueness and sparsity properties of the interpolator are discussed.\n\n**Post rebuttal** I have increased my score to 5 from 4. I can appreciate the theoretical contribution, while the practical relevance still seems to be a weakness.\n\nSTRENGTHS:\n- The paper analyzes the effect of bias norm regularization, which has not been considered much before, showing novel and original results. The effect is not trivial, and the regularization problem is identified as minimizing a functional norm with a weight factor, which leads to interesting uniqueness and sparsity properties of minimum norm interpolators.\n- Proofs are provided for most of the theorems and lemmas in a mathematically sound way. It is especially ingenious to prove the uniqueness and sparsity properties of the minimum norm interpolator by using a dynamic programming algorithm that obtains the interpolator.\n\nWEAKNESSES:\n- The benefit of the considered sparsity regarding the number of a function’s kinks is unclear. Why would the number of kinks be important? Can it be used to explain generalization properties in any way? Also, the sparsity in terms of kinks seems quite different from the one considered in usual machine learning contexts, e.g., having zero values for most weights or representations except for a few. Maybe addressing these points and providing some intuitive explanations would be helpful to enforce the claim of the paper.\n- The condition for the minimum norm interpolator to be the sparsest interpolator is quite restrictive according to **Assumption 1**. Moreover, **Appendix A** provides an example of when the sparsest and minimal norm ones are largely different in the number of kinks. These points can lower the significance of the sparsity result.\n- Due to the weight factor of $\\sqrt{1+x^2}$ in evaluating $\\mathcal{F}_1$ norm from bias regularization, the shift invariance property of the norm disappears. Can this have any negative impact on using the function norm to explain the nature of a function? Also, the connection between this weight factor and the sparsity property regarding the number of kinks is not intuitive. Rather, it seems that containing this weight factor would make the location of kinks not far from zero.\n\nQUESTIONS:\nIn addition to the questions posed in the **Weaknesses** section, I have the following questions:\n- What would be the practical benefit of considering bias regularization? As far as I know, a common practice for applying weight decay in neural network training is only regularizing the weight norm. This method has already shown many empirical successes so far.\n\n**[Minor comments]**\n- How is the ‘activation cone’ defined in line 188?\n- Providing some explanations about deriving (5) would be helpful.\n- **Corollary 1** is corrected in **Appendix E.6**, and I think the corrected version should come to the main text.\n\nLIMITATIONS:\nThe limitations of the results are discussed in **Conclusion** and **Appendix A**. For other possible limitations, please refer to the weaknesses and questions above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-26"}