SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\min  _ {\substack {f \\ \forall i \in [ n ], f (x _ {i}) = y _ {i}}} \left\| \sqrt {1 + x ^ {2}} f ^ {\prime \prime} \right\| _ {\text {TV}} = \min  _ {(s _ {i _ {0} - 1}, s _ {i _ {0}}) \in \Lambda} g _ {i _ {0}} \left(s _ {i _ {0}}, s _ {i _ {0} - 1}\right) + c _ {i _ {0} - 1} \left(s _ {i _ {0} - 1}\right) + c _ {i _ {0}} \left(s _ {i _ {0}}\right) \tag{5}
$$
where the set  $\Lambda$  and the functions  $g_{i}$  and  $c_{i}$  are defined in Equations (6) to (8) below.
Let us describe the dynamic program defining the functions  $c_{i}$ , which characterises the minimal norm interpolator thanks to Lemma 2. First define for any  $i \in [n - 1]$ , the slope  $\delta_{i} \coloneqq \frac{y_{i + 1} - y_{i}}{x_{i + 1} - x_{i}}$ ; the function
$$
g _ {i + 1} \left(s _ {i + 1}, s _ {i}\right) := \sqrt {\left(x _ {i + 1} \left(s _ {i + 1} - \delta_ {i}\right) - x _ {i} \left(s _ {i} - \delta_ {i}\right)\right) ^ {2} + \left(s _ {i + 1} - s _ {i}\right) ^ {2}} \text {f o r a n y} \left(s _ {i + 1}, s _ {i}\right) \in \mathbb {R} ^ {2}; \tag {6}
$$
$$
\text {a n d t h e i n t e r v a l s} \quad S _ {i} (s) := \left\{ \begin{array}{l} (- \infty , \delta_ {i} ] \text {i f} s > \delta_ {i} \\ \{\delta_ {i} \} \text {i f} s = \delta_ {i} \\ [ \delta_ {i}, + \infty) \text {i f} s <   \delta_ {i} \end{array} \right. \qquad \text {f o r a n y} s \in \mathbb {R}.
$$
The set  $\Lambda$  is then the union of three product spaces given by
$$
\Lambda := (- \infty , \delta_ {i _ {0} - 1}) \times (\delta_ {i _ {0} - 1}, + \infty) \cup \left\{\left(\delta_ {i _ {0} - 1}, \delta_ {i _ {0} - 1}\right) \right\} \cup \left(\delta_ {i _ {0} - 1}, + \infty\right) \times (- \infty , \delta_ {i _ {0} - 1}). \tag {7}
$$
Finally, we define the functions  $c_{i}:\mathbb{R}\to \mathbb{R}_{+}$  recursively as  $c_{1} = c_{n}\equiv 0$  and
$$
c _ {i + 1}: s _ {i + 1} \mapsto \min  _ {s _ {i} \in S _ {i} (s _ {i + 1})} g _ {i + 1} \left(s _ {i + 1}, s _ {i}\right) + c _ {i} \left(s _ {i}\right) \quad \text {f o r a n y} i \in \{1, \dots , i _ {0} - 2 \}
$$
$$
c _ {i}: s _ {i} \mapsto \min  _ {s _ {i + 1} \in S _ {i} (s _ {i})} g _ {i + 1} \left(s _ {i + 1}, s _ {i}\right) + c _ {i + 1} \left(s _ {i + 1}\right) \quad \text {f o r a n y} i \in \left\{i _ {0}, \dots , n - 1 \right\}. \tag {8}
$$
Equation (8) defines a dynamic program with a continuous state space. Intuitively for  $i \geq i_0$ , the variable  $s_i$  accounts for the left derivative at the point  $x_i$ . The term  $g_{i+1}(s_{i+1}, s_i)$  is the minimal cost (in neuron norm) for reaching the point  $(x_{i+1}, y_{i+1})$  with a slope  $s_{i+1}$ , knowing that the left slope is  $s_i$  at the point  $(x_i, y_i)$ . Similarly, the interval  $S_i(s_i)$  gives the reachable slopes at  $x_{i+1}$ , knowing the slope in  $x_i$  is  $s_i$ . Finally,  $c_i(s_i)$  holds for the minimal cost of fitting all the points  $(x_{i+1}, y_{i+1}), \ldots, (x_n, y_n)$  when the left derivative in  $(x_i, y_i)$  is given by  $s_i$ . It is defined recursively by minimising the sum of the cost for reaching the next point  $(x_{i+1}, y_{i+1})$  with a slope  $s_{i+1}$ , given by  $g_{i+1}(s_{i+1}, s_i)$ ; and the cost of fitting all the points after  $x_{i+1}$ , given by  $c_{i+1}$ . This recursive definition is illustrated in Figure 1 below. A symmetric definition holds for  $i < i_0$ .
The idea to derive Equation (6) is to first use Lemma 1 to get a finite representation of a minimal function  $f^{*}$ . From there, the minimal cost for connecting the point  $(x_{i + 1},y_{i + 1})$  with  $(x_{i},y_{i})$  is done by using a single kink in between. The restrictions given by  $s_i$  and  $s_{i + 1}$  then yield a unique possible kink. Minimizing its neuron norm then yields Equation (6)
Remark 1. Equation (5) actually considers the junction of two dynamic programs: a first one corresponding to the points with negative  $x$  values and a second one for positive values. This separation around  $x = 0$  is not needed for Lemma 2, but allows for a cleaner analysis in Section 5. Lemmas 1 and 2 also hold for any arbitrary choice of  $i_0$ . In particular for  $i_0 = 1$ , Equation (5) would not consider the junction of two dynamic programs anymore, but a single one.
4See Equation (21) in the Appendix for a mathematical definition.
Here, a single kink is used in the interval  $[x_i, x_{i+1}]$ , thanks to Lemma 3.
Figure 1: Recursive definition of the dynamic program for  $i \geq i_0$ .
Remark 2. The assumption  $x_{1} < 0$  and  $x_{n} \geq 0$  is not fundamental, but is only required to properly define the junction mentioned in Remark 1. If all the  $x$  values are positive (or negative by symmetry), the analysis of the right term in Equation (5) is simplified, since there is no junction to consider. In particular, all the results from Section 5 hold without this assumption. These results are proven in the hardest case  $x_{1} < 0$  and  $x_{n} \geq 0$  in Appendix E, from which other cases can be directly deferred.
Lemma 2 formulates the minimisation of the representational cost among the interpolating functions as a simpler dynamic program on the sequence of slopes at each  $x_{i}$ . This equivalence is the key technical result of this work, from which Section 5 defers many properties on the minimiser(s) of Equation (4).
5 Properties of minimal norm interpolator
Thanks to the dynamic program formulation given by Lemma 2, this section derives key properties on the interpolating functions of minimal representational cost. In particular, it shows that Equation (4) always admits a unique minimum. Moreover, under some condition on the training set, this minimising function has the smallest number of kinks among the set of interpolators.
Theorem 2. The following optimisation problem admits a unique minimiser:
$$
\inf  _ {f} \left\| \sqrt {1 + x ^ {2}} f ^ {\prime \prime} \right\| _ {\mathrm {T V}} \quad s u c h t h a t \forall i \in [ n ],   f (x _ {i}) = y _ {i}.
$$
The proof of Theorem 2 uses the correspondence between interpolating functions and sequences of slopes  $(s_i)_{i\in [n]}\in S$ , where the set  $S$  is defined by Equation (23) in Appendix D.2. In particular, we show that the following problem admits a unique minimiser:
$$
\min  _ {\mathbf {s} \in \mathcal {S}} \sum_ {i = 1} ^ {n - 1} g _ {i + 1} \left(s _ {i + 1}, s _ {i}\right). \tag {9}
$$
We note in the following  $\mathbf{s}^* \in S$  the unique minimiser of the problem in Equation (9). From this sequence of slopes  $\mathbf{s}^*$ , the unique minimising function of Equation (4) can be recovered. Moreover,  $\mathbf{s}^*$  minimises the dynamic program given by the functions  $c_i$  as follows:
$$
c _ {i + 1} \left(s _ {i + 1} ^ {*}\right) = g _ {i + 1} \left(s _ {i + 1} ^ {*}, s _ {i} ^ {*}\right) + c _ {i} \left(s _ {i} ^ {*}\right) \quad \text {f o r a n y} i \in \left[ i _ {0} - 2 \right]
$$
$$
c _ {i} \left(s _ {i} ^ {*}\right) = g _ {i + 1} \left(s _ {i + 1} ^ {*}, s _ {i} ^ {*}\right) + c _ {i + 1} \left(s _ {i + 1} ^ {*}\right) \quad \text {f o r a n y} i \in \left\{i _ {0}, \dots , n - 1 \right\}.
$$
Using simple properties of the functions  $c_{i}$  given by Lemma 7 in Appendix E, properties on  $\mathbf{s}^*$  can be derived besides the uniqueness of the minimal norm interpolator. Lemma 3 below gives a first intuitive property of this minimiser, which proves helpful in showing the main result of the section.
Lemma 3. For any  $i \in [n]$ ,  $s_i^* \in [\min(\delta_{i-1}, \delta_i), \max(\delta_{i-1}, \delta_i)]$ , where  $\delta_0 := \delta_1$  and  $\delta_n := \delta_{n-1}$  by convention.
A geometric interpretation of Lemma 3 is that the optimal (left or right) slope in  $x_{i}$  is between the line joining  $(x_{i - 1},y_{i - 1})$  with  $(x_{i},y_{i})$  and the line joining  $(x_{i},y_{i})$  with  $(x_{i + 1},y_{i + 1})$ .
5.1 Recovering a sparsest interpolator
We now aim at characterising when the minimiser of Equation (4) is among the set of sparsest interpolators, in terms of number of kinks. Before describing the minimal number of kinks required
to fit the data in Lemma 4, we partition  $[x_1,x_n)$  into intervals of the form  $[x_{n_k},x_{n_{k + 1}})$  where
$$
n _ {0} = 1 \text {a n d f o r a n y} k \geq 0 \text {s u c h t h a t} n _ {k} <   n,
$$
$$
n _ {k + 1} = \min  \left\{j \in \left\{n _ {k} + 1, \dots , n - 1 \right\} \mid \operatorname {s i g n} \left(\delta_ {j} - \delta_ {j - 1}\right) \neq \operatorname {s i g n} \left(\delta_ {j - 1} - \delta_ {j - 2}\right) \right\} \cup \{n \}, \tag {10}
$$
and  $\mathrm{sign}(0) := 0$  by convention. If we note  $f_{\mathrm{lin}}$  the canonical piecewise linear interpolator, it is either convex, concave or affine on every interval  $[x_{n_k - 1}, x_{n_{k + 1}}]$ . This partitioning thus splits the space into convex, concave and affine parts of  $f_{\mathrm{lin}}$ , as illustrated by Figure 2 on a toy example. This partition is crucial in describing the sparsest interpolators, thanks to Lemma 4.
Figure 2: Partition given by  $(n_k)_k$  on a toy example.
Lemma 4. If we denote by  $\| f''\| _0$  the cardinality of the support of the measure  $f^{\prime \prime}$
$$
\min_{\substack{f\\\forall i,f(x_{i}) = y_{i}}}\| f^{\prime \prime}\|_{0} = \sum_{k\geq 1}\left\lceil \frac{n_{k + 1} - n_{k}}{2}\right\rceil \mathbb{1}_{\delta_{n_{k} - 1}\neq \delta_{n_{k}}}.
$$
Lemma 4's proof idea is that for any interval  $[x_{k - 1}, x_{k + 1})$  where  $f_{\mathrm{lin}}$  is convex (resp. concave) non affine, any function requires at least one positive (resp. negative) kink to fit the three data points in this interval. The result then comes from counting the number of such disjoint intervals and showing that a specific interpolator exactly reaches this number.
The minimal number of kinks required to interpolate the data is given by Lemma 4. Before giving the main result of this section, we introduce the following assumption on the data  $(x_{k},y_{k})_{k\in [n]}$ .
Assumption 1. For the sequence  $(n_k)_k$  defined in Equation (10):
$$
n _ {k + 1} - n _ {k} \leq 3 \text {o r} \delta_ {n _ {k}} = \delta_ {n _ {k} - 1} \quad f o r a n y k \geq 0.
$$
Assumption 1 exactly means there are no 6 (or more) consecutive points  $x_{k}, \ldots, x_{k+5}$  such that  $f_{\mathrm{lin}}$  is convex (without 3 aligned points) or concave on  $[x_{k}, x_{k+5}]$ . This assumption depends a lot on the structure of the true model function (if there is any). For example, it holds if the truth is given by a piecewise linear function, while it may not if the truth is given by a quadratic function. Theorem 3 below shows that under Assumption 1, the minimal cost interpolator is amongst the sparsest interpolators, in number of its kinks.
Theorem 3. If Assumption 1 holds, then
$$
\underset {\forall i, f \left(x _ {i}\right) = y _ {i}} {\operatorname {a r g m i n}} \| \sqrt {1 + x ^ {2}} f ^ {\prime \prime} \| _ {\mathrm {T V}} \in \underset {\forall i, f \left(x _ {i}\right) = y _ {i}} {\operatorname {a r g m i n}} \| f ^ {\prime \prime} \| _ {0}. \tag {11}
$$
Theorem 3 states conditions under which the interpolating function  $f$  with the smallest representational cost  $\overline{R}_1(f)$  also has the minimal number of kinks, i.e. ReLU hidden neurons, among the set of interpolators. It illustrates how norm regularisation, and in particular adding the biases' norm to the weight decay, favors estimators with a small number of neurons. While training neural networks with norm regularisation, the final estimator can actually have many non-zero neurons, but they all align towards a few key directions. As a consequence, the obtained estimator is actually equivalent to a small width network, meaning they have the same output for every input  $x \in \mathbb{R}$ .
Recall that such a sparsity does not hold when the bias terms are not regularised. More precisely, some sparsest interpolators have a minimal representational cost in that case, but there are also minimal cost interpolators with an arbitrarily large (even infinite) number of kinks [Debarre et al., 2022]. There is thus no particular reason that the obtained estimator is sparse when minimising
the representational cost without penalising the bias terms. Section 6 empirically illustrates this difference of sparsity in the recovered estimators, depending on whether or not the bias parameters are penalised in the norm regularisation.
The generalisation benefit of this sparsity remains unclear. Indeed, generalisation bounds in the literature often rely on the parameters' norm rather than the network width (i.e., sparsity level). The relation between sparsest and min norm interpolators is important to understand in the particular context of implicit regularisation. In particular, while Boursier et al. [2022] conjectured that the implicit bias for regression problem was towards min norm interpolators, Chistikov et al. [2023] recently proved that the implicit bias could sometimes instead lead to sparsest interpolators. Our result suggests that both min norm and sparsest interpolators often coincide, which could explain the prior belief of convergence towards min norm interpolators. Yet, Theorem 3 and Chistikov et al. [2023] instead suggest that, at least in some situations, implicit bias favors sparsest interpolators, yielding different estimators<sup>6</sup>.
Remark 3. Theorem 3 states that sparse recovery, given by Equation (11), occurs if Assumption 1 holds. When  $n_{k+1} - n_k \geq 4$ , i.e., there are convex regions of  $f_{\mathrm{lin}}$  with at least 6 points, Appendix A gives a counterexample where Equation (11) does not hold. However, Equation (11) can still hold under weaker data assumptions than Assumption 1. In particular, Appendix A gives a necessary and sufficient condition for sparse recovery when there are convex regions with exactly 6 points. When we allow for convex regions with at least 7 points, it however becomes much harder to derive conditions where sparse recovery still occurs.
Remark 4. The counterexample presented in Appendix A reveals an unexpected outcome: minimal representational cost interpolators may not necessarily belong to the sparest interpolators. This finding is closely related to the idea that it may not be generally feasible to characterize the implicit regularisation of gradient descent as minimising parameters norm [Vardi and Shamir, 2021, Chistikov et al., 2023]. In particular, Vardi and Shamir [2021], Chistikov et al. [2023] rely on examples where minimal norm interpolators are not the sparsest ones; and the implicit regularisation instead favors the latter. We believe that this inherent limitation is one of the underlying reason for the different implicit regularization effects observed in other settings such as matrix factorization [Gunasekar et al., 2017, Razin and Cohen, 2020, Li et al., 2020a].
5.2 Application to classification
In the binary classification setting, max-margin classifiers, defined as the minimiser of the problem
$$
\min  _ {f} \bar {R} (f) \quad \text {s u c h t h a t} \forall i \in [ n ], y _ {i} f \left(x _ {i}\right) \geq 1, \tag {12}
$$
are known to be the estimators of interest. Indeed, gradient descent on the cross entropy loss  $l(\hat{y}, y) = \log (1 + e^{-\hat{y} y})$  converges in direction to such estimators [Lyu and Li, 2019, Chizat and Bach, 2020]. Theorem 3 can be used to characterise max- margin classifiers, leading to Corollary 1.
Corollary 1.
$$
\operatorname *{argmin}_{\substack{f\\ \forall i\in [n],y_{i}f(x_{i})\geq 1}}\overline{R}_{1}(f)\subset \operatorname *{argmin}_{\substack{f\\ \forall i\in [n],y_{i}f(x_{i})\geq 1}}\| f^{\prime \prime}\|_{0}.
$$
Theorem 3 yields that the max-margin classifier is among the sparsest margin classifiers, when a free skip connection is allowed. We believe that the left minimisation problem admits a unique solution. However, uniqueness cannot be directly derived from Theorem 3, but would instead require another thorough analysis, using an adapted dynamic programming reformulation. Since the uniqueness property is of minor interest, we here prefer to focus on a direct corollary of Theorem 3. We emphasise that no data assumptions are required for classification tasks, apart from being univariate.
This section compares, through Figure 3, the estimators that are obtained with and without counting the bias terms in the regularisation, when training a one-hidden ReLU layer neural network. The code is made available at github.com/eboursier/penalising_biases.
Note that this nuance only holds for regression tasks. Instead, it is known that implicit regularisation favors min norm interpolators in classification tasks [Chizat and Bach, 2020], which always coincide with sparsest interpolators in that case (see Corollary 1 in Section 5.2) for univariate data.
For this experiment, we train neural networks by minimising the empirical loss, regularised with the  $\ell_2$  norm of the parameters (either with or without the bias terms) with a regularisation factor  $\lambda = 10^{-3}$ . Each neural network has  $m = 200$  hidden neurons and all parameters are initialised i.i.d. as centered Gaussian variables of variance  $1 / \sqrt{m}$  (similar results are observed for larger initialisation scales). There is no free skip connection here, which illustrates its benignity: the results that are expected by the above theory also happen without free skip connection. Experiments with a free skip connection are given in Appendix B and yield similar observations.
(a) Penalising bias terms in the  $\ell_2$  regularisation.
(b) Ignoring the bias terms in the  $\ell_2$  regularisation.
Figure 3: Final estimator when training one-hidden layer network with  $\ell_2$  regularisation. The green dots correspond to the data and the green line is the estimated function. Each blue star represents a hidden neuron  $(w_j, b_j)$  of the network: its  $x$ -axis value is given by  $-b_j / w_j$ , which coincides with the position of the kink of its associated ReLU; its  $y$ -axis value is given by the output weight  $a_j$ .
As predicted by our theoretical study, penalising the bias terms in the  $\ell_2$  regularisation enforces the sparsity of the final estimator. The estimator of Figure 3a indeed counts 2 kinks (the smallest number required to fit the data), while in Figure 3b, the directions of the neurons are scattered. More precisely, the estimator is almost smooth near  $x = -0.5$ , while the sparse estimator of Figure 3a is clearly not differentiable at this point. Also, the estimator of Figure 3b includes a clear additional kink at  $x = 0$ . Figure 3 thus illustrates that counting the bias terms in regularisation can lead to sparser estimators.
This work studies the importance of parameters' norm for one hidden ReLU layer neural networks in the univariate case. In particular, the parameters' norm required to represent a function is given by  $\left\| \sqrt{1 + x^2} f'' \right\|_{\mathrm{TV}}$  when allowing for a free skip connection. In comparison to weight decay, which omits the bias parameters in the norm, an additional  $\sqrt{1 + x^2}$  weighting term appears in the representational cost. This weighting is of crucial importance since it implies uniqueness of the minimal norm interpolator. Moreover, it favors sparsity of this interpolator in number of kinks. Minimising the parameters' norm (with the biases), which can be either obtained by explicit or implicit regularisation when training neural networks, thus leads to sparse interpolators. We believe this sparsity is a reason for the good generalisation properties of neural networks observed in practice.
Although these results provide some understanding of minimal norm interpolators, extending them to more general and difficult settings remains open. Even if the representational cost might be described in the multivariate case [as done by Ongie et al., 2019, without bias penalisation], characterising minimal norm interpolators seems very challenging in that case. Characterising minimal norm interpolators, with no free skip connection, also presents a major challenge for future work.
For small initialisations, both methods yield sparse estimators, since implicit regularisation of the bias terms is significant in that case. Our goal is only to illustrate the differences in the minimisers of the two problems (with and without bias penalisation), without any optimisation consideration.
The authors thank Gal Vardi, for suggesting the proof of Corollary 1, through a direct use of Theorem 3. The authors also thank Claire Boyer, Julien Fageot and Loucas Pillaud-Vivien for very helpful discussions and feedback.
Francis Bach. Breaking the curse of dimensionality with convex neural networks. The Journal of Machine Learning Research, 18(1):629-681, 2017.
Andrew R Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory, 39(3):930-945, 1993.
Peter Bartlett. For valid generalization the size of the weights is more important than the size of the network. Advances in neural information processing systems, 9, 1996.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849-15854, 2019.
Yoshua Bengio, Nicolas Roux, Pascal Vincent, Olivier Delalleau, and Patrice Marcotte. Convex neural networks. Advances in neural information processing systems, 18, 2005.
Etienne Boursier, Loucas Pillaud-Vivien, and Nicolas Flammarion. Gradient flow dynamics of shallow relu networks for square loss and orthogonal inputs. arXiv preprint arXiv:2206.00939, 2022.
Emmanuel J Candès and Carlos Fernandez-Granda. Towards a mathematical theory of superresolution. Communications on pure and applied Mathematics, 67(6):906-956, 2014.
Scott Shaobing Chen, David L Donoho, and Michael A Saunders. Atomic decomposition by basis pursuit. SIAM review, 43(1):129-159, 2001.
Dmitry Chistikov, Matthias Englert, and Ranko Lazic. Learning a neuron by a shallow relu network: Dynamics and implicit bias for correlated inputs. arXiv preprint arXiv:2306.06479, 2023.
Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory, pages 1305-1338. PMLR, 2020.
Thomas Debarre, Quentin Denoyelle, Michael Unser, and Julien Fageot. Sparsest piecewise-linear regression of one-dimensional data. Journal of Computational and Applied Mathematics, 406: 114044, 2022.
Weinan E and Stephan Wojtowytsch. Representation formulas and pointwise properties for barron functions, 2021.
Carlos Fernandez-Granda. Super-resolution of point sources via convex programming. Information and Inference: A Journal of the IMA, 5(3):251-303, 2016.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. *Deep Learning*. MIT Press, 2016. http://www.deeplearningbook.org.
Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in Neural Information Processing Systems, 30, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Ziwei Ji and Matus Telgarsky. The implicit bias of gradient descent on nonseparable data. In Conference on Learning Theory, pages 1772-1798. PMLR, 2019.
Yifei Wang, Jonathan Lacotte, and Mert Pilanci. The hidden convex optimization landscape of regularized two-layer relu networks: an exact characterization of optimal solutions. In International Conference on Learning Representations, 2021.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
A Discussing Assumption 1
Theorem 3 requires Assumption 1, which assumes that there are no convex (or concave) regions of  $f_{\mathrm{lin}}$  with at least 6 data points. Actually, when there is a convex (or concave) region with exactly 6 data points, i.e.  $n_{k + 1} = n_k + 4$ , Theorem 3 holds (for this region) if and only if for  $i = n_k + 1$ :