SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {\left\langle u _ {i} , w _ {i - 1} \right\rangle \left\langle u _ {i + 1} , w _ {i + 1} \right\rangle}{\| w _ {i - 1} \| \| w _ {i + 1} \|} - \left\langle u _ {i}, u _ {i + 1} \right\rangle \leq \sqrt {\| u _ {i} \| ^ {2} - \frac {\left\langle u _ {i} , w _ {i - 1} \right\rangle^ {2}}{\| w _ {i - 1} \| ^ {2}}} \sqrt {\| u _ {i + 1} \| ^ {2} - \frac {\left\langle u _ {i + 1} , w _ {i + 1} \right\rangle^ {2}}{\| w _ {i + 1} \| ^ {2}}} \tag {13}
$$
where  $u_{i} = (x_{i},1)$  .  $w_{i - 1} = \frac{\delta_i - \delta_{i - 1}}{\delta_i - \delta_{i - 2}} (x_i,1) + \frac{\delta_{i - 1} - \delta_{i - 2}}{\delta_i - \delta_{i - 2}} (x_{i - 1},1);$
and  $u_{i + 1} = (x_{i + 1},1)$  .  $w_{i + 1} = \frac{\delta_{i + 2} - \delta_{i + 1}}{\delta_{i + 2} - \delta_i} (x_{i + 2},1) + \frac{\delta_{i + 1} - \delta_i}{\delta_{i + 2} - \delta_i} (x_{i + 1},1).$
The proof of this result (omitted here) shows that the problem
$$
\min  _ {(s _ {i}, s _ {i + 1}) \in [ \delta_ {i - 1}, \delta_ {i} ] \times [ \delta_ {i}, \delta_ {i + 1} ]} g _ {i} (s _ {i}, s _ {i - 1} ^ {*}) + g _ {i + 1} (s _ {i + 1, s _ {i}}) + g _ {i + 2} (s _ {i + 2} ^ {*}, s _ {i + 1})
$$
is minimised for  $(s_i, s_{i+1}) = (\delta_i, \delta_i)$  if and only if Equation (13) holds, which corresponds to the (unique) sparsest way to interpolate the data on this convex region. To show that the minimum is reached at that point, we can first notice that  $s_{i-1}^* = \delta_{i-2}$  and  $s_{i+2}^* = \delta_{i+2}$ . Then, it requires a meticulous study of the directional derivatives of the (convex but non-differentiable) objective function at the point  $(\delta_i, \delta_i)$ .
Figure 4 below illustrates a case of 6 data points, where the condition of Equation (13) does not hold. Clearly, the minimal norm interpolator differs from the (unique) sparsest interpolator in that case.
Figure 4: Case of difference between minimal norm interpolator and sparsest interpolator.
When considering more than 6 points, studying the minimisation problem becomes cumbersome and no simple condition of sparse recovery can be derived. When generating random data with large convex regions, e.g. 35 points, the minimal norm interpolator is rarely among the sparsest interpolators. Moreover, it seems that its number of kinks could be arbitrarily close to 34, which is the trivial upper bound of the number of kinks given by Lemma 3; while the sparsest interpolators only have 17 kinks.
Figure 5 shows the minimiser of Equation (4) on the toy example of Figure 2. The minimising function is computed thanks to the dynamic program given by Lemma 2. Although the variables of this dynamic program are continuous, we can efficiently solve it by approximating the constraint space of each slope  $s_i$  as a discrete grid of  $[\delta_{i-1}, \delta_i]$  thanks to Lemma 3. For the data used in Figure 5, Assumption 1 holds. It is clear that the minimiser is very sparse, counting only 4 kinks. The partition given by Figure 2 then shows that this is indeed the smaller possible number of kinks, thanks to Lemma 4. On the other hand, the canonical piecewise linear interpolator  $f_{\mathrm{lin}}$  is not as sparse and counts 7 kinks here.
Figure 5: Minimiser of Equation (4) on a toy data example.
(a) Final estimator when penalising bias terms in the  $\ell_2$  regularisation.
Figure 6: Final estimators when training one-hidden ReLU neural networks with  $\ell_2$  regularisation. The green dots correspond to the data, while the green line is the estimated function. Each blue star represents a hidden neuron  $(w_j, b_j)$  of the network: its  $x$ -axis value is given by  $-b_j / w_j$ , which coincides with the position of the kink of its associated ReLU; its  $y$ -axis value is given by the output layer weight  $a_j$ .
Figure 6 considers the exact same setting as Figure 3 in Section 6. The only difference is that we here allow a free skip connection in the neural network architecture, which represents the setting exactly described by Theorem 3.
(b) Final estimators when training one-hidden ReLU neural networks with  $\ell_2$  regularisation and a free skip connection.
Similar observations can be made: the obtained estimator when counting the bias terms in regularisation only has 2 kinks, while the estimator obtained by omitting the biases in the regularisation is much smoother (and thus much less sparse in the number of kinks). The only difference is that the latter estimator here does not have a clear kink at  $x = 0$ , but is instead even smoother on the interval  $[-0.5, 0]$ . This is explained by the presence of more scattered kinks in this interval. Despite this slight difference, the main observation remains unchanged: the estimator is a sparsest one when counting the bias terms, while it counts a lot of kinks (and is even smooth) when omitting the biases.
Theorem 4 below extends the characterisation of the representational cost  $\overline{R}_1(f)$  of Theorem 1.
Theorem 4. For any Lipschitz function  $f: \mathbb{R} \to \mathbb{R}$ ,
$$
\overline {{R}} _ {1} (f) = \left\| \sqrt {1 + x ^ {2}} f ^ {\prime \prime} \right\| _ {\mathrm {T V}} = \int_ {\mathbb {R}} \sqrt {1 + x ^ {2}} \mathrm {d} | f ^ {\prime \prime} | (x)
$$
and  $\begin{array}{r}\overline{R} (f) = \left\| \sqrt{1 + x^2} f''\right\|_{\mathrm{TV}} + D(x_f,\mathcal{C}_f), \end{array}$
$$
x _ {f} = \left(f ^ {\prime} (+ \infty) + f ^ {\prime} (- \infty), f (0) - \int_ {\mathbb {R}} | x | \mathrm {d} f ^ {\prime \prime} (x)\right)
$$
$$
\mathcal {C} _ {f} = \left\{\int_ {\mathbb {R}} \varphi (x) \mathrm {d} f ^ {\prime \prime} (x), - \int_ {\mathbb {R}} x \varphi (x) \mathrm {d} f ^ {\prime \prime} (x) | \| \varphi \| _ {\infty} \leq 1 \right\}
$$
$$
D \left(x _ {f}, \mathcal {C} _ {f}\right) = \inf  _ {x \in \mathcal {C} _ {f}} \| x - x _ {f} \|.
$$
For any non-Lipschitz function,  $\overline{R}_1(f) = \overline{R}(f) = +\infty$ .
Proof. We only prove the equality on  $\overline{R}(f)$  here. The other part of Theorem 4 can be directly deduced from this proof. First assume that  $\overline{R}(f)$  is finite. We can then consider some  $\mu \in \mathcal{M}(\mathbb{S}_1)$  such that for any  $x \in \mathbb{R}$ ,  $f(x) = \int_{\mathbb{S}_1} \sigma(wx + b) \, \mathrm{d}\mu(w, b)$ . Note that  $f$  is necessarily  $\| \mu \|_{\mathrm{TV}}$ -Lipschitz, which proves the second part of Theorem 4. Without loss of generality, we can parameterise  $\mathbb{S}_1$  on  $\theta \in [-\frac{\pi}{2}, \frac{3\pi}{2})$  with  $T^{-1}(\theta) = (\cos \theta, \sin \theta)$ . If we note  $\nu = T_{\#} \mu$  the pushforward measure of  $\mu$  by  $T$ , we then have
$$
f (x) = \int_ {- \frac {\pi}{2}} ^ {\frac {3 \pi}{2}} \sigma (x \cos \theta + \sin \theta) \mathrm {d} \nu (\theta). \tag {14}
$$
Since the total variation of  $\mu$  and thus  $\nu$  is bounded, we can derive under the integral sign:
$$
f ^ {\prime} (x) = \int_ {- \frac {\pi}{2}} ^ {\frac {3 \pi}{2}} \cos \theta \mathbb {1} _ {x \cos \theta + \sin \theta \geq 0} \mathrm {d} \nu (\theta).
$$
Now note that  $x\cos \theta +\sin \theta \geq 0$  if and only if  $\theta \in [-\arctan (x),\pi -\arctan (x)]$  since  $\theta \in [-\frac{\pi}{2},\frac{3\pi}{2})$  i.e.
$$
f ^ {\prime} (x) = \int_ {- \arctan x} ^ {\pi - \arctan x} \cos \theta \mathrm {d} \nu (\theta).
$$
When deriving this expression over  $x$ , we finally get for the distribution  $f''$
$$
\begin{array}{l} \mathrm {d} f ^ {\prime \prime} (x) = - \frac {\cos (\pi - \arctan (x)) \mathrm {d} \nu (\pi - \arctan (x)) - \cos (- \arctan (x)) \mathrm {d} \nu (- \arctan (x))}{1 + x ^ {2}} \\ = \frac {\cos (\arctan (x)) (\mathrm {d} \nu (\pi - \arctan (x)) + \mathrm {d} \nu (- \arctan (x)))}{1 + x ^ {2}}. \\ \end{array}
$$
This equality is straightforward for continuous distributions  $\nu$ . Extending it to any distribution  $\nu$  requires some extra work but can be obtained following the typical definition of distributional derivative.
Defining  $\nu_{+}(\theta) = \nu (\theta) + \nu (\pi +\theta)$  for any  $\theta \in [-\frac{\pi}{2},\frac{\pi}{2})$  and noting that  $\cos (\arctan x) = \frac{1}{\sqrt{1 + x^2}}$
$$
\forall x \in \mathbb {R}, \sqrt {1 + x ^ {2}} \mathrm {d} f ^ {\prime \prime} (x) = \frac {\mathrm {d} \nu_ {+} (- \arctan (x))}{1 + x ^ {2}}.
$$
Or equivalently, for any  $\theta \in (-\frac{\pi}{2}, \frac{\pi}{2})$
$$
\frac {\mathrm {d} f ^ {\prime \prime} (- \tan \theta)}{\cos \theta} = \cos^ {2} (\theta) \mathrm {d} \nu_ {+} (\theta). \tag {15}
$$
Similarly to the proof of Savarese et al. [2019],  $f''$  fixes  $\nu_{+}$  and the only degree of freedom is on  $\nu_{\perp}(\theta) \coloneqq \nu(\theta) - \nu(\pi + \theta)$ . The proof now determines which valid  $\nu_{\perp}$  minimises  $\| \mu \|_{\mathrm{TV}} = \| \nu \|_{\mathrm{TV}}$ . Equation (14) implies the following condition on  $\nu_{\perp}$
$$
\begin{array}{l} f (x) = \frac {1}{2} \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} \sigma (- x \cos \theta - \sin \theta) d (\nu_ {+} + \nu_ {\perp}) (\theta) + \frac {1}{2} \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} \sigma (x \cos \theta + \sin \theta) d (\nu_ {+} - \nu_ {\perp}) (\theta) \\ = \frac {1}{2} \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} | x \cos \theta + \sin \theta | \mathrm {d} \nu_ {+} (\theta) + \frac {x}{2} \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} \cos \theta \mathrm {d} \nu_ {\perp} (\theta) + \frac {1}{2} \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} \sin \theta \mathrm {d} \nu_ {\perp} (\theta). \\ \end{array}
$$
While  $\nu_{+}$  is given by  $f^{\prime \prime}$ ,  $\nu_{\perp}$  holds for affine part of  $f$ . The above equality directly leads to the following condition on  $\nu_{\perp}$
$$
\left\{ \begin{array}{l} \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} \cos \theta d v _ {\perp} (\theta) = f ^ {\prime} (+ \infty) + f ^ {\prime} (\infty) \\ \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} \sin \theta d v _ {\perp} (\theta) = f (0) - \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} | \sin \theta | d v _ {+} (\theta) \end{array} . \right. \tag {16}
$$
Now note that  $2\| \mu \|_{\mathrm{TV}} = \| \nu_{+} + \nu_{\perp}\|_{\mathrm{TV}} + \| \nu_{+} - \nu_{\perp}\|_{\mathrm{TV}}$ , so that  $\overline{R}(f)$  is given by
$2\overline{R}(f) = \min_{\nu_{\perp}} \| \nu_{+} + \nu_{\perp} \|_{\mathrm{TV}} + \| \nu_{+} - \nu_{\perp} \|_{\mathrm{TV}}$  such that  $\nu_{\perp}$  verifies Equation (16).
Lemma 5 below then implies<sup>8</sup>
$$
\bar {R} (f) = \left\| \nu_ {+} \right\| _ {\mathrm {T V}} + D \left(x _ {f}, \mathcal {C} _ {f}\right),
$$
where  $x_{f}$  and  $\mathcal{C}_f$  are defined in Theorem 4. Equation (15) leads with a simple change of variable when  $\overline{R}(f)$  is finite to
$$
\overline {{R}} (f) = \left\| \sqrt {1 + x ^ {2}} f ^ {\prime \prime} \right\| _ {\mathrm {T V}} + D (x _ {f}, \mathcal {C} _ {f}).
$$
Reciprocally, when  $\left\| \sqrt{1 + x^2} f''\right\|_{\mathrm{TV}}$  is finite, we can define  $\nu_{+}$  as in Equation (15) and  $\nu_{\perp}$  as a sum of Dirac's in  $-\frac{\pi}{2}$  and 0 verifying Equation (16). The corresponding  $\mu$  is then of finite total variation, implying that  $\overline{R}(f)$  is finite. This ends the proof of the first part of Theorem 4:
For  $\overline{R}_1(f)$ , the analysis is simpler since there is no constraint on  $\nu_{\perp}$ , whose optimal choice is then given by  $\nu_{\perp} = 0$ .
Lemma 5. The minimisation program
$$
\min  _ {\nu_ {\perp}} \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} \mathrm {d} | \nu_ {+} + \nu_ {\perp} | + \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} \mathrm {d} | \nu_ {+} - \nu_ {\perp} | \tag {17}
$$
such that  $\left(\int_{-\pi /2}^{\pi /2}\cos \theta \mathrm{d}\nu_{\perp}(\theta),\int_{-\pi /2}^{\pi /2}\sin \theta \mathrm{d}\nu_{\perp}(\theta)\right) = (a,b)$
$$
2 \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} \mathrm {d} | \nu_ {+} | + 2 \min  _ {u \in \mathcal {C}} \| (a, b) - u \|, \tag {18}
$$
$$
w h e r e \mathcal {C} = \left\{\left(\int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} \cos (\theta) \varphi (\theta) \mathrm {d} \nu_ {+} (\theta), \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} \sin (\theta) \varphi (\theta) \mathrm {d} \nu_ {+} (\theta)\right) | \| \varphi \| _ {\infty} \leq 1 \right\}.
$$
Proof. For any  $\nu_{\perp}$  in the constraint set of Equation (17), we can use a decomposition  $\nu_{\perp} = \varphi \nu_{+} + \mu_{2}$  where  $\| \varphi \|_{\infty}\leq 1$ . It then comes pointwise
$$
\begin{array}{l} \left| \nu_ {+} + \nu_ {\perp} \right| + \left| \nu_ {+} - \nu_ {\perp} \right| \leq 2 \left| \mu_ {2} \right| + \left| (1 + \varphi) \nu_ {+} \right| + \left| (1 - \varphi) \nu_ {+} \right| \tag {19} \\ = 2 \left| \mu_ {2} \right| + 2 \left| \nu_ {+} \right|. \\ \end{array}
$$
As a consequence, if we note  $v$  the infimum given by Equation (17):
$$
v \leq 2 \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} \mathrm {d} | \nu_ {+} | + 2 \min  _ {(\varphi , \mu_ {2}) \in \Gamma} \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} \mathrm {d} | \mu_ {2} |,
$$
$$
\Gamma = \left\{(\varphi , \mu_ {2}) \mid \| \varphi \| _ {\infty} \leq 1 \text {a n d} \int_ {- \pi / 2} ^ {\pi / 2} (\cos \theta , \sin \theta) (\varphi (\theta) \mathrm {d} \nu_ {+} (\theta) + \mathrm {d} \mu_ {2} (\theta)) = (a, b) \right\}.
$$
Moreover for a fixed  $\nu_{\perp}$ , we can choose  $(\varphi, \mu_2)$  as:
$$
\left\{ \begin{array}{l} \varphi = \mathrm {s i g n} (\frac {\mathrm {d} \nu_ {\perp}}{\mathrm {d} \nu_ {+}}) \min  \left(\left| \frac {\mathrm {d} \nu_ {\perp}}{\mathrm {d} \nu_ {+}} \right|, 1\right), \\ \mu_ {2} = \nu_ {\perp} - \varphi \nu_ {+}, \end{array} \right.
$$
where  $\frac{\mathrm{d}\nu_{\perp}}{\mathrm{d}\nu_{+}}$  denotes by abuse of notation the Radon-Nikodym derivative  $\frac{\mathrm{d}\nu_a}{\mathrm{d}\nu_+}$ , with the Lebesgue decomposition  $\nu_{\perp} = \nu_{a} + \nu_{s}$  with  $\nu_{a} \ll \nu_{+}$  and  $\nu_{s} \perp \nu_{+}$ . For this choice, Equation (19) becomes an equality, which directly implies that
$$
v = 2 \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} d | \nu_ {+} | + 2 \min  _ {(\varphi , \mu_ {2}) \in \Gamma} \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} d | \mu_ {2} |.
$$
${}^{8}$  A change of variable is also necessary to observe that the min of Lemma 5 is equal to  $D\left( {{x}_{f},{\mathcal{C}}_{f}}\right)$  .
It now remains to prove that  $\min_{u\in \mathcal{C}}\| (a,b) - u\| ^2 = \min_{(\varphi ,\mu_2)\in \Gamma}\int_{-\frac{\pi}{2}}^{\frac{\pi}{2}}\mathrm{d}|\mu_2|$ . Fix in the following  $\varphi$  such that  $\| \varphi \|_{\infty}\leq 1$  and note
$$
\left\{ \begin{array}{l} x = a - \int_ {- \pi / 2} ^ {\pi / 2} \cos \theta   \varphi (\theta) \mathrm {d} \nu_ {+} (\theta), \\ y = b - \int_ {- \pi / 2} ^ {\pi / 2} \sin \theta   \varphi (\theta) \mathrm {d} \nu_ {+} (\theta). \end{array} \right.
$$
It now suffices to show that for any fixed  $\varphi$ :
$$
\min  _ {\mu_ {2} \text {s . t .} (\varphi , \mu_ {2}) \in \Gamma} \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} \mathrm {d} | \mu_ {2} | = \| (x, y) \|.
$$
The constraint set is actually  $\left\{\mu_2 \mid \int_{-\pi/2}^{\pi/2} (\cos \theta, \sin \theta) \, \mathrm{d}\mu_2(\theta) = (x, y)\right\}$ . Now define
$$
\theta^ {*} = \arcsin \left(\frac {\operatorname {s i g n} (x) y}{\sqrt {x ^ {2} + y ^ {2}}}\right) \quad \text {a n d} \quad \mu_ {2} ^ {*} = \operatorname {s i g n} (x) \sqrt {x ^ {2} + y ^ {2}} \delta_ {\theta^ {*}},
$$
where  $\delta_{\theta^{*}}$  is the Dirac distribution located at  $\theta^{*}$ . This definition is only valid if  $x \neq 0$ , otherwise we choose  $\mu_2^* = -y\delta_{-\frac{\pi}{2}}$ .
Note that  $\mu_{2}^{*}$  is in the constraint set and  $\int_{-\frac{\pi}{2}}^{\frac{\pi}{2}}\mathrm{d}|\mu_2| = \| (x,y)\|$ , i.e.
$$
\min  _ {\mu_ {2} \text {s . t .} (\varphi , \mu_ {2}) \in \Gamma} \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} \mathrm {d} | \mu_ {2} | \leq \| (x, y) \|.
$$
Now consider any  $\mu_{2}$  in the constraint set and decompose  $\mu_{2} = \mu_{2}^{+} - \mu_{2}^{-}$  with  $(\mu_2^+, \mu_2^-) \in \mathcal{M}_+([-\frac{\pi}{2}, \frac{\pi}{2}))^2$ . Define
$$
(x _ {+}, y _ {+}) = \left(\int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} \cos \theta \mathrm {d} \mu_ {2} ^ {+}, \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} \sin \theta \mathrm {d} \mu_ {2} ^ {+}\right)
$$
$$
\left(x _ {-}, y _ {-}\right) = \left(\int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} \cos \theta \mathrm {d} \mu_ {2} ^ {-}, \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} \sin \theta \mathrm {d} \mu_ {2} ^ {-}\right)
$$
$$
\int \cos^ {2} (\theta) \mathrm {d} \mu_ {2} ^ {+} (\theta) \int \mathrm {d} \mu_ {2} ^ {+} (\theta) \geq x _ {+} ^ {2},
$$
$$
\int \sin^ {2} (\theta) \mathrm {d} \mu_ {2} ^ {+} (\theta) \int \mathrm {d} \mu_ {2} ^ {+} (\theta) \geq y _ {+} ^ {2}.
$$
Summing these two inequalities yields
$$
\int \mathrm {d} \mu_ {2} ^ {+} \geq \sqrt {x _ {+} ^ {2} + y _ {+} ^ {2}}.
$$
$$
\int \mathrm {d} \mu_ {2} ^ {-} \geq \sqrt {x _ {-} ^ {2} + y _ {-} ^ {2}}.
$$
Recall that  $\int \mathrm{d}|\mu_2| = \int \mathrm{d}\mu_2^+ +\int \mathrm{d}\mu_2^-.$  By triangle inequality, this yields:
$$
\begin{array}{l} \int \mathrm {d} | \mu_ {2} | \geq \| (x _ {+}, y _ {+}) \| + \| (x _ {-}, y _ {-}) \| \\ \geq \left\| \left(x _ {+}, y _ {+}\right) - \left(x _ {-}, y _ {-}\right) \right\| = \| (x, y) \|. \\ \end{array}
$$
As a consequence:
$$
\min  _ {\mu_ {2} \text {s . t .} (\varphi , \mu_ {2}) \in \Gamma} \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} \mathrm {d} | \mu_ {2} | \geq \| (x, y) \|.
$$
We finally showed that
$$
\min  _ {\mu_ {2} \text {s . t .} (\varphi , \mu_ {2}) \in \Gamma} \int_ {- \frac {\pi}{2}} ^ {\frac {\pi}{2}} \mathrm {d} | \mu_ {2} | = \left\| (a, b) - \left(\int_ {- \pi / 2} ^ {\pi / 2} \cos \theta \varphi (\theta) \mathrm {d} \nu_ {+} (\theta), \int_ {- \pi / 2} ^ {\pi / 2} \sin \theta \varphi (\theta) \mathrm {d} \nu_ {+} (\theta)\right) \right\|.
$$
This leads to Lemma 5 when taking the infimum over  $\varphi$ .
D Proof of Section 4
We first need to show the existence of a minimum. Using the definition of  $\overline{R}_1(f)$  and Theorem 1, Equation (4) is equivalent to
$$
\inf  _ {\mu , a, b} \| \mu \| _ {\mathrm {T V}} \quad \text {s u c h t h a t f o r a n y} i \in [ n ], f _ {\mu , a, b} \left(x _ {i}\right) = y _ {i}. \tag {20}
$$
Consider a sequence  $(\mu_j, a_j, b_j)_j$  such that  $f_{\mu_j, a_j, b_j}(x_i) = y_i$  for any  $i$  and  $j$  and  $\|\mu_j\|_{\mathrm{TV}}$  converges to the infimum of Equation (20). The sequence  $\|\mu_j\|_{\mathrm{TV}}$  is necessarily bounded. This also implies that both  $(a_j)$  and  $(b_j)$  are bounded. Since the space of finite signed measures on  $\mathbb{S}_1$  is a Banach space, there is a subsequence converging weakly towards some  $(\mu, a, b)$ . By weak convergence,  $(\mu, a, b)$  is in the constraints set of Equation (20) and  $\|\mu\|_{\mathrm{TV}} = \lim_j \|\mu_j\|_{\mathrm{TV}}$ .  $(\mu, a, b)$  is thus a minimiser of Equation (20). We thus proved the existence of a minimum for Equation (4), which is reached for  $f_{\mu, a, b}$ .
Define for the sake of the proof the activation cones  $C_i$  as
$$
C _ {0} = \left\{\theta \in \mathbb {R} ^ {2} \mid \forall i = 1, \dots , n, \langle \theta , (x _ {i}, 1) \rangle \geq 0 \right\},
$$
$$
C _ {i} = \left\{\theta \in \mathbb {R} ^ {2} \mid \langle \theta , (x _ {i + 1}, 1) \rangle \geq 0 > \langle \theta , (x _ {i}, 1) \rangle \right\} \quad \text {f o r a n y} i = 1, \dots , i _ {0} - 2,
$$
$$
C _ {i _ {0} - 1} = \left\{\theta \in \mathbb {R} ^ {2} \mid \langle \theta , \left(x _ {i _ {0}}, 1\right) \rangle > 0 > \langle \theta , \left(x _ {i _ {0} - 1}, 1\right) \rangle \right\}, \tag {21}
$$
$$
C _ {i} = \left\{\theta \in \mathbb {R} ^ {2} \mid \langle \theta , (x _ {i + 1}, 1) \rangle > 0 \geq \langle \theta , (x _ {i}, 1) \rangle \right\} \quad \text {f o r a n y} i = i _ {0}, \dots , n - 1,
$$
$$
C _ {n} = \left\{\theta \in \mathbb {R} ^ {2} \backslash \{0 \} \mid \forall i = 1, \dots , n, \langle \theta , (x _ {i}, 1) \rangle \leq 0 \right\rangle \}.
$$
As the  $x_{i}$  are ordered, note that  $(C_0,C_1, - C_1,\dots ,C_{n - 1}, - C_{n - 1},C_n)$  forms a partition of  $\mathbb{R}^2$ . To prove Lemma 1, it remains to show that any minimiser  $(\mu_{a},b)$  of Equation (20) has a function  $f_{\mu ,a,b}$  of the form
$$
f _ {\mu , a, b} (x) = \tilde {a} x + \tilde {b} + \sum_ {i = 1} ^ {n - 1} \tilde {a} _ {i} \sigma (\langle \theta_ {i}, (x, 1) \rangle) \quad \text {w h e r e} \theta_ {i} \in C _ {i}.
$$
Let  $f$  be a minimiser of Equation (4). Let  $\mu, a, b$  be a minimiser of Equation (20) such that  $f_{\mu, a, b} = f$ . Define  $\tilde{\mu}, \tilde{a}, \tilde{b}$  as
$$
\mathrm {d} \tilde {\mu} (\theta) = \left\{ \begin{array}{l l} \mathrm {d} \mu (\theta) + \mathrm {d} \mu (- \theta) & \quad \text {f o r} \theta \in C _ {i} \text {f o r a n y} i = 1, \ldots , n - 1 \\ 0 & \quad \text {f o r} \theta \in - C _ {i} \text {f o r a n y} i = 1, \ldots , n - 1 \\ \mathrm {d} \mu (\theta) & \quad \text {o t h e r w i s e}, \end{array} \right.
$$
$$
\tilde {a} = a - \sum_ {j = 1} ^ {n - 1} \int_ {- C _ {j}} \theta_ {1} \mathrm {d} \mu (\theta),
$$
$$
\tilde {b} = b - \sum_ {j = 1} ^ {n - 1} \int_ {- C _ {j}} \theta_ {2} \mathrm {d} \mu (\theta).
$$
Thanks to the identity  $\sigma(u) - u = \sigma(-u)$ ,  $f_{\mu, a, b} = f_{\tilde{\mu}, \tilde{a}, \tilde{b}}$ . Moreover,  $\|\tilde{\mu}\|_{\mathrm{TV}} \leq \|\mu\|_{\mathrm{TV}}$ , so we can assume w.l.o.g. that the support of  $\mu$  is included<sup>10</sup> in  $\bigcup_{i=0}^{n} C_i$ . In that case, for any  $i = 1, \ldots, n$
$$
\begin{array}{l} f (x _ {i}) = a x _ {i} + b + \sum_ {j = 0} ^ {i - 1} \int_ {C _ {j}} \langle \theta , (x _ {i}, 1) \rangle \mathrm {d} \mu (\theta) \\ = a x _ {i} + b + \sum_ {j = 0} ^ {i - 1} \left\langle \int_ {C _ {j}} \theta \mathrm {d} \mu (\theta), (x _ {i}, 1) \right\rangle . \tag {22} \\ \end{array}
$$
To see that, we can first consider the difference  $f_{\mu_j,a_j,b_j}(x_1) - f_{\mu_j,a_j,b_j}(x_2)$  to show that  $(a_j)_j$  is bounded. This then leads to the boundedness of  $(b_j)_j$  when considering  $f_{\mu_j,a_j,b_j}(x_1)$ .
<sup>10</sup>We here transform the triple  $(\mu, a, b)$ , but the corresponding function  $f$  remains unchanged.
First, the reduction
$$
\tilde {a} = a + \int_ {C _ {0}} \theta_ {1} \mathrm {d} \mu (\theta)
$$
$$
\tilde {b} = b + \int_ {C _ {0}} \theta_ {2} \mathrm {d} \mu (\theta)
$$
$$
\tilde {\mu} = \mu_ {| \bigcup_ {i = 1} ^ {n - 1} C _ {i}},
$$
does not increase the total variation of  $\mu$  and still interpolates the data. As a consequence, the support of  $\mu$  is included in  $\bigcup_{i=1}^{n-1} C_i$ . Now let  $\mu = \mu_+ - \mu_-$  be the Jordan decomposition of  $\mu$  and define for any  $i \in [n-1]$
$$
\alpha_ {i} = \int_ {C _ {i}} \theta \mathrm {d} \mu_ {+} (\theta) \quad \text {a n d} \quad \beta_ {i} = \int_ {C _ {i}} \theta \mathrm {d} \mu_ {-} (\theta).
$$
Note that  $\alpha_{i}$  and  $\beta_{i}$  are both in the positive convex cone  $C_i$ . For  $\theta_{i} \coloneqq \alpha_{i} - \beta_{i}$ , Equation (22) rewrites
$$
f (x _ {i}) = a x _ {i} + b + \sum_ {j = 1} ^ {i - 1} \langle \theta_ {i}, (x _ {i}, 1) \rangle .
$$
If  $\theta_{i}\in \overline{C}_{i}\cup \overline{-C_{i}}$  , we can then define
$$
\tilde {\mu} = \mu - \mu_ {| C _ {i}} + \| \theta_ {i} \| \delta_ {\frac {\theta_ {i}}{\| \theta_ {i} \|}}.
$$
Thanks to Equation (22), the function  $f_{\tilde{\mu},a,b}$  still interpolates the data and
$$
\| \tilde {\mu} \| _ {\mathrm {T V}} \leq \| \mu \| _ {\mathrm {T V}} - \left\| \mu_ {| C _ {i}} \right\| _ {\mathrm {T V}} + \| \theta_ {i} \|.
$$
By minimisation of  $\| \mu \|_{\mathrm{TV}}$ , this is an equality. Moreover as  $\mu$  is a measure on the sphere,
$$
\begin{array}{l} \left\| \mu_ {| C _ {i}} \right\| _ {\mathrm {T V}} = \int_ {C _ {i}} \| \theta \| \mathrm {d} \mu_ {+} (\theta) + \int_ {C _ {i}} \| \theta \| \mathrm {d} \mu_ {-} (\theta) \\ \geq \left\| \int_ {C _ {i}} \theta \mathrm {d} \mu_ {+} (\theta) \right\| + \left\| \int_ {C _ {i}} \theta \mathrm {d} \mu_ {-} (\theta) \right\| \\ = \| \alpha_ {i} \| + \| \beta_ {i} \| \geq \| \theta_ {i} \|. \\ \end{array}
$$