SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \frac {\partial L _ {\eta}}{\partial u _ {j}} (a, u) = \int_ {0} ^ {1} \frac {\partial f _ {\eta} (x ; a , u)}{\partial u _ {j}} \left(f _ {\eta} (x; a, u) - f ^ {*} (x)\right) \mathrm {d} x \\ = - a _ {j} \int_ {0} ^ {1} \sigma_ {\eta} ^ {\prime} (x - u _ {j}) \left(\sum_ {k = 0} ^ {m} a _ {k} \sigma_ {\eta} (x - u _ {k}) - f ^ {*} (x)\right) d x. \tag {10} \\ \end{array}
$$
$$
\begin{array}{l} \frac {\partial L _ {\eta}}{\partial a _ {j}} (a, u) = \int_ {0} ^ {1} \frac {\partial f _ {\eta} (x ; a , u)}{\partial a _ {j}} \left(f _ {\eta} (x; a, u) - f ^ {*} (x)\right) d x \\ = \int_ {0} ^ {1} \sigma_ {\eta} (x - u _ {j}) \left(\sum_ {k = 0} ^ {m} a _ {k} \sigma_ {\eta} (x - u _ {k}) - f ^ {*} (x)\right) \mathrm {d} x \\ = H _ {\eta , j} (u) ^ {\top} a - b _ {\eta , j} (u). \tag {11} \\ \end{array}
$$
We now move on to a series to lemmas that will be helpful in the proofs of Appendix B.
Lemma 1. For  $\eta \geqslant 0$  and  $u\in \mathbb{R}^m$ , we have
$$
\left\| b _ {\eta} (u) - b _ {0} (u) \right\| \leqslant M \eta \sqrt {m + 1} \quad \text {a n d} \quad \left\| b _ {\eta} (u) \right\| \leqslant M \sqrt {m + 1}.
$$
Proof. For any  $j \in \{0, \dots, m\}$ ,
$$
\begin{array}{l} \left| b _ {\eta , j} (u) - b _ {0, j} (u) \right| = \left| \int_ {0} ^ {1} f ^ {*} (x) \left(\sigma_ {\eta} (x - u _ {j}) - \sigma_ {0} (x - u _ {j})\right) d x \right| \\ \leqslant \| f ^ {*} \| _ {\infty} \int_ {0} ^ {1} | \sigma_ {\eta} (x - u _ {j}) - \sigma_ {0} (x - u _ {j}) | d x \\ \leqslant M \eta , \\ \end{array}
$$
where in the last step we use that  $\| f^{*}\|_{\infty}\leqslant M$  and that  $\sigma_{\eta}(x) = 0$  for  $x\leq -\eta /2,\sigma_{\eta}(x)\in [0,1]$  for  $-\eta /2 < x < \eta /2$  and  $\sigma_{\eta}(x) = 1$  for  $x\geq \eta /2$
$$
\left| b _ {\eta , j} (u) \right| = \left| \int_ {0} ^ {1} f ^ {*} (x) \sigma_ {\eta} (x - u _ {j}) \mathrm {d} x \right| \leqslant \| f ^ {*} \| _ {\infty} \leqslant M.
$$
Lemma 2. For  $\eta \geqslant 0$  and  $u\in \mathcal{U}_{\eta}$ ,  $H_{\eta}(u) = H_0(u) + D_{\eta}$ , where  $D_{\eta}$  is a diagonal matrix whose elements are independent of  $u$  and bounded in absolute value by  $\eta /2$ .
Proof. Let  $i, j \in \{0, \dots, m\}$ , and denote  $c = \max(u_i, u_j, 0)$ . Then
$$
H _ {0, i j} (u) = \int_ {0} ^ {1} \sigma_ {0} (x - u _ {i}) \sigma_ {0} (x - u _ {j}) d x = 1 - c.
$$
If  $i = j = 0$ ,  $\max(u_i, u_j) = -\eta/2$ , and  $H_{\eta, ij}(u) = 1 = H_{0, ij}(u)$ . If  $i = j \neq 0$ ,
$$
\begin{array}{l} H _ {\eta , i j} (u) = \int_ {0} ^ {1} \sigma_ {\eta} (x - c) ^ {2} d x \\ = 1 - c - \frac {\eta}{2} + \int_ {c - \eta / 2} ^ {c + \eta / 2} \sigma_ {\eta} (x - c) ^ {2} d x \\ = H _ {0, i j} (u) - \frac {\eta}{2} + \eta \int_ {- 1 / 2} ^ {1 / 2} \sigma^ {2}. \\ \end{array}
$$
Note that the last integral is non-negative and less than 1, hence  $|H_{\eta ,ij}(u) - H_{0,ij}(u)| \leqslant \eta /2$ . Finally, if  $i \neq j$ , since  $|u_i - u_j| > \eta$ ,
$$
H _ {\eta , i j} (u) = \int_ {0} ^ {1} \sigma_ {\eta} (x - u _ {i}) \sigma_ {\eta} (x - u _ {j}) d x = \int_ {0} ^ {1} \sigma_ {\eta} (x - \max  (u _ {i}, u _ {j})) d x.
$$
Furthermore,  $0 < \max (u_{i},u_{j}) < 1 - \frac{\eta}{2}$ , thus
$$
H _ {\eta , i j} (u) = \int_ {0} ^ {1} \sigma_ {\eta} (x - c) d x = 1 - c - \frac {\eta}{2} + \int_ {c - \eta / 2} ^ {c + \eta / 2} \sigma_ {\eta} (x - c) d x = 1 - c,
$$
where the last equality comes from the oddness of  $\sigma - 1/2$ .
Lemma 3. For  $\eta >0$ , let  $a_{\eta}^{*}:u\in \mathcal{U}_{\eta}\mapsto a_{\eta}^{*}(u)$ . Then  $a_{\eta}^{*}$  is differentiable and for any  $u\in \mathcal{U}_{\eta}$
$$
\left\| \frac {\partial a _ {\eta} ^ {*} (u)}{\partial u} \right\| \leqslant \frac {8}{\Delta (u)} \left(2 (m + 1) \| a _ {\eta} ^ {*} (u) \| + M\right).
$$
Proof. By Proposition 1 (whose proof does not rely on this lemma), for  $u \in \mathcal{U}_{\eta}$ ,  $L_{\eta}(\cdot, u)$  has a unique minimizer  $a_{\eta}^{*}(u)$ , which is equal to  $H_{\eta}(u)^{-1}b_{\eta}(u)$  by (11). Furthermore,  $H_{\eta}$  and  $b_{\eta}$  are differentiable with respect to  $u$ , hence  $a_{\eta}^{*}$  is also differentiable with respect to  $u$ , and we have
$$
\frac {\partial a _ {\eta} ^ {*} (u)}{\partial u _ {k}} = - H _ {\eta} (u) ^ {- 1} \frac {\partial H _ {\eta}}{\partial u _ {k}} (u) a _ {\eta} ^ {*} (u) + H _ {\eta} (u) ^ {- 1} \frac {\partial b _ {\eta}}{\partial u _ {k}} (u).
$$
Denote  $w_{k}(u) \coloneqq \frac{\partial H_{\eta}}{\partial u_{k}}(u)a_{\eta}^{*}(u)$  and  $W(u)$  the  $(m + 1)\times (m + 1)$  matrix formed by stacking column-wise the vectors  $(w_{k}(u))_{0\leqslant k\leqslant m}$ . Then
$$
\frac {\partial a _ {\eta} ^ {*} (u)}{\partial u} = - H _ {\eta} (u) ^ {- 1} W (u) + H _ {\eta} (u) ^ {- 1} \frac {\partial b _ {\eta}}{\partial u} (u).
$$
We now estimate the Frobenius norm of the matrix  $W(u)$ . By Lemma 2, for  $u \in \mathcal{U}_{\eta}$ ,  $H_{\eta}(u) = H_0(u) + D_{\eta}$ . Take  $i,j \in \{0,\dots,m\}$ , then
$$
H _ {\eta , i j} (u) = H _ {0, i j} (u) + D _ {\eta , i j} = \int_ {0} ^ {1} \sigma_ {0} (x - u _ {i}) \sigma_ {0} (x - u _ {j}) d x + D _ {\eta , i j} = 1 - \max (u _ {i}, u _ {j}, 0) + D _ {\eta , i j}.
$$
Hence  $\frac{\partial H_{\eta,ij}}{\partial u_k} = 0$  if  $i,j\neq k$ . Further, if  $i = k$  and  $j\neq k$
$$
\left| \frac {\partial H _ {\eta , i j}}{\partial u _ {k}} (u) \right| = \left| \frac {\partial}{\partial u _ {i}} (1 - \max (u _ {i}, u _ {j})) \right| \leqslant 1.
$$
Of course, the bound  $\left|\frac{\partial H_{\eta,ij}}{\partial u_k}(u)\right| \leqslant 1$  also holds when  $j = k$  and  $i \neq j$ . Finally, a similar bound shows that  $\left|\frac{\partial H_{\eta,ij}}{\partial u_k}(u)\right| \leqslant 2$  when  $i = j = k$ .
As a consequence, for  $k,i\in \{0,\dots ,m\}$
$$
| w _ {k, i} (u) | \leqslant \sum_ {j = 0} ^ {m} \left| \frac {\partial H _ {\eta , i j}}{\partial u _ {k}} (u) \right| \left| a _ {\eta , j} ^ {*} (u) \right| \leqslant \left\{ \begin{array}{l l} | a _ {\eta , k} ^ {*} (u) | & \text {i f} i \neq k  , \\ | a _ {\eta , k} ^ {*} (u) | + \| a _ {\eta} ^ {*} (u) \| _ {1} & \text {i f} i = k  . \end{array} \right.
$$
$$
\begin{array}{l} \| W (u) \| _ {\mathrm {F}} = \left(\sum_ {i = 0} ^ {m} \sum_ {k = 0} ^ {m} | w _ {k, i} (u) | ^ {2}\right) ^ {1 / 2} \leqslant \left(\sum_ {i = 0} ^ {m} \left(\sum_ {k = 0} ^ {m} | w _ {k, i} (u) |\right) ^ {2}\right) ^ {1 / 2} \\ \leqslant \left(\sum_ {i = 0} ^ {m} \left(2 \| a _ {\eta} ^ {*} (u) \| _ {1}\right) ^ {2}\right) ^ {1 / 2} = 2 \sqrt {m + 1} \| a _ {\eta} ^ {*} (u) \| _ {1}. \\ \end{array}
$$
With a reasoning similar to the above, note that  $\frac{\partial b_{\eta}}{\partial u}(u)$  is a diagonal matrix with diagonal entries in  $[-M, M]$ . Finally, putting these elements together, using Proposition 1 and that  $\|W(u)\| \leqslant \|W(u)\|_{\mathrm{F}}$ , we obtain
$$
\left\| \frac {\partial a _ {\eta} ^ {*} (u)}{\partial u} \right\| \leqslant \| H _ {\eta} (u) ^ {- 1} \| \| W (u) \| _ {F} + \| H _ {\eta} (u) ^ {- 1} \| \left\| \frac {\partial b _ {\eta} (u)}{\partial u} \right\| \leqslant \frac {8}{\Delta (u)} \Big (2 \sqrt {m + 1} \| a _ {\eta} ^ {*} (u) \| _ {1} + M \Big).
$$
The following lemma gives exact formulae for the derivative of the loss  $L_{\eta}$  with respect to the positions of the neurons, evaluated for  $a = a_0^* (u)$ , that is the best piecewise constant approximation of  $f^{*}$  with subdivision  $\{u_{1},\ldots ,u_{m}\}$ . Note that the formulae are the same as in Section 4.2, but the derivation is slightly more intricate since we consider here the loss  $L_{\eta}$  and not  $L_{0}$ .
Lemma 4. Take  $\eta >0$  and  $u\in \mathcal{U}_{\eta}$  such that there are at least two neurons on each piece  $[v_i,v_{i + 1}]$  of  $f^{*}$ . Then, if  $u_{j}$  does not flank a discontinuity of  $f^{*}$ ,
$$
\frac {\partial L _ {\eta}}{\partial u _ {j}} (a _ {0} ^ {*} (u), u) = 0.
$$
Furthermore, for a discontinuity  $v_{i}$ , denote  $u_{i}^{\mathrm{L}}$  is the closest neuron to its left and  $u_{i}^{\mathrm{R}}$  the closest neuron to its right. If  $v_{i} - u_{i}^{\mathrm{L}} \geqslant \frac{\eta}{2}$  and  $u_{i}^{\mathrm{R}} - v_{i} \geqslant \frac{\eta}{2}$ , then
$$
\begin{array}{l} \frac {\partial L _ {\eta}}{\partial u _ {i} ^ {\mathrm {L}}} (a _ {0} ^ {*} (u), u) = - \frac {1}{2} \frac {(u _ {i} ^ {\mathrm {R}} - v _ {i}) ^ {2}}{(u _ {i} ^ {\mathrm {R}} - u _ {i} ^ {\mathrm {L}}) ^ {2}} (f _ {i} ^ {*} - f _ {i - 1} ^ {*}) ^ {2}, \\ \frac {\partial L _ {\eta}}{\partial u _ {i} ^ {\mathrm {R}}} (a _ {0} ^ {*} (u), u) = \frac {1}{2} \frac {(v _ {i} - u _ {i} ^ {\mathrm {L}}) ^ {2}}{(u _ {i} ^ {\mathrm {R}} - u _ {i} ^ {\mathrm {L}}) ^ {2}} (f _ {i} ^ {*} - f _ {i - 1} ^ {*}) ^ {2}. \\ \end{array}
$$
Proof. In this proof, let us denote for simplicity  $a = a_0^*(u)$ . At the condition that there is at least two neurons on each piece of  $f^*$ , Section 4.2 gives the optimal approximation  $f_0(x; a, u)$  of  $f^*$  that is piecewise constant with respect to the subdivision  $\{u_1, \ldots, u_m\}$ . As a consequence, we easily get the value of  $a$ . Namely, if  $u_j$  does not flank a discontinuity of  $f^*$ , the value of  $f_0(x; a, u)$  is locally constant around  $u_j$ , thus  $a_j = 0$ . Plugging into (10), we obtain
$$
\frac {\partial L _ {\eta}}{\partial u _ {j}} (a, u) = 0.
$$
Further, for a discontinuity  $v_{i}$ , denote respectively  $a_{i}^{\mathrm{L}}$  and  $a_{i}^{\mathrm{R}}$  the coefficients associated to  $u_{i}^{\mathrm{L}}$  and  $u_{i}^{\mathrm{R}}$ . At  $u_{i}^{\mathrm{L}}$ , the value of  $f_{0}(x; a, u)$  jumps from  $f_{i-1}^{*}$  to  $\frac{v_{i} - u_{i}^{\mathrm{L}}}{u_{i}^{\mathrm{R}} - u_{i}^{\mathrm{L}}} f_{i-1}^{*} + \frac{u_{i}^{\mathrm{R}} - v_{i}}{u_{i}^{\mathrm{R}} - u_{i}^{\mathrm{L}}} f_{i}^{*}$ , thus
$$
a _ {i} ^ {\mathrm {L}} = \frac {v _ {i} - u _ {i} ^ {\mathrm {L}}}{u _ {i} ^ {\mathrm {R}} - u _ {i} ^ {\mathrm {L}}} f _ {i - 1} ^ {*} + \frac {u _ {i} ^ {\mathrm {R}} - v _ {i}}{u _ {i} ^ {\mathrm {R}} - u _ {i} ^ {\mathrm {L}}} f _ {i} ^ {*} - f _ {i - 1} ^ {*} = \frac {u _ {i} ^ {\mathrm {R}} - v _ {i}}{u _ {i} ^ {\mathrm {R}} - u _ {i} ^ {\mathrm {L}}} (f _ {i} ^ {*} - f _ {i - 1} ^ {*}).
$$
$$
a _ {i} ^ {\mathrm {R}} = \frac {v _ {i} - u _ {i} ^ {\mathrm {L}}}{u _ {i} ^ {\mathrm {R}} - u _ {i} ^ {\mathrm {L}}} (f _ {i} ^ {*} - f _ {i - 1} ^ {*}).
$$
We now compute, using (10),
$$
\begin{array}{l} \frac {\partial L _ {\eta}}{\partial u _ {i} ^ {\mathrm {L}}} (a, u) = - a _ {i} ^ {\mathrm {L}} \int_ {0} ^ {1} \sigma_ {\eta} ^ {\prime} (x - u _ {i} ^ {\mathrm {L}}) \left(f _ {\eta} (x; a, u) - f ^ {*} (x)\right) \mathrm {d} x \\ = - a _ {i} ^ {\mathrm {L}} \int_ {u _ {i} ^ {\mathrm {L}} - \eta / 2} ^ {u _ {i} ^ {\mathrm {L}} + \eta / 2} \sigma_ {\eta} ^ {\prime} (x - u _ {i} ^ {\mathrm {L}}) \left(f _ {\eta} (x; a, u) - f ^ {*} (x)\right) \mathrm {d} x. \\ \end{array}
$$
Using that  $\Delta(u) > 2\eta$  and that there are at least two neurons on each piece of  $f^{*}$ , we have that  $u_{i}^{\mathrm{L}} - v_{i-1} \geqslant 2\eta$ . Since, in addition, by assumption,  $v_{i} - u_{i}^{\mathrm{L}} \geqslant \frac{\eta}{2}$ , we get that for  $x \in [u_{i}^{\mathrm{L}} - \frac{\eta}{2}, u_{i}^{\mathrm{L}} + \frac{\eta}{2}]$ ,  $f^{*}(x) = f_{i-1}^{*}$ . Moreover, using again  $\Delta(u) \geqslant 2\eta$  that  $\sigma_{\eta}$  is equal to  $\sigma_{0}$  on  $(- \infty, -\eta/2]$  and  $[\eta/2, \infty)$ , we have for  $x \in [u_{i}^{\mathrm{L}} - \frac{\eta}{2}, u_{i}^{\mathrm{L}} + \frac{\eta}{2}]$ ,
$$
f _ {\eta} (x; a, u) = \sum_ {k = 0} ^ {m} a _ {k} \sigma_ {\eta} (x - u _ {k}) = f _ {0} \left(u _ {i} ^ {\mathrm {L}} - \frac {\eta}{2}; a, u\right) + a _ {i} ^ {\mathrm {L}} \sigma_ {\eta} (x - u _ {i} ^ {\mathrm {L}}) = f _ {i - 1} ^ {*} + a _ {i} ^ {\mathrm {L}} \sigma_ {\eta} (x - u _ {i} ^ {\mathrm {L}}).
$$
$$
\begin{array}{l} \frac {\partial L _ {\eta}}{\partial u _ {i} ^ {\mathrm {L}}} (a, u) = - a _ {i} ^ {\mathrm {L}} \int_ {u _ {i} ^ {\mathrm {L}} - \eta / 2} ^ {u _ {i} ^ {\mathrm {L}} + \eta / 2} \sigma_ {\eta} ^ {\prime} \left(x - u _ {i} ^ {\mathrm {L}}\right) a _ {i} ^ {\mathrm {L}} \sigma_ {\eta} \left(x - u _ {i} ^ {\mathrm {L}}\right) \mathrm {d} x \\ = - \frac {\left(a _ {i} ^ {\mathrm {L}}\right) ^ {2}}{2} \left(\sigma_ {\eta} \left(\frac {\eta}{2}\right) ^ {2} - \sigma_ {\eta} \left(- \frac {\eta}{2}\right) ^ {2}\right) \\ = - \frac {\left(a _ {i} ^ {\mathrm {L}}\right) ^ {2}}{2} \\ = - \frac {1}{2} \frac {(u _ {i} ^ {\mathrm {R}} - v _ {i}) ^ {2}}{(u _ {i} ^ {\mathrm {R}} - u _ {i} ^ {\mathrm {L}}) ^ {2}} (f _ {i} ^ {*} - f _ {i - 1} ^ {*}) ^ {2}. \\ \end{array}
$$
The computation of  $\frac{\partial L_{\eta}}{\partial u_i^{\mathrm{R}}}(a, u)$  is similar.
Lemma 5. Consider  $\eta \geqslant 0$  and  $u\in \mathcal{U}_{\eta}$  such that there are at least two neurons on each piece  $[v_i,v_{i + 1}]$  of  $f^{*}$ . Then, for all  $x\in [0,1]$ ,  $|f_{\eta}(x;a_0^* (u),u)|\leqslant M$ .
Proof. In the case where  $\eta = 0$ , the result easily follows from the expressions for  $f_0(x; a_0^*(u), u)$  provided in Section 4.2. We now assume  $\eta > 0$ .
Denote  $A_{k}^{*}(u) = \sum_{j=0}^{k} a_{0,j}^{*}(u)$  (with the convention  $A_{-1}^{*}(u) = 0$ ). Recall the convention  $u_{0} = -\eta / 2$ . We compute
$$
\begin{array}{l} f _ {\eta} (x; a _ {0} ^ {*} (u), u) = \sum_ {k = 0} ^ {m} a _ {0, k} ^ {*} (u) \sigma_ {\eta} (x - u _ {k}) \\ = \sum_ {k = 0} ^ {m} \left(A _ {k} ^ {*} (u) - A _ {k - 1} ^ {*} (u)\right) \sigma_ {\eta} (x - u _ {k}) \\ = \sum_ {k = 0} ^ {m - 1} A _ {k} ^ {*} (u) \left(\sigma_ {\eta} (x - u _ {k}) - \sigma_ {\eta} (x - u _ {k + 1})\right) + A _ {m} ^ {*} (u) \sigma_ {\eta} (x - u _ {m}) \\ \end{array}
$$
Note that  $A_k^*(u) = \lim_{x \to u_k +} f_0(x; a_0^*(u), u)$ , and thus, from the case  $\eta = 0$ , we have  $|A_k^*(u)| \leqslant M$ . Moreover,  $\sigma_\eta$  is increasing and the  $u_k$  are in increasing order. We thus get
$$
\begin{array}{l} \left| f _ {\eta} \left(x; a _ {0} ^ {*} (u), u\right) \right| \leqslant M \left(\sum_ {k = 0} ^ {m - 1} \left(\sigma_ {\eta} \left(x - u _ {k}\right) - \sigma_ {\eta} \left(x - u _ {k + 1}\right)\right) + \sigma_ {\eta} \left(x - u _ {m}\right)\right) \\ = M \sigma_ {\eta} (x - u _ {0}) \leqslant M. \\ \end{array}
$$
Lemma 6. Consider  $\eta >0$  and  $u\in \mathcal{U}_{\eta}$  such that there are at least two neurons on each piece  $[v_i,v_{i + 1}]$  of  $f^{*}$ . Then, for  $j\in \{0,\ldots ,m\}$ ,
$$
\left| a _ {0, j} ^ {*} (u) \right| \leqslant 2 M
$$
and, for any  $a\in \mathbb{R}^{m + 1}$
$$
\left| \frac {\partial L _ {\eta}}{\partial u _ {j}} (a, u) - \frac {\partial L _ {\eta}}{\partial u _ {j}} \left(a _ {0} ^ {*} (u), u\right) \right| \leqslant 2 M (\sqrt {m + 1} + 1) \| a - a _ {0} ^ {*} (u) \| + \sqrt {m + 1} \| a - a _ {0} ^ {*} (u) \| ^ {2}.
$$
Proof. The first statement of the Lemma comes from the explicit formulae for  $a_0^*(u)$  given in the proof of Lemma 4, namely each  $a_{0,j}^*(u)$  is either zero or less in magnitude than the gap between two pieces of  $f^*$  that is less than  $2M$ .
By (10), we have
$$
\begin{array}{l} \left| \frac {\partial L _ {\eta}}{\partial u _ {j}} (a, u) - \frac {\partial L _ {\eta}}{\partial u _ {j}} \left(a _ {0} ^ {*} (u), u\right) \right| \\ = \left| a _ {j} \int_ {0} ^ {1} \sigma_ {\eta} ^ {\prime} (x - u _ {j}) \left(f _ {\eta} (x; a, u) - f ^ {*} (x)\right) \mathrm {d} x \right. \\ \left. - a _ {0, j} ^ {*} (u) \int_ {0} ^ {1} \sigma_ {\eta} ^ {\prime} (x - u _ {j}) \left(f _ {\eta} (x; a _ {0} ^ {*} (u), u) - f ^ {*} (x)\right) \mathrm {d} x \right| \\ \leqslant \left| a _ {j} - a _ {0, j} ^ {*} (u) \right| \int_ {0} ^ {1} \sigma_ {\eta} ^ {\prime} (x - u _ {j}) \left| f _ {\eta} \left(x; a _ {0} ^ {*} (u), u\right) - f ^ {*} (x) \right| d x \\ + \left| a _ {j} \right| \int_ {0} ^ {1} \sigma_ {\eta} ^ {\prime} (x - u _ {j}) \left| f _ {\eta} (x; a, u) - f _ {\eta} (x; a _ {0} ^ {*} (u), u) \right| d x. \\ \end{array}
$$
We bound the two terms separately. For the first term, we use Lemma 5.
$$
\begin{array}{l} \int_ {0} ^ {1} \sigma_ {\eta} ^ {\prime} (x - u _ {j}) | f _ {\eta} (x; a _ {0} ^ {*} (u), u) - f ^ {*} (x) | \leqslant \int_ {0} ^ {1} \sigma_ {\eta} ^ {\prime} (x - u _ {j}) \left(\left| f _ {\eta} (x; a _ {0} ^ {*} (u), u) \right| + \left| f ^ {*} (x) \right|\right) \\ \leqslant 2 M \int_ {0} ^ {1} \sigma_ {\eta} ^ {\prime} (x - u _ {j}) \mathrm {d} x \leqslant 2 M. \\ \end{array}
$$
We now continue with the second term.
$$
\left| f _ {\eta} (x; a, u) - f _ {\eta} \left(x; a _ {0} ^ {*} (u), u\right) \right| = \left| \sum_ {k = 0} ^ {m} \left(a _ {k} - a _ {0, k} ^ {*} (u)\right) \sigma_ {\eta} \left(x - u _ {k}\right) \right| \leqslant \| a - a _ {0} ^ {*} (u) \| _ {1},
$$
$$
\begin{array}{l} \int_ {0} ^ {1} \sigma_ {\eta} ^ {\prime} (x - u _ {j}) | f _ {\eta} (x; a, u) - f _ {\eta} (x; a _ {0} ^ {*} (u), u) | d x \leqslant \| a - a _ {0} ^ {*} (u) \| _ {1} \int_ {0} ^ {1} \sigma_ {\eta} ^ {\prime} (x - u _ {j}) d x \\ \leqslant \left\| a - a _ {0} ^ {*} (u) \right\| _ {1}. \\ \end{array}
$$
Returning to our initial upper bound, we obtain, using the first statement of the Lemma,
$$
\begin{array}{l} \left| \frac {\partial L _ {\eta}}{\partial u _ {j}} (a, u) - \frac {\partial L _ {\eta}}{\partial u _ {j}} \left(a _ {0} ^ {*} (u), u\right) \right| \leqslant 2 M \| a - a _ {0} ^ {*} (u) \| + \left(\left| a _ {0, j} ^ {*} (u) \right| + \left| a _ {j} - a _ {0, j} ^ {*} (u) \right|\right) \| a - a _ {0} ^ {*} (u) \| _ {1} \\ \leqslant 2 M \| a - a _ {0} ^ {*} (u) \| + \left(2 M + \| a - a _ {0} ^ {*} (u) \|\right) \sqrt {m + 1} \| a - a _ {0} ^ {*} (u) \| \\ = 2 M (\sqrt {m + 1} + 1) \| a - a _ {0} ^ {*} (u) \| + \sqrt {m + 1} \| a - a _ {0} ^ {*} (u) \| ^ {2}. \\ \end{array}
$$
Lemma 7. For  $\eta \geqslant 0$  and  $u\in \mathcal{U}_{\eta}$
$$
\left\| a _ {\eta} ^ {*} (u) - a _ {0} ^ {*} (u) \right\| \leqslant \frac {1 6 M \sqrt {m + 1} \eta}{\Delta (u)}.
$$
Proof. By (11),
$$
H _ {\eta} (u) a _ {\eta} ^ {*} (u) = b _ {\eta} (u)
$$
and by (11) and by Lemma 2,
$$
H _ {\eta} (u) a _ {0} ^ {*} (u) = H _ {0} (u) a _ {0} ^ {*} (u) + D _ {\eta} a _ {0} ^ {*} (u) = b _ {0} (u) + D _ {\eta} a _ {0} ^ {*} (u).
$$
According to Proposition 1 (whose proof does not rely on this lemma),  $H_{\eta}(u)$  is invertible with  $\| H_{\eta}(u)^{-1} \| \leqslant 8 / \Delta(u)$ . We thus have
$$
\begin{array}{l} \left\| a _ {\eta} ^ {*} (u) - a _ {0} ^ {*} (u) \right\| = \left\| H _ {\eta} (u) ^ {- 1} \left(H _ {\eta} (u) a _ {\eta} ^ {*} (u) - H _ {\eta} (u) a _ {0} ^ {*} (u)\right) \right\| \\ \leqslant \frac {8}{\Delta (u)} \| b _ {\eta} (u) - b _ {0} (u) - D _ {\eta} a _ {0} ^ {*} (u) \| \\ \leqslant \frac {8}{\Delta (u)} \left(\| b _ {\eta} (u) - b _ {0} (u) \| + \| D _ {\eta} a _ {0} ^ {*} (u) \|\right) \\ \leqslant \frac {8}{\Delta (u)} \left(\| b _ {\eta} (u) - b _ {0} (u) \| + \frac {\eta}{2} \| a _ {0} ^ {*} (u) \|\right). \\ \end{array}
$$
The result then unfolds from Lemmas 1 and 6.
Lemma 8. Let  $\eta >0$ ,  $u\in \mathbb{R}^m$  and  $a,a'\in \mathbb{R}^{m + 1}$ . Then
$$
\left\| \nabla_ {u} L _ {\eta} (a, u) \right\| \leqslant \sqrt {m + 1} \| a \| ^ {2} + M \| a \|,
$$
$$
\left\| \nabla_ {a} L _ {\eta} (a, u) \right\| \leqslant \sqrt {m + 1} (\| a \| \sqrt {m + 1} + M).
$$
As a consequence of the second inequality, by the fundamental theorem of calculus for line integrals,
$$
\left| L _ {\eta} (a, u) - L _ {\eta} \left(a ^ {\prime}, u\right) \right| \leqslant \sqrt {m + 1} \left(\max  (\left\| a \right\|, \left\| a ^ {\prime} \right\|) \sqrt {m + 1} + M\right) \left\| a - a ^ {\prime} \right\|.
$$
Proof. Recall that, for all  $j \in \{1, \dots, m\}$ , and for all  $a, u \in \mathbb{R}^m$
$$
\frac {\partial L _ {\eta}}{\partial u _ {j}} (a, u) = - a _ {j} \int_ {0} ^ {1} \sigma_ {\eta} ^ {\prime} (x - u _ {j}) \Big (\sum_ {k = 0} ^ {m} a _ {k} \sigma_ {\eta} (x - u _ {k}) - f ^ {*} (x) \Big) \mathrm {d} x,
$$
$$
\frac {\partial L _ {\eta}}{\partial a _ {j}} (a, u) = \int_ {0} ^ {1} \sigma_ {\eta} (x - u _ {j}) \left(\sum_ {k = 0} ^ {m} a _ {k} \sigma_ {\eta} (x - u _ {k}) - f ^ {*} (x)\right) \mathrm {d} x.
$$
From the first equality, we have
$$
\begin{array}{l} \left| \frac {\partial L _ {\eta}}{\partial u _ {j}} (a, u) \right| \leqslant | a _ {j} | \int_ {0} ^ {1} | \sigma_ {\eta} ^ {\prime} (x - u _ {j}) | \left(\sum_ {k = 1} ^ {m} | a _ {k} | \sigma_ {\eta} (x - u _ {k}) + | f ^ {*} (x) |\right) d x \\ \leqslant | a _ {j} | \left(\| a \| _ {1} + M\right) \int_ {0} ^ {1} \left| \sigma_ {\eta} ^ {\prime} (x - u _ {j}) \right| d x \\ \leqslant \left| a _ {j} \right| \left(\left\| a \right\| _ {1} + M\right). \\ \end{array}
$$
$$
\left\| \nabla_ {u} L _ {\eta} (a, u) \right\| \leqslant \| a \| \left(\left\| a \right\| _ {1} + M\right) \leqslant \sqrt {m + 1} \| a \| ^ {2} + M \| a \|.
$$
Similarly, from the second equality, we have
$$
\left| \frac {\partial L _ {\eta}}{\partial a _ {j}} (a, u) \right| \leqslant \| a \| _ {1} + M.
$$
$$
\left\| \nabla_ {a} L _ {\eta} (a, u) \right\| \leqslant \sqrt {m + 1} \left(\| a \| _ {1} + M\right) = \sqrt {m + 1} \left(\| a \| \sqrt {m + 1} + M\right).
$$
Lemma 9. Consider  $\eta \geqslant 0$  and  $u\in \mathcal{U}_{\eta}$  such that there is a neuron at distance less than  $\eta$  from each discontinuity of  $f^{*}$  and  $3\eta \leqslant \Delta v$ . Then
$$
\int_ {0} ^ {1} \left| f _ {\eta} (x; a _ {\eta} ^ {*} (u), u) - f ^ {*} (x) \right| ^ {2} \mathrm {d} x \leqslant 6 M ^ {2} \eta n.
$$
Proof. By definition of  $a_{\eta}^{*}(u)$
$$
\int_ {0} ^ {1} \left| f _ {\eta} \left(x; a _ {\eta} ^ {*} (u), u\right) - f ^ {*} (x) \right| ^ {2} \mathrm {d} x = \min  _ {a \in \mathbb {R} ^ {m + 1}} \int_ {0} ^ {1} \left| f _ {\eta} \left(x; a, u\right) - f ^ {*} (x) \right| ^ {2} \mathrm {d} x.
$$
Thus it is enough to exhibit some  $a$  for which the latter integral is smaller than  $6M^2\eta n$  to conclude.
We construct such an  $a$  as follows: set  $a_0 = f^*(0)$ , and for each discontinuity  $v_i$ , set the coefficient of a neuron at distance less than  $\eta$  to the value  $f_{i}^{*} - f_{i-1}^{*}$  and set all other neurons to zero. Note that the active neurons are distinct since  $3\eta \leqslant \Delta v$ .
Then the neural network is equal to the target function everywhere except on an interval of size  $3\eta /2$  around each discontinuity, where they disagree (in infinite norm) by at most  $2M$ .
Lemma 10. Let  $m$  be a positive integer and  $u_{1},\ldots ,u_{m}$  be i.i.d. uniform random variables in [0, 1]. Assume that
$$
m \geqslant \frac {6}{\Delta v} \left(4 + \log n + \log \frac {1}{\delta}\right).
$$
Then, with probability at least  $1 - \delta$ , the vector  $u$  is  $D$ -good with  $D = \frac{\delta}{6(m + 1)^2}$ .
Proof. We define the following events:
(a)  $A$  is the event "there are at least 6 positions  $u_{j}$  in each interval  $[v_i,v_{i + 1}]$  for  $i\in \{0,\ldots ,n - 1\}$ ",
(b)  $B$  is the event  $\Delta (u)\geqslant D$
(c) for all  $i \in \{1, \dots, n - 1\}$ ,  $E_i$  is the event "there are at least one neuron on the left and on the right of  $v_i$ " and  $C_i$  is the event "  $E_i$  holds and  $|u_i^{\mathrm{R}} + u_i^{\mathrm{L}} - 2v_i| \geqslant D$ ".
Note that by Definition 2,  $u$  is  $D$ -good if and only if the event  $A \cap B \cap (\bigcap_{i} C_{i})$  holds. To show that this holds with high probability, we bound the probability of the complement
$$
\begin{array}{l} \left(A \cap B \cap \left(\bigcap_ {i} C _ {i}\right)\right) ^ {c} = A ^ {c} \cup B ^ {c} \cup \left(\bigcup_ {i} C _ {i} ^ {c}\right) = A ^ {c} \cup B ^ {c} \cup \left(\bigcup_ {i} \left(C _ {i} ^ {c} \cap A\right)\right) \\ \subset A ^ {c} \cup B ^ {c} \cup \left(\bigcup_ {i} \left(C _ {i} ^ {c} \cap E _ {i}\right)\right) \quad (\text {a s} A \subset E _ {i}). \\ \end{array}
$$
$$
\mathbb {P} (u \text {i s n o t} D \text {- g o o d}) \leqslant \mathbb {P} \left(A ^ {c}\right) + \mathbb {P} \left(B ^ {c}\right) + \sum_ {i = 1} ^ {n - 1} \mathbb {P} \left(C _ {i} ^ {c} \cap E _ {i}\right).
$$
Below, we bound separately the three terms of the right hand side.