SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we give precise mathematical statements leading to the convergence of the gradient flow to a global optimum, first in the two-timescale limit  $\varepsilon \to 0$ , then in the two-timescale regime with  $\varepsilon$  small but positive. All proofs are relegated to the Appendix.
5.1 In the two-timescale limit
This section analyses rigorously the two-timescale limit (5), which we recall for convenience:
$$
a ^ {*} (u) = \underset {a} {\arg \min } L (a, u), \quad \frac {\mathrm {d} u}{\mathrm {d} \tau} = - \nabla_ {u} L \left(a ^ {*} (u), u\right). \tag {8}
$$
We start by giving a rigorous meaning to these equations. First, for  $L$  to be differentiable in  $u$ , we require the parameter  $\eta$  of the non-linearity to be positive. Second, for  $a^*(u)$  to be well-defined, we need  $u \mapsto L(a, u)$  to have a unique minimum. Obviously, if the  $u_i$  are not distinct, then the features  $\{\sigma_\eta(. - u_1), \ldots, \sigma_\eta(. - u_m)\}$  are not independent and thus the minimum can not be unique. We restrict the state space of our dynamics to circumvent this issue. For  $u \in [0, 1]^m$ , we denote
$$
\Delta (u) = \min  _ {0 \leqslant j, k \leqslant m + 1, j \neq k} | u _ {j} - u _ {k} |,
$$
with the convention that  $u_0 = -\eta / 2$  and  $u_{m+1} = 1 + \eta / 2$ . Further, we define  $\mathcal{U} = \{u \in [0,1]^m \mid \Delta(u) > 2\eta\}$ . The proposition below shows that  $\mathcal{U}$  gives a good candidate for a set supporting solutions of (8).
Proposition 1. For  $u \in \mathcal{U}$ , the Hessian  $H(u)$  of the quadratic function  $L(.,u)$  is positive definite and its smallest eigenvalue is greater than  $\Delta(u)/8$ . In particular,  $L(.,u)$  has a unique minimum  $a^*(u)$ .
The bound on the Hessian is useful in the following, in particular in the proof of the following result.
Proposition 2. Let  $G(u) = \nabla_{u}L(a^{*}(u),u)$  for  $u\in \mathcal{U}$ . Then  $G:\mathcal{U}\to \mathbb{R}^m$  is Lipschitz-continuous.
Then, the Picard-Lindelöf theorem (see, e.g., Luk, 2017 for a self-contained presentation and Arnold, 1992 for a textbook) guarantees, for any initialization  $u(0) \in \mathcal{U}$ , the existence and unicity of a maximal solution of (8) taking values in  $\mathcal{U}$ . This solution is defined on a maximal interval  $[0, T_{\max})$  where it could be that  $T_{\max} < \infty$  if  $u$  hits the boundary of  $\mathcal{U}$ . However, the results below show that the target function  $f^{*}$  is recovered before this happens (with high probability over the initialization), and thus that this notion of solution is sufficient for our purposes. To this aim, we first define some sufficient conditions that the initialization should satisfy.
Definition 2. Let  $D$  be a positive real. We say that a vector of positions  $u \in [0,1]^m$  is  $D$ -good if
(a) for all  $i \in \{0, \dots, n - 1\}$ , there are at least 6 positions  $u_j$  in each interval  $[v_i, v_{i + 1}]$ .
(b)  $\Delta (u)\geqslant D$  , and
(c) for all  $i \in \{1, \dots, n - 1\}$ , denoting  $u_i^{\mathrm{L}}$  the position closest to the left of  $v_i$  and  $u_i^{\mathrm{R}}$  the position closest to the right, we have  $|u_i^{\mathrm{R}} + u_i^{\mathrm{L}} - 2v_i| \geqslant D$ .
Condition (a) is related to the fact that the derivation in Section 4.2 is valid only if there are at least two neurons per piece. This requirement that the neurons be distributed on every piece of the target seems to be necessary for our result to hold, and we provide in Appendix D.1 a counter-example where recovery fails otherwise. Condition (b) indicates that the neurons have to be sufficiently spaced at initialization, which is not surprising since we have to guarantee that  $\Delta(u(\tau)) > 2\eta$ , that is,  $u(\tau) \in \mathcal{U}$ , for all  $\tau$  until the recovery of  $f^*$  happens. Finally, condition (c) also helps to control the distance between neurons: although  $u_i^{\mathrm{L}}$  and  $u_i^{\mathrm{R}}$  move towards each other, as shown by (7), their distance can be controlled throughout the dynamics as a function of  $|u_i^{\mathrm{R}} + u_i^{\mathrm{L}} - 2v_i|$ .
We can now state the Proposition showing the recovery in finite time. The proof resembles the sketch of Section 4.2 with additional technical details since we need to control the distance between neurons, and the fact that  $\eta >0$  makes the dynamics more delicate to describe.
Proposition 3. Let  $f^{*} \in \mathcal{F}_{n,\Delta v,\Delta f,M}$ . Assume that the initialization  $u(0)$  is  $D$ -good with  $D = 2^{13/2}(m + 1)^{1/2}M\eta^{1/2}(\Delta f)^{-1}$ . Then the maximal solution of (8) taking values in  $\mathcal{U}$  is defined at least on  $[0,\mathcal{T}]$  for  $\mathcal{T} = 6 / (\Delta f)^2$ , and at the end of this time interval, there is a neuron at distance less than  $\eta$  from each discontinuity of  $f^{*}$ .
This Proposition is the main building block to show recovery in the next Theorem, along with some high-probability bounds to ensure that an i.i.d. uniform initialization is  $D$ -good.
Theorem 2. Let  $\xi, \delta > 0$ , and  $f^*$  a piecewise constant function from  $\mathcal{F}_{n,\Delta v,\Delta f,M}$ . Assume that the neural network has  $m$  neurons with
$$
m \geqslant \frac {6}{\Delta v} (4 + \log n + \log \frac {1}{\delta}).
$$
Assume that, at initialization, the positions  $u_{1}, \ldots, u_{m}$  of the neurons are i.i.d. uniformly distributed on  $[0, 1]$ . Then there exists  $Q$  depending on  $\xi, \delta, m, \Delta f, M$  such that, if
$$
\eta \leqslant Q,
$$
then, with probability at least  $1 - \delta$ , the maximal solution to the two-timescale limit (8) is defined at least until  $\mathcal{T} = \frac{6}{(\Delta f)^2}$ , and
$$
\int_ {0} ^ {1} \left| f ^ {*} (x) - f (x; a ^ {*} (u (\mathcal {T})), u (\mathcal {T})) \right| ^ {2} \mathrm {d} x \leqslant \xi .
$$
Furthermore, we have  $Q = \frac{C}{M^2} \min \left( \frac{\delta^2 (\Delta f)^2}{(m + 1)^5}, \frac{\xi}{n} \right)$  for some universal constant  $C > 0$ .
5.2 From the two-timescale limit to the two-timescale regime
We now briefly explain how the proof for the two-timescale limit can be adapted for the gradient flow problem in the two-timescale regime (1), that is with a small but non-vanishing  $\varepsilon$ . First note that the existence and uniqueness of the maximal solution to the dynamics (1) follow from the local Lipschitz-continuity of  $\nabla_{a}L$  and  $\nabla_{u}L$  with respect to both their variables.
The heuristics of Section 4.1 indicate that, for  $\varepsilon$  small enough, at any time  $t$ , the weights  $a(t)$  are close to  $a^{*}(u(t))$ , the global minimizer of  $L(\cdot ,u(t))$ . The next Proposition formalizes this intuition.
Proposition 4. Assume that  $a(0) = 0$  and that, for all  $s \in [0,t]$ , there at least 2 positions  $u_{j}(s)$  in each interval  $[v_i,v_{i + 1}]$  and  $\Delta (u(s))\geqslant D / 2$  for some  $D\geqslant 32\eta$ . Finally, assume that  $\varepsilon \leqslant 2^{-16}D^{2}M^{-2}(m + 1)^{-5 / 2}$ . Then
$$
\left\| a (t) - a ^ {*} (u (t)) \right\| \leqslant 3 M \sqrt {m + 1} \exp^ {- \frac {D}{1 6} t} + \frac {2 ^ {1 7} M ^ {3} (m + 1) ^ {3}}{D ^ {2}} \varepsilon .
$$
The crucial condition in the Proposition is  $\Delta(u(s)) \geqslant D/2$ ; it is useful to control the conditioning of the quadratic form  $L(\cdot, u(s))$ . The Proposition shows that  $\|a(t) - a^{*}(u(t))\|$  is upper bounded by the sum of two terms; the first term is a consequence of the initial gap between  $a(0)$  and  $a^{*}(u(0))$  and decays exponentially quickly. The second term is negligible in the regime  $\varepsilon \ll 1$ .
Armed with this Proposition, we show that the two-timescale regime has the same behavior as the two-timescale limit and thereby prove Theorem 1.
6 Numerical experiments and discussion
Numerical illustration in the setting of Section 2. We first compare the dynamics of the gradient flow in the two-timescale limit presented in Section 4.2 with the dynamics of SGD. To simulate the SGD dynamics, we assume that we have access to noisy observations of the value of  $f^{*} \in \mathcal{F}_{n,\Delta v,\Delta f,M}$ : let  $(X_{p},Y_{p})_{p\geqslant 1}$  be i.i.d. random variables such that  $X_{p}$  is uniformly distributed on  $[0,1]$ , and  $Y_{p} = f^{*}(X_{p}) + N_{p}$  where  $N_{p}$  is additive noise. The (one-pass) SGD updates are then given by
$$
a _ {p + 1} = a _ {p} - h \nabla_ {a} \ell \left(X _ {p + 1}, Y _ {p + 1}; a _ {p}, u _ {p}\right), \tag {9}
$$
$$
u _ {p + 1} = u _ {p} - \varepsilon h \nabla_ {u} \ell (X _ {p + 1}, Y _ {p + 1}; a _ {p}, u _ {p}),
$$
with  $\ell(X, Y; a, u) = \frac{1}{2}(Y - f(X; a, u))^2$ . The experimental settings, as well as additional results, are given in the Appendix.
Remarkably, the dynamics of SGD in the two-timescale regime with  $\eta$  small match closely the gradient flow in the two-timescale limit with  $\eta = 0$ , as illustrated in Figure 3. This validates the use
of the gradient flow to understand the training dynamics with SGD. Both dynamics are close until the two-timescale limit achieves perfect recovery of the target function, at which point the SGD stabilizes to a small non-zero error. The fact that SGD does not achieve perfect recovery is not surprising, since SGD is performed with  $\eta > 0$  and  $f^{*}$  is not in the span of  $\{1, \sigma_{\eta}(x - u_1), \ldots, \sigma_{\eta}(x - u_m)\}$  for any  $u_1, \ldots, u_m$  and for  $\eta > 0$ . On the contrary, we simulated the dynamics of gradient flow for  $\eta = 0$ , as presented in Section 4.2, enabling perfect recovery in that case.
(a) Loss  $L$  as a function of the number of steps  $p$
(b) Plot of the functions after  $p = 2.7 \cdot 10^{7}$  steps
Figure 3: Comparison between the SGD (9) with  $\eta = 4\cdot 10^{-3}$  in the two-timescale regime ( $\varepsilon = 2\cdot 10^{-5}$ ) and the gradient flow in the two-timescale limit (5) with  $\eta = 0$ . In the left-hand plot, to align the SGD and the two-timescale limit, we take  $\tau = \varepsilon hp$ . In the right-hand plot, the target function is in blue, the gradient flow in the two-timescale limit is in green, and the SGD is in orange.
Next, we compare the SGD dynamics in the two-timescale regime ( $\varepsilon \ll 1$ ) and outside of this regime ( $\varepsilon \approx 1$ ). In Figure 4, we see that the network trained by SGD (in orange) in the two-timescale regime  $\varepsilon = 2 \cdot 10^{-5}$ , achieves near-perfect recovery. If we change  $\varepsilon$  to 1, while keeping all other parameters equal, the algorithm fails to recover the target function (Figure 5). This shows that, in our setting with a moderate number of neurons  $m$ , recovery can fail away from the two-timescale regime. It could seem that we are favouring the two-timescale regime by running it for more steps. In fact, it is not the case since both regimes are run until convergence. We refer to Appendix C for details.
Note that the dynamics of the neurons in Figures 4 and 5 are different. In the two-timescale regime, only the neurons closest to a discontinuity move significantly, while the others do not. These dynamics correspond to the sketch of Section 4. Interestingly, it means that in this regime, the neural network learns a sparse representation of the target function, meaning that only  $n$  out of the  $m$  neurons are active after training. On the contrary, when  $\varepsilon = 1$ , all neurons move to align with discontinuities of the target function, thus the learned representation is not sparse. Furthermore, since the number of neurons is moderate, one of the discontinuities is left without any neuron.
(a) At initialization  $p = 0$
Figure 4: Simulation in the two-timescale regime ( $\varepsilon = 2 \cdot 10^{-5}$ ). The target function is in blue and the SGD (9) is in orange with  $\eta = 4 \cdot 10^{-3}$ ,  $h = 10^{-5}$ . The positions  $u_{1},\ldots ,u_{m}$  of the neurons are indicated with vertical dotted lines. In a first short phase, only the weights  $a_1,\dots,a_m$  of the neurons evolve to match as best as possible the target function (second plot). Then, in a longer phase, the neuron closest to each target discontinuity moves towards it (third plot). Recovery is achieved.
(b) After  $p = 5.4 \cdot 10^{6}$  steps
(c) After  $p = 1.8 \cdot 10^{8}$  steps
(a) At initialization  $p = 0$
(b) After  $p = 10^4$  steps
Figure 5: Simulation outside of the two-timescale regime  $(\varepsilon = 1)$ . The target function is in blue and the SGD (9) is in orange with  $\eta = 4 \cdot 10^{-3}$ ,  $h = 10^{-5}$ . The positions  $u_{1}, \ldots, u_{m}$  of the neurons are indicated with vertical dotted lines. The dynamics create a zone with no neuron, hindering recovery.
(c) After  $p = 10^6$  steps
Discussion. The two-timescale regime decouples the dynamics of the two layers of the neural network. As a consequence, it is a useful theoretical tool to simplify the evolution equations. In this paper showcasing the approach, the two-timescale regime enables to show the alignment of the neurons with the discontinuities of the target function in the piecewise constant 1D case, and thus to prove recovery. A full general understanding of the impact of the two-timescale regime is an open question, which is left for future work. We provide in the following some practical evidence of the applicability of this regime to other settings (higher-dimensional problems, ReLU networks, finite sample size). Additional technical difficulties significantly complicate the proof in these settings, but we believe that there is no fundamental reason that our mathematical approach should not apply.
Higher dimensions. We consider piecewise constant functions on  $\mathbb{R}^d$  with pieces that are cuboids aligned with the axes of the space (see Figure 6). Neural networks are of the form  $f(x;a,u) = a_0 + \sum_{j=1}^{m}\sum_{k=1}^{d}a_{jk}\sigma_\eta(x_k - u_{jk})$ , where the  $j$ -th neuron has  $d$ -dimensional position  $(u_{jk})_{1\leqslant k\leqslant d}$  and weight  $(a_{jk})_{1\leqslant k\leqslant d}$ . The results are similar to the 1D case: convergence to a global minimum is obtained in the two-timescale regime but not in the standard regime.
(a) After training in the standard regime
Figure 6: 2D experiment with a piecewise-constant target (each shade of blue depicts a constant piece). The orange lines show the positions of the neurons after training by SGD. In the standard regime, a discontinuity of the target at  $x = 0.3$  is not covered by a neuron. In the two-timescale regime, all the target discontinuities are covered. See Appendix C for more results with  $d = 2, 10$ .
(b) After training in the two-timescale regime
ReLU networks. Appendix C reports the case of using ReLU activations to approximate piecewise-affine targets. A similar conclusion holds.
Finite sample size. We believe that it should be possible to generalize our results to finite sample sizes (say, for single-pass SGD), following a perturbation analysis similar to the one of Section 5.2, with additional terms due to random sampling. Numerically, Figure 3 shows that SGD indeed closely follows the behavior of the two-timescale limit. Some ideas about the proof are provided in Appendix D.2.
The authors thank Emmanuel Abbé for many discussions and suggestions on this project, and Quentin Berthet for pointing them towards the envelope theorem. The authors are grateful towards the Simons Institute for the Theory of Computing where this work was initiated. P.M. is supported by a grant from Région Île-de-France, by a Google PhD Fellowship and by Mines Paris - PSL.
Emmanuel Abbe, Enric Boix Adserà, and Theodor Misiakiewicz. Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics. In Gergely Neu and Lorenzo Rosasco, editors, Proceedings of Thirty Sixth Conference on Learning Theory, volume 195 of Proceedings of Machine Learning Research, pages 2552-2623. PMLR, 2023.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via overparameterization. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 242-252, 2019.
Vladimir Arnold. Ordinary Differential Equations. Springer Berlin Heidelberg, 1992.
Nils Berglund and Barbara Gentz. Noise-induced phenomena in slow-fast dynamical systems: a sample-paths approach. Springer Science & Business Media, 2006.
Kim Border. Miscellaneous notes on optimization theory and related topics, 2015. URL: https://healy.econ.ohio-state.edu/kcb/AddedByPJ/Maximization.pdf (version: 2023-02-08).
Vivek Borkar. Stochastic approximation with two time scales. Systems & Control Letters, 29(5): 291-294, 1997.
Vivek Borkar. Stochastic approximation: a dynamical systems viewpoint, volume 48. Springer, 2009.
Léon Bottou, Frank Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM review, 60(2):223-311, 2018.
Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over-parameterized models using optimal transport. Advances in Neural Information Processing Systems, 31, 2018.
Alexandru Damian, Jason Lee, and Mahdi Soltanolkotabi. Neural networks can learn representations with gradient descent. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pages 5413-5452. PMLR, 2022.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1675-1685, 2019.
Sebastian Goldt, Madhu Advani, Andrew Saxe, Florent Krzakala, and Lenka Zdeborova. Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup. Journal of Statistical Mechanics: Theory and Experiment, 2020(12):124010, 2020.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic. SIAM Journal on Optimization, 33(1):147-180, 2023.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339, 2018.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems, 31, 2018.
Yunyong Ko, Dongwon Lee, and Sang-Wook Kim. Not all layers are equal: A layer-wise adaptive approach toward large-scale DNN training. In Proceedings of the ACM Web Conference 2022, page 1851-1859, 2022.
Fei-Fei Li, Jiajun Wu, and Ruohan Gao. Deep learning for computer vision course, 2022. URL: https://cs231n.github.io/transfer-learning/ (version: 2023-02-02).
Jonathan Luk. Notes on existence and uniqueness theorems for ODEs, 2017. URL: http://web.stanford.edu/~jluk/math63CMspring17/Existence.170408.pdf (version: 2023-01-19).
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33):E7665-E7671, 2018.
Youngmin Ro and Jin Young Choi. AutoLR: Layer-wise pruning and auto-tuning of learning rates in fine-tuning of deep networks. Proceedings of the AAAI Conference on Artificial Intelligence, 35 (3):2486-2494, 2021.
Grant Rotskoff and Eric Vanden-Eijnden. Trainability and accuracy of neural networks: An interacting particle system approach. arXiv preprint arXiv:1805.00915, 2018.
David Saad and Sara Solla. On-line learning in soft committee machines. Phys. Rev. E, 52:4225-4243, Oct 1995.
Itay Safran, Gal Vardi, and Jason D Lee. On the effective number of linear regions in shallow univariate relu networks: Convergence guarantees and implicit bias. Advances in Neural Information Processing Systems, 35, 2022.
Bharat Singh, Soham De, Yangmuzi Zhang, Thomas Goldstein, and Gavin Taylor. Layer-specific adaptive learning rates for deep networks. In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), pages 364-368, 2015.
Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of neural networks: a central limit theorem. Stochastic Processes and their Applications, 130(3):1820-1852, 2020.
Csaba Szepesvári. Algorithms for reinforcement learning. Springer, 2010.
Rodrigo Veiga, Ludovic Stephan, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborova. Phase diagram of stochastic gradient descent in high-dimensional two-layer neural networks. Advances in Neural Information Processing Systems, 35, 2022.
Yang You, Igor Gitman, and Boris Ginsburg. Scaling SGD batch size to 32k forImagenet training. arXiv preprint arXiv:1708.03888, 2017.
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes overparameterized deep ReLU networks. Machine Learning, 109(3):467-492, 2020.
Organization of the Appendix In Section A, we introduce additional notations that will be used throughout the Appendix, then proceed to prove useful technical lemmas. We proceed in Section B to prove the results presented in the main text. Section C contains details about our experimental settings as well as some additional simulations. Section D gives a couple of additional insights about our results.
A Additional notations and technical lemmas
For a vector  $a$ , we denote  $\|a\|$  its  $\ell^2$ -norm,  $\|a\|_1$  its  $\ell^1$ -norm and  $\|a\|_{\infty}$  its  $\ell^{\infty}$ -norm. For matrices  $H$ ,  $\|H\|$  denotes the operator norm associated to the  $\ell^2$  norm and  $\|H\|_{\mathrm{F}}$  denotes the Frobenius norm. For real-valued functions  $f$ ,  $\|f\|_{\infty}$  denotes the supremum norm.
In all of the Appendix, we denote  $u_0 = -\eta / 2$  and  $u_{m+1} = 1 + \eta / 2$ . Note that  $\sigma_\eta(x - u_0) = 1$  for all  $x \in [0,1]$ , meaning that  $\sigma_\eta(\cdot - u_0)$  corresponds to the bias term. This notation allows us to treat the bias term in a unified fashion with respect to the other terms of  $f(x; a, u)$ . Since  $u_i \in (0,1)$  for  $i \in \{1, \ldots, m\}$ , we assume in the following w.l.o.g. that the  $(u_i)_{0 \leqslant i \leqslant m+1}$  are ordered in increasing order. Note that we prove in the following that the  $(u_i)_{1 \leqslant i \leqslant m}$  do not cross during the dynamics, so they remain ordered throughout the dynamics.
The proofs involve comparisons of some quantities when  $\eta > 0$  and when  $\eta = 0$ . To avoid confusion, we make explicit the dependency of  $L$  on  $\eta \geqslant 0$ , i.e., we let  $L_{\eta}(a, u)$  in place of  $L(a, u)$  of the main paper, and similarly, when the arg  $\min$  is well-defined and unique,
$$
a_{\eta}^{*}(u) = \operatorname *{arg  min}_{a\in \mathbb{R}^{m + 1}}L_{\eta}(a,u).
$$
in place of  $a^*(u)$ . Similarly, we now make explicit the dependence of  $f$  on  $\eta \geqslant 0$ , i.e., we denote
$$
f _ {\eta} (x; a, u) = a _ {0} + \sum_ {j = 1} ^ {m} a _ {j} \sigma_ {\eta} (x - u _ {j}) = \sum_ {j = 0} ^ {m} a _ {j} \sigma_ {\eta} (x - u _ {j}).
$$
The Hessian of the quadratic function  $L_{\eta}(\cdot ,u)$  is denoted  $H_{\eta}(u)\in \mathbb{R}^{(m + 1)\times (m + 1)}$  (in place of  $H(u)$ ), and satisfies that, for  $i,j\in \{0,\ldots ,m\}$ ,
$$
H _ {\eta , i j} (u) = \int_ {0} ^ {1} \sigma_ {\eta} (x - u _ {i}) \sigma_ {\eta} (x - u _ {j}) \mathrm {d} x.
$$
Also let, for  $\eta \geqslant 0$  and  $u\in \mathbb{R}^m$ ,  $b_{\eta}(u)\in \mathbb{R}^{m + 1}$  such that, for  $j\in \{0,\dots ,m\}$
$$
b _ {\eta , j} (u) = \int_ {0} ^ {1} f ^ {*} (x) \sigma_ {\eta} (x - u _ {j}) \mathrm {d} x.
$$
Finally, we let  $\mathcal{U}_{\eta}$  in place of  $\mathcal{U}$  in the paper.
With these notations, we have, for  $\eta \geqslant 0$  and  $a, u \in \mathbb{R}^m$ ,