SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Urte Adomaityte
urte.adomaityte@kcl.ac.uk
Gabriele Sicuro
gabriele.sicuro@kcl.ac.uk
Pierpaolo Vivo
pierpaolo.vivo@kcl.ac.uk
We characterise the learning of a mixture of two clouds of data points with generic centroids via empirical risk minimisation in the high dimensional regime, under the assumptions of generic convex loss and convex regularisation. Each cloud of data points is obtained via a double-stochastic process, where the sample is drawn from a Gaussian distribution whose variance is itself a random parameter sampled from a scalar distribution  $\varrho$ . As a result, our analysis covers a large family of data distributions, including the case of power-law-tailed distributions with no covariance, and allows us to test recent "Gaussian universality" claims. We study the generalisation performance of the obtained estimator, we analyse the role of regularisation, and we analytically characterise the separability transition.
Generalised linear models (GLMs) are still ubiquitous in the theoretical research on machine learning, despite their simplicity. Their nontrivial phenomenology is often amenable to complete analytical treatment and has been key to understanding the unexpected behavior of large and complex architectures. Random features models [61, 47, 24], for example, allowed to clarify many aspects of the well-known double-descent phenomenon in neural networks [54, 8, 9]. A line of research spanning more than three decades [68, 74] has considered a variety of GLMs to investigate a number of aspects of learning in high dimensions. Yet, a crucial assumption adopted in many such theoretical models is that the covariates are obtained from a Gaussian distribution, or from a mixture of Gaussian distributions [42, 49, 3, 37, 43]. Although such a Gaussian design has served as a convenient working hypothesis (in some cases experimentally and theoretically justified [67, 44]), it is however not obvious how much it limits the scope of the results. It is reasonable to expect structure, heavy tails, and large fluctuations in real (often non-Gaussian) data [1, 62] to play an important role in the learning process, and it would be therefore desirable to include structured and heavy-tailed-distributed covariates in our theoretical toolbox.
This paper presents, to the best of our knowledge for the first time, the exact asymptotics for classification tasks on covariates obtained from a mixture of heavy-tailed distributions. We focus on supervised binary classification assuming that the sample size  $n$  and the dimensionality  $d$  of the space where the covariates live are both sent to infinity, keeping their ratio  $n / d = \alpha$  fixed. The paper fits therefore in the line of works on exact high-dimensional asymptotics for classification learning via a GLM [49, 43], but, crucially, we relax the usual Gaussian data hypothesis by including in our
analysis power-law-tailed distributions with possibly no covariance. The mixture is obtained from two distributions, each centered around a centroid  $\mu \in \mathbb{R}^d$  and resulting from a double stochastic process. Namely, each sample point is obtained from a distribution  $\mathcal{N}(\boldsymbol{\mu},\boldsymbol{\Sigma})$  whose covariance  $\boldsymbol{\Sigma} = \Delta \boldsymbol{I}_d \in \mathbb{R}^{d\times d}$  is itself a random variable so that  $\Delta$  has density  $\varrho$ , supported on  $\mathbb{R}_*^+$ . Using, for example, an appropriately parametrised inverse-Gamma distribution for  $\varrho$ , such a superstatistical construction (as known in the physics literature [5, 7]) can provide, for example, a heavy-tailed, Cauchy-like data distribution with infinite covariance. The replica method [48], an analytical tool widely adopted in statistical physics, provides asymptotic formulas for a generic density  $\varrho$ , allowing us to study the effects of non-Gaussianity on the performance curves, and test Gaussian universality hypotheses [59] in such a classification task.
Motivation and related works Learning a rule to classify data points clustered in clouds in high dimension is a classical problem in statistics [32]. Its ubiquity is exemplified by the recently observed neural collapse phenomenon in deep neural networks, in which the last layer classifier was found to operate on features clustered in clouds around the vertices of a simplex [38, 56]; more recently, Seddik et al. [67] showed that Gaussian mixtures suitably describe the deep learning representation of GAN data. In theoretical models, data points are typically assumed to be organised in  $K$  Gaussian clouds, each corresponding to a class. Each class  $k$ ,  $k \in \{1, \dots, K\}$ , is centered around a mean vector  $\pmb{\mu}_k$  and has covariance  $\pmb{\Sigma}_k$ . The binary classification case,  $K = 2$ , is the simplest, and possibly most studied, setting. For this case, Mai and Liao [46] considered an ERM task with generic convex loss and ridge regularisation in the high-dimensional regime  $n$ ,  $d \to +\infty$  with  $n / d \in (0, +\infty)$ . In this setting, they gave a precise prediction of the classification error, showing the optimality of the square loss in the unregularised case. Their results have been extended by Mignacco et al. [49], who showed that the presence of a regularisation can actually drastically affect the performance, improving it. In the same setting, the Bayes optimal estimator has been studied in both the supervised and the semi-supervised setting [49, 41]; almost-Bayes-optimal solutions have been put in relation to wide flat landscape regions [3]. In the context of binary classification, the maximum number  $n$  of samples that can be perfectly fitted by a linear model in dimension  $d$  [49, 16, 37] has been the topic of investigation since the seminal work of Cover [12] and is related to the classical storage capacity problems on networks [20, 21, 39]. The corresponding separability transition value  $\alpha = n / d$  is remarkably associated with the existence transition of the maximum likelihood estimator [70, 76]. The precise asymptotic characterisation of the test error in learning to classify  $K \geq 2$  Gaussian clouds with generic means and covariances has been recently obtained by Loureiro et al. [43]. Within this line of research, rigorous results have been obtained by a variety of methods, such as Gordon's inequality technique [31, 71, 49] or mapping to approximate message passing schemes [4, 36, 10, 43, 27].
As previously mentioned, the working hypothesis of Gaussian design is widely adopted in high-dimensional classification problems. On top of being a convenient technical hypothesis, this assumption has been justified in terms of a "Gaussian universality" principle. In other words, in a number of circumstances, non-Gaussian features distributions are found to be effectively described by Gaussian ones with matching first and second moments as far as the asymptotic properties of the estimators are concerned [44]. Such a "universality property" has been rigorously proven for example in compressed sensing [50] or in the case of LASSO with non-Gaussian dictionaries [55]. A Gaussian equivalence principle introduced by Goldt et al. [30] has been proven to hold for a wide class of generalised linear estimation problems [47]. Extending work by Hu and Lu [34], Montanari and Saeed [51] recently proved such a principle in a GLM under the assumption of pointwise normality of the distribution of the features. This crucial assumption can be intended, roughly speaking, as the assumption of sub-Gaussian decay of a marginal of the feature distribution in any direction (see also [25, 13] for further developments).
The Gaussian universality principle, however, can break down by relaxing some of the aforementioned assumptions on the feature distribution. Montanari and Saeed [51] for example showed that pointwise normality is a necessary hypothesis for universality. Studying regression tasks on an elliptically distributed dataset, El Karoui [18] showed that claims of "universality" obtained in the Gaussian setting require serious scrutiny, as the statistics of the estimators might strongly depend on the non-Gaussianity of the covariates [18, 69, 72]. More recently, Pesce et al. [59] showed that a structured Gaussian mixture cannot be effectively described by a single Gaussian cloud. Building upon a series of contributions related to the asymptotic performance of models in the proportional high-dimensional limit [33, 47, 28, 24, 30, 44], we aim precisely to explore the validity of Gaussian universality within classification problems.
In this paper, we work in a "superstatistical" data setting [5, 7], meaning that we superpose the statistics of Gaussian data distribution with an assumed distribution of its variance. Such a construction is adopted in a number of disciplines and contexts to go beyond Gaussianity, albeit is known under different names. In statistical physics, it is known as "superstatistics" and is employed in the analysis of non-equilibrium and non-linear systems [6]. In Bayesian modeling, it is common to refer to hierarchical priors and models [22, 23], while in probability, statistics, and actuarial sciences such distributions are known as compound probability distributions [63], or doubly-stochastic models [60, 66]. Crucially, this construction allows us to consider a very large family of non-Gaussian distributions, which include, but are not limited to, any power-law decay and Cauchy-like with possible infinite-covariance parametrisations.
Our contributions In this manuscript we provide the following results.
- We study a classification task on a non-Gaussian mixture model (see Eq. (1) below) via a generalised linear model (GLM) and we analytically derive, using the replica method [14, 19, 48], an asymptotic characterisation of the statistics of the empirical risk minimisation (ERM) estimator. Our results go therefore beyond the usual Gaussian assumption for the dataset, and include for example the case of covariates obtained from a mixture of distributions with infinite variance. The analysis is performed in the high-dimensional, proportional limit and for any convex loss and convex regularisation. By using this result, we provide asymptotic formulas for the generalisation, training errors and training loss.
- We analyse the performance of this ERM task on a specific family of dataset distributions by using different convex loss functions (quadratic and logistic) and ridge regularisation. We show in particular that, in the case of two balanced clusters with a specific non-Gaussian distribution, the optimal ridge regularisation strength  $\lambda^{\star}$  is finite, at odds with the Gaussian case, for which  $\lambda^{\star} \to \infty$  [49]. In this respect, by considering distributions with matching first and second moments, we analytically show that the performances of the analysed GLM do depend in general on higher moments, and therefore the "Gaussian universality principle" breaks down when heavy-tailed distributions are considered.
- We derive the separability threshold on a large family of non-Gaussian dataset distributions, possibly with unbounded covariance, generalising the known asymptotics for the separability of Gaussian clouds [49]. The result of Cover [12] is recovered in the case of infinite distribution width. Under some moment conditions, we derive the Bayes-optimal performance in the case of binary classifications with symmetric centroids, generalising the argument in Ref. [49].
- We finally extend recent results on Gaussian universality in the Gaussian mixture model with random labels [25, 59] to the case of non-Gaussian distributions. We show that the universal formula for the square training loss at zero regularisation found in Ref. [25] holds in our more general setting as well.
2 Main result
Dataset construction We consider the task of classifying two data clusters in the  $d$ -dimensional space  $\mathbb{R}^d$ . The dataset  $\mathcal{D} := \{(x^\nu, y^\nu)\}_{\nu \in [n]}$  is obtained by extracting  $n$  independent datapoints  $x^\nu$ , each associated with a label  $y^\nu \in \{-1, 1\}$  (a more general setting, involving a multiclass classification task, is discussed in Appendix A). The data points are correlated with the labels via a law  $P(x, y)$  which we assume to have the form
$$
P (\boldsymbol {x}, y) = \delta_ {y, 1} \rho P (\boldsymbol {x} \mid \boldsymbol {\mu} _ {+}) + \delta_ {y, - 1} (1 - \rho) P (\boldsymbol {x} \mid \boldsymbol {\mu} _ {-}), \quad \rho \in (0, 1), \quad \boldsymbol {\mu} _ {\pm} \in \mathbb {R} ^ {d}. \tag {1a}
$$
Here  $P(\pmb{x}|\pmb{\mu})$  is a distribution with mean  $\pmb{\mu}$ , and the mean vectors  $\pmb{\mu}_{\pm} \in \mathbb{R}^{d}$  are distributed according to some density, such that  $\mathbb{E}\left[\| \pmb{\mu}\|^{2}\right] = \Theta(1)$ , and correspond to the center of the two clusters. The scalar quantity  $\rho$  weighs the relative contribution of the two clusters: in the following, we will denote  $\rho_{+} = \rho = 1 - \rho_{-}$ . Each cluster distribution  $P(\pmb{x}|\pmb{\mu})$  around a vector  $\pmb{\mu}$  is assumed to have the form
$$
P (\boldsymbol {x} \mid \boldsymbol {\mu}) := \mathbb {E} _ {\Delta} \left[ \mathcal {N} \left(\boldsymbol {x} \mid \boldsymbol {\mu}, \Delta \boldsymbol {I} _ {d}\right) \right], \tag {1b}
$$
where  $\mathcal{N}(\pmb{x}|\pmb{\mu},\pmb{\Sigma})$  is a Gaussian distribution with mean  $\pmb{\mu}\in \mathbb{R}^{d}$  and covariance  $\pmb{\Sigma}\in \mathbb{R}^{d\times d}$ , and  $\Delta$  is randomly distributed with some density  $\varrho$  with support on  $\mathbb{R}_*^+ := (0, +\infty)$ . The family of "elliptic-like" distributions in Eq. (1b) has been extensively studied, for instance, by the physics community, in the context of superstatistics [5, 7]. Mixtures of normals in the form of Eq. 1b are a central tool in Bayesian statistics [65] due to their ability to approximate any distribution given a sufficient number
of components [53, 2, 29]. Although the family in Eq. (1b) is not the most general of such mixtures, it is sufficient to include a large family of power-law-tailed densities and to allow us to go beyond the usual Gaussian approximation. El Karoui [18] considered the statistical properties, in the same asymptotic proportional regime as here, of ridge-regularised regression estimators on datasets with distribution as in Eq. 1b, under the assumption (here relaxed) that  $\mathbb{E}[\Delta^2 ] < + \infty$ . This elliptic family includes a large class of distributions with properties markedly different from the ones of Gaussians. For example, the inverse-Gamma with density  $\varrho (\Delta) = (2\pi \Delta^{3})^{-1 / 2}\mathrm{e}^{-\frac{1}{2\Delta}}$  leads to a  $d$ -dimensional Cauchy-like distribution  $P(x|\mu)\propto (1 - \| \pmb {\mu} - \pmb {x}\| ^2)^{-\frac{d + 1}{2}}$ , having as marginals Cauchy distributions and  $\mathbb{E}[\| x - \pmb {\mu}\| ^2 ] = +\infty$ . We will perform our classification task by searching for a set of parameters  $(w^{\star},b^{\star})$ , called weights and bias, respectively, that will allow us to construct an estimator via a certain classifier  $\varphi \colon \mathbb{R}\to \{-1,1\}$
$$
\boldsymbol {x} \mapsto \varphi \left(\frac {\boldsymbol {x} ^ {\top} \boldsymbol {w} ^ {\star}}{\sqrt {d}} + b ^ {\star}\right). \tag {2}
$$
By means of the above law, we will predict the label for a new, unseen data point  $\pmb{x}$  sampled from the same law  $P(\pmb{x},\pmb{y})$ . Our analysis will be performed in the high-dimensional limit where both the sample size  $n$  and dimensionality  $d$  are sent to infinity, with  $n / d \equiv \alpha$  kept constant.
Learning task In the most general setting, the parameters are estimated by minimising an empirical risk function in the form
$$
\left(\boldsymbol {w} ^ {\star}, b ^ {\star}\right) \equiv \underset { \begin{array}{c} \boldsymbol {w} \in \mathbb {R} ^ {d} \\ b \in \mathbb {R} \end{array} } {\arg \min } \mathcal {R} (\boldsymbol {w}, b) \quad \text {w h e r e} \quad \mathcal {R} (\boldsymbol {w}, b) \equiv \sum_ {\nu = 1} ^ {n} \ell \left(y ^ {\nu}, \frac {\boldsymbol {w} ^ {\top} \boldsymbol {x} ^ {\nu}}{\sqrt {d}} + b\right) + \lambda r (\boldsymbol {w}). \tag {3}
$$
Here  $\ell$  is a strictly convex loss function with respect to its second argument, and  $r$  is a strictly convex regularisation function with the parameter  $\lambda \geq 0$  tuning its strength.
State evolution equations. Let us now present our main result, namely the exact asymptotic characterisation of the distribution of the estimator  $\pmb{w}^{\star} \in \mathbb{R}^{d}$  and of  ${}^{1}/\sqrt{a}\pmb{w}^{\star \top}X \in \mathbb{R}^{n}$ , where  $X \in \mathbb{R}^{d \times n}$  is the concatenation of the  $n$  dataset column vectors  $\pmb{x}^{\nu} \in \mathbb{R}^{d}$ ,  $\nu \in [n]$ . Such an asymptotic characterisation is performed via a set of order parameters satisfying a system of self-consistent "state-evolution" equations, which we will solve numerically. The asymptotic expressions for generalisation and training errors, in particular, are written in terms of their solutions, the order parameters. In the following, we use the shorthand  $\ell_{\pm}(u) \equiv \ell(\pm 1, u)$ , and given a function  $\Phi: \{-1, 1\} \to \mathbb{R}$ , we write  $\Phi_{\pm} \equiv \Phi(\pm 1)$  and  $\mathbb{E}_{\pm}[\Phi_{\pm}] := \rho_{+}\Phi_{+} + \rho_{-}\Phi_{-}$ .
Result 2.1 Let  $\zeta \sim \mathcal{N}(0,1)$  and  $\Delta \sim \varrho$ , both independent from other quantities. Let also be  $\xi \sim \mathcal{N}(\mathbf{0},I_d)$ . In the setting described above, given  $\phi_1\colon \mathbb{R}^d\to \mathbb{R}$  and  $\phi_{2}\colon \mathbb{R}^{n}\to \mathbb{R}$ , the estimator  $w^{\star}$  and the vector  $z^{\star} := \frac{1}{\sqrt{d}} w^{\star}X\in \mathbb{R}^{n}$  verify:
$$
\phi_ {1} \left(\boldsymbol {w} ^ {\star}\right) \xrightarrow [ n, d \rightarrow + \infty ]{\mathrm {P}} \mathbb {E} _ {\xi} \left[ \phi_ {1} (\boldsymbol {g}) \right], \quad \phi_ {2} \left(z ^ {\star}\right) \xrightarrow [ n, d \rightarrow + \infty ]{\mathrm {P}} \mathbb {E} _ {\zeta} \left[ \phi_ {2} (\boldsymbol {h}) \right], \tag {4}
$$
where we have introduced the proximal  $\pmb{g} \in \mathbb{R}^d$ , defined as
$$
\boldsymbol {g} := \arg \min  _ {\boldsymbol {w}} \left(\hat {v} \frac {\| \boldsymbol {w} \| ^ {2}}{2} - \sqrt {d} \sum_ {k = \pm} \hat {m} _ {k} \boldsymbol {w} ^ {\top} \boldsymbol {\mu} _ {k} - \sqrt {\hat {q}} \boldsymbol {\xi} ^ {\top} \boldsymbol {w} + \lambda r (\boldsymbol {w})\right) \tag {5}
$$
and where we also have introduced the proximal for the loss  $\pmb{h} \in \mathbb{R}^n$ , obtained by concatenating  $\rho_{+}n$  quantities  $h_{+}$  with  $\rho_{-}n$  quantities  $h_{-}$ , with  $h_{\pm}$  given by
$$
h _ {\pm} := \arg \min  _ {u} \left[ \frac {(u - \omega_ {\pm}) ^ {2}}{2 \Delta v} + \ell_ {\pm} (u) \right], \quad \text {w h e r e} \omega_ {\pm} := m _ {\pm} + b + \sqrt {q \Delta} \zeta . \tag {6}
$$
The collection of parameters  $(q, m_{\pm}, \nu, \hat{q}, \hat{m}_{\pm}, \hat{\nu}, b)$  appearing in the equations above is given by the fixed-point solution of the following self-consistent equations:
$$
\left\{ \begin{array}{l} m _ {\pm} = \frac {1}{\sqrt {d}} \mathbb {E} _ {\boldsymbol {\xi}} \left[ \boldsymbol {g} ^ {\top} \boldsymbol {\mu} _ {\pm} \right], \\ q = \frac {1}{d} \mathbb {E} _ {\boldsymbol {\xi}} \left[ \| \boldsymbol {g} \| ^ {2} \right], \\ v = \frac {1}{d} \hat {q} ^ {- 1 / 2} \mathbb {E} _ {\boldsymbol {\xi}} \left[ \boldsymbol {g} ^ {\top} \boldsymbol {\xi} \right], \end{array} \right. \quad \left\{ \begin{array}{l l} \hat {q} = \alpha \mathbb {E} _ {\pm , \zeta , \Delta} \left[ \Delta f _ {\pm} ^ {2} \right], \\ \hat {v} = - \alpha q ^ {- 1 / 2} \mathbb {E} _ {\pm , \Delta , \zeta} \left[ \sqrt {\Delta} f _ {\pm} \zeta \right], \\ \hat {m} _ {\pm} = \alpha \rho_ {\pm} \mathbb {E} _ {\Delta , \zeta} \left[ f _ {\pm} \right], \end{array} \right. \quad f _ {\pm} := \frac {h _ {\pm} - \omega_ {\pm}}{v \Delta}, (7)
$$
The derivation of the result above is given, in a multiclass setting, in Appendix A using the replica method, which can be put on rigorous ground by a mapping of the risk minimisation problem into an approximate message-passing iteration scheme [17, 43, 26]. As in the purely Gaussian case, the state evolution equations naturally split the dependence on the loss function  $\ell$ , which is relevant in the computation of  $h_{\pm}$ , and on the regularisation function  $r$ , entering in the computation of  $\pmb{g}$ . In the most general setting, we are required to solve two convex minimisation problems, consisting of the computation of the proximals  $h_{\pm}$  and  $\pmb{g}$  [57]; the convexity of the problem guarantees the uniqueness of the fixed point solution. Result 2.1 implies the following.
Result 2.2 Under the assumptions of Result 2.1, the training loss  $\epsilon_{\ell}$ , the training error  $\epsilon_{t}$ , and the test (or generalisation) error  $\epsilon_{g}$  are given, in the proportional asymptotic limit, by
$$
\frac {1}{n} \sum_ {\nu = 1} ^ {n} \ell \left(y ^ {\nu}, \frac {w ^ {\star \top} x ^ {\nu}}{\sqrt {d}} + b ^ {\star}\right)\rightarrow \mathbb {E} _ {\pm , \zeta , \Delta} [ \ell_ {\pm} (h _ {\pm}) ] =: \epsilon_ {\ell},
$$
$$
\frac {1}{n} \sum_ {\nu = 1} ^ {n} \mathbb {I} \left(\varphi \left(\frac {w ^ {\star \tau} x ^ {\nu}}{\sqrt {d}} + b ^ {\star}\right) \neq y ^ {\nu}\right)\rightarrow \mathbb {E} _ {\pm , \zeta , \Delta} \left[ \mathbb {I} \left(\varphi \left(h _ {\pm}\right) \neq \pm 1\right)\right] =: \epsilon_ {t}, \tag {8}
$$
$$
\mathbb {E} _ {(y, x)} \left[ \mathbb {I} \left(\varphi \left(\frac {w ^ {\star_ {\top}} x}{\sqrt {d}} + b ^ {\star}\right) \neq y\right) \right] = \mathbb {E} _ {\pm , \Delta , \zeta} [ \mathbb {I} (\varphi (\omega_ {\pm}) \neq \pm 1) ] =: \epsilon_ {g}.
$$
Result 2.3 Under the hypothesis that  $\mathbb{E}[\Delta] < +\infty$  and  $\mathbb{E}[\Delta^{-2}] < +\infty$ , the Bayes optimal test error for binary classification is
$$
\epsilon_ {\mathrm {g}} ^ {\mathrm {B O}} = \rho_ {+} \mathbb {E} \left[ \hat {\Phi} \left(\kappa_ {+}\right) \right] + \rho_ {-} \mathbb {E} \left[ \hat {\Phi} \left(\kappa_ {-}\right) \right], \quad \kappa_ {\pm} := \frac {1}{\sqrt {\Delta_ {0} ^ {\star} \left(1 + \frac {\mathbb {E} [ \Delta ] \mathbb {E} [ \Delta^ {- 2} ]}{\alpha \mathbb {E} [ \Delta^ {- 1} ] ^ {2}}\right)}} \left(1 \pm \frac {\Delta_ {0}}{2} \left(1 + \frac {1}{\alpha \mathbb {E} [ \Delta^ {- 1} ]}\right) \ln \frac {\rho_ {+}}{\rho_ {-}}\right) (9)
$$
where the expectation is over the two identically distributed random variables  $\Delta_0, \Delta_0^\star \sim \varrho$ , and  $\hat{\Phi}(x)$  is the complementary error function.
Quadratic loss with ridge regularisation. Explicit choices for  $\ell$  or  $r$  can highly simplify the fixed-point equations above. As an example, let us consider the case of quadratic loss  $\ell(y,x) = \frac{1}{2}(y - x)^2$  with ridge regularisation  $r(\boldsymbol{w}) = \frac{1}{2}\|\boldsymbol{w}\|^2$ . In this setting, an explicit formula for the proximal  $h_{\pm}$  in Eq. (6) and for the proximal  $\boldsymbol{g}$  in Eq. (5) can be easily found. As a result, we can obtain explicit expressions for the corresponding set of state evolution equations which become particularly suitable for a fast numerical solution and explicit the dependence on higher moments of  $\Delta$ :
$$
\left\{ \begin{array}{l} m _ {\pm} = \frac {\sum_ {k = \pm} \hat {m} _ {k} \mu_ {k} ^ {\top} \mu_ {\pm}}{\lambda + \hat {v}}, \\ q = \frac {\| \sum_ {k = \pm} \hat {m} _ {k} \mu_ {k} \| ^ {2} + \hat {q}}{(\lambda + \hat {v}) ^ {2}}, \\ v = \frac {1}{\lambda + \hat {v}}, \end{array} \right. \quad \left\{ \begin{array}{l} \hat {q} = \alpha \mathbb {E} _ {\pm} [ (\pm 1 - m _ {\pm} - b) ^ {2} ] \mathbb {E} _ {\Delta} \left[ \frac {\Delta}{(1 + v \Delta) ^ {2}} \right] + \alpha q \mathbb {E} _ {\Delta} \left[ \frac {\Delta^ {2}}{(1 + v \Delta) ^ {2}} \right], \\ \hat {v} = \alpha \mathbb {E} _ {\Delta} \left[ \frac {\Delta}{1 + v \Delta} \right], \\ \hat {m} _ {\pm} = \alpha \rho_ {\pm} (\pm 1 - m _ {\pm} - b) \mathbb {E} _ {\Delta} \left[ \frac {1}{1 + v \Delta} \right]. \end{array} \right. \tag {10}
$$
3 Application to synthetic datasets