Title: Classification of Heavy-tailed Features in High Dimensions: a Superstatistical Approach
Venue: NeurIPS 2023
Accepted: yes
Keywords: Classification

Gaussian Mixture Model

Superstatistics

Empirical Risk Minimization

Replica theory

Power-law distribution
Abstract: We characterise the learning of a mixture of two clouds of data points with generic centroids via empirical risk minimisation in the high dimensional regime, under the assumptions of generic convex loss and convex regularisation. Each cloud of data points is obtained via a double-stochastic process, where the sample is obtained from a Gaussian distribution whose variance is itself a random parameter sampled from a scalar distribution $\varrho$. As a result, our analysis covers a large family of data distributions, including the case of power-law-tailed distributions with no covariance, and allows us to test recent ''Gaussian universality'' claims. We study the generalisation performance of the obtained estimator, we analyse the role of regularisation, and we analytically characterise the separability transition.

Classification of Heavy-tailed Features in High Dimensions: a Superstatistical Approach

Abstract

We characterise the learning of a mixture of two clouds of data points with generic centroids via empirical risk minimisation in the high dimensional regime, under the assumptions of generic convex loss and convex regularisation. Each cloud of data points is obtained via a double-stochastic process, where the sample is drawn from a Gaussian distribution whose variance is itself a random parameter sampled from a scalar distribution /u1D71A . As a result, our analysis covers a large family of data distributions, including the case of power-law-tailed distributions with no covariance, and allows us to test recent 'Gaussian universality' claims. We study the generalisation performance of the obtained estimator, we analyse the role of regularisation, and we analytically characterise the separability transition.

1 Introduction

Generalised linear models (GLMs) are still ubiquitous in the theoretical research on machine learning, despite their simplicity. Their nontrivial phenomenology is often amenable to complete analytical treatment and has been key to understanding the unexpected behavior of large and complex architectures. Random features models [61, 47, 24], for example, allowed to clarify many aspects of the well-known double-descent phenomenon in neural networks [54, 8, 9]. A line of research spanning more than three decades [68, 74] has considered a variety of GLMs to investigate a number of aspects of learning in high dimensions. Yet, a crucial assumption adopted in many such theoretical models is that the covariates are obtained from a Gaussian distribution, or from a mixture of Gaussian distributions [42, 49, 3, 37, 43]. Although such a Gaussian design has served as a convenient working hypothesis (in some cases experimentally and theoretically justified [67, 44]), it is however not obvious how much it limits the scope of the results. It is reasonable to expect structure, heavy tails, and large fluctuations in real (often non-Gaussian) data [1, 62] to play an important role in the learning process, and it would be therefore desirable to include structured and heavy-tailed-distributed covariates in our theoretical toolbox.

This paper presents, to the best of our knowledge for the first time, the exact asymptotics for classification tasks on covariates obtained from a mixture of heavy-tailed distributions. We focus on supervised binary classification assuming that the sample size /u1D45B and the dimensionality /u1D451 of the space where the covariates live are both sent to infinity, keeping their ratio /u1D45B / /u1D451 = /u1D6FC fixed. The paper fits therefore in the line of works on exact high-dimensional asymptotics for classification learning via a GLM [49, 43], but, crucially, we relax the usual Gaussian data hypothesis by including in our

analysis power-law-tailed distributions with possibly no covariance. The mixture is obtained from two distributions, each centered around a centroid /u1D741 ∈ R /u1D451 and resulting from a double stochastic process. Namely, each sample point is obtained from a distribution N ( /u1D741 , /u1D6BA ) whose covariance /u1D6BA = Δ /u1D470 /u1D451 ∈ R /u1D451 × /u1D451 is itself a random variable so that Δ has density /u1D71A , supported on R + ∗ . Using, for example, an appropriately parametrised inverse-Gamma distribution for /u1D71A , such a superstatistical construction (as known in the physics literature [5, 7]) can provide, for example, a heavy-tailed, Cauchy-like data distribution with infinite covariance. The replica method [48], an analytical tool widely adopted in statistical physics, provides asymptotic formulas for a generic density /u1D71A , allowing us to study the effects of non-Gaussianity on the performance curves, and test Gaussian universality hypotheses [59] in such a classification task.

Motivation and related works Learning a rule to classify data points clustered in clouds in high dimension is a classical problem in statistics [32]. Its ubiquity is exemplified by the recently observed neural collapse phenomenon in deep neural networks, in which the last layer classifier was found to operate on features clustered in clouds around the vertices of a simplex [38, 56]; more recently, Seddik et al. [67] showed that Gaussian mixtures suitably describe the deep learning representation of GAN data. In theoretical models, data points are typically assumed to be organised in /u1D43E Gaussian clouds, each corresponding to a class. Each class /u1D458 , /u1D458 ∈ { 1 , . . . , /u1D43E } , is centered around a mean vector /u1D741 /u1D458 and has covariance /u1D6BA /u1D458 . The binary classification case, /u1D43E = 2 , is the simplest, and possibly most studied, setting. For this case, Mai and Liao [46] considered an ERM task with generic convex loss and ridge regularisation in the high-dimensional regime /u1D45B , /u1D451 → + ∞ with /u1D45B / /u1D451 ∈ ( 0 , + ∞ ) . In this setting, they gave a precise prediction of the classification error, showing the optimality of the square loss in the unregularised case. Their results have been extended by Mignacco et al. [49], who showed that the presence of a regularisation can actually drastically affect the performance, improving it. In the same setting, the Bayes optimal estimator has been studied in both the supervised and the semi-supervised setting [49, 41]; almost-Bayes-optimal solutions have been put in relation to wide flat landscape regions [3]. In the context of binary classification, the maximum number /u1D45B of samples that can be perfectly fitted by a linear model in dimension /u1D451 [49, 16, 37] has been the topic of investigation since the seminal work of Cover [12] and is related to the classical storage capacity problems on networks [20, 21, 39]. The corresponding separability transition value /u1D6FC = /u1D45B / /u1D451 is remarkably associated with the existence transition of the maximum likelihood estimator [70, 76]. The precise asymptotic characterisation of the test error in learning to classify /u1D43E ≥ 2 Gaussian clouds with generic means and covariances has been recently obtained by Loureiro et al. [43]. Within this line of research, rigorous results have been obtained by a variety of methods, such as Gordon's inequality technique [31, 71, 49] or mapping to approximate message passing schemes [4, 36, 10, 43, 27].

As previously mentioned, the working hypothesis of Gaussian design is widely adopted in highdimensional classification problems. On top of being a convenient technical hypothesis, this assumption has been justified in terms of a 'Gaussian universality' principle. In other words, in a number of circumstances, non-Gaussian features distributions are found to be effectively described by Gaussian ones with matching first and second moments as far as the asymptotic properties of the estimators are concerned [44]. Such a 'universality property' has been rigorously proven for example in compressed sensing [50] or in the case of LASSO with non-Gaussian dictionaries [55]. A Gaussian equivalence principle introduced by Goldt et al. [30] has been proven to hold for a wide class of generalised linear estimation problems [47]. Extending work by Hu and Lu [34], Montanari and Saeed [51] recently proved such a principle in a GLM under the assumption of pointwise normality of the distribution of the features. This crucial assumption can be intended, roughly speaking, as the assumption of sub-Gaussian decay of a marginal of the feature distribution in any direction (see also [25, 13] for further developments).

The Gaussian universality principle, however, can break down by relaxing some of the aforementioned assumptions on the feature distribution. Montanari and Saeed [51] for example showed that pointwise normality is a necessary hypothesis for universality. Studying regression tasks on an elliptically distributed dataset, El Karoui [18] showed that claims of 'universality' obtained in the Gaussian setting require serious scrutiny, as the statistics of the estimators might strongly depend on the nonGaussianity of the covariates [18, 69, 72]. More recently, Pesce et al. [59] showed that a structured Gaussian mixture cannot be effectively described by a single Gaussian cloud. Building upon a series of contributions related to the asymptotic performance of models in the proportional high-dimensional limit [33, 47, 28, 24, 30, 44], we aim precisely to explore the validity of Gaussian universality within classification problems.

In this paper, we work in a 'superstatistical' data setting [5, 7], meaning that we superpose the statistics of Gaussian data distribution with an assumed distribution of its variance. Such a construction is adopted in a number of disciplines and contexts to go beyond Gaussianity, albeit is known under different names. In statistical physics, it is known as 'superstatistics' and is employed in the analysis of non-equilibrium and non-linear systems [6]. In Bayesian modeling, it is common to refer to hierarchical priors and models [22, 23], while in probability, statistics, and actuarial sciences such distributions are known as compound probability distributions [63], or doubly-stochastic models [60, 66]. Crucially, this construction allows us to consider a very large family of non-Gaussian distributions, which include, but are not limited to, any power-law decay and Cauchy-like with possible infinite-covariance parametrisations.

Our contributions In this manuscript we provide the following results.

Westudy a classification task on a non-Gaussian mixture model (see Eq. (1) below) via a generalised linear model (GLM) and we analytically derive, using the replica method [14, 19, 48], an asymptotic characterisation of the statistics of the empirical risk minimisation (ERM) estimator. Our results go therefore beyond the usual Gaussian assumption for the dataset, and include for example the case of covariates obtained from a mixture of distributions with infinite variance. The analysis is performed in the high-dimensional, proportional limit and for any convex loss and convex regularisation. By using this result, we provide asymptotic formulas for the generalisation, training errors and training loss.

We analyse the performance of this ERM task on a specific family of dataset distributions by using different convex loss functions (quadratic and logistic) and ridge regularisation. We show in particular that, in the case of two balanced clusters with a specific non-Gaussian distribution, the optimal ridge regularisation strength /u1D706 ★ is finite, at odds with the Gaussian case, for which /u1D706 ★ →∞ [49]. In this respect, by considering distributions with matching first and second moments, we analytically show that the performances of the analysed GLM do depend in general on higher moments, and therefore the 'Gaussian universality principle' breaks down when heavy-tailed distributions are considered.

We derive the separability threshold on a large family of non-Gaussian dataset distributions, possibly with unbounded covariance, generalising the known asymptotics for the separability of Gaussian clouds [49]. The result of Cover [12] is recovered in the case of infinite distribution width.

Under some moment conditions, we derive the Bayes-optimal performance in the case of binary classifications with symmetric centroids, generalising the argument in Ref. [49].

We finally extend recent results on Gaussian universality in the Gaussian mixture model with random labels [25, 59] to the case of non-Gaussian distributions. We show that the universal formula for the square training loss at zero regularisation found in Ref. [25] holds in our more general setting as well.

2 Main result

Dataset construction We consider the task of classifying two data clusters in the /u1D451 -dimensional space R /u1D451 . The dataset D /colonequal {( /u1D499 /u1D708 , /u1D466 /u1D708 )} /u1D708 ∈ [ /u1D45B ] is obtained by extracting /u1D45B independent datapoints /u1D499 /u1D708 , each associated with a label /u1D466 /u1D708 ∈ { -1 , 1 } (a more general setting, involving a multiclass classification task, is discussed in Appendix A). The data points are correlated with the labels via a law /u1D443 ( /u1D499 , /u1D466 ) which we assume to have the form

Here /u1D443 ( /u1D499 | /u1D741 ) is a distribution with mean /u1D741 , and the mean vectors /u1D741 ± ∈ R /u1D451 are distributed according to some density, such that E [ ‖ /u1D741 ‖ 2 ] = Θ ( 1 ) , and correspond to the center of the two clusters. The scalar quantity /u1D70C weighs the relative contribution of the two clusters: in the following, we will denote /u1D70C + = /u1D70C = 1 -/u1D70C -. Each cluster distribution /u1D443 ( /u1D499 | /u1D741 ) around a vector /u1D741 is assumed to have the form

where N ( /u1D499 | /u1D741 , /u1D6BA ) is a Gaussian distribution with mean /u1D741 ∈ R /u1D451 and covariance /u1D6BA ∈ R /u1D451 × /u1D451 , and Δ is randomly distributed with some density /u1D71A with support on R + ∗ /colonequal ( 0 , + ∞ ) . The family of 'elliptic-like' distributions in Eq. (1b) has been extensively studied, for instance, by the physics community, in the context of superstatistics [5, 7]. Mixtures of normals in the form of Eq. 1b are a central tool in Bayesian statistics [65] due to their ability to approximate any distribution given a sufficient number

of components [53, 2, 29]. Although the family in Eq. (1b) is not the most general of such mixtures, it is sufficient to include a large family of power-law-tailed densities and to allow us to go beyond the usual Gaussian approximation. El Karoui [18] considered the statistical properties, in the same asymptotic proportional regime as here, of ridge-regularised regression estimators on datasets with distribution as in Eq. 1b, under the assumption (here relaxed) that E [ Δ 2 ] < + ∞ . This elliptic family includes a large class of distributions with properties markedly different from the ones of Gaussians. For example, the inverse-Gamma with density /u1D71A ( Δ ) = ( 2 /u1D70B Δ 3 ) -1 / 2 e -1 2 Δ leads to a /u1D451 -dimensional Cauchy-like distribution /u1D443 ( /u1D499 | /u1D741 ) ∝ ( 1 - ‖ /u1D741 -/u1D499 ‖ 2 ) -/u1D451 + 1 2 , having as marginals Cauchy distributions and E [ ‖ /u1D499 -/u1D741 ‖ 2 ] = + ∞ . We will perform our classification task by searching for a set of parameters ( /u1D498 ★ , /u1D44F ★ ) , called weights and bias , respectively, that will allow us to construct an estimator via a certain classifier /u1D711 : R → { -1 , 1 }

By means of the above law, we will predict the label for a new, unseen data point /u1D499 sampled from the same law /u1D443 ( /u1D499 , /u1D466 ) . Our analysis will be performed in the high-dimensional limit where both the sample size /u1D45B and dimensionality /u1D451 are sent to infinity, with /u1D45B / /u1D451 ≡ /u1D6FC kept constant.

Learning task In the most general setting, the parameters are estimated by minimising an empirical risk function in the form

Here ℓ is a strictly convex loss function with respect to its second argument, and /u1D45F is a strictly convex regularisation function with the parameter /u1D706 ≥ 0 tuning its strength.

State evolution equations. Let us now present our main result, namely the exact asymptotic characterisation of the distribution of the estimator /u1D498 ★ ∈ R /u1D451 and of 1 / √ /u1D451 /u1D498 ★ ᵀ /u1D47F ∈ R /u1D45B , where /u1D47F ∈ R /u1D451 × /u1D45B is the concatenation of the /u1D45B dataset column vectors /u1D499 /u1D708 ∈ R /u1D451 , /u1D708 ∈ [ /u1D45B ] . Such an asymptotic characterisation is performed via a set of order parameters satisfying a system of self-consistent 'stateevolution' equations, which we will solve numerically. The asymptotic expressions for generalisation and training errors, in particular, are written in terms of their solutions, the order parameters. In the following, we use the shorthand ℓ ±( /u1D462 ) ≡ ℓ (± 1 , /u1D462 ) , and given a function Φ : { -1 , 1 } → R , we write Φ ± ≡ Φ (± 1 ) and E ±[ Φ ±] /colonequal /u1D70C + Φ + + /u1D70C -Φ -.

Result 2.1 Let /u1D701 ∼ N ( 0 , 1 ) and Δ ∼ /u1D71A , both independent from other quantities. Let also be /u1D743 ∼ N ( 0 , /u1D470 /u1D451 ) . In the setting described above, given /u1D719 1 : R /u1D451 → R and /u1D719 2 : R /u1D45B → R , the estimator /u1D498 ★ and the vector /u1D49B ★ /colonequal 1 √ /u1D451 /u1D498 ★ /u1D47F ∈ R /u1D45B verify:

where we have introduced the proximal /u1D488 ∈ R /u1D451 , defined as and where we also have introduced the proximal for the loss /u1D489 ∈ R /u1D45B , obtained by concatenating /u1D70C + /u1D45B quantities ℎ + with /u1D70C -/u1D45B quantities ℎ -, with ℎ ± given by

The collection of parameters ( /u1D45E , /u1D45A ± , /u1D463 , ˆ /u1D45E , ˆ /u1D45A ± , ˆ /u1D463 , /u1D44F ) appearing in the equations above is given by the fixed-point solution of the following self-consistent equations:

The derivation of the result above is given, in a multiclass setting, in Appendix A using the replica method, which can be put on rigorous ground by a mapping of the risk minimisation problem into an approximate message-passing iteration scheme [17, 43, 26]. As in the purely Gaussian case, the state evolution equations naturally split the dependence on the loss function ℓ , which is relevant in the computation of ℎ ± , and on the regularisation function /u1D45F , entering in the computation of /u1D488 . In the most general setting, we are required to solve two convex minimisation problems, consisting of the computation of the proximals ℎ ± and /u1D488 [57]; the convexity of the problem guarantees the uniqueness of the fixed point solution. Result 2.1 implies the following.

Result 2.2 Under the assumptions of Result 2.1, the training loss /u1D716 ℓ , the training error /u1D716 /u1D461 , and the test (or generalisation) error /u1D716 /u1D454 are given, in the proportional asymptotic limit, by

Result 2.3 Under the hypothesis that E [ Δ ] < + ∞ and E [ Δ -2 ] < + ∞ , the Bayes optimal test error for binary classification is

where the expectation is over the two identically distributed random variables Δ 0 , Δ ★ 0 ∼ /u1D71A , and ˆ Φ ( /u1D465 ) is the complementary error function.

Quadratic loss with ridge regularisation. Explicit choices for ℓ or /u1D45F can highly simplify the fixedpoint equations above. As an example, let us consider the case of quadratic loss ℓ ( /u1D466 , /u1D465 ) = 1 2 ( /u1D466 -/u1D465 ) 2 with ridge regularisation /u1D45F ( /u1D498 ) = 1 2 ‖ /u1D498 ‖ 2 . In this setting, an explicit formula for the proximal ℎ ± in Eq. (6) and for the proximal /u1D488 in Eq. (5) can be easily found. As a result, we can obtain explicit expressions for the corresponding set of state evolution equations which become particularly suitable for a fast numerical solution and explicit the dependence on higher moments of Δ :

In this section, we compare our theoretical predictions with the results of numerical experiments for a large family of data distributions. The results have been obtained using ridge ℓ 2 -regularisation, and both quadratic and logistic losses, with various data cluster balances /u1D70C . We will also assume, without loss of generality, that /u1D741 ± = ± 1 √ /u1D451 /u1D741 , where /u1D741 ∼ N ( 0 , /u1D470 /u1D451 ) . The synthetic data sets will be produced using, an inverse-Gamma-distributed variance Δ , with density parametrised as

depending on the shape parameter /u1D44E > 0 and on the scale parameter /u1D450 > 0 . As a result, each class will be distributed as which produces a cloud centered in /u1D741 . The reason for our choice is that the distribution /u1D71A /u1D44E , /u1D450 allows us to easily explore different regimes by changing two parameters only. Indeed, the distribution has, for /u1D44E > 1 , covariance matrix /u1D6BA = E [( /u1D499 -/u1D741 ) ⊗ ( /u1D499 -/u1D741 ) ᵀ ] = /u1D70E 2 /u1D470 /u1D451 , where /u1D70E 2 = /u1D450 /u1D44E -1 . By fixing /u1D70E 2 and taking the limit /u1D44E → + ∞ , /u1D443 ( /u1D499 | /u1D741 ) → N ( /u1D499 | /u1D741 , /u1D70E 2 /u1D470 /u1D451 ) , i.e., the Gaussian case discussed by Mignacco et al. [49]. For /u1D44E ∈ ( 0 , 1 ] , instead, /u1D70E 2 = + ∞ . Note that the choice of inverse-Gamma distribution is not that unusual. It has been adopted, for example, to describe non-Gaussian data in quantitative finance [15, 40] or econometrics models [52]. Finally, we will construct our label estimator as in Eq. (2) with /u1D711 ( /u1D465 ) = sign ( /u1D465 ) .

3.1 Finite-covariance case

0.4

0

0.2

0.1

0.10

a→∞

α=10

t

0.05

a=4.5

α=2.5

a=1.5

0.00

0.2

0.1

0.0

2

+

OL

Let us start by considering a data distribution as in Eq. (11) with shape parameter /u1D44E = /u1D450 + 1 > 1 , which is therefore given by

and decays as ‖ /u1D499 ‖ -2 /u1D44E -1 in the radial direction for ‖ /u1D499 -/u1D741 ‖ /greatermuch 0 . As a consequence, the distribution has lim /u1D451 1 / /u1D451 E [ ‖ /u1D499 -/u1D741 ‖ /u1D458 ] = + ∞ for /u1D458 ≥ 2 /u1D44E . With this choice, all elements of the family in Eq. (12) have the same covariance /u1D6BA = E [( /u1D499 -/u1D741 ) ⊗ ( /u1D499 -/u1D741 ) ᵀ ] = /u1D470 /u1D451 as /u1D44E is varied , including the Gaussian limit /u1D443 ( /u1D499 | /u1D741 ) → N ( /u1D741 , /u1D470 /u1D451 ) obtained for /u1D44E → + ∞ .

In Fig. 1 we present the results of our numerical experiments using the square loss and small regularisation. An excellent agreement between the theoretical predictions and the results of numerical experiments is found for a range of values of /u1D44E and sample complexity /u1D6FC , both for balanced, i.e., equally sized, and unbalanced clusters of data (the plot for this case can be found in Appendix A.3). The test error /u1D716 /u1D454 presents the classical interpolation peak at /u1D6FC = 1 , smoothened by the presence of a non-zero regularisation strength /u1D706 , and the typical double-descent behavior [54]. As /u1D44E is increased, the results of Mignacco et al. [49] for Gaussian clouds with /u1D443 ( /u1D499 | /u1D741 ) = N ( /u1D499 | /u1D741 , /u1D470 /u1D451 ) are approached. The /u1D44E → + ∞ curves correspond, therefore, to the Gaussian mixture model we would have constructed by simply fitting the first and second moments of each class in the finite/u1D44E case. The plot shows that, at given population covariance /u1D6BA = /u1D470 /u1D451 , the classification of power-law distributed clouds is associated with different, and in particular smaller, errors than the corresponding task on Gaussian clouds with the same covariance: in this sense, no Gaussian universality principle holds. This (perhaps) counter-intuitive effect stems from the fact that, although for all classes /u1D6BA = /u1D470 /u1D451 , in our example the mean absolute deviation is smaller for finite /u1D44E

The learning process benefits from the fact that as /u1D44E decreases while keeping the variance fixed, points move on average closer to their means /u1D741 ± to compensate heavier tails. Finally, we observed that test errors are systematically smaller for unbalanced clusters than for balanced clusters (see Appendix A.3).

The same numerical experiment was repeated using the logistic loss for training. Once again, in Fig. 2, we focus on /u1D70C = 1 / 2 and show that the theoretical predictions of the generalisation and training errors agree with the results of numerical experiments for a range of sample complexity values /u1D6FC and various values of /u1D44E . Just as for the square loss, the test error /u1D716 /u1D454 of the Gaussian model ( /u1D44E → + ∞ ) is larger than the one observed for power-law distributed clouds at finite values of /u1D44E . In the /u1D706 → 0 limit, the typical interpolation cusp in the generalisation error is observed: the cusp occurs at different values of /u1D6FC as /u1D44E is changed; interpolation occurs at smaller values of /u1D6FC for larger /u1D44E (i.e., for 'more Gaussian' distributions). A comparison with the test error /u1D716 /u1D461 confirms that this cusp occurs at the value of /u1D6FC where the training error becomes non-zero and the two training data clouds become non-separable [70, 16]. This sharp transition in /u1D6FC also corresponds to the existence transition of maximum-likelihood estimator in high-dimensional logistic regression, analysed for Gaussian data in Refs. [70, 49] and, in our setting, in Section 4 below. For larger regularisation strength, the cusp smoothens, and the training error becomes non-zero at smaller values of /u1D6FC .

3.2 Infinite-covariance case

The family of distributions specified by Eq. (11) allows us to also consider the case of infinite covariance, i.e., /u1D70E 2 = + ∞ . To test our formulas in this setting, we considered the case in which each cloud is obtained by 'contaminating' a standard Gaussian with an infinite-covariance distribution as in Eq. (11b) with /u1D450 = 1 and /u1D44E < 1 [35]. In other words, we use the density /u1D71A ( Δ ) = /u1D45F /u1D71A /u1D44E , 1 ( Δ ) + ( 1 -/u1D45F ) /u1D6FF ( Δ -1 ) , with /u1D45F ∈ [ 0 , 1 ] interpolation parameter, for Δ in Eq. (1). Each class has therefore infinite covariance for 0 < /u1D45F ≤ 1 and the Gaussian case is recovered for /u1D45F = 0 . Fig. 3 collects our results for two balanced clouds, analysed using square loss and logistic loss. Good agreement between the theoretical predictions and the results of numerical experiments is found for a range of values of sample complexity and of the ratio /u1D45F . In this case, the finite-variance case /u1D45F = 0 corresponds, not surprisingly, to the lowest test error with both square loss and logistic loss, which grows with /u1D45F .

3.3 The role of regularisation in the classification of non-Gaussian clouds

0.4

E

9

0.2

8←

0.1

a=10

t

α=2.5

a=1.5

0.0

0.2

0.0

8

10

a

One of the main results in the work of Mignacco et al. [49] is related to the effect of regularisation on the performances in a classification task on a Gaussian mixture. They observed that, for all values of the sample complexity /u1D6FC , the optimal ridge classification performance on a pair of balanced clouds is obtained for an infinite regularisation strength /u1D706 , using both hinge loss and square loss. We tested the robustness of this result beyond the purely Gaussian setting. In Fig. 4 (left), we plot the test error obtained using square loss and different regularisation strengths /u1D706 on the dataset distribution as in Eq. (12) with /u1D44E = 2 . We observe that, both in the balanced case and in the unbalanced case, the optimal regularisation strength /u1D706 ★ is finite and, moreover, /u1D6FC -dependent. On the other hand, if we fix /u1D6FC and let /u1D44E grow towards + ∞ , the optimal /u1D706 ★ grows as well to recover the result of Mignacco et al. [49] of diverging optimal regularisation for balanced Gaussian clouds, as shown in Fig. 4 (center)

0.50

0.45

0.40

0.35

0.30

r=1

r=0.8

0.25

r=0.6

0.20

r=0.4

r=0.2

0.15

r=0

0.10

0

1

2

3

5

0.50

r=1

0.45

r=0.75

r=0.5

0.40

r=0.25

0.35

r=0

9

0.30

0.25

0.20

0.15

0

2

4

6

8

10

0.5

入=1e-05

0.275

p=0.5

入= 0.1

p=0.49

0.4

入=1

入=10

0.250

p=0.25

入=100

0.3

BO

10²

0.225

0.2

0.200

0.175

101

入=1e-05

0.150

0.35

入=0.1

入=1

0.30

入=10

0.125

BO

0.25

0.100

0.20

8←

Q=100

a=10

Q = 4.5

100

0.15

0.075

Q=50

a=2.5

α = 1.5

2

3

100

102

104

101

102

入

a

4 The separability threshold

By studying the training error /u1D716 /u1D461 with logistic loss at zero regularisation we can obtain information on the boundary between the regime in which the training data are perfectly separable and the regime in which they are not. In other words, we can extract the value of the so-called separability transition complexity /u1D6FC ★ [70, 49, 43]. Once again, a complete characterisation of this transition point is available in the Gaussian mixture case, where /u1D6FC ★ can be explicitly given as a function of /u1D70E 2 [49].

It is not trivial to extend this result to the general case of non-Gaussian mixtures. It is, however, possible to derive /u1D6FC ★ within the large family of 'superstatistical' models we are considering here under the assumption that Result 2.1 holds. In this case, in Appendix A.3 we prove the following.

Result 4.1 In the considered double-stochastic model, data are linearly separable for /u1D6FC < /u1D6FC ★ , where

Fig. 5 shows the values of /u1D6FC ★ for different choices of the shape parameters /u1D44E and /u1D450 in the case of two balanced clouds of datapoints distributed as in Eq. (11b) as predicted by Eq. (14). In Fig. 5 (left), we fixed /u1D70E 2 = /u1D450 /u1D44E -1 < + ∞ , and plotted the separability threshold /u1D6FC ★ for a range of values of the shape parameter /u1D44E > 1 . As /u1D44E grows, the known threshold value for the Gaussian mixture case is recovered [49]. The double limit /u1D44E →∞ and /u1D70E 2 →∞ , on the other hand, provides the Cover [12] transition value /u1D6FC ★ = 2 , as expected. In Fig. 5 (right), instead, we analyse the case of infinite-variance clusters, by fixing /u1D44E ∈ ( 0 , 1 ) and by varying the scale parameter /u1D450 in Eq. (11b), which controls the spread (width) of the distribution. The Cover transition is therefore correctly recovered as /u1D450 diverges for all values of /u1D44E .

103

103

α=1/2

102

102

2

101

2

101

100

100

92

=1

10-1

g²=5

10-1

² =50

10-2

10-5

10-4

10-3

10-2

10-1

100

101

10-2

10-5 10-4 10-3 10-2 10-1

100

101

102

a-1

C

5 Random labels in the teacher-student scenario and Gaussian universality

Our analysis can be extended to the case in which a point /u1D499 is labeled not according to its class but by a 'teacher', represented by a probabilistic law /u1D443 0 ( /u1D466 | /u1D70F ) and a vector /u1D73D 0 ∈ R /u1D451 so that each dataset element is independently generated with the joint distribution

Let us assume now that /u1D73D 0 is such that lim /u1D451 1 / √ /u1D451 /u1D73D ᵀ 0 /u1D741 ± = 0 , i.e., the teacher is 'uncorrelated' with the structure of the mixture. In Appendix A.5 we show that, if ℓ ( /u1D466 , /u1D702 ) = ℓ ( -/u1D466 , -/u1D702 ) and /u1D443 0 ( /u1D466 | /u1D70F ) = /u1D443 0 ( -/u1D466 | -/u1D70F ) , the task is equivalent to a classification problem on a single cloud centered in the origin, i.e., to the case /u1D741 ± = 0 , so that /u1D443 ( /u1D499 , /u1D466 ) = /u1D443 0 ( /u1D466 /barex /barex 1 / √ /u1D451 /u1D73D ᵀ 0 /u1D499 ) E [ N ( /u1D499 /barex /barex 0 , Δ /u1D470 /u1D451 ) ] . This mean-invariance, recently proven in Ref. [25] in the pure Gaussian setting, holds therefore in our non-Gaussian setting as well.

A special, relevant case of 'uncorrelated teacher' is the one of random labels , where /u1D443 0 ( /u1D466 | /u1D70F ) = 1 / 2 ( /u1D6FF /u1D466 , + 1 + /u1D6FF /u1D466 , -1 ) . This apparently vacuous model is relevant in the study of the storage capacity problem [21, 39, 11] and of the separability threshold [12], and it has also been an effective tool to better understand deep learning [45, 75]. In this case, the following universal result holds with respect to our setup.

Result 5.1 In the random label setting studied via ridge regression case with zero regularisation, the asymptotic training loss is independent of the distribution of Δ , and given by the universal formula

The proof of the previous result is given in Appendix A.5. Note that this result does not require /u1D70E 2 /colonequal E [ Δ ] to be finite, and it includes, as a special case, the Gaussian mixture model with random labels considered in Ref. [25, 59]. Therein, Eq. (16) has been obtained under the Gaussian design hypothesis. It turns out therefore that Eq. (16) holds for a much larger ensemble, which includes heavy-tailed distributions with possibly infinite covariance. On the other hand, universality does not hold, e.g., for the training error.

In Fig. 6 we exemplify these results by using ridge-regularised square loss. The numerical experiments correspond to a classification task on a dataset of two clouds centered in /u1D741 ± , /u1D741 + = -/u1D741 - ∼ N ( 0 , 1 / /u1D451 /u1D470 /u1D451 ) , obtained using the distribution

where /u1D443 ( /u1D499 | /u1D741 ±) is the distribution in Eq. (11b) depending on the shape parameters /u1D44E and /u1D450 . These results are compared with the theoretical prediction obtained for a single cloud, with the same parameters /u1D44E and /u1D450 , but centered in the origin, so that /u1D443 ( /u1D499 , /u1D466 ) = /u1D6FF /u1D466 , + 1 + /u1D6FF /u1D466 , -1 2 /u1D443 ( /u1D499 | /u1D741 = 0 ) : the agreement confirms the 'mean-invariance' property stated above. As anticipated, as we are considering random labels, the training loss /u1D716 ℓ follows Eq. (16) in all cases, whereas the test and training errors are not universal. Note that, in place of the test error (which is trivially /u1D716 /u1D454 = 1 / 2 ) we plot the mean square error on the pre-activations ˆ /u1D716 /u1D454 /colonequal E ( /u1D466 , /u1D499 ) [( /u1D466 -1 / √ /u1D451 /u1D499 ᵀ /u1D498 ★ ) 2 ] . Also, observe that this quantity is divergent in the case of infinite covariance, hence is absent from the plot for /u1D44E ∈ ( 0 , 1 ] .

6 Conclusions and perspectives

The framework introduced in this work has allowed us to perform an exact asymptotic analysis of the learning of a mixture of two data point clouds with possibly power-law tails. In our dataset, each sample point /u1D499 is labeled by /u1D466 ∈ { -1 , 1 } and obtained as /u1D499 = /u1D466 /u1D741 + √ Δ /u1D49B , where /u1D741 ∈ R /u1D451 , /u1D49B ∼ N ( 0 , /u1D470 /u1D451 ) and Δ ∼ /u1D71A . We have derived exact expressions for the test and training errors and training loss achieved by a GLM in the limit of large size /u1D45B of the dataset and large dimensionality /u1D451 under the assumption that /u1D6FC = /u1D45B / /u1D451 is kept fixed. We have shown that the performance crucially depends on higher-order moments of the data distribution, so that a Gaussian universality assumption is not feasible. Moreover, we have shown the effects of non-Gaussianity on the optimal regularisation strength and derived an exact formula for the critical /u1D6FC ★ at which the two clouds become linearly inseparable. Finally, we have considered the so-called random labels setting (in which sample label assignment is randomised) and shown that, in this case, the training square loss takes a universal asymptotic form independent of the distribution of Δ : this result further supports the crucial role of labels' randomness in some recent universality results [25].

In conclusion, we have provided an exactly solvable toy model exhibiting power-law tails and possibly unbounded covariance, highlighting the fact that a dataset deviation from Gaussianity in high dimensions cannot, in general, be neglected. The application of the introduced theoretical framework to real datasets is an interesting direction to explore in the future: the main difficulty, in this case, is the choice of the best distribution /u1D71A given the observed dataset, a problem that has a long tradition in the context of Bayesian estimation [2, 23] 1 .

References

R.J. Adler, R.E. Feldman, and M.S. Taqqu, editors. A Practical Guide to Heavy Tails: Statistical Techniques and Applications . Birkhauser Boston Inc., USA, 1998. ISBN 0817639519.

D. Alspach and H. Sorenson. Nonlinear Bayesian estimation using Gaussian sum approximations. IEEE Trans. Autom. Control , 17(4):439-448, 1972.

C. Baldassi, E.M. Malatesta, M. Negri, and R. Zecchina. Wide flat minima and optimal generalization in classifying high-dimensional gaussian mixtures. J. Stat. Mech.: Theory Exp. , 2020(12):124012, 2020.

M. Bayati and A. Montanari. The dynamics of message passing on dense graphs, with applications to compressed sensing. IEEE Trans. Inf. Theory , 57(2):764-785, 2011.

C. Beck. Superstatistics: Theory and applications. Contin. Mech. Thermodyn. , 16, 2003.

C. Beck. Recent developments in superstatistics. Braz. J. Phys. , 39:357-363, 2008.

C. Beck and E.G.D. Cohen. Superstatistics. Physica A , 322:267-275, 2003.

M. Belkin, D. Hsu, S. Ma, and S. Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proc. Natl. Acad. Sci. U.S.A. , 116(32):15849-15854, 2019.

M. Belkin, A. Rakhlin, and A.B. Tsybakov. Does data interpolation contradict statistical optimality? In K. Chaudhuri and M. Sugiyama, editors, Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics , volume 89 of Proc. Mach. Learn. Res. , pages 1611-1619. PMLR, 2019.

R. Berthier, A. Montanari, and P.-M. Nguyen. State evolution for approximate message passing with non-separable functions. Inf. Inference , 9(1):33-79, 2020.

N. Brunel, J.-P. Nadal, and G. Toulouse. Information capacity of a perceptron. JJ. Phys. A: Math. Gen. , 25(19):5017, 1992.

T.M. Cover. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE Trans. Comput. , EC-14(3):326-334, 1965.

Y. Dandi, L. Stephan, F. Krzakala, B. Loureiro, and L. Zdeborová'. Universality laws for gaussian mixtures in generalized linear models. arXiv:2302.08933 , 2023.

P. del Giudice, S. Franz, and M. Virasoro. Perceptron beyond the limit of capacity. J. Physique , 50:121-134, 1989.

D. Delpini and G. Bormetti. Minimal model of financial stylized facts. Phys. Rev. E , 83:041111, 2011.

A. Deng, Z.and Kammoun and C. Thrampoulidis. A model of double descent for highdimensional binary linear classification. Inf. Inference , 11, 2021.

D.L. Donoho, A. Maleki, and A. Montanari. Message-passing algorithms for compressed sensing. Proc. Natl. Acad. Sci. U.S.A. , 106(45):18914-18919, Nov 2009.

N. El Karoui. On the impact of predictor geometry on the performance on high-dimensional ridge-regularized generalized robust regression estimators. Probab. Theory Relat. Fields , 170, 2018.

S. Franz, D.J. Amit, and M.A. Virasoro. Prosopagnosia in high capacity neural networks storing uncorrelated classes. J. Physique , 51(5):387-408, 1990.

E Gardner. The space of interactions in neural network models. J. Phys. A: Math. Gen. , 21(1): 257, 1988.

E. Gardner and B. Derrida. Three unfinished works on the optimal storage capacity of networks. J. Phys. A: Math. Gen. , 22(12):1983, 1989.

A. Gelman and J.1/ Hill. Data Analysis Using Regression and Multilevel/Hierarchical Models . Analytical Methods for Social Research. Cambridge University Press, 2006.

A. Gelman, J.B. Carlin, H.S. Stern, D.B. Dunson, A. Vehtari, and D.B. Rubin. Bayesian Data Analysis, Third Edition . Chapman & Hall/CRC Texts in Statistical Science. Taylor & Francis, 2013.

F. Gerace, B. Loureiro, F. Krzakala, M. Mézard, and L. Zdeborová. Generalisation error in learning with random features and the hidden manifold model. In ICML , pages 3452-3462, 2020.

F. Gerace, F. Krzakala, B. Loureiro, L. Stephan, and L. Zdeborová. Gaussian universality of perceptrons with random labels. arXiv:2205.13303 , 2023.

C. Gerbelot and R. Berthier. Graph-based approximate message passing iterations. arXiv preprint arXiv:2109.11905 , 2021.

C. Gerbelot, A. Abbara, and F. Krzakala. Asymptotic errors for teacher-student convex generalized linear models (or: How to prove kabashima's replica formula). IEEE Trans. Inf. Theory , 69 (3):1824-1852, 2023.

B. Ghorbani, S. Mei, T. Misiakiewicz, and A. Montanari. Limitations of lazy training of two-layers neural network. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32, 2019.

J.K. Ghosh and R.V. Ramamoorthi. Bayesian Nonparametrics . Springer Series in Statistics. Springer New York, 2006.

S. Goldt, M. Mézard, F. Krzakala, and L. Zdeborová. Modeling the influence of data structure on learning in neural networks: The hidden manifold model. Phys. Rev. X , 10:041044, 2020.

Y. Gordon. Some inequalities for gaussian processes and applications. Isr. J. Math. , 50:265-289, 1985.

T. Hastie, R. Tibshirani, and J.H. Friedman. The elements of statistical learning: data mining, inference, and prediction , volume 2. Springer, 2009.

T. Hastie, A. Montanari, S. Rosset, and R.J. Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. Ann. Stat. , 50(2):949 - 986, 2022.

H. Hu and Y.M. Lu. Universality laws for high-dimensional learning with random features. arXiv:2009.07669 , 2022.

P.J. Huber. Robust estimation of a location parameter. Ann. Math. Stat. , 35(1):73-101, 1964.

A. Javanmard and A. Montanari. State evolution for general approximate message passing algorithms, with applications to spatial coupling. Inf. Inference , 2(2):115-144, 2013.

G.R. Kini and C. Thrampoulidis. Phase transitions for one-vs-one and one-vs-all linear separability in multiclass gaussian mixtures. In ICASSP 2021 , pages 4020-4024, 2021.

V. Kothapalli, E. Rasromani, and V. Awatramani. Neural collapse: A review on modelling principles and generalization. arXiv:2206.04041 , 2022.

W. Krauth and M. Mézard. Storage capacity of memory networks with binary couplings. J. Physique , 50(20):3057-3066, 1989.

N. Langrené, G. Lee, and Z. Zhu. Switching to non-affine stochastic volatility: A closed-form expansion for the inverse gamma model. Int. J. Theor. Appl. Finance , 19, 2015.

M. Lelarge and L. Miolane. Asymptotic bayes risk for gaussian mixture in a semi-supervised setting. In 2019 IEEE 8th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing , pages 639-643, 2019.

M. Lelarge and L. Miolane. Asymptotic bayes risk for gaussian mixture in a semi-supervised setting. In 2019 IEEE 8th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP) , pages 639-643, 2019.

B. Loureiro, G. Sicuro, C. Gerbelot, A. Pacco, F. Krzakala, and L. Zdeborová. Learning Gaussian Mixtures with Generalized Linear Models: Precise Asymptotics in High-dimensions. Advances in Neural Information Processing Systems , 34:10144-10157, 2021.

B. Loureiro, C. Gerbelot, H. Cui, S. Goldt, F. Krzakala, M. Mézard, and L. Zdeborová. Learning curves of generic features maps for realistic datasets with a teacher-student model. J. Stat. Mech.: Theory Exp. , 2022(11):114001, 2022.

H. Maennel, I.M. Alabdulmohsin, I.O. Tolstikhin, R. Baldock, O. Bousquet, S. Gelly, and D. Keysers. What do neural networks learn when trained with random labels? In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 19693-19704. Curran Associates, Inc., 2020.

X. Mai and Z. Liao. High dimensional classification via regularized and unregularized empirical risk minimization: Precise error and optimal loss. arXiv:1905.13742 , 2020.

S. Mei and A. Montanari. The Generalization Error of Random Features Regression: Precise Asymptotics and the Double Descent Curve. Commun. Pure Appl. Math. , 75, 2019.

M. Mézard, G. Parisi, and M.A. Virasoro. Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications , volume 9. World Scientific Publishing Company, 1987.

F. Mignacco, F. Krzakala, Y. Lu, P. Urbani, and L. Zdeborová. The role of regularization in classification of high-dimensional noisy Gaussian mixture. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proc. Mach. Learn. Res. , pages 6874-6883. PMLR, 2020.

A. Montanari and P.-M. Nguyen. Universality of the elastic net error. In 2017 IEEE ISIT , pages 2338-2342. IEEE, 2017.

A. Montanari and B.N. Saeed. Universality of empirical risk minimization. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of 35th Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 4310-4312. PMLR, 2022.

D.B. Nelson. Arch models as diffusion approximations. J. Econom. , 45(1-2):7-38, 1990.

V. Nestoridis, S. Schmutzhard, and V. Stefanopoulos. Universal series induced by approximate identities and some relevant applications. J. Approx. Theory , 163(12):1783-1797, 2011.

M. Opper, W. Kinzel, J. Kleinz, and R. Nehl. On the ability of the optimal perceptron to generalise. J. Phys. A: Math. Gen. , 23(11):L581, 1990.

A. Panahi and B. Hassibi. A universal analysis of large-scale regularized least squares solutions. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

V. Papyan, X.Y. Han, and D.L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proc. Natl. Acad. Sci. U.S.A. , 117(40):24652-24663, 2020.

N. Parikh and S. Boyd. Proximal algorithms. Found. Trends Optim. , 1(3):127-239, 2014.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. J. Mach. Learn. Res. , 12:2825-2830, 2011.

L. Pesce, F. Krzakala, B. Loureiro, and L. Stephan. Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimation. arXiv:2302.08923 , 2023.

M.A. Pinsky and S. Karlin. An Introduction to Stochastic Modeling: Fourth Edition , pages 1-563. Elsevier Inc, December 2010. ISBN 9780123814166.

A. Rahimi and B. Recht. Random features for large-scale kernel machines. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems , volume 20. Curran Associates, Inc., 2007.

W.J. Reed and B.D. Hughes. From gene families and genera to incomes and internet file sizes: Why power laws are so common in nature. Phys. Rev. E , 66:067103, 2002.

H.E. Robbins. Asymptotically subminimax solutions of compound statistical decision problems. 1985.

S. Rosset, J. Zhu, and T. Hastie. Margin maximizing loss functions. In S. Thrun, L. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing Systems , volume 16. MIT Press, 2003.

P.E. Rossi. Bayesian Non- and Semi-parametric Methods and Applications . Princeton University Press, 2014.

D. Schnoerr, R. Grima, and G. Sanguinetti. Cox process representation and inference for stochastic reaction-diffusion processes. Nat. Comm. , 7:11729, 05 2016.

M.E.A. Seddik, C. Louart, M. Tamaazousti, and R. Couillet. Random matrix theory proves that deep learning representations of GAN-data behave as Gaussian mixtures. In H.D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proc. Mach. Learn. Res. , pages 8573-8582. PMLR, 2020.

H. S. Seung, H. Sompolinsky, and N. Tishby. Statistical mechanics of learning from examples. Phys. Rev. A , 45:6056-6091, 1992.

L. Steinberger and H. Leeb. Conditional predictive inference for stable algorithms. Ann. Stat. , 51(1):290 - 311, 2018.

P. Sur and E.J. Candès. A modern maximum-likelihood theory for high-dimensional logistic regression. Proc. Natl. Acad. Sci. U.S.A. , 116(29):14516-14525, 2019.

C. Thrampoulidis, S. Oymak, and B. Hassibi. Regularized linear regression: A precise analysis of the estimation error. In Peter Grünwald, Elad Hazan, and Satyen Kale, editors, Proceedings of The 28th Conference on Learning Theory , volume 40 of Proc. Mach. Learn. Res. , pages 1683-1709, Paris, France, 2015. PMLR.

C. Thrampoulidis, E. Abbasi, and B. Hassibi. Precise error analysis of regularized /u1D45A -estimators in high dimensions. IEEE Trans. Inf. Theory , 64(8):5592-5628, 2018.

Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, ˙ Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods , 17:261-272, 2020. doi: 10.1038/s41592-019-0686-2.

T.L.H. Watkin, A. Rau, and M. Biehl. The statistical mechanics of learning a rule. Rev. Mod. Phys. , 65:499-556, 1993.

C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning (still) requires rethinking generalization. Commun. ACM , 64(3):107-115, 2021.

Q. Zhao, P. Sur, and E.J. Candès. The asymptotic distribution of the MLE in high-dimensional logistic models: Arbitrary covariance. Bernoulli , 28(3):1835 - 1861, 2022.

Peer reviews:
comments: SUMMARY:
The paper is concerned with binary classification, when the data comes from two point clouds that are superposition of Gaussian distribution. This model allows for data distribution with fat tails. The authors analyse the performance of empirical risk minimization in the high-dimensional regime where the number of training samples and the dimension jointly diverge. Using the replica method from statistical physics, they reduce the computation of e.g the training loss / generalization error to the resolution of self-consistent equations.
In the third section of the paper, the authors apply their main result on experiments with synthetic data.

STRENGTHS:
The paper is clearly written, and the mathematical derivations are easy to follow. The application of the replica method on a mixture of superposition of Gaussian distributions is new, to the best of my knowledge.

WEAKNESSES:
The paper lacks an experiment on real data to showcase situations in which the data model used in this paper (superposition of Gaussian) is more realistic / useful than simply using a mixture of two Gaussians.

QUESTIONS:
Could the computations be easily extended to  : 
1) other types of estimators, e.g. Bayesian estimators that sample from a posterior distribution instead of ERM 
2) more generic covariance for the Gaussian distributions. For instance instead of $N(\mu, \Delta I_d)$, use $N(\mu, \Delta \Sigma)$ where $\Sigma is a generic covariance matrix.

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the asymptotic behavior of Generalized Linear Models (GLM) when the number of training samples $n$, and the dimension of the feature-space $d$ both go to infinity, but the ratio $n/d$ is fixed to some known bounded value $\\alpha$. Moreover, authors assume the training data points are drawn from a mixture of two heavy-tailed distributions, which is different from the usual Gaussian assumption in most of the existing works (the mentioned heavy-tailed distributions are constructed by combining uncountably infinitely many Gaussian distributions).\n\nPaper claims to achieve a non-trivial asymptotic characterization of this problem setting, and also try to validate it via a number of experiments on synthetic data. Paper has a number of shortcomings, therefore my current vote is borderline reject. Presentation of the main results needs to be significantly improved, and also I would like to see the comments from other reviewers with more expertise in this particular field to assess the level of technical contribution in this work.\n\nSTRENGTHS:\n- Paper is well-written (at least in most parts), and the literature review part in the introduction section is very informative.\n- I have not completely checked the proofs, however, I have not noticed any mathematical mistakes. The technical validity of the theoretical part looks fine. I have not checked the experimental parts.\n\nWEAKNESSES:\n- All the theoretical derivations are based on asymptotics, while any result in the non-asymptotic case would be far more interesting.\n- I suggest presenting the results more formally, i.e., in the form of Theorems, Lemmas, and etc. Otherwise, the actual level of technical contribution in this work becomes hard to assess. Right now, there are no theorems inside the manuscript. Also, the explanations from L.130 to L.158 are vague (please see the questions section).\n- The main motivation behind this work is to assume non-Gaussian distributions (with a possibly infinite covariance) as the components of the mixture model which generates the data. I am concerned with how much this setting would look imporntant and/or interesting to the community. Due to the Gaussian universality principle, the analysis based on the Gaussian assumptions applies (more or less) to all \"Gaussian-like\" distributions (which covers almost all distributions with bounded moments) as well. Heavy-tailed distributions with power-law tails which do not have a bounded covariance are of course excluded from this list, but how important are they? IMO, authors have not given enough motivation regarding this issue.\n- The process which is used to generate the above-mentioned heavy-tailed distributions is very specific: superposition of an uncountably many Gaussians, or equivalently assuming that the covariance matrix of the Gaussian itself is a R.V. with an inverse-Gamma distribution. Authors have not discussed the limitation of this process. How general is it? does it include almost all heavy-tailed distributions?\n- I have not completely checked the proofs in supp. However, the mathematical tools used for deriving the results are not sophisticated. Not using sophisticated math or not relying on existing elegant theorems is fine, as long as an important problem has been solved or an interesting discovery has been made. This again takes us to a previous comment, on the importance level of this problem setting. I am not familiar with this particular line of research, so I have to wait for other reviewers to comment on that.\n\nQUESTIONS:\nL.130 to L.158: Results are not properly presented. I suggest using a formal theorem and a set of lemmas.\n\nL.139, Eq (4): What are $\\boldsymbol{g}$ and $\\boldsymbol{h}$?\n\nL.140, Eq (5): What are $h_{\\pm}$, $\\omega_{\\pm}$, $q$ and ... Actually this list can go on.\n\nThe main theoretical contributions are presented in Eq (8) and Eq (9). However, the vague explanation preceding them, would impose a huge negative impact on the potential reader.\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is focused on the non-Gaussian mixture model and asymptotical investigation of the asymptotic characterization of the statistics of the empirical risk minimization estimator. The paper takes under consideration the models with two clusters and applies their analysis to the convex loss functions and regularizers. The empirical evaluations investigate the theoretical aspects in practice.\n\nSTRENGTHS:\n- The theoretical analysis of generalized linear models is an exciting direction, especially in the contents of high-dimensional data. \n- The empirical evaluation of synthetic use cases seems to confirm the theoretical investigations.\n\nWEAKNESSES:\n- The paper is chaotic and very difficult to follow. It makes the work very difficult to understand. The listed contusions are not defined to the point. They are focused on studying and analyzing, not on practical outcomes. The work is not even summarized (taking into account that there is still some space in the manuscript, it seems to be strange) in the conclusions section, and limitations are not discussed well. Some explanations of crucial symbols are missing in the paper, and the motivations behind some steps are not explained well.\n\n- The proposed theoretical investigation is limited to the models with two classes. How can the results scale to multiclass scenarios?\n\n- The empirical evaluation is limited only to artificial cases. The problem investigated by the authors is very practical, and it is crucial to provide some empirical evaluation using real datasets. \n\n- There are many ways to go beyond the Gaussian distribution. Normalizing flows may be used to model the distributions for each of the considered clusters as an alternative to this approach. It would be beneficial to discuss this issue in the paper and even provide some empirical comparison to the approach.\n\nQUESTIONS:\nPlease refer to the remarks from the Weaknesses section.\n\nLIMITATIONS:\nThe paper is not organized properly, it is problematic to identify contributions and a strong plot in the work. Empirical evaluation on real cases is missing.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper derive a theory for training and generalization error when classifying a large number of points from a non-Gaussian high-dimensional data distribution. The data model is a double-stochastic process where a parameter is sampled from a scalar distribution and then a sample is taken from a Gaussian distribution with this parameter as variance. A self-consistent mean-field theory is provided for the case where the number of points is large and proportional to the dimensionality and the equations can be numerically solved for logistic and square losses. The theory is applied to data-sets with finite and infinite covariance, to study the role of regularization, and to estimate the separability threshold for such data. This highlighting both cases of “Gaussian universality” where the results coincide with previous “Gaussian” literature and deviation from such universality.\n\nSTRENGTHS:\n*\tOriginality: the tasks and methods are not new, and previous contributions are very well introduced. The originality of the work lies in the successful calculation of the theory for the non-Gaussian case, which is a valuable contribution. Furthermore, the work provides a basis for analyzing when an extrapolation beyond the Gaussian case is justified.\n *\tSignificance: the paper is important in highlighting where non-Gaussian data may diverge from the Gaussian case discussed in the literature, and as such it opens a venue for future work to use non-Gaussian analysis of real-worlds data, which is an important direction. The conclusions about test error in Gaussian vs non-Gaussian cases is non-trivial (the inversion between figure 1+2 and 3) and as is the finding or optimal finite regularization value for the non-Gaussian case. \n *\tClarity: the paper is in general well-written and can be served as exemplar for providing complicated theoretical results without sacrificing the clarity of the ideas.\n *\tQuality: the paper seems technically very sound, with impressive combination of theory and simulations.\n\nWEAKNESSES:\n*\tClarity: some of the notations and ideas presented are only hastily introduced, with two prominent examples being “Superstatistical Features” (from the title) and “uncountable superposition” (from the abstract, which seem overly complicated. To me, a presentation through “double stochastic” process (as in my summary) is straight-forward and require no extra jargon.\nAnother avenue for improving the understanding of the reader may lie in “Quadratic loss with ridge regularisation” where results are more amendable to interpretation. The authors should have provided more intuition for those results and furthermore point out where does non-Gaussian enters in the self-consistent equations (i.e., what part is shared with the Gaussian case).\n *\tOriginality: the main part of the work focus on reproducing known results from Gaussian literature and exploring the deviation from them for the non-Gaussian case. In that sense, there is no originality in this work beyond the (impressive) achievement of providing a theory which describe this non-Gaussian case.\n *\tSignificance: the work would have been more influential if it provided new tools which can be applied to datasets, where a small number of shape parameters is fit to non-Gaussian data and the ability to classify this data can be predicted from theory and then compared to actual classification of the data.\nCases where the Gaussian case predicts the behavior for the non-Gaussian case might deserve a fuller theoretical analysis, perhaps through the analysis suggested for “Quadratic loss with ridge regularisation” above.\n\nQUESTIONS:\n* Why bother with the classifier estimator phi? Is there any reasonable choice beyond sign?\n * Why do you refer to z* as “the matrix”?\n * Are all the 8 (or 10) order parameters scalars?\n * Can you clarify the interpretation of the proximal h and g? Do their distribution is a mean-field version of some real-worlds quantity?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations of their work. Those include the diagonal structure of the covariance matrix (conditional on the value of delta), the use of K=2 which leads to lack of discussion about the mean of the distribution (because they do not affect anything for K=2 beyond their norm), and the resulting theory solvable only numerically.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-19"}