Title: Universality laws for Gaussian mixtures in generalized linear models
Venue: NeurIPS 2023
Accepted: yes
Keywords: theoretical analysis

high-dimensional statistics

Universality

weak convergence

mixture models

sampling

statistical physics
Abstract: A recent line of work in high-dimensional statistics working under the Gaussian mixture hypothesis has led to a number of results in the context of empirical risk minimization, Bayesian uncertainty quantification, separation of kernel methods and neural networks, ensembling and fluctuation of random features. We provide rigorous proofs for the applicability of these results to a general class of datasets $(\mathbf{x_i},y_i, {i=1,\dots,n})$ containing independent samples from a mixture distribution $\sum_{c\in\mathcal{C}} \rho_{c}P_{c}^{\mathbf{x}}$. Specifically, we consider the hypothesis class of generalized linear models $\hat{y} = F(\mathbf{\Theta}^{\top}\mathbf{x})$ and investigate the asymptotic joint statistics of a family of generalized linear estimators $(\mathbf{\Theta}^{(1)}, \dots, \mathbf{\Theta}^{(M)})$, obtained either from (a) minimizing an empirical risk $\hat{R_n}^{(m)}(\mathbf{\Theta}^{(m)};\mathbf{X},\mathbf{y})$ or (b) sampling from the associated Gibbs measure $\exp(-\beta n \hat{R_n}^{(m)}(\mathbf{\Theta}^{(m)};\mathbf{X},\mathbf{y}))$. Our main contribution is to characterize under which conditions the asymptotic joint statistics of this family depends (on a weak sense) only on the means and covariances of the class conditional features distribution $P_{c}^{\mathbf{x}}$. This allows us to prove the universality of different quantities of interest, including training, generalization errors, as well as the geometrical properties and correlations of the estimators.

Universality laws for Gaussian mixtures in generalized linear models

Abstract

A recent line of work in high-dimensional statistics working under the Gaussian mixture hypothesis has led to a number of results in the context of empirical risk minimization, Bayesian uncertainty quantification, separation of kernel methods and neural networks, ensembling and fluctuation of random features. We provide rigorous proofs for the applicability of these results to a general class of datasets ( x i , y i ) i =1 ,...,n containing independent samples from mixture distribution ∑ c ∈C ρ c P x c . Specifically, we consider the hypothesis class of generalized linear models ˆ y = F ( Θ ⊤ x ) and investigate the asymptotic joint statistics of a family of generalized linear estimators ( Θ (1) , . . . , Θ ( M ) ) , obtained either from (a) minimizing an empirical risk ˆ R ( m ) n ( Θ ( m ) ; X , y ) or (b) sampling from the associated Gibbs measure exp( -βn ˆ R ( m ) n ( Θ ( m ) ; X , y )) . Our main contribution is to characterize under which conditions the asymptotic joint statistics of this family depend (on a weak sense) only on the means and covariances of the class conditional feature distribution P x c . This allows us to prove the universality of different quantities of interest, including training and generalization errors, as well as the geometrical properties and correlations of the estimators.

A recurrent topic in high-dimensional statistics is the investigation of the typical properties of signal processing and machine learning methods on synthetic, i.i.d. Gaussian data, a scenario often known under the umbrella of Gaussian design [1, 2, 3, 4, 5]. A less restrictive assumption arises when considering that many machine learning tasks deal with data partitioned into a fixed number of classes. In these cases, the data distribution is naturally described by a mixture model , where each sample is generated conditionally on the class. In other words: data is generated by first sampling the class assignment and then generating the input conditioned on the class. Arguably the simplest example of such distributions is that of a Gaussian mixture , which shall be our focus in this work.

Gaussian mixtures are a popular model in high-dimensional statistics since, besides being an universal approximator, they often lead to mathematically tractable problems. Indeed, a recent line of work has analyzed the asymptotic performance of a large class of machine learning problems in the proportional high-dimensional limit under the Gaussian mixture data assumption, see e.g. [6, 7, 8, 9, 10, 11, 12]. The goal of the present work is to show that this assumption, and the conclusions derived therein, are far more general than previously anticipated.

A recent line of work [13, 14], initiated by the work of [15] for Kernel matrices, posits that for generalized linear estimation on non-linear feature maps satisfying certain regularity conditions and a 'one-dimensional CLT', the data distribution can be replaced by equivalent Gaussian data without affecting the training and generalization errors. This was recently proven by [16] for empirical

SPOC, EPFL, Switzerland

risk minimization under the setup of strongly convex objectives, and extended to a larger class of objectives by [17].

However, there is strong empirical evidence that Gaussian universality holds in a more general sense [18]. First, existing results rely on the assumption of a target function depending on linear projections in the latent or feature space. This excludes the rich class of classification on mixture distributions, where the target function is given by the label. For such distributions, a more appropriate equivalent distribution is given by a mixture of Gaussians. Such a 'Gaussian mixture equivalence" has been conjectured and used in existing works, such as [11, 12] and was found to closely approximately classification on real datasets.

Furthermore, equivalence with mixtures of Gaussians has been observed to hold not only for training, generalization errors but other quantities of the estimators such as overlaps, variance, etc. For instance, [19, 20] empirically observed that the equivalence holds even while considering the joint distribution of multiple uncertainty estimators or ensembles of multiple random feature mappings. This suggests the equivalence of the distributions of the minimizers themselves.

Our results fill these gaps and provide rigorous justification for the universality in all the aforementioned works. Namely, we show that the joint statistics of multiple generalized estimators obtained either from ERM or sampling on a mixture model asymptotically agrees (in a weak sense) with the statistics of estimators from the same class trained on a Gaussian mixture model with matching first and second order moments. Our main contributions are as follows:

· Through a generalization of recent developments in the Gaussian equivalence principle [14, 17, 16], we prove the universality of empirical risk minimization and sampling for a generic mixture distribution and an equivalent mixture of Gaussians. In particular, we show that a Gaussian mixture observed through a random feature map is also a Gaussian mixture in the high-dimensional limit, a fact used for instance (without rigorous justification) in [11, 12, 21, 22].

· A consequence of our results is that, with conditions on the matrix weights, data generated by conditional Generative Adversarial Networks (cGAN) behave as a Gaussian mixture when observed through the prism of generalized linear models (kernels, feature maps, etc...), as illustrated in Figs 1 and 2. This further generalizes the work of [23] that only considered the universality of Gram matrices for GAN generated data through the prism of random matrix theory.

· We construct a unified framework involving multiple sets of parameters arising from simultaneous minimization of different objectives as well as sampling from Gibbs distributions defined by the empirical risk. Through the design of suitable reductions and a convexity-based argument, we establish conditions for the asymptotic universality of arbitrary functions of the set of minimizers or samples from different Gibbs distributions (Theorem 4). For instance, it includes ensembling [20]), and the uncertainty quantification and Bayesian setting assumed (without proof) in [19, 24].

· Finally, we show that for multi-class classification, the conditions leading to universality hold for a large class of functions of the minimizers, such as overlaps and sparsity measures, leading to the equivalence between their distributions of themselves, and provide a theorem for their weak convergence (Theorem 5).

· As a technical contribution of independent interest, our proof of Theorem 5 demonstrates a principled approach for leveraging existing results on the exact asymptotics for simple data distributions (such as for Gaussian mixture models in [12]) to prove the weak convergence and universality of the joint-empirical measure of the estimators and parameters (means, covariances) of the data-distribution.

Related works Universality is an important topic in applied mathematics, as it motivates the scope of tractable mathematical models. It has been extensively studied in the context of random matrix theory [25, 26], signal processing problems [1, 2, 3, 27, 28, 29, 30, 31] and kernel methods [15, 32, 33]. Closer to us is the recent stream of works that investigated the Gaussian universality of the asymptotic error of generalized linear models trained on non-linear features, starting from singlelayer random feature maps [34, 35, 16, 36, 37] and later extended to single-layer NTK [17] and deep random features [38]. These results, together with numerical observations that Gaussian universality holds for more general classes of features, led to the formulation of different Gaussian equivalence conjectures [13, 14, 18]. Our results build on the line of works on the proofs of these conjectures, through the use of the one-dimensional CLT (Central Limit Theorem), stated formally in [14] who proved it for random features of Gaussian data. We generalize this principle to a Gaussian equivalence conditioned on the cluster assignment in a mixture-model with a corresponding conditional 1d-CLT (Assumption 10).

A complementary line of research has investigated cases in which the distribution of the features is multi-modal, suggesting a Gaussian mixture universality class instead [39, 23, 40]. A bridge between these two lines of work has been recently investigated with random labels and teachers in [21, 22]. Our results provide rigorous extensions of Gaussian universality to the setups of mixture models as well as uncertainty quantificatio and ensembling.

1 Setting and models

Consider a supervised learning problem where the training data ( x i , y i ) ∈ R p × Y , i ∈ [ n ] := { 1 , · · · , n } is drawn i.i.d. from a mixture distribution:

with c i a categorical random variable denoting the cluster assignment for the i th example x i . Let µ c , Σ c denote the mean and covariance of P x c , and k = |C| . Further, assume that the labels y i are generated from the following target function:

where η : R 3 → R is a general label generating function, Θ ⋆ ∈ R k × p and ε i is an i.i.d source of randomness. It is important to stress that the class labels (2) are themselves not constrained to arise from a simple function of the inputs x i . For instance, the functional form in (2) includes the case where the labels are exclusively given by a function of the mixture index y i = η ( c i ) . This will allow us to handle complex targets, such as data generated using conditional Generative Adversarial Networks (cGANs).

In this manuscript, we will be interested in hypothesis classes defined by parametric predictors of the form y Θ ( x ) = F ( Θ ⊤ x ) , where Θ ∈ R k × p are the parameters and F : R k →Y a possibly nonlinear function. For a given loss function ℓ : R k ×Y → R + and regularization term r : R k × p → R + , define the (regularized) empirical risk over the training data:

where we have defined the feature matrix X ∈ R p × n by stacking the features x i column-wise and the labels y i in a vector y ∈ Y n . In what follows, we will be interested in the following two tasks:

(i) Minimization: in a minimization task, the statistician's goal is to find a good predictor by minimizing the empirical risk (3), possibly over a constraint set S p :

This encompasses diverse settings such as generalized linear models with noise, two-layer networks with a constant number of neurons and fixed second layer, mixture classification, but also the random label setting (with η ( Θ ⊤ ⋆ x i , ε i , c i ) = ε i ). In the following, we denote ̂ R ⋆ n ( X , y ) := min Θ ̂ R n ( Θ ; X , y )

(ii) Sampling: here, instead of minimizing the empirical risk (3), the statistician's goal is to sample from a Gibbs distribution that weights different hypothesis according to their empirical error:

where µ is reference prior measure and β > 0 is a parameter known as the inverse temperature . Note that minimization can be seen as a particular example of sampling when β →∞ , since in this limit the above measure peaks on the global minima of (4).

Applications of interestSo far, the setting defined above is quite generic, and the motivation to study this problem might not appear evident to the reader. Therefore, we briefly discuss a few scenarios of interest which are covered by this model.

Latentspace

Generativemodel

Data

Featuremap

Linearmodel

88

8888

ZERd~GMM

XERP~GMM

EffectiveGaussian

Data E RD

MixtureModel

(i) Conditional GANs (cGANs): These were introduced by [41] as a generative model to learn mixture distributions. Once trained in samples from the target distribution, they define a function Ψ that maps Gaussian mixtures (defining the latent space) to samples from the target mixture that preserve the label structure. In other words, conditioned on the label:

The connection to model (1) is immediate. This scenario was extensively studied by [39, 23, 40], and is illustrated in Fig. 1. In Fig. 2 we report on a concrete experiment with a cGAN trained on the fashion-MNIST dataset.

(ii) Multiple objectives: Our framework also allows to characterize the joint statistics of estimators ( Θ 1 , . . . , Θ M ) obtained from empirical risk minimization and/or sampling from different objective functions ˆ R m n defined on the same training data ( X , y ) . This can be of interest in different scenarios. For instance, [19, 24] has characterized the correlation in the calibration of different uncertainty measures of interest, e.g. last-layer scores and Bayesian training of last-layer weights. This crucially depends on the correlation matrix ˆ Θ erm Θ ⊤ Bayes ∈ R k × k which fits our framework.

(iii) Ensemble of features: Another example covered by the multi-objective framework above is that of ensembling. Let ( z i , y i ) ∈ R d ×Y denote some training data from a mixture model akin to (1). A popular ensembling scheme often employed in the context of deep learning [42] is to take a family of M feature maps z i ↦→ x ( m ) i = φ m ( z i ) (e.g. neural network features trained from different random initialization) and train M independent learners:

Prediction on a new sample z is then made by ensembling the independent learners, e.g. by taking their average ˆ y = 1 / M ∑ M m =1 ˆ Θ ( m ) ⊤ erm φ m ( z ) . A closely related model was studied in [43, 44, 20].

Note that in all the applications above, having the labels depending on the features X would not be natural, since they are either generated from a latent space, as in ( i ) , or chosen by the statistician, as in ( ii ) , ( iii ) . Indeed, in these cases the most natural label model is given by the mixture index y = c itself, which is a particular case of (2). This highlights the flexibility of our target model with respect to prior work [17]. Instead, [16] assumes that the target is a function of a latent variable , which would correspond to a mismatched setting. The discussion here can be generalized also to this case, but require an additional assumption discussed in Appendix B.

Universality Given these tasks, the goal of the statistician is to characterize different statistical properties of these predictors. These can be, for instance, point performance metrics such as the empirical and population risks, or uncertainty metrics such as the calibration of the predictor or moments of the posterior distribution (5). These examples, as well as many different other quantities of interest, are functions of the joint statistics of the pre-activations ( Θ ⊤ ⋆ x , Θ ⊤ x ) , for x either a test or

0.00

0.02

0.04

0.06

0.08

0.10

test misclassification error

cGAN

GMM

0.0

0.5

1.0

1.5

2.0

2.5

3.0

sample complexity = n/d

0.00

0.01

0.02

0.03

0.04

0.05

train loss

0.0

0.1

0.2

0.3

0.4

0.5

test mse

cGAN

GMM

0.0

0.5

1.0

1.5

2.0

2.5

3.0

sample complexity = n/d

0.00

0.02

0.04

0.06

0.08

training sample from (1). For instance, in a Gaussian mixture model, where x ∼ ∑ c ∈C ρ c N ( µ c , Σ c ) , the sufficient statistics are simply given by the first two moments of these pre-activations. However, for a general mixture model (1), the sufficient statistics will generically depend on all moments of these pre-activations. Surprisingly, our key result in this work is to show that in the high-dimensional limit this is not the case. In other words, under some conditions which are made precise in Section 2, we show that expectations with respect to (1) can be exchanged by expectations over a Gaussian mixture with matching moments. This can be formalized as follows. Define an equivalent Gaussian data set ( g i , y i ) n i =1 ∈ R p ×Y with samples drawn i.i.d. from the equivalent Gaussian mixture model :

We recall that µ c , Σ c denotes the mean and covariance of P x c from (1). Consider a family of estimators ( Θ 1 , · · · , Θ M ) defined by minimization (3) and/or sampling (5) over the training data ( X , y ) from the mixture model (1). Let h be a statistical metric of interest. Then, in the proportional high-dimensional limit where n, p →∞ at a fixed α = n / d > 0 , and where ⟨·⟩ denote the expectation with respect to the Gibbs distribution (5), we define universality as:

The goal of the next section is to make this statement precise.

2 Main results

We now present the main theoretical contributions of the present work and discuss its consequences. Our proofs for Theorems 4 and 6 build upon existing results on the universality of empirical risk minimization for uni-model distributions [16, 17] and therefore rely on similar technical regularity and concentration assumptions. Concretely, our work relies on the following assumptions:

Assumption 1 (Loss and regularization) . The loss function ℓ : R k +1 → R is nonnegative and Lipschitz, and the regularization function r : R p × k → R is locally Lipschitz, with constants independent from p .

Assumption 2 (Boundedness and concentration) . The constraint set S p is a compact subset of R k × p . Further, there exists a constant M > 0 such that for any c ≥ 0 ,

where ∥·∥ ψ 2 is the sub-gaussian norm, and K p ⊆ R p is such that S p ⊆ K k p .

Assumption 3 (Labels) . The labeling function η is Lipschitz, the teacher vector Θ belongs to S p , and the noise variables ε i are i.i.d sub-gaussian with ∥ ε i ∥ ψ 2 ≤ M for some constant M > 0 .

Those three assumptions are fairly technical, and it is possible that the universality properties proven in this article hold irrespective of these conditions. The crucial assumption in our theorems is that of a conditional one-dimensional CLT :

Assumption 4. For any Lipschitz function φ : R → R ,

where x and g denote samples from the given mixture distribution and the equivalent gaussian mixture distribution in equations (1) and (8) respectively.

The above assumption is a generalization of the 'one-dimensional CLT" underlying the line of work based on the Gaussian equivalence (GE) Principle [13, 14, 17, 16]. The above assumption splits the proof of universality for a general mixture distribution into two parts. First, one shows that asymptotic universality of an observable h can be reduced to the proof of a one-dimensional CLT. Second, one proves this CLT holds for the particular class of features of interest. This proof scheme streamlines universality proofs. Our work provides a general proof of the first step in Theorem 4, conditioned on the second, later showing that Assumption 4 holds for some natural feature maps of interest, i.e. random features applied to a Gaussian mixture (Theorem 6). However, Assumption 4 is expected to hold for a large class of features, as supported by our empirical observations in Figure 2 and arguments in Appendix C.

2.1 Universality of Mixture Models

We start by proving the universality of the free energy for a Gibbs distribution defined through the objective ̂ R n ( Θ ; X , y ) for the data distribution (1) and its equivalent Gaussian mixture (8).

Theorem 1 (Universality of Free Energy) . Let µ p ( Θ ) be a sequence of Borel probability measures with compact supports S p . Define the following free energy function:

Under Assumptions 1-4 on X and S p , for any bounded differentiable function Φ with bounded Lipschitz derivative, we have:

When µ p corresponds to discrete measures supported on an ϵ -net in S p , using the reduction from Lemma 1 to Theorem 1 in [17], we obtain the following corollary:

Corollary 2 (Universality of Training Error) . For any bounded Lipschitz function Φ : R → R :

In particular, for any E ∈ R , and denoting P -→ the convergence in probability:

The full theorem, as well as its proof, is presented in Appendix A, along with additional remarks and an intuitive sketch. The proof combines the conditional 1d-CLT in Assumption 4 with the

interpolation of the free-energy in [17]. For strongly-convex losses, one may alternatively use the Lindeberg's method as in [16].

In a nutshell, this theorem shows that the multi-modal data generated by any generative neural network is equivalent to a finite mixture of Gaussian in high-dimensions: in other words, a finite mixture of Gaussians leads to the same loss as for data generated by (for instance) a cGAN. Since the function ℓ : R k +1 → R need not be convex, we can take

where Ψ is an already pretrained neural network. In particular, if x is the output of all but the last layer of a neural net, we can view Ψ as the averaging procedure for a small committee machine.

Note that Corollary 2 depends crucially on Assumption 4 (the one-dimensional CLT), which is by no means evident. We discuss the conditions on the weights matrix for which it can be proven in Section 2.4. However, one can observe empirically that the validity of Corollary 2 goes well beyond what can be currently proven. A number of numerical illustrations of this property can be found in the work of [23, 39, 40], who already derived similar (albeit more limited) results using random matrix theory. Additionally, we observed that even with trained GANs, when we observed data through a random feature map [46], the Gaussian mixture universality is well obeyed. This scenario is illustrated in Fig. 1, with a concrete example in Fig. 2. Even though we did not prove the one-dimensional CLT for arbitrary learned matrices, and worked with finite moderate sizes, the realistic data generated by our cGAN behaves extremely closely to those generated by the corresponding Gaussian mixture.

A second remark is that the interest of Corollary 2 lies in the fact that it requires only a finite mixture to approximate the loss. Indeed, while we could use the standard approximation results (e.g. the Stone-Weierstrass theorem) to approximate the data density to arbitrary precision by Gaussian mixtures, this would require a diverging number of Gaussian in the mixture. The fact that loss is captured with finite C is key to our approach.

2.2 Convergence of expectations for Joint Minimization and Sampling

Our next result establishes a general relationship between the differentiability of the limit of expected training errors or free energies for empirical risk minimization or free energies for sampling and the universality of expectations of a given function of a set of parameters arising from multiple objectives. As a motivating example, consider the uncertainty quantification in Section 1 that uses both Bayesian and ERM estimators [19, 24]. The parameters ˆ Θ erm and Θ Bayes are obtained through empirical risk minimization and posterior sampling respectively on the same sequence of training data. In general, the inputs used in different objectives could be different but have some correlation structure. In the setup of ensembling (Equation 7), they are correlated through the feature mapping z i ↦→ x ( m ) i = φ m ( z i ) . In light of these considerations, we present the following general setup: Consider a sequence of M risks:

with possibly different losses, regularizers and datasets. For simplicity, we assume that the objectives are defined on parameters having the same dimension Θ ∈ R p × k . We aim to minimize M 1 of them:

and the M 2 remaining parameters are independently sampled from a family of Gibbs distributions:

where M = M 1 + M 2 . The joint distribution of the x i = ( x (1) i , . . . , x ( M ) i ) is assumed to be a mixture of the form (1). However, we assume that the labels y ( m ) i only depend on the vectors x ( m ) i :

The equivalent Gaussian inputs g i = ( g (1) i , . . . , g ( M ) i ) and their labels y ( G ) are defined as in (8).

Statistical metric and free energy Our goal is to study statistical metrics for some function h : R M × k × p → R of the form h ( Θ (1) , · · · , Θ ( M ) ) . For instance, the metric h could be the population risk (a.k.a. generalization error), or some overlap between Θ and Θ ⋆ . We define the following coupling free energy function:

where P ( M 1 +1): M denotes the product measure of the P m defined in (16). This gives rise to the following joint objective:

In particular, when s = 0 we have f n, 0 = 0 and the problem reduces to the joint minimization problem in (15). Our first result concerns the universality of the minimum of the above problem:

Proposition 3 (Universality for joint minimization and sampling) . Under Assumptions 1-4, for any s > 0 and bounded Lipschitz function Φ : R → R , and denoting ̂ R ⋆ n,s ( X , y ) := min ̂ R n,s ( Θ ; X , y ) :

The proof uses a reduction to Corollary 2, and can be found in App. A.5. The next result concerns the value of h at the minimizers point ( ˆ Θ (1) , . . . , ˆ Θ ( M ) ) . We make the following additional assumptions: Assumption 5 (Differentiable Limit) . There exists a neighborhood of 0 such that the function q n ( s ) = E [ ̂ R ⋆ n,s ( G , y ( G )) ] converges pointwise to a function q ( s ) that is differentiable at 0 .

The above assumption stems from the convexity based argument used to prove Theorem 4.

For a fixed realization of the dataset X , we denote by 〈 h ( Θ (1) , · · · , Θ ( M ) ) 〉 X the expected value of h when ( ˆ Θ (1) , . . . , ˆ Θ ( M 1 ) ) are obtained through the minimization of (15) and ( Θ ( M 1 +1) , . . . , Θ ( M ) ) are sampled according to the Boltzmann distributions (16).

Assumption 6. With high probability on X , G , the value 〈 h ( Θ (1) , · · · , Θ ( M ) ) 〉 X (resp. the same for G ) is independent from the chosen minimizers in (15) .

The above assumption is motivated by the fact that commonly non-convex problems contain minima exhibiting a specific symmetry. For example, all the global minima for a two-layer neural network are permutation invariant. Assumption 6 reflects that the quantity h respects these symmetries by taking the same value at each global minimum. This can be replaced by the stronger condition of a unique minimizer. Then the following holds:

Theorem 4. Under Assumptions 1-6, we have:

Proof Sketch: Our proof relies on the observation that q n ( s ) is a concave function of s . Further:

This allows us to leverage a result of convex analysis relating the convergence of a sequence of convex or concave functions to the convergence of the corresponding derivatives, bypassing the more involved probabilistic arguments in [16, 17]. Our approach also generalizes in a straightforward manner to the setup of multiple objectives.

The above result shows that the expected value of h ( Θ (1) , · · · , Θ ( M ) ) for a multi-modal data satisfying the 1d CLT is equivalent to that of a mixture of Gaussians. The full theorem is presented and proven in Appendix A.

2.3 Universal Weak Convergence

Theorem 4 provides a general framework for proving the equivalence of arbitrary functions of parameters obtained by minimization/sampling on a given mixture dataset and the equivalent gaussian mixture distribution. However, it relies on the assumption of a differentiable limit of the free energy (Assumption 5). If the assumption holds for a sequence of functions belonging to dense subsets of particular classes of functions, it allows us to prove convergence of minimizers themselves, in a weak sense. We illustrate this through a simple setup considered in [12], which precisely characterized the asymptotic distribution of the minimizers of empirical risk with GMM data in the strictly convex case. Consider the following setup:

where W ∈ R |C|× d , b ∈ R |C| and y i ∈ R |C| is the one-hot encoding of the class index c i . We make the following assumptions:

Assumption 7. All of the covariance matrices Σ c are diagonal, with strictly positive eigenvalues ( σ c,i ) i ∈ [ d ] , and there exists a constant M > 0 such that for any c ∈ C we have σ c,i ≤ M and ∥ µ c ∥ 2 ≤ M .

Secondly, since we aim at obtaining a result on the weak convergence of the estimators, we assume the same weak convergence for the means and covariances, and that the regularization only depends on the empirical measure of W .

Assumption 8. The empirical distribution of the µ c and Σ c converges weakly as follows:

Assumption 9. The regularizer r ( · ) is a pseudo-Lipshitz function of finite-order having the following form: r ( W ) = ∑ d i =1 ψ r ( W i ) , for some convex, differentiable function ψ r : R → R . This includes, in particular the squared reqularization r ( W ) = ∑ d i =1 W 2 i .

We briefly comment on the choice of the above assumptions. The boundedness of the Σ c and µ in Assumption 7 guarantees that we are in a case covered both by [12] and by the assumptions of Theorem 4. The diagonal property of the Σ c in 7, as well as the joint convergence in Assumption 8, ensure that we can view the minimization problem 22 ensures that W ⋆ , b ⋆ converge towards a well-defined limit. Finally, the separability assumption on r in assumption 9 responds to the fact that we aim for a result on the empirical coordinate distribution of W ⋆ , b ⋆

Under these conditions, the joint empirical measure of the minimizers and of the data moments converges weakly to a fixed limit, indepdendent of the data-distribution:

Theorem 5. Assume that conditions 1-9 hold, and further that the function ℓ ( · , y ) + r ( · ) is convex, coercive and differentiable. Then, for any bounded-Lipschitz function: Φ : R 3 |C| → R , we have:

where ˜ p is a measure on R 3 |C| , that is determined by the so-called replica equations .

Proof Sketch The proof starts with the observation that the nonlinear system of (replica) equations in [12] describes the joint-empirical measure of the parameters, means and covariances of the mixtures in a self-consistent manner. Furthermore, for h ( W ) having bounded second derivatives, the perturbation term sh ( W ) can be absorbed into the regularizer. We then utilize topological and analytical arguments to relate the weak convergence to the differentiability Assumption 5 for functions that can be expressed as expectations w.r.t the joint empirical measure in 24. More details can be found in Appendix A.8.

In particular, the above result implies the universality of the overlaps of the minimizers with means, covariances, as well as their geometrical properties such as L p norms.

2.4 One-dimensional CLT for Random Features

We finally show a conditional one-dimensional CLT for a random features map applied to a mixture of gaussians, in the vein of those shown in [14, 16, 17]. Concretely, we consider the following setup:

where the feature matrix F ∈ R p × d has i.i.d N (0 , 1 / d ) entries. This setup is much more permissive than the ones in [16, 17], that restrict the samples z to standard normal vectors. However, we do require some technical assumptions:

Assumption 10. The activation function σ is thrice differentiable, with ∥ σ ( i ) ∥ ≤ M for some M > 0 , and we have E g ∼N (0 , 1) [ σ ( g )] = 0 . Additionally, the cluster means and covariances of z satisfy for all c ∈ C ∥ µ z c ∥ ≤ M, ∥ Σ z c ∥ op ≤ M for some constant M > 0 .

We also place ourselves in the proportional regime, i.e. a regime where p/d ∈ [ γ -1 , γ ] for some γ > 0 . For simplicity, we will consider the case k = 1 ; and the constraint set S p as follows:

for a given η > 0 . We show in the appendix the following theorem:

Theorem 6. Under Assumption 10, and with high probability on the feature matrix F , the data X satisfy the concentration assumption 2, as well as the one-dimensional CLT of Assumption 4. Consequently, the results of Theorems 1 and 4 apply to X and their Gaussian equivalent G .

Proof Sketch Our proof proceeds by defining the following neuron-wise activation functions:

We subsequently control the effects of the means, covariances and the dimensions of the inputs to prove a result analogous to the one-dimensional CLT for random features in [16, 17, 14]. While we prove the above result for random weights, we note, however that the non-asymptotic results in [16, 14] also hold for deterministic matrices satisfying approximate orthogonality conditions. Therefore, we expect the one-dimensional CLT to approximately hold for a much larger class of feature maps. Finally, we also note that the above extension of the one-dimensional CLT to mixture of gaussians also provides a proof for the asymptotic error for random features in [11].

Conclusions We demonstrate the universality of the Gaussian mixture assumption in highdimension for various machine learning tasks such as empirical risk minimization, sampling and ensembling, in a variety of settings including random features or GAN generated data. We also show that universality holds for a large class of functions, and provide a weak convergence theorem. These results, we believe, vindicate the classical theoretical line of works on the Gaussian mixture design. We hope that our results will stimulate further research in this area. We also believe it crucial to understand the limitations of our extended universality framework, for instance in the cases of data with low-dimensional structure or sparsity.

References

David Donoho and Jared Tanner. Observed universality of phase transitions in high-dimensional geometry, with implications for modern data analysis and signal processing. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences , 367(1906):4273-4293, 2009.

Satish Babu Korada and Andrea Montanari. Applications of the lindeberg principle in communications and statistical learning. IEEE Transactions on Information Theory , 57(4):2440-2450, 2011.

Hatef Monajemi, Sina Jafarpour, Matan Gavish, null null, David L. Donoho, Sivaram Ambikasaran, Sergio Bacallado, Dinesh Bharadia, Yuxin Chen, Young Choi, Mainak Chowdhury, Soham Chowdhury, Anil Damle, Will Fithian, Georges Goetz, Logan Grosenick, Sam Gross, Gage Hills, Michael Hornstein, Milinda Lakkam, Jason Lee, Jian Li, Linxi Liu, Carlos SingLong, Mike Marx, Akshay Mittal, Hatef Monajemi, Albert No, Reza Omrani, Leonid Pekelis, Junjie Qin, Kevin Raines, Ernest Ryu, Andrew Saxe, Dai Shi, Keith Siilats, David Strauss, Gary Tang, Chaojun Wang, Zoey Zhou, and Zhen Zhu. Deterministic matrices matching the compressed sensing phase transitions of gaussian random matrices. Proceedings of the National Academy of Sciences , 110(4):1181-1186, 2013.

Emmanuel J Candès, Pragya Sur, et al. The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression. The Annals of Statistics , 48(1):2742, 2020.

Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences , 117(48):30063-30070, 2020.

Xiaoyi Mai and Zhenyu Liao. High dimensional classification via empirical risk minimization: Improvements and optimality. arXiv: 1905.13742 , 2019.

Francesca Mignacco, Florent Krzakala, Yue Lu, Pierfrancesco Urbani, and Lenka Zdeborova. The role of regularization in classification of high-dimensional noisy gaussian mixture. In International Conference on Machine Learning , pages 6874-6883. PMLR, 2020.

Hossein Taheri, Ramtin Pedarsani, and Christos Thrampoulidis. Optimality of least-squares for classification in gaussian-mixture models. In 2020 IEEE International Symposium on Information Theory (ISIT) , pages 2515-2520. IEEE, 2020.

Ganesh Ramachandra Kini and Christos Thrampoulidis. Phase transitions for one-vs-one and one-vs-all linear separability in multiclass gaussian mixtures. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 40204024. IEEE, 2021.

Ke Wang and Christos Thrampoulidis. Benign overfitting in binary classification of gaussian mixtures. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 4030-4034. IEEE, 2021.

Maria Refinetti, Sebastian Goldt, Florent Krzakala, and Lenka Zdeborová. Classifying highdimensional gaussian mixtures: Where kernel methods fail and neural networks succeed. In International Conference on Machine Learning , pages 8936-8947. PMLR, 2021.

Bruno Loureiro, Gabriele Sicuro, Cedric Gerbelot, Alessandro Pacco, Florent Krzakala, and Lenka Zdeborová. Learning gaussian mixtures with generalized linear models: Precise asymptotics in high-dimensions. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 10144-10157. Curran Associates, Inc., 2021.

Sebastian Goldt, Marc Mézard, Florent Krzakala, and Lenka Zdeborová. Modelling the influence of data structure on learning in neural networks: the hidden manifold model. Physical Review X , 10:041044, 2019.

Sebastian Goldt, Bruno Loureiro, Galen Reeves, Florent Krzakala, Marc Mezard, and Lenka Zdeborova. The gaussian equivalence of generative models for learning with shallow neural networks. In Joan Bruna, Jan Hesthaven, and Lenka Zdeborova, editors, Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference , volume 145 of Proceedings of Machine Learning Research , pages 426-471. PMLR, 16-19 Aug 2022.

Noureddine El Karoui. The spectrum of kernel random matrices. The Annals of Statistics , 38(1):1-50, 2010.

Hong Hu and Yue M. Lu. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory , pages 1-1, 2022.

Andrea Montanari and Basil N. Saeed. Universality of empirical risk minimization. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 4310-4312. PMLR, 02-05 Jul 2022.

Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mezard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with a teacher-student model. Advances in Neural Information Processing Systems , 34, 2021.

Lucas Clarté, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Theoretical characterization of uncertainty in high-dimensional linear classification. arXiv: 2202.03295 , 2022.

Bruno Loureiro, Cedric Gerbelot, Maria Refinetti, Gabriele Sicuro, and Florent Krzakala. Fluctuations, bias, variance & ensemble of learners: Exact asymptotics for convex losses in high-dimension. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 14283-14314. PMLR, 17-23 Jul 2022.

Federica Gerace, Florent Krzakala, Bruno Loureiro, Ludovic Stephan, and Lenka Zdeborová. Gaussian universality of linear classifiers with random labels in high-dimension. arXiv: 2205.13303 , 2022.

Luca Pesce, Florent Krzakala, Bruno Loureiro, and Ludovic Stephan. Are gaussian data all you need? extents and limits of universality in high-dimensional generalized linear estimation. arXiv preprint arXiv:2302.08923 , 2023.

Mohamed El Amine Seddik, Cosme Louart, Mohamed Tamaazousti, and Romain Couillet. Random matrix theory proves that deep learning representations of GAN-data behave as Gaussian mixtures. In Proceedings of the 37th International Conference on Machine Learning , ICML'20, pages 8573-8582. JMLR.org, July 2020.

Lucas Clarté, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. A study of uncertainty quantification in overparametrized high-dimensional models. arXiv: 2210.12760 , 2022.

Terence Tao and Van Vu. Random matrices: Universality of local eigenvalue statistics. Acta Mathematica , 206(1):127 - 204, 2011.

Terence Tao and Van Vu. Random matrices: universal properties of eigenvectors. Random Matrices: Theory and Applications , 01(01):1150001, 2012.

Ashkan Panahi and Babak Hassibi. A universal analysis of large-scale regularized least squares solutions. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Andrea Montanari and Phan-Minh Nguyen. Universality of the elastic net error. In 2017 IEEE International Symposium on Information Theory (ISIT) , pages 2338-2342, 2017.

Ehsan Abbasi, Fariborz Salehi, and Babak Hassibi. Universality in learning from linear measurements. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Alia Abbara, Antoine Baker, Florent Krzakala, and Lenka Zdeborová. On the universality of noiseless linear estimation with respect to the measurement matrix. Journal of Physics A: Mathematical and Theoretical , 53(16):164001, mar 2020.

Rishabh Dudeja, Subhabrata Sen, and Yue M. Lu. Spectral universality of regularized linear regression with nearly deterministic sensing matrices. arXiv: 2208.02753 , 2022.

Yue M. Lu and Horng-Tzer Yau. An equivalence principle for the spectrum of random innerproduct kernel matrices. arXiv: 2205.06308 , 2022.

Theodor Misiakiewicz. Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regression. arXiv: 2204.10425 , 2022.

Andrea Montanari, Feng Ruan, Youngtak Sohn, and Jun Yan. The generalization error of max-margin linear classifiers: High-dimensional asymptotics in the overparametrized regime. arXiv: 1911.01544 , 2019.

Federica Gerace, Bruno Loureiro, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. Generalisation error in learning with random features and the hidden manifold model. In International Conference on Machine Learning , pages 3452-3462. PMLR, 2020.

Oussama Dhifallah and Yue M. Lu. A precise performance analysis of learning with random features. arXiv: 2008.11904 , 2020.

Tengyuan Liang and Pragya Sur. A precise high-dimensional asymptotic theory for boosting and minimumℓ 1 -norm interpolated classifiers. The Annals of Statistics , 50(3):1669 - 1695, 2022.

Dominik Schröder, Hugo Cui, Daniil Dmitriev, and Bruno Loureiro. Deterministic equivalent and error universality of deep random features learning. arXiv: 2302.00401 , 2023.

Cosme Louart, Zhenyu Liao, and Romain Couillet. A random matrix approach to neural networks. The Annals of Applied Probability , 28(2):1190-1248, 2018.

Mohamed El Amine Seddik, Cosme Louart, Romain Couillet, and Mohamed Tamaazousti. The unexpected deterministic and universal behavior of large softmax classifiers. In International Conference on Artificial Intelligence and Statistics , pages 1045-1053. PMLR, 2021.

Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv: 1411.1784 , 2014.

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d'Ascoli, Giulio Biroli, Clément Hongler, and Matthieu Wyart. Scaling description of generalization with number of parameters in deep learning. Journal of Statistical Mechanics: Theory and Experiment , 2020(2):023401, feb 2020.

Stéphane d'Ascoli, Maria Refinetti, Giulio Biroli, and Florent Krzakala. Double trouble in double descent: Bias and variance (s) in the lazy regime. In International Conference on Machine Learning , pages 2280-2290. PMLR, 2020.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv: 1708.07747 , 2017.

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems , volume 20. Curran Associates, Inc., 2007.

Francesco Guerra. Broken replica symmetry bounds in the mean field spin glass model. Communications in mathematical physics , 233(1):1-12, 2003.

R Tyrrell Rockafellar. Convex analysis , volume 18. Princeton university press, 1970.

Michael Celentano, Andrea Montanari, and Yuting Wei. The lasso with general gaussian designs with applications to hypothesis testing. arXiv: 2007.13716 , 2020.

J. W. Lindeberg. Eine neue Herleitung des Exponentialgesetzes in der Wahrscheinlichkeitsrechnung. Mathematische Zeitschrift , 15(1):211-225, December 1922.

Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

S. G. Bobkov. On concentration of distributions of random weighted sums. The Annals of Probability , 31(1):195-215, January 2003.

Michel Ledoux. The concentration of measure phenomenon , volume 89 of Math. Surv. Monogr. American Mathematical Society (AMS), Providence, RI, 2001.

Karl Pearson. On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science , 2(11):559-572, November 1901.

Mark A. Kramer. Nonlinear principal component analysis using autoassociative neural networks. AIChE Journal , 37(2):233-243, 1991.

Elena Facco, Maria d'Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific Reports , 7(1):12140, 2017.

Stefano Spigler, Mario Geiger, and Matthieu Wyart. Asymptotic learning curves of kernel methods: empirical data versus teacher-student paradigm. Journal of Statistical Mechanics: Theory and Experiment , 2020(12):124001, dec 2020.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv: 1312.6114 , 2013.

Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 1530-1538, Lille, France, 07-09 Jul 2015. PMLR.

Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Jens Behrmann, Paul Vicol, Kuan-Chieh Wang, Roger Grosse, and Joern-Henrik Jacobsen. Understanding and mitigating exploding inverses in invertible neural networks. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , volume 130 of Proceedings of Machine Learning Research , pages 1792-1800. PMLR, 13-15 Apr 2021.

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations , 2018.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019.

Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. arXiv: 1606.03657 , 2016.

Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, ˙ Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods , 17:261-272, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn classification problems with a finite number of classes under a supervised setting, if the feature vectors are generated from a mixture of distributions conditioned on the class labels and the learning is performed using generalized linear models, a phenomenon called ``Gaussian mixture equivalence (GME)'' simplifying the problem is known to occur. GME is the phenomenon in the high-dimensional limit that when considering a model in which the original conditional distribution of the feature vector is replaced by a Gaussian distribution having the same first- and second-order moments as the original ones, the quantities of interest such as generalization error coincide between the original model with the non-trivial class-conditional distribution and the reduced model with the Gaussian distribution. This paper rigorously proves this GME actually occurs under some (partly mild, partly strong) conditions by extending the method of reference [17] to the case involving multiple sets of parameters, which may be determined from the minimization of different objectives or by sampling from different Gibbs distributions.\n\nSTRENGTHS:\nOriginality & Significance: Although GME has been assumed in previous studies and meaningful results have been shown under the assumption, the results of this paper clarifies the sufficient conditions under which GME can actually occur by the rigorous proof. This provides a mathematical foundation for attempts to understand complicated machine learning/statistical models' behavior through the analysis of simple Gaussian mixture models. The originality and the significance are thus high. \n\n\nClarity & Quality: The main text mainly provides the statements about the theorems, assumptions, and their consequences/applicabilities, and most of the proof details, even including the intuitive arguments, are deferred to the appendix. The statements are also not easy to understand, because many things are explained in the limited pages and the relationships among the theorems, assumptions, and consequences/applicabilities are not evident. The distinction between what has been proven in previous studies and what has been reiterated here is also not clear. Hence the clarity is not high. Nevertheless, the argument can be understood if one reads the manuscript carefully, and considering the importance of what is proven, also with the nice numerical experiment completely supporting the proposition, it is considered that the quality is high.\n\n\nHere I enumerate some typos found and uneasy expressions for better clarity:\n\nLine 162: The sentence is not easy to understand.\n\nBelow Line 200: The quantity $\\ell'$ is not explained.\n\nLine 225: be be -> be\n\nLine 246: Theorem 2 -> Corollary 2\n\nLine 262: a result convex -> a result of convex\n\nLine 305,325: gaussian -> Gaussian\n\nWEAKNESSES:\n- Proof technicalities seem to much rely on [17], and there does not appear to have been much technical progress when extending it to the current case. \n\n- Assumption 4 is the key assumption in the proof but when it does hold is not clear. Maybe this is the difficult point of the problem, and hence some more explanations in words are welcome.\n\nQUESTIONS:\n- Is it possible to extend the proof to the case where $\\bm{x}_i$ has correlations among $i \\in [n]$? An interesting example is the orthogonal ensemble under $p>n$. Some discussions are welcome.\n\nLIMITATIONS:\nThe authors address their research limitations well. I think there is no concern about the potential societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors investigated the joint statistical behaviors of a set of genealized linear estimators obtained from either empirical risk minimization or sampling, for data and labels drawn from a mixture model, in the high-dimensional regime where where the data dimension $p$ and the sample size $n$ both go to infinity.\n\nIn particular, they establish sufficient conditions under which the asymptotic joint statistics of this family depend only on the means and covariances of the mixture model.\n\nThe obtained results extend a few previous efforts: for instance, provide alternative proof to the fact that Gaussian mixture observed through a random feature map is akin to a Gaussian mixture [11, 12, 21, 22, 16]; and that data generated by conditional Generative Adversarial Networks (cGAN) are like Gaussian mixture [23], etc.\n\nSome theoretical results are presented in a not-so-precise manner and some clarifications are needed. The paper is in general well written otherwise.\n\nSTRENGTHS:\nThe topic focuses on the emerging topic of universality of ML methods in the high-dimensional regime.\nThe obtained theoretical results are strong and improve prior arts in many ways.\nI did not check the detailed proofs, but the proof sketch look compelling.\n\nWEAKNESSES:\nThere are a lot of assumptions in the paper, and it somewhat makes the paper less easy to grasp and compare to existing works, some more explanations are needed.\nSome clarifications are needed to better understand the statements in the paper, as well as the major contribution of this paper.\n\nQUESTIONS:\nI have the following minor comments or questions:\n\n1. Assumption 2 indeed describes the type of (mixture) data distribution under study, some more explanation and/or comparison would be helpful here. \n2. Corollary 2 seems to be a corollary of Theorem 1, and if this is the case, it should be announced to hold under \"the same setting of Theorem 1\" or under \"Assumption 1-4\".\n2. Theorem 1 and Corollary 2 hold as long as one takes the limit of $n,p \\to \\infty$? Should some assumption be put on their ratio $p/n$?\n3. last paragraph in page 6 \"this would require a diverging number of Gaussian in the mixture\": would the authors elaborate on this, with some reference or more details for example?\n4. it would be helpful to comment on the necessity and/or importance of Assumption 7-9, otherwise they are a bit difficult to digest and compare.\n5. Also, the dimension $n,p$ are NOT involved in Theorem 5?\n6. as expectations w.r.t the joint empirical measure in 24 -> in (24) or Equation 24.\n7. could the authors elaborate on the replica equations in Theorem 5: can they be numerically evaluated (with ease)?\n\nLIMITATIONS:\nThis work is theoretical and I do not see any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
The submission studies the universality phenomena for general mixture model. The authors show the universality of empirical risk minimization and sampling and several extensions.

STRENGTHS:
The authors give good introduction and literature review.

WEAKNESSES:
1. The main contribution lies in the technical part. However, it lacks novelty given the results in Hu and Lu 2022, Montanari and Saeed 2022. The assumptions 1-4 and Section 2.1 are almost identical to the main results in Montanari and Saeed 2022 without mentioning or comparisons at the beginning. The proofs for Theorem 1 and Corollary 2 only recap the previous results in the appendix. Section 2.2 gives some simple extensions of those results in Montanari and Saeed 2022 to multiple risk case.

2. The paper has plenty of inconsistencies and lack of explanations in the course of writing. For example, line 109, the notation of \eta is not consistent to equation (2). The three applications in Section 1 are hard to follow given the model setting. 

3. Some of the notations are not proper: For example, line 101, it would be better to write \hat{y}=F(\hat{\Theta}^T x) to denote estimated parameters. I suggest the authors modify the notations in the whole paper to fit their own topic and interests instead of using those in Montanari and Saeed 2022.

4. Also, it would be better to explain the notations immediately after introducing.

5. The statements of the main theorems and proofs in the appendix is lack of polish. It would be better to write remarks after the theorems to improve the readability.

QUESTIONS:
Could the authors explain in more details the technical differences between their submission and the previous analyses?

LIMITATIONS:
No
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proves that the performance and other quantities of interest output by a number of algorithms in the generalized linear models class are the same as if the data were replaced by Gaussians with the same means and covariance matrices. This universality result suggests that the performance of this entire class of algorithms (generalized linear models) depends only on the first- and second-order moments of the data. This theoretical study justifies the use of a Gaussian mixture model in the study of generalized linear algorithms.\nThe particularity of the study consists in proving this important phenomenon rigorously even on restrictive assumptions.\n\nSTRENGTHS:\n1-The question of the universality of Gaussian mixtures in the analysis of the performance of generalized linear algorithms is a subject that has long remained empirical and has known very few rigorous and clear answers. This paper by providing a rigorous answer, makes it possible to make an important step in the resolution of this question.\n\n2- The paper is generally well written with very few spelling mistakes and consistency in notations. The motivation and structure of the paper seems to be quite clear.\n\n3- Although not having checked the theoretical calculations, the approach seems to be rigorous and well conducted. The results are consistent.\n\nWEAKNESSES:\n1- Assumption 4 and Assumption 5 need to be better explained in terms of their intuitive meaning and how they are binding. Although there is some discussion that it is very restrictive and experiments tend to suggest that the results should be valid beyond that, it would be important to provide more intuitive explanations of what it means and why it is restrictive.\n\n2- In the experiment shown in Figure 2, there is a systematic bias between the performance of the CGAN and the GMM. It would be interesting to discuss this aspect, providing explanations if necessary. It would have been useful to show the limits of universality in a general way (even in the appendix). Are there cases where this holds less well, and this would be due to what phenomenon, or are there algorithms that are more resistant to this universality in the class of linear algorithms?\n\n3- Although I appreciate the authors' effort to provide experiments on real data, it would have been interesting to do so with different data structures and slightly more challenging data such as time series and language classification. In particular, we could ask the question of universality for highly structured data such as very sparse data (like speech encoding) or time series. It seems somewhat counter-intuitive that for these highly structured data, the mean and variance alone are sufficient to characterize performance. Recently, in random matrix theory, universality has been focused on a class of random vectors that come from the concentration of the measure and it is not obvious that highly sparse data are concentrated and therefore that universality holds for this type of data, even through the prism of a linear model.\n\nQUESTIONS:\n1- What is the intuitive meaning of Ass 4 and Ass 5? How restrictive they are?\n\n2- Is the universality behave the same whatever the type of structure considered (highly sparse data, time series)? Do some experiments could be provided on some of these datasets? Are these some cases of failure of the universality noticed in experiments?\n\nLIMITATIONS:\nThe main statement of the paper seems quite strong even though the mathematics seems rigorous. In this case, it would be really important to provide a wide set of experiments over several datasets (with different structure) and several linear algorithms. This would be more reassuring than checking the calculations line by line for the reader.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}