Universality laws for Gaussian mixtures in generalized linear models
Authors: Yatin Dandi
Ludovic Stephan
Florent Krzakala
Bruno Loureiro
Lenka Zdeborova
Venue: NeurIPS 2023 poster (2023)
Keywords: theoretical analysis
high-dimensional statistics
Universality
weak convergence
mixture models
sampling
statistical physics
TL;DR: Exploring the universality of Gaussian mixtures in high-dimensional statistics, this work rigorously proves their applicability across various machine learning tasks such as ERM, uncertainty quantification, and ensembling.

Abstract: A recent line of work in high-dimensional statistics working under the Gaussian mixture hypothesis has led to a number of results in the context of empirical risk minimization, Bayesian uncertainty quantification, separation of kernel methods and neural networks, ensembling and fluctuation of random features. We provide rigorous proofs for the applicability of these results to a general class of datasets $(\mathbf{x_i},y_i, {i=1,\dots,n})$ containing independent samples from a mixture distribution $\sum_{c\in\mathcal{C}} \rho_{c}P_{c}^{\mathbf{x}}$. Specifically, we consider the hypothesis class of generalized linear models $\hat{y} = F(\mathbf{\Theta}^{\top}\mathbf{x})$ and investigate the asymptotic joint statistics of a family of generalized linear estimators $(\mathbf{\Theta}^{(1)}, \dots, \mathbf{\Theta}^{(M)})$, obtained either from (a) minimizing an empirical risk $\hat{R_n}^{(m)}(\mathbf{\Theta}^{(m)};\mathbf{X},\mathbf{y})$ or (b) sampling from the associated Gibbs measure $\exp(-\beta n \hat{R_n}^{(m)}(\mathbf{\Theta}^{(m)};\mathbf{X},\mathbf{y}))$. Our main contribution is to characterize under which conditions the asymptotic joint statistics of this family depends (on a weak sense) only on the means and covariances of the class conditional features distribution $P_{c}^{\mathbf{x}}$. This allows us to prove the universality of different quantities of interest, including training, generalization errors, as well as the geometrical properties and correlations of the estimators.

Universality laws for Gaussian mixtures in generalized linear models
Yatin Dandi, Ludovic Stephan, Florent Krzakala Idephics, EPFL. Switzerland
Bruno Loureiro
Lenka Zdeborova SPOC, EPFL, Switzerland
DI, Ecole Normale Supericure, Paris, France
Abstract
A recent line of work in high-dimensional statistics working under the Gaussian mixture hypothesis has led to number of results in the context of empirical risk minimization, Bayesian uncertainty quantification, separation of kernel methods and neural networks, ensembling and fluctuation of random features. We pro- vide rigorous proofs for the applicability of these results to a general class of datasets (r,, y=) containing independent samples from mixture distribution Eeec PeP Specifically, we consider the hypothesis class of generalized linear models y F(@ x) and investigate the asymptotic joint statisties of family of generalized linear estimators (0(1).... ...... (MM),, obtained either from (a) minimiz- ing an empirical risk /.(m)) (o (m), x, y) or (b) sampling from the associated Gibbs measure exp(-3mnft) (mm) (m); X, y)). Our main contribution is to characterize under which conditions the asymptotic joint statistics of this family depend (on weak sense) only on the means and covariances of the class conditional feature distribution P This allows us to prove the universality of different quantities interest, including training and generalization errors, well as the geometrical properties and correlations the estimators.
recurrent topie in high-dimensional statistics is the investigation of the typical properties of signal processing and machine learning methods on synthetic, i.i.d. Gaussian data, a scenario often known under the umbrella of Gaussian design [1, 2, 4, 5]. A less restrictive assumption arises when considering that many machine learning tasks deal with data partitioned into fixed number of classes. In these cases, the data distribution is naturally described by mixture model, where each sample is generated conditionally on the class. In other words: data is generated by first sampling the class assignment and then generating the input conditioned the class. Arguably the simplest example of such distributions is that of a Gaussian mixture, which shall be our focus in this work.
popular model in high-dimensional statistics since, besides being universal approximator, they often lead to mathematically tractable problems. Indeed, a recent line of work has analyzed the asymptotic performance of a large class of machine learning problems in the proportional high-dimensional limit under the Gaussian mixture data assumption, see e.g. [6, 7, 8, 9, 10, 11, 12). The goal of the present work is show that this assumption, and the conclusions derived therein, far more general than previously anticipated.
Gaussian mixtures
A recent line of work [13, 14], initiated by the work of [15] for Kernel matrices, posits that for generalized linear estimation on non-linear feature maps satisfying certain regularity conditions and a "one fone-dimensional CLT", the data distribution can be replaced by equivalent Gaussian data without affecting the training and generalization errors. This was recently proven by [16] for empirical
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

risk minimization under the setup of strongly convex objectives, and extended to larger class of objectives by [17].
However, there is strong empirical evidence that Gaussian universality holds in a more general sense [18). First, existing results rely on the assumption of a target function depending on linear projections in the latent or feature space. This excludes the rich class of classification on mixture distributions, where the target function is given by the label. For such distributions, a more appropriate equivalent distribution is given by mixture of Gaussians. Such a "Gaussian mixture equivalence" has been conjectured and used in existing works, such as [11, 12] and was found to closely approximately classification on real datasets.
Furthermore, equivalence with mixtures of Gaussians has been observed to hold not only for training, generalization errors but other quantities of the estimators such as overlaps, variance, etc. For instance, [19, 20] empirically observed that the equivalence holds even while considering the joint distribution of multiple uncertainty estimators or ensembles of multiple random feature mappings. This suggests the equivalence of the distributions of the minimizers themselves.
Our results fill these gaps and provide rigorous justification for the universality in all the aforemen- tioned works. Namely, we show that the joint statistics of multiple generalized estimators obtained either from ERM or sampling on mixture model asymptotically agrees (in a weak sense) with the statistics of estimators from the same class trained on Gaussian mixture model with matching first and second order moments. Our main contributions are as follows:
Through generalization of recent developments in the Gaussian equivalence principle [14, 17, 16], we prove the universality of empirical risk minimization and sampling for a generic mixture distribution and an equivalent mixture of Gaussians. In particular, we show that a Gaussian mixture observed through random feature map is also a Gaussian mixture in the high-dimensional limit, a fact used for instance (without rigorous justification) in [11, 12, 21, 22].
A consequence of our results is that, with conditions on the matrix weights, data generated by Gaussian mixture when observed
conditional Generative Adversarial Networks (cGAN) behave through the prism of generalized linear models (kernels, feature maps, etc..., as illustrated in Figs 1 and 2. This further generalizes the work of [23] that only considered the universality of Gram matrices for GAN generated data through the prism of random matrix theory.
We construct unified framework involving multiple sets of parameters arising from simultaneous minimization of different objectives as well as sampling from Gibbs distributions defined by the empirical risk. Through the design of suitable reductions and convexity- based argument, establish conditions for the asymptotic universality of arbitrary functions of the set of minimizers or samples from different Gibbs distributions (Theorem 4). For instance, includes ensembling [20]), and the uncertainty quantification and Bayesian setting assumed (without proof) in [19, 24).
Finally, we show that for multi-class classification, the conditions leading to universality hold for a large class of functions of the minimizers, such as overlaps and sparsity measures, leading to the equivalence between their distributions of themselves, and provide a theorem for their weak convergence (Theorem 5).
As technical contribution of independent interest, our proof of Theorem 5 demonstrates a princi- pled approach for leveraging existing results on the exact asymptotics for simple data distributions (such as for Gaussian mixture models in [12)) to prove the weak convergence and universality of the joint-empirical measure of the estimators and parameters (means, covariances) of the data-distribution.
it motivates the scope of tractable mathematical models. It has been extensively studied in the context of random matrix theory [25, 26], signal processing problems [1, 2, 3, 27, 28, 29, 30. 31] and kernel methods [15, 32, 33]. Closer to us is the recent stream of works that investigated the Gaussian universality of the asymptotic error of generalized linear models trained on non-linear features, starting from single- layer random feature maps [34, 35, 16, 36. 37] and later extended to single-layer NTK [17] and deep random features [38]. These results, together with numerical observations that Gaussian universality holds for more general classes of features, led to the formulation of different Gaussian equivalence conjectures [13, 14, 18]. Our results build on the line of works on the proofs of these conjectures, through the use of the one-dimensional CLT (Central Limit Theorem), stated formally in [14] who proved it for random features of Gaussian data. We generalize this principle to a Gaussian equivalence conditioned on the cluster assignment in mixture-modee with corresponding conditional Id-CLT (Assumption 10).
Related works Universality is an important topic in applied mathematics,
2

---

A complementary line of research has investigated cases in which the distribution of the features is multi- modal, suggesting a Gaussian mixture universality class instead [39. 23, 40]. A bridge between these two lines of work has been recently investigated with random labels and teachers in [21, 22]. Our results provide rigorous extensions Gaussian universality the setups of mixture models as well as uncertainty quantificatio and ensembling.
1 Setting and models
Consider a supervised learning problem where the training data (2),))) E IP V. i E [n] = " n is drawn i.idd from a mixture distribution:
32 Σ PePe
P(G = c) = Pes
LEC
with C; a categorical random variable denoting the cluster assignment for the ith example x, Let
Jes De denote the mean and covariance of P and |C]. Further, assume that the labels y are generated from the following target function:
ye (x) = ( S1, ci),
where 7) 3 -> R is general label generating function, O E & and e; is an i.i.d source of randomness. important to stress that the class labels (2) are themselves not constrained to arise from simple function of the inputs Z;. For instance, the functional form in (2) includes the case where the labels are exclusively given by a function the mixture index yi = n(()). This will allow us to handle complex targets, such data generated using conditional Generative Adversarial Networks (cGANs).
In this manuscript, we will be interested in hypothesis classes defined by parametric predictors of the form we (z) F(O" x). where O RxP are the parameters and F: I² + J' a possibly non- linear function. For given loss function f R$ y -> R4 and regularization term Rgxx - R.,
define the (regularized) empirical risk over the training data:
R. (O; = 71
e T. 11.
where we have defined the feature matrix X E RPX= by stacking the features column-wise and the labels y: in a vector y E y"" what follows, we will be interested in the following two tasks:
(i) Minimization: in minimization task, the statistician's 's goal is to find good predictor by minimizing the empirical risk (3), possibly over constraint set Sp:
e em X, y) arg min R., (e; ØES,
This encompasses diverse settings such as generalized linear models with noise, two-layer networks with a constant number of neurons and fixed second layer, mixture classification, but also the random label setting (with 17( C4, = E2). In the following, we denote R; y) = mine (O; x,y)
(ii) Sampling: here, instead of minimizing the empirical risk (3), the statistician's 's goal is to sample from a Gibbs distribution that weights different hypothesis according to their empirical error:
Bayes X, y) PBayes ( exp (- [-BnR, ( X, .)) dy(()
(5) where / is reference prior measure and 3 0 is a parameter known the inverse temperature. Note that minimization can he seen as a particular example of sampling when 33 - oc, since in this limit the above measure peaks the global minima of (4).
Applications of interest- So far, the setting defined above is quite generic, and the motivation to study this problem might not appear evident to the reader. Therefore, we briefly discuss a few scenarios of interest which are covered by this model.
3

---

Latent space
Generative model
Data
Feature map
Linear model
E Rd GMM
GMM
Effective Gaussian Mixture Model
Data RD
Figure 1: Illustration of Corollary 2: high-dimensional data generated by generative neural networks starting from a mixture of Gaussian in latent space (z E IR are (with conditions on the weights matrices) equivalent, in high-dimension and for generalized linear models, data sampled from Gaussian mixture. A concrete example is shown in Fig. 2.
ii Conditional GANs (cGANs): These were introduced by [41] as generative model learn mixture distributions. Once trained in samples from the target distribution, they define a function that maps Gaussian mixtures (defining the latent space) to samples from the target mixture that preserve the label structure. In other words, conditioned on the label:
Ec
2 N (e) 1 = (12,c) P
(6)
The connection to model (1) is immediate. This scenario was extensively studied by [39, 23, 40], and is illustrated in Fig. 1. In Fig. 2 we report on concrete experiment with a cGAN trained on the Mashion-MNITT dataset.
(ii) Multiple objectives: Our framework also allows to characterize the joint statisties of estimators (O....., ①M) obtained from empirical risk minimization and/or sampling from different objective functions fr defined on the same training data (X, y). This can be of interest in different scenarios. For instance, [19, 24] has characterized the correlation in the calibration of different uncertainty measures of interest, e.g. last-layer scores and Bayesian training of last-layer weights. This crucially depends on the correlation matrix ② ecm O Bayes E * which fits our framework.
(iii) Ensemble of features: Another example covered by the multi-objective framework above is that of ensembling. Let (zi, V:) E IR ) denote some training data from mixture model akin to (1). A popular ensembling scheme often employed in the context of deep learning [42] is take family of
M feature maps 2; + x (m) = pm (z,) (e.g. neural network features trained from different random initialization) and train M independent learners:
(m) crm
E arg min
Σ e(e
x (m) v.) + +
OES,
Prediction on new sample x is then made by ensembling the independent learners, e.g. by taking
their average ý 1/M MM (m)T (m).)) A closely related model was studied in [43, 44, 20].
Note that in all the applications above, having the labels depending on the features X would not be natural, since they are either generated from a latent space, as in (t), or chosen by the statistician, as in (it), (dit). Indeed, these cases the most natural label model is given by the mixture index e itself, which is a particular case of (2). This highlights the flexibility of our target model with respect to prior work [17]. Instead, [16] assumes that the target a function of a latent variable, which would correspond to mismatched setting. The discussion here can be generalized also to this case, but require an additional assumption discussed in Appendix B.
Universality Given these tasks, the goal of the statistician is to characterize different statistical properties of these predictors. These can be, for instance, point performance metrics such as the empirical and population risks, or uncertainty metrics such the calibration of the predictor or moments of the posterior distribution (5). These examples, as well as many different other quantities of interest, are functions of the joint statisties of the pre-activations (e) x. θ Tx), for æ either test

---

cSAN GMM
CGAN
Rrcatin
GMM
0.05
0.08
0.04
0.06
0.03
0.04
0.02
0.01
0.02
0.00
0.00
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
sample complexity m/d
sample complexity nid
Figure 2: Illustration of the universality scenario described in Fig.1. 1. Logistic (left) & ridge (right) regression test (up) and training (bottom) errors are shown versus the sample complexity a "/4 for odd vs. even binary classification task two data models: Blue dots data generated from conditional GAN [41] trained on the fashion-MNIST dataset [45] and pre-processed with a random features map + tanh(Wæ) with Gaussian weights W E 1176x778, Red dots are the 10- clusters Gaussian mixture model with means and covariances matching each fashion-MNIST cluster conditioned on labels (&g regularization is À 10-4) Details on the simulations discussed in Appendix D.
training sample from (1). For instance, in Gaussian mixture model, where E Pe PeNN Ex), sufficient statistics are simply given by the first two moments of these pre-activations. However, for general mixture model (1), the sufficient statistics will generically depend on all moments of these aeevvavivaiios. Surprisingly, our key result this work is to show that the nigh-dimenssonal limit this is not the case. In other words, under some conditions which made precise in Section 2, we show that expectations with respect to (I) can be exchanged by expectations over a Gaussian mixture with matching moments. This can be formalized follows. Define equivalent Gaussian data set (g,, yt) E IR y with samples drawn i.i.d. from the equivalent Gaussian mixture model:
Σ PeNN(Me: Ec), rEC
y: (G) n -,,,,,,,,..)))
9:
We recall that E. denotes the mean and covariance of P from (1). Consider a family of estimators (0).... O.) defined by minimization (3) and/or sampling (5) over the training data (X, y) from the mixture model (1). Let h be a statistical metric of interest. Then, in the proportional high-dimensional limit where n. p - oc a fixed a ==/aa > O, and where (.) denote the expectation with respect to the Gibbs distribution (5), we define universality as:
Ex (
xM))
EG ((((),. OM))
22000
The goal of the next section is to make this statement precise.
Main results
We now present the main theoretical contributions of the present work and discuss its consequences. Our proofs for Theorems and build upon existing results on the universality of empirical risk minimization for uni-model distributions [16, 17] and therefore rely on similar technical regularity and concentration assumptions. Concretely, our work relies on the following assumptions:
Assumption 1 (Loss and regularization). The loss function é R&+1 + R is nonnegative and Lipschitz, and the regularization function 1Rpxk + R is locally Lipschitz, with constants independent from p.
5

---

Assumption 2 (Boundedness and concentration). The constraint set Sp is a compact subset of R Further, there exists constant M such that for any 2 0,
sup
||0| x=1 S M,
sup 0EKp,1101125
| 0012 - M,
and Mell S M
(10)
@EKp.1101l:51
where I1-1ez is the sub-gaussian norm, and Kp S I&P is such that Sp SKy.
Assumption 3 (Labels). The labeling function is Lipschitz, the teacher vector Q belongs to Sp. and the noise variables E are i.i.d sub- bub-gaussian with ||e:]|02 M for some constant M >
Those three assumptions are fairly technical, and it is possible that the universality properties proven in this article hold irrespective of these conditions. The crucial assumption our theorems is that of a conditional one mne-dimensionl CLT:
Assumption 4. For any Lipschitz function yp RR - R,
lim sup 22 BEKp
E [(p0Ta x) = E
g) Co = =c] =0, Ve C
(11)
where and denote samples from the given mixture distribution and the equivalent gaussian mixture distribution in equations (1) and (8) respectively.
The above assumption is generalization of the -one-dimeniional CLT" underlying the line of work based on the Gaussian equivalence (GE) Principle [13, 14, 17, 16]. The above assumption splits the proof of universality for general mixture distribution into two parts. First, one shows that asymptotic universality of an observable can be reduced to the proof of one-dimensional CLT. Second, one proves this CLT holds for the particular class of features of interest. This proof scheme streamlines universality proofs. Our work provides general proof of the first step Theorem conditioned on the second, later showing that Assumption 4 holds for some natural feature maps of interest, i.c. random features applied to a Gaussian mixture (Theorem 6). However, Assumption is expected to hold for large class of features, as supported by empirical observations in Figure and arguments in Appendix C.
2.1 Universality of Mixture Models
We start by proving the universality of the free energy for a Gibbs distribution defined through the objective R., (@; ; X, y) for the data distribution (1) and its equivalent Gaussian mixture (8).
Theorem (Universality of Free Energy). Let (p) be sequence of Borel probability measures with compact supports Sp. Define the following free energy function:
/ exp (-3nR,, (O: (pp(()
(X) log
(12)
3n
Under Assumptions 1-4 X and Sp. for any bounded differentiable function & with bounded Lipschitz derivative, we have:
lim 00
E D (fa,a
-E (fr, =.
When Up corresponds to discrete measures supported on an e-net in Sp, using the reduction from
Lemma I to Theorem 1 in [17], obtain the following corollary:
Corollary 2 (Universality of Training Error). For any bounded functton & R
R:
lim a.p
x (
E
(xxx,)(()) -
TE ( (R])C.
In particular, for any E E R, and denoting P the convergence in probability:
(x P. and only if (G y())) 3,
(13)
The full theorem, as well as proof, is presented in Appendix A, along with additional remarks and an intuitive sketch. The proof combines the conditional Id-CLT in Assumption with the

---

interpolation of the free-energy in [17]. For strongly-convex losses, one may alternatively use the Lindeberg's 's method as in [16].
In a nutshell, this theorem shows that the multi-modal data generated by any generative neural network is equivalent to a finite mixture of Gaussian in high-dimensions: in other words, a finite mixture of Gaussians leads to the same loss as for data generated by (for instance) cGAN. Since the function & [ R need not be convex, we can take
(T 3) = L'( (Tou), y),
where is an already pretrained neural network. In particular, if the output of all but the last layer of a neural net, we can view V as the averaging procedure for small committee machine.
Note that Corollary depends erucially Assumption 4 (the one-dimensional CLT), which is by no means evident. We discuss the conditions on the weights matrix for which it can be proven in Section 2.4. However, one can observe empirically that the validity of Corollary goes well beyond what can be currently proven. A number of numerical illustrations of this property can be found in the work of [23, 39, 40], who already derived similar (albeit more limited) results using random matrix theory. Additionally. we observed that even with trained GANs, when we observed data through random feature map [46], the Gaussian mixture universality is well obeyed. This scenario is illustrated in Fig. 1, with a concrete example in Fig. 2. Even though we did not prove the one-dimensional CLTT for arbitrary learned matrices, and worked with finite moderate sizes, the realistic data generated by our cGAN behaves extremely closely to those generated by the corresponding Gaussian mixture.
second remark is that the interest of Corollary lies in the fact that it requires only a finite mixture to approximate the loss. Indeed, while we could use the standard approximation results (e.g. the Stone-Wo Weierstrass theorem) approximate the data density arbitrary precision by Gaussian mixtures, this would require diverging number of Gaussian in the mixture. The fact that loss is captured with finite C is key our approach.
2.2 Convergence of expectations for Joint Minimization and Sampling
Our next result establishes a general relationship between the differentiability of the limit of expected training errors free energies for empirical risk minimization free energies for sampling and the universality of expectations of a given function of a set of parameters arising from multiple objectives. As motivating example, consider the uncertainty quantification in Section that uses both Bayesian and ERM estimators [19, 24]. The parameters Ô and O Bayes are obtained through empirical risk minimization and posterior sampling respectively on the same sequence of training data. In general, the inputs used in different objectives could be different but have some correlation structure. In the setup of ensembling (Equation 7), they are correlated through the feature mapping
2j at; (m) pm (2.). In light of these considerations, we present the following general setup: Consider sequence of M risks:
(m)) (o; X (m), (m))y(mm) (m))
1 Σ em ① n
(m) x; yi (m)) + rim (+), m [M]
(14)
with possibly different losses, regularizers and datasets. For simplicity, we assume that the objectives
are defined on parameters having the same dimension E B We aim to minimize Mj of them:
(X) E arg min R(()) (O; (ma), (m)), @es(")
m E [Mi]
(15)
and the M2 remaining parameters are independently sampled from a family of Gibbs distributions:
(m) Pm (0) x exp (-Bm, (m))
(0: x (m)
y (m))) ddm (9), m E [M, + M.),
(16)
where M M2 M2. The joint distribution of the x; (z) (1)
....., x (M) is assumed be
mixture of the form (I). However, we assume that the labels y:" only depend on the vectors æ; (m))
(m)
y (m) (x(m)) (")) (m) (m)
a
(m))
c.).
(17)
The equivalent Gaussian inputs gi (g] ..... .....g.t A and their labels GG) are defined as in (8).
7

---

Statistical metric and free energy Our goal is to study statistical metrics for some function : Rxxxxp + R of the form h(@ θ (M) For instance, the metric h could be the
population risk (a.k.a. generalization error), or some overlap between O and ① We define the following coupling free energy function:
(O Mi), x,y)
- - log
f
(()
e ((O))
d
(18)
n
where P(M++ +) denotes the product measure of the Pm defined (16). This gives rise to the following joint objective:
Mi). x,9)
Σ R.th) ( (O(m) x x(m) y (m) ( MiJ, X,)).
(m)
(19)
In particular, when 0 we have fn.o and the problem reduces to the joint minimization problem in (15). Our first result concerns the universality of the minimum of the above problem:
Proposition (Universality for joint minimization and sampling). Under Assumptions 1 1-4, for any > and bounded Lipschitz function + R + R, and denoting (X. = min R (O; y)):
lim 000
E $
(Ri, (X,
y(x)) E T (R......
The proof uses reduction to Corollary 2, and can be found App. A.5. The next result concerns value of h at the minimizers point (é(")...., (())) We make the following additional assumptions: Assumption 5 (Differentiable Limit). There exists a neighhorhood of 0 such that the function (s) E (G. converges pointwise a function ((s) that is differentiable O.
The above assumption stems from the convexity based argument used to prove Theorem 4.
For fixed realization of the dataset X, we denote by (h(@()) . ((M))) x the expected value of h when (+(1), ..... e (Mi)) are obtained through the minimization of (15) and (O.,,+))...,,,,M)))) are sampled according to the Boltzmann distributions (16).
...., (M)
Assumption 6. With high probability on X, G, the value (h(O ... (1.00)))))) (resp. the same for G.) is independent from the chosen minimizers (15).
The above assumption is motivated by the fact that commonly non-convex problems contain minima exhibiting a specific symmetry. For example, all the global minima for a two-layer neural network are permutation invariant. Assumption 6 reflects that the quantity h respects these symmetries by taking the same value each global minimum. This can be replaced by the stronger condition of unique minimizer. Then the following holds:
Theorem 4. Under Assumptions 1-6, we have:
(((())).....M -E ( (0t),..... (00)))).MMMM))
)(M))
11 0,
lim
(20)
N,P-+00
Proof Sketch:
Our proof relies on the observation that yn (8) is concave function of s. Further:
% (0) F [ (0 e (M)))
(21)
This allows us to leverage a result of convex analysis relating the convergence of a sequence of convex concave functions the convergence of the corresponding derivatives, bypassing the more involved probabilistic arguments in [16, 17]. Our approach also generalizes in straightforward manner to the setup of multiple objectives.
The above result shows that the expected value of (0(1) ( for multi-modal data satisfying the 1d CLT equivalent that of mixture of Gaussians. The full theorem is presented and proven in Appendix A.

---

2.3 Universal Weak Convergence
Theorem 4 provides general framework for proving the equivalence of arbitrary functions of parameters obtained by minimization/sampiinn on a given mixture dataset and the equivalent gaussian mixture distribution. However, it relies on the assumption of differentiable limit of the free energy (Assumption 5). If the assumption holds for sequence of functions belonging to dense subsets of weak
particular classes of functions, allows us to prove convergence of minimizers themselves, sense. We illustrate this through simple setup considered in [12], which precisely characterized the asymptotic distribution of the minimizers of empirical risk with GMM data in the strictly convex case. Consider the following setup:
(Wx, 6xx== arg min W.b
Wx
+b.31) Ar(W), vd
(22)
where W E Riilidd E IR CI and yr E RRII is the one-hot encoding of the class index C₁. We make the following assumptions:
Assumption 7. All of the covariance matrices Ee are diagonal, with stricily positive eigen- values (de,) and there exists constant M 0 such that for any e E we have Jc,i S and || S M.
Secondly, since we aim at obtaining a result on the weak convergence of the estimators, we assume the same weak convergence for the means and covariances, and that the regularization only depends on the empirical measure of W.
Assumption 8. The empirical distribution of the pee and Ee converges weakly as follows:
1
EI
II 8( Vdpe,))oloc-o σe,)) OEC
C
p(ar,p)
(23)
d=+00
Assumption 9. The regularizer r(-) is a pseudo-Lipshitz, function of finite-order having the following
form: r(W)= = Ei= (W.),, some convex, differentiable function v., : :R + R. This includes, in particular the squared reqularization r(W) = Σ W?
We briefly comment on the choice of the above assumptions. The boundedness of the E, and J in Assumption 7 guarantees that we are in case covered both by [12] and by the assumptions of Theorem 4. The diagonal property of the E. in 7, as well as the joint convergence 8, ensure that we view the minimization problem 22 ensures that W*, 6" converge towards well-defined limit. Finally, the separability assumption on r in assumption responds to the fact that we aim for result on the empirical coordinate distribution of W* ,
Assumption
Under these conditions, the joint empirical measure of the minimizers and of the data moments converges weakly to a fixed limit, indepdendent of the data dita-disttibution:
Theorem 5. Assume that conditions 1-9 hold, and further that the function E( y) + +r()) is convex, coercive and differentiable. Then, for any bounded- hounded-Lipschitz function: & : R3CC| + R. we have:
E
(((((()ax))) (Hes)eec, (des)cec)
Eg (.0(,,,,)))
(24)
m/d=a>0
where P is a measure on 33|0| that is determined by the so-called replica equations.
Proof Sketch The proof starts with the observation that the nonlinear system of (replica) equations in [12] describes the joint-empirical measure of the parameters, means and covariances of the mixtures in a self-consistent manner. Furthermore, for (W) having bounded second derivatives, the perturbation term (W) can be absorbed into the regularizer. We then utilize topological and analytical arguments relate the weak convergence to the differentiability Assumption 5 for functions that can be expressed as expectations w.r.t the joint empirical measure in 24. More details can be found in Appendix A.8.
In particular, the above result implies the universality of the overlaps of the minimizers with means, covariances, well as their geometrical properties such as L norms.

---

2.4 One-dimensional CLT for Random Features
We finally show conditional one- dimensional CLT for random features map applied to a mixture of gaussians, in the vein of those shown in [14, 16, 17]. Concretely, we consider the following setup:
8; o(Fz),
Zi Σ E.) LEC
(25)
where the feature matrix F Rexd has i..d (0, 1/)) entries. This setup is much permissive than the ones in [16, 17], that restrict the samples z to standard normal vectors. However, we do require some technical assumptions:
Assumption 10. The activation function is thrice differentiable, with ||o S M for some M and we have (0.1) [g]] = 0. Additionally, the cluster means and covariances of x satisfy for all E C || S M,
- lo S M for some constant M > O.
We also place ourselves in the proportional regime, i.e. regime where p/d E (- for some 7 0. For simplicity, will consider the case 1; and the constraint set Sp follows:
Sp = (0 E Rd ||0|2 - R, |00| S Cp-")
(26)
for given 17 > O. We show in the appendix the following theorem:
Theorem 6. Under Assumption 10, and with high probubility on the feature matrix F, the data X satisfy the concentration assumption 2, as well as the one-dimensional CLT of Assumption 4. Consequently, the results of Theorems I and 4 apply X and their Gaussian equivalent G.
Proof Sketch Our proof proceeds by defining the following neuron-wise activation functions:
T2,e(u) + f. p.).
(27)
We subsequently control the effects of the means, covariances and the dimensions of the inputs to prove result analogous to the miedimensioonll CLT for random features in [16, 17, 14). While we prove the above result for random weights, we note, however that the non-asymptotic results in [16, 14] also hold for deterministic matrices satisfying approximate orthogonality conditions. Therefore, we expect the one-dimensional CLT to approximately hold for a much larger class of feature maps. Finally, we also note that the above extension the noeddimensional CLT mixture of gaussians also provides proof for the asymptotic error for random features in [1]].
Conclusions We demonstrate the universality of the Gaussian mixture assumption in high- dimension for various machine learning tasks such as empirical risk minimization, sampling and ensembling, in variety of settings including random features or GAN generated data. We also show that universality holds for large class of functions, and provide a weak convergence theorem. These results, believe, vindicate the classical theoretical line of works on the Gaussian mixture design. We hope that our results will stimulate further research in this area. We also believe it crucial to understand the limitations of extended universality framework, for instance in the cases of data with low-dimensional structure or sparsity.
Acknowledgements
We acknowledge funding from the ERC under the European Union's Horizon 2020 Research and Innovation Program Grant Agreement 714608-SMI le, as well as by the Swiss National Science Foundation grant SNES OperaGOST, 100021200390.
10

---

References
[1] David Donoho and Jared Tanner. Observed universality of phase transitions in high-dimensional geometry, with implications for modern data analysis and signal processing. Philosophi- cal Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 367(1996))4273-4299 2009.
[2] Satish Babu Korada and Andrea Montanari. Applications of the lindeberg principle in commu- nications and statistical learning. IEEE Transactions Information Theory, 57(4):2440 2450224400-00 2011.
[3] Hatef Monajemi, Sina Jafarpour, Matan Gavish, null null, David L. Donoho, Sivaram Am- bikasaran, Sergio Bacallado, Dinesh Bharadia, Yuxin Chen, Young Choi, Mainak Chowdhury, Soham Chowdhury, Anil Damle, Will Fithian, Georges Goetz, Logan Grosenick, Sam Gross, Gage Hills, Michael Hornstein, Milinda Lakkam, Jason Lee, Jian Li, Linxi Liu, Carlos Sing- Long, Mike Marx, Akshay Mittal, Hatef Monajemi, Albert No, Reza Omrani, Leonid Pekelis, Junjie Qin, Kevin Raines, Ernest Ryu, Andrew Saxe, Dai Shi, Keith Siilats, David Strauss, Gary Tang, Chaojun Wang. Zoey Zhou, and Zhen Zhu. Deterministic matrices matching the compressed sensing phase transitions of gaussian random matrices. Proceedings of the National Academy of Sciences, 110(4):1811-1188 2013.
[4] Emmanuel Candès, Pragya Sur, et al. The phase transition for the existence the maximum likelihood estimate in high-dimensional logistic regression. The Annals of Statistics, 48(1):27- 42, 2020.
[5] Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117748):30063330077) 2020.
[6] Xiaoyi Mai and Zhenyu Liao. High dimensional classification via empirical risk minimization: Improvements and optimality. arXiv: /905. 13742, 2019.
[7] Francesca Mignacco, Florent Krzakala, Yue Lu, Pierfrancesco Urbani, and Lenka Zdeborova. The role of regularization in classification of high-dimensional noisy gaussian mixture. In International Conference on Machine Learning, pages 6874-6883. PMLR, 2020.
[8] Hossein Taheri, Ramtin Pedarsani, and Christos Thrampoulidis. Optimality of least-squares for classification in gaussian-mixture models. In 2020 IEEE International Symposium on Information Theory (ISIT), pages
IEEE, 2020.
[9] Ganesh Ramachandra Kini and Christos Thrampoulidis. Phase transitions for one-vs-one and one-vs-all linear separability in multiclass gaussian mixtures. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4020- 4024. IEEE, 2021.
[10] Ke Wang and Christos Thrampoulidis. Benign overfitting in binary classification of gaussian mixtures. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 0030-4034. IEEE, 2021.
[11] Maria Refinetti, Sebastian Goldt, Florent Krzakala, and Lenka Zdeborová. Classifying high- dimensional gaussian mixtures: Where kernel methods fail and neural networks succeed. In International Conference Machine Learning, pages 8936-8947. PMLR, 2021.
[12] Bruno Loureiro, Gabriele Sicuro, Cedric Gerbelot, Alessandro Pacco, Florent Krzakala, and Lenka Zdeborová. Learning gaussian mixtures with generalized linear models: Precise asymp- toties in high-dimensions. In M. Ranzato, A. Beygelzimer, Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neurul Information Processing Systems, volume 34, pages 10144-10157. Curran Associates, Inc., 2021.
[13] Sebastian Goldt, Mare Mézard, Florent Krzakala, and Lenka Zdeborová. Modelling the influence of data structure on learning in neural networks: the hidden manifold model. Physical Review X, 10:041044, 2019.
11

---

[14] Sebastian Goldt, Bruno Loureiro, Galen Reeves, Florent Krzakala, Mare Mezard, and Lenka Zdeborova. The gaussian equivalence of generative models for learning with shallow neural networks. In Joan Bruna, Jan Hesthaven, and Lenka Zdeborova, editors, Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, volume 145 of Proceedings of Machine Learning Research, pages 426-471, PMLR, 16-19 Aug 2022.
[15] Noureddine EI Karoui. The spectrum of kernel random matrices. 38(1):1-50, 2010.
The Annals of Statistics,
[16] Hong Hu and Yue M. Lu. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory, pages 1-1, 2022.
[17] Andrea Montanari and Basil N. Saeed. Universality of empirical risk minimization. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pages 4310-4312. PMLR, 02-05 Jul 2022.
[18] Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mezard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with teacher-student model. Advances in Neural Information Processing Systems, 34, 2021.
[19] Lucas Clarté, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Theoretical characteri- zation of uncertainty in high-dimensional linear classification. arXiv: 2202.03295, 2022.
[20] Bruno Loureiro, Cedric Gerbelot, Maria Refinetti, Gabriele Sicuro, and Florent Krzakala. Fluctuations, bias, variance & ensemble of learners: Exact asymptotics for convex losses in high-dimension. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 14283-14314. PMLR, 17-23 Jul 2022.
[21] Federica Gerace, Florent Krzakala, Bruno Loureiro, Ludovic Stephan, and Lenka Zdeborová. Gaussian universality of linear classifiers with random labels in high-dimension. arXiv: 2205.13303, 2022.
[22] Luca Pesce, Florent Krzakala, Bruno Loureiro, and Ludovic Stephan. Are gaussian data all you need? extents and limits of universality nigh-dimensional generalized linear estimation. arXiv preprint arXiv:2302.08993, 2023.
[23] Mohamed EI Amine Seddik, Cosme Louart, Mohamed Tamaazousti, and Romain Couillet. Random matrix theory proves that deep learning representations of GAN-data behave as Gaussian mixtures. In Proceedings of the 37th International Conference on Machine Learning, ICML'20, 20, pages 8573-8582. JML,.org, July 2020.
[24] Lucas Clarté, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. A study of uncertainty quantification in overparametrized high-dimensional models. arXiv: 2210. 12760, 2022.
[25] Terence Tao and Van Vu. Random matrices: Universality of local eigenvalue statistics. Acta Mathematica, 206(1):127 - 204, 2011.
[26] Terence Tao and Van Vu. Random matrices: universal properties of cigenvectors. Random Matrices: Theory and Applications, 01(01):115000, 2012.
[27] Ashkan Panahi and Babak Hassibi. A universal analysis of large-scale regularized least squares solutions. In Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[28] Andrea Montanari and Phan-Minh Nguyen. Universality of the elastic net error. In 2017 IEEE International Symposium on Information Theory (ISTT), pages 2338-2349 2017.
[29] Ehsan Abbasi, Fariborz Salehi, and Babak Hassibi. Universality in learning from linear measurements. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
12

---

[30] Alia Abbara. Antoine Baker, Florent Krzakala, and Lenka Zdeborová. On the universality of noiseless linear estimation with respect the measurement matrix. Journal of Physics A: Mathematical and Theoretical, 53(16):16400) mar 2020.
[31] Rishabh Dudeja, Subhabrata Sen, and Yue M. Lu. Spectral universality of regularized linear regression with nearly deterministic sensing matrices. arXiv: 2208.02753. 2022.
[32] Yue M. Lu and Horn-TTzer Yau. An equivalence principle for the spectrum of random inner- product kernel matrices. arXiv: 2205.06308 2022.
[33] Theodor Misiakiewiez. Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regression. arXiv: 2204. 10425, 2022.
[34] Andrea Montanari, Feng Ruan, Youngtak Sohn, and Jun Yan. The generalization error of max-margin linear classifiers: High-dimensional asymptotics in the overparametrized regime. arXiv: 1911.01544, 2019.
[35] Federica Gerace, Bruno Loureiro, Florent Krzakala, Mare Mézard, and Lenka Zdeborová. Generalisation error in learning with random features and the hidden manifold model. In International Conference Machine Learning, pages 3452-3462. PMLR, 2020.
[36] Oussama Dhifallah and Yue M. Lu. A precise performance analysis of learning with random features. arXiv: 2008. 1 1904, 2020.
[37] Tengyuan Liang and Pragya Sur. A precise high-dimensional asymptotic theory for boosting and minimum- -norm interpolated classifiers. The Annals of Statistics, 50(3):1669 1695, 2022.
[38] Dominik Schröder, Hugo Cui, Daniil Dmitriev, and Bruno Loureiro. Deterministic equivalent and error universality of deep random features learning. arXiv: 2302.00401, 2023.
[39] Cosme Louart, Zhenyu Liao, and Romain Couillet. A random matrix approach to neural networks. The Annals of Applied Probability, 28(2):1 190-1248, 2018.
[40] Mohamed El Amine Seddik, Cosme Louart, Romain Couillet. and Mohamed Tamaazousti. The unexpected deterministic and universal behavior of large softmax classifiers. In International Conference on Artificial Intelligence and Statistics, pages 1045-1053. PMLR, 2021.
[41] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv: 1411.1784, 2014.
[42] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable pre- dictive uncertainty estimation using deep ensembles. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and Garnett, editors, Advances in Neural Informa- lion Processing Systems, volume 30. Curran Associates, Inc., 2017.
[43] Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d' A'scolii Giulio Biroli, Clément Hongler, and Matthieu Wyart. Scaling description of generalization with number of parameters in deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2020(2):023401, feb 2020.
[44] Stéphane Ascoli, Maria Refinetti, Giulio Biroli, and Florent Krzakala. Double trouble in double descent: Bias and variance (s) in the lazy regime. In International Conference Machine Learning, pages 2290-2290. PMLR, 2020.
[45] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv: 1708. 0708007747, 2017.
[46] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In J. Platt,
D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007.
[47] Francesco Guerra. Broken replica symmetry bounds in the mean field spin glass model. Communications in mathematical physics, 233(1):1-12, 2003.
13

---

[48] R Tyrrell Rockafellar. Convex analysis, volume 18. Princeton university press, 1970.
[49] Michael Celentano, Andrea Montanari, and Yuting Wei. The lasso with general gaussian designs with applications to hypothesis testing. arXiv: 2007.13716, 2020.
[50] J. W. Lindeberg. Eine neue Herleitung des Exponentialgesetzes in der Wahnscheinlichkeissrcchh nung. Mathematische Zeitschrift, 15(1):211-22) December 1922.
[51] Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
[52] S. G. Bobkov. On concentration of distributions of random weighted sums. The Annals of Probability, 31(1):195-215, January 2003.
[53] Michel Ledoux. The concentration of measure phenomenon, volume 89 of Math. Surv. Monogr. American Mathematical Society (AMS), Providence, RI, 2001.
[54] Karl Pearson. On lines and planes of closest fit to systems of points in space. The London, Edin- burgh, and Dublin Philosophical Magazine and Journal of Science, 2(15):55--572, November 1901.
[55] Mark A. Kramer. Nonlinear principal component analysis using autoassociative neural networks. AIChE Journal, 37(2):233-245 1991.
[56] Elena Facco, Maria d' Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimension of datasets by minimal neighhorhood information. Scientific Reports, 7(1):12140, 2017.
[57] Stefano Spigler, Mario Geiger, and Matthieu Wyart. Asymptotic learning curves of kernel methods: empirical data versus teacher-student paradigm. Journal of Statistical Mechanics: Theory and Experiment, 2020(12):124000) dee 2020.
[58) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
[59] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv: 1312.6114. 2013.
[60] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1530-1538, Lille, France, 07-09 Jul 2015. PMLR.
[61] Peter I. Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalizee margin bounds for neural networks. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, vol- ume 30. Curran Associates, Inc., 2017.
[62] Jens Behrmann, Paul Vicol, Kuan-Chiel Wang, Roger Grosse, and Joern-Henrik Jacobsen. Understanding and mitigating exploding inverses in invertible neural networks. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 1792-1800. PMLR, 13-15 Apr 2021.
[63] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
14

---

[64] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[65] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. arXiv: 1606.03657, 2016.
[66] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson. K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson. C Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E, A. Quintero, Charles Harris, Anne M. Archibald, Antônio Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272 2020.
15


Reviews:
[{"review_id": "tXyCIJDQ8C", "reviewer": "Reviewer_JkJQ", "summary": "The paper proves that the performance and other quantities of interest output by a number of algorithms in the generalized linear models class are the same as if the data were replaced by Gaussians with the same means and covariance matrices. This universality result suggests that the performance of this entire class of algorithms (generalized linear models) depends only on the first- and second-order moments of the data. This theoretical study justifies the use of a Gaussian mixture model in the study of generalized linear algorithms.\nThe particularity of the study consists in proving this important phenomenon rigorously even on restrictive assumptions.", "questions": "1- What is the intuitive meaning of Ass 4 and Ass 5? How restrictive they are?\n\n2- Is the universality behave the same whatever the type of structure considered (highly sparse data, time series)? Do some experiments could be provided on some of these datasets? Are these some cases of failure of the universality noticed in experiments?", "limitations": "The main statement of the paper seems quite strong even though the mathematics seems rigorous. In this case, it would be really important to provide a wide set of experiments over several datasets (with different structure) and several linear algorithms. This would be more reassuring than checking the calculations line by line for the reader.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1-The question of the universality of Gaussian mixtures in the analysis of the performance of generalized linear algorithms is a subject that has long remained empirical and has known very few rigorous and clear answers. This paper by providing a rigorous answer, makes it possible to make an important step in the resolution of this question.\n\n2- The paper is generally well written with very few spelling mistakes and consistency in notations. The motivation and structure of the paper seems to be quite clear.\n\n3- Although not having checked the theoretical calculations, the approach seems to be rigorous and well conducted. The results are consistent.", "weaknesses": "1- Assumption 4 and Assumption 5 need to be better explained in terms of their intuitive meaning and how they are binding. Although there is some discussion that it is very restrictive and experiments tend to suggest that the results should be valid beyond that, it would be important to provide more intuitive explanations of what it means and why it is restrictive.\n\n2- In the experiment shown in Figure 2, there is a systematic bias between the performance of the CGAN and the GMM. It would be interesting to discuss this aspect, providing explanations if necessary. It would have been useful to show the limits of universality in a general way (even in the appendix). Are there cases where this holds less well, and this would be due to what phenomenon, or are there algorithms that are more resistant to this universality in the class of linear algorithms?\n\n3- Although I appreciate the authors' effort to provide experiments on real data, it would have been interesting to do so with different data structures and slightly more challenging data such as time series and language classification. In particular, we could ask the question of universality for highly structured data such as very sparse data (like speech encoding) or time series. It seems somewhat counter-intuitive that for these highly structured data, the mean and variance alone are sufficient to characterize performance. Recently, in random matrix theory, universality has been focused on a class of random vectors that come from the concentration of the measure and it is not obvious that highly sparse data are concentrated and therefore that universality holds for this type of data, even through the prism of a linear model.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and suggestions which will help improving this work. Below, we address specific comments and questions:\n\n> Assumption 4 and Assumption 5 need to be better explained in terms of their intuitive meaning and how they are binding. Although there is some discussion that it is very restrictive and experiments tend to suggest that the results should be valid beyond that, it would be important to provide more intuitive explanations of what it means and why it is restrictive.\n\n> What is the intuitive meaning of Ass 4 and Ass 5? How restrictive they are?\n\nAssumptions 4 and 5 conceptually play a very different role in the proof of Theorem 4.\n\nOn one hand, Assumption 5 is a technical restriction, mainly stemming from the convexity based argument used to prove Theorem 4. We expect it to work in a fairly generic setting, and we show it also holds in the setting of Theorem 5.\n\nOn the other hand, Assumption 4 plays a crucial role. As explained in Appendix A.2, the key idea is to split the proof of universality in two steps. First, one shows that asymptotic universality of an observable $h$ can be reduced to the proof of a one-dimensional CLT. Second, one proves this CLT holds for the particular class of features of interest.\n\nThis proof scheme streamlines universality proofs. Our work provides a general proof of the first step (Theorem 4) conditioned on the second (Assumption 4), later showing that Assumption 4 holds for some natural feature maps of interest, e.g. random features applied to a Gaussian mixture (Theorem 6). However, Theorem 4 holds independently, cutting in half the work of proving universality for other feature maps.\n\nWe will clarify the relative importance of each assumption in the revised version.\n\n\n\n\n\n> 2- In the experiment shown in Figure 2, there is a systematic bias between the performance of the CGAN and the GMM. It would be interesting to discuss this aspect, providing explanations if necessary. It would have been useful to show the limits of universality in a general way (even in the appendix). Are there cases where this holds less well, and this would be due to what phenomenon, or are there algorithms that are more resistant to this universality in the class of linear algorithms?\n}}\n\nA possible explanation for the systematic bias in Figure 2 is that performing classification using linear classifiers on GMM data with the same mean, covariance as the cGAN data is ``easier\" since it only requires capturing the conditional first two moments. Indeed, we expect assumption 4 to only hold approximately for real data. Additionally, our results are asymptotic and can suffer from finite size effects which usually translate to a systematic bias.\n\nWe agree with the reviewer that understanding the limitations of universality is an interesting and important problem. We will add further discussions about it in the final version.\n\n\n> 3- Although I appreciate the authors' effort to provide experiments on real data, it would have been interesting to do so with different data structures and slightly more challenging data such as time series and language classification. In particular, we could ask the question of universality for highly structured data such as very sparse data (like speech encoding) or time series. It seems somewhat counter-intuitive that for these highly structured data, the mean and variance alone are sufficient to characterize performance. Recently, in random matrix theory, universality has been focused on a class of random vectors that come from the concentration of the measure and it is not obvious that highly sparse data are concentrated and therefore that universality holds for this type of data, even through the prism of a linear model.\n\n> 2- Is the universality behave the same whatever the type of structure considered (highly sparse data, time series)? Do some experiments could be provided on some of these datasets? Are these some cases of failure of the universality noticed in experiments?\n\n> The main statement of the paper seems quite strong even though the mathematics seems rigorous. In this case, it would be really important to provide a wide set of experiments over several datasets (with different structure) and several linear algorithms. This would be more reassuring than checking the calculations line by line for the reader.\n\nOur universality results are as strong as the one-dimensional CLT. In particular, since one-dimensional projections of (very) sparse data are not Gaussian, we expect universality to break down in this case. For time series, given the high correlation between data points, the chosen directions of $\\mathcal S_p$ should probably satisfy strong conditions to ensure that Assumptions 2 and 4 hold. \n\nOn the other hand, positive examples are already abound in the literature. For instance, [11-13, 19-21] all compare the results of their exact Gaussian predictions in various settings with the performances on real-world datasets, and find a very good agreement between the two. Our work aims at providing a general framework to justify those previous results, hence we only included an illustrative example.\n\nFinally, for a explicit synthetic example breaking universality, we refer the reviewer to the systematic investigation by Goldt and Ingrosso (*Data-driven emergence of convolutional structure in neural networks*, PNAS, 2022)."}, {"review_id": "ALVj1Og1V6", "reviewer": "Reviewer_KxLS", "summary": "The submission studies the universality phenomena for general mixture model. The authors show the universality of empirical risk minimization and sampling and several extensions.", "questions": "Could the authors explain in more details the technical differences between their submission and the previous analyses?", "limitations": "No", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The authors give good introduction and literature review.", "weaknesses": "1. The main contribution lies in the technical part. However, it lacks novelty given the results in Hu and Lu 2022, Montanari and Saeed 2022. The assumptions 1-4 and Section 2.1 are almost identical to the main results in Montanari and Saeed 2022 without mentioning or comparisons at the beginning. The proofs for Theorem 1 and Corollary 2 only recap the previous results in the appendix. Section 2.2 gives some simple extensions of those results in Montanari and Saeed 2022 to multiple risk case.\n\n2. The paper has plenty of inconsistencies and lack of explanations in the course of writing. For example, line 109, the notation of \\eta is not consistent to equation (2). The three applications in Section 1 are hard to follow given the model setting. \n\n3. Some of the notations are not proper: For example, line 101, it would be better to write \\hat{y}=F(\\hat{\\Theta}^T x) to denote estimated parameters. I suggest the authors modify the notations in the whole paper to fit their own topic and interests instead of using those in Montanari and Saeed 2022.\n\n4. Also, it would be better to explain the notations immediately after introducing.\n\n5. The statements of the main theorems and proofs in the appendix is lack of polish. It would be better to write remarks after the theorems to improve the readability.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and suggestions which will help improving this work. Below, we address specific comments and questions:\n\n> 1. The main contribution lies in the technical part. However, it lacks novelty given the results in Hu and Lu 2022, Montanari and Saeed 2022. The assumptions 1-4 and Section 2.1 are almost identical to the main results in Montanari and Saeed 2022 without mentioning or comparisons at the beginning. The proofs for Theorem 1 and Corollary 2 only recap the previous results in the appendix. Section 2.2 gives some simple extensions of those results in Montanari and Saeed 2022 to multiple risk case.\n\n> Could the authors explain in more details the technical differences between their submission and the previous analyses?\n\nWe refer the reviewer to the general response regarding a summary of the technical and conceptual novelty of our work. To address their specific points:\n\n- Section 2.1, and our Assumptions 1-4, generalize Assumptions 1-5 and Theorem 1 of [17] (Montanari and Saeed (2022)) from a single Gaussian to the Gaussian mixture setting as in eqs. (1),(8). The main differences are outlined in lines 633-645 in Appendix A. The most important distinction is in point (ii), where we allow the labeling function to depend on the mixture label of each datapoint.\nAlthough our proof omits the technical details identical to the ones in [17](Montanari and Saeed (2022)), the proof does require considerable adaptations. \n   In particular, we introduce an additional dependency by coupling the mixture labels of the datapoints $x_i$ and their Gaussian counterparts, that we handle by controlling the terms corresponding to each mixture separately.\n   This is detailed in Section A.4 of the appendix (lines 627-632 and 662-664).\n- About section 2.2. What is considered \"simple\" is of course in the eye of the beholder. We respectfully disagree: constructing the reduction from multiple parameters, such as required in Bayesian inference and ensembling, to a single equivalent Boltzmann distribution on a lifted parameter space (Equation (49) in the Appendix) is technically non-trivial from our point of view. This leads to our theorem 4 which has a much broader applicability than the ones in previous literature. We refer to concrete such cases in the common answer. Furthermore, as we mentioned in the general response, our proof of Theorem 4 relies on a novel convexity-based argument absent in earlier works.\n\n> 2. The paper has plenty of inconsistencies and lack of explanations in the course of writing. For example, line 109, the notation of $\\eta$ is not consistent to equation (2). The three applications in Section 1 are hard to follow given the model setting.\n\n> 3. Some of the notations are not proper: For example, line 101, it would be better to write $\\hat{y}=F(\\hat{\\Theta}^T x)$ to denote estimated parameters. I suggest the authors modify the notations in the whole paper to fit their own topic and interests instead of using those in Montanari and Saeed 2022.\n\nWe will modify some of the notation in our paper to improve readability and consistency. \nFor clarification, the definition of $\\hat{y}$ in line 101 is applicable for any $\\Theta$ and is used to define the empirical risk objective in line 103. The estimated parameters $\\hat{\\Theta}$ are introduced only later in equation 4. \n\n> 4. Also, it would be better to explain the notations immediately after introducing.\n\n> 5. The statements of the main theorems and proofs in the appendix is lack of polish. It would be better to write remarks after the theorems to improve the readability.\n\nWe thank the reviewer for the suggestions. We will add explanations for the notations and remarks after the theorems in the final version."}, {"review_id": "zz9KJVBpF7", "reviewer": "Reviewer_Wbxr", "summary": "In this paper, the authors investigated the joint statistical behaviors of a set of genealized linear estimators obtained from either empirical risk minimization or sampling, for data and labels drawn from a mixture model, in the high-dimensional regime where where the data dimension $p$ and the sample size $n$ both go to infinity.\n\nIn particular, they establish sufficient conditions under which the asymptotic joint statistics of this family depend only on the means and covariances of the mixture model.\n\nThe obtained results extend a few previous efforts: for instance, provide alternative proof to the fact that Gaussian mixture observed through a random feature map is akin to a Gaussian mixture [11, 12, 21, 22, 16]; and that data generated by conditional Generative Adversarial Networks (cGAN) are like Gaussian mixture [23], etc.\n\nSome theoretical results are presented in a not-so-precise manner and some clarifications are needed. The paper is in general well written otherwise.\n", "questions": "I have the following minor comments or questions:\n\n1. Assumption 2 indeed describes the type of (mixture) data distribution under study, some more explanation and/or comparison would be helpful here. \n2. Corollary 2 seems to be a corollary of Theorem 1, and if this is the case, it should be announced to hold under \"the same setting of Theorem 1\" or under \"Assumption 1-4\".\n2. Theorem 1 and Corollary 2 hold as long as one takes the limit of $n,p \\to \\infty$? Should some assumption be put on their ratio $p/n$?\n3. last paragraph in page 6 \"this would require a diverging number of Gaussian in the mixture\": would the authors elaborate on this, with some reference or more details for example?\n4. it would be helpful to comment on the necessity and/or importance of Assumption 7-9, otherwise they are a bit difficult to digest and compare.\n5. Also, the dimension $n,p$ are NOT involved in Theorem 5?\n6. as expectations w.r.t the joint empirical measure in 24 -> in (24) or Equation 24.\n7. could the authors elaborate on the replica equations in Theorem 5: can they be numerically evaluated (with ease)? ", "limitations": "This work is theoretical and I do not see any potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The topic focuses on the emerging topic of universality of ML methods in the high-dimensional regime.\nThe obtained theoretical results are strong and improve prior arts in many ways.\nI did not check the detailed proofs, but the proof sketch look compelling.", "weaknesses": "There are a lot of assumptions in the paper, and it somewhat makes the paper less easy to grasp and compare to existing works, some more explanations are needed.\nSome clarifications are needed to better understand the statements in the paper, as well as the major contribution of this paper.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and suggestions which will help improving this work. Below, we address specific comments and questions:\n\n> There are a lot of assumptions in the paper, and it somewhat makes the paper less easy to grasp and compare to existing works, some more explanations are needed. Some clarifications are needed to better understand the statements in the paper, as well as the major contribution of this paper.\n\n>1. Assumption 2 indeed describes the type of (mixture) data distribution under study, some more explanation and/or comparison would be helpful here. \n\nAssumption 2 guarantees that several quantities of interest throughout the proof stay bounded with high probability.\nWe do not consider this assumption as inherently limiting, it should be possible to relax it through the use of a different proof technique. For instance, the bounded norm condition on the means $\\mu_c$ can be removed for coercive losses or with strongly convex regularization, since without loss of generality the constraint set can be restricted to parameters such that $\\theta^\\top \\mu_c$ remains bounded. We will add further details and explanations in the final version.\n\n>2. Corollary 2 seems to be a corollary of Theorem 1, and if this is the case, it should be announced to hold under \"the same setting of Theorem 1\" or under \"Assumption 1-4\".}}\n\nThanks for pointing this out. We will update the statement of Corollary 2. \n\n> 3. Theorem 1 and Corollary 2 hold as long as one takes the limit of $n,p\\to\\infty$? Should some assumption be put on their ratio $n/p$?\n\n> 6. Also, the dimension $n,p$ are NOT involved in Theorem 5? \n\n\nAll of our results hold for $n/p \\rightarrow \\alpha$ for some constant $\\alpha$: this was mentioned in the appendix, but we will emphasize it in the main text. Regarding Theorem 5, this is a typo; $d$ should be replaced by $p$ in the statements of section 2.3.\n\n\n> 4. last paragraph in page 6 \"this would require a diverging number of Gaussian in the mixture\": would the authors elaborate on this, with some reference or more details for example?\n\nWe do not have a proof that a specific family distribution actually requires a diverging number of Gaussians to approximate; however, the Stein-Weierstrass theorem does not entail dimension-independent bounds on the number of Gaussians required, hence those types of results are not enough to ensure that Assumption 4 holds.\n\n> 5. it would be helpful to comment on the necessity and/or importance of Assumption 7-9, otherwise they are a bit difficult to digest and compare.\n\nWe can classify Assumptions 7-9 into three categories:\n- The boundedness of the $\\Sigma_c$ and $\\mu$ simply guarantee that we are in a case covered both by [12] and by the assumptions of Theorem 4.\n- The diagonal property of the $\\Sigma_c$, as well as the joint convergence, ensure that we can view the minimization problem (22) as a finite-dimensional approximation of an infinite-dimension problem. This guarantees that there is a well-defined object that $W^\\star, b^\\star$ will converge towards.\n- Finally, the separability assumption on $r$ (which is satisfied by most regularizers in practice, e.g. any $\\lVert \\cdot \\rVert_p^p$) responds to the fact that we aim for a result on the empirical coordinate distribution of $W^\\star, b^\\star$; hence we need to be able to express $r$ only as a function of this empirical distribution.\n\n\n> 8. could the authors elaborate on the replica equations in Theorem 5: can they be numerically evaluated (with ease)?\n\nThe replica equations are borrowed from the work of [12](Loureiro et al. (2021)) which derived them from the state evolution of an Approximate Message Passing scheme for generalized linear models trained on Gaussian mixtures. Despite looking cumbersome, these are scalar equations which are easy to evaluate numerically."}, {"review_id": "guYL4pao3M", "reviewer": "Reviewer_tZtP", "summary": "In classification problems with a finite number of classes under a supervised setting, if the feature vectors are generated from a mixture of distributions conditioned on the class labels and the learning is performed using generalized linear models, a phenomenon called ``Gaussian mixture equivalence (GME)'' simplifying the problem is known to occur. GME is the phenomenon in the high-dimensional limit that when considering a model in which the original conditional distribution of the feature vector is replaced by a Gaussian distribution having the same first- and second-order moments as the original ones, the quantities of interest such as generalization error coincide between the original model with the non-trivial class-conditional distribution and the reduced model with the Gaussian distribution. This paper rigorously proves this GME actually occurs under some (partly mild, partly strong) conditions by extending the method of reference [17] to the case involving multiple sets of parameters, which may be determined from the minimization of different objectives or by sampling from different Gibbs distributions. \n", "questions": "- Is it possible to extend the proof to the case where $\\bm{x}_i$ has correlations among $i \\in [n]$? An interesting example is the orthogonal ensemble under $p>n$. Some discussions are welcome. \n", "limitations": "The authors address their research limitations well. I think there is no concern about the potential societal impact. \n", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Originality & Significance: Although GME has been assumed in previous studies and meaningful results have been shown under the assumption, the results of this paper clarifies the sufficient conditions under which GME can actually occur by the rigorous proof. This provides a mathematical foundation for attempts to understand complicated machine learning/statistical models' behavior through the analysis of simple Gaussian mixture models. The originality and the significance are thus high. \n\n\nClarity & Quality: The main text mainly provides the statements about the theorems, assumptions, and their consequences/applicabilities, and most of the proof details, even including the intuitive arguments, are deferred to the appendix. The statements are also not easy to understand, because many things are explained in the limited pages and the relationships among the theorems, assumptions, and consequences/applicabilities are not evident. The distinction between what has been proven in previous studies and what has been reiterated here is also not clear. Hence the clarity is not high. Nevertheless, the argument can be understood if one reads the manuscript carefully, and considering the importance of what is proven, also with the nice numerical experiment completely supporting the proposition, it is considered that the quality is high.\n\n\nHere I enumerate some typos found and uneasy expressions for better clarity:\n\nLine 162: The sentence is not easy to understand.\n\nBelow Line 200: The quantity $\\ell'$ is not explained.\n\nLine 225: be be -> be\n\nLine 246: Theorem 2 -> Corollary 2\n\nLine 262: a result convex -> a result of convex\n\nLine 305,325: gaussian -> Gaussian\n", "weaknesses": "- Proof technicalities seem to much rely on [17], and there does not appear to have been much technical progress when extending it to the current case. \n\n- Assumption 4 is the key assumption in the proof but when it does hold is not clear. Maybe this is the difficult point of the problem, and hence some more explanations in words are welcome. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and suggestions which will help improving this work. Below, we address specific comments and questions:\n\n> Proof technicalities seem to much rely on [17], and there does not appear to have been much technical progress when extending it to the current case.\n\nWe refer to the general response regarding the technical and conceptual novelty of our work.\n\n> Assumption 4 is the key assumption in the proof but when it does hold is not clear. Maybe this is the difficult point of the problem, and hence some more explanations in words are welcome.\n\nAs discussed in the reply to *Reviewer JkJQ* and explained in Appendix A.2, the key idea is to split the proof of universality in two steps. First, one shows that asymptotic universality of an observable $h$ can be reduced to the proof of a one-dimensional CLT. Second, one proves this CLT holds for the particular class of features of interest.\n\nThis proof scheme streamlines universality proofs. Our work provides a general proof of the first step (Theorem 4) conditioned on the second (Assumption 4), later showing that Assumption 4 holds for some natural feature maps of interest, e.g. random features applied to a Gaussian mixture (Theorem 6). However, Theorem 4 holds independently, cutting in half the work of proving universality for other feature maps.\n\nWe will clarify the importance of this assumption in the revised version.\n\n> Is it possible to extend the proof to the case where $x_{i}$ has correlations among $i\\in[n]$? An interesting example is the orthogonal ensemble under $p>n$. Some discussions are welcome.\n\nThis is a very interesting question ! On one hand, the iid property of the $x_i$ is crucial to ensure that eq. (34) is an iid sum, and hence that we can bound each term separately; extending it to correlated data requires a major innovation in the proof technique.\n\nOn the other hand, assume that $(x_i)\\_{i\\in [n]}$ are random orthogonal vectors. For a given $\\theta$, the random vector $(\\theta^\\top x_i)\\_{i\\in [n]}$ is a random projection of a spherical vector. Since this distribution converges weakly to a Gaussian, and the empirical risk $\\hat{\\mathcal{R}}\\_n(\\theta; x, y(x))$ is a Lipschitz function in $(\\theta^\\top x_i, \\theta^{\\star \\top} x_i)\\_{i\\in [n]}$, we might expect that:\n$$\\hat{\\mathcal{R}}\\_n(\\theta; X, y(X)) \\approx \\hat{\\mathcal R}\\_n(\\theta; G, y(G)),$$\n\nwhere $G$ is a matrix of i.i.d Gaussians, which is the crucial first step to prove universality. This property may also hold for other choices of distribution for $X$, as long as the gaussian equivalent $G$ is chosen accordingly. Hence, even though the proof does not seem to translate that well to the weakly dependent setting, we expect that a suitably defined universality property can also hold."}]

Author rebuttal:
# General Response

We thank the reviewers for their valuable comments and suggestions. We will incorporate them into a revised version of our paper. In our understanding, the main concern across the reviews is the significance and technical novelty of our work with respect with the literature. Below, we address this concern in detail.

## Significance

Our results build on the long line of works on the proof of the Gaussian equivalence (GE) Principle, starting from the work [14] (Goldt et al. (2022)) who proved a 1d CLT theorem for random features of Gaussian data analogous to Assumption 4 in our work. The universality of the empirical risk minimizer for the random features model was then proven by [16](Hu and Lu (2022)) for Gaussian design. [17](Montanari and Saeed (2022)) provided an alternative proof for GE which relaxed the conditions of [16](Hu and Lu (2022)) from strongly convex to almost convex losses.

However, none of these results cover the different settings in which GE was empirically observed to hold. Covering them is precisely the goal of our work. Concretely:

- While [16](Hu and Lu (2022)) (see eq. (8)) and [17](Montanari and Saeed (2022)) (see Ass. 5) only consider the case where the equivalent model is a single-mode Gaussian, our Theorem 1 and Corollary 2 encompasses the much richer Gaussian Mixture model. This allows us to provide theoretical grounding to the works [21](Gerace et al. (2022)), [22](Pesce et al. (2023)), [12](Loureiro et al. (2021)).
- Existing proofs only consider the empirical minimization setting (eq. (1) in [16], eq. (8) in [17]), for a single estimator. Since our Theorem 4 extends to any ensemble of ERM objectives and (non-necessarily optimal) Bayes estimators, our results are applicable to a wide range of problems such as ensembling [20](Loureiro et al. (2022)) or uncertainty quantification [19](Clarté et al. (2022)), [24](Clarté et al. (2022)). 
- Our one-dimensional CLT result extends the existing work from the input being a single Gaussian of zero mean (Theorem 2 of [16] and Corollary 2 of [17]) to the input being a Gaussians mixture with general non-zero mean (our Theorem 6). These kind of random features were for example used in [11](Refinetti et al (2021)) for Gaussian mixture classification. 

Further, our results bridge the works of [38](Louart et al. (2018)); [23,39](Seddik et al. (2020, 2021)) for universality of conditional generative models to Gaussian equivalence of the empirical risk minimizer. 

Hence, our results fill a significant gap in the rigorous understanding of GE which **is not covered by existing works**.

## Technical novelty

Concerning mathematical novelty, our proofs build on many existing techniques but also introduce several new ones that we anticipate to be of independent interest: 

- **Theorem 4**: While the technique of adding a perturbation to the loss is well known, our proof leverages tools from convex analysis to cut short an otherwise lengthy argument from existing works. This approach is more general, enabling the proof for a general class of Gibbs measures involving multiple objectives as well as the empirical convergence of other observables. Previous proofs were restricted to the particular case of the generalization and training errors.
- **Theorem 5**: The starting point of our proof is the non-trivial observation that the nonlinear system of equations in [12](Loureiro et al. (2021)) can be described in a self-consistent manner using the joint-empirical measure of the means and covariances of the mixtures. Leveraging novel topological and analytical arguments, we relate weak convergence to the differentiability assumption (Assumption 5) of Theorem 4. This argument bridges exact asymptotic results for specific distributions to the universality of the corresponding estimators in a larger class of distributions, and we believe it can be of independent interest.

To summarize, our results significantly extends previous literature and provides rigorous guarantees to existing heuristic results. 

This discussion will be reflected in the revised manuscript.

Meta-review: The paper provides a theoretical answer to the empirical question about the universality of Gaussian mixtures in the analysis of the performance of generalized linear algorithms. The work is well-regarded for its theoretical rigor and is considered to be a step forward in the field. However, there are several points where the reviewers felt the paper could be improved. 

**Strengths:**
- Theoretical Rigor: All reviewers appreciated the rigorous theoretical framework presented in the paper.
- Originality and Significance: Reviewers noted the work's contribution in providing a mathematical foundation for understanding complex ML/statistical models through Gaussian mixture models.
- Empirical Consistency: Reviewers found the results to be consistent and the empirical section promising.

**Weaknesses:**
- Assumptions: Multiple reviewers pointed out that certain assumptions (especially Assumptions 4 and 5) need more intuitive explanations. The restrictive nature of these assumptions should be elaborated upon.
- Clarity and Readability: Several reviewers noted that the paper is hard to follow at times due to its technical depth and use of assumptions. Some inconsistencies in notation and lack of clarity in writing were also noted.
- Novelty: One reviewer pointed out that much of the technical content and assumptions seem to recap or rely heavily on previous work, notably those by Montanari and Saeed (2022).  This point should be explained more clearly in the final version.   
- Experimental Scope: Reviewers suggested expanding the range of data types and structures studied, to more convincingly demonstrate the universality claimed in the paper.  Also, the presentation of "applications of interest" in the current manuscript should be improved for clarity in the final version.  

The authors are strongly encouraged to address the mentioned concerns in making the final version.
Decision: Accept (poster)