Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixtures

1 Introduction . In modern machine learning , complex models such as deep neural networks have become increasingly popular . These complicated models are capable of fitting noisy training data sets , while at the same time achieving small test errors . In fact , this benign overfitting phenomenon is not a unique feature of deep learning . Even for kernel methods and linear models , [ 5 ] demonstrated that interpolators on the noisy training data can still perform near optimally on the test data . A series of recent works [ 4 , 20 , 11 , 2 ] theoretically studied how over-parameterization can achieve small population risk . In particular in [ 2 ] the authors considered the setting where the data are generated from a ground-truth linear model with noise , and established a tight population risk bound for the minimum norm linear interpolator with a matching lower bound . More recently , [ 23 ] further studied benign overfitting in ridge regression , and established non-asymptotic generalization bounds for over-parametrized ridge regression . They showed that those bounds are tight for a range of regularization parameter values . Notably , these results cover arbitrary covariance structure of the data , and give a nice characterization of how the spectrum of the data covariance matrix affects the population risk in the over-parameterized regime . Very recently , benign overfitting has also been studied in the setting of linear classification [ 6 , 19 , 25 ] . Specifically , [ 19 ] studied the setting where the data inputs are Gaussian and the labels are generated 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . from a ground truth linear model with label flipping noise , and showed equivalence between the hardmargin support vector machine ( SVM ) solution and the minimum norm interpolator to study benign overfitting . [ 6 , 25 ] studied the benign overfitting phenomenon in sub-Gaussian/Gaussian mixture models and established population risk bounds for the maximum margin classifier . [ 6 ] leveraged the implicit bias of gradient descent for logistic regression [ 22 ] to establish the risk bound . [ 25 ] established an equivalence result between classification and regression for isotropic Gaussian mixture models . While these results have offered valuable insights into the benign overfitting phenomenon for ( sub- ) Gaussian mixture classification , they still have certain limitations . Unlike the results in the regression setting where the eigenvalues of the data covariance matrix play a key role , the current results for Gaussian/sub-Gaussian mixture models do not show the impact of the spectrum of the data covariance matrix on the risk . In this paper , we study the benign overfitting phenomenon in a general sub-Gaussian mixture model that covers both the isotropic and anisotropic settings , where the d-dimensional features from two classes have the same covariance matrix Σ but have different means µ and −µ respectively . We consider the over-parameterized setting where d is larger than the sample size n , and prove a risk bound for the maximum margin classifier . We show that under certain conditions on eigenvalues of Σ , the mean vector µ and the sample size n , the maximum margin classifier for this problem is identical to the minimum norm interpolator . We then utilize this result to establish a tight population risk bound of the maximum margin classifier . Our result reveals how the eigenvalues of the covariance matrix Σ affect the benign property of the classification problem , and is tighter and more general than existing results on sub-Gaussian/Gaussian mixture models . The contributions of this paper are as follows : • We establish a tight population risk bound for the maximum margin classifier . Our bound works for both the isotropic and anisotropic settings , which is more general than existing results in [ 6 , 25 ] . When reducing our bound to the setting studied in [ 6 ] , our result gives a bound exp ( −Ω ( n‖µ‖42/d ) ) , where n is the training sample size . Our bound is tighter than the risk bound exp ( −Ω ( ‖µ‖42/d ) ) in [ 6 ] by a factor of n in the exponent . Our result also gives a tighter risk bound than that in [ 25 ] 1 in the so-called “ low SNR setting ” : our result suggests that ‖µ‖42 = ω ( d/n ) suffices to ensure an o ( 1 ) population risk , while [ 25 ] requires ‖µ‖42 = ω ( ( d/n ) 3/2 ) . • We establish population risk lower bounds achieved by the maximum margin classifier under two different settings . In both settings , the lower bounds match our population risk upper bound up to some absolute constants . This suggests that our population risk bound is tight . • Our analysis reveals that for a class of high-dimensional anisotropic sub-Gaussian mixture models , the maximum margin linear classifier on the training data can achieve small population risk under mild assumptions on the sample size n and mean vector µ . Specifically , suppose that the eigenvalues of Σ are { λk = k−α } dk=1 for some parameter α ∈ [ 0 , 1 ) , and treat the sample size n as a constant . Then our result shows that to achieve o ( 1 ) population risk , the following conditions on ‖µ‖2 suffice : ‖µ‖2 = ω ( d 1/4−α/2 ) , if α ∈ [ 0 , 1/2 ) , ω ( ( log ( d ) ) 1/4 ) , if α = 1/2 , ω ( 1 ) . if α ∈ ( 1/2 , 1 ) . More specifically , when α = 1/2 , the condition on the mean vector µ only has a logarithmic dependency on the dimension d , and when α ∈ ( 1/2 , 1 ) , the condition on µ for benign overfitting is dimension free . • Our proof of the population risk bound introduces some tight intermediate results , which may be of independent interest . Specifically , our proof utilizes the polarization identity to establish equivalence between the maximum margin classifier and the minimum norm interpolator . This is , to the best of our knowledge , the first equivalence result between classification and regression for anisotropic sub-Gaussian mixture models . Additional Related Work Our study is closely related to the phenomenon of double descent studied in recent works . [ 3 , 4 ] showed experimental results and provided theoretical analyses on some 1After we posted our paper on arXiv , we noticed that the authors of [ 25 ] updated their paper a week later to include the results for mixture of anisotropic Gaussians and sharpened their bounds . Our results are still more general as we covered the mixture of anisotropic sub-Gaussian , and we also proved a matching lower bound . specific models to demonstrate that the risk curve versus over-parameterization has a double descent shape . These results can therefore indicate that over-parameterization can be beneficial to achieve small test risk . [ 11 , 26 ] studied the double descent phenomenon in linear regression under the setting where the dimension d and sample size n can grow simultaneously but have a fixed ratio , and showed that the population risk exhibits a double descent curve with respect to the ratio . More recently , [ 17 , 15 , 18 ] further extended the setting to random feature models and studied double descent when the sample size , data dimension and the number of random features have fixed ratios . Our work is also related to the studies of implicit bias , which analyze the impact of training algorithms when the over-parameterized models have multiple global minima . Specifically , [ 22 ] showed that if the training data are linearly separable , then gradient descent on unregularized logistic regression converges directionally to the maximum margin linear classifier on the training data set . [ 13 ] further studied the implicit bias of gradient descent for logistic regression on non-separable data . [ 9 ] studied the implicit bias of various optimization methods for generic objective functions . [ 10 , 1 ] established implicit bias results for matrix factorization problems . More recently , [ 16 ] showed that gradient flow for learning homogeneous neural networks with logistic loss maximizes the normalized margin on the training data set . These studies of implicit bias offer a handle for us to connect the over-parameterized logistic regression with the maximum margin classifiers for linear models . 2 Problem Setting and Notation . Notations . We use lower case letters to denote scalars , and use lower/upper case bold face letters to denote vectors/matrices respectively . For a vector v , we denote by ‖v‖2 the ` 2-norm of v. For a matrix A , we use ‖A‖2 , ‖A‖F to denote its spectral norm and Frobinuous norm respectively , and use tr ( A ) to denote its trace . For a vector v ∈ Rd and a positive definite matrix A , we define ‖v‖A = √ v > Av . For an integer n , we denote [ n ] = { 1 , 2 , . . . , n } . We also use standard asymptotic notations O ( · ) , Ω ( · ) , o ( · ) , and ω ( · ) . Let { an } and { bn } be two sequences . If there exists a constant C > 0 such that |an| ≤ C|bn| for all large enough n , then we denote an = O ( bn ) . We denote an = Ω ( bn ) if bn = O ( an ) . Moreover , we write an = o ( bn ) if lim |an/bn| = 0 and an = ω ( bn ) if lim |an/bn| = ∞ . We also use Õ ( · ) and Ω̃ ( · ) to hide some logarithmic terms in Big-O and Big-Omega notations . At last , for a random variable Z , we denote by ‖Z‖ψ2 and ‖Z‖ψ1 the sub-Gaussian and subexponential norms of Z respectively . Sub-Gaussian Mixture Model . We consider a model where the feature vectors are generated from a mixture of two sub-Gaussian distributions with means µ and −µ and the same covariance matrix Σ . We assume that each data pair ( x , y ) are generated independently from the following procedure : 1 . The label y ∈ { +1 , −1 } is generated as a Rademacher random variable . 2 . A random vector u ∈ Rd is generated from a distribution such that the entries of u are independent sub-Gaussian random variables with E [ uj ] = 0 , E [ u2j ] = 1 and ‖uj‖ψ2 ≤ σu for all j ∈ [ d ] . 3 . Let Σ be a positive definite matrix with eigenvalue decomposition Σ = VΛV > , where Λ = diag { λ1 , . . . , λd } and V is an orthonormal matrix consisting of the eigenvectors of Σ . We calculate the random vector q based on u as q = VΛ1/2u . This ensures that q has mean zero and a covariance matrix Σ . 4 . The feature is given as x = y ·µ+ q , where µ ∈ Rd is a vector . Clearly , the mean of x is µ when y = 1 and is −µ when y = −1 . We consider n training data points ( xi , yi ) generated independently from the above procedure , and denote X = yµ > + Q , where X = [ x1 , . . . , xn ] > , Q = [ q1 , . . . , qn ] > ∈ Rn×d , and y = [ y1 , . . . , yn ] > ∈ { ±1 } n. For any θ ∈ Rd , the population risk of the linear classifier x→ 〈θ , x〉 is defined as : R ( θ ) = P ( y · 〈θ , x〉 < 0 ) . In this paper , we consider the maximum margin linear classifier θ̂SVM , i.e. , the solution to the hard-margin support vector machine : θ̂SVM = argmin ‖θ‖22 , subject to yi · 〈θ , xi〉 ≥ 1 , i ∈ [ n ] , and study its population risk R ( θ̂SVM ) . A recent work [ 6 ] has studied a similar sub-Gaussian mixture model under an assumption that tr ( Σ ) = Ω ( d ) , and considered additional label flipping noises . In this paper , we do not introduce the label flipping noises for simplicity , but we consider a general covariance matrix Σ to cover the general anisotropic setting . It is worth noting that although our model is not exactly the same as [ 6 ] because we don ’ t have additional label flipping noise , there is still noise in our model because of the nature of sub-Gaussian mixture model . For example , consider a mixture of two Gaussian distributions . The two Gaussian clusters have non-trivial overlap , and the Bayes optimal classifier has non-zero Bayes risk as long as ‖µ‖2 < ∞ . Therefore , the Bayes optimal classifier and the interpolating classifier are generally quite different . In general , a model is appropriate for the study of benign overfitting whenever the optimal classifier has non-zero Bayes risk . Our model is rather general and covers the following examples . Example 2.1 ( Gaussian mixture model ) . The most straight-forward example is when the data are generated from Gaussian mixtures N ( µ , Σ ) and N ( −µ , Σ ) . This is covered by our model when the sub-Gaussian vector u is a standard Gaussian random vector . Example 2.2 ( Rare/weak feature model ) . The rare-weak model is a special case of the Gaussian mixture model where Σ = I and µ is a sparse vector with s non-zero entries equaling γ . The rare/weak feature model was originally investigated by [ 8 , 14 ] , and was recently studied by [ 6 ] . Connection to Over-parameterized Logistic Regression . Our study of the maximum margin classifier is closely related to over-parameterized logistic regression . In logistic regression , we consider the following empirical loss minimization problem : min θ∈Rd L ( θ ) : = 1 n n∑ i=1 log [ 1 + exp ( −yi · 〈θ , xi〉 ) ] . We solve the above optimization problem with gradient descent θ ( t+1 ) = θ ( t ) − η · ∇L ( θ ( t ) ) , ( 2.1 ) where η > 0 is the learning rate . In the over-parameterized setting where d n , it is evident that the training data points are linearly separable with high probability ( for example , XX > is invertible with high probability and the minimum norm interpolator θ̂LS = X > ( XX > ) −1y separates the training data. ) . For linearly separable data , a series of recent works have studied the implicit bias of ( stochastic ) gradient descent for logistic regression [ 22 , 13 , 21 ] . These results demonstrate that among all linear classifiers that can classify the training data correctly , gradient descent will converge to the one that maximizes the ` 2 margin . Such an implicit bias result is summarized in the following lemma . Lemma 2.3 ( Theorem 3 in [ 22 ] ) . Suppose that the training data set { ( xi , yi ) } is linearly separable . Then as long as η > 0 is small enough , the gradient descent iterates θ ( t ) for logistic regression defined in ( 2.1 ) has the following direction limit : lim t→∞ θ ( t ) ‖θ ( t ) ‖2 = θ̂SVM ‖θ̂SVM‖2 , where θ̂SVM is the maximum margin classifier . Lemma 2.3 suggests that our risk bound of the maximum margin classifier θ̂SVM directly implies a risk bound for the over-parameterized logistic regression trained by gradient descent .

Summary: This paper proves new risk bounds for overfitting in the over-parameterized linear classification setting when the data is generated by a sub-Gaussian mixture. In certain regimes, the results show that overfitting can be benign, that is, overfitting still leads to a good test error. An important property of the proposed upper bound on the risk is that it is not directly dependent on the dimension, but rather on certain norms of the covariance matrix. This is useful because, as shown in the paper, if the eigenvalues of the covariance matrix decay fast enough, then requirements for benign overfitting become dimension independent.