Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolation

1 Introduction . Modern deep neural networks are overparameterized with respect to the amount of training data and achieve zero training error , yet generalize well on test data . Recent analysis has shown that fitting of noise in regression tasks can in fact be relatively benign for sufficiently high-dimensional linear models [ BLLT20 , BHX20 , HMRT19 , MVSS20 , KLS20 ] . However , these analyses do not 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . directly extend to classification , which requires separate treatment . In fact , very recent progress on sharp analysis of interpolating binary classifiers [ MNS+20 , CL21 , WT21 , CGB21 ] revealed highdimensional regimes in which binary classification generalizes well , but the corresponding regression task does not work and/or the success can not be predicted by classical margin-based bounds . In an important separate development , these same high-dimensional regimes admit an equivalence of loss functions used at training time . The support vector machine ( SVM ) , which arises from minimizing the logistic loss using gradient descent [ SHN+18 , JT19 ] , was recently shown to satisfy a high-probability equivalence to interpolation , which arises from minimizing the squared loss [ MNS+20 , HMX21 ] . This equivalence suggests that interpolation is ubiquitous in very overparmaeterized settings , and can arise naturally as a consequence of the optimization procedure even when this is not explicitly encoded or intended . Moreover , this equivalence to interpolation and corresponding analysis implies that the SVM can generalize even in regimes where classical learning theory bounds are not predictive . In the logistic model case [ MNS+20 ] and Gaussian binary mixture model case [ CL21 , WT21 , CGB21 ] , it is shown that good generalization of the SVM is possible beyond the realm in which classical margin-based bounds apply . These analyses lend theoretical grounding to the surprising hypothesis that squared loss can be equivalent to , or possibly even superior , to the cross-entropy loss for classification tasks . This hypothesis was supported empirically on kernel machines in Ryan Rifkin ’ s doctoral dissertation work [ Rif02 , RK04 ] , and more recently in overparameterized neural networks [ HB20 , PL20 ] . These compelling perspectives have thus far been limited to regression and binary classification settings . In contrast , most success stories and surprising new phenomena of modern machine learning have been recorded in multiclass classification settings , which appear naturally in a host of applications that demand the ability to automatically distinguish between large numbers of different classes ; for example , the popular ImageNet dataset [ RDS+15 ] contains on the order of 1000 classes . Whether a ) good generalization beyond effectively low-dimensional regimes where margin-based bounds are predictive is possible , and b ) equivalence of squared loss and cross-entropy loss holds in multiclass settings remained open problems . This paper makes significant progress towards a complete understanding of the optimization and generalization properties of high-dimensional linear multiclass classification , both for unconditional Gaussian covariates ( where labels are generated via a multinomial logistic model ) , and high-dimensional Gaussian mixture models . Our contributions are listed in more detail below . 1.1 Our Contributions • We establish a deterministic sufficient condition under which the multiclass SVM solution has a very simple and symmetric structure : it is identical to the solution of the One-vs-All ( OvA ) SVM classifier that uses the one-hot encoded labels . Moreover , the constraints at both solutions are active . Geometrically , this means that all data points are support vectors . • This implies a surprising equivalence between traditionally different formulations of multiclass SVM , which in turn are equivalent to the minimum-norm interpolating ( MNI ) classifier on one-hot label vectors . Thus , the outcomes of training with cross-entropy ( CE ) loss and squared loss are identical . • Next , for data following a Gaussian-mixtures model ( GMM ) or a Multinomial logistic model ( MLM ) , we show that the above sufficient condition is satisfied with high-probability under sufficient effective overparameterization depending on the number of classes , and on quantities related to the data covariance . Our numerical results show excellent agreement with our theoretical findings . • Subsequently , we provide novel bounds on the error of the MNI classifier for data generated either from the GMM or the MLM and characterize overparmeterization conditions under which benign overfitting occurs . A direct outcome of our results is that benign overfitting occurs under these conditions regardless of whether the cross-entropy loss or squared loss is used during training . Figure 1 describes our contributions and their implications through a flowchart . To the best of our knowledge , these are the first results characterizing a ) equivalence of loss functions , and b ) generalization of interpolating solutions in the multiclass setting . The multiclass setting poses several challenges over and above the recently studied binary case . When presenting our results in later sections , we discuss in detail how our analysis circumvents these challenges . 1.2 Related Work . Multiclass classification and the impact of training loss functions . There is a classical body of work on algorithms for multiclass classification , e.g. , [ WW98 , BB99 , DB95 , CS02 , LLW04 ] and several empirical studies of their comparative performance [ RK04 , F0̈2 , ASS01 ] ( also see [ HYS16 , GCOZ17 , KS18 , BEH20 , DCO20 , HB20 , PL20 ] for recent such studies in the context of deep nets ) . Many of these ( e.g . [ RK04 , HB20 , BEH20 ] ) have found that least-squares minimization yields competitive test classification performance to cross-entropy minimization . Our proof of equivalence of the SVM and MNI solutions under sufficient overparameterization provides theoretical support for this line of work . This is a consequence of the implicit bias of gradient descent run on the CE and squared losses leading to the multiclass SVM [ SHN+18 , JT19 ] and MNI [ EHN96 ] respectively . Numerous classical works investigated consistency [ Zha04 , LLW04 , TB07 , PGS13 , PS16 ] and finitesample behavior , e.g. , [ KP02 , CKMY16 , LDBK15 , Mau16 , LDZK19 ] of multiclass classification algorithms in the underparameterized regime . In contrast , our focus is on the highly overparameterized regime , where the typical uniform convergence techniques can not apply . Binary classification error analyses in overparameterized regime . The recent wave of analyses of the minimum- ` 2-norm interpolator ( MNI ) in high-dimensional linear regression ( an incomplete list is [ BLLT20 , BHX20 , HMRT19 , MVSS20 , KLS20 ] ) prompted researchers to consider to what extent the phenomena of benign overfitting and double descent [ BHMM19 , GJS+20 ] can be proven to occur in classification tasks . Even the binary classification setting turns out to be significantly more challenging to study owing to the discontinuity of the 0 1 test loss function . Sharp asymptotic formulas for the generalization error of binary classification algorithms in the linear high-dimensional regime have been derived in several recent works [ Hua17 , SC19 , MLC19 , SAH19 , TPT20 , TPT21 , DKT21 , MRSY19 , KA21 , LS20 , SAH20 , AKLZ20 , Lol20 , DL20 ] . These formulas are solutions to complicated nonlinear systems of equations that typically do not admit closed-form expressions . A separate line of work provides non-asymptotic error bounds for both the MNI classifier and the SVM classifier [ CL21 , MNS+20 , WT21 , CGB21 ] ; in particular , [ MNS+20 ] analyzed the SVM in a Gaussian covariates model by explicitly connecting its solution to the MNI solution . Subsequently , [ WT21 ] also took this route to analyze the SVM and MNI in mixture models , which turn out to be more technically involved . Even more recently , [ CGB21 ] provided extensions of the result by [ WT21 ] to sub-Gaussian mixtures . While these non-asymptotic analyses are only sharp in their dependences on n and p , they provide closed-form generalization expressions in terms of easily interpretable summary statistics . Interestingly , these results imply good generalization of the SVM beyond the regime in which margin-based bounds are predictive . Specifically , [ MNS+20 ] identifies a separating regime for Gaussian covariates in which corresponding regression tasks would not generalize . In the Gaussian mixture model , margin-based bounds [ SFBL98 , BM03 ] ( as well as corresponding recently derived mistake bounds on interpolating classifiers [ LR21 ] ) would require the intrinsic signal-to-noise-ratio ( SNR ) to scale at least as ! ( p1/2 ) for good generalization ; however , the analyses of [ CL21 , WT21 , CGB21 ] show that good generalization is possible for significantly lower SNR scaling as ! ( p1/4 ) The above error analyses are specialized to the binary case , where closed-form error expressions are easy to derive [ MNS+20 ] . The only related work applicable to multiclass settings is [ TOS20 ] , which also highlights the numerous challenges of obtaining a sharp error analysis in multiclass settings . Specifically , [ TOS20 ] derived sharp generalization formulas for multiclass least-squares in underparameterized settings ; extensions to the overparameterized regime and other losses beyond least-squares remained wide open . Finally , [ KT21 ] recently derived sharp phase-transition thresholds for the feasibility of OvA-SVM on multiclass Gaussian mixture data in the linear high-dimensional regime . However , their result does not cover the more challenging multiclass-SVM that we investigate here . Other SVM analyses . The number of support vectors in binary SVM has been characterized in lowdimensional separable and non-separable settings [ DOS99 , BG01 , MO05 ] and scenarios have been identified in which there is vanishing fraction of support vectors , as this implies good generalization1 via PAC-Bayes sample compression bounds [ Vap13 ] . In the highly overparameterized regime that we consider , perhaps surprisingly , the opposite behavior occurs : all training points become support vectors with high probability [ DOS99 , BG01 , MO05 , MNS+20 , HMX21 ] . In particular , [ HMX21 ] provided sharp non-asymptotic sufficient conditions for this phenomenon for both isotropic and anisotropic settings . The techniques in [ MNS+20 , HMX21 ] are highly specialized to the binary SVM and its dual , where a simple complementary slackness condition directly implies the property of interpolation . In contrast , the complementary slackness condition for the case of multiclass SVM does not directly imply interpolation ; in fact , the operational meaning of “ all training points becoming support vectors '' is unclear in the multiclass SVM . Our proof of deterministic equivalence goes beyond the complementary slackness condition and uncovers a surprising symmetric structure2 by showing equivalence of multiclass SVM to a symmetric OvA classifier . Notation For a vector v 2 Rp , let kvk2 = pP p i=1 v 2 i , kvk1 = P p i=1 |vi| , kvk1 = maxi { |vi| } . v > 0 is interpreted elementwise . 1 / 0 denote the all-ones / all-zeros vectors and ei denotes the i-th standard basis vector . For a matrix M , kMk2 denotes its 2 ! 2 operator norm and kMkF denotes the Frobenius norm . denotes the Hadamard product . [ n ] denotes the set { 1 , 2 , ... , n } . We also use standard “ Big O '' notations ⇥ ( · ) , ! ( · ) , e.g. , see [ CLRS09 , Chapter 3 ] . Finally , we write N ( µ , ⌃ ) for the ( multivariate ) Gaussian distribution of mean µ and covariance matrix ⌃ , and , Q ( x ) = P ( Z > x ) , Z ⇠ N ( 0 , 1 ) for the Q-function of a standard normal . Throughout , constants refer to numbers that do not depend on the problem dimensions n or p .

Summary: “Benign overfitting” is a recently revealed phenomenon that high-dimensional models achieving zero training loss can still have good generalization. Existing works studying benign overfitting are mostly focused on the regression setting or the binary classification setting. In this paper, the authors establish benign overfitting guarantees for multi-class classification problems. It is shown that empirical risk minimization with cross-entropy loss is equivalent to the min-norm interpolating (MNI) solutions and the one-vs-all SVM classifier, and then derive error bounds on the accuracy of the MNI classifier.