Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolation

1 Introduction . Modern deep neural networks are overparameterized with respect to the amount of training data and achieve zero training error , yet generalize well on test data . Recent analysis has shown that fitting of noise in regression tasks can in fact be relatively benign for sufficiently high-dimensional linear models [ BLLT20 , BHX20 , HMRT19 , MVSS20 , KLS20 ] . However , these analyses do not 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . directly extend to classification , which requires separate treatment . In fact , very recent progress on sharp analysis of interpolating binary classifiers [ MNS+20 , CL21 , WT21 , CGB21 ] revealed highdimensional regimes in which binary classification generalizes well , but the corresponding regression task does not work and/or the success can not be predicted by classical margin-based bounds . In an important separate development , these same high-dimensional regimes admit an equivalence of loss functions used at training time . The support vector machine ( SVM ) , which arises from minimizing the logistic loss using gradient descent [ SHN+18 , JT19 ] , was recently shown to satisfy a high-probability equivalence to interpolation , which arises from minimizing the squared loss [ MNS+20 , HMX21 ] . This equivalence suggests that interpolation is ubiquitous in very overparmaeterized settings , and can arise naturally as a consequence of the optimization procedure even when this is not explicitly encoded or intended . Moreover , this equivalence to interpolation and corresponding analysis implies that the SVM can generalize even in regimes where classical learning theory bounds are not predictive . In the logistic model case [ MNS+20 ] and Gaussian binary mixture model case [ CL21 , WT21 , CGB21 ] , it is shown that good generalization of the SVM is possible beyond the realm in which classical margin-based bounds apply . These analyses lend theoretical grounding to the surprising hypothesis that squared loss can be equivalent to , or possibly even superior , to the cross-entropy loss for classification tasks . This hypothesis was supported empirically on kernel machines in Ryan Rifkin ’ s doctoral dissertation work [ Rif02 , RK04 ] , and more recently in overparameterized neural networks [ HB20 , PL20 ] . These compelling perspectives have thus far been limited to regression and binary classification settings . In contrast , most success stories and surprising new phenomena of modern machine learning have been recorded in multiclass classification settings , which appear naturally in a host of applications that demand the ability to automatically distinguish between large numbers of different classes ; for example , the popular ImageNet dataset [ RDS+15 ] contains on the order of 1000 classes . Whether a ) good generalization beyond effectively low-dimensional regimes where margin-based bounds are predictive is possible , and b ) equivalence of squared loss and cross-entropy loss holds in multiclass settings remained open problems . This paper makes significant progress towards a complete understanding of the optimization and generalization properties of high-dimensional linear multiclass classification , both for unconditional Gaussian covariates ( where labels are generated via a multinomial logistic model ) , and high-dimensional Gaussian mixture models . Our contributions are listed in more detail below . 1.1 Our Contributions • We establish a deterministic sufficient condition under which the multiclass SVM solution has a very simple and symmetric structure : it is identical to the solution of the One-vs-All ( OvA ) SVM classifier that uses the one-hot encoded labels . Moreover , the constraints at both solutions are active . Geometrically , this means that all data points are support vectors . • This implies a surprising equivalence between traditionally different formulations of multiclass SVM , which in turn are equivalent to the minimum-norm interpolating ( MNI ) classifier on one-hot label vectors . Thus , the outcomes of training with cross-entropy ( CE ) loss and squared loss are identical . • Next , for data following a Gaussian-mixtures model ( GMM ) or a Multinomial logistic model ( MLM ) , we show that the above sufficient condition is satisfied with high-probability under sufficient effective overparameterization depending on the number of classes , and on quantities related to the data covariance . Our numerical results show excellent agreement with our theoretical findings . • Subsequently , we provide novel bounds on the error of the MNI classifier for data generated either from the GMM or the MLM and characterize overparmeterization conditions under which benign overfitting occurs . A direct outcome of our results is that benign overfitting occurs under these conditions regardless of whether the cross-entropy loss or squared loss is used during training . Figure 1 describes our contributions and their implications through a flowchart . To the best of our knowledge , these are the first results characterizing a ) equivalence of loss functions , and b ) generalization of interpolating solutions in the multiclass setting . The multiclass setting poses several challenges over and above the recently studied binary case . When presenting our results in later sections , we discuss in detail how our analysis circumvents these challenges . 1.2 Related Work . Multiclass classification and the impact of training loss functions . There is a classical body of work on algorithms for multiclass classification , e.g. , [ WW98 , BB99 , DB95 , CS02 , LLW04 ] and several empirical studies of their comparative performance [ RK04 , F0̈2 , ASS01 ] ( also see [ HYS16 , GCOZ17 , KS18 , BEH20 , DCO20 , HB20 , PL20 ] for recent such studies in the context of deep nets ) . Many of these ( e.g . [ RK04 , HB20 , BEH20 ] ) have found that least-squares minimization yields competitive test classification performance to cross-entropy minimization . Our proof of equivalence of the SVM and MNI solutions under sufficient overparameterization provides theoretical support for this line of work . This is a consequence of the implicit bias of gradient descent run on the CE and squared losses leading to the multiclass SVM [ SHN+18 , JT19 ] and MNI [ EHN96 ] respectively . Numerous classical works investigated consistency [ Zha04 , LLW04 , TB07 , PGS13 , PS16 ] and finitesample behavior , e.g. , [ KP02 , CKMY16 , LDBK15 , Mau16 , LDZK19 ] of multiclass classification algorithms in the underparameterized regime . In contrast , our focus is on the highly overparameterized regime , where the typical uniform convergence techniques can not apply . Binary classification error analyses in overparameterized regime . The recent wave of analyses of the minimum- ` 2-norm interpolator ( MNI ) in high-dimensional linear regression ( an incomplete list is [ BLLT20 , BHX20 , HMRT19 , MVSS20 , KLS20 ] ) prompted researchers to consider to what extent the phenomena of benign overfitting and double descent [ BHMM19 , GJS+20 ] can be proven to occur in classification tasks . Even the binary classification setting turns out to be significantly more challenging to study owing to the discontinuity of the 0 1 test loss function . Sharp asymptotic formulas for the generalization error of binary classification algorithms in the linear high-dimensional regime have been derived in several recent works [ Hua17 , SC19 , MLC19 , SAH19 , TPT20 , TPT21 , DKT21 , MRSY19 , KA21 , LS20 , SAH20 , AKLZ20 , Lol20 , DL20 ] . These formulas are solutions to complicated nonlinear systems of equations that typically do not admit closed-form expressions . A separate line of work provides non-asymptotic error bounds for both the MNI classifier and the SVM classifier [ CL21 , MNS+20 , WT21 , CGB21 ] ; in particular , [ MNS+20 ] analyzed the SVM in a Gaussian covariates model by explicitly connecting its solution to the MNI solution . Subsequently , [ WT21 ] also took this route to analyze the SVM and MNI in mixture models , which turn out to be more technically involved . Even more recently , [ CGB21 ] provided extensions of the result by [ WT21 ] to sub-Gaussian mixtures . While these non-asymptotic analyses are only sharp in their dependences on n and p , they provide closed-form generalization expressions in terms of easily interpretable summary statistics . Interestingly , these results imply good generalization of the SVM beyond the regime in which margin-based bounds are predictive . Specifically , [ MNS+20 ] identifies a separating regime for Gaussian covariates in which corresponding regression tasks would not generalize . In the Gaussian mixture model , margin-based bounds [ SFBL98 , BM03 ] ( as well as corresponding recently derived mistake bounds on interpolating classifiers [ LR21 ] ) would require the intrinsic signal-to-noise-ratio ( SNR ) to scale at least as ! ( p1/2 ) for good generalization ; however , the analyses of [ CL21 , WT21 , CGB21 ] show that good generalization is possible for significantly lower SNR scaling as ! ( p1/4 ) The above error analyses are specialized to the binary case , where closed-form error expressions are easy to derive [ MNS+20 ] . The only related work applicable to multiclass settings is [ TOS20 ] , which also highlights the numerous challenges of obtaining a sharp error analysis in multiclass settings . Specifically , [ TOS20 ] derived sharp generalization formulas for multiclass least-squares in underparameterized settings ; extensions to the overparameterized regime and other losses beyond least-squares remained wide open . Finally , [ KT21 ] recently derived sharp phase-transition thresholds for the feasibility of OvA-SVM on multiclass Gaussian mixture data in the linear high-dimensional regime . However , their result does not cover the more challenging multiclass-SVM that we investigate here . Other SVM analyses . The number of support vectors in binary SVM has been characterized in lowdimensional separable and non-separable settings [ DOS99 , BG01 , MO05 ] and scenarios have been identified in which there is vanishing fraction of support vectors , as this implies good generalization1 via PAC-Bayes sample compression bounds [ Vap13 ] . In the highly overparameterized regime that we consider , perhaps surprisingly , the opposite behavior occurs : all training points become support vectors with high probability [ DOS99 , BG01 , MO05 , MNS+20 , HMX21 ] . In particular , [ HMX21 ] provided sharp non-asymptotic sufficient conditions for this phenomenon for both isotropic and anisotropic settings . The techniques in [ MNS+20 , HMX21 ] are highly specialized to the binary SVM and its dual , where a simple complementary slackness condition directly implies the property of interpolation . In contrast , the complementary slackness condition for the case of multiclass SVM does not directly imply interpolation ; in fact , the operational meaning of “ all training points becoming support vectors '' is unclear in the multiclass SVM . Our proof of deterministic equivalence goes beyond the complementary slackness condition and uncovers a surprising symmetric structure2 by showing equivalence of multiclass SVM to a symmetric OvA classifier . Notation For a vector v 2 Rp , let kvk2 = pP p i=1 v 2 i , kvk1 = P p i=1 |vi| , kvk1 = maxi { |vi| } . v > 0 is interpreted elementwise . 1 / 0 denote the all-ones / all-zeros vectors and ei denotes the i-th standard basis vector . For a matrix M , kMk2 denotes its 2 ! 2 operator norm and kMkF denotes the Frobenius norm . denotes the Hadamard product . [ n ] denotes the set { 1 , 2 , ... , n } . We also use standard “ Big O '' notations ⇥ ( · ) , ! ( · ) , e.g. , see [ CLRS09 , Chapter 3 ] . Finally , we write N ( µ , ⌃ ) for the ( multivariate ) Gaussian distribution of mean µ and covariance matrix ⌃ , and , Q ( x ) = P ( Z > x ) , Z ⇠ N ( 0 , 1 ) for the Q-function of a standard normal . Throughout , constants refer to numbers that do not depend on the problem dimensions n or p .

Summary: In this paper, the authors study benign overfitting in multiclass linear classification. They show the equivalence between one-vs-all SVM and multiclass SVM with either cross entropy or squared loss under a sufficient condition. Furthermore they show that when overparametrization occurs, the sufficient condition holds with high probability.