Support vector machines and linear regression coincide with very high-dimensional features

1 Introduction . The support vector machine ( SVM ) and ordinary least squares ( OLS ) are well-weathered approaches to fitting linear models , but they are associated with different learning tasks : classification and regression . In this paper , we study the case in which the models return exactly the same hypothesis for sufficiently high-dimensional data . The hard-margin SVM is a linear classification model that finds the separating hyperplane that maximizes the minimum margin of error for every training sample . If the training data ( x1 , y1 ) , . . . , ( xn , yn ) ∈ Rd × { ±1 } are linearly separable , then the resulting linear classifier is x 7→ sign ( xTwSVM ) , where wSVM is the solution to the following optimization problem : wSVM = argmin w∈Rd ∥w∥2 such that yiw Txi ≥ 1 , ∀i ∈ [ n ] . ( 1 ) An example xi is a support vector if the corresponding constraint is satisfied with equality , and the optimal solution wSVM is a linear combination of these support vectors . Ordinary least squares regression finds the linear function that best fits the training data ( x1 , y1 ) , . . . , ( xn , yn ) ∈ Rd × R according to the sum of squared errors . When the solution is not unique , it is natural to take the solution of minimum Euclidean norm ; this is the convention we adopt . Taking X : = [ x1| . . . |xn ] T ∈ Rn×d and y : = ( y1 , . . . , yn ) , the solution is the hypothesis x 7→ wTOLSx where wOLS is the solution to the following : wOLS = argminw∈Rd ∥w∥2 such that XTXw = XTy . In many high-dimensional settings ( e.g. , where X has full row rank ) , the solution may in fact interpolate the training data , i.e. , wOLS = argmin w∈Rd ∥w∥2 such that w Txi = yi , ∀i ∈ [ n ] . ( 2 ) 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Although the optimization problems in ( 1 ) and ( 2 ) are very different , they have been observed to coincide in very high-dimensional regimes . The study of this support vector proliferation ( SVP ) phenomenon—in which every training example is a support vector—was recently initiated by Muthukumar et al . [ 35 ] and Hsu et al . [ 23 ] . Roughly speaking , they show that SVP occurs when d = Ω ( n log n ) for a broad class of sample distributions , and that SVP does not occur when d = O ( n ) in an idealized isotropic Gaussian case . SVP is a phenomenon that connects linear classification and linear regression , topics that have received renewed attention due to the break-down of classical analyses of these methods in high-dimensions . For instance , some analyses of SVM that are based on the number of support vectors become vacuous when this number becomes large [ e.g. , 19 , 20 , 44 ] . Similarly , overparameterized linear regression is typically only studied in noisy settings with explicit regularization . It was not until recently that SVM and OLS have been meaningfully analyzed in these regimes ( see Section 1.2 ) , and the connection between the two approaches via SVP has played an important analytical role [ 9 , 35 , 46 ] . In this work , we further examine support vector proliferation with the goal of broadly understanding when and why SVMs and OLS coincide . We pose and study the following questions : 1 . How general is the SVP phenomena ? What relationship between d and n determines if the solutions to ( 1 ) and ( 2 ) coincide ? We close the log n gap from the prior work of Hsu et al . [ 23 ] by showing that d ≳ n log n is necessary for SVP to occur under a model of independent subgaussian features , even with constant probability . Our lower-bounds hold for a broad class of distributions over xi , and they match the upper-bounds from [ 23 ] . This demonstrates that SVP is extremely unlikely to occur in the much-studied d = Θ ( n ) setting . 2 . Is there a sharp threshold separating the occurrence and non-occurrence of this phenomenon ? Is this threshold universal across all “ reasonable ” distributions over each xi ? We hypothesize that a sharp phase transition occurs at d = 2n log n. We rigorously prove this hypothesis for isotropic Gaussian features and quantitatively bound the width of the transition . We experimentally observe the same transition for a wide range of other distributions . 3 . Is support vector proliferation specific to the ℓ2 SVM problem ? If ( 1 ) and ( 2 ) are generalized to instead minimize ℓp norms , does this still occur at the same rate ? We re-frame this question with a geometric characterization of the dual of the SVM optimization problem for ℓp norms . We conjecture that a similar phase transition occurs for ℓ1 , but also that it requires much larger dimension d ; this is supported by preliminary experiments . 1.1 Outline of our results . Section 2 introduces the SVM and OLS approaches in full generality , our λ-anisotropic subgaussian data model , and prior results about SVP . Several equivalent characterizations of SVP are established ( Proposition 1 ) for use in subsequent sections . Section 3 characterizes when SVP does not occur for a broad range of distributions ( Theorem 3 ) . Our lower-bound on the dimension required for SVP matches the upper-bounds from [ 23 ] in the isotropic Gaussian setting , resolving the open question from that work , and also gives new lower-bounds for anisotropic cases . The proof works by tightly controlling the spectrum of the Gram matrix and establishing anti-concentration via the Berry-Esseen Theorem . Section 4 establishes a sharp threshold of d = 2nlog n for SVP in the case of isotropic Gaussian samples , and also characterize the width of the phase transition ( Theorem 4 ) . Section 5 provides empirical evidence that the sharp threshold observed in Section 4 holds for a wide range of random variables . Rigorous statistical methodology inspired by Donoho and Tanner [ 16 ] is used to test our “ universality hypothesis ” that the probability of SVP does not depend on the underlying sample distribution as d and n become large . Section 6 asks the questions about SVP from the preceding sections in the context of ℓ1-SVM and minimum ℓ1-norm interpolation . Specifically , the SVP threshold for ℓ1 is conjectured to occur for d = ω ( n log n ) . Evidence for this conjecture is provided in a simulation study and in geometric arguments about random linear programs . 1.2 Related work . Prior works connecting SVP and generalization . Muthukumar et al . [ 35 ] initiate the study of SVP in part to facilitate generalization analysis of the SVM in very high-dimensional settings . Their work , as well as the contemporaneous work of Chatterji and Long [ 10 ] , shows that the SVM enjoys low test error in certain regimes where classical learning-theoretic analyses would otherwise yield vacuous error bounds . ( In fact , one of the settings in [ 10 ] requires polynomially-higher dimension than is typically studied : d = Ω ( n2 log n ) . ) The coincidence between SVM and OLS identified by Muthukumar et al . was also more recently used by Wang and Thrampoulidis [ 46 ] and Cao et al . [ 9 ] for analyses of linear classification in very high dimensions under different data distributions . The generalization analysis of Muthukumar et al . concerns a data model inspired by the spiked covariance model of Wang and Fan [ 47 ] . They identify a regime of overparameterization where the hard-margin SVM classifier has good generalization ( i.e. , classification risk going to 0 ) even when all the training samples are supoort vectors . Our new lower bound can be regarded as establishing a limit on this approach to the analysis of SVM ; specifically , if the ( effective ) dimension is not sufficiently large , the OLS and SVM solutions may not coincide . Prior analyses of number of support vectors . Besides its relevance to generalization analysis , the number of support vectors in an SVM model is an interesting quantity to study in its own right . Hsu et al . [ 23 ] sharpen and extend the analysis of Muthukumar et al . [ 35 ] about SVP in the independent features model that we also adopt . They prove that SVM on n samples with d independent subgaussian components coincides with OLS when d = Ω ( n log n ) with probability tending to 1 . They also give a converse result stating that the coincidence fails with constant probability when d = O ( n ) in the isotropic Gaussian feature model . ( We give these results here as Theorems 1 and 2 respectively . ) Our results generalize and tighten the latter bound to tell an asymptotically sharp story about the phase transition for both isotropic and anisotropic random vectors with subgaussian components . Our specific analysis for the isotropic Gaussian case gives the exact point of the phase transition . The number of support vectors is also studied in the context of variants of SVM [ 3 , 39 ] , including the soft-margin SVM [ 12 ] and the ν-SVM [ 37 ] . In these cases , the asymptotic number of support vectors is shown to be related to the noise rate in the problem . The setups we study are linearly separable , which makes it possible to study the hard-margin SVM ( without regularization ) . The hard-margin SVM is also of interest because it captures the implicit bias of gradient descent on the logistic loss objective for linear predictors [ 25 , 38 ] . Phase transitions have been studied in the context of linear classification [ 8 , 13 , 27 , 40 ] , and SVMs in particular [ 7 , 14 , 28 , 30 ] , but most study qualitative changes in behavior other than support vector proliferation . The most relevant is the study of Buhot and Gordon [ 7 ] , who employ techniques from statistical physics to show the existence of phase transitions for the generalization error , margin size , and number of support vectors as n and d = Θ ( n ) become arbitrarily large . While they characterize the fraction of samples that are support vectors , they do not address our question about when all samples are support vectors , not just a large fraction . Indeed , our results demonstrate that their regime where d grows linearly with n will not exhibit support vector proliferation when n and d in the limit . Overparameterized linear regression . There has been a recent flurry of analyses of overparameterized linear regression models [ e.g. , 4 , 5 , 21 , 24 , 29 , 32–35 , 47 , 48 ] . Many of these analyses are carried out in the d = Θ ( n ) asymptotic regime , whereas our work studies a phase transition that occurs in a much higher-dimensional regime . The notions of effective dimensions we use are present in the analyses of Bartlett et al . [ 4 ] and Muthukumar et al . [ 35 ] , and the latter work identifies regimes where SVM and OLS coincide and enjoy good performance for both classification and regression . High-dimensional geometry and universality . Our conjecture about support vector proliferation for ℓ1-SVMs derives inspiration from studies of high-dimensional geometric phase transitions , particularly those by [ 1 , 2 , 15 ] . These results consider the geometry of random polytopes . Amelunxen and Bürgisser [ 1 ] establish phase transitions on the feasibility and boundedness of the solutions to random linear programs , Amelunxen et al . [ 2 ] extend these results to characterize when ℓ1-norm minimizing solutions to sparse recovery problems are exactly correct , and Donoho and Tanner [ 15 ] bound the number of faces of random polytopes . We also borrow heavily from [ 16 ] when designing our experiments in Section 5 to test the universality hypothesis .

Summary: This manuscript investigates the phenomena of \emph{Support Vector proliferation} (SVP) [1,2], in which all points in the training set $x_{i}\in\mathbb{R}^{d}$, $i=1,\cdots, n$ become support vectors, i.e. satisfy $y_{i}\hat{w}^{\top}x_{i}=1$, where $\hat{w}\in\mathbb{R}^{d}$ is the SVM estimator and $y_{i}\in\{-1,1\}$ are the training labels. SVP implies that the SVM estimator coincides with interpolating least-norm linear regression, a fact which has been used in many recent works to study overparametrisation in generalised linear models. In particular, the authors focus in a data model defined by fixed labels $y_{i}$ and sub-Gaussian features $x_{i}$ with zero mean and diagonal covariance $\lambda_{k}$, $k=1, \cdots, d$ (i.e. the labels are independent of the features). The key results are:  - Different equivalent characterisations of SVP for generic $\ell_{p}$, $1< p< \infty$ norm SVM (proposition 1).   - Establishing a lower bound showing that one needs at least $d=n\log{n}$ for SVP to occur in the anisotropic model defined above and $\ell_{2}$-SVM  (Theorem 3). This result tightens the a previous bound from [2] (Theorem 1).  - Establishing a sharp transition at $d = 2n\log{n}$ for SVP in the particular case of isotropic Gaussian data $x_{i}\sim\mathcal{N}(0,I_{d})$, also for $\ell_{2}$-SVM (Theorem 4).  The authors also provide numerical simulations corroborating the theorems, and suggesting that their results apply beyond the theorem's assumptions. Finally, they also provide a conjecture for the equivalent result in the $\ell_1$-SVM case.   [1] Vidya Muthukumar, Adhyyan Narang, Vignesh Subramanian, Mikhail Belkin, Daniel Hsu, and Anant Sahai. *Classification vs regression in overparameterized regimes: Does the loss function matter?*, 2020.  [2]: Daniel Hsu, Vidya Muthukumar, and Ji Xu. *On the proliferation of support vectors in high dimensions*.  In Twenty-Fourth International Conference on Artificial Intelligence and Statistics, 391 2021.