Support vector machines and linear regression coincide with very high-dimensional features

1 Introduction . The support vector machine ( SVM ) and ordinary least squares ( OLS ) are well-weathered approaches to fitting linear models , but they are associated with different learning tasks : classification and regression . In this paper , we study the case in which the models return exactly the same hypothesis for sufficiently high-dimensional data . The hard-margin SVM is a linear classification model that finds the separating hyperplane that maximizes the minimum margin of error for every training sample . If the training data ( x1 , y1 ) , . . . , ( xn , yn ) ∈ Rd × { ±1 } are linearly separable , then the resulting linear classifier is x 7→ sign ( xTwSVM ) , where wSVM is the solution to the following optimization problem : wSVM = argmin w∈Rd ∥w∥2 such that yiw Txi ≥ 1 , ∀i ∈ [ n ] . ( 1 ) An example xi is a support vector if the corresponding constraint is satisfied with equality , and the optimal solution wSVM is a linear combination of these support vectors . Ordinary least squares regression finds the linear function that best fits the training data ( x1 , y1 ) , . . . , ( xn , yn ) ∈ Rd × R according to the sum of squared errors . When the solution is not unique , it is natural to take the solution of minimum Euclidean norm ; this is the convention we adopt . Taking X : = [ x1| . . . |xn ] T ∈ Rn×d and y : = ( y1 , . . . , yn ) , the solution is the hypothesis x 7→ wTOLSx where wOLS is the solution to the following : wOLS = argminw∈Rd ∥w∥2 such that XTXw = XTy . In many high-dimensional settings ( e.g. , where X has full row rank ) , the solution may in fact interpolate the training data , i.e. , wOLS = argmin w∈Rd ∥w∥2 such that w Txi = yi , ∀i ∈ [ n ] . ( 2 ) 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Although the optimization problems in ( 1 ) and ( 2 ) are very different , they have been observed to coincide in very high-dimensional regimes . The study of this support vector proliferation ( SVP ) phenomenon—in which every training example is a support vector—was recently initiated by Muthukumar et al . [ 35 ] and Hsu et al . [ 23 ] . Roughly speaking , they show that SVP occurs when d = Ω ( n log n ) for a broad class of sample distributions , and that SVP does not occur when d = O ( n ) in an idealized isotropic Gaussian case . SVP is a phenomenon that connects linear classification and linear regression , topics that have received renewed attention due to the break-down of classical analyses of these methods in high-dimensions . For instance , some analyses of SVM that are based on the number of support vectors become vacuous when this number becomes large [ e.g. , 19 , 20 , 44 ] . Similarly , overparameterized linear regression is typically only studied in noisy settings with explicit regularization . It was not until recently that SVM and OLS have been meaningfully analyzed in these regimes ( see Section 1.2 ) , and the connection between the two approaches via SVP has played an important analytical role [ 9 , 35 , 46 ] . In this work , we further examine support vector proliferation with the goal of broadly understanding when and why SVMs and OLS coincide . We pose and study the following questions : 1 . How general is the SVP phenomena ? What relationship between d and n determines if the solutions to ( 1 ) and ( 2 ) coincide ? We close the log n gap from the prior work of Hsu et al . [ 23 ] by showing that d ≳ n log n is necessary for SVP to occur under a model of independent subgaussian features , even with constant probability . Our lower-bounds hold for a broad class of distributions over xi , and they match the upper-bounds from [ 23 ] . This demonstrates that SVP is extremely unlikely to occur in the much-studied d = Θ ( n ) setting . 2 . Is there a sharp threshold separating the occurrence and non-occurrence of this phenomenon ? Is this threshold universal across all “ reasonable ” distributions over each xi ? We hypothesize that a sharp phase transition occurs at d = 2n log n. We rigorously prove this hypothesis for isotropic Gaussian features and quantitatively bound the width of the transition . We experimentally observe the same transition for a wide range of other distributions . 3 . Is support vector proliferation specific to the ℓ2 SVM problem ? If ( 1 ) and ( 2 ) are generalized to instead minimize ℓp norms , does this still occur at the same rate ? We re-frame this question with a geometric characterization of the dual of the SVM optimization problem for ℓp norms . We conjecture that a similar phase transition occurs for ℓ1 , but also that it requires much larger dimension d ; this is supported by preliminary experiments . 1.1 Outline of our results . Section 2 introduces the SVM and OLS approaches in full generality , our λ-anisotropic subgaussian data model , and prior results about SVP . Several equivalent characterizations of SVP are established ( Proposition 1 ) for use in subsequent sections . Section 3 characterizes when SVP does not occur for a broad range of distributions ( Theorem 3 ) . Our lower-bound on the dimension required for SVP matches the upper-bounds from [ 23 ] in the isotropic Gaussian setting , resolving the open question from that work , and also gives new lower-bounds for anisotropic cases . The proof works by tightly controlling the spectrum of the Gram matrix and establishing anti-concentration via the Berry-Esseen Theorem . Section 4 establishes a sharp threshold of d = 2nlog n for SVP in the case of isotropic Gaussian samples , and also characterize the width of the phase transition ( Theorem 4 ) . Section 5 provides empirical evidence that the sharp threshold observed in Section 4 holds for a wide range of random variables . Rigorous statistical methodology inspired by Donoho and Tanner [ 16 ] is used to test our “ universality hypothesis ” that the probability of SVP does not depend on the underlying sample distribution as d and n become large . Section 6 asks the questions about SVP from the preceding sections in the context of ℓ1-SVM and minimum ℓ1-norm interpolation . Specifically , the SVP threshold for ℓ1 is conjectured to occur for d = ω ( n log n ) . Evidence for this conjecture is provided in a simulation study and in geometric arguments about random linear programs . 1.2 Related work . Prior works connecting SVP and generalization . Muthukumar et al . [ 35 ] initiate the study of SVP in part to facilitate generalization analysis of the SVM in very high-dimensional settings . Their work , as well as the contemporaneous work of Chatterji and Long [ 10 ] , shows that the SVM enjoys low test error in certain regimes where classical learning-theoretic analyses would otherwise yield vacuous error bounds . ( In fact , one of the settings in [ 10 ] requires polynomially-higher dimension than is typically studied : d = Ω ( n2 log n ) . ) The coincidence between SVM and OLS identified by Muthukumar et al . was also more recently used by Wang and Thrampoulidis [ 46 ] and Cao et al . [ 9 ] for analyses of linear classification in very high dimensions under different data distributions . The generalization analysis of Muthukumar et al . concerns a data model inspired by the spiked covariance model of Wang and Fan [ 47 ] . They identify a regime of overparameterization where the hard-margin SVM classifier has good generalization ( i.e. , classification risk going to 0 ) even when all the training samples are supoort vectors . Our new lower bound can be regarded as establishing a limit on this approach to the analysis of SVM ; specifically , if the ( effective ) dimension is not sufficiently large , the OLS and SVM solutions may not coincide . Prior analyses of number of support vectors . Besides its relevance to generalization analysis , the number of support vectors in an SVM model is an interesting quantity to study in its own right . Hsu et al . [ 23 ] sharpen and extend the analysis of Muthukumar et al . [ 35 ] about SVP in the independent features model that we also adopt . They prove that SVM on n samples with d independent subgaussian components coincides with OLS when d = Ω ( n log n ) with probability tending to 1 . They also give a converse result stating that the coincidence fails with constant probability when d = O ( n ) in the isotropic Gaussian feature model . ( We give these results here as Theorems 1 and 2 respectively . ) Our results generalize and tighten the latter bound to tell an asymptotically sharp story about the phase transition for both isotropic and anisotropic random vectors with subgaussian components . Our specific analysis for the isotropic Gaussian case gives the exact point of the phase transition . The number of support vectors is also studied in the context of variants of SVM [ 3 , 39 ] , including the soft-margin SVM [ 12 ] and the ν-SVM [ 37 ] . In these cases , the asymptotic number of support vectors is shown to be related to the noise rate in the problem . The setups we study are linearly separable , which makes it possible to study the hard-margin SVM ( without regularization ) . The hard-margin SVM is also of interest because it captures the implicit bias of gradient descent on the logistic loss objective for linear predictors [ 25 , 38 ] . Phase transitions have been studied in the context of linear classification [ 8 , 13 , 27 , 40 ] , and SVMs in particular [ 7 , 14 , 28 , 30 ] , but most study qualitative changes in behavior other than support vector proliferation . The most relevant is the study of Buhot and Gordon [ 7 ] , who employ techniques from statistical physics to show the existence of phase transitions for the generalization error , margin size , and number of support vectors as n and d = Θ ( n ) become arbitrarily large . While they characterize the fraction of samples that are support vectors , they do not address our question about when all samples are support vectors , not just a large fraction . Indeed , our results demonstrate that their regime where d grows linearly with n will not exhibit support vector proliferation when n and d in the limit . Overparameterized linear regression . There has been a recent flurry of analyses of overparameterized linear regression models [ e.g. , 4 , 5 , 21 , 24 , 29 , 32–35 , 47 , 48 ] . Many of these analyses are carried out in the d = Θ ( n ) asymptotic regime , whereas our work studies a phase transition that occurs in a much higher-dimensional regime . The notions of effective dimensions we use are present in the analyses of Bartlett et al . [ 4 ] and Muthukumar et al . [ 35 ] , and the latter work identifies regimes where SVM and OLS coincide and enjoy good performance for both classification and regression . High-dimensional geometry and universality . Our conjecture about support vector proliferation for ℓ1-SVMs derives inspiration from studies of high-dimensional geometric phase transitions , particularly those by [ 1 , 2 , 15 ] . These results consider the geometry of random polytopes . Amelunxen and Bürgisser [ 1 ] establish phase transitions on the feasibility and boundedness of the solutions to random linear programs , Amelunxen et al . [ 2 ] extend these results to characterize when ℓ1-norm minimizing solutions to sparse recovery problems are exactly correct , and Donoho and Tanner [ 15 ] bound the number of faces of random polytopes . We also borrow heavily from [ 16 ] when designing our experiments in Section 5 to test the universality hypothesis .

Summary: This paper investigates the phenomenon of support vector proliferation (SVP), that is, when linear classification with minimum-$\ell_p$-norm hard-margin SVM and minimum-$\ell_p$-norm interpolation yield the same solution. A previous paper showed for $p=2$ that SVP occurs with high probability for certain distributions with (effective) input dimension $d = \Omega(n \log n)$, but not for $d = O(n)$. The authors close this gap by showing that SVP does not occur for $d = o(n \log n)$. The authors further specify a more precise bound for standard Gaussian inputs and provide empirical evidence showing similar behavior for other distributions. Finally, the authors present some other theoretical and empirical results to illuminate the behavior for different $p$.