The Effect of the Intrinsic Dimension on the Generalization of Quadratic Classifiers

1 Introduction . We revisit the problem of supervised classification using quadratic features of the data . We do so to highlight the influence of properties of data distrbution on the generalization error . Most of the existing results on this error only use a bound on the support of the distribution . By leveraging results from matrix concentration , we show an improved bound that uses more refined properties of the data distribution , like the second moment matrix . The use of the second moment matrix in the error bound shows that the intrinsic dimension of the data distribution plays an important role . This is of particular interest because it is widely believed that real-world data distributions have nice properties that allow classifiers , namely neural networks , to avoid the worst-case sample complexities predicted by generalization bounds Jiang * et al . ( 2020 ) . Indeed , assumptions like the manifold hypothesis , which state that the data lies on lower dimensional embedded manifold , are often made to explain the practical success of some generative methods . A recent paper by Pope et al . ( 2021 ) computes estimates of this true dimensionality of common machine learning datasets and shows that they are much lower than the ambient dimension of the pixel space [ 0 , 1 ] d. It is therefore important that properties of the data distribution , going beyond simple bounds on the support , intervene in the study of generalization . This influence of intrinsic dimension on generalization has been recently observed in the context of differentiating neural networks and from their kernel approximations , like the neural tangent kernel Jacot et al . ( 2018 ) or random feature models Yehudai and Shamir ( 2019 ) . In particular , Ghorbani et al . ( 2020 ) observe that neural networks seem to require fewer samples than kernel methods to learn when the data distribution is isotropic . We show that a similar phenomenon occurs in the simpler setting of quadratic classifiers , which leads to a better understanding of the causes . An improvement in sample complexity on isotropic data 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . distributions can be proved when comparing nuclear-norm constrained quadratic classifiers and the corresponding kernel method ( Frobenius norm constrained classifiers ) . The study of quadratic classifiers can serve as an important first step in understanding how neural networks take advantage of the intrinsic dimension to learn with fewer samples Du and Lee ( 2018 ) ; Bai and Lee ( 2020 ) . The nuclear-norm constraint is a natural one to study in this context . Indeed , when applying weight-decay ( or ` 2-regularization ) on a single-hidden layer neural network with quadratic activations , the regularization is in effect encouraging a low nuclear norm of the coefficient matrix of the quadratic polynomial . A better understanding of quadratics is also a worthwhile goal in its own right : complex architectures like those in Jayakumar et al . ( 2020 ) use quadratics as building blocks , attention layers Vaswani et al . ( 2017 ) , which have seen great success in language processing tasks , are multiplicative interactions . For these reasons , we present theoretical and practical developments of nuclear-norm regularization for quadratic classification . We summarize our contributions as follows Rademacher complexity bounds . We present a new bound on the Rademacher complexity of quadratic classifiers with a nuclear norm constraint c.f . Theorem 1 . It improves upon the previously known bound , implied by the results by Kakade et al . ( 2012 ) , by up to a square-root factor of the dimension , depending on the distribution of the data c.f . Lemma 4 . As a consequence of our bound , we draw attention to a clear difference between the complexity of nuclear-norm constrained and Frobenius-norm constrained quadratic classifiers . When the input data distribution is nearly-isotropic , the former enjoys a reduced dependency on the dimension . In contrast , the complexity of Frobenius-norm constrained classifiers has the same dependency on the dimension , independently of how isotropic the input data distribution is ( Corollary 2 ) . This observation motivates the use of data whitening pre-processing steps , which are commonly used in practice : such transformation might bring the second-order moment ( covariance ) matrix of the distribution close to the identity matrix and thus to nearly-isotropicity . Computable generalization bounds . The refined Rademacher complexity bound that we obtain depends on the often unknown second-order moment of the distribution , rather than simple bounds on the diameter of the support as in ( Kakade et al. , 2012 ) . Even though useful in theory , it is desirable in practice to obtain bounds that can be computed from a sample . We overcome this difficulty in Theorem 3 , where we provide high-probability computable generalization error bounds for nuclear-norm constrained quadratic classifiers . Experiments . We illustrate our theoretical results on synthetic data . We show how the isotropy of the input distribution plays a major role in the generalization properties of quadratic classifiers . As the dimension increases and the sample size remains proportional to it , we observe a constant generalization gap for the nuclear-norm constrained classifier . In contrast , for SVMs , the gap grows at a predicted √ d rate . In the case of anisotropic distributions , we observe similar performance for both regularization schemes . 1.1 Related work . Kakade et al . ( 2012 ) provides generalization error bounds for the more general problem of learning a linear classifier over matrices . An upper bound for quadratic classifiers with a nuclear norm constraint can be derived as a consequence of their results c.f . Corollary 1 . To the best of our knowledge , it is the only known bound for the hypothesis class we study , and thus the one we compare to . Our analysis improves the dependency on the dimension . See subsection 2.1 for a technical discussion . Because of the generality of the results in Kakade et al . ( 2012 ) , it is only natural that the implied bound in some particular case is not the tightest . We precisely give a step towards tight complexity estimates for classification with quadratic polynomials . We look on our results as relevant , given the simplicity and widespread use of linear learning over features . Wimalawarne et al . ( 2016 ) study linear classifiers over higher-order tensor spaces , using constraints on generalized notions of the nuclear norm . The problem we study is thus a particular case . Generalization error bounds via Rademacher complexity are provided , but they apply only under a highly restrictive assumption : the entries of the tensor are independent standard normal random variables . In contrast we only require a boundedness condition . Srebro et al . ( 2005 ) ; Srebro and Shraibman ( 2005 ) develop the theory of nuclear-norm regularization for matrix completion . Bounds on the Rademacher complexity of the class of matrices with bounded nuclear-norm are obtained in Srebro ( 2005 ) . However , in this setting matrices are understood as mappings from an index pair to a value , and the generalization error measures how well the missing entries of the target matrix can be predicted . In contrast , in our setting the matrix corresponds to the coefficients of a quadratic polynomial , so the bounds are not comparable . Pontil and Maurer ( 2013 ) study nuclear-norm regularization in the context of multi-task learning ( Caruana , 1997 ) . Rademacher complexity bounds are obtained for nuclear-norm constrained multitask classifiers . In this setting , each row of the matrix corresponds to a linear classifier for a different task , and each task corresponds to a different distribution over data-label pairs . Matrices with bounded trace norm also have been studied by Amit et al . ( 2007 ) and Yu et al . ( 2014 ) in the related multi-class classification and multi-label learning setting , respectively . Yu et al . ( 2014 ) remark essential differences between Rademacher complexity bounds of nuclear- and Frobenius-norm constrained linear classifiers , similar to our conclusion in Corollary 2 . In all such problems , however , the setting is not comparable to ours : the matrix acts as a linear mapping of the sample , rather than as a quadratic . Thus , the analysis is not analogous and requires a different set of tools in our case . The analysis in Yu et al . ( 2014 ) is closest in spirit to ours , as their bound also depends on the intrinsic dimension of the distribution , and the largest eigenvalue of the second moment ( covariance ) matrix . However , we go the extra mile and achieve bounds that can be computed from the sample at hand ( Theorem 3 ) , as the true second moment ( covariance ) matrix is usually unknown . The papers Du and Lee ( 2018 ) ; Bai and Lee ( 2020 ) establish a similar result as ours for a different norm for a non-convex parametrization of quadratic polynomials . The reparametrization consits of writing the coefficient matrix as a sum of m rank 1 matrices which facilitates analogies to neural networks . It is straighforward to see that their studied norm ‖ · ‖2,4 is , in essence , a way of upper bounding the nuclear-norm by using the reparametrization . 2 Rademacher complexity bounds . Notation . Throughout this section x ∈ Rd is a random variable with distribution µ , and Xn = ( x1 , . . .xn ) is a sample of i.i.d . random variables drawn from µ . The second moment matrix of µ is denoted by Σ : = E [ xxT ] . Note that , for centered ( mean zero ) random variables , this notion coincides with the covariance matrix . For a square symmetric matrix A we denote with ‖A‖F , ‖A‖2 and ‖A‖tr its Frobenius-norm , spectral-norm and nuclear-norm , respectively . The notation . stands for less than or equal , but hides constants independent of the dimension or number of samples . The notation x ≈ y means that there exist constants c , C > 0 such that cy ≤ x ≤ Cy . We consider a binary classifier obtained from a homogeneous quadratic polynomial . Such a function can be parametrized as fA ( x ) : = xTAx , where A is a square symmetric matrix containing the coefficients of the monomials . In order to control the complexity of a quadratic polynomial we choose a matrix norm ‖·‖ and consider only elements in a constrained set : Q‖·‖ , λ : = { fA ( x ) = xTAx : ‖A‖ ≤ λ } , Q‖·‖ : = Q‖·‖,1 ( 1 ) We quantify the complexity of such function classes using the classical notion of Rademacher complexity . It is well known that high probability generalization error bounds can be obtained in terms of this quantity ( Koltchinskii and Panchenko , 2002 ; Bartlett and Mendelson , 2003 ) . For this reason , we focus only on deriving upper bounds on this complexity measure . Definition 1 ( Rademacher complexity ) . Let σ be uniformly distributed over the set { −1 , 1 } n and let Xn = [ x1 , . . . , xn ] ⊆ Rd be an i.i.d . sample drawn according to µ . For a class of functions F : Rd → R we define the empirical Rademacher complexity and the Rademacher complexity ( with respect to µ ) of F , respectively , as : R̂ ( F ; Xn ) : = Eσ [ sup f∈F 1 n n∑ i=1 σif ( xi ) ] , Rn , µ ( F ) = E [ R̂ ( F ; Xn ) ] ( 2 ) Our bounds depend on the distribution through its intrinsic dimension ( Tropp , 2015 , Section 7 ) , which measures how much the probability density concentrates near low-dimensional subspaces . Definition 2 . The intrinsic dimension of a distribution µ is the ratio 1 ≤ r ( Σ ) : = tr ( Σ ) /‖Σ‖2 ≤ d. We are now ready to state our main results about the Rademacher complexity of homogeneous quadratic polynomials with nuclear norm constraint : Theorem 1 . Let x ∈ Rd such that ‖x‖22 . E‖x‖22 almost surely and suppose n & r ( Σ ) log d. It holds that Rn , µ ( Q‖·‖tr , λ ) . λ √ r ( Σ ) log d n ‖Σ‖2 ( 3 ) Now we proceed to prove Theorem 1 . First , we only focus on the class Q‖·‖ corresponding to the unit nuclear-norm ball . This is justified by well-known technical result Lemma 1 , whose proof is included for completeness in Appendix D. Lemma 1 . Rn , µ ( Q‖·‖ , λ ) ≤ λRn , µ ( Q‖·‖ ) . The backbone of Theorem 1 is Lemma 3 , which relates the Rademacher complexity of a class of functions to concentration of empirical means to expectations with respect to the dual norm . It makes use of the technical Lemma 2 , whose proof is included for completeness in Appendix B. Lemma 2 . For all even n ∈ N , it holds that ∑n k=0|2k − n| ( n k ) < √ n2n Lemma 3 . Denote by ‖·‖∗ the dual norm of ‖·‖ . DefineMk : = E ‖Σk − Σ‖∗ , Σk : = 1 k ∑k i=1 xix T i The Rademacher complexity of the class Q‖·‖ can be upper bounded as follows : Rn , µ ( Q‖·‖ ) ≤ 1 n2n−1 n∑ k=1 k ( n k ) Mk + ‖Σ‖∗√ n ( 4 ) Proof . We first compute an upper bound on the empirical Rademacher complexity . The result will follow after taking expectation of the bound over the sample Xn = [ x1 , . . . , xn ] . By definition of the dual norm , using the basic algebraic identity xTAx = 〈A , xxT 〉 we have R̂ ( Q‖·‖ ; Xn ) = 1 n Eσ sup ‖A‖≤1 〈 A , n∑ i=1 σixix T i 〉 = 1 n Eσ ∥∥∥∥∥ n∑ i=1 σixix T i ∥∥∥∥∥ ∗ ( 5 ) We now compute the expectation in eq . 5 . There is a bijection between the possible configurations of the Rademacher variable σ ∈ { −1 , 1 } n and the power set of [ n ] , namely σ 7→ { i ∈ [ n ] : σi = 1 } . This allows us to write eq . 5 as : R̂ ( Q‖·‖ ; Xn ) = 1 n Eσ ∥∥∥∥∥ n∑ i=1 σixix T i ∥∥∥∥∥ ∗ = 1 n2n ∑ B⊆ [ n ] ∥∥∥∥∥∑ i∈B xix T i − ∑ i∈Bc xix T i ∥∥∥∥∥ ∗︸ ︷︷ ︸ : =DB ( 6 ) Let ΣB : = |B|−1 ∑ i∈B xix T i . Using the triangle inequality , we can bound DB as : DB ≤|B|‖ΣB − Σ‖∗ + ||B| − |Bc||‖Σ‖∗ + |Bc|‖ΣBc − Σ‖∗ ( 7 ) To obtain a bound on the Rademacher complexity , we need now sum over B ⊆ [ n ] the terms on the right hand side of eq . 7 , and take expectation with respect to the sample Xn . First , we will deal with the sum of the second term in eq . 7 , as it is actually a deterministic value . We can sum over B ⊆ [ n ] by grouping together subsets B of the same cardinality |B| = k. We obtain : ∑ B⊆ [ n ] ||B| − |Bc||‖Σ‖∗ = n∑ k=0 ( n k ) |2k − n|‖Σ‖∗ ≤ √ n2n‖Σ‖∗ ( 8 ) where the last inequality follows from Lemma 2 . Finally , we compute the expectation of the sum over B ⊆ [ n ] of the first and third term in eq . 7 . After taking the sum , both terms become equal by symmetry . It suffices to bound the sum of the first term . Notice that because the variables x1 , . . . , xn are i.i.d. , the distribution of ΣB depends only on the size of the set B . Using the same counting argument as in eq . 8 we arrive at : E ∑ B⊆ [ n ] |B|‖ΣB − Σ‖∗ = ∑ B⊆ [ n ] |B|E‖ΣB − Σ‖∗ = n∑ k=1 k ( n k ) E‖Σk − Σ‖∗︸ ︷︷ ︸ =Mk ( 9 ) Combining the bounds in eq . 8 and eq . 9 , and dividing by n2n we obtain the result . Note that the proof of Lemma 3 follows from technical arguments but the final result is not to be found in the literature , in this form or a similar one . In particular , it is completely unrelated to the result by Vershynin ( 2011 ) with which it only shares the fairly trivial split of Rademacher random variables preceding eq . 6 . Lemma 3 provides a way to derive Rademacher complexity bounds from a bound on the expected deviations Mk defined in Lemma 3 , and might be of independent interest . In the particular case where the norm in consideration is the nuclear-norm , this lemma will be used to establish Theorem 1 as a simple application of a well-known non-asymptotic bound for the convergence of the empirical second moment ( covariance ) matrix to the true second moment ( covariance ) matrix . Proof of Theorem 1 . Recall that the dual norm of the nuclear-norm is the spectral-norm . The value of Mk in Lemma 3 measures the average deviation of the empirical second moment matrix Σk to the true Σ , in spectral-norm . Our assumption that ‖x‖22 . E‖x‖22 almost surely , implies the concentration result ( Vershynin , 2018 , Theorem 5.6.1 ) , which concludes that Mk . ( √ r ( Σ ) log d k + r ( Σ ) log d k ) ‖Σ‖2 ( 10 ) Plug this in eq . 4 , and use the bound √ k ≤ √ n for k ≤ n to obtain the inequality Rn , µ ( Q‖·‖tr ) . ( √ r ( Σ ) log d n + r ( Σ ) log d n ) ‖Σ‖2 ( 11 ) By assumption n & r ( Σ ) log d , so that the first term in eq . 11 is the largest . The second term is of smaller order and thus ends up hidden by the notation .. We conclude Rn , µ ( Q‖·‖tr ) . √ r ( Σ ) log d n ‖Σ‖2 ( 12 ) Invoking Lemma 1 we obtain the desired result .

Summary: This paper computes Rademacher complexity estimates for regularised quadratic function classes, and shows that the choice of nuclear norm regularisation makes the complexity to scale with the effective dimension of the covariance (trace / largest eigenvalue) of the underlying data distribution. Numerical experiments are also given to demonstrate this effect and show that Frobenius norm regularisation behaves differently. The results are used to bound the generalisation error of quadratic classifiers.