Fast Regression for Structured Inputs

1 INTRODUCTION . Given a matrix A ∈ Rn×d and a vector b ∈ Rn , the goal of linear regression is to find a vector x ∈ Rd that best fits A to b . However for d < n , the linear system is overconstrained and thus there may not exist a vector x such that Ax = b . Thus the Lp regression problem seeks to find a vector x̃ ∈ Rd such that ‖Ax̃− b‖p ≤ ( 1 + ε ) min x∈Rd ‖Ax− b‖p , for some approximation parameter ε > 0 . Here for a vector y ∈ Rn , ‖y‖pp = ∑n i=1 |yi|p . The Lp regression problem is a central question in statistical data analysis that has numerous applications in machine learning , data science , and applied mathematics ( Friedman et al. , 2001 ; Chatterjee & Hadi , 2006 ) and thus has received a considerable amount of attention . A direct way to solve the Lp regression problem is to use an iterative solver ( Adil et al. , 2019b ; a ) that makes poly ( n ) calls to a linear system solver . However , the linear system solver uses time poly ( n , d ) , which can be prohibitively slow for n d. Therefore , a standard approach for Lp regression is to first sample a number of rows of A down to a smaller matrix M that generally has poly ( d , 1ε , log n ) rows by generating a sampling and rescaling matrix S such that M = SA and ‖SAx − Sb‖p is a “ good ” approximation to ‖Ax − b‖p for all x ∈ Rd . Subsequently , an iterative solver can now be used on M , which serves as a proxy for the original problem but has smaller input size , thus allowing for more efficient computation . Unfortunately , to handle worst-case inputs , M requires sampling O ( dmax ( 1 , p/2 ) ) rows , so that not only is the overall runtime exponential in p , but also the sampling is only helpful if n dp/2 . Fortunately , there are many applications in which the design matrix A is known to have additional structure that can be leveraged to design more efficient algorithms . For example , Vandermonde matrices are used in the polynomial regression problem , which has been studied for over 200 years ( Gergonne , 1974 ) and recently motivated by applications to machine learning ( Kalai et al. , 2008 ) , applied statistics ( MacNeill , 1978 ) , and computer graphics ( Pratt , 1987 ) . The goal is to fit a continuous signal , which can be measured at time choices t1 , . . . , tn to form a measurement vector b ∈ Rn , using the best degree d polynomial whose coefficients can be represented by the vector x ∈ Rd+1 . Thus , entry Ai , j of the feature matrix A is the value of the j-th polynomial basis at time ti so that the problem has been restated as the Lp regression problem with a Vandermonde matrix A , where the entries of each row ai form a sequence 1 , ti , ( ti ) 2 , . . . , ( ti ) d−1 for some parameter ti . Regression problems with Vandermonde matrices also arise in the settings of Fourier fitting ( Avron et al. , 2019 ) and Toeplitz covariance estimation ( Eldar et al. , 2020 ) . Shi & Woodruff ( 2019 ) show that for Vandermonde matrices A , the time used to compute the smaller subsampled matrix M can be improved upon the naı̈ve approach by using fast Vandermonde matrix-vector multiplication . However , the number of rows of M is still exponential in p so that unfortunately , the runtime of the iterative solver for Lp regression still uses time exponential in p , which suffices for small p but crucially does not obtain any improvements for the important problem of L∞ regression . 1.1 OUR CONTRIBUTIONS . We first show that for Lp regression on Vandermonde matrices and its noisy perturbations , the size of the subsampled matrix M and therefore the runtime of the iterative solver only needs polynomial dependence in p. Theorem 1.1 . Given ε ∈ ( 0 , 1 ) and p ≥ 1 , a Vandermonde matrix A ∈ Rn×d , and b ∈ Rd , there exists an algorithm that creates a sampling matrix S ∈ Rm×n with m = O ( p2d ε3 log 2 n ) rows so that with high probability , ( 1− ε ) ‖Ax− b‖p ≤ ‖SAx− Sb‖p ≤ ( 1 + ε ) ‖Ax− b‖p , for all x ∈ Rd . The algorithm uses O ( n log2 n ) + p3d3/2 poly ( 1 ε , log n ) total time and with high probability , returns a vector x̂ ∈ Rd such that ‖Ax̂− b‖p ≤ ( 1 + ε ) min x∈Rd ‖Ax− b‖p . The best known previous work has runtime either ( 1 ) O ( n log2 n+ dq · poly ( 1/ε ) ) time ( Shi & Woodruff , 2019 ) , where q = ω ∈ [ 2 , 3 ) is the exponent for matrix multiplication for p ∈ [ 1 , 4 ] and q = p/2 + C for some fixed constant C > 0 for p > 4 , which has an exponential dependence in p , or ( 2 ) using poly ( n ) solves on the n × d linear system Adil et al . ( 2019b ; a ) . By contrast , our work only has a polynomial dependence on p. In fact , we show that to obtain a ( 1 + ε ) -approximation to the fundamental problem of L∞ regression , it suffices to consider Lp regression for p = O ( logn ε ) and since our results have polynomial dependence on p rather than exponential , we obtain sublinear size coresets for L∞ regression : Theorem 1.2 . Given ε ∈ ( 0 , 1 ) , a Vandermonde matrix A ∈ Rn×d , and b ∈ Rd , there exists an algorithm that uses O ( n log2 n ) + d3/2 poly ( 1 ε , log n ) time that computes a sampling matrix S ∈ Rm×n with m = d poly ( 1 ε , log n ) such that with high probability , ( 1− ε ) ‖Ax− b‖∞ ≤ ‖SAx− Sb‖∞ ≤ ( 1 + ε ) ‖Ax− b‖∞ for all x ∈ Rd . In particular , min x∈Rd ‖SAx− Sb‖∞ ≤ ( 1 + ε ) min x∈Rd ‖Ax− b‖∞ . To the best of our knowledge , this is the first known dimensionality reduction for L∞ regression with provable guarantees for any ( nontrivial ) input matrix . We introduce the following definition to capture the “ true dimension ” of regression problems for structured input matrices . Definition 1.3 ( Rank of Regression Problem ) . Given an integer p ≥ 1 and a matrix A ∈ Rn×d , suppose there exists a matrix M ∈ Rn×t and a fixed function f : Rd → Rt so that for all x ∈ Rd , |〈ai , x〉|p = |〈mi , f ( x ) 〉| . Then we call t the rank of the Lp regression problem . Our result relies on the following key structural property that we prove about the p-fold tensor product of rows of the Vandermonde matrix . Lemma 1.4 . For integer p ≥ 1 , the rank of the Lp regression problem on a Vandermonde matrix A ∈ Rn×d is O ( dp ) . Lemma 1.4 implies that the Lp loss function for a row of a Vandermonde matrix can potentially be expressed as a linear combination of O ( dp ) variables even though the entries of the measurement vector b can be arbitrary . By comparison , the Lp loss function for a row ai of a general matrix A is |〈ai , x〉 − bi|pp can only be expressed as a linear combination ofO ( pdp ) variables , corresponding to each of the dk k-wise product of coordinates of x , for each k ∈ [ p ] . As a corollary of Lemma 1.4 , we obtain a small coreset for Lp regression ( as well as L∞ regression ) on a Vandermonde matrix , which can thus be used as a preconditioner . We summarize these results in Table 1 . We generalize Lemma 1.4 to similar guarantees for Lp regression on a matrix A that is the sum of a low-rank matrix L and a sparse matrix S that has at most s non-zero entries per row . Theorem 1.5 . Given ε ∈ ( 0 , 1 ) and p ≥ 1 , a matrix A ∈ Rn×d such that A = K + S for a rank k matrix K and a s-sparse matrix S , and b ∈ Rd , there exists an algorithm that uses O ( nk + ns ) + poly ( ds , kp , sp , 1ε ) time , and with high probability , returns a vector x̂ ∈ Rd with ‖Ax̂− b‖p ≤ ( 1 + ε ) min x∈Rd ‖Ax− b‖p . Similarly , we obtain efficient guarantees for Lp regression on a matrix A that is the sum of a Vandermonde matrix V and a sparse matrix S that has at most s non-zero entries per row . Theorem 1.6 . Given ε ∈ ( 0 , 1 ) and p ≥ 1 , a matrix A ∈ Rn×d such that A = V + S for a Vandermonde matrix V and an s-sparse matrix S , and b ∈ Rd , there exists an algorithm that usesO ( T ( A ) ) log n+ poly ( p2d , ds , sp , log n , 1ε ) ) time and with high probability , returns a vector x̂ ∈ Rd such that ‖Ax̂− b‖p ≤ ( 1 + ε ) min x∈Rd ‖Ax− b‖p . Surprisingly , our methods even yield practical algorithms for general matrices with absolutely no structure . Although the rank of the Lp regression problem for general matrices is O ( dp ) , we still obtain the optimal sample complexity ( see Li et al . ( 2021 ) for a lower bound ) of roughly O ( dp/2 ) . Furthermore , the advantage of our approach is that we now only need to perform Lq Lewis weight sampling for q ∈ [ 1 , 4 ] . There are known efficient iterative methods for computing the Lq Lewis weights for q ∈ [ 1 , 4 ] , e.g. , see Section 2.1 or Section 3 in Cohen & Peng ( 2015 ) . By contrast , previous methods relied on computing general Lp weights for p > 4 , which required solving a large convex program , e.g. , through semidefinite programming , see Section 4 in Cohen & Peng ( 2015 ) . Finally , we experimentally validate our theory on synthetic data . In particular , we consider matrices and response vectors that are motivated by existing lower bounds for subsampling and sketching methods . For Vandermonde regression , our experiments demonstrate that the number of rows needed is subexponential in p , reinforcing how structured matrices outperform the worst-case bound . For unstructured matrix Lp regression , we demonstrate that our Lq Lewis Weight subsampling scheme is effective and accurate . For instance , we can find a near-optimal solution vector by solving the regression problem on just 2 % of the input matrix ’ s rows .

Summary: This paper studies ways of subsampling tall-and-dense p-norm regression involving structured Vandermonde matrices. It shows that in this setting with additional structure, sampling by Lewis weights produces poly(p) * d row sized samples for all values of p. Theoretically, this has two major advantages: it avoids more complicated computations of Lewis weights for higher values of p, and the sample size required is significantly better than the d^{p/2} bound for general matrices.  The algorithm is empirically evaluated on 25k-by-10 synthetic matrices, where it was observed that about 1k rows suffice to give sketches that incur at most 10% relative error. Furthermore, additional experiments provide cases where this type of sampling do significantly better than sampling uniformly.