Fast Regression for Structured Inputs

1 INTRODUCTION . Given a matrix A ∈ Rn×d and a vector b ∈ Rn , the goal of linear regression is to find a vector x ∈ Rd that best fits A to b . However for d < n , the linear system is overconstrained and thus there may not exist a vector x such that Ax = b . Thus the Lp regression problem seeks to find a vector x̃ ∈ Rd such that ‖Ax̃− b‖p ≤ ( 1 + ε ) min x∈Rd ‖Ax− b‖p , for some approximation parameter ε > 0 . Here for a vector y ∈ Rn , ‖y‖pp = ∑n i=1 |yi|p . The Lp regression problem is a central question in statistical data analysis that has numerous applications in machine learning , data science , and applied mathematics ( Friedman et al. , 2001 ; Chatterjee & Hadi , 2006 ) and thus has received a considerable amount of attention . A direct way to solve the Lp regression problem is to use an iterative solver ( Adil et al. , 2019b ; a ) that makes poly ( n ) calls to a linear system solver . However , the linear system solver uses time poly ( n , d ) , which can be prohibitively slow for n d. Therefore , a standard approach for Lp regression is to first sample a number of rows of A down to a smaller matrix M that generally has poly ( d , 1ε , log n ) rows by generating a sampling and rescaling matrix S such that M = SA and ‖SAx − Sb‖p is a “ good ” approximation to ‖Ax − b‖p for all x ∈ Rd . Subsequently , an iterative solver can now be used on M , which serves as a proxy for the original problem but has smaller input size , thus allowing for more efficient computation . Unfortunately , to handle worst-case inputs , M requires sampling O ( dmax ( 1 , p/2 ) ) rows , so that not only is the overall runtime exponential in p , but also the sampling is only helpful if n dp/2 . Fortunately , there are many applications in which the design matrix A is known to have additional structure that can be leveraged to design more efficient algorithms . For example , Vandermonde matrices are used in the polynomial regression problem , which has been studied for over 200 years ( Gergonne , 1974 ) and recently motivated by applications to machine learning ( Kalai et al. , 2008 ) , applied statistics ( MacNeill , 1978 ) , and computer graphics ( Pratt , 1987 ) . The goal is to fit a continuous signal , which can be measured at time choices t1 , . . . , tn to form a measurement vector b ∈ Rn , using the best degree d polynomial whose coefficients can be represented by the vector x ∈ Rd+1 . Thus , entry Ai , j of the feature matrix A is the value of the j-th polynomial basis at time ti so that the problem has been restated as the Lp regression problem with a Vandermonde matrix A , where the entries of each row ai form a sequence 1 , ti , ( ti ) 2 , . . . , ( ti ) d−1 for some parameter ti . Regression problems with Vandermonde matrices also arise in the settings of Fourier fitting ( Avron et al. , 2019 ) and Toeplitz covariance estimation ( Eldar et al. , 2020 ) . Shi & Woodruff ( 2019 ) show that for Vandermonde matrices A , the time used to compute the smaller subsampled matrix M can be improved upon the naı̈ve approach by using fast Vandermonde matrix-vector multiplication . However , the number of rows of M is still exponential in p so that unfortunately , the runtime of the iterative solver for Lp regression still uses time exponential in p , which suffices for small p but crucially does not obtain any improvements for the important problem of L∞ regression . 1.1 OUR CONTRIBUTIONS . We first show that for Lp regression on Vandermonde matrices and its noisy perturbations , the size of the subsampled matrix M and therefore the runtime of the iterative solver only needs polynomial dependence in p. Theorem 1.1 . Given ε ∈ ( 0 , 1 ) and p ≥ 1 , a Vandermonde matrix A ∈ Rn×d , and b ∈ Rd , there exists an algorithm that creates a sampling matrix S ∈ Rm×n with m = O ( p2d ε3 log 2 n ) rows so that with high probability , ( 1− ε ) ‖Ax− b‖p ≤ ‖SAx− Sb‖p ≤ ( 1 + ε ) ‖Ax− b‖p , for all x ∈ Rd . The algorithm uses O ( n log2 n ) + p3d3/2 poly ( 1 ε , log n ) total time and with high probability , returns a vector x̂ ∈ Rd such that ‖Ax̂− b‖p ≤ ( 1 + ε ) min x∈Rd ‖Ax− b‖p . The best known previous work has runtime either ( 1 ) O ( n log2 n+ dq · poly ( 1/ε ) ) time ( Shi & Woodruff , 2019 ) , where q = ω ∈ [ 2 , 3 ) is the exponent for matrix multiplication for p ∈ [ 1 , 4 ] and q = p/2 + C for some fixed constant C > 0 for p > 4 , which has an exponential dependence in p , or ( 2 ) using poly ( n ) solves on the n × d linear system Adil et al . ( 2019b ; a ) . By contrast , our work only has a polynomial dependence on p. In fact , we show that to obtain a ( 1 + ε ) -approximation to the fundamental problem of L∞ regression , it suffices to consider Lp regression for p = O ( logn ε ) and since our results have polynomial dependence on p rather than exponential , we obtain sublinear size coresets for L∞ regression : Theorem 1.2 . Given ε ∈ ( 0 , 1 ) , a Vandermonde matrix A ∈ Rn×d , and b ∈ Rd , there exists an algorithm that uses O ( n log2 n ) + d3/2 poly ( 1 ε , log n ) time that computes a sampling matrix S ∈ Rm×n with m = d poly ( 1 ε , log n ) such that with high probability , ( 1− ε ) ‖Ax− b‖∞ ≤ ‖SAx− Sb‖∞ ≤ ( 1 + ε ) ‖Ax− b‖∞ for all x ∈ Rd . In particular , min x∈Rd ‖SAx− Sb‖∞ ≤ ( 1 + ε ) min x∈Rd ‖Ax− b‖∞ . To the best of our knowledge , this is the first known dimensionality reduction for L∞ regression with provable guarantees for any ( nontrivial ) input matrix . We introduce the following definition to capture the “ true dimension ” of regression problems for structured input matrices . Definition 1.3 ( Rank of Regression Problem ) . Given an integer p ≥ 1 and a matrix A ∈ Rn×d , suppose there exists a matrix M ∈ Rn×t and a fixed function f : Rd → Rt so that for all x ∈ Rd , |〈ai , x〉|p = |〈mi , f ( x ) 〉| . Then we call t the rank of the Lp regression problem . Our result relies on the following key structural property that we prove about the p-fold tensor product of rows of the Vandermonde matrix . Lemma 1.4 . For integer p ≥ 1 , the rank of the Lp regression problem on a Vandermonde matrix A ∈ Rn×d is O ( dp ) . Lemma 1.4 implies that the Lp loss function for a row of a Vandermonde matrix can potentially be expressed as a linear combination of O ( dp ) variables even though the entries of the measurement vector b can be arbitrary . By comparison , the Lp loss function for a row ai of a general matrix A is |〈ai , x〉 − bi|pp can only be expressed as a linear combination ofO ( pdp ) variables , corresponding to each of the dk k-wise product of coordinates of x , for each k ∈ [ p ] . As a corollary of Lemma 1.4 , we obtain a small coreset for Lp regression ( as well as L∞ regression ) on a Vandermonde matrix , which can thus be used as a preconditioner . We summarize these results in Table 1 . We generalize Lemma 1.4 to similar guarantees for Lp regression on a matrix A that is the sum of a low-rank matrix L and a sparse matrix S that has at most s non-zero entries per row . Theorem 1.5 . Given ε ∈ ( 0 , 1 ) and p ≥ 1 , a matrix A ∈ Rn×d such that A = K + S for a rank k matrix K and a s-sparse matrix S , and b ∈ Rd , there exists an algorithm that uses O ( nk + ns ) + poly ( ds , kp , sp , 1ε ) time , and with high probability , returns a vector x̂ ∈ Rd with ‖Ax̂− b‖p ≤ ( 1 + ε ) min x∈Rd ‖Ax− b‖p . Similarly , we obtain efficient guarantees for Lp regression on a matrix A that is the sum of a Vandermonde matrix V and a sparse matrix S that has at most s non-zero entries per row . Theorem 1.6 . Given ε ∈ ( 0 , 1 ) and p ≥ 1 , a matrix A ∈ Rn×d such that A = V + S for a Vandermonde matrix V and an s-sparse matrix S , and b ∈ Rd , there exists an algorithm that usesO ( T ( A ) ) log n+ poly ( p2d , ds , sp , log n , 1ε ) ) time and with high probability , returns a vector x̂ ∈ Rd such that ‖Ax̂− b‖p ≤ ( 1 + ε ) min x∈Rd ‖Ax− b‖p . Surprisingly , our methods even yield practical algorithms for general matrices with absolutely no structure . Although the rank of the Lp regression problem for general matrices is O ( dp ) , we still obtain the optimal sample complexity ( see Li et al . ( 2021 ) for a lower bound ) of roughly O ( dp/2 ) . Furthermore , the advantage of our approach is that we now only need to perform Lq Lewis weight sampling for q ∈ [ 1 , 4 ] . There are known efficient iterative methods for computing the Lq Lewis weights for q ∈ [ 1 , 4 ] , e.g. , see Section 2.1 or Section 3 in Cohen & Peng ( 2015 ) . By contrast , previous methods relied on computing general Lp weights for p > 4 , which required solving a large convex program , e.g. , through semidefinite programming , see Section 4 in Cohen & Peng ( 2015 ) . Finally , we experimentally validate our theory on synthetic data . In particular , we consider matrices and response vectors that are motivated by existing lower bounds for subsampling and sketching methods . For Vandermonde regression , our experiments demonstrate that the number of rows needed is subexponential in p , reinforcing how structured matrices outperform the worst-case bound . For unstructured matrix Lp regression , we demonstrate that our Lq Lewis Weight subsampling scheme is effective and accurate . For instance , we can find a near-optimal solution vector by solving the regression problem on just 2 % of the input matrix ’ s rows .

Summary: This paper considers the problem of $L_p$-regression. I.e. given an $n\times d$ matrix $A$ and a vector $b\in\mathbb{R}^n$, compute an approximation $\tilde{x}\in\mathbb{R}^d$ such that ||A\tilde{x}-b||_p <= (1+\epsilon)\min_{x\in\mathbb{R}^d} ||Ax-b||_p. In particular, the focus of this paper is on the case of large $p$ and when the matrix $A$ has structure (specifically when A is the Vandermonde matrix or matrices that are sum of sparse and Vandermonde/low rank matrices) and the goal is to perform the task in sub/near-linear time.   This paper focuses on the well studied method of sub-sampling the rows of $A$ (and re-scaling it) to reduce the problem size where more expensive iterative methods can be applied.  For simplicity we only talk about the results for the case of A being Vandermonde matrix. The results in this paper bring down the previous best know runtime bound of $O(n\log^2{n})+d^{p/2+C}\mathrm{poly}(1/\epsilon)$ (for $p>4$ and some fixed constant $C$) to $O(n\log^2{n})+p^3d^{3/2}\mathrm{poly}(1/\epsilon,\log{n})$. Thus, for values $d >n^{2/p}$, this presents an improvement. The improvement is especially stark when we take $p=\infty$, for which case this paper presents the _first_ sublinear-time $L_\infty$ regression.  Technically the results in the paper follow from roughly three steps. The first step is to argue that for the structured matrices considered in this paper the rank of the regression problem is much smaller than the bound of $d^{\Omega(p)}$ for arbitrary $A$. This gives the desired result if $b=\mathbf{0}$ but to handle general $b$ needs more work. In particular, if all the entries of $b$ were roughly the same then the rank result is enough. Thus, it seems like the bulk of the technical work is in showing how to round and truncate $b$ and then in the third step solve the regression problem in "groups" where in each group, the sub-vector of $b$ is essentially "fixed."  There are some experimental results presented that validate the results in the paper on Vandermonde matrix as well as results for general matrices (where the paper represents a quadratic improvement over the best-known results).