Dynamic Least-Squares Regression

1 INTRODUCTION . The problem of least-squares regression ( LSR ) dates back to Gauss in 1821 ( Stigler , 1981 ) , and is the backbone of statistical inference ( Hastie et al. , 2001 ) , signal processing ( Rabiner & Gold , 1975 ) , convex optimization ( Bubeck , 2015 ) , control theory ( Chui , 1990 ) and network routing ( Lee & Sidford , 2014 ; Madry , 2013 ) . Given an overdetermined ( n d ) linear system A 2 Rn⇥d , b 2 Rn , the goal is to find the solution vector x that minimizes the mean squared error ( MSE ) min x2Rn kAx bk2 . ( 1 ) Among many other loss functions ( e.g. , ` p ) that have been studied for linear regression , ` 2-regression has been the most popular choice as it is at the same time robust to outliers , and admits a highaccuracy efficient solution . The computational task of least-squares regression arises naturally in high-dimensional statistics and has been the central of focus . The exact closed-form solution is given by the well-known Normal equation x ? = ( A > A ) 1A > b , which requires O ( nd2 ) time to compute using naive matrixmultiplication , or O ( nd ! 1 ) ⇡ O ( nd1.37 ) time using fast matrix-multiplication ( FMM ) ( Strassen , 1969 ) for the current FMM exponent of ! ⇡ 2.37 ( Le Gall , 2014 ; Alman & Williams , 2021 ) . Despite the elegance and simplicity of this closed-form solution , in practice the latter runtime is often too slow , especially in modern data analysis applications where both the dimension of the feature space ( d ) and the size of datasets ( n ) are overwhelmingly large . A more modest objective in attempt to circumvent this computational overhead , is to seek an ✏-accurate solution that satisfies kAx bk2 ( 1 + ✏ ) min x2Rd kAx bk2 . This was the primary motivation behined the development of the sketch-and-solve paradigm , where the idea is to first compress the matrix into one with fewer ( ⇠ d/✏2 ) rows and then to compute the standard LSR solution but over the smaller matrix . A long line of developments on this framework culminates in algorithms that run in close to input-sparsity time ( Sarlos , 2006 ; Clarkson & Woodruff , 2017 ; Nelson & Nguyên , 2013 ; Chepurko et al. , 2021 ) . In particular , a direct application of sketchand-solve yields an algorithm runs in eO ( nnz ( A ) ✏ 1 + d ! ) 1 , which is near optimal in the “ low 1In this paper we use eO ( · ) to hide polylogarithmic terms , and we use O✏ ( · ) to hide poly ( log d , ✏ 1 ) terms . precision ” regime ( ✏ = 1/ poly ( log d ) ) . Interestingly , when combined with more sophisticated ideas of preconditioning and ( conjugate ) gradient descent , the runtime of this algorithm in terms of the error ✏ can be further improved to eO ( nnz ( A ) log ( 1/✏ ) + d ! ) , which yields a high precision algorithm , i.e. , it can efficiently solve the problem to within polynomial accuracy ✏ = 1/ poly ( d ) . Dynamic least-squares In many real-world scenarios of the aforementioned applications , data is evolving in an online fashion either by nature or by design , and such applications require maintaining the solution ( 1 ) adaptively , where rows of the data matrix and their corresponding labels ( A ( t ) , b ( t ) ) arrive one-by-one incrementally . This is known as the dynamic least-squares regression problem . The origins of dynamic least-squares regression was in control theory of the 1950 ’ s ( Plackett , 1950 ) , in the context of dynamical linear systems . In this setup , the data matrix [ A ( t ) , b ( t ) ] corresponds to the set of measurement and it evolves in an online ( incremental ) fashion , and the goal is to efficiently maintain the ( exact ) solution to a noisy linear system b : = A ( t ) x ( t ) + ⇠ ( t ) without recomputing the LSR solution from scratch . The recursive least-squares ( RLS ) framework and the celebrated Kalman filter ( Kalman , 1960 ) provide a rather simple update rule for maintaining an exact solution for this problem , by maintaining the sample covariance matrix and using Woodburry ’ s identity ( which assert that an incremental update to [ A ( t ) , b ( t ) ] translates into a rank-1 update to the sample covariance matrix ) , and hence each update can be computed in O ( d2 ) time ( Kalman , 1960 ) . Beyond this classic motivation for dynamic LSR , a more timely motivation comes from modern deep learning applications : Most neural networks need to be frequently re-trained upon arrival on new training data , in order to improve prediction accuracy , and it is desirable to avoid recomputing weights from scratch . This problem of efficient incremental training of DNNs has been studied before in elastic machine learning ( Liberty et al. , 2020 ) and in the context of continual learning ( Parisi et al. , 2019 ) . Our work sheds light on this question by analyzing the minimal computational resources required for ` 2 loss-minimization . Despite the rich and versatile literature on static LSR , the understanding of the dynamic counterpart was so far quite limited : The previous best known result requires O ( d2 ) amortized update time ( by a direct application of the Woodbury identity ) . The basic questions we address in this papers are : Is it possible to achieve faster update time for maintaining an exact solution ? How about a smallapproximate solution – Is it then possible to achieve amortized O ( d ) or even input-sparsity time ? In this paper , we settle both of these questions and present an essentially complete characterization of the dynamic complexity of LSR . 1.1 OVERVIEW OF OUR RESULTS . Our first result is a negative answer to the first question above of maintaining exact ( or polynomialaccuracy ) LSR solutions in the dynamic setting – We prove that Kalman ’ s approach is essentially optimal , assuming the popular Online Matrix-Vector ( OMv ) Conjecture ( Henzinger et al. , 2015 ) 2 : Theorem 1.1 ( Hardness of exact dynamic LSR , informal ) . Assuming the OMv Conjecture , any dynamic algorithm that maintains an ✏ = 1/ poly ( d ) -approximate solution for the dynamic LSR problem over T = poly ( d ) iterations , must have ⌦ ( d2 o ( 1 ) ) amortized update time per iteration . Theorem 1.1 separates the static and the dynamic complexities of the exact LSR problem : As mentioned above , the static problem can be solved by batching rows together using FMM in time O ( Td ! 1 ) , whereas the dynamic problem requires ⌦ ( Td2 ) by Theorem 1.1 . Indeed , the implication Theorem 1.1 is stronger , it also separates the static and dynamic complexity of approximate LSR problem under the high precision regime , it asserts that a polylogarithmic dependence on the precision ( i.e . d poly ( log ( 1/✏ ) ) ) on update time is impossible ( assuming OMv ) , in sharp contrast to the static case . We next focus on an approximate version of this classic online problem , dynamic ✏-LSR , where the goal is to efficiently maintain , during all iterations t 2 [ T ] , an ✏-approximate solution under incremental row-updates to A ( t ) and labels b ( t ) , where efficiency is measured by the amortized 2This conjecture postulates that multiplying a fixed d ⇥ d matrix A with an online matrix B , column-bycolumn ( ABi ) , requires d3 o ( 1 ) time , in sharp contrast to the batch setting where this can be done using FMM in d ! ⌧ d3 time . See Section 4. update time for inserting a new row . A natural complexity benchmark for this dynamic problem is the aforementioned best static sketch-and-solve solution , which for n = T is eO ( nnz ( A ( T ) ) ✏ 1 + d ! ) = eO ( nnz ( A ) ✏ 1 ) for T d. Our main result is a provably efficient and practical dynamic data structure , whose total running time essentially matches the complexity of the offline problem : Theorem 1.2 ( Main result , informal version of Theorem 3.1 ) . For any accuracy parameter ✏ > 0 , there is a randomized dynamic data structure which , with probability at least 0.9 , maintains an ✏- approximate solution to the dynamic LSR problem simultaneously for all iterations t 2 [ T ] , with total update time O ( ✏ 2 nnz ( A ( T ) ) log ( T ) + ✏ 6d3 log5 ( T ) ) . Theorem 1.2 almost matches the fastest static sketching-based solution , up to polylogarithmic terms and the additive FMM term . When T d , this theorem shows that amortized update time of our algorithm is O ( d1+o ( 1 ) ) . 1.2 RELATED WORK . Sketching and sampling The least squares regression as a fundamental problem has been extensively studied in the literature . A long line of work ( Ailon & Chazelle , 2006 ; Clarkson & Woodruff , 2017 ; Nelson & Nguyên , 2013 ; Avron et al. , 2017 ; Cohen et al. , 2015 ; Woodruff , 2014 ; 2021 ) have focused on using dimension reduction technique ( sketching or sampling ) to speedup the computation task , culminates into algorithms that run in eO ( nnz ( A ) log ( 1/✏ ) + d ! ) time ( Clarkson & Woodruff , 2017 ) . See Appendix A for detailed discussions . Regression in online , streaming , and sliding window models Least-squares regressions have also been studied in various computational models , though the focus of these models are generally not the ( amortized ) running time . Our algorithm uses techniques developed by Cohen et al . ( 2020 ) , where they study the regression problem in the online model , with the goal of maintaining a spectral approximation of data matrix in the online stream . Their algorithm only needs to store O✏ ( d ) rows but the amortized running time is still ⌦ ( d2 ) ( see Section 3.1 for detailed discussions ) . In the streaming model , the main focus is the space complexity , and a direct application of random Gaussian sketch or count sketch reduces the space complexity to eO ( d2✏ 1 ) and it is shown to be tight ( Clarkson & Woodruff , 2009 ) . Recent work of ( Braverman et al. , 2020 ) studies regressions and other numerical linear algebra tasks in the sliding window model , where data come in an online stream and only the most recent updates form the underlying data set . The major focus of a sliding window model is still the space complexity , and there is no amortized running time guarantee . Disparity from online learning Our work crucially differs from online learning literature ( Hazan , 2019 ) , in that the main bottleneck in online regret-minimization and bandit problems is informationtheoretic , whereas the challenge in our loss-minimization problem is purely computational . See Appendix A for detailed discussions .

Summary: This paper studies incremental least-squares regression, where the goal is to maintain an $(1+\epsilon)$-approximate solution to $\min_x \left\Vert Ax-b\right\Vert_2^2$ for some $A\in\mathbb{R}^{n\times d}$, under row insertions to $\begin{pmatrix}A & b\end{pmatrix}$, while keeping the total runtime as low as possible.   It builds on a work of Cohen et al. (2020), where it is shown that if, upon insertion of a row, we only keep (a multiple of) it with probability roughly equal to its current leverage score (otherwise we discard it), then the total number of rows that will be inserted is only $\widetilde{O}(d)$ and the matrix spectrally approximates the "true" matrix where all rows are kept. The main contribution of this paper is to make this algorithm faster, by improving the leverage score computation procedure. Instead of directly computing the leverage scores, the authors use the Johnson-Lindenstrauss lemma to be able to quickly approximate them. Then the main thing to be taken care of is maintaining certain dimension-reduced matrices that arise. In total the asymptotic amortized runtime goes down from $\widetilde{O}(d\cdot nnz(A))$ to $\widetilde{O}(nnz(A))$.   Additionally, the authors give a negative result, which shows that, under the OMv conjecture, the amortized runtime is $\Omega(d^2)$ if we require a high-precision solution ($\epsilon=1/\mathrm{poly}(d)$).  The theoretical result is accompanied by some synthetic and real experiments. In both, it seems that the runtime improvement is significant (e.g. up to 4x speedup in the real data when $\epsilon=1$).