Dynamic Least-Squares Regression

1 INTRODUCTION . The problem of least-squares regression ( LSR ) dates back to Gauss in 1821 ( Stigler , 1981 ) , and is the backbone of statistical inference ( Hastie et al. , 2001 ) , signal processing ( Rabiner & Gold , 1975 ) , convex optimization ( Bubeck , 2015 ) , control theory ( Chui , 1990 ) and network routing ( Lee & Sidford , 2014 ; Madry , 2013 ) . Given an overdetermined ( n d ) linear system A 2 Rn⇥d , b 2 Rn , the goal is to find the solution vector x that minimizes the mean squared error ( MSE ) min x2Rn kAx bk2 . ( 1 ) Among many other loss functions ( e.g. , ` p ) that have been studied for linear regression , ` 2-regression has been the most popular choice as it is at the same time robust to outliers , and admits a highaccuracy efficient solution . The computational task of least-squares regression arises naturally in high-dimensional statistics and has been the central of focus . The exact closed-form solution is given by the well-known Normal equation x ? = ( A > A ) 1A > b , which requires O ( nd2 ) time to compute using naive matrixmultiplication , or O ( nd ! 1 ) ⇡ O ( nd1.37 ) time using fast matrix-multiplication ( FMM ) ( Strassen , 1969 ) for the current FMM exponent of ! ⇡ 2.37 ( Le Gall , 2014 ; Alman & Williams , 2021 ) . Despite the elegance and simplicity of this closed-form solution , in practice the latter runtime is often too slow , especially in modern data analysis applications where both the dimension of the feature space ( d ) and the size of datasets ( n ) are overwhelmingly large . A more modest objective in attempt to circumvent this computational overhead , is to seek an ✏-accurate solution that satisfies kAx bk2 ( 1 + ✏ ) min x2Rd kAx bk2 . This was the primary motivation behined the development of the sketch-and-solve paradigm , where the idea is to first compress the matrix into one with fewer ( ⇠ d/✏2 ) rows and then to compute the standard LSR solution but over the smaller matrix . A long line of developments on this framework culminates in algorithms that run in close to input-sparsity time ( Sarlos , 2006 ; Clarkson & Woodruff , 2017 ; Nelson & Nguyên , 2013 ; Chepurko et al. , 2021 ) . In particular , a direct application of sketchand-solve yields an algorithm runs in eO ( nnz ( A ) ✏ 1 + d ! ) 1 , which is near optimal in the “ low 1In this paper we use eO ( · ) to hide polylogarithmic terms , and we use O✏ ( · ) to hide poly ( log d , ✏ 1 ) terms . precision ” regime ( ✏ = 1/ poly ( log d ) ) . Interestingly , when combined with more sophisticated ideas of preconditioning and ( conjugate ) gradient descent , the runtime of this algorithm in terms of the error ✏ can be further improved to eO ( nnz ( A ) log ( 1/✏ ) + d ! ) , which yields a high precision algorithm , i.e. , it can efficiently solve the problem to within polynomial accuracy ✏ = 1/ poly ( d ) . Dynamic least-squares In many real-world scenarios of the aforementioned applications , data is evolving in an online fashion either by nature or by design , and such applications require maintaining the solution ( 1 ) adaptively , where rows of the data matrix and their corresponding labels ( A ( t ) , b ( t ) ) arrive one-by-one incrementally . This is known as the dynamic least-squares regression problem . The origins of dynamic least-squares regression was in control theory of the 1950 ’ s ( Plackett , 1950 ) , in the context of dynamical linear systems . In this setup , the data matrix [ A ( t ) , b ( t ) ] corresponds to the set of measurement and it evolves in an online ( incremental ) fashion , and the goal is to efficiently maintain the ( exact ) solution to a noisy linear system b : = A ( t ) x ( t ) + ⇠ ( t ) without recomputing the LSR solution from scratch . The recursive least-squares ( RLS ) framework and the celebrated Kalman filter ( Kalman , 1960 ) provide a rather simple update rule for maintaining an exact solution for this problem , by maintaining the sample covariance matrix and using Woodburry ’ s identity ( which assert that an incremental update to [ A ( t ) , b ( t ) ] translates into a rank-1 update to the sample covariance matrix ) , and hence each update can be computed in O ( d2 ) time ( Kalman , 1960 ) . Beyond this classic motivation for dynamic LSR , a more timely motivation comes from modern deep learning applications : Most neural networks need to be frequently re-trained upon arrival on new training data , in order to improve prediction accuracy , and it is desirable to avoid recomputing weights from scratch . This problem of efficient incremental training of DNNs has been studied before in elastic machine learning ( Liberty et al. , 2020 ) and in the context of continual learning ( Parisi et al. , 2019 ) . Our work sheds light on this question by analyzing the minimal computational resources required for ` 2 loss-minimization . Despite the rich and versatile literature on static LSR , the understanding of the dynamic counterpart was so far quite limited : The previous best known result requires O ( d2 ) amortized update time ( by a direct application of the Woodbury identity ) . The basic questions we address in this papers are : Is it possible to achieve faster update time for maintaining an exact solution ? How about a smallapproximate solution – Is it then possible to achieve amortized O ( d ) or even input-sparsity time ? In this paper , we settle both of these questions and present an essentially complete characterization of the dynamic complexity of LSR . 1.1 OVERVIEW OF OUR RESULTS . Our first result is a negative answer to the first question above of maintaining exact ( or polynomialaccuracy ) LSR solutions in the dynamic setting – We prove that Kalman ’ s approach is essentially optimal , assuming the popular Online Matrix-Vector ( OMv ) Conjecture ( Henzinger et al. , 2015 ) 2 : Theorem 1.1 ( Hardness of exact dynamic LSR , informal ) . Assuming the OMv Conjecture , any dynamic algorithm that maintains an ✏ = 1/ poly ( d ) -approximate solution for the dynamic LSR problem over T = poly ( d ) iterations , must have ⌦ ( d2 o ( 1 ) ) amortized update time per iteration . Theorem 1.1 separates the static and the dynamic complexities of the exact LSR problem : As mentioned above , the static problem can be solved by batching rows together using FMM in time O ( Td ! 1 ) , whereas the dynamic problem requires ⌦ ( Td2 ) by Theorem 1.1 . Indeed , the implication Theorem 1.1 is stronger , it also separates the static and dynamic complexity of approximate LSR problem under the high precision regime , it asserts that a polylogarithmic dependence on the precision ( i.e . d poly ( log ( 1/✏ ) ) ) on update time is impossible ( assuming OMv ) , in sharp contrast to the static case . We next focus on an approximate version of this classic online problem , dynamic ✏-LSR , where the goal is to efficiently maintain , during all iterations t 2 [ T ] , an ✏-approximate solution under incremental row-updates to A ( t ) and labels b ( t ) , where efficiency is measured by the amortized 2This conjecture postulates that multiplying a fixed d ⇥ d matrix A with an online matrix B , column-bycolumn ( ABi ) , requires d3 o ( 1 ) time , in sharp contrast to the batch setting where this can be done using FMM in d ! ⌧ d3 time . See Section 4. update time for inserting a new row . A natural complexity benchmark for this dynamic problem is the aforementioned best static sketch-and-solve solution , which for n = T is eO ( nnz ( A ( T ) ) ✏ 1 + d ! ) = eO ( nnz ( A ) ✏ 1 ) for T d. Our main result is a provably efficient and practical dynamic data structure , whose total running time essentially matches the complexity of the offline problem : Theorem 1.2 ( Main result , informal version of Theorem 3.1 ) . For any accuracy parameter ✏ > 0 , there is a randomized dynamic data structure which , with probability at least 0.9 , maintains an ✏- approximate solution to the dynamic LSR problem simultaneously for all iterations t 2 [ T ] , with total update time O ( ✏ 2 nnz ( A ( T ) ) log ( T ) + ✏ 6d3 log5 ( T ) ) . Theorem 1.2 almost matches the fastest static sketching-based solution , up to polylogarithmic terms and the additive FMM term . When T d , this theorem shows that amortized update time of our algorithm is O ( d1+o ( 1 ) ) . 1.2 RELATED WORK . Sketching and sampling The least squares regression as a fundamental problem has been extensively studied in the literature . A long line of work ( Ailon & Chazelle , 2006 ; Clarkson & Woodruff , 2017 ; Nelson & Nguyên , 2013 ; Avron et al. , 2017 ; Cohen et al. , 2015 ; Woodruff , 2014 ; 2021 ) have focused on using dimension reduction technique ( sketching or sampling ) to speedup the computation task , culminates into algorithms that run in eO ( nnz ( A ) log ( 1/✏ ) + d ! ) time ( Clarkson & Woodruff , 2017 ) . See Appendix A for detailed discussions . Regression in online , streaming , and sliding window models Least-squares regressions have also been studied in various computational models , though the focus of these models are generally not the ( amortized ) running time . Our algorithm uses techniques developed by Cohen et al . ( 2020 ) , where they study the regression problem in the online model , with the goal of maintaining a spectral approximation of data matrix in the online stream . Their algorithm only needs to store O✏ ( d ) rows but the amortized running time is still ⌦ ( d2 ) ( see Section 3.1 for detailed discussions ) . In the streaming model , the main focus is the space complexity , and a direct application of random Gaussian sketch or count sketch reduces the space complexity to eO ( d2✏ 1 ) and it is shown to be tight ( Clarkson & Woodruff , 2009 ) . Recent work of ( Braverman et al. , 2020 ) studies regressions and other numerical linear algebra tasks in the sliding window model , where data come in an online stream and only the most recent updates form the underlying data set . The major focus of a sliding window model is still the space complexity , and there is no amortized running time guarantee . Disparity from online learning Our work crucially differs from online learning literature ( Hazan , 2019 ) , in that the main bottleneck in online regret-minimization and bandit problems is informationtheoretic , whereas the challenge in our loss-minimization problem is purely computational . See Appendix A for detailed discussions .

Summary: The paper considers solving the regression problem min_x |Ax-b|_2 in the online setting, where A in R^{n x d} and b in R^n are given row by row, one at each time. The main task is to maintain a good approximate solution x (meaning that |Ax-b|_2 <= (1+eps)*OPT) throughout this process, with the update time as small as possible. The paper shows that if there are T updates to A (the initial A may not be empty), the total update time will be O(eps^{-2} nnz(A) log T + poly(eps^{-1} d log T)).   The first term O(eps^{-1}nnz(A)log T) matches the runtime in a typical sketching algorithm up to the log T factor, while the second term poly(eps^{-1} d log T) is much better in the dependence on T than the runtime of a simple sketching algorithm, which would be poly(d/eps)*T. This means that the amortized cost of each update is much smaller when T >> d.  The paper also proves a lower bound of Omega(d^{2-o(1)}) amortized update time for T = poly(d) updates, assuming the Online Matrix-vector conjecture (which is on the amortized runtime of multiplying a matrix with a vector in the online setting).