Superfast CUR Approximation of Low Rank Matrices ∗

Low rank approximation of a matrix (hereafter referred to as LRA) is a fundamental subject of numerical linear algebra and data mining and analysis and is an enormously popular research subject of the last decade. The arithmetic time and memory space of the classical SVD-based optimal solution and of the faster algorithms of the 1990s have been dramatically decreased. We achieve new substantial progress by relying on our novel techniques but even more so on our new insight into the problem. We study LRA in a transparent form of CUR where C and R are submatrices made up of l input columns and k input rows, respectively, the integers k and l are small relatively to the input dimensions, and an l×k matrix U unifies the submatrices C and R in the product. Cross-approximation algorithms routinely compute accurate CUR LRAs by using much fewer memory cells and arithmetic operations than the input matrix has entries. We call such algorithms superfast, assuming that they are also superefficient in using memory space. So far, however, no one proved that any superfast algorithm outputs accurate LRA of the worst case input. We first show why such a proof has been elusive: for any matrix size m × n we specify a family of 2mn matrices such that for almost 50% of them any superfast LRA algorithm fails to improve the poor trivial approximation by the matrix filled with zeros. ∗Some results of this paper have been presented at the Workshop on Fast Direct Solvers, November 12– 13, 2016, Purdue University, West Lafayette, Indiana; the SIAM Conference on Computational Science and Engineering, February–March 2017, Atlanta, Georgia, USA, and the INdAM Meeting – Structured Matrices in Numerical Linear Algebra: Analysis, Algorithms and Applications, Cortona, Italy, September 4–8, 2017. Also see [PLSZ16]. 1 ar X iv :1 71 0. 07 94 6v 3 [ m at h. N A ] 4 N ov 2 01 7 Then in contrast, however, we prove that the class of all such hard inputs is narrow – cross-approximation and even more primitive superfast algorithms compute reasonably accurate LRAs (i) to any matrix allowing close LRA except for small norm perturbations of matrices of an algebraic variety of a smaller dimension, (ii) to the average matrix allowing close LRA, (iii) to the average sparse matrix allowing close LRA and (iv) with a high probability to any matrix allowing close LRA if it is pre-processed fast with a Gaussian, SRHT or SRFT multiplier, Moreover empirically the output LRAs remain accurate when we perform the computations superfast by replacing such a multiplier with one of our sparse and structured multipliers. Our study provides new insights into CUR LRA and reveals, demonstrates and enhances the power of simplified heuristics, the average case analysis, randomized preprocessing and cross-approximation. Our work should encourage further exploration of these techniques and their wider application. A part of our progress is due to our synergistic combination of the techniques proposed in the communities of Numerical Linear Algebra and Computer Science. This should help bring together the research efforts in the two communities. Our auxiliary results can be of independent interest. For example, we improve a decade-old estimate for the norm of the inverse of a Gaussian matrix, prove such an estimate also in the case of sparse Gaussian matrices, describe a very simple superfast algorithm for the transition from any LRA to CUR LRA and present some novel advanced pre-processing techniques for fast and superfast computation of LRA. Fast LRA algorithms are applied to various highly important computational areas, for example, to tensor decomposition. Our progress should lead to more efficient applications. Moreover superfast LRA provides new opportunities, not available for fast LRA, as we demonstrate by means of dramatic acceleration of the Conjugate Gradient algorithms and the bottleneck stage of the construction of low rank generators for the Fast Multipole Method.