A generalization of the randomized singular value decomposition

1 INTRODUCTION . Computing the singular value decomposition ( SVD ) is a fundamental linear algebra task in machine learning ( Paterek , 2007 ) , statistics ( Wold et al. , 1987 ) , and signal processing ( Alter et al. , 2000 ; Van Der Veen et al. , 1993 ) . The SVD of an m × n real matrix A with m ≥ n is a factorization of the form A = UΣV∗ , where U is an m × m orthogonal matrix of left singular vectors , Σ is an m × n diagonal matrix with entries σ1 ( A ) ≥ · · · ≥ σn ( A ) ≥ 0 , and V is an n × n orthogonal matrix of right singular vectors ( Golub & Van Loan , 2013 ) . The SVD plays a central role because truncating it after k terms provides a best rank k approximation to A in the spectral and Frobenius norm ( Eckart & Young , 1936 ; Mirsky , 1960 ) . Since computing the SVD of a large matrix can be computationally infeasible , there are various alternative algorithms that compute near-best rank k matrix approximations from matrix-vector products ( Halko et al. , 2011 ; Martinsson & Tropp , 2020 ; Nakatsukasa , 2020 ; Nyström , 1930 ; Williams & Seeger , 2001 ) . The randomized SVD uses matrixvector products with standard Gaussian random vectors and is one of the most popular algorithms for constructing a low-rank approximation to A ( Halko et al. , 2011 ; Martinsson & Tropp , 2020 ) . Currently , the randomized SVD is used and theoretically justified when it uses matrix-vector products with standard Gaussian random vectors . In this paper , we consider the following generalizations . Generalization 1 . We generalize the randomized SVD when the matrix-vector products are with multivariate Gaussian random vectors . Our theory allows for multivariate Gaussian random input vectors that have a general symmetric positive semi-definite covariance matrix . A key novelty of our work is that prior knowledge of A can be exploited to design covariance matrices that achieve lower approximation errors than the randomized SVD with standard Gaussian vectors . Generalization 2 . We generalize the randomized SVD to Hilbert–Schmidt ( HS ) operators ( Hsing & Eubank , 2015 ) . We design a practical algorithm for learning HS operators using random input functions , sampled from a Gaussian process ( GP ) . Examples of applications include learning integral kernels such as Green ’ s functions associated with linear partial differential equations ( Boullé & Townsend , 2022 ; Boullé et al. , 2021 ) . The choice of the covariance kernel in the GP is crucial and impacts both the theoretical bounds and numerical results of the randomized SVD . This leads us to introduce a new covariance kernel based on weighted Jacobi polynomials for learning HS operators . One of the main advantages of this kernel is that it is directly expressed as a Karhunen–Loève expansion ( Karhunen , 1946 ; Loève , 1946 ) so that it is faster to sample functions from the associated GP than using a standard squaredexponential kernel . In addition , we show that the smoothness of the functions sampled from a GP with the Jacobi kernel can be controlled as it is related to the decay rate of the kernel ’ s eigenvalues . Contributions . We summarize our novel contributions as follows : 1 . We provide new theoretical bounds for the randomized SVD for matrices or HS operators when using random input vectors generated from any multivariate Gaussian distribution . This shows when it is beneficial to use nonstandard Gaussian random vectors in the randomized SVD for constructing low-rank approximations . 2 . We generalize the randomized SVD to HS operators and provide numerical examples to learn integral kernels . 3 . We propose a covariance kernel based on weighted Jacobi polynomials and show that one can select the smoothness of the sampled random functions by choosing the decay rate of the kernel eigenvalues . 2 BACKGROUND : THE RANDOMIZED SVD FOR MATRICES . The randomized SVD computes a near-best rank k approximation to a matrixA . First , one performs the matrix-vector products y1 = Ax1 , . . . , yk+p = Axk+p , where x1 , . . . , xk+p are standard Gaussian random vectors with identically and independently distributed entries and p ≥ 1 is an oversampling parameter . Then , one computes the economized QR factorization [ y1 · · · yk+p ] = QR , before forming the rank ≤ k + p approximant QQ∗A . Note that if A is symmetric , one can form QQ∗A by computing Q ( AQ ) ∗ via matrix-vector products involving A . The quality of the rank ≤ k + p approximant QQ∗A is characterized by the following theorem . Theorem 1 ( Halko et al . ( 2011 ) ) Let A be an m × n matrix , k ≥ 1 an integer , and choose an oversampling parameter p ≥ 4 . If Ω ∈ Rn× ( k+p ) is a standard Gaussian random matrix and QR = AΩ is the economized QR decomposition of AΩ , then for all u , t ≥ 1 , ‖A−QQ∗A‖F ≤ ( 1 + t √ 3k p+ 1 ) √√√√ n∑ j=k+1 σ2j ( A ) + ut √ k + p p+ 1 σk+1 ( A ) , ( 1 ) with failure probability at most 2t−p + e−u 2 . The squared tail of the singular values of A , i.e. , √∑n j=k+1 σ 2 j ( A ) , gives the best rank k approxi- mation error to A in the Frobenius norm . This result shows that the randomized SVD can compute a near-best low-rank approximation to A with high probability . 3 GENERALIZED RANDOMIZED SVD FOR MATRICES AND OPERATORS . The theory behind the randomized SVD has been recently extended to nonstandard covariance matrices and HS operators ( Boullé & Townsend , 2022 ) . However , the probability bounds , generalizing Theorem 1 , are not sharp enough to emphasize the improved performance of covariance matrices with prior information over the standard randomized SVD . We provide new bounds for GPs with nonstandard covariance matrices in Theorem 2 . An upper bound on the expectation is also available in the Appendix ( see Lemma 7 ) . While Theorem 2 is formulated with matrices , the same result holds for HS operators in infinite dimensions . For a fixed target rank 1 ≤ k ≤ n , we define V1 ∈ Rn×k and V2 ∈ Rn× ( n−k ) to be the matrices containing the first k and last n − k right singular vectors of A , respectively , and denote by Σ2 ∈ R ( n−k ) × ( n−k ) , the diagonal matrix with entries σk+1 ( A ) , . . . , σn ( A ) . We consider a symmetric positive semi-definite covariance matrix K ∈ Rn×n , with kth largest eigenvalue λk > 0 . Theorem 2 Let A be an m × n matrix , k ≥ 1 an integer , and choose an oversampling parameter p ≥ 4 . If Ω ∈ Rn× ( k+p ) is a Gaussian random matrix , where each column is sampled from a multivariate Gaussian distribution with covariance matrix K ∈ Rn×n , and QR = AΩ is the economized QR decomposition of AΩ , then for all u , t ≥ 1 , ‖A−QQ∗A‖F ≤ ( 1 + ut √ ( k + p ) 3k p+ 1 βk γk ) √√√√ n∑ j=k+1 σ2j ( A ) , ( 2 ) with failure probability at most t−p + [ ue− ( u 2−1 ) /2 ] k+p . Here , γk = k/ ( λ1 Tr ( ( V∗1KV1 ) −1 ) ) ) denotes the covariance quality factor , and βk = Tr ( Σ22V ∗ 2KV2 ) / ( λ1‖Σ2‖2F ) , where λ1 is the largest eigenvalue of K. The factors γk and βk , measuring the quality of the covariance matrix to learn A in Theorem 2 , can be respectively bounded ( Boullé & Townsend 2022 , Lem . 2 ; Lemma 9 ) using the eigenvalues λ1 ≥ · · · ≥ λn of the covariance matrix K and the singular values of A as : 1 γk ≤ 1 k n∑ j=n−k+1 λ1 λj , βk ≤ n∑ j=k+1 λj−k λ1 σ2j ( A ) / n∑ j=k+1 σ2j ( A ) . This shows that the performance of the generalized randomized SVD depends on the decay rate of the sequence { λj } . The quantities αk and βk depend on how much prior information of the k+ 1 , . . . , n right singular vectors of A is encoded in K. In the ideal situation where these singular vectors are known , then one can define K such that βk = 0 for λk+1 = · · · = λn = 0 . In particular , this highlights that a suitably chosen covariance matrix can outperform the randomized SVD with standard Gaussian vectors ( see Section 5.1 for a numerical example ) . 3.1 RANDOMIZED SVD FOR HILBERT–SCHMIDT OPERATORS . We now describe the randomized SVD for learning HS operators ( see Algorithm 1 ) . The algorithm is implemented in the Chebfun software system ( Driscoll et al. , 2014 ) , which is a MATLAB package for computing with functions . The Chebfun implementation of the randomized SVD for HS operators uses Chebfun ’ s capabilities , which offer continuous analogues of several matrix operations like the QR decomposition and numerical integration . Indeed , the continuous analogue of a matrixvector multiplication AΩ for an HS integral operator F ( see Hsing & Eubank , 2015 for definitions and properties of HS operators ) , with kernel G : D ×D → R , is ( Ff ) ( x ) = ∫ D G ( x , y ) f ( y ) dy , x ∈ D , f ∈ L2 ( D ) , where D ⊂ Rd with d ≥ 1 , and L2 ( D ) is the space of square-integrable functions on D. Algorithm 1 Randomized SVD for HS operators Input : HS integral operator F with kernel G ( x , y ) , number of samples k > 0 Output : Approximation Gk of G 1 : Define a GP covariance kernel K 2 : Sample the GP k times to generate a quasimatrix of random functions Ω = [ f1 . . . fk ] 3 : Evaluate the integral operator at Ω , Y = [ F ( f1 ) . . .F ( fk ) ] 4 : Orthonormalize the columns of Y , Q = orth ( Y ) = [ q1 . . . qk ] 5 : Compute an approximation to G by evaluating the adjoint of F 6 : Initialize Gk ( x , y ) to 0 7 : for i = 1 : k do 8 : Gk ( x , y ) ← Gk ( x , y ) + qi ( x ) ∫ D G ( z , y ) qi ( z ) dz 9 : end for The algorithm takes as input an integral operator that we aim to approximate . Note that we focus here on learning an integral operator , but other HS operators would work similarly . The first step of the randomized SVD for HS operators consists of generating an∞× k quasimatrix Ω by sampling a GP k times , where k is the target rank ( see Section 4 ) . An ∞ × k quasimatrix is an ordered set of k functions ( Townsend & Trefethen , 2015 ) , and generalizes the notion of matrix to infinite dimensions . Therefore , each column of Ω is an object , consisting of a polynomial approximation of a smooth random function sampled from the GP in the Chebyshev basis . After evaluating the HS operator at Ω to obtain a quasimatrix Y , we use the QR algorithm ( Townsend & Trefethen , 2015 ) to obtain an orthonormal basis Q for the range of the columns of Y . Then , the randomized SVD for HS operators requires the left-evaluation of the operator F or , equivalently , the evaluation of its adjoint Ft satisfying : ( Ftf ) ( x ) = ∫ D G ( y , x ) f ( y ) dy , x ∈ D. We evaluate the adjoint of F at each column vector of Q to construct an approximation Gk of G. Finally , the approximation error between the operator kernel G and the learned kernel Gk can be computed in the L2-norm , corresponding to the HS norm of the integral operator .

Summary: This paper proposes a generalization approach for the randomized SVD. In the existing approach of the randomized SVD, a standard Gaussian random matrix is used to reduce the size of a matrix to perform SVD. The proposed approach uses a multivariate Gaussian distribution with a covariance matrix instead of the standard Gaussian random matrix in performing SVD. Since the covariance matrix can reflect prior knowledge of the given matrix, the proposed approach can improve the approximation quality of the randomized SVD. By using a synthetic dataset, this paper conducts experiments to show the effectiveness of the proposed approach.