An Efficient and Reliable Tolerance-Based Algorithm for Principal Component Analysis

1 INTRODUCTION . 1.1 THE TRUNCATED SVD AND PCA . Let A be an m×n real matrix and A = UΣV T its singular value decomposition ( SVD ) . The rank-k truncated SVD ( rank-k TSVD ) of A is the matrix Ak : = UkΣkV TK , where Uk and Vk are the first k columns of U and V , respectively , and Σk is the leading k × k block of Σ . The columns uj and vj of Uk and Vk are the left and right singular vectors of A , respectively , and the diagonal entries σ1 ( A ) ≥ · · · ≥ σk ( A ) ≥ 0 are the singular values of A . One common use of TSVD is in principal component analysis ( PCA ) . This is a dimensionality reduction technique that aims to find the directions in which the data varies the most . It turns out that these directions are given by the top k right singular vectors vj , 1 ≤ j ≤ k. Projecting the original data onto these directions then transforms the data from a high-dimensional space into a lower-dimensional one . Typically , k is chosen so that these principal directions explain a certain amount of variance in the data . For example , if the user wanted to explain 99 % of the variance , they would choose k so that ∑k i=1 σi ( A ) 2/ ∑n i=1 σi ( A ) 2 ≥ 0.99 . PCA is used to compress data in a variety of settings such as images , training data for machine learning algorithms , and even neural network weight matrices ( Xue et al. , 2013 ) . In this work , we will be interested in a slightly different version of PCA . Let ε be a user-prescribed tolerance , and instead choose k so that σk ( A ) ≥ ε ≥ σk+1 ( A ) . Since the ith principal component explains σi ( A ) 2/ ∑n i=1 σi ( A ) 2 fraction of the total variance , we are essentially ignoring any components that explain less than ε2/ ∑n i=1 σi ( A ) 2 fraction of the total variance . This can be interpreted as discarding principal components corresponding to noise , where ε describes the size of the noise . Despite its utility and importance , the main drawback of using TSVD for PCA is that it is expensive , especially when the user needs only the top few singular values/vectors . Thus , a large body of research has been devoted to finding faster ways of computing it without sacrificing too much accuracy . 1.2 PRIOR WORK . The literature on fast , approximate TSVD algorithms typically assume the user knows what rank k to use . Recent work uses randomization to reduce the run time while still maintaining a high level of accuracy . See , for example , Rokhlin et al . ( 2010 ) and Halko et al . ( 2010 ) and the references therein for typical examples of these types of algorithms . In Musco & Musco ( 2015 ) , the authors present a randomized algorithm based on block Krylov subspace methods to compute an approximate TSVD . For a matrix A , rank k , and tolerance ε , the algorithm produces a matrix Z whose columns approximate the top k left singular vectors of A and such that ∥∥A− ZZTA∥∥ 2 ≤ ( 1 + ε ) ∥A−Ak∥2 . They also prove stronger bounds on the quality of the singular vectors , which is important for PCA . This algorithm is especially suited to sparse matrices , which can be multiplied quickly . In some cases , the user may not know ahead of time what k to use , so it is useful to consider algorithms which accept a desired precision ε as input rather than rank . Algorithms in this vein incrementally build a matrix Q with orthogonal columns and another matrix B until ∥A−QB∥ < ε . Typically , the number of columns of Q ( or the number of rows of B ) is quite small so that QB is a compact approximation of the original data A . An approximate TSVD of A can then be produced from the SVD of B . Recent work again uses randomization to reduce the run time . We present a prototypical example of this style of algorithm in Algorithm 1 ( Yu et al. , 2018 ) . See Halko et al . ( 2010 ) and Martinsson & Voronin ( 2016 ) for more examples . While these algorithms guarantee a small approximation error , there are no guarantees on the accuracy of the singular values or vectors . We will compare the accuracy for Algorithm 1 to the proposed algorithm in Experiments . Algorithm 1 The randQB EI algorithm for the fixed-precision problem Input : an m× n matrix A ; desired accuracy tolerance ε ; block size b Output : Q , B such that ∥A−QB∥F < ε Q = [ ] ; B = [ ] ; E = ∥A∥2F for i = 1 , 2 , 3 , . . . do Ωi = randn ( n , b ) Qi = orth ( AΩi −Q ( BΩi ) ) Qi = orth ( Qi −Q ( QTQi ) ) Bi = Q T i A Q = [ Q , Qi ] B = [ B Bi ] E = E − ∥Bi∥2F if E < ε2 then stop end for 1.3 OUR WORK . In this work , we propose an algorithm that , for a matrix A , accuracy tolerance δ , and singular value tolerance ε , produces an approximate TSVD Ãk satisfying : 1 . The rank k of Ãk does not exceed the true rank of A , determined by the tolerance ε as described above , 2. σj ( Ãk ) ≥ ( 1− δ ) σj ( A ) for 1 ≤ j ≤ k , 3 . ∥∥∥A− Ãk∥∥∥ 2 ≤ 1+δ1−δ ε ≈ ( 1 + 2δ ) ε , and 4 . If k coincides with the true rank , then ∥∥∥A− Ãk∥∥∥ 2 ≤ ( 1+δ ) σk+1 ( A ) = ( 1+δ ) ∥A−Ak∥2 , i.e . the truncation error is a factor of 1 + δ from optimal . These properties are verified in the Appendix . The algorithm thus yields a high-quality approximation to TSVD and can be used in applications as an approximate PCA . The algorithm is fast for matrices whose singular values decay quickly when ε is set so that k will be relatively small . 2 PRELIMINARIES . Unless otherwise stated , we will consider matrices with more rows than columns . For a matrix with more columns than rows , apply the algorithm to its transpose . 2.1 FLIP-FLOP SPECTRUM REVEALING QR . The proposed algorithm is essentially a tolerance-based version of Flip-Flop Spectrum Revealing QR ( FFQR ) ( Feng et al. , 2019 ) . For a matrix A and integers k ≤ l , FFQR produces an approximation to the rank-k TSVD Ak whose accuracy depends on the ratio σk+1 ( A ) /σl+1 ( A ) . Thus , if A has rapidly decaying singular values , FFQR will be close to TSVD . FFQR is computed as follows . Let A be an m × n matrix ( m ≥ n ) and k ≤ l. Perform l steps of Randomized QR with Column Pivoting ( RQRCP ) ( Duersch & Gu , 2017 ) to get the factorization AΠ = QR = Q ( R11 R12 0 R22 ) , where Π is an n× n permutation matrix , Q is an m×m orthogonal matrix , R is m× n , and R11 is an l × l upper triangular matrix . The next phase of FFQR involves performing extra “ spectrum-revealing ” column swaps on R and using Givens rotations to restore its upper trapezoidal form . These swaps ensure ∥R22∥2 = O ( σl ( A ) ) . See Xiao et al . ( 2017 ) for more details . Next , perform l steps of QR on RT to get RT = PLT = ( P1 P2 ) ( L11 0 L21 L22 ) T , where P is an n× n orthogonal matrix , P1 is its leading l columns , L is an m× n matrix , and L11 is l × l lower triangular . Putting the above together yields A = QRΠT = Q ( L11 0 L21 L22 ) PTΠT . Discard L22 ( as in truncated QRCP ) and approximate ( L11 L21 ) with its rank-k TSVD ÛkΣ̂kV̂ Tk : Q ( L11 0 L21 L22 ) PTΠT ≈ Q ( L11 L21 ) PT1 Π T ≈ Q ( ÛkΣ̂kV̂ Tk ) PT1 ΠT Setting Ũk : = QÛk , Σ̃k : = Σ̂k , Ṽk : = ΠP1V̂k gives the rank-k approximation A ≈ ŨkΣ̃kṼ Tk . In Feng et al . ( 2019 ) , the authors prove the following bounds for FFQR . Given ε > 0 and g > 1 , there are matrix-dependent quantities g1 ≤ √ 1+ε 1−ε , g2 ≤ g , τ ≤ g1g2 √ ( l + 1 ) ( n− l ) , and τ̂ ≤ g1g2 √ l ( n− l ) such that for 1 ≤ j ≤ k , σj ( Σk ) ≥ σj ( A ) 4 √ 1 + min ( 2τ̂4 , τ4 ( 2 + 4τ̂4 ) ( σl+1 ( A ) σj ( A ) ) 4 ) and ∥∥∥A− ŨkΣ̃kṼ Tk ∥∥∥ 2 ≤ σk+1 ( A ) 4 √ 1 + 2τ4 ( σl+1 ( A ) σk+1 ( A ) ) 4 . 2.2 THE QLP DECOMPOSITION . The basis of FFQR is the QLP decomposition ( Stewart , 1999 ) . Let A be an m× n matrix . Perform QRCP on A to obtain AΠ = QR and then perform QRCP on RT to get RTΠ1 = PLT , where L is lower triangular . Putting these together yields A = QΠ1LPTΠT . This is the pivoted QLP decomposition of A. Stewart observed that the diagonal entries Lii of L closely track the singular values of A . For the proposed algorithm , we choose not to pivot when factoring RT . In this case , Lii will not track σi ( A ) as well . We will discuss a partial remedy for this below . The advantage of not pivoting is that , just as in FFQR , we do not have to finish computing R before computing L. Once we have performed l steps of QRCP on A , we can compute the first l rows of L and thus have access to its first l diagonal entries . 3 A FAST , APPROXIMATE , TOLERANCE-BASED PCA ALGORITHM . 3.1 BLOCKED FFQR . Since we do not know what l is beforehand , we compute R and L incrementally in blocks . Select a block size b and perform b steps of RQRCP to get AΠ1 = Q1 ( R ( b ) 11 R ( b ) 12 0 R ( b ) 22 ) , where R ( b ) 11 is b × b upper triangular . The first b rows of R are essentially done since subsequent steps of RQRCP will only permute the columns of R ( b ) 12 . Perform QR on them ( to keep the notation simple , we write this as an LQ factorization ) : ( R ( b ) 11 R ( b ) 12 ) = ( L11 0 ) P T 1 , where L11 is b × b lower triangular , and P1 is orthogonal . We have just computed the first b rows of L and know the first b diagonal entries . For the next block , continue RQRCP for another b steps . The permutation matrix Π2 in this block will affect only columns b + 1 through n , leaving the first b columns untouched . Thus , Π2 can be written in block form as Π2 = ( Ib 0 0 Π̃2 ) , where Ib is the b× b identity matrix and Π̃2 is an ( n− b ) × ( n− b ) permutation matrix . We now have AΠ1Π2 = Q2Q1 R ( b ) 11 R ( b ) 12 Π̃2 0 R ( 2b ) 11 R ( 2b ) 12 0 0 R ( 2b ) 22 , where R ( 2b ) 11 is b× b upper triangular . Since the first b rows have changed , we must account for this in the previous LQ : ( R ( b ) 11 R ( b ) 12 Π̃2 ) = ( R ( b ) 11 R ( b ) 12 ) Π2 = ( L11 0 ) P T 1 Π2 . Now apply the matrix ΠT2 P1 to the newly completed rows ( 0 R ( 2b ) 11 R ( 2b ) 12 ) and perform LQ on the last n− b columns to get ( 0 R ( 2b ) 11 R ( 2b ) 12 ) ΠT2 P1 = ( L21 L22 0 ) P T 2 , where L22 is b× b lower triangular . The orthogonal matrix P2 affects only the last n − b columns and can therefore be written in block form as P2 = ( Ib 0 0 P̃2 ) . Hence , ( L11 0 ) = ( L11 0 ) PT2 and ( R ( b ) 11 R ( b ) 12 Π̃2 0 R ( 2b ) 11 R ( 2b ) 12 ) = ( L11 0 0 L21 L22 0 ) PT2 P T 1 Π2 , showing that we have computed the first 2b rows of L. We can continue this procedure , computing b rows of L at a time . Once we decide to stop , we finish the remaining rows of L by applying the orthogonal matrices from all previous LQ factorizations to the last rows of R. For example , if we wanted to stop after 2 blocks , apply ΠT2 P1P2 to ( 0 0 R ( 2b ) 22 ) to get ( 0 0 R ( 2b ) 22 ) ΠT2 P1P2 = ( L31 L32 L33 ) and the partial QLP decomposition AΠ1Π2 = Q2Q1 R ( b ) 11 R ( b ) 12 Π̃2 0 R ( 2b ) 11 R ( 2b ) 12 0 0 R ( 2b ) 22 = Q2Q1 ( L11 0 0L21 L22 0 L31 L32 L33 ) PT2 P T 1 Π2 . Afterwards , spectrum-revealing swaps can be performed if desired . For each swap and uppertrapezoidal restoration , some nonzero entries will appear above the diagonal in L. These are easily eliminated with Givens rotations . 3.2 DETERMINING l We now derive a criterion to determine when to stop factoring in blocked FFQR and to find l. Let ε be the tolerance parameter , and define the rank k by σk+1 ( A ) ≤ ε ≤ σk ( A ) . One could use the bounds derived in Feng et al . ( 2019 ) , using the diagonal entries of L to estimate the singular values of A and stopping when σl+1 ( A ) /σk+1 ( A ) is sufficiently small . However , the dimension-dependent bounds for τ and τ̂ are impractical , so we will use a different bound . In Feng et al . ( 2019 ) , the authors prove that σj ( A ) 4 ≤ σj ( Σk ) 4+2 ∥R22∥42 , 1 ≤ j ≤ k. Rearranging this inequality gives σj ( Σk ) ≥ σj ( A ) 4 √ 1− 2 ∥R22∥42 σj ( A ) 4 , 1 ≤ j ≤ k. They also prove the following bound on the truncation error : ∥∥∥A− ŨkΣ̃kṼ Tk ∥∥∥ 2 ≤ σk+1 ( A ) 4 √ 1 + 2 ∥R22∥42 σk+1 ( A ) 4 . ( 1 ) These bounds hold even without spectrum-revealing swaps . So , if ∥R22∥2 /σk+1 ( A ) , is small , then the leading k singular values of A will be revealed up to a certain number of digits and ŨkΣ̃kṼ Tk will be a nearly optimal rank-k approximation . In practice , the above two bounds are sufficient because ∥R22∥2 = O ( σl ( A ) ) already , without extra swaps . The earlier bounds still have theoretical value in that they show the algorithm works well when A has rapidly decaying singular values . The factors 4 √ 1− 2∥R22∥ 4 2 σj ( A ) 4 and 4 √ 1 + 2 ∥R22∥42 σk+1 ( A ) 4 are equal to 1 − 12 ∥R22∥42 σj ( A ) 4 and 1 + 12 ∥R22∥42 σk+1 ( A ) 4 , respectively , up to first order . Introduce an accuracy parameter δ , and say we have 12 ∥R22∥42 σk+1 ( A ) 4 ≤ δ . Then we have σj ( Σ̃k ) ≥ σj ( A ) ( 1− δ ) , 1 ≤ j ≤ k , and ∥∥∥A− ŨkΣ̃kṼ Tk ∥∥∥ 2 ≤ σk+1 ( A ) ( 1+ δ ) up to first order . This means that ≈ − log δ digits of the top k singular values of A and optimal truncation error have been computed correctly . We can rewrite 12 ∥R22∥42 σk+1 ( A ) 4 ≤ δ as ∥R22∥2 ≤ σk+1 ( A ) 4 √ 2δ . This is the tolerance-based criterion to determine l. 3.2.1 ESTIMATING σk+1 AND ∥R22∥2 To estimate ∥R22∥2 and σk+1 ( A ) accurately , we use Stewart ’ s observation that the diagonal entries Lii of L closely track the singular values σi ( A ) of A . As stated above , Lii will not track σi ( A ) as well because we are not pivoting when factoring RT . A partial remedy is simply to sort the Lii ’ s . We show below that the resulting tracking behavior is similar in quality to that of fully pivoted QLP . Let L ( j ) be the j-th largest diagonal entry of L in magnitude , i.e . ∣∣L ( 1 ) ∣∣ ≥ ∣∣L ( 2 ) ∣∣ ≥ · · · ≥ ∣∣L ( n ) ∣∣ . In light of Stewart ’ s observation , we will assume that there are constants α and β such that α ∣∣L ( j ) ∣∣ ≤ σj ( A ) ≤ β ∣∣L ( j ) ∣∣ , 1 ≤ j ≤ n. The values of α and β will be estimated empirically below . A simple way to interpret these inequalities is that for each diagonal entry Ljj , there is a singular value of A in the interval [ α |Ljj | , β |Ljj | ] . Consider { Ljj : β |Ljj | ≤ ε } . This just corresponds to all the intervals [ α |Ljj | , β |Ljj | ] contained in ( −∞ , ε ] . For each Ljj in the set , there is a singular value σi ( A ) in the corresponding interval . Thus σi ( A ) ≤ ε . Since σk+1 ( A ) is the largest singular value of A less than or equal to ε , we must have σi ( A ) ≤ σk+1 ( A ) , which then implies α |Ljj | ≤ σk+1 ( A ) . This yields a lower bound on σk+1 ( A ) , namely max { α |Ljj | : β |Ljj | ≤ ε } . Since we will not know all the Ljj ’ s , we can obtain only a sub-optimal lower bound sk+1 on σk+1 ( A ) . Initialize sk+1 = 0 . After i blocks of blocked FFQR , update sk+1 = max { α |Ljj | : β |Ljj | ≤ ε and j ≤ ib } . To estimate ∥R22∥2 = σ1 ( R22 ) , we will use the first diagonal entry L11 of L in the fully pivoted QLP factorization . First , consider a general matrix A , and perform QRCP : AΠ = QR . Then in fully pivoted QLP , |L11| is just the largest row norm of R. As noted in Stewart ( 1999 ) , the largest row of R is usually among the first few rows . Thus we can estimate ∥A∥2 using max1≤i≤q ∥R ( i , : ) ∥2 , for some small integer q . This requires only q steps of QRCP . We can apply this idea to estimate ∥R22∥2 . After i steps of QRCP on A , the R factor has the form ( R ( i ) 1 0 R ( i ) 22 ) , where R ( i ) 1 is i×n upper triangular . The R factor in the QRCP factorization of R ( i ) 22 is just R ( n ) 1 ( i+1 : m , i+1 : n ) . Thus , QRCP-factoring A automatically yields the QRCP factorizations of all the trailing blocks R ( i ) 22 . Using the 2-norm estimation scheme in the previous paragraph , after j steps of QRCP , we have the upper bound ∥∥∥R ( i ) 22 ∥∥∥ 2 ≤ β max i≤ι≤i+q−1 ∥∥∥R ( j ) 1 ( ι , : ) ∥∥∥ 2 : = β ∥∥∥R ( i ) 22 ∥∥∥ j , q for 1 ≤ i ≤ j − q + 1 . Putting these estimates together will give us the final stopping criterion . After each block , we first update sk+1 with the newly computed Ljj ’ s and then check if ∥∥∥R ( i ) 22 ∥∥∥ j , q ≤ 1β sk+1 4 √ 2δ for some i . The smallest i for which this inequality holds will be l + 1 . Algorithm 2 Approximate , tolerance-based PCA Inputs : A , tolerance ε , accuracy δ , block size b , number of rows q , oversampling size p for RQRCP Outputs : Rank k , Ũk , Σ̃k , Ṽk c← 0 , sk+1 ← 0 while c < n do Perform steps c+ 1 to c+ b of RQRCP on A ; update Q , R , and Π Compute rows c+ 1 to c+ b of L ; update P for j = c+ 1 : c+ b do if |Ljj | ≤ ε/β and α |Ljj | ≥ sk+1 then sk+1 ← α |Ljj | end if end for for i = 1 : c+ b− q + 1 do if ∥∥∥R ( i ) 22 ∥∥∥ c+b , q ≤ 1β sk+1 4 √ 2δ then l← i− 1 exit while loop end if end for c← c+ b end while Compute rows c+ b+ 1 to m of L. Compute TSVD ÛkΣ̂kV̂k of L ( : , 1 : l ) , where k satisfies σk ( L ( : , 1 : l ) ) ≥ ε ≥ σk+1 ( L ( : , 1 : l ) ) . Ũk ← QÛk , Σ̃k ← Σ̂k , Ṽk ← ΠP1V̂k

Summary: This paper develops an algorithm that approximates the principal components corresponding to the top singular values. The algorithm requires the user specified tolerance instead of the number of top singular values, which is often the case in practice. It presents the detailed algorithm and evaluate its performance using numerical experiments.