An Efficient and Reliable Tolerance-Based Algorithm for Principal Component Analysis

1 INTRODUCTION . 1.1 THE TRUNCATED SVD AND PCA . Let A be an m×n real matrix and A = UΣV T its singular value decomposition ( SVD ) . The rank-k truncated SVD ( rank-k TSVD ) of A is the matrix Ak : = UkΣkV TK , where Uk and Vk are the first k columns of U and V , respectively , and Σk is the leading k × k block of Σ . The columns uj and vj of Uk and Vk are the left and right singular vectors of A , respectively , and the diagonal entries σ1 ( A ) ≥ · · · ≥ σk ( A ) ≥ 0 are the singular values of A . One common use of TSVD is in principal component analysis ( PCA ) . This is a dimensionality reduction technique that aims to find the directions in which the data varies the most . It turns out that these directions are given by the top k right singular vectors vj , 1 ≤ j ≤ k. Projecting the original data onto these directions then transforms the data from a high-dimensional space into a lower-dimensional one . Typically , k is chosen so that these principal directions explain a certain amount of variance in the data . For example , if the user wanted to explain 99 % of the variance , they would choose k so that ∑k i=1 σi ( A ) 2/ ∑n i=1 σi ( A ) 2 ≥ 0.99 . PCA is used to compress data in a variety of settings such as images , training data for machine learning algorithms , and even neural network weight matrices ( Xue et al. , 2013 ) . In this work , we will be interested in a slightly different version of PCA . Let ε be a user-prescribed tolerance , and instead choose k so that σk ( A ) ≥ ε ≥ σk+1 ( A ) . Since the ith principal component explains σi ( A ) 2/ ∑n i=1 σi ( A ) 2 fraction of the total variance , we are essentially ignoring any components that explain less than ε2/ ∑n i=1 σi ( A ) 2 fraction of the total variance . This can be interpreted as discarding principal components corresponding to noise , where ε describes the size of the noise . Despite its utility and importance , the main drawback of using TSVD for PCA is that it is expensive , especially when the user needs only the top few singular values/vectors . Thus , a large body of research has been devoted to finding faster ways of computing it without sacrificing too much accuracy . 1.2 PRIOR WORK . The literature on fast , approximate TSVD algorithms typically assume the user knows what rank k to use . Recent work uses randomization to reduce the run time while still maintaining a high level of accuracy . See , for example , Rokhlin et al . ( 2010 ) and Halko et al . ( 2010 ) and the references therein for typical examples of these types of algorithms . In Musco & Musco ( 2015 ) , the authors present a randomized algorithm based on block Krylov subspace methods to compute an approximate TSVD . For a matrix A , rank k , and tolerance ε , the algorithm produces a matrix Z whose columns approximate the top k left singular vectors of A and such that ∥∥A− ZZTA∥∥ 2 ≤ ( 1 + ε ) ∥A−Ak∥2 . They also prove stronger bounds on the quality of the singular vectors , which is important for PCA . This algorithm is especially suited to sparse matrices , which can be multiplied quickly . In some cases , the user may not know ahead of time what k to use , so it is useful to consider algorithms which accept a desired precision ε as input rather than rank . Algorithms in this vein incrementally build a matrix Q with orthogonal columns and another matrix B until ∥A−QB∥ < ε . Typically , the number of columns of Q ( or the number of rows of B ) is quite small so that QB is a compact approximation of the original data A . An approximate TSVD of A can then be produced from the SVD of B . Recent work again uses randomization to reduce the run time . We present a prototypical example of this style of algorithm in Algorithm 1 ( Yu et al. , 2018 ) . See Halko et al . ( 2010 ) and Martinsson & Voronin ( 2016 ) for more examples . While these algorithms guarantee a small approximation error , there are no guarantees on the accuracy of the singular values or vectors . We will compare the accuracy for Algorithm 1 to the proposed algorithm in Experiments . Algorithm 1 The randQB EI algorithm for the fixed-precision problem Input : an m× n matrix A ; desired accuracy tolerance ε ; block size b Output : Q , B such that ∥A−QB∥F < ε Q = [ ] ; B = [ ] ; E = ∥A∥2F for i = 1 , 2 , 3 , . . . do Ωi = randn ( n , b ) Qi = orth ( AΩi −Q ( BΩi ) ) Qi = orth ( Qi −Q ( QTQi ) ) Bi = Q T i A Q = [ Q , Qi ] B = [ B Bi ] E = E − ∥Bi∥2F if E < ε2 then stop end for 1.3 OUR WORK . In this work , we propose an algorithm that , for a matrix A , accuracy tolerance δ , and singular value tolerance ε , produces an approximate TSVD Ãk satisfying : 1 . The rank k of Ãk does not exceed the true rank of A , determined by the tolerance ε as described above , 2. σj ( Ãk ) ≥ ( 1− δ ) σj ( A ) for 1 ≤ j ≤ k , 3 . ∥∥∥A− Ãk∥∥∥ 2 ≤ 1+δ1−δ ε ≈ ( 1 + 2δ ) ε , and 4 . If k coincides with the true rank , then ∥∥∥A− Ãk∥∥∥ 2 ≤ ( 1+δ ) σk+1 ( A ) = ( 1+δ ) ∥A−Ak∥2 , i.e . the truncation error is a factor of 1 + δ from optimal . These properties are verified in the Appendix . The algorithm thus yields a high-quality approximation to TSVD and can be used in applications as an approximate PCA . The algorithm is fast for matrices whose singular values decay quickly when ε is set so that k will be relatively small . 2 PRELIMINARIES . Unless otherwise stated , we will consider matrices with more rows than columns . For a matrix with more columns than rows , apply the algorithm to its transpose . 2.1 FLIP-FLOP SPECTRUM REVEALING QR . The proposed algorithm is essentially a tolerance-based version of Flip-Flop Spectrum Revealing QR ( FFQR ) ( Feng et al. , 2019 ) . For a matrix A and integers k ≤ l , FFQR produces an approximation to the rank-k TSVD Ak whose accuracy depends on the ratio σk+1 ( A ) /σl+1 ( A ) . Thus , if A has rapidly decaying singular values , FFQR will be close to TSVD . FFQR is computed as follows . Let A be an m × n matrix ( m ≥ n ) and k ≤ l. Perform l steps of Randomized QR with Column Pivoting ( RQRCP ) ( Duersch & Gu , 2017 ) to get the factorization AΠ = QR = Q ( R11 R12 0 R22 ) , where Π is an n× n permutation matrix , Q is an m×m orthogonal matrix , R is m× n , and R11 is an l × l upper triangular matrix . The next phase of FFQR involves performing extra “ spectrum-revealing ” column swaps on R and using Givens rotations to restore its upper trapezoidal form . These swaps ensure ∥R22∥2 = O ( σl ( A ) ) . See Xiao et al . ( 2017 ) for more details . Next , perform l steps of QR on RT to get RT = PLT = ( P1 P2 ) ( L11 0 L21 L22 ) T , where P is an n× n orthogonal matrix , P1 is its leading l columns , L is an m× n matrix , and L11 is l × l lower triangular . Putting the above together yields A = QRΠT = Q ( L11 0 L21 L22 ) PTΠT . Discard L22 ( as in truncated QRCP ) and approximate ( L11 L21 ) with its rank-k TSVD ÛkΣ̂kV̂ Tk : Q ( L11 0 L21 L22 ) PTΠT ≈ Q ( L11 L21 ) PT1 Π T ≈ Q ( ÛkΣ̂kV̂ Tk ) PT1 ΠT Setting Ũk : = QÛk , Σ̃k : = Σ̂k , Ṽk : = ΠP1V̂k gives the rank-k approximation A ≈ ŨkΣ̃kṼ Tk . In Feng et al . ( 2019 ) , the authors prove the following bounds for FFQR . Given ε > 0 and g > 1 , there are matrix-dependent quantities g1 ≤ √ 1+ε 1−ε , g2 ≤ g , τ ≤ g1g2 √ ( l + 1 ) ( n− l ) , and τ̂ ≤ g1g2 √ l ( n− l ) such that for 1 ≤ j ≤ k , σj ( Σk ) ≥ σj ( A ) 4 √ 1 + min ( 2τ̂4 , τ4 ( 2 + 4τ̂4 ) ( σl+1 ( A ) σj ( A ) ) 4 ) and ∥∥∥A− ŨkΣ̃kṼ Tk ∥∥∥ 2 ≤ σk+1 ( A ) 4 √ 1 + 2τ4 ( σl+1 ( A ) σk+1 ( A ) ) 4 . 2.2 THE QLP DECOMPOSITION . The basis of FFQR is the QLP decomposition ( Stewart , 1999 ) . Let A be an m× n matrix . Perform QRCP on A to obtain AΠ = QR and then perform QRCP on RT to get RTΠ1 = PLT , where L is lower triangular . Putting these together yields A = QΠ1LPTΠT . This is the pivoted QLP decomposition of A. Stewart observed that the diagonal entries Lii of L closely track the singular values of A . For the proposed algorithm , we choose not to pivot when factoring RT . In this case , Lii will not track σi ( A ) as well . We will discuss a partial remedy for this below . The advantage of not pivoting is that , just as in FFQR , we do not have to finish computing R before computing L. Once we have performed l steps of QRCP on A , we can compute the first l rows of L and thus have access to its first l diagonal entries . 3 A FAST , APPROXIMATE , TOLERANCE-BASED PCA ALGORITHM . 3.1 BLOCKED FFQR . Since we do not know what l is beforehand , we compute R and L incrementally in blocks . Select a block size b and perform b steps of RQRCP to get AΠ1 = Q1 ( R ( b ) 11 R ( b ) 12 0 R ( b ) 22 ) , where R ( b ) 11 is b × b upper triangular . The first b rows of R are essentially done since subsequent steps of RQRCP will only permute the columns of R ( b ) 12 . Perform QR on them ( to keep the notation simple , we write this as an LQ factorization ) : ( R ( b ) 11 R ( b ) 12 ) = ( L11 0 ) P T 1 , where L11 is b × b lower triangular , and P1 is orthogonal . We have just computed the first b rows of L and know the first b diagonal entries . For the next block , continue RQRCP for another b steps . The permutation matrix Π2 in this block will affect only columns b + 1 through n , leaving the first b columns untouched . Thus , Π2 can be written in block form as Π2 = ( Ib 0 0 Π̃2 ) , where Ib is the b× b identity matrix and Π̃2 is an ( n− b ) × ( n− b ) permutation matrix . We now have AΠ1Π2 = Q2Q1 R ( b ) 11 R ( b ) 12 Π̃2 0 R ( 2b ) 11 R ( 2b ) 12 0 0 R ( 2b ) 22 , where R ( 2b ) 11 is b× b upper triangular . Since the first b rows have changed , we must account for this in the previous LQ : ( R ( b ) 11 R ( b ) 12 Π̃2 ) = ( R ( b ) 11 R ( b ) 12 ) Π2 = ( L11 0 ) P T 1 Π2 . Now apply the matrix ΠT2 P1 to the newly completed rows ( 0 R ( 2b ) 11 R ( 2b ) 12 ) and perform LQ on the last n− b columns to get ( 0 R ( 2b ) 11 R ( 2b ) 12 ) ΠT2 P1 = ( L21 L22 0 ) P T 2 , where L22 is b× b lower triangular . The orthogonal matrix P2 affects only the last n − b columns and can therefore be written in block form as P2 = ( Ib 0 0 P̃2 ) . Hence , ( L11 0 ) = ( L11 0 ) PT2 and ( R ( b ) 11 R ( b ) 12 Π̃2 0 R ( 2b ) 11 R ( 2b ) 12 ) = ( L11 0 0 L21 L22 0 ) PT2 P T 1 Π2 , showing that we have computed the first 2b rows of L. We can continue this procedure , computing b rows of L at a time . Once we decide to stop , we finish the remaining rows of L by applying the orthogonal matrices from all previous LQ factorizations to the last rows of R. For example , if we wanted to stop after 2 blocks , apply ΠT2 P1P2 to ( 0 0 R ( 2b ) 22 ) to get ( 0 0 R ( 2b ) 22 ) ΠT2 P1P2 = ( L31 L32 L33 ) and the partial QLP decomposition AΠ1Π2 = Q2Q1 R ( b ) 11 R ( b ) 12 Π̃2 0 R ( 2b ) 11 R ( 2b ) 12 0 0 R ( 2b ) 22 = Q2Q1 ( L11 0 0L21 L22 0 L31 L32 L33 ) PT2 P T 1 Π2 . Afterwards , spectrum-revealing swaps can be performed if desired . For each swap and uppertrapezoidal restoration , some nonzero entries will appear above the diagonal in L. These are easily eliminated with Givens rotations . 3.2 DETERMINING l We now derive a criterion to determine when to stop factoring in blocked FFQR and to find l. Let ε be the tolerance parameter , and define the rank k by σk+1 ( A ) ≤ ε ≤ σk ( A ) . One could use the bounds derived in Feng et al . ( 2019 ) , using the diagonal entries of L to estimate the singular values of A and stopping when σl+1 ( A ) /σk+1 ( A ) is sufficiently small . However , the dimension-dependent bounds for τ and τ̂ are impractical , so we will use a different bound . In Feng et al . ( 2019 ) , the authors prove that σj ( A ) 4 ≤ σj ( Σk ) 4+2 ∥R22∥42 , 1 ≤ j ≤ k. Rearranging this inequality gives σj ( Σk ) ≥ σj ( A ) 4 √ 1− 2 ∥R22∥42 σj ( A ) 4 , 1 ≤ j ≤ k. They also prove the following bound on the truncation error : ∥∥∥A− ŨkΣ̃kṼ Tk ∥∥∥ 2 ≤ σk+1 ( A ) 4 √ 1 + 2 ∥R22∥42 σk+1 ( A ) 4 . ( 1 ) These bounds hold even without spectrum-revealing swaps . So , if ∥R22∥2 /σk+1 ( A ) , is small , then the leading k singular values of A will be revealed up to a certain number of digits and ŨkΣ̃kṼ Tk will be a nearly optimal rank-k approximation . In practice , the above two bounds are sufficient because ∥R22∥2 = O ( σl ( A ) ) already , without extra swaps . The earlier bounds still have theoretical value in that they show the algorithm works well when A has rapidly decaying singular values . The factors 4 √ 1− 2∥R22∥ 4 2 σj ( A ) 4 and 4 √ 1 + 2 ∥R22∥42 σk+1 ( A ) 4 are equal to 1 − 12 ∥R22∥42 σj ( A ) 4 and 1 + 12 ∥R22∥42 σk+1 ( A ) 4 , respectively , up to first order . Introduce an accuracy parameter δ , and say we have 12 ∥R22∥42 σk+1 ( A ) 4 ≤ δ . Then we have σj ( Σ̃k ) ≥ σj ( A ) ( 1− δ ) , 1 ≤ j ≤ k , and ∥∥∥A− ŨkΣ̃kṼ Tk ∥∥∥ 2 ≤ σk+1 ( A ) ( 1+ δ ) up to first order . This means that ≈ − log δ digits of the top k singular values of A and optimal truncation error have been computed correctly . We can rewrite 12 ∥R22∥42 σk+1 ( A ) 4 ≤ δ as ∥R22∥2 ≤ σk+1 ( A ) 4 √ 2δ . This is the tolerance-based criterion to determine l. 3.2.1 ESTIMATING σk+1 AND ∥R22∥2 To estimate ∥R22∥2 and σk+1 ( A ) accurately , we use Stewart ’ s observation that the diagonal entries Lii of L closely track the singular values σi ( A ) of A . As stated above , Lii will not track σi ( A ) as well because we are not pivoting when factoring RT . A partial remedy is simply to sort the Lii ’ s . We show below that the resulting tracking behavior is similar in quality to that of fully pivoted QLP . Let L ( j ) be the j-th largest diagonal entry of L in magnitude , i.e . ∣∣L ( 1 ) ∣∣ ≥ ∣∣L ( 2 ) ∣∣ ≥ · · · ≥ ∣∣L ( n ) ∣∣ . In light of Stewart ’ s observation , we will assume that there are constants α and β such that α ∣∣L ( j ) ∣∣ ≤ σj ( A ) ≤ β ∣∣L ( j ) ∣∣ , 1 ≤ j ≤ n. The values of α and β will be estimated empirically below . A simple way to interpret these inequalities is that for each diagonal entry Ljj , there is a singular value of A in the interval [ α |Ljj | , β |Ljj | ] . Consider { Ljj : β |Ljj | ≤ ε } . This just corresponds to all the intervals [ α |Ljj | , β |Ljj | ] contained in ( −∞ , ε ] . For each Ljj in the set , there is a singular value σi ( A ) in the corresponding interval . Thus σi ( A ) ≤ ε . Since σk+1 ( A ) is the largest singular value of A less than or equal to ε , we must have σi ( A ) ≤ σk+1 ( A ) , which then implies α |Ljj | ≤ σk+1 ( A ) . This yields a lower bound on σk+1 ( A ) , namely max { α |Ljj | : β |Ljj | ≤ ε } . Since we will not know all the Ljj ’ s , we can obtain only a sub-optimal lower bound sk+1 on σk+1 ( A ) . Initialize sk+1 = 0 . After i blocks of blocked FFQR , update sk+1 = max { α |Ljj | : β |Ljj | ≤ ε and j ≤ ib } . To estimate ∥R22∥2 = σ1 ( R22 ) , we will use the first diagonal entry L11 of L in the fully pivoted QLP factorization . First , consider a general matrix A , and perform QRCP : AΠ = QR . Then in fully pivoted QLP , |L11| is just the largest row norm of R. As noted in Stewart ( 1999 ) , the largest row of R is usually among the first few rows . Thus we can estimate ∥A∥2 using max1≤i≤q ∥R ( i , : ) ∥2 , for some small integer q . This requires only q steps of QRCP . We can apply this idea to estimate ∥R22∥2 . After i steps of QRCP on A , the R factor has the form ( R ( i ) 1 0 R ( i ) 22 ) , where R ( i ) 1 is i×n upper triangular . The R factor in the QRCP factorization of R ( i ) 22 is just R ( n ) 1 ( i+1 : m , i+1 : n ) . Thus , QRCP-factoring A automatically yields the QRCP factorizations of all the trailing blocks R ( i ) 22 . Using the 2-norm estimation scheme in the previous paragraph , after j steps of QRCP , we have the upper bound ∥∥∥R ( i ) 22 ∥∥∥ 2 ≤ β max i≤ι≤i+q−1 ∥∥∥R ( j ) 1 ( ι , : ) ∥∥∥ 2 : = β ∥∥∥R ( i ) 22 ∥∥∥ j , q for 1 ≤ i ≤ j − q + 1 . Putting these estimates together will give us the final stopping criterion . After each block , we first update sk+1 with the newly computed Ljj ’ s and then check if ∥∥∥R ( i ) 22 ∥∥∥ j , q ≤ 1β sk+1 4 √ 2δ for some i . The smallest i for which this inequality holds will be l + 1 . Algorithm 2 Approximate , tolerance-based PCA Inputs : A , tolerance ε , accuracy δ , block size b , number of rows q , oversampling size p for RQRCP Outputs : Rank k , Ũk , Σ̃k , Ṽk c← 0 , sk+1 ← 0 while c < n do Perform steps c+ 1 to c+ b of RQRCP on A ; update Q , R , and Π Compute rows c+ 1 to c+ b of L ; update P for j = c+ 1 : c+ b do if |Ljj | ≤ ε/β and α |Ljj | ≥ sk+1 then sk+1 ← α |Ljj | end if end for for i = 1 : c+ b− q + 1 do if ∥∥∥R ( i ) 22 ∥∥∥ c+b , q ≤ 1β sk+1 4 √ 2δ then l← i− 1 exit while loop end if end for c← c+ b end while Compute rows c+ b+ 1 to m of L. Compute TSVD ÛkΣ̂kV̂k of L ( : , 1 : l ) , where k satisfies σk ( L ( : , 1 : l ) ) ≥ ε ≥ σk+1 ( L ( : , 1 : l ) ) . Ũk ← QÛk , Σ̃k ← Σ̂k , Ṽk ← ΠP1V̂k

Summary: This paper presents a PCA algorithm that terminates after approximate singular values fall below a user provided threshold. The proposed method is based on the FFQR algorithm of Feng et al, but includes a tolerance based stopping criteria. It is stated that the algorithm runs $O(mnl)$ time. Experimental results are provided for comparison between TSVD and randQB_EI of Yue et al.