Neural Networks for Principal Component Analysis: A New Loss Function Provably Yields Ordered Exact Eigenvectors 

1 INTRODUCTION . Ranking among the most widely-used and valuable statistical tools , Principal Component Analysis ( PCA ) represents a given set of data within a new orthogonal coordinate system in which the data are uncorrelated and the variance of the data along each orthogonal axis is successively ordered from the highest to lowest . The projection of data along each axis gives what are called principal components . Theoretically , eigendecomposition of the covariance matrix provides exactly such a transformation . For large data sets , however , classical decomposition techniques are infeasible and other numerical methods , such as least squares approximation schemes , are practically employed . An especially notable instance is the problem of dimensionality reduction , where only the largest principal components—as the best representative of the data—are desired . Linear autoencoders ( LAEs ) are one such scheme for dimensionality reduction that is applicable to large data sets . An LAE with a single fully-connected and linear hidden layer , and Mean Squared Error ( MSE ) loss function can discover the linear subspace spanned by the principal components . This subspace is the same as the one spanned by the weights of the decoder . However , it failure to identify the exact principal directions . This is due to the fact that , when the encoder is transformed by some matrix , transforming the decoder by the inverse of that matrix will yield no change in the loss . In other words , the loss possesses a symmetry under the action of a group of invertible matrices , so that directions ( and orderings/permutations thereto ) will not be discriminated . The early work of Bourlard & Kamp ( 1988 ) and Baldi & Hornik ( 1989 ) connected LAEs and PCA and demonstrated the lack of identifiability of principal components . Several methods for neural networks compute the exact eigenvectors ( Rubner & Tavan , 1989 ; Xu , 1993 ; Kung & Diamantaras , 1990 ; Oja et al. , 1992 ) , but they depend on either particular network structures or special optimization methods . It was recently observed ( Plaut , 2018 ; Kunin et al. , 2019 ) that regularization causes the left singular vectors of the decoder to become the exact eigenvectors , but recovering them still requires an extra decomposition step . As Plaut ( 2018 ) point out , no existent method recovers the eigenvectors from an LAE in an optimization-independent way on a standard network — this work fills that void . Moreover , analyzing the loss surface for various architectures of linear/non-linear neural networks is a highly active and prominent area of research ( e.g . Baldi & Hornik ( 1989 ) ; Kunin et al . ( 2019 ) ; Pretorius et al . ( 2018 ) ; Frye et al . ( 2019 ) ) . Most of these works extend the results of Baldi & Hornik ( 1989 ) for shallow LAEs to more complex networks . However , most retain the original MSE loss , and they prove the same critical point characterization for their specific architecture of interest . Most notably Zhou & Liang ( 2018 ) extends the results of Baldi & Hornik ( 1989 ) to deep linear networks and shallow RELU networks . In contrast in this work we are going after a loss with better loss surface properties . We propose a new loss function for performing PCA using LAEs . We show that with the proposed loss function , the decoder converges to the exact ordered unnormalized eigenvectors of the sample covariance matrix . The idea is simple : for identifying p principal directions we build up a total loss function as a sum of p squared error losses , where the ith loss function identifies only the first i principal directions . This approach breaks the symmetry since minimizing the first loss results in the first principal direction , which forces the second loss to find the first and the second . This constraint is propagated through the rest of the losses , resulting in all p principal components being identified . For the new loss we prove that all local minima are global minima . Consequently , the proposed loss function has both theoretical and practical implications . Theoretically , it provides better understanding of the loss surface . Specifically , we show that any critical point of our loss L is a critical point of the original MSE loss but not vice versa , and conclude that L eliminates those undesirable global minima of the original loss ( i.e. , exactly those which suffer from the invariance ) . Given that the set of critical points of L is a subset of critical points of MSE loss , many of the previous work on loss surfaces of more complex networks likely extend . In light of the removal of undesirable global minima through L , examining more complex networks is certainly a very promising direction . As for practical consequences , we show that the loss and its gradients can be compactly vectorized so that their computational complexity is no different from the MSE loss . Therefore , the loss L can be used to perform PCA/SVD on large datasets using any method of optimization such as Stochastic Gradient Descent ( SGD ) . Chief among the compellingly reasons to perform PCA/SVD using this method is that , in recent years , there has been unprecedented gains in the performance of very large SGD optimizations , with autoencoders in particular successfully handling larger numbers of high-dimensional training data ( e.g. , images ) . The loss function we offer is attractive in terms of parallelizability and distributability , and does not prescribe any single specific algorithm or implementation , so stands to continue to benefit from the arms race between SGD and its competitors . More importantly , this single loss function ( without an additional post hoc processing step ) fits seamlessly into optimization pipelines ( where SGD is but one instance ) . The result is that the loss allows for PCA/SVD computation as single optimization layer , akin to an instance of a fully differentiable building block in a NN pipeline Amos & Kolter ( 2017 ) , potentially as part of a much larger network . 2 THE PROPOSED LOSS FUNCTION AND REVIEW OF FINAL RESULTS . Let X ∈ Rn×m and Y ∈ Rn×m be the input and output matrices , where m centered sample points , each n-dimensional , are stacked column-wise . Let xj ∈ Rn and yj ∈ Rn be the jth sample input and output ( i.e . the jth column of X and Y , respectively ) . Define the loss function L ( A , B ) as L ( A , B ) : = p∑ i=1 m∑ j=1 ‖yj −AIi ; pBxj‖22 = p∑ i=1 ‖Y −AIi ; pBX‖2F ( 1 ) where 〈· , ·〉F and ‖·‖F are the Frobenius inner product and norm , Ii ; p is a p× p matrix with all elements zero except the first i diagonal elements being one . ( Or , equivalently , the matrix obtained by setting the last p− i diagonal elements of a p×p identity matrix to zero , e.g . I2 ; 3 = [ 1 0 0 0 1 0 0 0 0 ] . ) In what follows , we shall denote the transpose of matrix M by M ′ . Moreover , the matrices A ∈ Rn×p , and B ∈ Rp×n can be viewed as the weights of the decoder and encoder parts of an LAE . The results are based on the following standard assumptions that hold generically : Assumption 1 . For an input X and an output Y , let Σxx : = XX ′ , Σxy : = XY ′ , Σyx : = Σ′xy and Σyy = Y Y ′ be their sample covariance matrices . We assume • The input and output data are centered ( zero mean ) . • Σxx , Σxy , Σyx and Σyy are positive definite ( of full rank and invertible ) . • The covariance matrix Σ : = ΣyxΣ−1xxΣxy is of full rank with n distinct eigenvalues λ1 > λ2 > · · · > λn . • The decoder matrix A has no zero columns . Claim . The main result of this work proved in Theorem 2 is as follows : If the above assumptions hold then all the local minima of L ( A , B ) are achieved iff A and B are of the form A = U1 : pDp B = D−1p U ′ 1 : pΣyxΣ −1 xx , where the ith column of U1 : p is the unit eigenvector of Σ : = ΣyxΣ−1xxΣxy corresponding to the i th largest eigenvalue and Dp is a diagonal matrix with nonzero diagonal elements . In other words , A contains ordered unnormalized eigenvectors of Σ corresponding to the p largest eigenvalues . Moreover , all the local minima are global minima with the value of the loss function at those global minima being L ( A , B ) = p Tr ( Σyy ) − p∑ i=1 ( p− i+ 1 ) λi , where λi is the ith largest eigenvalue of Σ : = ΣyxΣ−1xxΣxy . In the case of autoencoder ( Y = X ) : Σ = Σxx . Finally , while L ( A , B ) in the given form containsO ( p ) matrix products , we will show that it can be evaluated with constant ( less than 7 ) matrix products independent of the value p . 3 NOTATION . In this paper , the underlying field is always R , and positive semidefinite matrices are symmetric by definition . The following constant matrices are used extensively throughout . The matrices Tp ∈ Rp×p and Sp ∈ Rp×p are defined as ( Tp ) ij = ( p− i+ 1 ) δij , i.e . Tp = diag ( p , p− 1 , · · · , 1 ) , ( 2 ) ( Sp ) ij=p−max ( i , j ) +1 , i.e . Sp= p p− 1 · · · 2 1 p− 1 p− 1 · · · 2 1 ... ... . . . 2 1 2 2 2 2 1 1 1 1 1 1 , e.g . S4= 4 3 2 13 3 2 12 2 2 1 1 1 1 1 . ( 3 ) Another matrix that will appear in the formulation is Ŝp : = T−1p SpT −1 p . Clearly , the diagonal matrix Tp is positive definite . As shown in Lemma 2 , Sp and Ŝp are positive definite as well . 4 MAIN THEOREMS . The general strategy to prove the above claim is as follows . First the analytical gradients of the loss is derived in a matrix form in Propositions 1 and 2 . We compare the gradients with that of the original Minimum Square Error ( MSE ) loss . Next , we analyze the loss surface by solving the gradient equations which yields the general structure of critical points based on the rank of the decoder matrix A . Next , we delineate several interesting properties of the critical points , notably , any critical point of the loss is also a critical point for the MSE loss but not the other way around . Finally , by performing second order analysis on the loss in Theorem 2 the exact equations for local minima are derived which is shown to be as claimed . Let L̃ ( A , B ) and L ( A , B ) be the original loss , and the proposed loss function , respectively , i.e. , L̃ ( A , B ) : = m∑ j=1 ‖yj −ABxj‖22 = ‖Y −ABX‖2F L ( A , B ) : = p∑ i=1 m∑ j=1 ‖yj −AIi ; pBxj‖22 = p∑ i=1 ‖Y −AIi ; pBX‖2F The first step is to calculate the gradients with respect to A and B and set them to zero to derive the implicit expressions for the critical points . In order to do so , first , in Lemma 5 , for a fixed A , we derive the directional ( Gateaux ) derivative of the loss with respect to B along an arbitrary direction W ∈ Rp×n , denoted as dBL ( A , B ) W , i.e . dBL ( A , B ) W = lim‖W ‖F→0 L ( A , B + W ) − L ( A , B ) ‖W ‖F . As shown in the proof of the lemma , dBL ( A , B ) W is derived by writing the norm in the loss as an inner product , opening it up using linearity of inner product , dismiss second order terms in W ( i.e . O ( ‖W ‖2 ) ) and rearrange the result as the inner product between the gradient with respect to B , and the direction W , which yields dBL ( A , B ) W = −2 Tr ( W ′ ( TpA′Σyx − ( Sp ◦ ( A′A ) ) BΣxx ) ) = −2〈TpA′Σyx − ( Sp ◦ ( A′A ) ) BΣxx , W 〉F , ( 4 ) where , ◦ is the Hadamard product and the constant matrices Tp and Sp , were defined in the beginning . Second , the same process is done in Lemma 6 , to derive dAL ( A , B ) V ; the derivative of L with respect to A in an arbitrary direction V ∈ Rn×p , for a fixed B , which is then set to zero to derive the implicit expressions for the critical points . The results are formally stated in the two following propositions . Proposition 1 . For any fixed matrix A ∈ Rn×p the function L ( A , B ) is convex in the coefficients of B and attains its minimum for any B satisfying the equation ( Sp ◦ ( A′A ) ) BΣxx = TpA′Σyx , ( 5 ) where ◦ is the Hadamard ( element-wise ) product operator , and Sp and Tp are constant matrices defined in the previous section . Further , if A has no zero column , then L ( A , B ) is strictly convex in B and has a unique minimum when the critical B is B = B̂ ( A ) = ( Sp ◦ ( A′A ) ) −1TpA′ΣyxΣ−1xx , ( 6 ) and in the autoencoder case it becomes B = B̂ ( A ) = ( Sp ◦ ( A′A ) ) −1TpA′ . ( 6′ ) The proof is given in appendix A.2 . Remark 1 . Note that as long as A has no zero column , Sp ◦ ( A′A ) is nonsingular ( we will explain the reason soon ) . In practice , A with zero columns can always be avoided by nudging the zero columns of A during the gradient decent process . Proposition 2 . For any fixed matrix B ∈ Rp×n the function L ( A , B ) is a convex function in A . Moreover , for a fixed B , the matrix A that satisfies A ( Sp ◦ ( BΣxxB′ ) ) =ΣyxB′Tp ( 7 ) is a critical point of L ( A , B ) . The proof is given in appendix A.3 . The pair ( A , B ) is a critical point of L if they make dBL ( A , B ) W and dAL ( A , B ) V zero for any pair of directions ( V , W ) . Therefore , the implicit equations for critical points are given below , next to the ones derived by Baldi & Hornik ( 1989 ) for L̃ ( A , B ) . For L̃ ( A , B ) : A′ABΣxx = A ′Σyx , ABΣxxB ′ = ΣyxB ′ . For L ( A , B ) : ( Sp ◦ ( A′A ) ) BΣxx = TpA′Σyx , A ( Sp ◦ ( BΣxxB′ ) ) = ΣyxB′Tp . Remark 2 . Notice the similarity , and the difference only being the presence of Hadamard product by Sp in the left and by diagonal Tp in the right . Therefore , practically , the added computational cost of evaluating the gradients is negligible compare to that of MSE loss . The next step is to determine the structure of ( A , B ) that satisfies the above equations , and find the subset of those solutions that account for local minima . For the original loss , the first expression ( A′ABΣxx = A′Σyx ) is used to solve for B and put it in the second to derive an expression solely based on A . Obviously , in order to solve the first expression for B , two cases are considered separately : the case where A is of full rank p , so A′A is invertible , and the case of A being of rank r < p. Here we do the same but there is a twist ; for us there is only one case . The reason is as long as ( not necessarily full rank ) A has no zero column , Sp ◦ ( A′A ) is positive definite and hence , invertible . This is discussed in detail in Lemma 2 and we briefly explain it here . As shown in the lemma , Sp is positive definite and by Shur product theorem for any A ( of any rank ) , Sp ◦ ( A′A ) is positive semidefinite . However , as a result of Oppenheim inequality ( Horn & Johnson ( 2012 ) , Thm 7.8.16 ) , that in our case translates to det ( Sp ) ∏ i ( A ′A ) ii ≤ det ( Sp ◦ ( A′A ) ) , as long as A has no zero column , ∏ i ( A ′A ) ii > 0 and therefore , det ( Sp ◦ ( A′A ) ) > 0 . Here , we assume A of any rank r ≤ p has no zero column ( since this can be easily avoided in practice ) and consider Sp ◦ ( A′A ) to be always invertible . Therefore , ( A , B ) define a critical point of losses L̃ and L if For L̃ ( A , B ) and full rank A : B = B̂ ( A ) = ( A′A ) −1A′ΣyxΣ −1 xx , ABΣxxB ′ = ΣyxB ′ . For L ( A , B ) and no zero column A : B = B̂ ( A ) = ( Sp ◦ ( A′A ) ) −1TpA′ΣyxΣ−1xx , A ( Sp ◦ ( BΣxxB′ ) ) = ΣyxB′Tp . Before , we state the main theorem we need the following definitions . First , a rectangular permutation matrix Πr ∈ Rr×p is a matrix that each column consists of at most one nonzero element with the value 1 . If the rank of Πr is r with r < p then clearly , Πr has p − r zero columns . Also , by taking away those zero columns the resultant r × r submatrix of Πr is a standard square permutation matrix . Second , under the conditions provided in Assumption 1 , the matrix Σ : = ΣyxΣ−1xxΣxy has an eigenvalue decomposition Σ = UΛU ′ , where the ith column of U , denoted as ui , is an eigenvector of Σ corresponding to the ith largest eigenvalue of Σ , denoted as λi . Also , Λ = diag ( λ1 , · · · , λn ) is the diagonal vector of ordered eigenvalues of Σ , with λ1 > λ2 > · · · > λn > 0 . We use the following notation to organize a subset of eigenvectors of Σ into a rectangular matrix . Let for any r ≤ p , Ir = { i1 , · · · , ir } ( 1 ≤ i1 < · · · < ir < n ) be any ordered r−index set . Define UIr ∈ Rn×p as UIr = [ ui1 , · · · , uir ] . That is the columns of UIr are the ordered orthonormal eigenvectors of Σ associated with eigenvalues λi1 < · · · < λir . Clearly , when r = p , we have UIr = [ ui1 , · · · , uip ] corresponding to an p−index set Ip = { i1 , · · · , ip } ( 1 ≤ i1 < · · · < ip < n ) . Similarly , we define ΛIr ∈ Rp×p as ΛIr = diag ( λi1 , · · · , λir ) . Theorem 1 . Let A ∈ Rn×p and B ∈ Rp×n such that A is of rank r ≤ p. Under the conditions provided in Assumption 1 and the above notation , The matrices A and B define a critical point of L ( A , B ) if and only if for any given r−index set Ir , and a nonsingular diagonal matrix D ∈ Rr×r , A and B are of the form A = UIrCD , ( 8 ) B = B̂ ( A ) = D−1ΠCU ′ IrΣyxΣ −1 xx , ( 9 ) where , C ∈ Rr×p is of of full rank r with nonzero and normalized columns such that ΠC : = ( Sp ◦ ( C ′C ) ) −1 TpC ′ is a rectangular permutation matrix of rank r and CΠC = Ir . For all 1 ≤ r ≤ p , such C always exists . In particular , if matrix A is of full rank p , i.e . r = p , the two given conditions on ΠC are satisfied iff the invertible matrix C is any squared p × p permutation matrix Π . In this case ( A , B ) define a critical point of L ( A , B ) iff they are of the form A = UIpΠD , ( 10 ) B = B̂ ( A ) = D−1Π′U ′IpΣyxΣ −1 xx . ( 11 ) The proof is given in appendix A.4 . Remark 3 . The above theorem provides explicit equations for the critical points of the loss surface in terms of the rank of the decoder matrixA and the eigenvectors of Σ . This explicit structure allows us to further analyze the loss surface and its local/global minima . Here , we provide a proof sketch for the above theorem to make the claims more clear . Again as a reminder , the EVD of Σ : = ΣyxΣ−1xxΣxy is Σ = UΛU ′ . For both L̃ and L , the corresponding B̂ ( A ) is replaced by B on the RHS of critical point equations . For the loss L ( A , B ) , as shown in the proof of the theorem , results in the following identity U ′A ( Sp ◦ ( B̂ΣxxB̂ ′ ) ) A′U = Λ∆ , ( 12 ) where ∆ : = U ′ATp ( Sp ◦ ( A′A ) ) −1TpA′U is symmetric and positive semidefinite . The LHS of eq . ( 12 ) is symmetric so the RHS is symmetric too , so Λ∆ = ( Λ∆ ) ′ = ∆′Λ′ = ∆Λ . Therefore , ∆ commutes with the diagonal matrix of eigenvalues Λ . Since eigenvalues are assumed to be distinct , ∆ has to be diagonal as well . By Lemma 2 Tp ( Sp ◦ ( A′A ) ) −1Tp is positive definite and U is an orthogonal matrix . Therefore , r = rank ( A ) = rank ( ∆ ) = rank ( U ′∆U ) , which implies that the diagonal matrix ∆ , has r nonzero and positive diagonal entries . There exists an r−index set Ir corresponding to the nonzero diagonal elements of ∆ . Forming a diagonal matrix ∆Ir ∈ Rr×r by filling its diagonal entries ( in order ) by the nonzero diagonal elements of ∆ , we have U∆U ′ = UIr∆IrU ′ Ir Def of ∆ ====⇒ ATp ( Sp ◦ ( A′A ) ) −1TpA′ = UIr∆IrU ′Ir , ( 13 ) which indicates that the matrix A has the same column space as UIr . Therefore , there exists a full rank matrix C̄ ∈ Rr×p such that A = UIrC̄ . Since A has no zero column , C̄ has no zero column . Further , by normalizing the columns of C̄ we can write A = UIrCD , where D ∈ Rp×p is diagonal that contains the norms of columns of C̄ . Baldi & Hornik ( 1989 ) did something similar for full rank A for the loss L̃ to derive ( AL̃ = UIpC̃ ) . But their C̃ can be any invertible p × p matrix . However , in our case , the matrix C ∈ Rr×p corresponding to rank r ≤ p matrix A , has to satisfy eq . ( 13 ) by replacing A by UIrCD and eq . ( 12 ) by replacing B̂ ( A ) by B̂ ( UIrCD ) . In the case of Baldi & Hornik ( 1989 ) , for the original loss L̃ , equations similar to eq . ( 13 ) and eq . ( 12 ) appear but they are are satisfied trivially by any invertible matrix C̃ . Simplifying those equations by using A = UIrCD after some algebraic manipulation results in the following two conditions for C : CTp ( Sp ◦ ( C ′C ) ) −1 TpC ′ =∆Ir and ( 14 ) C ( Sp ◦ ( ( Sp ◦ ( C ′C ) ) −1TpC ′ΛIrCTp ( Sp ◦ ( C ′C ) ) −1 ) ) C ′ =ΛIr∆Ir . ( 15 ) As detailed in proof of Theorem 1 , solving for C leads to its specific structure as laid out in the theorem . Remark 4 . Note that when A is of rank r < p with no zero columns then the invariant matrix C is not necessarily a rectangular permutation matrix but ΠC : = ( Sp ◦ ( C ′C ) ) −1 TpC ′ is a rectangular permutation matrix with CΠC = Ir . It is only when r = p that the invariant matrix C becomes a permutation matrix . Nevertheless , as we show in the following corollary , the global map is always ∀r ≤ p : G = AB = UIrU ′IrΣyxΣ−1xx . It is possible to find further structure ( in terms of block matrices ) for the invariant matrix C when r < p. However , this is not necessary as we soon show that all rank deficient matrix As are saddle points for the loss and ideally should be passed by during the gradient decent process . Based on some numerical results our conjecture is that when r < p the matrix C can only start with a r × k rectangular permutation matrix of rank r with r ≤ k ≤ p and the rest of p− k columns of C is arbitrary as long as none of the columns are identically zero . Corollary 1 . Let ( A , B ) be a critical point of L ( A , B ) under the conditions provided in Assumption 1 and rankA = r ≤ p. Then the following are true 1 . The matrix BΣxxB′ is a p× p diagonal matrix of rank r. 2 . For all 1 ≤ r ≤ p , for any critical pair ( A , B ) , the global map G : = AB becomes G = UIrU ′ IrΣyxΣ −1 xx . ( 16 ) For the autoencoder case ( Y = X ) the global map is simply G = UIrU ′ Ir . 3 . ( A , B ) is also the critical point of the classical loss L̃ ( A , B ) = ∑p i=1 ‖Y −ABX‖ 2 F . The proof is given in appendix A.5 . Remark 5 . The above corollary implies that L ( A , B ) not only does not add any extra critical point compare to the original loss L̃ ( A , B ) , it provides the same global map G : = AB . It only limits the structure of the invariance matrix C as described in Theorem 1 so that the decoder matrix A can recover the exact eigenvectors of Σ. Lemma 1 . The loss function L ( A , B ) can be written as L ( A , B ) = pTr ( Σyy ) − 2 Tr ( ATpBΣxy ) + Tr ( B′ ( Sp ◦ ( A′A ) ) BΣxx ) . ( 17 ) The above identity shows that the number of matrix operations required for computing the loss L ( A , B ) is constant and thereby independent of the value of p. The proof is given in appendix A.6 . Theorem 2 . Let A∗ ∈ Rn×p and B∗ ∈ Rp×n such that A∗ is of rank r ≤ p. Under the conditions provided in Assumption 1 , ( A∗ , B∗ ) define a local minima of the proposed loss function iff they are of the form A∗ = U1 : pDp ( 18 ) B∗ = D−1p U ′ 1 : pΣyxΣ −1 xx , ( 19 ) where the ith column of U1 : p is a unit eigenvector of Σ : = ΣyxΣ−1xxΣxy corresponding the i th largest eigenvalue and Dp is a diagonal matrix with nonzero diagonal elements . In other words , A∗ contains ordered unnormalized eigenvectors of Σ corresponding to the p largest eigenvalues . Moreover , all the local minima are global minima with the value of the loss function at those global minima being L ( A∗ , B∗ ) = p Tr ( Σyy ) − p∑ i=1 ( p− i+ 1 ) λi , ( 20 ) where λi is the ith largest eigenvalue of Σ . The proof is given in appendix A.7 . Remark 6 . Finally , the second and third assumptions we made in the beginning in Assumption 1 can be relaxed by requiring only Σxx to be full rank . The output data can have a different dimension than the input . That is Y ∈ Rn×m and X ∈ Rn′×m , where n 6= n′ . The reason is that the given loss function structurally is very similar to MSE loss and can be represented as a Frobenius norm on the space of n × m matrices . In this case the covariance matrix Σ : = ΣyxΣ−1xxΣxy is still n × n. Clearly , for under-constrained systems with n < n′ the full rank assumption of Σ holds . For the overdetermined case , where n′ > n the second and third assumptions in Assumption 1 can be relaxed : we only require Σxx to be full rank since this is the only matrix that is inverted in the theorems . Note that if p > min ( n′ , n ) then ΛIp : the p× p diagonal matrix of eigenvalues of Σ for a p-index-set Ip bounds to have some zeros and will be say rank r < p , which in turn , results in the encoder A with rank r. However , the Theorem 1 is proved for encoder of any rank r ≤ p. Finally , following theorem 2 then the first r columns of the encoder A converges to ordered eigenvectors of Σ while the p − r remaining columns span the kernel space of Σ . Moreover , Σ need not to have distinct eigenvectors . In this case ∆Ir becomes a block diagonal matrix , where the blocks correspond to identical eigenvalues ΣIr . In this case , the corresponding eigenvectors in A ∗ are not unique but they span the respective eigenspace .

Summary: This paper proposes a new loss function to compute the exact ordered eigenvectors of a dataset. The loss is motivated from the idea of computing the eigenvectors sequentially. However doing so would be computationally expensive, and the authors show that the loss function they propose (sum of sequential losses) has the same order (constant less than 7) of computational complexity as using the squared loss. A proof of the correctness of the algorithm is given, along with experiments to verify its performance.