EigenGame: PCA as a Nash Equilibrium

1 INTRODUCTION . The principal components of data are the vectors that align with the directions of maximum variance . These have two main purposes : a ) as interpretable features and b ) for data compression . Recent methods for principal component analysis ( PCA ) focus on the latter , explicitly stating objectives to find the k-dimensional subspace that captures maximum variance ( e.g. , ( Tang , 2019 ) ) , and leaving the problem of rotating within this subspace to , for example , a more efficient downstream singular value ( SVD ) decomposition step1 . This point is subtle , yet critical . For example , any pair of twodimensional , orthogonal vectors spans all of R2 and , therefore , captures maximum variance of any two-dimensional dataset . However , for these vectors to be principal components , they must , in addition , align with the directions of maximum variance which depends on the covariance of the data . By learning the optimal subspace , rather than the principal components themselves , objectives focused on subspace error ignore the first purpose of PCA . In contrast , modern nonlinear representation learning techniques focus on learning features that are both disentangled ( uncorrelated ) and low dimensional ( Chen et al. , 2016 ; Mathieu et al. , 2018 ; Locatello et al. , 2019 ; Sarhan et al. , 2019 ) . It is well known that the PCA solution of the d-dimensional dataset X ∈ Rn×d is given by the eigenvectors of X > X or equivalently , the right singular vectors of X . Impractically , the cost of computing the full SVD scales with O ( min { nd2 , n2d } ) -time and O ( nd ) -space ( Shamir , 2015 ; Tang , 2019 ) . For moderately sized data , randomized methods can be used ( Halko et al. , 2011 ) . Beyond this , stochastic—or online—methods based on Oja ’ s rule ( Oja , 1982 ) or power iterations ( Rutishauser , 1971 ) are common . Another option is to use streaming k-PCA algorithms such as Frequent Directions ( FD ) ( Ghashami et al. , 2016 ) or Oja ’ s algorithm2 ( Allen-Zhu and Li , 2017 ) with storage complexity O ( kd ) . Sampling or sketching methods also scale well , but again , focus on the top-k subspace ( Sarlos , 2006 ; Cohen et al. , 2017 ; Feldman et al. , 2020 ) . In contrast to these approaches , we view each principal component ( equivalently eigenvector ) as a player in a game whose objective is to maximize their own local utility function in controlled competition with other vectors . The proposed utility gradients are interpretable as a combination of Oja ’ s rule and a generalized Gram-Schmidt process . We make the following contributions : • A novel formulation of PCA as finding the Nash equilibrium of a suitable game , • A sequential , globally convergent algorithm for approximating the Nash on full-batch data , 1After learning the top-k subspace V ∈ Rd×k , the rotation can be recovered via an SVD of XV . 2FD approximates the top-k subspace ; Oja ’ s algorithm approximates the top-k eigenvectors . • A decentralized algorithm with experiments demonstrating the approach as competitive with modern streaming k-PCA algorithms on synthetic and real data , • In demonstration of the scaling of the approach , we compute the top-32 principal components of the matrix of RESNET-200 activations on the IMAGENET dataset ( n ≈ 106 , d ≈ 20 ·106 ) . Each of these contributions is important . Novel formulations often lead to deeper understanding of problems , thereby , opening doors to improved techniques . In particular , k-player games are in general complex and hard to analyze . In contrast , PCA has been well-studied . By combining the two fields we hope to develop useful analytical tools . Our specific formulation is important because it obviates the need for any centralized orthonormalization step and lends itself naturally to decentralization . And lastly , theory and experiments support the viability of this approach for continued research . 2 PCA AS AN EIGEN-GAME . We adhere to the following notation . Vectors and matrices meant to approximate principal components ( equivalently eigenvectors ) are designated with hats , v̂ and V̂ respectively , whereas true principal components are v and V . Subscripts indicate which eigenvalue a vector is associated with . For example , vi is the ith largest eigenvector . In this work , we will assume each eigenvalue is distinct . By an abuse of notation , vj < i refers to the set of vectors { vj |j ∈ { 1 , . . . , i− 1 } } and are also referred to as the parents of vi ( vi is their child ) . Sums over indices should be clear from context , e.g. , ∑ j < i = ∑i−1 j=1 . The Euclidean inner product is written 〈u , v〉 = u > v . We denote the unit sphere by Sd−1 and simplex by ∆d−1 in d-dimensional ambient space . Outline of derivation As argued in the introduction , the PCA problem is often mis-interpreted as learning a projection of the data into a subspace that captures maximum variance ( equiv . maximizing the trace of a suitable matrix R introduced below ) . This is in contrast to the original goal of learning the principal components . We first develop the intuition for deriving our utility functions by ( i ) showing that only maximizing the trace of R is not sufficient for recovering all principal components ( equiv . eigenvectors ) , and ( ii ) showing that minimizing off-diagonal terms in R is a complementary objective to maximizing the trace and can recover all components . We then consider learning only the top-k and construct utilities that are consistent with findings in ( i ) and ( ii ) , equal the true eigenvalues at the Nash of the game we construct , and result in a game that is amenable to analysis . Derivation of player utilities . The eigenvalue problem for a symmetric matrix X > X = M ∈ Rd×d is to find a matrix of d orthonormal column vectors V ( implies V is full-rank ) such that MV = V Λ with Λ diagonal . Given a solution to this problem , the columns of V are known as eigenvectors and corresponding entries in Λ are eigenvalues . By left-multiplying by V > and recalling V > V = V V > = I by orthonormality ( i.e. , V is unitary ) , we can rewrite the equality as V > MV = V > V Λ unitary = Λ . ( 1 ) Let V̂ denote a guess or estimate of the true eigenvectors V and define R ( V̂ ) def= V̂ > MV̂ . The PCA problem is often posed as maximizing the trace of R ( equiv . minimizing reconstruction error ) : max V̂ > V̂=I { ∑ i Rii = Tr ( R ) = Tr ( V̂ > MV̂ ) = Tr ( V̂ V̂ > M ) = Tr ( M ) } . ( 2 ) Surprisingly , the objective in ( 2 ) is independent of V̂ , so it can not be used to recover all ( i.e. , k = d ) the eigenvectors of M— ( i ) . Alternatively , Equation ( 1 ) implies the eigenvalue problem can be phrased as ensuring all off-diagonal terms of R are zero , thereby ensuring R is diagonal— ( ii ) : min V̂ > V̂=I ∑ i6=j R2ij . ( 3 ) It is worth further examining the entries of R in detail . Diagonal entries Rii = 〈v̂i , Mv̂i〉 are recognized as Rayleigh quotients because ||v̂i|| = 1 by the constraints . Off-diagonal entries Rij = 〈v̂i , Mv̂j〉 measure alignment between v̂i and v̂j under a generalized inner product 〈· , ·〉M . So far , we have considered learning all the eigenvectors . If we repeat the logic for the top-k eigenvectors with k < d , then by Equation ( 1 ) , R must still be diagonal . V is not square , so V V > 6= I , but assuming V is orthonormal as before , we have V V > = P is a projection matrix . Left-multiplying Equation ( 1 ) by V now reads ( PM ) V = V Λ so we are solving an eigenvalue problem for a subspace of M . If we only desire the top-k eigenvectors , maximizing the trace encourages learning a subspace spanned by the top-k eigenvectors , but does not recover the eigenvectors themselves . On the other hand , Equation ( 3 ) places no preference on recovering large over small eigenvectors , but does enforce the columns of V̂ to actually be eigenvectors . The preceding exercise is intended to introduce minimizing the off-diagonal terms of R as a possible complementary objective for solving top-k PCA . Next , we will use these two objectives to construct utility functions for each eigenvector v̂i . We want to combine the objectives to take advantage of both their strengths . A valid proposal is max V̂ > V̂=I ∑ i Rii − ∑ i6=j R2ij . ( 4 ) However , this objective ignores the natural hierarchy of the top-k eigenvectors . For example , v̂1 is penalized for aligning with v̂k and vice versa , but v̂1 , being the estimate of the largest eigenvector , should be free to search for the direction that captures the most variance independent of the locations of the other vectors . Instead , first consider solving for the top-1 eigenvector , v1 , in which case R = [ 〈v̂1 , Mv̂1〉 ] is a 1× 1 matrix . In this setting , Equation ( 3 ) is not applicable because there are no off-diagonal elements , so maxv̂ > 1 v̂1=1〈v̂1 , Mv̂1〉 is a sensible utility function for v̂1 . If considering the top-2 eigenvectors , v̂1 ’ s utility remains as before , and we introduce a new utility for v̂2 . Equation ( 3 ) is now applicable , so v̂2 ’ s utility is max v̂ > 2 v̂2=1 , v̂ > 1 v̂2=0 〈v̂2 , Mv̂2〉 − 〈v̂2 , Mv̂1〉2 〈v̂1 , Mv̂1〉 ( 5 ) where we have divided the off-diagonal penalty by 〈v1 , Mv1〉 so a ) the two terms in Equation ( 5 ) are on a similar scale and b ) for reasons that ease analysis . Additionally note that the constraint v̂ > 1 v̂2 = 0 may be redundant at the optimum ( v̂∗1 = v1 , v̂ ∗ 2 = v2 ) because the second term , 〈v̂∗2 , Mv̂∗1〉2 = 〈v2 , Mv1〉2 = Λ211〈v2 , v1〉2 , already penalizes such deviations ( Λii is the ith largest eigenvector ) . These reasons motivate the following set of objectives ( utilities ) , one for each vector i ∈ { 1 , . . . , k } : max v̂ > i v̂i=1 { ui ( v̂i|v̂j < i ) = v̂ > i Mv̂i − ∑ j < i ( v̂ > i Mv̂j ) 2 v̂ > j Mv̂j = ||Xv̂i||2 − ∑ j < i 〈Xv̂i , Xv̂j〉2 〈Xv̂j , Xv̂j〉 } ( 6 ) where the notation ui ( ai|b ) emphasizes that player i adjusts ai to maximize a utility conditioned on b . It is interesting to note that by incorporating knowledge of the natural hierarchy ( see Figure 1 ) , we are immediately led to constructing asymmetric utilities , and thereby , inspired to formulate the PCA problem as a game , rather than a direct optimization problem as in Equation ( 4 ) . A key concept in games is a Nash equilibrium . A Nash equilibrium specifies a variable for each player from which no player can unilaterally deviate and improve their outcome . In this case , V̂ is a ( strict- ) Nash equilibrium if and only if for all i , ui ( v̂i|v̂j < i ) > ui ( zi|v̂j < i ) for all zi ∈ Sd−1 . Theorem 2.1 ( PCA Solution is the Unique strict-Nash Equilibrium ) . Assume that the top-k eigenvalues of X > X are positive and distinct . Then the top-k eigenvectors form the unique strictNash equilibrium of the proposed game in Equation ( 6 ) .3 The proof is deferred to Appendix L. Solving for the Nash of a game is difficult in general . Specifically , it belongs to the class of PPADcomplete problems ( Gilboa and Zemel , 1989 ; Daskalakis et al. , 2009 ) . However , because the game is hierarchical and each player ’ s utility only depends on its parents , it is possible to construct a sequential algorithm that is convergent by solving each player ’ s optimization problem in sequence .

Summary: The authors present new insights on PCA analysis by reconceiving it in terms of a Nash equilibrium among different players, related to the different components. The importance of an objective function minimizing the off-diagonal elements of R is emphasized. The insights lead to parallel algorithms and are demonstrated on large scale problems, which is nice. Overall the new insights can be very valuable and also inspiring for future work and for new developments,  from a broader perspective.