EigenGame: PCA as a Nash Equilibrium

1 INTRODUCTION . The principal components of data are the vectors that align with the directions of maximum variance . These have two main purposes : a ) as interpretable features and b ) for data compression . Recent methods for principal component analysis ( PCA ) focus on the latter , explicitly stating objectives to find the k-dimensional subspace that captures maximum variance ( e.g. , ( Tang , 2019 ) ) , and leaving the problem of rotating within this subspace to , for example , a more efficient downstream singular value ( SVD ) decomposition step1 . This point is subtle , yet critical . For example , any pair of twodimensional , orthogonal vectors spans all of R2 and , therefore , captures maximum variance of any two-dimensional dataset . However , for these vectors to be principal components , they must , in addition , align with the directions of maximum variance which depends on the covariance of the data . By learning the optimal subspace , rather than the principal components themselves , objectives focused on subspace error ignore the first purpose of PCA . In contrast , modern nonlinear representation learning techniques focus on learning features that are both disentangled ( uncorrelated ) and low dimensional ( Chen et al. , 2016 ; Mathieu et al. , 2018 ; Locatello et al. , 2019 ; Sarhan et al. , 2019 ) . It is well known that the PCA solution of the d-dimensional dataset X ∈ Rn×d is given by the eigenvectors of X > X or equivalently , the right singular vectors of X . Impractically , the cost of computing the full SVD scales with O ( min { nd2 , n2d } ) -time and O ( nd ) -space ( Shamir , 2015 ; Tang , 2019 ) . For moderately sized data , randomized methods can be used ( Halko et al. , 2011 ) . Beyond this , stochastic—or online—methods based on Oja ’ s rule ( Oja , 1982 ) or power iterations ( Rutishauser , 1971 ) are common . Another option is to use streaming k-PCA algorithms such as Frequent Directions ( FD ) ( Ghashami et al. , 2016 ) or Oja ’ s algorithm2 ( Allen-Zhu and Li , 2017 ) with storage complexity O ( kd ) . Sampling or sketching methods also scale well , but again , focus on the top-k subspace ( Sarlos , 2006 ; Cohen et al. , 2017 ; Feldman et al. , 2020 ) . In contrast to these approaches , we view each principal component ( equivalently eigenvector ) as a player in a game whose objective is to maximize their own local utility function in controlled competition with other vectors . The proposed utility gradients are interpretable as a combination of Oja ’ s rule and a generalized Gram-Schmidt process . We make the following contributions : • A novel formulation of PCA as finding the Nash equilibrium of a suitable game , • A sequential , globally convergent algorithm for approximating the Nash on full-batch data , 1After learning the top-k subspace V ∈ Rd×k , the rotation can be recovered via an SVD of XV . 2FD approximates the top-k subspace ; Oja ’ s algorithm approximates the top-k eigenvectors . • A decentralized algorithm with experiments demonstrating the approach as competitive with modern streaming k-PCA algorithms on synthetic and real data , • In demonstration of the scaling of the approach , we compute the top-32 principal components of the matrix of RESNET-200 activations on the IMAGENET dataset ( n ≈ 106 , d ≈ 20 ·106 ) . Each of these contributions is important . Novel formulations often lead to deeper understanding of problems , thereby , opening doors to improved techniques . In particular , k-player games are in general complex and hard to analyze . In contrast , PCA has been well-studied . By combining the two fields we hope to develop useful analytical tools . Our specific formulation is important because it obviates the need for any centralized orthonormalization step and lends itself naturally to decentralization . And lastly , theory and experiments support the viability of this approach for continued research . 2 PCA AS AN EIGEN-GAME . We adhere to the following notation . Vectors and matrices meant to approximate principal components ( equivalently eigenvectors ) are designated with hats , v̂ and V̂ respectively , whereas true principal components are v and V . Subscripts indicate which eigenvalue a vector is associated with . For example , vi is the ith largest eigenvector . In this work , we will assume each eigenvalue is distinct . By an abuse of notation , vj < i refers to the set of vectors { vj |j ∈ { 1 , . . . , i− 1 } } and are also referred to as the parents of vi ( vi is their child ) . Sums over indices should be clear from context , e.g. , ∑ j < i = ∑i−1 j=1 . The Euclidean inner product is written 〈u , v〉 = u > v . We denote the unit sphere by Sd−1 and simplex by ∆d−1 in d-dimensional ambient space . Outline of derivation As argued in the introduction , the PCA problem is often mis-interpreted as learning a projection of the data into a subspace that captures maximum variance ( equiv . maximizing the trace of a suitable matrix R introduced below ) . This is in contrast to the original goal of learning the principal components . We first develop the intuition for deriving our utility functions by ( i ) showing that only maximizing the trace of R is not sufficient for recovering all principal components ( equiv . eigenvectors ) , and ( ii ) showing that minimizing off-diagonal terms in R is a complementary objective to maximizing the trace and can recover all components . We then consider learning only the top-k and construct utilities that are consistent with findings in ( i ) and ( ii ) , equal the true eigenvalues at the Nash of the game we construct , and result in a game that is amenable to analysis . Derivation of player utilities . The eigenvalue problem for a symmetric matrix X > X = M ∈ Rd×d is to find a matrix of d orthonormal column vectors V ( implies V is full-rank ) such that MV = V Λ with Λ diagonal . Given a solution to this problem , the columns of V are known as eigenvectors and corresponding entries in Λ are eigenvalues . By left-multiplying by V > and recalling V > V = V V > = I by orthonormality ( i.e. , V is unitary ) , we can rewrite the equality as V > MV = V > V Λ unitary = Λ . ( 1 ) Let V̂ denote a guess or estimate of the true eigenvectors V and define R ( V̂ ) def= V̂ > MV̂ . The PCA problem is often posed as maximizing the trace of R ( equiv . minimizing reconstruction error ) : max V̂ > V̂=I { ∑ i Rii = Tr ( R ) = Tr ( V̂ > MV̂ ) = Tr ( V̂ V̂ > M ) = Tr ( M ) } . ( 2 ) Surprisingly , the objective in ( 2 ) is independent of V̂ , so it can not be used to recover all ( i.e. , k = d ) the eigenvectors of M— ( i ) . Alternatively , Equation ( 1 ) implies the eigenvalue problem can be phrased as ensuring all off-diagonal terms of R are zero , thereby ensuring R is diagonal— ( ii ) : min V̂ > V̂=I ∑ i6=j R2ij . ( 3 ) It is worth further examining the entries of R in detail . Diagonal entries Rii = 〈v̂i , Mv̂i〉 are recognized as Rayleigh quotients because ||v̂i|| = 1 by the constraints . Off-diagonal entries Rij = 〈v̂i , Mv̂j〉 measure alignment between v̂i and v̂j under a generalized inner product 〈· , ·〉M . So far , we have considered learning all the eigenvectors . If we repeat the logic for the top-k eigenvectors with k < d , then by Equation ( 1 ) , R must still be diagonal . V is not square , so V V > 6= I , but assuming V is orthonormal as before , we have V V > = P is a projection matrix . Left-multiplying Equation ( 1 ) by V now reads ( PM ) V = V Λ so we are solving an eigenvalue problem for a subspace of M . If we only desire the top-k eigenvectors , maximizing the trace encourages learning a subspace spanned by the top-k eigenvectors , but does not recover the eigenvectors themselves . On the other hand , Equation ( 3 ) places no preference on recovering large over small eigenvectors , but does enforce the columns of V̂ to actually be eigenvectors . The preceding exercise is intended to introduce minimizing the off-diagonal terms of R as a possible complementary objective for solving top-k PCA . Next , we will use these two objectives to construct utility functions for each eigenvector v̂i . We want to combine the objectives to take advantage of both their strengths . A valid proposal is max V̂ > V̂=I ∑ i Rii − ∑ i6=j R2ij . ( 4 ) However , this objective ignores the natural hierarchy of the top-k eigenvectors . For example , v̂1 is penalized for aligning with v̂k and vice versa , but v̂1 , being the estimate of the largest eigenvector , should be free to search for the direction that captures the most variance independent of the locations of the other vectors . Instead , first consider solving for the top-1 eigenvector , v1 , in which case R = [ 〈v̂1 , Mv̂1〉 ] is a 1× 1 matrix . In this setting , Equation ( 3 ) is not applicable because there are no off-diagonal elements , so maxv̂ > 1 v̂1=1〈v̂1 , Mv̂1〉 is a sensible utility function for v̂1 . If considering the top-2 eigenvectors , v̂1 ’ s utility remains as before , and we introduce a new utility for v̂2 . Equation ( 3 ) is now applicable , so v̂2 ’ s utility is max v̂ > 2 v̂2=1 , v̂ > 1 v̂2=0 〈v̂2 , Mv̂2〉 − 〈v̂2 , Mv̂1〉2 〈v̂1 , Mv̂1〉 ( 5 ) where we have divided the off-diagonal penalty by 〈v1 , Mv1〉 so a ) the two terms in Equation ( 5 ) are on a similar scale and b ) for reasons that ease analysis . Additionally note that the constraint v̂ > 1 v̂2 = 0 may be redundant at the optimum ( v̂∗1 = v1 , v̂ ∗ 2 = v2 ) because the second term , 〈v̂∗2 , Mv̂∗1〉2 = 〈v2 , Mv1〉2 = Λ211〈v2 , v1〉2 , already penalizes such deviations ( Λii is the ith largest eigenvector ) . These reasons motivate the following set of objectives ( utilities ) , one for each vector i ∈ { 1 , . . . , k } : max v̂ > i v̂i=1 { ui ( v̂i|v̂j < i ) = v̂ > i Mv̂i − ∑ j < i ( v̂ > i Mv̂j ) 2 v̂ > j Mv̂j = ||Xv̂i||2 − ∑ j < i 〈Xv̂i , Xv̂j〉2 〈Xv̂j , Xv̂j〉 } ( 6 ) where the notation ui ( ai|b ) emphasizes that player i adjusts ai to maximize a utility conditioned on b . It is interesting to note that by incorporating knowledge of the natural hierarchy ( see Figure 1 ) , we are immediately led to constructing asymmetric utilities , and thereby , inspired to formulate the PCA problem as a game , rather than a direct optimization problem as in Equation ( 4 ) . A key concept in games is a Nash equilibrium . A Nash equilibrium specifies a variable for each player from which no player can unilaterally deviate and improve their outcome . In this case , V̂ is a ( strict- ) Nash equilibrium if and only if for all i , ui ( v̂i|v̂j < i ) > ui ( zi|v̂j < i ) for all zi ∈ Sd−1 . Theorem 2.1 ( PCA Solution is the Unique strict-Nash Equilibrium ) . Assume that the top-k eigenvalues of X > X are positive and distinct . Then the top-k eigenvectors form the unique strictNash equilibrium of the proposed game in Equation ( 6 ) .3 The proof is deferred to Appendix L. Solving for the Nash of a game is difficult in general . Specifically , it belongs to the class of PPADcomplete problems ( Gilboa and Zemel , 1989 ; Daskalakis et al. , 2009 ) . However , because the game is hierarchical and each player ’ s utility only depends on its parents , it is possible to construct a sequential algorithm that is convergent by solving each player ’ s optimization problem in sequence .

Summary: Principal component analysis (PCA) is a well-known dimensionality reduction and feature learning technique in the literature that leads to uncorrelated features. While there are a plethora of algorithms for PCA, along with accompanying analysis, a majority of these works have been developed from an optimization perspective. This paper differs from existing works in that it motivates the $k$-PCA problem, which involves learning the $k$-dominant eigen vectors of the sample covariance matrix, as a competitive game between $k$ players in which each player is supposed to estimate one of the eigen vectors and the PCA solution is the unique strict-Nash equilibrium. The main contributions of the paper in this regard are the following: