Generative Principal Component Analysis

of order √ k logL m , where m is the number of samples . Moreover , we provide a variant of the classic power method , which projects the calculated data onto the range of the generative model during each iteration . We show that under suitable conditions , this method converges exponentially fast to a point achieving the above-mentioned statistical rate . This rate is conjectured in ( Aubin et al. , 2019 ; Cocola et al. , 2020 ) to be the best possible even when we only restrict to the special case of spiked matrix models . We perform experiments on various image datasets for spiked matrix and phase retrieval models , and illustrate performance gains of our method to the classic power method and the truncated power method devised for sparse principal component analysis . 1 INTRODUCTION . Principal component analysis ( PCA ) is one of the most popular techniques for data processing and dimensionality reduction ( Jolliffe , 1986 ) , with an abundance of applications such as image recognition ( Hancock et al. , 1996 ) , gene expression data analysis ( Alter et al. , 2000 ) , and clustering ( Ding & He , 2004 ; Liu & Tan , 2019 ) . PCA seeks to find the directions that capture maximal variances in vector-valued data . In more detail , letting x1 , x2 , . . . , xm be m realizations of a random vector x ∈ Rn with a population covariance matrix Σ̄ ∈ Rn×n , PCA aims to reconstruct the top principal eigenvectors of Σ̄ . The first principal eigenvector can be computed as follows : u1 = arg max w∈Rn wTΣw s.t . ‖w‖2 = 1 , ( 1 ) where the empirical covariance matrix is defined as Σ : = 1m ∑m i=1 ( xi − c ) ( xi − c ) T , with c : = 1 m ∑m i=1 xi . In addition , subsequent principal eigenvectors can be estimated by similar optimization problems subject to being orthogonal to the previous vectors . PCA is consistent in the conventional setting where the dimension of the data n is relatively small compared to the sample size m ( Anderson , 1962 ) , but leads to rather poor estimates in the highdimensional setting wherem n. In particular , it has been shown in various papers that the empirical principal eigenvectors are no longer consistent estimates of their population counterparts ( Nadler , ∗Corresponding authors . 2008 ; Johnstone & Lu , 2009 ; Jung & Marron , 2009 ; Birnbaum et al. , 2013 ) . In order to tackle the curse of dimensionality , a natural approach is to impose certain structural constraints on the principal eigenvectors . A common assumption is that the principal eigenvectors are sparse , and this gives rise to the problem of sparse principal component analysis ( SPCA ) ( Zou et al. , 2006 ) . In particular , for recovering the top principal eigenvector , the optimization problem of SPCA is given by u1 = arg max w∈Rn wTΣw s.t . ‖w‖2 = 1 , ‖w‖0 ≤ s , ( 2 ) where ‖w‖0 = | { i : wi 6= 0 } | denotes the number of non-zero entries of w , and s ∈ N represents the sparsity level . In addition to reducing the effective number of parameters , the sparsity assumption also enhances the interpretability ( Zou et al. , 2006 ) . Departing momentarily from the PCA problem , recent years have seen tremendous advances in deep generative models in a wide variety of real-world applications ( Foster , 2019 ) . This has motivated a new perspective of the related problem of compressed sensing ( CS ) , in which the standard sparsity assumption is replaced by a generative modeling assumption . That is , the underlying signal is assumed to lie near the range of a ( deep ) generative model ( Bora et al. , 2017 ) . The authors of ( Bora et al. , 2017 ) characterized the number of samples required to attain an accurate reconstruction , and also presented numerical results on image datasets showing that compared to sparsity-based methods , generative priors can lead to large reductions ( e.g. , a factor of 5 to 10 ) in the number of measurements needed to recover the signal up to a given accuracy . Additional numerical and theoretical results concerning inverse problems using generative models have been provided in ( Van Veen et al. , 2018 ; Dhar et al. , 2018 ; Heckel & Hand , 2019 ; Jalal et al. , 2020 ; Liu & Scarlett , 2020a ; Ongie et al. , 2020 ; Whang et al. , 2020 ; Jalal et al. , 2021 ; Nguyen et al. , 2021 ) , among others . In this paper , following the developments in both PCA/SPCA and inverse problems with generative priors , we study the use of generative priors in principal component analysis ( GPCA ) , which gives a generative counterpart of SPCA in ( 2 ) , formulated as follows : u1 = arg max w∈Rn wTΣw s.t . w ∈ Range ( G ) , ( 3 ) where G is a ( pre-trained ) generative model , which we assume has a range contained in the unit sphere of Rn.1 Similarly to SPCA , the motivation for this problem is to incorporate prior knowledge on the vector being recovered ( or alternatively , a prior preference ) , and to permit meaningful recovery and theoretical bounds even in the high-dimensional regime m n . 1.1 RELATED WORK . In this subsection , we summarize some relevant works , which can roughly be divided into ( i ) the SPCA problem , and ( ii ) signal recovery with generative models . SPCA : It has been proved that the solution of the SPCA problem in ( 2 ) attains the optimal statistical rate √ s log n/m ( Vu & Lei , 2012 ) , where m is the number of samples , n is the ambient dimension , and s is the sparsity level of the first principal eigenvector . However , due to the combinatorial constraint , the computation of ( 2 ) is intractable . To address this computational issue , in recent years , an extensive body of practical approaches for estimating sparse principal eigenvectors have been proposed in the literature , including ( d ’ Aspremont et al. , 2007 ; Vu et al. , 2013 ; Chang et al. , 2016 ; Moghaddam et al. , 2006 ; d ’ Aspremont et al. , 2008 ; Jolliffe et al. , 2003 ; Zou et al. , 2006 ; Shen & Huang , 2008 ; Journée et al. , 2010 ; Hein & Bühler , 2010 ; Kuleshov , 2013 ; Yuan & Zhang , 2013 ; Asteris et al. , 2011 ; Papailiopoulos et al. , 2013 ) , just to name a few . Notably , statistical guarantees for several approaches have been provided . The authors of ( Yuan & Zhang , 2013 ) propose the truncated power method ( TPower ) , which adds a truncation operation to the power method to ensure the desired level of sparsity . It is shown that this approach attains the optimal statistical rate under appropriate initialization . Most approaches for SPCA only focus on estimating the first principal eigenvector , with a certain deflation method ( Mackey , 2008 ) being leveraged to reconstruct the rest . However , there are some exceptions ; for instance , an iterative thresholding approach is proposed in ( Ma , 2013 ) , and is shown to attain a near-optimal statistical rate 1Similarly to ( Liu et al. , 2020 ; 2021a ) , we assume that the range of G is contained in the unit sphere for convenience . Our results readily transfer to general ( unnormalized ) generative models by considering its normalized version . See Remark 1 for a detailed discussion . when estimating multiple individual principal eigenvectors . In addition , the authors of ( Cai et al. , 2013 ) propose a regression-type method that gives an optimal principal subspace estimator . Both works ( Ma , 2013 ; Cai et al. , 2013 ) rely on the assumption of a spiked covariance model to ensure a good initial vector . To avoid the spiked covariance model assumption , the work ( Wang et al. , 2014 ) proposes a two-stage procedure that attains the optimal subspace estimator in polynomial time . Signal recovery with generative models : Since the seminal work ( Bora et al. , 2017 ) , there has been a substantial volume of papers studying various inverse problems with generative priors . One of the problems more closely related to PCA is spectral initialization in phase retrieval , which amounts to solving an eigenvalue problem . Phase retrieval with generative priors has been studied in ( Hand et al. , 2018 ; Hyder et al. , 2019 ; Jagatap & Hegde , 2019 ; Wei et al. , 2019 ; Shamshad & Ahmed , 2020 ; Aubin et al. , 2020 ; Liu et al. , 2021b ) . In particular , the work ( Hand et al. , 2018 ) models the underlying signal as being in the range of a fully-connected ReLU neural network with no offsets , and all the weight matrices of the ReLU neural network are assumed to have i.i.d . zero-mean Gaussian entries . In addition , the neural network needs to be sufficiently expansive in the sense that ni ≥ Ω ( ni−1 log ni−1 ) , where ni is the width of the i-th layer . Under these assumptions , the authors establish favorable global optimization landscapes for the corresponding objective , and derive a near-optimal sample complexity upper bound . They minimize the objective function directly over the latent space in Rk using gradient descent , which may suffer from local minima in general optimization landscapes ( Hyder et al. , 2019 ; Shah & Hegde , 2018 ) . In ( Aubin et al. , 2020 ) , the assumptions on the neural network are similar to those in ( Hand et al. , 2018 ) , relaxing to general activation functions ( beyond ReLU ) and ni ≥ Ω ( ni−1 ) . The authors focus on the high dimensional regime where n , m , k →∞with the ratiom/n being fixed , and assume that the input vector in Rk is drawn from a separable distribution . They derive sharp asymptotics for the information-theoretically optimal performance and for the associated approximate message passing ( AMP ) algorithm . Both works ( Hand et al. , 2018 ; Aubin et al. , 2020 ) focus on noiseless phase retrieval . When only making the much milder assumption that the generative model is Lipschitz continuous , with no assumption on expansiveness , Gaussianity , and offsets , a spectral initialization step ( similar to that of sparse phase retrieval ) is typically required in order to accurately reconstruct the signal ( Netrapalli et al. , 2015 ; Candès et al. , 2015 ) . The authors of ( Liu et al. , 2021b ) propose an optimization problem similar to ( 3 ) for the spectral initialization for phase retrieval with generative models . It was left open in ( Liu et al. , 2021b ) how to solve ( or approximate sufficiently accurately ) the optimization problem in practice . Understanding the eigenvalues of spiked random matrix models has been a central problem of random matrix theory , and spiked matrices have been widely used in the statistical analysis of SPCA . Recently , theoretical guarantees concerning spiked matrix models with generative priors have been provided in ( Aubin et al. , 2019 ; Cocola et al. , 2020 ) . In particular , in ( Aubin et al. , 2019 ) , the assumptions are similar to those in ( Aubin et al. , 2020 ) , except that the neural network is assumed to have exactly one hidden layer . The Bayes-optimal performance is analyzed , and it is shown that the AMP algorithm can attain this optimal performance . In addition , the authors of ( Aubin et al. , 2019 ) propose the linearized approximate message passing ( LAMP ) algorithm , which is a spectral algorithm specifically designed for single-layer feedforward neural networks with no bias terms . The authors show its superiority to classical PCA via numerical results on the Fashion-MNIST dataset . In ( Cocola et al. , 2020 ) , the same assumptions are made as those in ( Hand et al. , 2018 ) on the neural network , and the authors demonstrate the benign global geometry for a nonlinear least squares objective . Similarly to ( Hand et al. , 2018 ) , the objective is minimized over Rk using a gradient descent algorithm , which can get stuck in local minima for general global geometries .

Summary: This work studies PCA under the contraint that the principal eigenvectors lie in the range of some fixed generative model, e.g., a neural network with fixed weights.  More precisely, given an $L$-Lipschitz function $G : \mathbb R^k \mapsto R(G)$ the authors try to find a vector $w \in R(G)$ that maximizes $w^T V w$, where $V = \bar{V} + E$, is a (noisy) version of the "true" psd matrix $\bar{V}$.  The corresponding true solution $\bar{x}$ is defined to be the top eigenvector of $\bar{V}$.  They assume that the range $R(G)$ of $G$ is a subset of the unit ball of $\mathbb R^n$ and that the error matrix $E$ is not too large in the sense that for any sets $S_1, S_2$ with $m = \Omega( \log( |S_1| |S_2|))$ it holds that $|s_1^T E s_2| \leq \sqrt{ \log(|S_1| |S_2|)/ m} \|s_1\|_2 \|s_2\|_2$.  This error bound is satisfied (via standard concentration inequalities) by the spiked covariance and phase retrieval model assuming that the number of samples $m$ is large enough.  The authors show two main results.  First, they show that under the above assumptions the global maximizer given the noisy matrix $V$, i.e., the maximizer $\hat{v}$ of $w^T V w$ constrained on $R(G)$ has error roughly $\|\hat{v} - \bar{x}\|_2 \leq \sqrt{k \log L/m}$.  Next, they show that the power iteration algorithm together with a projection step onto the contraint set $R(G)$ will also eventually converge to a vector with similar error.  They also give experimental results showing that adding this projection step in the power iteration leads to improved recovery for the spiked covariance and phase retrieval problems for the MNIST and Fashion-MNIST datasets in the case where the generative model $G$ is a pre-trained variational autoencoder of small latent dimension $k$.