Generalized Eigenvalue Problems with Generative Priors

Authors: zhaoqiang  liu
wen  li
junren  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:00:50

Abstract: Generalized eigenvalue problems (GEPs) find applications in various fields of science and engineering. For example, principal component analysis, Fisher's discriminant analysis, and canonical correlation analysis are specific instances of GEPs and are widely used in statistical data processing. In this work, we study GEPs under generative priors, assuming that the underlying leading generalized eigenvector lies within the range of a Lipschitz continuous generative model. Under appropriate conditions, we show that any optimal solution to the corresponding optimization problems attains the optimal statistical rate. Moreover, from a computational perspective, we propose an iterative algorithm called the Projected Rayleigh Flow Method (PRFM) to approximate the optimal solution. We theoretically demonstrate that under suitable assumptions, PRFM converges linearly to an estimated vector that achieves the optimal statistical rate. Numerical results are provided to demonstrate the effectiveness of the proposed method.

Main paper: 
 introduction: 
 The generalized eigenvalue problem (GEP) plays an important role in various fields of science and engineering [72, 70, 23, 21] . For instance, it underpins numerous machine learning and statistical methods, such as principal component analysis (PCA), Fisher's discriminant analysis (FDA), and canonical correlation analysis (CCA) [85, 37, 61, 5, 28, 22, 48, 53] . In particular, the GEP for a symmetric matrix A ∈ R n×n and a positive definite matrix B ∈ R n×n is defined as Av i = λ i Bv i , i = 1, 2, . . . , n.(1) Here, λ i represent the generalized eigenvalues, and v i denote the corresponding generalized eigenvectors of the matrix pair (A, B). The eigenvalues are ordered such that λ 1 ≥ λ 2 . . . ≥ λ n . Throughout this paper, we focus on the setting of symmetric semi-definite GEPs.According to the Rayleigh-Ritz theorem [70] , the leading generalized eigenvector of (A, B) is the optimal solution to the following optimization problem:EQUATIONOr, in an equivalent form with respect to the Rayleigh quotient (up to the transformation of the norm of the optimal solutions):EQUATIONSubsequent generalized eigenvectors of (A, B) can be obtained through methods such as generalizations of Rayleigh quotient iteration [56, 19, 66] , subspace style iterations [99, 25, 89] , and the QZ method [64, 43, 45, 10] .In many practical applications, the matrices A and B may be contaminated by unknown perturbations, and we can only access approximate matrices Â and B based on m independent observations. Specifically, we haveEQUATIONwhere E and F are the perturbation matrices. In addition, in recent years, there has been an increasing interest in studying the GEP in the high-dimensional setting where m ≪ n, with the prominent assumption that the leading generalized eigenvector v 1 ∈ R n of the uncorrupted matrix pair (A, B) (cf. Eq. ( 1)) is sparse. This leads to the sparse generalized eigenvalue problem (SGEP), for which we can estimate the underlying signal v 1 based on the approximate matrices Â and B by solving the following constrained optimization problem:EQUATIONwhere ∥u∥ 0 = |{i : u i ̸ = 0}| represents the number of non-zero entries of u and s ∈ N is a parameter for the sparsity level. As mentioned in [84] , solving the non-convex optimization problem in Eq. ( 5) is challenging in the high-dimensional setting because B is singular and not invertible, preventing us from converting the SGEP to the sparse eigenvalue problem and making use of classical algorithms that require taking the inverse of B.Motivated by the remarkable success of deep generative models in a multitude of real-world applications, a new perspective on high-dimensional inverse problems has recently emerged. In this new perspective, the assumption that the underlying signal can be well-modeled by a pre-trained (deep) generative model replaces the common sparsity assumption. Specifically, in the seminal work [4] , the authors explore linear inverse problems with generative models and provide sample complexity upper bounds for accurate signal recovery. Furthermore, they presented impressive numerical results on natural image datasets, demonstrating that the utilization of generative models can result in substantial reductions in the required number of measurements compared to sparsity-based methods. Numerous subsequent works have built upon [4] , exploring various aspects of inverse problems under generative priors [86, 29, 42, 36, 67, 93, 35, 65] .In this work, we investigate the high-dimensional GEP under generative priors, which we refer to as the generative generalized eigenvalue problem (GGEP). The corresponding optimization problem becomes:EQUATIONwhere R(G) denotes the range of a pre-trained generative model G : R k → R n , with the condition that the input dimension k is much smaller than the output dimension n to enable accurate recovery of the signal using a small number of measurements or observations.This subsection provides a summary of existing works related to Sparse Generalized Eigenvalue Problems (SGEP) and inverse problems with generative priors.There have been extensive studies aimed at developing algorithms and providing theoretical guarantees for specific instances of SGEP. For instance, sparse principal component analysis (SPCA) is one of the most widely studied instances of SGEP, with numerous notable solvers in the literature, including the truncated power method [98] , Fantope-based convex relaxation method [88] , iterative thresholding approach [58] , regression-type method [6] , and sparse orthogonal iteration pursuit [91] , among others [63, 18, 59, 80, 30, 39, 1, 46, 68, 11, 90] . Additionally, significant developments for sparse FDA and sparse CCA, another two popular instances of SGEP, can be found in various works including [24, 95, 69, 57, 7, 15, 27, 79, 60, 13, 14, 44, 94, 20, 83, 96] .There are also many works that propose general approaches for the SGEP, including the projectionfree method proposed in [32] , the majorization-minimization approaches proposed in [82, 81] , the decomposition algorithm proposed in [97] , an inverse-free truncated Rayleigh-Ritz method in [9] , the linear programming based sparse estimation method in [75] , among others [76, 62, 84, 40, 8, 31] .Among the works related to SGEP, [84] is the most relevant to ours. In [84] , the authors proposed a two-stage computational method for SGEP that achieves linear convergence to a point achieving the optimal statistical rate. Their method first computes an initialization vector via convex relaxation, and then refines the initial guess by truncated gradient ascent, with the method for the second stage being referred to as the truncated Rayleigh flow method. Our work generalizes the truncated Rayleigh flow method to the more complex scenario of using generative priors, and we also establish a linear convergence guarantee to an estimated vector with the optimal statistical error.Inverse problems with generative priors: Since the seminal work [4] that investigated linear inverse problems with generative priors, various high-dimensional inverse problems have been studied using generative models. For example, under the generative modeling assumption, one-bit or more general single-index models have been investigated in [92, 38, 50, 52, 73, 41, 12] , spiked matrix models have been investigated in [3, 16, 17] , and phase retrieval problems have been investigated in [26, 33, 34, 92, 78, 2, 49, 54] .Among the works that study inverse problems using generative models, the recent work [51] is the most relevant to ours. Specifically, in [51] , the authors considered generative model-based PCA (GPCA) and proposed a practical projected power (PPower) method. Furthermore, they showed that provided a good initial guess, PPower converges linearly to an estimated vector with the optimal statistical error. However, as pointed out in [84] for the case of sparse priors, due to the singular matrix B of the GEP, the corresponding methods for PCA generally cannot be directly applied to solve GEPs. Therefore, our work, which provides a unified treatment to GEP under generative priors, broadens the scope of [51] .A detailed discussion on the technical novelty of this work compared to [51] and [84] is provided in Section 2.2.The main contributions of this paper can be summarized as follows:• We provide theoretical guarantees regarding the optimal solutions to the GGEP in Eq. ( 6).Particularly, we demonstrate that under appropriate conditions, the distance between any optimal solution to Eq. ( 6) and the underlying signal is roughly of order O( (k log L)/m), assuming that the generative model G is L-Lipschitz continuous with bounded k-dimensional inputs. Such a statistical rate is naturally conjectured to be optimal based on the informationtheoretic lower bound established in [51] for the simpler GPCA problem.• We propose an iterative approach to approximately solve the non-convex optimization problem in Eq. ( 6), which we refer to as the projected Rayleigh flow method (PRFM). We show that PRFM converges linearly to a point achieving the optimal statistical rate under suitable assumptions.• We have conducted simple proof-of-concept experiments to demonstrate the effectiveness of the proposed projected Rayleigh flow method.We use upper and lower case boldface letters to denote matrices and vectors, respectively. We write We use ∥X∥ 2→2 to denote the spectral norm of a matrix X. We use λ min (X) and λ max (X) to denote the minimum and maximum eigenvalues of X respectively. S n-1 := {x ∈ R n : ∥x∥ 2 = 1} represents the unit sphere in R n . The symbols C, C ′ , and C ′′ are absolute constants whose values may differ from line to line. We use standard Landau symbols for asymptotic notations, with the implied constants in Ω(•) terms being implicitly assumed to be sufficiently large and the implied constants in O(•) terms being implicitly assumed to be sufficiently small.[N ] = {1, 2, • • • , N }Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: We write the abstract and introduction based on the contributions and scope of this paper.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
preliminary: 
 In this section, we formally introduce the problem and overview the main assumptions that we adopt. Additionally, we discuss the technical distinctions of this work in comparison with the two closely related works [84] and [51] . In Appendix A, we show examples of popular statistical and machine-learning problems that can be expressed as GGEP.Recall that we assume that we have access to a matrix pair ( Â, B) in R n×n constructed from m independent observations, satisfying Â = A + E, B = B + F (see Eq. ( 4)).Here, E and F are the unknown perturbation matrices, (A, B) is the underlying matrix pair with A symmetric and B positive definite, and the generalized eigenvalues are λ i with corresponding generalized eigenvectors v i respectively (see Eq. ( 1)). The generalized eigenvalues are ordered such thatλ 1 ≥ λ 2 ≥ . . . ≥ λ n .For brevity, we fix the norms of the v i such that v ⊤ i Bv i = 1 (and v ⊤ i Bv j = 0 for i ̸ = j). In addition, we assume that we have an L-Lipschitz continuous generative model G : R k → R n with k ≪ n and the radius r > 0. Based on this model formulation for GEP and the generative model, we further make the following assumptions.First, we assume that the generative model G has bounded inputs in R k and it is normalized in the sense that its range is contained in the unit sphere of R n . Assumption 2.1. For the sake of convenience, we assume that the generative model G maps from B k (r) to S n-1 for some radius r > 0. Remark 2.2. Our results can be readily extended to general unnormalized generative models by considering their normalized variants. More specifically, according to [4] , for an ℓ-layer fully connected feed-forward neural network from R k to R n with popular activation functions such as ReLU and sigmoid, typically, it is L-Lipschitz continuous with L = n Θ(ℓ) . In addition, as mentioned in [50, 51] , for a general unnormalized L-Lipschitz continuous generative model G, we can consider a corresponding normalized generative model G : D → S n-1 with D := {z ∈ B k 2 (r) : ∥G(z)∥ 2 > R min } for some R min > 0, and G(z) = G(z)/∥G(z)∥ 2 . Then, we can set R min to be as small as 1/n Θ(ℓ) and r to be as large as n Θ(ℓ) without altering the scaling laws, which renders the dependence on R min and r very mild.Next, we impose assumptions on the generalized eigenvalues and the leading generalized eigenvector. Assumption 2.3. We suppose that the largest generalized eigenvalue is strictly greater than the second largest generalized eigenvalue, i.e., λ 1 > λ 2 . Furthermore, we assume that the normalized leading generalized eigenvector v * := v 1 /∥v 1 ∥ 2 lies in the range of the generative model G, meaningv * ∈ R(G) ⊆ S n-1 .We also make appropriate assumptions on the perturbation matrices E and F. In particular, following [84, Proposition 2] and [51, Assumption 2], we make the following assumption for the perturbation matrices. Similarly to that mentioned in [84] , it is easy to verify that under generative priors, for various statistical models that can be formulated as a GEP such as CCA, FDA, and SDR (discussed in Appendix A), Assumption 2.4 will be satisfied with high probability. Additionally, similarly to the proof for the spiked covariance model in [51, Appendix B.1] , it can be readily demonstrated that for the ( Â, B) constructed in Section 4.2, the corresponding perturbation matrices E and F satisfy Assumption 2.4 with high probability. Assumption 2.4. Let S 1 , S 2 be two finite sets in R n satisfying m = Ω(log(|S 1 | • |S 2 |)).Then, for all s 1 ∈ S 1 and s 2 ∈ S 2 , we haveEQUATIONwhere C is an absolute constant. Moreover, we have ∥E∥ 2→2 = O(n/m) and ∥F∥ 2→2 = O(n/m).Remark 2.5. Assumption 2.4 is closely related to classic assumptions about the Crawford number of the symmetric-definite matrix pair (A, B) [84, 8] . More specifically, we have cr(A, B) := minEQUATIONAdditionally, based on the L-Lipschitz continuity of G and [87, Lemma 5.2], for any δ > 0, there exists a δ-netEQUATIONwithEQUATIONit follows from Assumption 2.4 thatEQUATIONTherefore, if m = Ω k log 4Lr δ with a sufficiently large implied constant, the conditions thatEQUATIONnaturally hold, where c, c ′ are certain positive constants. 2 This leads to an assumption similar to [84, Assumption 1].2.2 Discussion on the Technical Novelty Compared to [51] and [84] Our analysis builds on techniques from existing works such as [51] and [84] , but these techniques are combined and extended in a non-trivial manner; we highlight some examples as follows:• We present a recovery guarantee regarding the global optimal solutions of GGEP in Theorem 3.1. Such guarantees have not been provided for SGEP in [84] . Although in [51, Theorem 1], the authors established a recovery guarantee regarding the global optimal solutions of GPCA, its proof is significantly simpler than ours and does not involve handling singular B or using the property of projection as in Eq. ( 78) in our proof.• The convergence guarantee for Rifle in [84] hinges on the generalized eigenvalue decomposition of ( ÂF , BF ), where F is a superset of the support of the underlying sparse signal, along with their Lemma 4, which follows directly from [98, Lemma 12] and characterizes the error induced by the truncation step. Additionally, the convergence guarantee of PPower in [51] only involves bounding the term related to the sample covariance matrix V (as seen in their Eq. ( 83)). In contrast, in our proof of Theorem 3.3, we make use of the generalized eigenvalue decomposition of (A, B), and we require the employment of significantly distinct techniques to manage the projection step (and bound a term involving both Â and B, see our Eq. ( 111)). This is evidenced in the three auxiliary lemmas we introduced in Appendix B.1 and the appropriate manipulation of the first and second terms on the right-hand side of Eq. (114).• Unlike in GPCA studied in [51] , where the underlying signal can be readily assumed to be a unit vector that is (approximately) within the range of the normalized generative model, in the formulation of the optimization problem for GGEP (refer to Eq. ( 6) and the corresponding analysis, we need to carefully address the distinct normalization requirements of the generative model and B.Input: Â, B, G, number of iterations T , step size η > 0, initial vector u 0 for t = 0, 1, . . . , T -1 doEQUATIONEQUATIONend for Output: u TQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations of the work performed by us in Section 5, which motivate our future work.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
main results: 
 Firstly, we present the following theorem, which pertains to the recovery guarantees in relation to the globally optimal solution of the GGEP in Eq. ( 6). Similar to [51, Theorem 1], Theorem 3.1 can be easily extended to the case where there is representation error, i.e., the underlying signalv * / ∈ R(G).Here, we focus on the case where v * ∈ R(G) (cf. Assumption 2.3) to avoid nonessential complications. The proof of Theorem 3.1 is deferred to Appendix B. Theorem 3.1. Suppose that Assumptions 2.1, 2.3, and 2.4 hold for the GEP and generative model G. Let û be a globally optimal solution to Eq. (6) for GGEP. Then, for any δ ∈ (0, 1)EQUATIONwhere C 1 is a positive constant depending on A and B.As noted in [4] , an ℓ-layer neural network generative model is typically L-Lipschitz continuous with L = n Θ(ℓ) . Then, under the typical scaling of L = n Θ(ℓ) , r = n Θ(ℓ) , and δ = 1/n Θ(ℓ) , the upper bound in Eq. ( 13) is of order O( (k log L)/m), which is naturally conjectured to be optimal based on the algorithm-independent lower bound provided in [51] for the simpler GPCA problem.Although Theorem 3.1 demonstrates that the estimator for the GGEP in Eq. ( 6) achieves the optimal statistical rate, in general, the optimization problem is highly non-convex, and obtaining the optimal solution is not feasible. To address this issue, we propose an iterative approach that can be regarded as a generative counterpart of the truncated Rayleigh flow method proposed in [84] . This approach is used to find an estimated vector that approximates a globally optimal solution to Eq. ( 6). The corresponding algorithm, which we refer to as the projected Rayleigh flow method (PRFM), is presented in Algorithm 1.In the iterative process of Algorithm 1, the following steps are performed:• Calculate ρ t in Eq. ( 14) to approximate the largest generalized eigenvalue λ 1 . Note that from Lemma B.1 in Appendix B.1, we obtain u ⊤ t Bu t > 0 for t > 0 under appropriate conditions.• In Eq. 15, we perform a gradient ascent operation and a projection operation onto the range of the generative model, where P G (•) denotes the projection function. This step is essentially analogous to the corresponding step in [84, Algorithm 1] . However, instead of seeking the support for the sparse signal, our aim is to project onto the range of the generative model. Furthermore, we adopt a simpler choice for the step size η in the gradient ascent operation. 3Remark 3.2. Specifically, for any u ∈ R n , P G (u) ∈ arg min w∈R(G) ∥w -u∥ 2 . We will assume implicitly that the projection step can be performed accurately, as in [77, 71, 51] , for the convenience of theoretical analysis. In practice, however, approximate methods might be necessary, such as gradient descent [77] or GAN-based projection methods [74] .We establish the following convergence guarantee for Algorithm 1. The proof of Theorem 3.3 can be found in Appendix C.Theorem 3.3. Suppose that Assumptions 2.1, 2.3, and 2.4 hold for the GEP and generative model G.LetEQUATIONandEQUATIONwhereEQUATIONThen, for any δ ∈ (0, 1)satisfying δ = O (k log 4Lr δ )/n , when m = Ω k log 4Lr δ , there exists a positive integer T 0 = O log m k log Lr δ, such that the sequence {∥u t -v * ∥ 2 } t≤T0 is monotonically decreasing, with the following inequality holds for all t ≤ T 0 :EQUATIONwhere C 2 is a positive constant depending on A, B, and η. Additionally, we have for all t ≥ T 0 thatEQUATIONTheorem 3.3 establishes the conditions under which Algorithm 1 converges linearly to a point that achieves the statistical rate of order O( (k log L)/m). Remark 3.4. Both inequalities in Eqs. ( 16) and ( 17) can be satisfied under appropriate conditions. More specifically, first, under suitable conditions on the underlying matrix pair (A, B) and the step size η, the condition in ( 16) can hold. Moreover, using of the inequality that 2 (1 -c)c ≤ 1 for any c ∈ [0, 1), we obtain that whenEQUATIONor equivalently,EQUATIONthe condition in Eq. ( 17) holds. Then, for example, when κ(B) := λ max (B)/λ min (B) is close to 1, and ν 0 is close to 1, (22) can be approximately simplified asEQUATION)Note that both the conditions γ 1 + γ 2 < 2 (in Eq. ( 16)) and γ 2 + 3γ 1 > 3 can be satisfied for appropriate γ 1 and γ 2 (under suitable conditions for A, B, and η), say γ 1 = 2 3 and γ 2 = 1.1. Remark 3.5. In certain practical scenarios, we may assume that the data only contains non-negative vectors, for example, in the case of image datasets. Additionally, during pre-training, we can set the activation function of the final layer of the neural network generative model to be a non-negative function, such as ReLU or sigmoid, further restricting the range of the generative model to the non-negative orthant. Therefore, the assumption that ν 0 := u ⊤ 0 v * > 0 is mild (in experiments, we simply set the initial vector u 0 to be u0 = [1, 1, . . . , 1] ⊤ / √ n ∈ R n ), and similar assumptions have been made in prior works such as [51] . As a result, we provide an upper bound on ∥ut -v * ∥ 2 instead of on min{∥u t -v * ∥ 2 , ∥u t + v * ∥ 2 }.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiments: 
 In this section, we conduct proof-of-concept numerical experiments on the MNIST dataset [47] to showcase the effectiveness of the proposed Algorithm 1. Additional results for MNIST and CelebA [55] are provided in Appendices D and E. 4 We note that these experiments are intended as a basic proof of concept rather than an exhaustive study, as our contributions are primarily theoretical in nature.The MNIST dataset consists of 60, 000 images of handwritten digits, each measuring 28 × 28 pixels, resulting in an ambient dimension of n = 784. We choose a pre-trained variational autoencoder (VAE) model as the generative model G for the MNIST dataset, with a latent dimension of k = 20. Both the encoder and decoder of the VAE are fully connected neural networks with two hidden layers, having an architecture of 20 -500 -500 -784. The VAE is trained using the Adam optimizer with a mini-batch size of 100 and a learning rate of 0.001 on the original MNIST training set. To approximately perform the projection step P G (•), we use a gradient descent method with the Adam optimizer, with a step size of 100 and a learning rate of 0.1. This approximation method has been used in several previous works, including [77, 71, 50, 51] . The reconstruction task is evaluated on a random subset of 10 images drawn from the testing set of the MNIST dataset, which is unseen by the pre-trained generative model. We compare our Algorithm 1 (denoted as PRFM) with the projected power method (denoted as PPower) proposed in [51] and the Rifle method (e.g., denoted as Rifle20 when the cardinality parameter is set to 20) proposed in [84] .To evaluate the performance of different algorithms, we employ the Cosine Similarity metric, which is calculated as CosSim v * , ũ) = ũ⊤ v * . Here, v * is the target signal that is contained in the unit sphere, and ũ denotes the normalized output vector of each algorithm. To mitigate the effect of local minima, we perform 10 random restarts and select the best result from these restarts. The average Cosine Similarity is calculated over the 10 test images and the 10 restarts. All experiments are carried out using Python 3.10.6 and PyTorch 2.0.0, with an NVIDIA RTX 3060 Laptop 6GB GPU.Firstly, we adhere to the experimental setup employed in [8] , where the underlying matrices A and B are set to be A = 4v * (v * ) ⊤ + I n and B = I n respectively. We sample m data points following N (0, A), and another m data points following N (0, B). The approximate matrices Â and B are constructed based on the sample covariances correspondingly. Specifically, we setEQUATIONHere, γ i are independently and identically distributed (i.i.d.) realizations of the standard normal distribution N (0, 1), z i are i.i.d. realizations of N (0, I n ), and w i are also i.i.d. realizations of N (0, I n ). α i , z i , and w i are independently generated. The leading generalized eigenvector v * is set to be the normalized version of the test image vector. Note that in this case, the generalized eigenvalues are λ 1 = 5 and λ 2 = . . . = λ n = 1. To ensure that the conditions in Eqs. ( 16) and ( 23) are both satisfied, we set the step size η for PRFM to be η = 7 32 . The step size η ′ for Rifle is set to 35/32 such that η ′ /ρ t-1 ≈ 7/32 = η. For all the methods, the initialization vector u 0 for is set to be the normalized vector of all ones, namely u 0 = [1, 1, . . . , 1] ⊤ / √ n ∈ R n , which naturally guarantees that ν 0 = u ⊤ 0 v * > 0 since the image vectors in the MNIST dataset contain only non-negative entries. For PPower, the input matrix set to be Â (ignoring the fact that B is not exactly the identity matrix; see [51, Algorithm 1] ). We vary the number of measurements m in {100, 150, 200, 250, 300, 350}. these figures, we can see that when the number of measurements m is relatively small compared to the ambient dimension n, sparsity-based methods Rifle20 and Rifle100 lead to poor reconstructions, and PPower and PRFM can achieve reasonably good reconstructions. Moreover, PRFM generally yields significantly better results than PPower. This is not surprising as PPower is not compatible with the case where B is not the identity matrix.Next, we also investigate the case whereEQUATIONwhere g i ∈ R n are independent standard Gaussian vectors andy i = (g ⊤ i v * ) 2 .The generation of Â corresponds to the phase retrieval model. Other experimental settings remain the same as those in the previous case where Â and B are generated from Eq. (24) . The corresponding numerical results are presented in Figures 1(b ) and 2(b). From these figures, we can see that PRFM also leads to best reconstructions in this case. 
conclusion and future work: 
 GEP encompasses numerous significant eigenvalue problems, motivating this paper's examination of GEP using generative priors, referred to as Generative GEP and GGEP for brevity. Specifically, we assume that the desired signal lies within the range of a specific pre-trained Lipschitz generative model. Unlike prior works that typically addressed high-dimensional settings through sparsity, the generative prior enables the accurate characterization of more intricate signals. We have demonstrated that the exact solver of GGEP attains the optimal statistical rate. Furthermore, we have devised a computational method that converges linearly to a point achieving the optimal rate under appropriate assumptions. Experimental results are provided to demonstrate the efficacy of our method.In the current work, for the sake of theoretical analysis, we adhere to the assumption made in prior works like [77, 71, 51] that the projection onto the range of the generative model can be executed accurately. However, in experiments, this projection step can only be approximated, consuming a substantial portion of the running time of our proposed method. Developing highly efficient projection methods with theoretical guarantees is of both theoretical and practical interest. Another intriguing area for future research is to provide guarantees for estimating multiple generalized eigenvectors (or the subspace they span) under generative modeling assumptions. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We have reported error bars suitably and correctly in Figure 2 . Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: As far as we can tell, this paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [NA] Justification: This paper does not use existing assets. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: GEP encompasses a broad range of eigenvalue problems, and in the following, we specifically discuss three noteworthy examples under generative priors.• Generative Principle Component Analysis (GPCA). Given a data matrix Σ that typically represents the covariance matrix of m observed data points with n features, the goal of PCA is to find a projection that maximizes variance. Under the generative prior, the generative PCA (GPCA) problem can be formulated as follows (assuming that the range of the generative model G is a subset of the unit sphere for simplicity)EQUATIONThis problem can be derived from the optimization problem for GGEP in Eq. ( 6) with Â = Σ and B = I n .• Generative Fisher's Discriminant Analysis (GFDA). Given m observations with n features belonging to K different classes, Fisher's discriminant analysis (FDA) aims to find a lowdimensional projection that maps the observations to a lower-dimensional space, where the between-class variance Σ b is large while the within-class variance Σ w is relatively small. Let ( Σb , Σw ) be the empirical version of (Σ b , Σ w ) constructed from the observations. Under generative priors, the goal of generative FDA (GFDA) is to solve the following optimization problem:EQUATIONwhich can be derived from the optimization problem for GGEP in Eq. ( 6) with Â = Σb and B = Σw .• Generative Canonical Correlation Analysis (GCCA). Given random vectors x, y ∈ R p and their realizations, let Σ xx and Σ yy be the corresponding covariance matrices, Σ xy be the cross-covariance matrix between x and y, and Σxx , Σyy , Σxy be the empirical version of Σ xx , Σ yy , Σ xy , respectively. Then, canonical correlation analysis aims to solve the following optimization problem:EQUATIONThe optimization problem in Eq. ( 28) can be equivalently formulated asEQUATIONwhereEQUATIONand u = u x u y . Imposing the generative prior on u and adding the corresponding constraint into Eq. ( 29) (leading to the optimization problem for GGEP in Eq. ( 6)), we derive generative CCA (GCCA).B Proof of Theorem 3.1 (Guarantees for Globally Optimal Solutions)B.1 Auxiliary Lemmas for the Proof of Theorem 3.1We present the following useful lemma, which shows that for any globally optimal solution û to GGEP, we have û⊤ Bû > 0 under appropriate conditions. Proof. For any u ∈ S n-1 , we have u ⊤ Bu ≥ λ min (B), where λ min (B) > 0 denotes the smallest eigenvalue of the positive definite matrix B. Additionally, let M be a (δ/L)-net of B k (r). From [87, Lemma 5.2], we know that there exists such a net withEQUATION)Since G is L-Lipschitz continuous, we have thatEQUATIONEQUATIONEQUATION)Since w ∈ M ′ is a unit vector, we have w ⊤ Bw ≥ λ min (B). Using Assumption 2. First, since dim(span{v 1 , v 2 , . . . , v n }) = n, we know that there exist coefficients g i such that the unit vector û can be written asEQUATIONLet d = 1/∥v 1 ∥ 2 . We have v * = dv 1 = v1 ∥v1∥2 ∈ R(G). Additionally, we have the following:EQUATION)Since v ⊤ i Bv i = 1 and for i ̸ = j, v ⊤ i Bv j = 0, if letting ṽi = B 1/2 v i , we observe that ṽ1 , . . . , ṽn form an orthonormal basis of R n . Then, if setting ũ = B 1/2 û = n i=1 g i ṽi , we have n i=1EQUATIONCombining Eqs. ( 40) and ( 41), we obtainEQUATIONIn addition, we haveEQUATIONwith (v * ) ⊤ Bv * = d 2 , û⊤ Bû ≥ λ min (B), and the numeratorEQUATIONEQUATIONwhere Eq. ( 45) follows from the conditions that û is an optimal solution to Eq. ( 6) and v * ∈ R(G) (note that it follows from Lemma B.1 that û⊤ Bû > 0 and (v * ) ⊤ Bv * > 0). Let M be a (δ/L)-net of B k (r). From [87, Lemma 5.2], we know that there exists such a net withEQUATIONNote that since G is L-Lipschitz continuous, we have thatEQUATIONwhere ū ∈ G(M ) satisfies ∥û -ū∥ 2 ≤ δ. For the first term in the right-hand side of Eq. ( 45), we have 6EQUATIONFrom Assumption 2.4 and Eq. ( 46), we obtainEQUATIONAdditionally, we haveEQUATIONEQUATIONEQUATIONEQUATION6 For brevity, throughout the following, we assume that both E and F are symmetric, and we make use of the equality that for any s1, s2 ∈ R n and any symmetric G ∈ R n×n , s T 1 Gs1 -s T 2 Gs2 = (s1 + s2) T G(s1 -s2). For the case that E and F are asymmetric, we can easily obtain similar results using the equality that for any s1, s2 ∈ R n and anyG ∈ R n×n , s T 1 Gs1 -s T 2 Gs2 = 2 s 1 +s 2 2 T G s 1 -s 2 2 + 2 s 1 -s 2 2 T G s 1 +s 2 2 .where we use ∥F∥ 2→2 = O(n/m) and δ ∈ (0, 1) in Eq. ( 53), and Eq. ( 54) follows from Assumption 2.4 and Eq. ( 46). Therefore, combining Eqs. ( 50) and ( 54), we obtain that the first term in the right-hand side of Eq. ( 49) can be upper bounded as follows:EQUATIONWe have a similar bound for the second term in the right-hand side of Eq. ( 49), which givesEQUATION)Moreover, for the second term in the right-hand side of Eq. ( 45), we haveEQUATIONwhereEQUATIONEQUATIONEQUATIONwithC ′ 1 denoting ∥A∥ 2→2 + C k log 4Lr δ m. Similarly to Eq. ( 50), we haveEQUATIONIn addition, we haveEQUATIONandEQUATIONEQUATION)Combining Eqs. ( 58), ( 62), ( 63), ( 64), (67) , we obtainEQUATION)Similarly, we have that the third term in the right-hand side of Eq. ( 45) can be bounded asEQUATIONThen, combining Eqs. ( 45), ( 56), ( 68), (69) , and setting δ = O(m/n) with a sufficiently small implied constant, we obtainEQUATION)Therefore, combining Eqs. ( 43) and ( 70), we obtainEQUATION)Combining Eqs. ( 42) and ( 71), we obtainEQUATIONFurthermore, recall that we set ũ = B 1/2 û and ṽ1 = B 1/2 v 1 , which leads toEQUATIONOn the other hand, we haveEQUATIONTherefore, combining Eqs. ( 75) and ( 76), we obtainEQUATION)Note that by the property of projection, we haveEQUATION)In addition, note that for any unit vectorsEQUATIONTherefore, we obtainEQUATION)Therefore, combining Eqs. ( 72) and ( 82), we obtainEQUATIONEQUATIONwhereC ′′ 1 := 20C ′ 1 •λmax(B) d 2 (λ1-λ2)•λ 2 min (B). Without loss of generality, we assume that û⊤ v ≥ 0. For this case, it follows from Eq. ( 84) thatEQUATIONThen, if settingEQUATION)Similarly, if assuming û⊤ v < 0, we obtainEQUATIONwhich gives the desired result.C.1 Auxiliary Lemmas for Theorem 3.3Before presenting the proof for Theorem 3.3, we first provide some useful lemmas. Lemma C.1. For any ρ ∈ (λ 2 , λ 1 ] and any x =EQUATION)In addition, we haveEQUATION)From Eq. ( 92), we obtainEQUATIONSimilarly, we haveEQUATIONLemma C.2. For any ρ ∈ (λ 2 , λ 1 ] and any x =EQUATIONProof. From Lemma C.1, we haveEQUATIONEQUATIONEQUATIONProof. We haveEQUATIONIn addition, if letting ε = 1 -x ⊤ v * = 1 -ν, we have ε < 1 andEQUATIONEQUATIONC.2 Complete Proof of Theorem 3.3EQUATION) which gives ∥h t+1 ∥ 2 2 = ∥u t+1 -v * ∥ 2 2 ≤ 2⟨ũ t+1 -v * , h t+1 ⟩ (110) = 2⟨h t -η(ρ t B -Â)u t , h t+1 ⟩ (111) = 2h ⊤ t h t+1 -2η⟨(λ 1 B -A)u t , h t+1 ⟩ + 2η⟨(λ 1 B -ρ t B + ( Â -A))u t , h t+1 ⟩ (112) = 2h ⊤ t h t+1 -2η⟨(λ 1 B -A)h t , h t+1 ⟩ + 2η⟨(λ 1 B -ρ t B)u t , h t+1 ⟩ + 2η⟨( Â -A)u t , h t+1 ⟩ (113) = 2h ⊤ t h t+1 -2η⟨(λ 1 B -A)h t , h t+1 ⟩ + 2η⟨(λ 1 -ρ t )Bu t , h t+1 ⟩ + 2η⟨ρ t (B -B)u t , h t+1 ⟩ + 2η⟨( Â -A)u t , h t+1 ⟩.(114) Let us bound the terms in the right-hand side of Eq. ( 114) separately.EQUATIONwe obtainEQUATIONEQUATIONMoreover, if lettingEQUATIONEQUATIONwhere we set ν t := u ⊤ t v * ,7 κ(B) = λ max (B)/λ min (B), and Eq. (120) follows from Lemma C.3.• The term 2η⟨(λ 1 -ρ t )Bu t , h t+1 ⟩: We haveEQUATIONNote that from the calculations in Lemma C.3 (cf. Eq. ( 106)), we know i≥2EQUATIONIn addition, we haveEQUATIONwhich givesEQUATIONSimilarly to Eq. ( 47), we know that u t can be written asEQUATIONwhere ūt ∈ G(M ) satisfies the condition that ∥u t -ūt ∥ 2 ≤ δ. Then, we haveEQUATIONEQUATIONEQUATIONEQUATIONwhere Eq. ( 132) follows from the condition that δ = O k log 4Lr δ /m . Similarly, we can obtain thatEQUATIONCombining Eqs. (124), (125), (127), (132), and (133), we obtainEQUATIONEQUATIONTherefore,EQUATIONwhere Eq. ( 139) follows from the setting that γ 2 = η(λ 1 -λ n )λ max (B).• The term 2ηρ t ⟨(B -B)u t , h t+1 ⟩: We haveEQUATIONEQUATIONTherefore, we obtainEQUATION• The term 2η⟨( Â -A)u t , h t+1 ⟩: Similarly to Eq. ( 145), we obtainEQUATIONCombining Eqs. ( 114), ( 122), ( 140), (145), and (146), we obtainEQUATIONThen, if lettingEQUATIONEQUATIONwe obtainEQUATIONwhich leads toEQUATIONThen, if δ > 0 satisfies the condition that δ = O k log 4Lr δ /n , we obtainEQUATIONwhere C 2 is a positive constant obtained from to Eq. ( 152), depending on A, B, and η. This completes the proof.In this section, we perform the experiments for the case that B in Eq. ( 24) is modified by w i ∼ N (0, Diag(2, 1, . . . , 1)) (other settings of the experiments remain unchanged), and thus B = E[ B] = Diag(2, 1, . . . , 1) and κ(B) = λ max (B)/λ min (B) = 2. The quantitative results are shown in the following table, from which we observe that our PRFM method also performs well in this case. m Rifle20 Rifle100 PPower PRFM 100 0.17 ± 0.02 0.27 ± 0.01 0.75 ± 0.03 0.80 ± 0.02 200 0.28 ± 0.01 0.43 ± 0.01 0.78 ± 0.01 0.86 ± 0.02 300 0.32 ± 0.01 0.50 ± 0.01 0.79 ± 0.01 0.91 ± 0.01The CelebA dataset contains over 200,000 face images of celebrities. Each input image is cropped for the deep convolutional generative adversarial networks (DCGAN) model with a latent dimension of k = 100. 8 Among 5 random restarts, we choose the best estimate. For the projection operator, the Adam optimizer is utilized with 100 steps and a learning rate of 0.1.Similar to MNIST, we also consider two cases where ( Â, B) is generated from Eqs. (24) and (25) . The experimental results are presented in Figures 3 and 4 . While the CelebA dataset is publicly accessible and widely used, we would like to point out the potential for issues such as ethnic-group switching during face-image reconstruction.• Bias and Stereotyping: If a generative model trained on the CelebA dataset exhibits ethnicgroup switching during reconstruction, it could reinforce harmful stereotypes. For example, if certain facial features are associated with a particular ethnicity in an inaccurate or unfair way, it can lead to the misrepresentation and marginalization of that ethnic group. This can also affect how society perceives different ethnicities, potentially leading to discrimination in areas such as employment, housing, and social interactions. • Invasion of Privacy and Consent: In the context of more sensitive data, if a generative model can manipulate or change aspects of an individual's appearance in a way that they did not consent to, it is a serious invasion of privacy. For example, if a model is used to generate images of someone in a different ethnic guise without their permission, it can cause emotional distress and harm to their self-identity. The use of such models on data where the individuals have not given proper consent for this type of manipulation can lead to legal and ethical dilemmas. • Social and Cultural Impact: Changing the ethnicity of a face in an image can have significant social and cultural implications. It can undermine the cultural identity of individuals and communities. For instance, if a model is used to "erase" or change the ethnic features of a cultural icon in an image, it can be seen as an act of cultural appropriation or disrespect. It can also affect the way cultural heritage is represented and passed down, as the authenticity of images related to a particular culture may be compromised by unethical generative model manipulations.NeurIPS Paper ChecklistNote that although we use notations such as cr(A, B) and λmin(B), both A and B are considered fixed matrices, and for the ease of presentation, we omit the dependence on them for relevant positive constants.In[84, Algorithm 1], the step size is η/ρt, which depends on t and is approximately equal to η/λ1.For the CelebA dataset, which is publicly accessible and widely used (containing face images of celebrities), we point out the potential ethical problems in Appendix E.Note that as mentioned in the footnote at the end of Remark 2.5, A and B are considered fixed matrices, and for brevity, we omit the dependence on them for the relevant positive constants.By proof of induction, using the same proof strategy to follow, we can show that the sequence {νt} t≥0 is monotonically non-decreasing. We omit the details for brevity.We follow the PyTorch DCGAN tutorial in https://pytorch.org/tutorials/beginner/dcgan_ faces_tutorial.html?highlight=dcgan to pre-train the model. 


Decision: This manuscript address how to solve symmetric (semi-definite) generalized eigenvalue problems when there is a reasonable generative prior that can be placed on the eigenvectors. Theoretical results are provided that highlight the quality of the optimal solution and practical algorithms are discussed.

Overall, the reviewers feel that there is a clear contribution here given both the prevalence of GEPs and the advent of using generative models as priors. While the algorithm is similar in spirit to prior work, the theoretical discussions highlight its potential efficacy. 

Nevertheless, there are some lingering concerns about the validity/sensibility of various assumptions and the limited computational experiments. The authors rebuttal did partially address the former point, though I do think that there are som lingering questions. Of note, some of the assumptions (such as an exact projection) do appear elsewhere and are not unreasonable to make when analyzing the problem—plus the limitation of this assumption is clearly discussed/addressed.

NB: one aspect of the presentation that could use some work is a lack of engagement with the broader symmetric GEP literature. For example, deflation (as alluded to in lines 25 and 26) is not commonly preferred for targeting multiple eigenvalue/vector pairs and many other algorithms exist (such as generalizations of Rayleigh quotient iteration, subspace style iterations, the QZ method, etc.). I would encourage the authors to provide some pointers to the broader literature for solving GEPs since that likely is quite relevant here. In addition, the authors should be more clear that their results are for symmetric semi-definite GEPs—this is not always clear. Admittedly the use of SGEP for sparse GEPs complicates this matter from a notational perspective, but the results are very clearly limited to the symmetric case and that should be more clear (e.g., in the title and abstract). There are also lots of works on more general (i.e., not symmetric) GEPs, but this work does not fall in that category.

Reviews: review: paper_summary: paper_summary: This submission proposes a way to solve Generalized Eigenproblems with a constraint on the eigenvectors to be in the range of some generative model. On top of a simple algorithm, the paper proposes bound on the optimal solution and some experiments on some toy data.
strength_weakness: strength_weakness: The presentation of the problem could be improved (for exemple, it is never said that Eq 3 is a Rayleigh quotient), section 2.2 gather folklore results that are not useful (in my opinion).
It is also difficult to follow how the algorithm 1 is derived from section 3. 
The numerical experiments are carried out on toy data and it makes it difficult to understand how it could be used in practice.
questions: questions: I am not quite sure to understand how the generative prior (and the projection onto the range of a pre trained model) is not redondant with the optimization problem itself. If the data are distributed according to some distribution, then the covariance of the data should be in this range. There is for sure something that I missed and it was not clear from the application but could you clarify this point ?

In Th 3.3, beside being a technical condition, is there any impact of the condition \gamma_1 + \gamma_2 < 2 ? Is it a reasonable constraint ?

In the main results, the $\mathcal{P}_G(\dot)$ has to be exact. In the experiments, it is approximated. How does this approximation impact the algorithm and the numerical results ?

The proposed algorithm is somewhat similar to a projected power method. How stable is it when k grows ? (As it is know to behave poorly as k grows).
On possible solution could be to solve the problem as the $\max_U Tr (U^T A U)$ with U in a genaralized Stiefel manifold (defined with B) and the generative prior could be enforced with a regularization. This should prove to be more stable than the power method-like method. 

Section C of the appendix is very interesting and should be included in the paper (although the argument about having a proof that is bigger than the competitors is clearly not a valid argument …)
limitations: limitations: Since the paper involves the use of a generative model, it could have some negative implication for the society. A discussion on that topic should be included.
paper_summary: paper_summary: The paper studies generalized eigenvalue problems under generative priors. They show that under suitable conditions on the prior assumptions on the perturbation matrices, the optimal solution vector of the corresponding optimization problem attains the statistically optimal rate. Furthermore, they provide an algorithm that under assumptions on the signal strength, step size, and initialization vector, converges linearly to a statistically optimal solution vector. They further supplement their theoretical analysis with experiments on the MNIST dataset, while comparing with other approaches for solving generative generalized eigenvalue problems.
strength_weakness: strength_weakness: - Regarding Computational Efficiency: As acknowledged by the authors, projection on the prior set may not be efficiently achievable in general and approximations may be required. Theoretical analysis taking the approximation into account would be good to see. Furthermore, even with the current approximate projection based off gradient descent, I would like to see how the proposed algorithm compares in compute time to other methods such as PPower.

- Regarding Eq (22): It seems to me that the condition assumed here essentially implies a local convergence guarantee. In Line 252, the authors mention that if $\nu_{0}$ is close to 1, then the conditions (21), (22) are easy to verify. However, achieving $\nu_{0}$ close to 1 itself seems like a hard problem since it would imply that you already have a pretty good initialization to start with, which may be hard to find in high dimensions.
questions: questions: - Regarding practical examples/instances:  Apart from the numerical experiments pointed out in the paper, are there other scenarios where assuming existence of such generative priors is reasonable/well-accepted?

- Regarding Assumption 2.4. : Outside the spiked covariance model, can the authors specify other covariance matrices where this assumption is satisfied?
limitations: limitations: The primary limitation seems to be assuming the existence of an exact projection step. The authors address this adequately.
paper_summary: paper_summary: This paper provides theoretical guarantees for the optimal solution of the generalised eigenvalue problem with generative priors. It also designs an algorithm to approximate the optimal solution within some guaranteed distance.
strength_weakness: strength_weakness: + Assumption 2.4 looks too strong and not very convincing. As shown in (12), this condition is even stronger than [72, Assumption 1]. At the first sight, I don't see any pair ($\mathbf{E}$,$\mathbf{F}$) which satisfies this assumption except the trivial one $\mathbf{E}=0$ and $\mathbf{F}=0$. Please find at least one class of $(\mathbf{E}, \mathbf{F})$ which satisfies this assumption. 
+ Similarly, it looks not interesting to only put three conditions (21), (22), (23) in Theorem 3.3 but don't identify (at least) a class of matrices ($\mathbf{A}, \mathbf{B})$ which satisfies these conditions. Please give some classes of $(\mathbf{A}, \mathbf{B})$ such that these conditions hold. It is more interesting if $\mathbf{B}$ is not the trivial one, i.e., $\mathbf{B}=\mathbf{I}_n$, as in the experiment since you are considering the generalised eigenvalue problem.
questions: questions: Please address my comments in the weakness. Additional notes:
+ Typo in (11): I think the constant should be $2C$ (not $\sqrt{2} C$). 
+ Typo: The second and third terms in (34) and (35) should be $(\mathbf{\hat{u}}-\mathbf{w})^T \mathbf{\hat{B}}\mathbf{w}$ and $\mathbf{w}^T \mathbf{\hat{B}}(\mathbf{\hat{u}}-\mathbf{w})$.
+ Typo: The second term in (49) should be $-$.
limitations: limitations: This is a theoretical research paper, hence the negative society impact of this work is not direct. The authors mention some technical limitations of this work though assumptions.

Comments: title: Reply to the author rebuttal
Responses to Ethics Reviewer hGfQ
Responses to Reviewer PACz
comment: Thank you very much for your detailed answers to my comments. Based on your answers, I raised my score to 5. Please add these to your revised version.
Dear Reviewer 5DuU,

We have taken your initial feedback into meticulous consideration in our responses. Could you please check whether our responses have appropriately addressed your concerns? If so, could you please kindly consider increasing your initial score accordingly? Certainly, we are more than happy to answer your further questions.

Thank you for your valuable time and dedicated effort in reviewing our work!

Best Regards,

Authors
Thank you for your detailed ethics reviews. We will include a brief acknowledgement of the ethical concern when applying our method to the CelebA dataset, which is a publicly accessible dataset consisting of face images of celebrities.
Thank you for your responses and for raising the score. We will incorporate a discussion concerning the application of generative models to more sensitive data/applications in the revised version.
I thanks the authors for their answer which covers some of my concerns and I will raise my score.

About the limitations and ethics, although the authors only work on pre-trained models with classical dataset, they are tinkering with generative models. The implications for other generative models (on more sensitive data/applications) could be huge and it would deserve some discussion.
I thank the authors for their response. I maintain my current score.