Title: An Iterative Algorithm for Differentially Private $k$-PCA with Adaptive Noise
Venue: NeurIPS 2025
Accepted: yes
Keywords: Differential Privacy

Stochastic PCA

Adaptive Noise
Abstract: Given $n$ i.i.d.random matrices $A_i \in \mathbb{R}^{d \times d}$ that share common expectation $\Sigma$, the objective of Differentially Private Stochastic PCA is to identify a subspace of dimension $k$ that captures the largest variance directions of $\Sigma$, while preserving differential privacy (DP) of each individual $A_i$. Existing methods either (i) require the sample size $n$ to scale super-linearly with dimension $d$, even under Gaussian assumptions on the $A_i$, or (ii) introduce excessive noise for DP even when the intrinsic randomness within $A_i$ is small.~\citet{liu2022dp} addressed these issues for sub-Gaussian data but only for estimating the top eigenvector ($k=1$) using their algorithm DP-PCA. We propose the first algorithm capable of estimating the top $k$ eigenvectors for arbitrary $k \leq d$, whilst overcoming both limitations above. For $k=1$, our algorithm matches the utility guarantees of DP-PCA, achieving near-optimal statistical error even when $n = \tilde{O}(d)$. We further provide a lower bound for general $k > 1$, matching our upper bound up to a factor of $k$, and experimentally demonstrate the advantages of our algorithm over comparable baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper consider differentialy private $k$-PCA, where the goal is to privately estimate the $k$ leading eigenvectors of the covariance matrix of data. The method and analysis is tailored to the stochastic setting, where the data vectors are drawn from a given distribution. The paper makes improvements over the prior works in essentially two ways: first, it improves the baseline method and analysis by Liu et al. (2022) which is for eastimating the leading eigenvectors of the covariance matrix, and second, the paper applies the deflation techniques presented by Jalumbati et al. (2024) to apply the single vector estimator $k$ times and carrying out deflation. Numerical comparisons to the baseline method by Liu et al. (2022) and also to the Analyze Gauss (Dwork et al., 2014) which works in the data-deterministic setting and where the whole covariance matrix is pertubed verify the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Improvement of the single vector method by Liu et al. (2022)\n\n- Novel analysis based on a novel analysis of the non-private method by Jain et al. (2016)\n\n- Clear novelty: method to estimate $k$ leading vectors, enabled by the deflation (Jalumbati, 2024)\n\n- The paper is very well written, it is nice that there are examples of data distributions that satisfy the assumptions of the analysis vs. exaples that do not satisfy.\n\n\nWeaknesses:\n\n- The experimental section is limited, as there are two examples with synthetic data and in both cases $k=2$. Comparisons with some real-world data with larger $k$ would strengthen the experimental section.\n\n- The selection of baseline methods is limited. The method by [Cai et al. (2024)](https://arxiv.org/pdf/2401.03820) is not considered in the baselines. But most importantly, the noisy power iteration is not considered at all. There are recent works, e.g.\n\n[Nicolas, J., Sabater, C., Maouche, M., Mokhtar, S. B., & Coates, M. (2024). Differentially private and decentralized randomized power method.](https://arxiv.org/pdf/2411.01931)\n\nthat are block iterations and consider computing simultaneously the leading $k$-eigensubspace for the covariance matrix. There is also the analysis of \n\n[Balcan, M. F., Du, S. S., Wang, Y., & Yu, A. W. An improved gap-dependency analysis of the noisy power method. In Conference on Learning Theory 2016](http://proceedings.mlr.press/v49/balcan16a.pdf)\n\nwhich gives an error analysis for the noisy power method, that could be directly compared to the results presented here.\n\nThe existing methods and analyses for the noisy power iteration, to the best of my knowledge, are for the data-deterministic setting. However, you could compare to that method using similar modifications that you make for the Analyze Gauss.\n\nQUESTIONS:\n- Can you comments on how the deflation method would compare to block iteration methods such as the noisy power method (Hardt and Price, 2014, Balcan et al., 2016) ? Both theoretically and numerically? As that is fundamentally a different approach (deflation vs. block iteration), I think it would be important to mention that approach.\n\n- Why did you not consider other baselines than Analyze Gauss and the method by Liu et al. ?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles differentially-private stochastic k-PCA. It combines a deflation (Algorithm 1 k-DP-PCA) with a new 1-PCA oracle (MODIFIEDDP-PCA). Under a sub-Gaussian data model, they authors prove $(\\epsilon,\\delta)$-DP and show that with only $n = \\tilde{O}(d)$ samples their estimator recovers a $k$-dimensional subspace whose retained variance has near-optimal dependence on $k, n$. An information-theoretic lower bound for the case of the spiked covariance model matches dependence on factors except $k$. The authors further supplement their theoretical results with experiments to show the linear-scaling of the error with data dimension $d$.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths :\n\nA clear rigorous proof of differential privacy and utility for k-PCA with a nearly tight lower bound and efficient implementation. Nice combination and generalization of the ideas of Jambulapati et al. (2024) and Liu et al (2022). I have been keeping up with the literature and I think the paper is overall well-written and a nice contribution to the privacy and PCA literature.\n\nWeaknesses/Questions : \n\nI don't see any obvious weaknesses apart from the ones the authors acknowledged in Section 6, so I will ask state some questions/suggestions instead.\n\n1. I would appreciate it if, apart from the lower bound, the authors could bring out their contributions over Jambulapati et al. (2024) and Liu et al (2022) and mention novel parts of their analysis, even as a sketch, as compared to these existing analysis.        \n\n2. Line 198-199 : The authors mention that Jambulapati et al's ePCA oracle definition doesn't extend to the stochastic case. However, based on my understanding, Jambulapati et al. do provide bounds for robust k-PCA where $1-\\epsilon$ fraction of the data is drawn iid from some distribution. Therefore, I am a little confused as to what the authors meant with this claim.\n\n3. Regarding dependence on $k$ : Again referencing Jambulapati et al., based on my understanding, for the case of robust heavy-tailed data, they show that in fact a union bound over the $k$ deflation steps is not necessary and a stability result in the first stage implies stability for the subsequent stages due to the nature of the deflation. This helps them save a $k$ factor in the sample complexity. \n\nI am wondering if such a trick would work in the DP case as well? \n\n4. Notion of $k$-PCA : The authors consider energy k-PCA. Another notion of PCA convergence is the correlation k-PCA which measures mis-alignment between estimated and true eigenspaces. I am just wondering if the authors believe that their results provide near-optimal guarantees for such a case as well.\n\nQUESTIONS:\nPlease see weaknesses section above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

comments: SUMMARY:
The proposed stochastic k-PCA algorithm not only has differential privacy and computational efficiency, but also overcomes the following two limitations: (i) require the sample size n to scale super-linearly with dimension d, even under Gaussian assumptions on the Ai, or (ii) introduce excessive noise for DP even when the intrinsic randomness within Ai is small.

STRENGTHS AND WEAKNESSES:
This paper proposes the first algorithm for stochastic k-PCA that is both differentially private and computationally efficient, supports any k≤d, and achieves near-optimal error.
Weaknesses: 
Liu et al. [2022a] addressed these issues for sub-Gaussian data but only for estimating the top eigenvector (k = 1) using their algorithm DP-PCA. This paper proposes a kind that can estimate the top k eigenvectors of any k≤d. For k = 1, the algorithm in this paper matches the utility guarantee of DP-PCA and improves the computational efficiency. It also provides a lower bound for the general k > 1. The essence of the paper can be regarded as an expansion of Liu et al. [2022a].

QUESTIONS:
1. In the experimental part, only k=2 was set to compare with other algorithms, and the advantages of the algorithm can not be demonstrated.
2. Line 230, “However, for other cases such as sub-Gaussian data we expect them to perform similarly.” This conclusion has not been verified.

LIMITATIONS:
This paper extends the algorithm DP-PCA of Liu et al. [2022a] and is capable of estimating the top k eigenvectors of any k≤d. However, no new solutions were proposed to overcome the two limitations (i) and (ii).
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 1
DATE: 2025-06-30

comments: SUMMARY:
The paper studies the problem of k-PCA in the differential privacy settings. Prior work Liu et al.[2022a] address the problem only when k=1. Recently Jambulapati et al. [2024] provides a k-to-1-PCA reduction. This paper extended the deflation analysis of Jambulapati et al 2024 to the stochastic setting. The proposed algorithm repeatedly applies the DP-PCA subroutine in Liu et al. 2022a to recover the top eigenvector and filter out the component in the recovered space. A complementing lowerbound showed that the algorithm achieves near matching statistical error upto a factor of k.

STRENGTHS AND WEAKNESSES:
Strengths: The paper addresses an important problem in the DP data analysis. The algorithm is simple to implement and achieves a near optimal error upto a factor of k.
Weakness: It is not clear there is significant technical novelty or contribution, at least not that I am aware of upon reading the paper. It seems to be a straightforward combination of Liu et al. [20222a] and Jambulapati et al. [2024]. The guarantee of the algorithm is off by a k factor, which might be improved to sqrt(k) if data is reused.

QUESTIONS:
1. How is ModifiedDP-PCA different from DP-PCA in Liu et al. [2022a]? If we simply project each matrix with PAP^T and apply DP-PCA, do we have the same result and guarantee?
2. Following the first question. Line 210 on page 7 says “Note that we cannot plug in the DP-PCA algorithm...since it only guarantees relative error on E[P] \Sigma E[P]”. What does this mean? Isn’t P an input to the algorithm which is deterministic?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of DP $k$-PCA. The goal is to privately approximate the sub-space corresponding to the $k$ largest eigenvalues of a matrix that is the expected value of a distribution over random matrices. The problem has been addressed in the past in the work of Liu et al. (2022) for the case where $k = 1$, i.e., the case where we want to perform top eigenvector estimation.\nThe present paper gives the first algorithm for private $k$-PCA under approx-DP for which it suffices for the number of samples to scale roughly linearly with $d$ (the dimension of the data) to obtain low statistical error. A core element of the approach involves the use of the deflation framework of Jambulapati et al. (2024). That work demonstrates how, given black-box access to an oracle for top eigenvector estimation, it becomes possible to estimate the top-$k$ subspace by using the oracle iteratively. The present work combines a stochastic version of that reduction with variants of the subroutines given in Liu et al. (2022), which leads to their algorithm for $k$-PCA. This necessitates using the notion of a stochastic e-PCA oracle, and showing that the well-known Oja's algorithm satisfies the requirements of the oracle's definition. Upper bounds are given when the input is constructed from Gaussian random vectors, as well as when it comes from the spiked covariance model. Additionally, a lower bound is given for the spiked covariance model, which exhibits a gap with the upper bound in the dependence on $k$. Finally, an empirical evaluation of the method is conducted.\n\nArun Jambulapati, Syamantak Kumar, Jerry Li, Shourya Pandey, Ankit Pensia, and Kevin Tian. Black-box k-to-1-pca reductions: Theory and applications. arXiv:2403.03905, 2024.\n\nXiyang Liu, Weihao Kong, Prateek Jain, and Sewoong Oh. Dp-pca: Statistically optimal and differentially private pca. In Neural Information Processing Systems (NeurIPS), 2022.\n\nSTRENGTHS AND WEAKNESSES:\nI think that this is a good paper, but there's a number of weaknesses too in my opinion. I am going to discuss both positive and negative aspects in detail below.\n\nStrengths:\n1) This is the first work that gives a computationally efficient private algorithm for top-$k$ PCA with roughly linear sample complexity and low error. I feel this alone gives the work significant merit.\n2) The use of the deflation method from the work of Jambulapati et al. (2024) in the context of privacy was something that I found nice at a conceptual level.\n\nWeaknesses:\n1) One issue has to do with the gap in the $k$-dependence between the upper and the lower bounds. This is a weakness, but I don't think it significantly affects the value of the paper, since this is the first algorithm for DP $k$-PCA, and I think it's not damning if it is not statistically optimal.\n2) By reading the main body, it was not entirely clear to me what the technical challenges were for modifying the subroutines of Liu et al. (2022), and to show that the requirements of a stochastic e-PCA oracle are satisfied. I have not read Liu et al. (2022) and Jambulapati et al. (2024), so I was hoping for a more detailed discussion, which would make the evaluation of the merits of the present work easier.\n\nOverall, I think this is a decent paper, but I want to wait for the discussion period before expressing strong opinions about whether it should be accepted or not. I am voting borderline accept for the time being.\n\nQUESTIONS:\nCan the authors provide a more detailed discussion and comparison with the results of Liu et al. (2022) and Jambulapati et al. (2024)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-23"}