DP-PCA: Statistically Optimal and Differentially Private PCA

Abstract: We study the canonical statistical task of  computing the principal component from    i.i.d.~data under differential privacy. Although extensively studied in literature,  existing solutions fall short on two key aspects: ($i$) even for Gaussian data, existing private algorithms   require the number of samples  $n$ to scale super-linearly with $d$, i.e., $n=\Omega(d^{3/2})$, to obtain non-trivial results while non-private PCA  requires only $n=O(d)$, and ($ii$) existing techniques suffer from a large error even when the variance in each data point is small. We propose DP-PCA method that uses a single-pass minibatch gradient descent style algorithm to overcome the above  limitations. For sub-Gaussian data, we provide nearly optimal statistical error rates even for $n=O(d \log d)$. 

Reviews and author responses:

 The current revise solves my questions. I will change my rating to 'accept'.

 - Re: Major weakness:
   - We thank the reviewer for their meticulous checking of the proof. We acknowledge that there were a few typos in the proof which we have now fixed. A corrected proof of ***the exactly the same statement of Theorem 5.3*** is provided in the revised pdf (with the changes emphasized in blue in Section C.1 in the new supplementary material). In particular, 
      - In Theorem C.2, the very last term in the equation is now fixed (and this has no effect on the final statement we want to prove). The reason is that the error rate scales as $\tau$ (which is not changed after this fixing of the typo), and the last term $|V|/e^{10n\phi \varepsilon}$, which is now revised, only needs to be larger than some universal constant. This is ensured by the condition that $d\geq C’ n \phi \varepsilon \simeq n\varepsilon\alpha \sqrt{(\lambda_1-\lambda_2)^2/(2\lambda_1 \lambda_2)}$, which is ensured by our choice of $\alpha$. 
      - In Lemma C.3, the final bound does not depend on $\alpha$ and we put a complete proof of the lemma in Section C.1.1. Intuitively, we are constructing the set by placing points on the intersection of the (d-1)-dim unit-sphere (which insures the unit norm constraint) and a “certain” ball with radius scaling as $\alpha$ (which allows us to control the ultimate loss). At the same time, since our goal of $(1/\sqrt{2})\alpha \leq || v-v’|| \leq \sqrt{2}\alpha$ scales linearly in $\alpha$, the construction is scale-invariant; for smaller $\alpha$ we will be packing the same number of points, but just closer to each other. We added a complete proof of this lemma, for completeness. 
      - In Eq.(20), the constants had typos and we fixed them. 
      - Now that we know Lemma C.3 does not depend on $\alpha$, we can apply this as we have written, even though  $\alpha$ depends on the problem dimension. 
- Re: Minor weaknesses:
   - All typos are fixed in the revised version. 
   - There are two ways to extend our framework to general rank-$r$ PCA. We will add a discussion in the final version that states that other methods provide error bounds for the general case. 
      - Hotelling’s deflation method: we can iteratively find the PCA component one by one, by alternating our DP-PCA and deflation. For example, in one step of the iteration, we only update the current iterate vector in the directions orthogonal to all the previously found PCA components. Repeating this $r$ steps gives the estimates of the top $r$ principal components. 
      - Oja’s algorithm: we can keep track of a $r$-dimensional subspace in the Oja’s update rule for PCA, and perform QR decomposition to keep the iterates on the Grassmannian manifold. It might be possible to extend the analysis of “Streaming k-PCA: Efficient guarantees for Oja’s algorithm, beyond rank-one updates” by De Huang, Jonathan Niles-Weed, Rachel Ward to analyze the private version, but we believe this is beyond the scope of this paper.
- Re: Question: 
   - That is a typo and Theorem 5.1 is now fixed and includes the $d\log(1/\delta)/\varepsilon$. 


 - Re: choice of learning rate: In the submitted manuscript, we discuss how to search for the appropriate learning rate. The main idea is that the same guarantees can be achieved with an adaptive search for the right learning rate done in differentially private manner. For convenience, we copy the text from the submission, which reads: 
> Currently, DP-PCA requires choices of the learning rates, $\eta_t$, that depend on possibly unknown quantities. Since we can privately evaluate the quality of our solution, one can instead run multiple instances of DP-PCA with varying  $\eta_t = c_1/(c_2 + t)$ and find the best choice of $c_1>0$ and $c_2>0$. Let $w_T(c_1,c_2)$ denote the resulting solution for one instance of $\{\eta_t = c_1/(c_2 + t)\}_{t=1}^T$. We first set a target error $\zeta$. For each round $i=1,\ldots$, we will run algorithm for $(c_1, c_2) = [2^{i-1},2^{-i+1}]\times [2^{-i+1}, 2^{-i+2}\ldots, 2^{i-1}]$ and $(c_1, c_2) = [2^{-i+1}, 2^{-i+2}\ldots, 2^{i-1}]\times [2^{i-1},2^{-i+1}]$, and compute each $\sin(w_T(c_1, c_2), v_1)$ privately, each with privacy budget $\varepsilon_i = \frac{\varepsilon}{2^{i+1}(2i-1)}, \delta_i = \frac{\delta}{2^{i+1}(2i-1)}$. We terminate the algorithm once there there is a $w_T(c_1, c_2)$ satisfies $\sin(w_T(c_1, c_2), v_1)\le \zeta$. It is clear that this search meta-algorithm terminates in logarithmic rounds, and the total sample complexity only blows up by a poly-log factor.” (line 252-261)

  The choice of the learning rate is inherited from Theorem 2.2 on the analysis of non-private PCA. It is an adaptive choice, as it decreases with the iteration $t$. This is critical in achieving the near-optimal sample complexities. This sheds light on the type of learning rates that efficiently performs private streaming PCA. We note that we have essentially assumed an oracle which sets the learning rate in this paper. An important question for future research in numerical investigations of various differentially private PCA algorithms is how to set the learning rate in a robust, private, and data-driven manner. We will clarify our choice of the learning rate in the final version. 



- Re: experiments
The main contribution of this paper is the introduction of the novel algorithm and its theoretical analysis. We believe the numerical investigation of various private PCA approaches is outside the scope of this paper and we leave it for future research. 


 - Response to question 1: It is a correct observation that since we primarily care about the error rate’s dependence on various parameters (and not the constant), we do not attempt to optimize the constants (like those that can be gained by a better splitting of the data).  

- Response to question 2: PCA for the heavy-tailed case is an exciting problem. We know that we need to “add more noise” since the fundamental trade-off is different for heavy-tailed samples, which are known to be achievable with non-efficient algorithms [“Differential privacy and robust statistics in high dimensions”, Xiyang Liu, Weihao Kong, and Sewoong Oh.] but no efficient algorithm is known. The main difference will be in carefully analyzing the clipping threshold, which will govern how much larger the noise is going to be for the heavy-tailed case. We believe our framework will still be applicable with heavy-tailed mean estimators, and we leave it for future research. 

- Response to question 3: For the case where there is no spectral gap (i.e. $\lambda_1 \simeq \lambda_2 $), the fundamental statistical trade-offs are known in [“Differential privacy and robust statistics in high dimensions”, Xiyang Liu, Weihao Kong, and Sewoong Oh.]. Combining the techniques from the heavy-tailed lower bound from this reference with the lower bound technique in our manuscript, we believe one can prove a heavy-tailed lower bound for the case of interest in our manuscript where there is a spectral gap (i.e. $\lambda_1-\lambda_2 >0$). An educated conjecture for this lower bound is 
$\sin(v,\hat v) \geq C \min \{ \kappa \Big(\sqrt{\frac{d}{n}} +(\frac{d}{n\varepsilon})^{(p-2)/p} \Big) , 1\}$, for $p$-th moment bounded distributions. 


 We thank the reviewer for the references and added them to the related work section in Section A of the supplementary materials. Comparisons to Imtiaz and Sarwate are also provided. The newly added parts are indicated by blue fonts. 

 This paper studies principle component analysis (PCA) under differential privacy (DP) constraints. The authors improve the sample complexity from prior SOTA to a near-optimal rate by proposing a new single-pass algorithm. This algorithm is novel, and it is based on a minibatch SGD algorithm where each step contains a private top eigenvalue estimation and a private mean estimation. Although the authors have not shown any simulation or experiment for their algorithm, they provide both upper bounds and lower bounds under various assumptions with solid theoretical analysis. Major strengths:

1. The authors point out current issues for the existing results performing PCA with privacy constraint and use a totally different approach to tackle this problem. They identify the two challenges of the intuitive Private Oja’s Algorithm and solve them by two subalgorithms which perfectly match the need.
2.  The authors provide both the upper bound and lower bound which vastly improves the understanding of the PCA problem under DP constraint. The analysis of the DP-PCA under Gaussian distributions shows that their algorithm is near-optimal.

Minor strengths:

1. This paper is well-written and greatly organized to help readers get familiar with the problem they are solving and the reason behind their solution. Section 6 is very helpful for understanding what is happening during the proof.
2. The assumptions and the corresponding reasons are clearly provided.

Major weakness:

1. I have checked most of the theorems with the proof in the appendix, and found all of them are solid and accurate, except for the lower bound, Theorem 5.3, which may invalidate the near-optimality: 
    1. In Section C.1, Line 635, the very last term in the equation is incorrect where from [3] Corollary 4, $\frac{\log(|V|)}{n\phi\varepsilon}$ should be $\frac{|V|}{\exp(10n\phi\varepsilon)}$; 
    2. In Line 640 Lemma C.3, the final part $c_1 d$ should contains $\alpha$. 
    3. In Line 644, the usage of Lemma F.4 seems to have wrong constants. 
    4. In Line 647, the choice of $\alpha$ depends on $d$ which is not permissible if we want to use Lemma C.3.  

Minor weakness: 

1. This paper only considers PCA as computing the first principal component. The paper [23], which is the tightest result they have compared with in Line 115, can be used to compute the top $k$ principal components under DP constraint. Therefore, the description that the rate of DP-PCA is better than existing results is correct but may not be very fair. It would be better if the authors provide a discussion on how to use DP-PCA to find top $k$ principal components.
2. Theorem 2.2 typo: $\beta$ should be $\xi$
3. Line 178 typo, $x_i$ should be $s_i$ 1. In Theorem 6.1, the rate on $B$ contains $d\log(1/\delta)$, but in Theorem 5.1, the rate on $n$, which I think should be almost the same as $B$ since $B=O(n/log^2 n)$ in Line 196, does not contain $d\log(1/\delta)$ as a whole term. Is there any typo? The authors has discussed the limitation of their analysis and algorithm, e.g., Line 121-123, Line 235-244.

 Principal Component Analysis (PCA) is a fundamental statistical tool with multiple applications and differential privacy (DP) is a widely accepted mathematical notion of privacy. This paper presents a algorithm called DP-PCA and shows that it achieves an error bounded for sub-Gaussian-like data.  The authors make the Oja’s algorithm private and show that it achieves near-optimal error if the data is strictly from a Gaussian distribution. Then they overcame the two remaining challenges: (i) restricted range of \epsilon=O(\sqrt{log(n/\delta)/n}); (ii) excessive noise for privacy. To overcome each challenge, it critically relies on two techniques: minibatch SGD and private mean estimation. The authors proposed DP-SGD based on the two ideas above. They use minibatch SGD of minibatch size B=O(n/log^{2}n) to allow for larger range of \epsilon=O(1) and use Private Mean Estimation to add an appropriate level of noise chosen adaptively according to Private Eigenvalue Estimation.
This paper considers using private estimation within an optimization in DP-PCA setting while the idea has already been proposed in abstract settings. 
 See below for the weakness DP-PCA requires choices of the learning rates. I think authors should better explain their choice of learning rate.

There are no experiments to validate the effectiveness of the proposed methods in the current paper.


 This paper studies the PAC task under differential privacy guarantee and propose private algorithms which achieves the optimal sample complexity.  Strengths:
1. This paper is well written and I enjoy the reading. The motivations for each proposed techniques and improvements, though simple, are clear.
2. This paper shows DP PCA algorithm of an asymptotically optimal sample complexity. Concrete examples for the lower bound are presented, especially for the effect on concentration assumptions. 

Weakness:
There is some unclarity in the algorithm and proof. Please see below.  1. In Algorithm 3, why not apply subsampling in the input A_i but instead only implement on split data? Is is because it only concerns the asymptotic  properties and there does not matter if the mini-batch is n/log n for log n iterations? 


2. Can you further drop the Gaussian tail assumption (A4) but instead apply heavy-tailed mean estimation as Algorithm 5 to implement Algorithm 3? It seems that changes in A4 only produce a worse dependence on the variance of the data but some bound can still be derived though not necessarily optimal. 

3. Can the techniques to construct the lower bound be generalized to the case when only the second or fourth moment of  tail is assumed? I am curious for heavy-tailed data, can the author claim anything interesting on the utility bound?  Please see above. 

 The authors propose a Differentially Private PCA (DP-PCA) based on a minibatch GD method to overcome the limitations of existing approaches. Nearly optimal statistical error rates are provided. A lower bound shows that sub-Gaussian style assumption is necessary
to obtain the optimal error rate.
 Strengths:
1) The paper is well organized.
2)  The proposed Private Oja’s Algorithm and DP-PCA are well explained and reproducible.

Weaknesses:
1) There are missing key references:
H. Imtiaz, A. Sarwate, "Differentially Private Distributed Principal Component Analysis", IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2018, pages 2206-2210, 2018.
H. Imtiaz, A. Sarwate, “Distributed Differentially-Private Algorithms for Matrix and Tensor Factorization” , IEEE Journal of Selected Topics in Signal Processing, December 2018.
S. Wang, J. Chang, "Differentially Private Principal Component Analysis Over Horizontally Partitioned Data", IEEE Conference on Dependable and Secure Computing, DSC 2018, 2018.
A. Gang, B. Xiang, W. Bajwa, “Distributed principal subspace analysis for partitioned big data: Algorithms, analysis, and implementation", IEEE Transactions on Signal and Information Processing over Networks, Volume 7, pages 699–715, 2021.
 Can the authors locate their work in comparison of the works of . Imtiaz and. Sarwate published in 2018? The authors have addressed adequately  the limitations of the proposed method.

Meta-review: The paper studies PCA under the constraint of differential privacy, and presents an improved algorithm which is based on a minibatch SGD where each step contains a private top eigenvalue estimation and a private mean estimation. The reviewers agree that the new algorithm is interesting and that the new results are important.

Decision: Accept