Privately Learning Subspaces

1 Introduction . Differentially private algorithms generally have a poor dependence on the dimensionality of their input . That is , their error or sample complexity grows polynomially with the dimension . For example , for the simple task of estimating the mean of a distribution supported on [ 0 , 1 ] d , we have per-coordinate error Θ ( √ d/n ) to attain differential privacy , where n is the number of samples . In contrast , the non-private error is Θ ( √ log ( d ) /n ) . This cost of dimensionality is inherent [ BUV14 , SU17 , DSS+15 ] . Any method with lower error is susceptible to tracing attacks ( a.k.a . membership inference attacks ) . However , these lower bounds only apply when the data distribution is “ high-entropy. ” This leaves open the posssibility that we can circumvent the curse of dimensionality when the data has an underlying low-dimensional structure . Data often does possess an underlying low-dimensional structure . For example , the gradients that arise in deep learning tend to be close to a low-dimensional subspace [ ACG+16 , LXT+17 , GARD18 , LFLY18 , LGZ+20 , ZWB20 , FT20 ] . Low dimensionality can arise from meaningful relationships that are at least locally linear , such as income versus tax paid . It can also arise because we are looking at a function of data with relatively few attributes . A long line of work [ BLR08 , HT10 , HR10 , Ull15 , BBNS19 , BCM+20 , ZWB20 , KRRT20 , etc . ] has shown how to exploit structure in the data to attain better privacy and accuracy . However , these approaches assume that this structure is known a priori or that it can be learned from non-private sources . This raises the question : Can we learn low-dimensional structure from the data subject to differential privacy ? We consider the simple setting where the data lies in Rd but is in , or very close to a linear subspace , of dimension k. We focus on the setting where k d and we develop algorithms whose sample 0The full version of this article is available online at https : //arxiv.org/abs/2106.00001 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . complexity does not depend on the ambient dimension d ; a polynomial dependence on the true dimension k is unavoidable . Our algorithms identify the subspace in question or , if the data is perturbed slightly , an approximation to it . Identifying the subspace structure is interesting in its own right , but it also can be used as a pre-processing step for further analysis – by projecting to the low-dimensional subspace , we ensure subsequent data analysis steps do not need to deal with high-dimensional data . 1.1 Our Contributions : Privately Learning Subspaces – Exact Case . We first consider the exact case , where the data X1 , · · · , Xn ∈ Rd are assumed to lie in a kdimensional subspace ( rather than merely being near to it ) – i.e. , rank ( A ) = k , where A =∑n i XiX T i ∈ Rd×d . In this case , we can also recover the subspace exactly . However , we must also make some non-degeneracy assumptions . We want to avoid a pathological input dataset such as the following . Suppose X1 , · · · , Xk are linearly independent , but Xk = Xk+1 = Xk+2 = · · · = Xn . While we can easily reveal the repeated data point , we can not reveal anything about the other points due to the privacy constraint . A natural non-degeneracy assumption would be to assume that the data points are in “ general position ” – that is , that there are no non-trivial linear dependencies among the data points . This means that every set of k data points spans the subspace or , equivalently , no subspace of dimension k − 1 contains more than k − 1 data points . This is a very natural assumption – if the data consists of n samples from a continuous distribution on the subspace , then this holds with probability 1 . We relax this assumption slightly and assume that no subspace of dimension k− 1 contains more than ` data points . We also assume that all points are non-zero . Note that we define subspaces to pass through the origin ; our results can easily be extended to affine subspaces . Theorem 1.1 ( Main Result – Exact Case ) . For all n , d , k , ` ∈ N and ε , δ > 0 satisfying n ≥ O ( ` + log ( 1/δ ) ε ) , there exists a randomized algorithm M : Rd×n → Skd satisfying the following . Here Skd denotes the set of all k-dimensional subspaces of Rd . • M is ( ε , δ ) -differentially private with respect to changing one column of its input . • Let X = ( X1 , · · · , Xn ) ∈ Rd×n . Suppose there exists a k-dimensional subspace S∗ ∈ Skd that contains all but ` of the points – i.e. , | { i ∈ [ n ] : Xi ∈ S∗ } | ≥ n− ` . Further suppose that any ( k − 1 ) -dimensional subspace contains at most ` points – i.e. , for all S ∈ Sk−1d , we have | { i ∈ [ n ] : Xi ∈ S } | ≤ ` . Then P [ M ( X ) = S∗ ] = 1 . The parameter ` in Theorem 1.1 can be thought of as a robustness parameter . Ideally the data points are in general position , in which case ` = k − 1 . If a few points are corrupted , then we increase ` accordingly ; our algorithm can tolerate the corruption of a small constant fraction of the data points . Theorem 1.1 is optimal in the sense that n ≥ Ω ( ` + log ( 1/δ ) ε ) samples are required . 1.2 Our Contributions : Privately Learning Subspaces – Approximate Case . Next we turn to the substantially more challenging approximate case , where the data X1 , · · · , Xn ∈ Rd are assumed to be close to a k-dimensional subspace , but are not assumed to be contained within that subspace . Our algorithm for the exact case is robust to changing a few points , but very brittle if we change all the points by a little bit . Tiny perturbations of the data points ( due to numerical errors or measurement imprecision ) could push the point outside the subspace , which would cause the algorithm to fail . Thus it is important to for us to cover the approximate case and our algorithm for the approximate is entirely different from our algorithm for the exact case . The approximate case requires us to precisely quantify how close the input data and our output are to the subspace and we also need to make quantitative non-degeneracy assumptions . It is easiest to formulate this via a distributional assumption . We will assume that the data comes from a Gaussian distribution where the covariance matrix has a certain eigenvalue gap . This is a strong assumption and we emphasize that this is only for ease of presentation ; our algorithm works under weaker assumptions . Furthermore , we stress that the differential privacy guarantee is worst-case and does not depend on any distributional assumptions . We assume that the data is drawn from a multivariate Gaussian N ( 0 , Σ ) . Let λ1 ( Σ ) ≥ λ2 ( Σ ) ≥ · · · ≥ λd ( Σ ) be the eigenvalues of Σ ∈ Rd×d . We assume that there are k large eigenvalues λ1 ( Σ ) , · · · , λk ( Σ ) – these represent the “ signal ” we want – and d − k small eigenvalues λk+1 ( Σ ) , · · · , λd ( Σ ) – these are the “ noise ” . Our goal is to recover the subspace spanned by the eigenvectors corresponding to the k largest eigenvalues λ1 ( Σ ) , · · · , λk ( Σ ) . Our assumption is that there is a large multiplicative gap between the large and small eigenvalues . Namely , we assume λk+1 ( Σ ) λk ( Σ ) ≤ 1poly ( d ) . Theorem 1.2 ( Main Result – Approximate Case ) . For all n , d , k ∈ N and α , γ , ε , δ > 0 satisfying n≥Θ ( k log ( 1/δ ) ε + ln ( 1/δ ) ln ( ln ( 1/δ ) /ε ) ε ) and γ2≤Θ ( εα2n d2k log ( 1/δ ) ·min { 1 k , 1 log ( k log ( 1/δ ) /ε ) } ) , there exists an algorithm M : Rd×n → Skd satisfying the following . Here Skd is the set of all k-dimensional subspaces of Rd represented as projection matricies – i.e. , Skd = { Π ∈ Rd×d : Π2 = Π = ΠT , rank ( Π ) = k } . • M is ( ε , δ ) -differentially private with respect to changing one column of its input . • Let X1 , · · · , Xn be independent samples from N ( 0 , Σ ) . Let λ1 ( Σ ) ≥ λ2 ( Σ ) ≥ · · · ≥ λd ( Σ ) be the eigenvalues of Σ ∈ Rd×d . Suppose λk+1 ( Σ ) ≤ γ2 · λk ( Σ ) . Let Π ∈ Skd be the projection matrix onto the subspace spanned by the eigenvectors corresponding to the k largest eigenvalues of Σ . Then P [ ‖M ( X ) −Π‖ ≤ α ] ≥ 0.7 . The sample complexity of our algorithm n = O ( k log ( 1/δ ) /ε ) is independent of the ambient dimension d ; this is ideal . However , there is a polynomial dependence on d in γ , which controls the multiplicative eigenvalue gap . This multiplicative eigenvalue gap is a strong assumption , but it is also a necessary assumption if we want the sample complexity n to be independent of the dimension d. In fact , it is necessary even without the differential privacy constraint [ CZ16 ] . That is , if we did not assume an eigenvalue gap that depends polynomially on the ambient dimension d , then it would be impossible to estimate the subspace with sample complexity n that is independent of the ambient dimension d even in the non-private setting . Our algorithm is based on the subsample and aggregate framework [ NRS07 ] and a differentially private histogram algorithm . These methods are generally quite robust and thus our algorithm is , too . For example , our algorithm can tolerate o ( n/k ) input points being corrupted arbitrarily . We also believe that our algorithm ’ s utility guarantee is robust to relaxing the Gaussianity assumption . All that we require in the analysis is that the empirical covariance matrix of a few samples from the distribution is sufficiently close to its expectation Σ with high probability . 1.3 Related Work . To the best of our knowledge , the problem of privately learning subspaces , as we formulate it , has not been studied before . However , a closely-related line of work is on Private Principal Component Analysis ( PCA ) and low-rank approximations . We briefly discuss this extensive line of work below , but first we note that , in our setting , all of these techniques have a sample complexity n that grows polynomially with the ambient dimension d. Thus , they do not evade privacy ’ s curse of dimensionality . However , we make a stronger assumption than these prior works – namely , we assume a large multiplicative eigenvalue gap . ( Many of the prior works consider an additive eigenvalue gap , which is a weaker assumption . ) There has been a lot of interest in Private PCA , matrix completion , and low-rank approximation . One motivation for this is the infamous Netflix prize , which can be interpreted as a matrix completion problem . The competition was cancelled after researchers showed that the public training data revealed the private movie viewing histories of many of Netflix ’ s customers [ NS06 ] . Thus privacy is a real concern for matrix analysis tasks . Many variants of these problems have been considered : Some provide approximations to the data matrix X = ( X1 , · · · , Xn ) ∈ Rd×n ; others approximate the covariance matrix A = ∑n i XiX T i ∈ Rd×d ( as we do ) . There are also different forms of approximation – we can either produce a subspace or an approximation to the entire matrix , and the approximation can be measured by different norms ( we consider the operator norm between projection matrices ) . Importantly , we define differential privacy to allow one data point Xi to be changed arbitrarily , whereas most of the prior work assumes a bound on the norm of the change or even assumes that only one coordinate of one vector can be changed . In the discussion below we focus on the techniques that have been considered for these problems , rather than the specific results and settings . [ DTTZ14 ] consider the simple algorithm which adds independent Gaussian noise to each of entries of the covariance matrix A , and then perform analysis on the noisy matrix . ( In fact , this algorithm predates the development of differential privacy [ BDMN05 ] and was also analyzed under differential privacy by McSherry and Mironov [ MM09 ] and Chaudhuri , Sarwate , and Sinha [ CSS12 ] . ) This simple algorithm is versatile and several bounds are provided for the accuracy of the noisy PCA . The downside of this is that a polynomial dependence on the ambient dimension d is inherent – indeed , they prove a sample complexity lower bound of n = Ω̃ ( √ d ) for any algorithm that identifies a useful approximation to the top eigenvector of A . This lower bound does not contradict our results because the relevant inputs do not satisfy our near low-rank assumption . [ HR12 ] and [ ABU18 ] apply techniques from dimensionality reduction to privately compute a lowrank approximation to the input matrix X . [ HR13 ] and [ HP13 ] use the power iteration method with noise injected at each step to compute low-rank approximations to the input matrix X . In all of these , the underlying privacy mechanism is still noise addition and the results still require the sample complexity to grow polynomially with the ambient dimension to obtain interesting guarantees . ( However , the results can be dimension-independent if we define differential privacy so that only one entry – as opposed to one column – of the matrix X can be changed by 1 . This is a significantly weaker privacy guarantee . ) [ BBDS12 ] and [ She19 ] also use tools from dimensionality reduction ; they approximate the covariance matrix A . However , they show that the dimensionality reduction step itself provides a privacy guarantee ( whereas the aforementioned results did not exploit this and relied on noise added at a later stage ) . [ She19 ] analyzes two additional techniques – the addition of Wishart noise ( i.e. , Y Y T where the columns of Y are independent multivariate Gaussians ) and sampling from an inverse Wishart distribution ( which has a Bayesian interpretation ) . [ CSS12 ] , [ KT13 ] , [ WSC+16 ] , and [ ADK+18 ] apply variants of the exponential mechanism [ MT07 ] to privately select a low-rank approximation to the covariance matrix A . This method is nontrivial to implement and analyse , but it ultimately requires the sample complexity to grow polynomially in the ambient dimension . [ GGB18 ] exploit smooth sensitivity [ NRS07 ] to release a low-rank approximation to the matrix A . This allows adding less noise than worst case sensitivity , under an eigenvalue gap assumption . However , the sample complexity n is polynomial in the dimension d. Limitations of Prior Work Given the great variety of techniques and analyses that have been applied to differentially private matrix analysis problems , what is missing ? We see that almost all of these techniques are ultimately based on some form of noise addition or the exponential mechanism . With the singular exception of the techniques of Sheffet [ She19 ] , all of these prior techniques satisfy pure1 or concentrated differential privacy [ BS16 ] . This is enough to conclude that these techniques can not yield the dimension-independent guarantees that we seek . No amount of postprocessing or careful analysis can avoid this limitation . This is because pure and concentrated differential privacy have strong group privacy properties , which means “ packing ” lower bounds [ HT10 ] apply . We briefly sketch why concentrated differential privacy is incompatible with dimension-independent guarantees . Let the input be X1 = X2 = · · · = Xn = ξ/ √ d for a uniformly random ξ ∈ { −1 , +1 } d. That is , the input is one random point repeated n times . If M satisfies O ( 1 ) -concentrated differential privacy , then it satisfies the mutual information bound I ( M ( X ) ; X ) ≤ O ( n2 ) [ BS16 ] . But , if M provides a meaningful approximation to X or A = XXT , then we must be able to recover an approximation to ξ from its output , whence I ( M ( X ) ; X ) ≥ Ω ( d ) , as the entropy of X is d bits . This gives a lower bound of n ≥ Ω ( √ d ) , even though X and A have rank k = 1 . The above example shows that , even under the strongest assumptions ( i.e. , the data lies exactly in a rank-1 subspace ) , any good approximation to the subspace , to the data matrix X , or to the covariance matrix A = XXT must require the sample complexity n to grow polynomially in the ambient 1Pure differential privacy ( a.k.a . pointwise differential privacy ) is ( ε , δ ) -differential privacy with δ = 0. dimension d if we restrict to techniques that satisfy concentrated differential privacy . Almost all of the prior work in this general area is subject to this restriction . To avoid a sample complexity n that grows polynomially with the ambient dimension d , we need fundamentally new techniques .

Summary: This paper studies the problem of learning low-dimensional structure from the data subject to differential privacy. The paper considers the setting where the data lies in or very close to a linear subspace of dimension (k<<d). For the exact case where all the data lies in a linear subspace, the proposed algorithm DPESE, which applies the GAP-MAX, can recover the subspace exactly when there are enough samples. For the approximate case, they assume that the data comes from a Gaussian distribution where the covariance matrix has a certain eigenvalue gap. This paper proposes the DPASE algorithm that is based on the subsample-and-aggregate algorithm. In both cases, the sample complexity the dimension-independent.