Privately Learning Subspaces

1 Introduction . Differentially private algorithms generally have a poor dependence on the dimensionality of their input . That is , their error or sample complexity grows polynomially with the dimension . For example , for the simple task of estimating the mean of a distribution supported on [ 0 , 1 ] d , we have per-coordinate error Θ ( √ d/n ) to attain differential privacy , where n is the number of samples . In contrast , the non-private error is Θ ( √ log ( d ) /n ) . This cost of dimensionality is inherent [ BUV14 , SU17 , DSS+15 ] . Any method with lower error is susceptible to tracing attacks ( a.k.a . membership inference attacks ) . However , these lower bounds only apply when the data distribution is “ high-entropy. ” This leaves open the posssibility that we can circumvent the curse of dimensionality when the data has an underlying low-dimensional structure . Data often does possess an underlying low-dimensional structure . For example , the gradients that arise in deep learning tend to be close to a low-dimensional subspace [ ACG+16 , LXT+17 , GARD18 , LFLY18 , LGZ+20 , ZWB20 , FT20 ] . Low dimensionality can arise from meaningful relationships that are at least locally linear , such as income versus tax paid . It can also arise because we are looking at a function of data with relatively few attributes . A long line of work [ BLR08 , HT10 , HR10 , Ull15 , BBNS19 , BCM+20 , ZWB20 , KRRT20 , etc . ] has shown how to exploit structure in the data to attain better privacy and accuracy . However , these approaches assume that this structure is known a priori or that it can be learned from non-private sources . This raises the question : Can we learn low-dimensional structure from the data subject to differential privacy ? We consider the simple setting where the data lies in Rd but is in , or very close to a linear subspace , of dimension k. We focus on the setting where k d and we develop algorithms whose sample 0The full version of this article is available online at https : //arxiv.org/abs/2106.00001 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . complexity does not depend on the ambient dimension d ; a polynomial dependence on the true dimension k is unavoidable . Our algorithms identify the subspace in question or , if the data is perturbed slightly , an approximation to it . Identifying the subspace structure is interesting in its own right , but it also can be used as a pre-processing step for further analysis – by projecting to the low-dimensional subspace , we ensure subsequent data analysis steps do not need to deal with high-dimensional data . 1.1 Our Contributions : Privately Learning Subspaces – Exact Case . We first consider the exact case , where the data X1 , · · · , Xn ∈ Rd are assumed to lie in a kdimensional subspace ( rather than merely being near to it ) – i.e. , rank ( A ) = k , where A =∑n i XiX T i ∈ Rd×d . In this case , we can also recover the subspace exactly . However , we must also make some non-degeneracy assumptions . We want to avoid a pathological input dataset such as the following . Suppose X1 , · · · , Xk are linearly independent , but Xk = Xk+1 = Xk+2 = · · · = Xn . While we can easily reveal the repeated data point , we can not reveal anything about the other points due to the privacy constraint . A natural non-degeneracy assumption would be to assume that the data points are in “ general position ” – that is , that there are no non-trivial linear dependencies among the data points . This means that every set of k data points spans the subspace or , equivalently , no subspace of dimension k − 1 contains more than k − 1 data points . This is a very natural assumption – if the data consists of n samples from a continuous distribution on the subspace , then this holds with probability 1 . We relax this assumption slightly and assume that no subspace of dimension k− 1 contains more than ` data points . We also assume that all points are non-zero . Note that we define subspaces to pass through the origin ; our results can easily be extended to affine subspaces . Theorem 1.1 ( Main Result – Exact Case ) . For all n , d , k , ` ∈ N and ε , δ > 0 satisfying n ≥ O ( ` + log ( 1/δ ) ε ) , there exists a randomized algorithm M : Rd×n → Skd satisfying the following . Here Skd denotes the set of all k-dimensional subspaces of Rd . • M is ( ε , δ ) -differentially private with respect to changing one column of its input . • Let X = ( X1 , · · · , Xn ) ∈ Rd×n . Suppose there exists a k-dimensional subspace S∗ ∈ Skd that contains all but ` of the points – i.e. , | { i ∈ [ n ] : Xi ∈ S∗ } | ≥ n− ` . Further suppose that any ( k − 1 ) -dimensional subspace contains at most ` points – i.e. , for all S ∈ Sk−1d , we have | { i ∈ [ n ] : Xi ∈ S } | ≤ ` . Then P [ M ( X ) = S∗ ] = 1 . The parameter ` in Theorem 1.1 can be thought of as a robustness parameter . Ideally the data points are in general position , in which case ` = k − 1 . If a few points are corrupted , then we increase ` accordingly ; our algorithm can tolerate the corruption of a small constant fraction of the data points . Theorem 1.1 is optimal in the sense that n ≥ Ω ( ` + log ( 1/δ ) ε ) samples are required . 1.2 Our Contributions : Privately Learning Subspaces – Approximate Case . Next we turn to the substantially more challenging approximate case , where the data X1 , · · · , Xn ∈ Rd are assumed to be close to a k-dimensional subspace , but are not assumed to be contained within that subspace . Our algorithm for the exact case is robust to changing a few points , but very brittle if we change all the points by a little bit . Tiny perturbations of the data points ( due to numerical errors or measurement imprecision ) could push the point outside the subspace , which would cause the algorithm to fail . Thus it is important to for us to cover the approximate case and our algorithm for the approximate is entirely different from our algorithm for the exact case . The approximate case requires us to precisely quantify how close the input data and our output are to the subspace and we also need to make quantitative non-degeneracy assumptions . It is easiest to formulate this via a distributional assumption . We will assume that the data comes from a Gaussian distribution where the covariance matrix has a certain eigenvalue gap . This is a strong assumption and we emphasize that this is only for ease of presentation ; our algorithm works under weaker assumptions . Furthermore , we stress that the differential privacy guarantee is worst-case and does not depend on any distributional assumptions . We assume that the data is drawn from a multivariate Gaussian N ( 0 , Σ ) . Let λ1 ( Σ ) ≥ λ2 ( Σ ) ≥ · · · ≥ λd ( Σ ) be the eigenvalues of Σ ∈ Rd×d . We assume that there are k large eigenvalues λ1 ( Σ ) , · · · , λk ( Σ ) – these represent the “ signal ” we want – and d − k small eigenvalues λk+1 ( Σ ) , · · · , λd ( Σ ) – these are the “ noise ” . Our goal is to recover the subspace spanned by the eigenvectors corresponding to the k largest eigenvalues λ1 ( Σ ) , · · · , λk ( Σ ) . Our assumption is that there is a large multiplicative gap between the large and small eigenvalues . Namely , we assume λk+1 ( Σ ) λk ( Σ ) ≤ 1poly ( d ) . Theorem 1.2 ( Main Result – Approximate Case ) . For all n , d , k ∈ N and α , γ , ε , δ > 0 satisfying n≥Θ ( k log ( 1/δ ) ε + ln ( 1/δ ) ln ( ln ( 1/δ ) /ε ) ε ) and γ2≤Θ ( εα2n d2k log ( 1/δ ) ·min { 1 k , 1 log ( k log ( 1/δ ) /ε ) } ) , there exists an algorithm M : Rd×n → Skd satisfying the following . Here Skd is the set of all k-dimensional subspaces of Rd represented as projection matricies – i.e. , Skd = { Π ∈ Rd×d : Π2 = Π = ΠT , rank ( Π ) = k } . • M is ( ε , δ ) -differentially private with respect to changing one column of its input . • Let X1 , · · · , Xn be independent samples from N ( 0 , Σ ) . Let λ1 ( Σ ) ≥ λ2 ( Σ ) ≥ · · · ≥ λd ( Σ ) be the eigenvalues of Σ ∈ Rd×d . Suppose λk+1 ( Σ ) ≤ γ2 · λk ( Σ ) . Let Π ∈ Skd be the projection matrix onto the subspace spanned by the eigenvectors corresponding to the k largest eigenvalues of Σ . Then P [ ‖M ( X ) −Π‖ ≤ α ] ≥ 0.7 . The sample complexity of our algorithm n = O ( k log ( 1/δ ) /ε ) is independent of the ambient dimension d ; this is ideal . However , there is a polynomial dependence on d in γ , which controls the multiplicative eigenvalue gap . This multiplicative eigenvalue gap is a strong assumption , but it is also a necessary assumption if we want the sample complexity n to be independent of the dimension d. In fact , it is necessary even without the differential privacy constraint [ CZ16 ] . That is , if we did not assume an eigenvalue gap that depends polynomially on the ambient dimension d , then it would be impossible to estimate the subspace with sample complexity n that is independent of the ambient dimension d even in the non-private setting . Our algorithm is based on the subsample and aggregate framework [ NRS07 ] and a differentially private histogram algorithm . These methods are generally quite robust and thus our algorithm is , too . For example , our algorithm can tolerate o ( n/k ) input points being corrupted arbitrarily . We also believe that our algorithm ’ s utility guarantee is robust to relaxing the Gaussianity assumption . All that we require in the analysis is that the empirical covariance matrix of a few samples from the distribution is sufficiently close to its expectation Σ with high probability . 1.3 Related Work . To the best of our knowledge , the problem of privately learning subspaces , as we formulate it , has not been studied before . However , a closely-related line of work is on Private Principal Component Analysis ( PCA ) and low-rank approximations . We briefly discuss this extensive line of work below , but first we note that , in our setting , all of these techniques have a sample complexity n that grows polynomially with the ambient dimension d. Thus , they do not evade privacy ’ s curse of dimensionality . However , we make a stronger assumption than these prior works – namely , we assume a large multiplicative eigenvalue gap . ( Many of the prior works consider an additive eigenvalue gap , which is a weaker assumption . ) There has been a lot of interest in Private PCA , matrix completion , and low-rank approximation . One motivation for this is the infamous Netflix prize , which can be interpreted as a matrix completion problem . The competition was cancelled after researchers showed that the public training data revealed the private movie viewing histories of many of Netflix ’ s customers [ NS06 ] . Thus privacy is a real concern for matrix analysis tasks . Many variants of these problems have been considered : Some provide approximations to the data matrix X = ( X1 , · · · , Xn ) ∈ Rd×n ; others approximate the covariance matrix A = ∑n i XiX T i ∈ Rd×d ( as we do ) . There are also different forms of approximation – we can either produce a subspace or an approximation to the entire matrix , and the approximation can be measured by different norms ( we consider the operator norm between projection matrices ) . Importantly , we define differential privacy to allow one data point Xi to be changed arbitrarily , whereas most of the prior work assumes a bound on the norm of the change or even assumes that only one coordinate of one vector can be changed . In the discussion below we focus on the techniques that have been considered for these problems , rather than the specific results and settings . [ DTTZ14 ] consider the simple algorithm which adds independent Gaussian noise to each of entries of the covariance matrix A , and then perform analysis on the noisy matrix . ( In fact , this algorithm predates the development of differential privacy [ BDMN05 ] and was also analyzed under differential privacy by McSherry and Mironov [ MM09 ] and Chaudhuri , Sarwate , and Sinha [ CSS12 ] . ) This simple algorithm is versatile and several bounds are provided for the accuracy of the noisy PCA . The downside of this is that a polynomial dependence on the ambient dimension d is inherent – indeed , they prove a sample complexity lower bound of n = Ω̃ ( √ d ) for any algorithm that identifies a useful approximation to the top eigenvector of A . This lower bound does not contradict our results because the relevant inputs do not satisfy our near low-rank assumption . [ HR12 ] and [ ABU18 ] apply techniques from dimensionality reduction to privately compute a lowrank approximation to the input matrix X . [ HR13 ] and [ HP13 ] use the power iteration method with noise injected at each step to compute low-rank approximations to the input matrix X . In all of these , the underlying privacy mechanism is still noise addition and the results still require the sample complexity to grow polynomially with the ambient dimension to obtain interesting guarantees . ( However , the results can be dimension-independent if we define differential privacy so that only one entry – as opposed to one column – of the matrix X can be changed by 1 . This is a significantly weaker privacy guarantee . ) [ BBDS12 ] and [ She19 ] also use tools from dimensionality reduction ; they approximate the covariance matrix A . However , they show that the dimensionality reduction step itself provides a privacy guarantee ( whereas the aforementioned results did not exploit this and relied on noise added at a later stage ) . [ She19 ] analyzes two additional techniques – the addition of Wishart noise ( i.e. , Y Y T where the columns of Y are independent multivariate Gaussians ) and sampling from an inverse Wishart distribution ( which has a Bayesian interpretation ) . [ CSS12 ] , [ KT13 ] , [ WSC+16 ] , and [ ADK+18 ] apply variants of the exponential mechanism [ MT07 ] to privately select a low-rank approximation to the covariance matrix A . This method is nontrivial to implement and analyse , but it ultimately requires the sample complexity to grow polynomially in the ambient dimension . [ GGB18 ] exploit smooth sensitivity [ NRS07 ] to release a low-rank approximation to the matrix A . This allows adding less noise than worst case sensitivity , under an eigenvalue gap assumption . However , the sample complexity n is polynomial in the dimension d. Limitations of Prior Work Given the great variety of techniques and analyses that have been applied to differentially private matrix analysis problems , what is missing ? We see that almost all of these techniques are ultimately based on some form of noise addition or the exponential mechanism . With the singular exception of the techniques of Sheffet [ She19 ] , all of these prior techniques satisfy pure1 or concentrated differential privacy [ BS16 ] . This is enough to conclude that these techniques can not yield the dimension-independent guarantees that we seek . No amount of postprocessing or careful analysis can avoid this limitation . This is because pure and concentrated differential privacy have strong group privacy properties , which means “ packing ” lower bounds [ HT10 ] apply . We briefly sketch why concentrated differential privacy is incompatible with dimension-independent guarantees . Let the input be X1 = X2 = · · · = Xn = ξ/ √ d for a uniformly random ξ ∈ { −1 , +1 } d. That is , the input is one random point repeated n times . If M satisfies O ( 1 ) -concentrated differential privacy , then it satisfies the mutual information bound I ( M ( X ) ; X ) ≤ O ( n2 ) [ BS16 ] . But , if M provides a meaningful approximation to X or A = XXT , then we must be able to recover an approximation to ξ from its output , whence I ( M ( X ) ; X ) ≥ Ω ( d ) , as the entropy of X is d bits . This gives a lower bound of n ≥ Ω ( √ d ) , even though X and A have rank k = 1 . The above example shows that , even under the strongest assumptions ( i.e. , the data lies exactly in a rank-1 subspace ) , any good approximation to the subspace , to the data matrix X , or to the covariance matrix A = XXT must require the sample complexity n to grow polynomially in the ambient 1Pure differential privacy ( a.k.a . pointwise differential privacy ) is ( ε , δ ) -differential privacy with δ = 0. dimension d if we restrict to techniques that satisfy concentrated differential privacy . Almost all of the prior work in this general area is subject to this restriction . To avoid a sample complexity n that grows polynomially with the ambient dimension d , we need fundamentally new techniques .

Summary: The paper studies the problem of learning a low-dimensional subspace from high-dimensional data in a differentially private manner.  The emphasis is on getting sample complexity that only depends on the dimension of the subspace but not the dimension of the original space. Such a guarantee is not achieved by previous techniques such as private PCA.  Formally, the paper studies two settings: the exact setting and the approximate setting.  In the exact setting, they assume that they are given $n$ points and almost all of them are contained in some $k$-dimensional subspace.  It is also necessary to make a non-degeneracy assumption that no $k-1$-dimensional subspace contains too many of the points.  In the approximate setting, they assume that all of the points are drawn from a Gaussian that is close to $k$-dimensional (i.e. its covariance matrix has $k$ large eigenvalues and $d-k$ much smaller eigenvalues).  In the first case, they obtain a sample complexity of $O(k + \log (1/\delta)/\epsilon)$  and in the second setting they obtain a sample complexity of  $O(k \log(1/\delta)/\epsilon)$ for achieving $(\epsilon, \delta)$ differential privacy.  Crucially, these are linear in $k$ and independent of $d$ and also their privacy guarantee is in the worst case, where any of the sample points may be altered arbitrarily.  It is also worth noting that for the approximate setting, they do not really use the Gaussian distribution assumption very strongly and only need that the sample covariances approximate the true covariances.