Differential Privacy Over Riemannian Manifolds

1 Introduction . Over the last decade we have seen a tremendous push for the development and application of methods in data privacy . This surge has been fueled by the production of large sophisticated datasets alongside increasingly complex data gathering technologies . One theme that has emerged with the proliferation of highly structured and dynamic data is the importance of exploiting underlying structures in the data or models to maximize utility while controlling disclosure risks . In this paper we consider the problem of achieving pure Differential Privacy , DP , when the statistical summary to be released takes values on a complete Riemannian manifold . Riemannian manifolds are used extensively in the analysis of data or parameters that are inherently nonlinear , meaning , either addition or scalar multiplication may cause the summary to leave the manifold , or such operations are not even well defined . Classic examples of such objects include spatio-temporal processes , covariance matrices , projections , rotations , compositional data , densities , and shapes . Traditional privacy approaches for handling such objects typically consist of utilizing an ambient or embedding space that is linear so that standard DP tools can be employed . For example , Karwa and Slavković [ 2016 ] considered the problem of releasing private degree sequences of a graph which required them to project back onto a particular convex hull as a post-processing step . Such an approach is a natural starting point and reasonable so long as the space doesn ’ t exhibit too much curvature . However , there are several interrelated motivations for working with the manifolds directly . First , if one employs an ambient space ( known as taking an extrinsic approach ) , then calculations such as the sensitivity may depend on the dimension of the ambient space , which will in turn impact 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the utility of the private statistical summary . For example , minimax rates in DP typically scale polynomially in the dimension [ e.g . Hardt and Talwar , 2010 , Bun et al. , 2018 , Kamath et al. , 2019 ] . Second , the Whitney embedding theorem states that , in the worst case , to embed a manifold in Euclidean space requires a space that is twice the dimension of the manifold . Third , if the manifold exhibits substantial curvature , then even small distances in the ambient space may result in very large distances on the manifold . Lastly , the choice of the ambient space may be arbitrary and one would ideally prefer if this choice did not play a role in the resulting statistical analysis . Related Literature : To the best of our knowledge , general manifolds have not been considered before in the DP literature . The closest works come from the literature on private covariance matrix estimation and principal components [ Blum et al. , 2005 , Awan et al. , 2019 , Amin et al. , 2019 , Kamath et al. , 2019 , Biswas et al. , 2020 , Wang and Xu , 2020 ] . While not always explicitly described in some works [ Wang et al. , 2013 , Wei et al. , 2016 ] , these objects lie in nonlinear manifolds , namely , the space of symmetric positive definite matrices ( SPDM ) and the space of projections matrices respectively , called the Stiefel manifold . For example , in Chaudhuri et al . [ 2013 ] they consider the problem of generating a synthetic PCA projection by using the matrix Bingham distribution , a distribution over the Stiefel manifold [ Khatri and Mardia , 1977 , Hoff , 2009 ] . In contrast , producing private covariance matrix estimates usually involves adding noise in a way that preserves symmetry , but does not use any deeper underlying manifold structure . A related problem comes from the literature on private manifold learning [ Choromanska et al. , 2016 , Vepakomma et al. , 2021 ] , though this is entirely distinct from the present work , which assumes the underlying manifold is known , usually because of some physical constraints on the data or statistical summaries . Contributions : In this paper we utilize tools from Differential Geometry that allow us to extend the Laplace mechanism for -Differential Privacy to general Riemannian manifolds . Under this framework , we consider the problem of privately estimating the Fréchet mean of data lying on a d-dimensional manifold . We are able to bound the global sensitivity of the mean and provide bounds on the magnitude of the privacy noise , as measured using the distance on the manifold , that match the optimal rates derived in Euclidean spaces . However , we demonstrate the influence of curvature of the space in understanding the sensitivity of the mean , and how the situation becomes especially challenging on positively curved spaces . We conclude by providing two specific numerical examples that elucidate this phenomenon : the first considers data coming from the space of positive definite matrices equipped with a geometry that results in negative curvature , while the second example considers data lying on the sphere , which has constant positive curvature . 2 Notation and Background . In this section we provide the basic notation , terminology , and mathematical concepts needed from differential geometry . For a more detailed treatment of differential geometry and Shape Analysis there are many excellent classic texts , e.g. , Gallot et al . [ 1990 ] , Lang [ 2002 ] , Dryden [ 2014 ] , Srivastava and Klassen [ 2016 ] , Lee [ 2018 ] , while an overview of DP can be found in Dwork and Roth [ 2014 ] . Throughout the paper we let M denote a d-dimensional complete Riemannian manifold . For m ∈ M , denote the corresponding tangent space as TmM . We assume M is equipped with a Riemannian metric { 〈· , ·〉m : m ∈ M } , which is a collection of inner products over the tangent spaces { TmM : m ∈M } that vary smoothly in m. Two quantities that will be used extensively in this work are that of the distance and volume induced from the Riemannian metric . Consider two points m1 , m2 ∈M and a smooth path γ : [ 0 , 1 ] →M such that γ ( 0 ) = m1 and γ ( 1 ) = m2 . The derivative γ̇ ( t ) represents the velocity of γ as it passes through the point γ ( t ) and can thus be identified as an element of the tangent space Tγ ( t ) M. We define the length of the curve as L ( γ ) : = ∫ 1 0 〈γ̇ ( t ) , γ̇ ( t ) 〉1/2γ ( t ) dt . The distance between m1 and m2 is the infimum over all possible paths connecting the two points ρ ( m1 , m2 ) : = inf γ : γ ( 0 ) =m1 γ ( 1 ) =m2 L ( γ ) . If this distance is achieved by a particular path , γ , then we say that γ is a geodesic . Geodesics generalize the concept of a straight line to nonlinear spaces , and are thus a natural tool to consider when generalizing DP perturbation mechanisms . The distance ρ ( · , · ) defines a valid distance metric overM ; we say thatM is complete if it is complete as a metric space . By the Hopf-Rinow theorem [ Lee , 2018 , Theorem 6.19 ] , this is equivalent to saying that to every pair of points there exists a minimizing geodesic , though if the points are far enough apart it need not be unique . In the next section we will use the exponential map , expm : TmM → M , which is a means of moving between the manifold and tangent spaces . If there exists a unique geodesic , γ , between between points γ ( 0 ) = m1 and γ ( 1 ) = m2 , then the exponential map is defined as the mapping expm1 ( γ̇ ( 0 ) ) = m2 . In other words , expm maps initial velocities to points onM through the use of geodesic curves . If the manifold is complete , then the exponential map is locally diffeomorphic , meaning that for anym ∈M there exists an open neighborhood Ur ( m ) ofM that is diffeomorphic to an open ball Br ( 0 ) centred at the origin of TmM . This ensures that the inverse exp−1m : Ur → Br ( 0 ) known as the inverse-exponential ( also known as logarithm ) map exists locally . The injectivity radius of m is the supremum of r 7→ Ur over all such radii r. The injectivity radius ofM , denoted by injM , is defined to be the infimum of the injectivity radii of all points m ∈M . The Riemannian metric can be used to define a notion of volume , called the Riemannian volume measure denoted as µ , which acts analogously to Lesbesgue measure in Euclidean space . To define the measure , it helps to employ a chart , ( U , φ ) , although the final definition will not depend on which chart we choose . Since φ : U → φ ( U ) ⊂ Rd is a homeomorphism , at each m ∈ U the inverse φ−1 induces a basis , ∂x1 , . . . , ∂xd on TmM and a corresponding dual basis dx1 , . . . , dxd on TmM∗ , the dual space of TmM . Then the Riemannian volume form is defined as √ |g|dx1 ∧ · · · ∧ dxd , where gij = 〈∂xi , ∂xj〉m and | · | is the absolute value of the determinant , which can be shown to be invariant to the choice of chart . This induces a volume over the set U and upon employing a partition of unity , one can define µ over the entire manifoldM , equipped with the Borel σ-algebra . 3 Manifold Perturbations and Differential Privacy . Denote the dataset as D = { x1 , . . . , xn } with the data coming from points xi collected from an arbitrary set X . We aim to release a statistical summary f ( D ) which takes values onM . Defining differential privacy over a Riemannian manifold presents no major challenge since it is a well defined concept over any measurable space [ Wasserman and Zhou , 2010 , Awan et al. , 2019 ] , which includes Riemannian manifolds equipped with the Borel σ-algebra . Denote the ( random ) sanitized version of f ( D ) as f̃ ( D ) . We can then define what it means for f̃ ( D ) to satisfy -DP . Definition 1 . A family of randomized summaries , { f̃ ( D ) ∈ M : D ∈ Xn } , is said to be - differentially private with > 0 , if for any adjacent database D′ , denoted as D ∼ D′ , differing in only one record we have P ( f̃ ( D ) ∈ A ) ≤ e P ( f̃ ( D′ ) ∈ A ) , for any measurable set A . In a similar fashion , we can extend the notion of sensitivity to Riemannian manifolds using the distance function . However , this definition is by no means canonical and intimately connected to the type of noise one intends to use [ Mirshani et al. , 2019 ] . Definition 2 . A summary f is said to have a global sensitivity of ∆ < ∞ , with respect to ρ ( · , · ) , if for any two adjacent databases D and D′ we have ρ ( f ( D ) , f ( D′ ) ) ≤ ∆ . With a bounded sensitivity , it still isn ’ t obvious how to produce a differentially private summary . In particular , we no longer have linear perturbations or classic noise mechanisms such as Laplace or Gaussian . However , the Riemannian structure allows us to use the volume measure as a base measure , similar to the Lebesgue measure on Euclidean spaces . We can then define a new distribution that can be viewed as a generalization of the Laplace distribution over manifolds [ e.g . Hajri et al. , 2016 , for SPDM ] . It is worth noting that this distribution is not equivalent to the multivariate Laplace in Euclidean settings , instead it is an instantiation of the K-norm distribution [ Hardt and Talwar , 2010 ] . Definition 3 . A probability measure P overM is called a Laplace distribution with footpoint η ∈M and rate σ > 0 if for any measurable set A we have P ( A ) = ∫ A C−1η , σe −ρ ( η , m ) /σ dµ ( m ) , where 0 < Cη , σ < ∞ is the normalizing constant and µ is the Riemannian volume measure overM . Unlike with Euclidean distance , the normalizing constant might only be finite for certain values of σ . The foot-point represents the center of the distribution but , in general , need not be equal to the mean of the distribution ( which also might not exist ) . For data privacy , one often has to restrict the data or parameter space anyway , in which case the Laplace distribution can be restricted so that the normalizing constant , mean , etc are all finite and well defined . The advantage in using such a mechanism is that sensitivity can be readily transferred into differential privacy . Theorem 1 . Let f : Xn →M be a summary with global sensitivity ∆ . Then the Laplace mechanism with footpoint f ( D ) and rate σ = 2∆/ satisfies -differential privacy . If the normalizing constant , Cη , σ does not depend on the footpoint , η , then one can take σ = ∆/ . Proof . ( Sketch ) The proof follows from a direct verification via the triangle inequality . In principle , it is possible to sample from the Laplace distribution using Markov Chain Monte Carlo ( MCMC ) . One can start the chain at η and then make small proposed steps by randomly selecting a direction and radius on the tangent space . The resulting tangent vector can be pushed to an element ofM via the exponential map . Alternatively , if enough structure is known , as we illustrate in Section 5 , one may be able to sample from the distribution directly or if the space is bounded then one can use rejection sampling . An interesting alternative to our approach that is still inherently intrinsic is to instead generate the sanitised summary on a particular tangent space and then map it to the manifold using the exponential map . On the surface , this seems like a reasonable idea , however there are some subtle technicalities that would have to be overcome . In particular , one has to choose which tangent space to work with . Ideally one would work with the tangent space at the summary of interest , but that isn ’ t private . If another plane is used , then there is the chance for more serious distortions from the noise . Likely these issues could be overcome , but would require additional work .

Summary: This paper studies the problem of releases a differentially private statistic when the data and the statistic lie on a Riemannian manifold. They give a formulation of the Laplacian mechanism in this setting. They then consider the specific example of.a Frechet mean to display the benefits of using the manifold structure rather than adding noise in the ambient (higher dimensional) space, then projecting back to the manifold.   The authors show through specific examples and experiments the benefit of using the manifold structure rather than adding noise in the ambient space. As a first paper on the topic this seems like a nice proof of concept. I would have liked it to go deeper in discussing the complexity of actually sampling from the Laplace mechanism on a manifold. In particular, assumption 1 felt like it simplified the problem considerably.