Title: On Differentially Private Subspace Estimation in a Distribution-Free Setting
Venue: NeurIPS 2024
Accepted: yes
Keywords: Differential Privacy

Private Subspace Estimation

Private PCA

Fingerprinting Codes
Abstract: Private data analysis faces a significant challenge known as the curse of dimensionality, leading to increased costs. However, many datasets possess an inherent low-dimensional structure. For instance, during optimization via gradient descent, the gradients frequently reside near a low-dimensional subspace. If the low-dimensional structure could be privately identified using a small amount of points, we could avoid paying for the high ambient dimension.

On the negative side, Dwork, Talwar, Thakurta, and Zhang (STOC 2014) proved that privately estimating subspaces, in general, requires an amount of points that has a polynomial dependency on the dimension. However, their bounds do not rule out the possibility to reduce the number of points for "easy" instances. Yet, providing a measure that captures how much a given dataset is "easy" for this task turns out to be challenging, and was not properly addressed in prior works.

Inspired by the work of Singhal and Steinke (NeurIPS 2021), we provide the first measures that quantify "easiness" as a function of multiplicative singular-value gaps in the input dataset, and support them with new upper and lower bounds. In particular, our results determine the first types of gaps that are sufficient and necessary for estimating a subspace with an amount of points that is independent of the dimension. Furthermore, we realize our upper bounds using a practical algorithm and demonstrate its advantage in high-dimensional regimes compared to prior approaches.

On Differentially Private Subspace Estimation in a Distribution-Free Setting

Abstract

Private data analysis faces a significant challenge known as the curse of dimensionality, leading to increased costs. However, many datasets possess an inherent low-dimensional structure. For instance, during optimization via gradient descent, the gradients frequently reside near a low-dimensional subspace. If the low-dimensional structure could be privately identified using a small amount of points, we could avoid paying for the high ambient dimension.

Onthe negative side, Dwork et al. [2014] proved that privately estimating subspaces, in general, requires an amount of points that has a polynomial dependency on the dimension. However, their bounds do not rule out the possibility to reduce the number of points for 'easy' instances. Yet, providing a measure that captures how much a given dataset is 'easy' for this task turns out to be challenging, and was not properly addressed in prior works.

Inspired by the work of Singhal and Steinke [2021], we provide the first measures that quantify 'easiness' as a function of multiplicative singular-value gaps in the input dataset, and support them with new upper and lower bounds. In particular, our results determine the first types of gaps that are sufficient and necessary for estimating a subspace with an amount of points that is independent of the dimension. Furthermore, we realize our upper bounds using a practical algorithm and demonstrate its advantage in high-dimensional regimes compared to prior approaches.

1 Introduction

Differentially private (DP) Dwork et al. [2006b] algorithms typically exhibit a significant dependence on the dimensionality of their input, as their error or sample complexity tends to grow polynomially with the dimension. This cost of dimensionality is inherent in many problems, as Bun et al. [2014], Steinke and Ullman [2017], Dwork et al. [2015] showed that any method that achieves lower error rates is vulnerable to tracing attacks (also known as, membership inference attacks). However, these lower bounds consider algorithms that guarantee accuracy for worst-case inputs and do not rule out the possibility of achieving higher accuracy for 'easy' instances.

Example: DP averaging. As a simple prototypical example, consider the task of DP averaging. In this task, the input dataset consists of d -dimensional points x 1 , . . . , x n ∈ R d , and the goal is to estimate their average 1 n ∑ n i =1 x i using a DP algorithm while minimizing the ℓ 2 additive error. One natural way to capture input 'easiness' for this task is via the maximal ℓ 2 distance between any two points (i.e., points that are closer to each other are considered 'easier'). Indeed, Tsfadia et al. [2022], Peter et al. [2024] showed that if the points are γ -close to each other, and we aim for an accuracy of λγ (i.e., an accuracy that is proportional to the 'easiness' parameter γ ), then it is sufficient and

necessary to use n = ˜ Θ( √ d/λ ) points. Equivalently, if we aim for an accuracy of α , then by applying these results with λ = α/γ , we obtain that the answer is n = ˜ Θ( γ √ d/α ) . This, in particular, implies that when γ ≤ α/ √ d (i.e., the points are very close to each other), then ˜ O (1) points are sufficient, but for γ = α/d 1 / 2 -Ω(1) , a polynomial dependency on d is necessary in general.

DP subspace estimation. In this work, we consider the more complex problem of DP subspace estimation: Given a dataset X = ( x 1 , . . . , x n ) ∈ ( R d ) n of unit norm points and a parameter k , estimate the topk subspace of Span { x 1 , . . . , x n } . The main goal of this work is to answer the following meta question:

Question 1.1. How should we quantify how 'easy' a given dataset is for DP subspace estimation?

Since the dimension d is very large in many settings, we aim at providing tight measures that smoothly eliminate the dependency on d as a function of input 'easiness'. In particular, we want to be able to identify when we can avoid paying on the ambient dimension d , and when a polynomial dependency on d is unavoidable.

1.1 Motivation: DP-SGD

To motivate the problem, consider the task of privately training large neural networks. The most commonly used tool to perform such a private training is the differentially-private stochastic gradient descent (DP-SGD) Abadi et al. [2016b], Bassily et al. [2014], Song et al. [2013] - a private variant of SGD that perturbs each gradient update with random noise vector drawn from an isotropic Gaussian distribution. However, this approach does not differentiate between 'easy' gradients and 'hard' ones, which results with high error rates when the ambient dimension - the number of parameters in the model - is large. However, empirical evidence and theoretical analysis indicate that while training some deep learning models, the gradients tend to live near a low-dimensional subspace Abadi et al. [2016a], Li et al. [2018b], Gur-Ari et al. [2018], Li et al. [2018a], Demeniconi and Chawla [2020], Zhou et al. [2021], Feng and Tu [2020], Li et al. [2022], Golatkar et al. [2022], Kairouz et al. [2020]. In particular, Gur-Ari et al. [2018] showed that in some cases, the low dimension is the number of classes in the dataset, and the gradients tend to be close and well-spread inside this subspace. If we could exploit such a low-dimensional structure into an (inexpensive) private and useful projection matrix, we could reduce the error of DP-SGD by making it dependent solely on the low dimension.

We start by defining the setting of DP subspace estimation more formally.

1.2 Subspace Estimation

We consider the setting of Dwork et al. [2014]. That is, the input dataset consists of n points of unit norm x 1 , . . . , x n ∈ S d := { v ∈ R d : ∥ v ∥ 2 = 1 } and a parameter k , and the goal is to output a k -dimensional projection matrix Π such that Π · X T is 'close' to X T as possible, where X denotes the n × d matrix whose rows are x 1 , . . . , x n . We measure the accuracy of our estimation using the 'usefulness' definition of Dwork et al. [2014]:

Definition 1.2 ( α -useful) . We say that a rankk projection matrix Π is α -useful for a matrix X ∈ ( S d ) n if for any k -rank projection matrix Π ′ :

where ∥·∥ F denotes the Frobenius norm. 1

Observe that any projection matrix is 1 -useful for any X (because ∥ X ∥ 2 F = ∑ n i =1 ∥ x i ∥ 2 2 = n ). Therefore, we will be interested in smaller values of α (e.g., 0 . 001 ).

1.3 Prior Works

Without privacy restrictions, we can find a 0 -useful (i.e., optimal) solution using Singular-Value Decomposition (SVD). The SVD of X is X = U Σ V T , where U ∈ R n × n and V ∈ R d × d are unitary

matrices, and Σ is an n × d diagonal matrix which has values σ 1 ≥ . . . ≥ σ min { n,d } ≥ 0 along the diagonal. The topk rows subspace of X is given by the span of the first k columns of V , and it can be computed, e.g., by applying Principal Component Analysis (PCA) on the covariance matrix A = X T X (the eigenvectors of A are the columns of V ).

With differential privacy, however, the problem is much harder, and Dwork et al. [2014] showed a lower bound of n ≥ ˜ Ω( k √ d ) for computing a 0 . 001 -useful k -rank projection matrix under (1 , Ω(1 /n 2 )) -DP. This bound, however, only holds for algorithms that provide accuracy for worstcase instances and does not rule out the possibility of achieving high accuracy with smaller values of n for input points that are very close to being in a k -dimensional subspace (i.e., 'easy" instances).

Perhaps the easiest instances are those that exactly lie in a k -dimensional subspace, and are wellspread within it (i.e., there is no ( k -1) -dimensional subspace that contains many points). Indeed, Singhal and Steinke [2021] and Ashtiani and Liaw [2022] developed ( ε, δ ) -DP algorithms for such instances that precisely recover the subspace using only n = ˜ Θ ε,δ ( k ) points. However, while these algorithms are robust to changing a few points, they are very brittle if we change all the points by a little bit.

One approach to smoothly quantify how much a dataset is 'easy' is to consider the additive-gap σ 2 k -σ 2 k +1 . Indeed, Dwork et al. [2014], Gonem and Gilad-Bachrach [2018] present ( ε, δ ) -DP algorithms that output 0 . 001 -useful projection using n = ˜ Θ ε,δ ( k √ d σ 2 k -σ 2 k +1 ) points. Yet, the downside of such additive-gap based approaches is their inherent dependency on the dimension d . Even in the extreme case where the points exactly lie in a k -dimensional subspace and well-spread within it, the additive gap σ 2 k -σ 2 k +1 is at most n/k , which still results with a polynomial dependency on d .

The only existing approach to eliminate the dependency on d in some non-exact cases is the one of Singhal and Steinke [2021] (their 'approximate' case). Rather than quantifying easiness as a function of the input dataset, they consider a setting where the points are sampled i.i.d. from some distribution, and implicitly measure how 'easy' a distribution is according to some stability notion. In particular, they show that a d -dimensional Gaussian N ( ⃗ 0 , Σ) with a multiplicative-gap σ k +1 (Σ) σ k (Σ) ≤ ˜ Θ ε,δ,k ( 1 d 2 ) is 'stable' enough for estimating the topk subspace of Σ with sample complexity that is independent of d . While Singhal and Steinke [2021] do not provide an answer to Question 1.1, they inspired our work to consider multiplicative singular-value gaps in the input dataset as a measure for easiness.

1.4 Defining Subspace Estimators

Towards answering Question 1.1, we consider mechanisms M that are parameterized by k , λ , and β , and satisfy the following utility guarantee: Given a dataset X = ( x 1 , . . . , x n ) ∈ ( S d ) n and a value γ as inputs, such that X is ' γ -easy' for k -subspace estimation, then with probability at least β over a random execution of M ( X,γ ) , the output Π is an λγ -useful rankk projection matrix for X . 2 In Definition 1.3, the ' γ -easy" property is abstracted by a predicate f . We also allow an additional parameter γ max to relax the utility for non-easy instances (i.e., we would not require a utility guarantee for instances that are not ' γ max -easy'). Furthermore, we only focus on cases in which X has at least k significant directions, which is formalized by requiring that σ 2 k ( X ) ≥ 0 . 01 · n/k (we refer to Remark 2.3 for how our upper bounds, stated in Theorem 1.6, can handle smaller values of σ k using an additional parameter or different assumptions).

Definition 1.3 ( ( k, λ, β, γ max , f ) -Subspace Estimator) . Let n, k, d ∈ N s.t. k ≤ min { n, d } , let β ∈ (0 , 1] , and let f : ( S d ) n × [0 , 1] →{ 0 , 1 } be a predicate. We say that M : ( S d ) n × [0 , 1] → R d × d is an ( k, λ, β, γ max , f ) -subspace estimator, if for every X ∈ ( S d ) n and γ ≤ γ max with σ 2 k ( X ) ≥ 0 . 01 · n/k and f ( X,γ ) = 1 , it holds that

1.5 Quantifying Easiness - Our Approach

In this work, we develop two types of smooth measures (captured by the predicate f in Definition 1.3) for input 'easiness', which are translated to the following two types of subspace estimators:

Definition 1.4 ( ( k, λ, β, γ max ) -Weak Subspace Estimator) . M : ( S d ) n × [0 , 1] → R d × d is called an ( k, λ, β, γ max ) -Weak Subspace Estimator , if it is ( k, λ, β, γ max , f ) -Subspace Estimator for the predicate f ( X,γ ) that outputs 1 iff ∑ n i = k +1 σ 2 i ( X ) ≤ γ 2 σ 2 k ( X ) .

Definition 1.5 ( ( k, λ, β, γ max ) -Strong Subspace Estimator) . M : ( S d ) n × [0 , 1] → R d × d is called an ( k, λ, β, γ max ) -Strong Subspace Estimator , if it is ( k, λ, β, γ max , f ) -Subspace Estimator for the predicate f ( X,γ ) that outputs 1 iff σ k +1 ( X ) ≤ γσ k ( X ) .

In both cases, we define f based on multiplicative singular-value gaps in the input dataset, but the difference is what type of gap the value γ bounds: Strong estimators depend solely on the gap σ k +1 /σ k without taking into account smaller singular values. Weak estimators, on the other, depend on the gap √ ∑ min { n,d } i = k +1 σ 2 i /σ k . Note that a strong estimator is, in particular, a weak one (with the same parameters). Also note that both measures smoothly converge to the exact k -subspace case: When each gap tends to zero, the points tend to be closer to a k -dimensional subspace.

We provide new upper and lower bounds for both types of estimators.

1.5.1 Our Upper Bounds

Theorem 1.6 (Weak estimator) . There exists an ( k, λ, β = 0 . 9 , γ max = Θ(min { 1 λ , 1 } )) -weak subspace estimator M : ( S d ) n × [0 , 1] → R d × d with n = ˜ O ε,δ ( k + min { k 2 √ d, kd } λ ) such that M ( · , γ ) is ( ε, δ ) -DP for every γ ∈ [0 , 1] .

Theorem 1.7 (Strong estimator) . There exists an ( k, λ, β = 0 . 8 , γ max = ˜ Θ(min { 1 λ , λ 2 k 2 d } )) -strong subspace estimator M : ( S d ) n × [0 , 1] → R d × d with n = ˜ O ε,δ ( k + k 3 d λ 2 ) such that M ( · , γ ) is ( ε, δ ) -DP for every γ ∈ [0 , 1] .

Both of our estimators provide a useful projection by outputting a matrix that is close (in Frobenius norm) to the projection matrix of the topk rows subspace. Their running time is n m · T ( m,d,k ) + ˜ O ( dkn ) for some m = ˜ Θ( k ) , where T ( m,d,k ) denotes the running time required to compute (non-privately) a projection matrix to the topk rows subspace of an m × d matrix. We refer to Section 2 for a detailed overview.

For simplifying the presentation and the formal utility guarantees, we assume that our algorithms know the values of γ (the bound on the multiplicative-gap) and of k beforehand. Yet, we show that both assumptions are not inherent, and we refer to Remark 2.4 for additional details.

We also remark that in both theorems, it is possible to increase the confidence β to any constant smaller than 1 without changing the asymptotic cost.

1.5.2 Our Lower Bounds

Theorem 1.8 (Lower bound for weak estimators) . If M : ( S d ) n × [0 , 1] → R d × d is a ( k, λ, β = 0 . 1 , γ max = Θ( 1 λ )) -weak subspace estimator for 1 ≤ λ ≤ Θ( d k log k ) and M ( · , γ ) is ( 1 , 1 50 nk ) -DP for every γ ∈ [0 , 1] , then n ≥ ˜ Ω ( √ kd ) .

Theorem 1.9 (Lower bound for strong estimators) . If M : ( S d ) n × [0 , 1] → R d × d is a ( k, λ, β = 0 . 1 , γ max = Θ ( 1 λ ) ) -strong subspace estimator for 1 ≤ λ ≤ Θ( d log k ) and M ( · , γ ) is ( 1 , 1 50 nk ) -DP for every γ , then n ≥ ˜ Ω ( k √ d λ ) .

Our lower bounds are more technically involved, and use a novel combination of generating hardinstances using the tools from Peter et al. [2024] for proving smooth lower bounds, and extracting sensitive vectors from useful projection matrices using ideas from the lower bound of Dwork et al. [2014]. Both lower bounds are proven by generating hard-instances that are ' γ -easy' for γ = 1 1000 λ .

We refer to Section 3 for a detailed overview.

We remark that Peter et al. [2024] recently proved a similar lower bound for the special case of k = 1 (estimating the top-singular vector). However, their result strongly relies on the similarity between

averaging and estimating top-singular vector in their hard instances, which does not hold for the case k ≥ 2 . Table 1 summarizes our bounds for k ≤ √ d .

1.5.3 Implications

We offer two formulations which have the property we seek: If we aim for an error α , and the dataset is ' γ -easy' for a very small γ , we take λ = α/γ to reduce the number of necessary and sufficient points.

For strong estimators, the rate n = n ( λ ) in Theorem 1.7 does not match the corresponding lower bound Theorem 1.9, and Theorem 1.7 is limited to small values of γ . 3 Yet, for small values of k , the strong-estimator bounds do imply that in order to privately compute an 0 . 001 -useful rankk projection with number of points that is independent of d , it is sufficient and necessary to have a gap σ k +1 /σ k of at most ∝ 1 / √ d . Table 2 summarizes our bounds for the special case of outputing 0 . 001 -useful projection matrix, using our two different types of input 'easiness'.

1.6 Empirical Evaluation

We believe that private subspace estimation of easy instances could find practical applications. Therefore, we made an effort to realize our upper bounds using a practical algorithm. In Section 4 we empirically compared our method to the additive-gap based approach of Dwork et al. [2014] for the task of privately estimating the empirical mean of points that are close to a small dimensional subspace, demonstrating the advantage of our approach in high-dimensional regimes.

1.7 Paper Organization

In Sections 2 and 3 we present proof overviews for our results, and the empirical evaluation is provided in Section 4. A limitations section is given in Section 5. Additional related work appears at Appendix A. Notations, definitions and general statements are given in Appendix B. Our upper bounds are proven in Appendix C, and our lower bounds are proven in Appendix D.

2 Upper Bounds - Overview

Both of our estimators (Theorems 1.6 and 1.7) follow the same structure, but with different parameters. Similarly to Singhal and Steinke [2021], our algorithms follow the sample-and-aggregate approach of Nissim et al. [2007]. That is, given a dataset X = ( x 1 , . . . , x n ) ∈ ( S d ) n , we partition the rows into t subsets, compute (non-privately) a projection matrix to the topk rows subspace of each subset,

and then privately aggregate the projection matrices Π 1 , . . . , Π t . For doing that, we need to argue that Π 1 , . . . , Π t are expected to be close to each other. In the Gaussian setting of Singhal and Steinke [2021], this holds by concentration properties of Gaussian distributions. In our setting, however, it is unreasonable to expect that arbitrary partitions will lead to similar subspaces. For instance, consider the matrix X whose first n/k rows are e 1 = (1 , 0 , . . . , 0) , the next n/k rows are e 2 = (0 , 1 , . . . , 0) , and so forth until e k . Even though X is rankk and has σ 2 1 = . . . = σ 2 k = n/k , if we simply partition the rows according to their order, then most of those subsets will induce a rank1 matrix which clearly does not represent the original matrix X . Therefore, we must consider a more clever partition that will guarantee a good representation of the topk rows subspace of X in each subset.

There is an extensive line of works who aim for methods to choose a small subset of rows that provides a good low-approximation for the original matrix (e.g., see Mahoney [2011] for a survey). Yet, most of these methods are data-dependent, and therefore seem less applicable for privacy.

In this work we show that by simply using a uniformly random partition into subsets of size ˜ Θ( k ) , then w.h.p. each subset induces a projection matrix that is close to the projection matrix of the topk rows subspace of X :

Lemma 2.1. Let X = ( x 1 , . . . , x n ) ∈ ( S d ) n with singular values σ 1 ≥ . . . ≥ σ min { n,d } ≥ 0 , and let γ 1 = σ k +1 σ k and γ 2 = √ ∑ min { n,d } i = k +1 σ 2 i σ k . Let X ′ ∈ ( S d ) m be a uniformly random m -size subset of the rows of X (without replacement). Let Π and Π ′ be the projection matrices to the topk rows subspace of X and X ′ , respectively. Assuming that σ 2 k ≥ 0 . 01 n/k , then the following holds for m = ˜ Θ( k ) :

Namely, Item 1 bounds the expected spectral norm distance of the projection matrices using the first type gap σ k +1 /σ k (which is used in the analysis of our strong estimator), and Item 2 bounds the expected Frobenius norm distance using the second type gap √ ∑ min { n,d } i = k +1 σ 2 i /σ k (which is used in the analysis of our weak estimator). We prove Lemma 2.1 in Appendix C.1.2.

The next step is to aggregate the non-private projection matrices Π 1 , . . . , Π t into a private one ˜ Π . We consider two types of aggregations. The first one simply treats each matrix as a d 2 vector and privately estimate the average of Π 1 , . . . , Π t . The second type (which outperforms the first one in most cases) follows a similar high-level structure of Singhal and Steinke [2021]. That is, to sample i.i.d. reference points p 1 , . . . , p q ∼ N ( ⃗ 0 , I d × d ) for q = Θ( k ) , privately average the qd -dimensional points { (Π j p 1 , . . . , Π j p q ) } t j =1 for obtaining a private ˜ P ∈ R q × d (whose i th row estimates the projection of p i onto the topk rows subspace of X ), and then compute the projection matrix of the topk rows subspace of ˜ P . But unlike Singhal and Steinke [2021] who perform this step using a histogram-based averaging that has the same flavor of Karwa and Vadhan [2018], we perform this step using FriendlyCore Tsfadia et al. [2022] that simplifies the construction and makes it practical in high dimensional regimes. We remark that in both aggregation types, we need a DP averaging algorithm that is resilient to a constant fraction of outliers (say, 20% ) since both items in Lemma 2.1 only guarantee that the expected number of outliers is no more than 10% . Fortunately, FriendlyCore can be utilized for such regimes of outliers (see Appendix B.8.3 for more details).

A few remarks are in order.

Remark 2.2. The first aggregation type (which privately estimate the average of Π 1 , . . . , Π t directly) outperforms the second type only for our weak estimator in the regime k ≥ √ d (as it inherently posses larger dependency in the dimension).

Remark 2.3. We could avoid the requirement σ 2 k ≥ 0 . 01 n/k by adding an additional parameter η such that σ 2 k ≥ η · n/k , and using subsets of size m = ˜ Θ( k/η ) (which would increase n by the same factor of 1 /η ). For readability purposes, we chose to avoid this additional parameter. Because our algorithms provide useful projection by estimating the actual topk projection, then such a

requirement is unavoidable if we would like to provide utility guarantees only as a function of the singular values. 6 In fact, any assumption that would imply that two random subsets induce similar topk projection matrices would suffice for the utility analysis.

Remark 2.4. To eliminate the knownγ assumption, we can replace the FriendlyCore-averaging step Tsfadia et al. [2022] (that requires to know the diameter) with their unknown diameter variant that gets two very rough bounds ξ min and ξ max , and performs a private binary search for estimating a good diameter ξ ∈ [ ξ min , ξ max ] in a preprocessing phase. This step only replaces the dependency on d with d +loglog( ξ max /ξ min ) in the asymptotic sample complexity (section 5.1.2. in Tsfadia et al. [2022]) and is very practical. In fact, we use this method in our empirical evaluation in Section 4.

For handling unknown values of k , note that our algorithms provide useful utility guarantees (compared to the additive-gap based ones) only when σ 2 k +1 ≪ 1 . So in cases where σ 2 k ≥ 2 · log( k/β ) /ε ′ for some (fixed) privacy budget ε ′ < ε (say, ε ′ = ε/ 10 ), we can privately determine w.p. 1 -β the right value of k using a simple ε ′ -DP method. The main observation is that the ℓ 1 sensitivity of the vector ( σ 2 1 , . . . , σ 2 n ) is at most 2 (Amin et al. [2019]), yielding that we can privately compute ( σ ′ 2 1 , . . . , σ ′ 2 n ) = ( σ 2 1 +Lap(2 /ε ′ ) , . . . , σ 2 n +Lap(2 /ε ′ )) , and then perform analysis on ( σ ′ 2 1 , . . . , σ ′ 2 n ) to set k as the first index i where σ 2 i ≥ log( i/β ) /ε ′ and σ 2 i +1 < log( i/β ) /ε ′ .

3 Lower Bounds - Overview

Our lower bounds (Theorems 1.8 and 1.9) use the recent framework of Peter et al. [2024] for generating smooth lower bounds for DP algorithms using Fingerprinting Codes (FPC), but require technically involved analysis due to the complex structure of this problem for k ≥ 2 .

Roughly speaking, let D be a distribution over {-1 , 1 } n 0 × d 0 that induces an optimal FPC codebook with d 0 = ˜ O ( n 2 0 ) (e.g., Tardos [2008], Peter et al. [2024]). The connection between FPC and DP (first introduced by Bun et al. [2014]) is that any DP algorithm, given a random codebook X = ( x j i ) i ∈ [ n 0 ] ,j ∈ [ d 0 ] ∼ D as input, cannot output a vector q = ( q 1 , . . . , q d 0 ) ∈ {-1 , 1 } d 0 that 'agrees" with most of the 'marked" columns of X (Formally, for b ∈ {-1 , 1 } , a columns x j = ( x j 1 , . . . , x j n ) is called b -marked if x j 1 = . . . = x j n = b , and q agrees with it if q j = b ).

Now consider a DP mechanism M : X n → W that satisfies some non-trivial accuracy guarantee. Peter et al. [2024] reduces the task of lower bounding n to the following task: (1) Generate from an FPC codebook X ∈ {-1 , 1 } n 0 × d 0 hard instances Y ∈ X n for M , and (2) Extract from the output w ∼ M ( Y ) a vector q ∈ {-1 , 1 } d 0 that agrees with most of the marked columns of X ( n 0 and d 0 are some functions of n , X and the weak accuracy guarantee of M ). Peter et al. [2024] proved that if there exists such generating algorithm G and extracting algorithm F (which even share a random secret that M does not see) such that G is neighboring-preserving (i.e., maps neighboring databases to neighboring databases), then it must hold that n 0 ≥ ˜ Ω( √ d 0 ) (Otherwise, M cannot be DP).

Warm-up: DP averaging. We first sketch how Peter et al. [2024] applied their framework with n 0 = n and d 0 = Θ( d/λ 2 ) for proving a lower bound for the simpler problem of DP averaging. In this setting, we are given a mechanism that guarantees λγ -accuracy ( ℓ 2 additive error) for γ -easy instances (i.e., points that are γ -close to each other in ℓ 2 norm). The generator G , given an FPC codebook X ∈ {-1 , 1 } n 0 × d 0 , uses the padding-and-permuting technique: It pads ℓ ≈ 10 4 λ 2 d 0 1 -marked columns and ℓ ( -1) -marked columns, and then permutes all the d = d 0 +2 ℓ columns of the new codebook X ′ using a random permutation π : [ d ] → [ d ] that is shared with the extractor F . The input Y to the algorithm would be the normalized rows of X ′ which are 1 100 λ -close to each other in ℓ 2 norm, so the mechanism has to output an 1 100 -accurate solution w . In particular, after rounding w to {-1 , 1 } d , the coordinates of w must agree with a vast majority of the marked columns, and also with a vast majority of the original marked columns that are located within π (1) , . . . , π ( d 0 ) as it cannot distinguish between them and the other marked columns (because π is hidden from it). The extractor F , given w and π , rounds w to {-1 , 1 } d and outputs q = ( w π (1) , . . . , w π ( d ) ) which agrees with most of the marked columns of X . Hence, we obtain the lower bound of n ≥ ˜ Ω( √ d 0 ) = ˜ Ω( √ d/λ ) .

1

武

Vad

B1

1

1

PAP(X)

B2

√a

1

B3

√a

1

√a

DP Subspace Estimation In our case, we are given a (weak or strong) subspace estimator M : ( S d ) n → R d × d that outputs an λγ -useful rankk projection matrix if σ k +1 ( X ) ≤ γσ k ( X ) (or √ ∑ min { n,d } i = k +1 σ i ( X ) 2 ≤ γσ k ( X ) ). We prove our lower bounds by applying the framework with n 0 = n/k and d 0 = Θ( αd ) , for some parameter α = α ( λ ) that will depend on the type of M we consider. In order to generate hard instances Y ∈ ( S d ) n for M given an FPC codebook X ∼ D ( X ∈ {-1 , 1 } ( n/k ) × d 0 ), we use a variation of the approach to Dwork et al. [2014]. Namely, our generator G samples k independent FPC codebooks A 1 , . . . , A k ∼ D where it plants A i = X for a random i ← [ k ] . Then for each j ∈ [ k ] , it applies (independently) the padding-and-permuting technique of Peter et al. [2024] where it pads ℓ 1 -marked columns and ℓ ( -1) -marked columns for ℓ ≈ d 0 2 α , and permute all the columns. This induces k matrices B 1 , . . . , B k ∈ {-1 , 1 } ( n/k ) × d (for d = d 0 +2 ℓ ) such that each B j is 'almost' rank1 and their vertical concatenation B ∈ {-1 , 1 } n × d is almost rankk . It provides Y = 1 √ d B as the input for M . See Figure 1 for graphical illustrations.

We remark that at this step, the main difference from Dwork et al. [2014] (who implicitly follow a similar paradigm) is that they use a fixed padding length of ℓ = 15 d 0 that suffices for the robustness properties that they need. On the other hand, we use Peter et al. [2024]'s observation that increasing the padding can handle low-accuracy regimes of many problems, and indeed we use the padding length ℓ to increase the k 'th singular value gap, which will be a function of the quality parameter λ .

The next step is to choose the right value of α = α ( λ ) such that M , on input Y , will have to output a useful projection matrix. We show that the input matrix Y has w.h.p. σ 1 ( Y ) 2 ≥ . . . ≥ σ k ( Y ) 2 ≥ (1 -O ( α )) · n k , which yields that ∑ min { n,d } i = k +1 σ i ( Y ) 2 ≤ O ( α ) n . If M is a weak estimator, then we simply use α = Θ ( 1 λ 2 k ) to guarantee that ∑ min { n,d } i = k +1 σ i ( Y ) 2 ≤ γ 2 · σ k ( Y ) 2 for γ = 1 1000 λ , which yields that by the utility guarantee of M , we get an 0 . 001 -useful projection matrix. If M is a strong estimator, then we use α = Θ ( 1 λ 2 ) (i.e., we decrease the padding length by a factor of k ). Yet, in order to meet the requirements of M , we must argue that w.h.p., σ k +1 ( Y ) 2 ≤ ˜ O ( α ) · n k , and this is more complex than the previous case. Here we use more internal properties of the fingerprinting distribution D . Namely, that in Peter et al. [2024]'s construction (which is also true for Tardos [2008]'s one), each entry of the codebook matrix has expectation 0 and the columns of the matrix are independent. Using known concentration bounds, this allows us to argue that if we pick a unit vector v that is orthogonal to the topk rows subspace of Y , then with probability at least 1 -exp( -˜ Ω( d )) it holds that ∥ Y v ∥ 2 2 ≤ ˜ O ( α ) · n k . Since σ 2 k is bounded by the supremum of ∥ Y v ∥ 2 2 under such unit vectors, we conclude the proof of this part using a net argument.

Finally, the last step, which is not trivial for k ≥ 2 , is to extract from an 0 . 001 -useful projection matrix ˜ Π for Y , a vector q ∈ {-1 , 1 } d 0 that with noticeable probability, strongly agrees with the marked columns of the original codebook X ∈ {-1 , 1 } n 0 × d 0 . For that, our extractor F uses the random permutations and the random location i (which are part of the shared secret between the

generator and the extractor) and follows the strategy of Dwork et al. [2014]. That is, it applies the i th invert permutation over the columns of ˜ Π (denote the resulting matrix by ˜ Π i ), chooses a vector u ∈ Span ( ˜ Π i ) that has the maximal agreement with half of the padding bits, and then simply outputs its first d 0 coordinates after rounding to {-1 , 1 } . The intuition is that an 0 . 001 -useful projection matrix must be also 0 . 001 -useful for at least one of the parts Y j = 1 √ d B j . Since all the Y j 's have the same distribution and the location i (where the original X is planted) is hidden from M , then it must be 0 . 001 -useful for Y i with probability at least β/k (where β denotes the success probability of M ). Given that this event occurs, the usefulness of ˜ Π implies that there must exists a vector in Span ( ˜ Π i ) that strongly agrees with half of the padding locations. But because all the marked columns (that includes the padding locations) are indistinguishable from the eyes of M who computed ˜ Π , then a similar agreement must hold for the marked columns of X .

4 Empirical Evaluation

We implemented a zCDP (Definition B.24) variant of our subspace estimation algorithm in Python (denoted by EstSubspace ), and in this section we present empirical results for the fundamental task of privately estimating the average of d dimensional points that approximately lie in a much smaller k -dimensional subspace. Namely, given a dataset X = ( x 1 , . . . , x n ) ∈ S n d , a parameter k , and zCDP parameters ρ, δ , we perform the following steps: (a) Compute a ( ρ/ 2 , δ ) -zCDP rankk projection matrix ˜ Π using EstSubspace that estimates the projection onto the topk rows subspace of X , (b) Compute a ρ/ 2 -zCDP estimation of the average of X using the Gaussian Mechanism: ˜ x = 1 n ∑ n i =1 x i + N ( 0 , σ 2 · I d × d ) for σ = 2 n √ ρ , and (c) Output ˆ x = ˜ Π · ˜ x .

The accuracy is measured by the ℓ 2 error from the average: ∥ ∥ ˆ x -1 n ∑ n i =1 x i ∥ ∥ 2 .

In all our experiments, we use ρ = 2 and δ = 10 -5 , t = 125 (the number of subsets in the sampleand-aggregate process), n = 2 tk data points, q = 10 · k (the number of reference points in the aggregation), and use the zCDP implementation of the FriendlyCore-based averaging algorithm of Tsfadia et al. [2022]. 7 All experiments were tested on a MacBook Pro Laptop with 8-core Apple M1 CPU with 16GB RAM.

Rather than using Tsfadia et al. [2022]'s algorithm for the known-diameter case, we use their unknown-diameter implementation with ξ min = 10 -6 and ξ max = 100 (see Remark 2.4 for details). Furthermore, we reduced the space complexity of our implementation from ˜ Θ( d 2 ) to ˜ Θ( kd ) . 8

In order to generate a synthetic dataset that approximately lie in a k -dimensional subspace, we initially sample uniformly random b 1 , . . . , b k ← {-1 , 1 } d and perform the following process to generate each data point: (i) Sample a random unit vector u in Span { b 1 , . . . , b k } , (ii) Sample a random noise vector ν ←{ 1 /τ, -1 /τ } d , and (iii) Output u + ν ∥ u + ν ∥ (note that higher τ results with data points that are closer to a k -dimensional subspace).

We compare our averaging method to two other approaches: The first one simply applies the Gaussian mechanism directly on X = ( x 1 , . . . , x n ) using the entire privacy budget ρ (i.e., without computing a projection matrix). The second one replaces our Step (a) by computing the projection matrix ˜ Π using a ( ρ/ 2 , δ ) -zCDP variant of the additive-gap based algorithm of Dwork et al. [2014] (see Appendix B.8.4 for more details). 9 The empirical results are presented in Figure 2. In all experiments, we perform 30 repetitions for generating each graph point which represents the trimmed average of

Averaging (Effect of Increasing d)

Averaging (Effect of Increasing k)

Averaging (Effect of Increasing lambda)

0.07

0.07

0.030

Dwork et al. (2014)

0.06

Dwork et al. (2014)

EstSubspace

0.06

Dwork et al. (2014)

EstSubspace

0.025

EstSubspace

0.05

0.05

0.020

error

0.04

error

0.04

error

0.015

o.03

0.010

0.02

0.02

0.01

0.01

FS000

0.00

0.00

0.000

40

60

80

100

120

140

160

180

2

4

6

8

10

12

1

2

3

4

6

7

sqrt(d)

k

tau/d

values between the 0 . 1 and 0 . 9 quantiles. We show the ℓ 2 error of our estimate on the Y -axis. The first graph illustrates the inherent dependency on d that Dwork et al. [2014]'s algorithm has, while our algorithm EstSubspace takes advantage of the closeness of the points to dimension k in order to eliminate this dependency. The second graph illustrates that when d is fixed, increasing k and n in the same rate a has similar affect on both EstSubspace and Dwork et al. [2014]'s algorithm. In the last graph we compare the accuracy of EstSubspace and Dwork et al. [2014]'s algorithm as a function of the closeness to a subspace k (measured in our experiments by the parameter τ ), and show in what regimes EstSubspace outperforms Dwork et al. [2014]'s algorithm.

5 Limitations and Future Directions

From a theoretical perspective, our work is the first to provide proper measures for how 'easy' a given dataset is which smoothly eliminates the dependency on the dimension d . Yet, closing the gap between our upper and lower bounds is still left open. Specifically, for weak estimator, there is still a gap of k 1 . 5 between Theorems 1.6 and 1.8. For strong estimators, the upper-bound rate n = n ( λ ) (Theorem 1.7) does not align with the one of the lower bound (Theorem 1.9), and it is left open to relax the restriction on γ max . One possible reason for some of these gaps (especially the dependency in k ) is that our upper bounds follow the approach of Singhal and Steinke [2021] to estimate (under some matrix norm) the projection matrix to the topk rows subspace (we do it in Frobenius norm). While estimating the projection matrix itself provides, in particular, a useful solution (Proposition B.16), the opposite direction is not true in general, and it could be possible to reduce the sample complexity by focusing on α -usefulness (Definition 1.2) directly, or alternatively, providing stronger lower bounds for estimating the projection matrix.

From a more practical standpoint, we empirically demonstrate the advantage of our approach in high-dimension regimes when the data is very close to a low-dimensional structure, which is directly translated to an advantage in private mean estimation of such instances. The downside of our approach is that it requires the points to be very close to a k -dimensional structure in order to be effective, which might not be sufficient for typical training scenarios in deep learning. It would be intriguing to explore if there is a connection between training parameters (e.g., the network structure) to the phenomena of gradients that are close to a low-dimensional subspace (mentioned in Section 1.1). If we could boost this closeness to regimes where our method achieves high accuracy, we could generate drastically improved private models. On the other hand, if we cannot do it, then our lower bounds indicate that improving DP-SGD via private subspace estimation might not be the right approach, and we should focus on different approaches for this task.

Acknowledgments and Disclosure of Funding

Work supported by a gift to Georgetown University.

The author would like to thank Edith Cohen and Jonathan Ullman for very useful discussions.

References

Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security , CCS '16, page 308-318, 2016a.

Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security , CCS '16, page 308-318, 2016b.

Amin, T. Dick, A. Kulesza, A. Munoz, and S. Vassilvitskii. Differentially private covariance estimation. In Advances in Neural Information Processing Systems , volume 32, 2019.

Arora, V. braverman, and J. Upadhyay. Differentially private robust low-rank approximation. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Ashtiani and C. Liaw. Private and polynomial time algorithms for learning gaussians and beyond. In Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 1075-1076. PMLR, 2022.

Bardenet and O.-A. Maillard. Concentration inequalities for sampling without replacement. Bernoulli , 21(3):1361 - 1385, 2015.

Bassily, A. Smith, and A. Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In Proceedings of the 2014 IEEE 55th Annual Symposium on Foundations of Computer Science , FOCS '14, page 464-473, USA, 2014.

Blocki, A. Blum, A. Datta, and O. Sheffet. The johnson-lindenstrauss transform itself preserves differential privacy. In 2012 IEEE 53rd Annual Symposium on Foundations of Computer Science , pages 410-419, 2012.

Blum, C. Dwork, F. McSherry, and K. Nissim. Practical privacy: The SuLQ framework. In C. Li, editor, PODS , pages 128-138. ACM, 2005.

Bun and T. Steinke. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Theory of Cryptography - 14th International Conference, TCC 2016 , volume 9985, pages 635-658, 2016.

Bun, J. Ullman, and S. P. Vadhan. Fingerprinting codes and the price of approximate differential privacy. In Symposium on Theory of Computing, STOC 2014 , pages 1-10, 2014.

T. Cai and A. Zhang. Rate-optimal perturbation bounds for singular subspaces with applications to high-dimensional statistics. The Annals of Statistics , 46(1):60 - 89, 2018.

Chaudhuri, A. Sarwate, and K. Sinha. Near-optimal differentially private principal components. In Advances in Neural Information Processing Systems , volume 25, 2012.

Chaudhuri, A. D. Sarwate, and K. Sinha. A near-optimal algorithm for differentially-private principal components. Journal of Machine Learning Research , 14, 2013.

Demeniconi and N. Chawla. Hessian based analysis of sgd for deep nets: Dynamics and generalization. In C. Demeniconi and N. Chawla, editors, Proceedings of the 2020 SIAM International Conference on Data Mining (SDM) , 2020.

Dwork, K. Kenthapadi, F. McSherry, I. Mironov, and M. Naor. Our data, ourselves: Privacy via distributed noise generation. In EUROCRYPT , volume 4004, pages 486-503, 2006a.

Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating noise to sensitivity in private data analysis. In TCC , volume 3876, pages 265-284, 2006b.

Dwork, K. Talwar, A. Thakurta, and L. Zhang. Analyze gauss: Optimal bounds for privacypreserving principal component analysis. In Proceedings of the 46th Annual ACM Symposium on Theory of Computing , STOC '14, pages 11-20. ACM, 2014. URL http://kunaltalwar.org/ papers/PrivatePCA.pdf .

Dwork, A. D. Smith, T. Steinke, J. R. Ullman, and S. P. Vadhan. Robust traceability from trace amounts. In IEEE 56th Annual Symposium on Foundations of Computer Science, FOCS 2015 , pages 650-669, 2015.

Feng and Y. Tu. How neural networks find generalizable solutions: Self-tuned annealing in deep learning, 2020.

Golatkar, A. Achille, Y. Wang, A. Roth, M. Kearns, and S. Soatto. Mixed differential privacy in computer vision. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8366-8376. IEEE Computer Society, 2022.

Gonem and R. Gilad-Bachrach. Smooth sensitivity based approach for differentially private pca. In Proceedings of Algorithmic Learning Theory , volume 83, pages 438-450. PMLR, 2018.

Gur-Ari, D. A. Roberts, and E. Dyer. Gradient descent happens in a tiny subspace, 2018.

Hardt and E. Price. The noisy power method: A meta algorithm with applications. In Advances in Neural Information Processing Systems , volume 27, 2014.

Hardt and A. Roth. Beating randomized response on incoherent matrices. In Proceedings of the Forty-Fourth Annual ACM Symposium on Theory of Computing , STOC '12, page 1255-1268, 2012.

Hardt and A. Roth. Beyond worst-case analysis in private singular vector computation. In Proceedings of the Forty-Fifth Annual ACM Symposium on Theory of Computing , STOC '13, page 331-340, 2013.

Kairouz, M. Ribero, K. Rush, and A. Thakurta. Dimension independence in unconstrained private ERM via adaptive preconditioning. CoRR , abs/2008.06570, 2020.

Kamath, A. Mouzakis, V. Singhal, T. Steinke, and J. Ullman. A private and computationallyefficient estimator for unbounded gaussians. In Proceedings of Thirty Fifth Conference on Learning Theory , volume 178, pages 544-572. PMLR, 02-05 Jul 2022.

Kapralov and K. Talwar. On differentially private low rank approximation. In Proceedings of the twenty-fourth annual ACM-SIAM symposium on Discrete algorithms , pages 1395-1414, 2013.

Karwa and S. Vadhan. Finite sample differentially private confidence intervals. In 9th Innovations in Theoretical Computer Science Conference (ITCS 2018) , 2018.

Kothari, P. Manurangsi, and A. Velingker. Private robust estimation by stabilizing convex relaxations. In Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 723-777. PMLR, 2022.

Li, H. Farkhoor, R. Liu, and J. Yosinski. Measuring the intrinsic dimension of objective landscapes. CoRR , abs/1804.08838, 2018a.

Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018b.

Li, D. Liu, T. Hashimoto, H. A. Inan, J. Kulkarni, Y. Lee, and A. G. Thakurta. When does differentially private learning not suffer in high dimensions? In Advances in Neural Information Processing Systems , 2022.

W. Mahoney. Randomized algorithms for matrices and data. Found. Trends Mach. Learn. , 3(2): 123-224, 2011.

McSherry and I. Mironov. Differentially private recommender systems: building privacy into the net. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining , KDD '09, pages 627-636, New York, NY, USA, 2009. ACM.

McSherry and K. Talwar. Mechanism design via differential privacy. In FOCS , pages 94-103. IEEE Computer Society, 2007.

J. Montgomery-Smith. The distribution of rademacher sums. Proceedings of the American Mathematical Society , 109(2):517-522, 1990.

Nissim, S. Raskhodnikova, and A. Smith. Smooth sensitivity and sampling in private data analysis. In STOC , pages 75-84. ACM, 2007.

Peter, E. Tsfadia, and J. R. Ullman. Smooth lower bounds for differentially private algorithms via padding-and-permuting fingerprinting codes. In The Thirty Seventh Annual Conference on Learning Theory, COLT 2024 , volume 247, pages 4207-4239, 2024.

Rényi. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics , page 547-561, 1961.

Scala. Hypergeometric tail inequalities: ending the insanity. arXiv preprint arXiv:1311.5939 , 2009.

Sheffet. Old techniques in differentially private linear regression. In Proceedings of the 30th International Conference on Algorithmic Learning Theory , volume 98 of Proceedings of Machine Learning Research , pages 789-827. PMLR, 2019.

Singhal and T. Steinke. Privately learning subspaces. Advances in Neural Information Processing Systems , 34, 2021. URL https://arxiv.org/abs/2106.00001 .

Song, K. Chaudhuri, and A. D. Sarwate. Stochastic gradient descent with differentially private updates. In 2013 IEEE Global Conference on Signal and Information Processing , pages 245-248, 2013.

Steinke and J. R. Ullman. Subgaussian tail bounds via stability arguments. CoRR , abs/1701.03493, 2017.

Tardos. Optimal probabilistic fingerprint codes. J. ACM , 55(2), 2008.

Tsfadia, E. Cohen, H. Kaplan, Y. Mansour, and U. Stemmer. Friendlycore: Practical differentially private aggregation. In International Conference on Machine Learning, ICML 2022 , volume 162 of Proceedings of Machine Learning Research , pages 21828-21863. PMLR, 2022.

Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018.

Wei, A. D. Sarwate, J. Corander, A. Hero, and V. Tarokh. Analysis of a privacy-preserving pca algorithm using random matrix theory. In 2016 IEEE Global Conference on Signal and Information Processing (GlobalSIP) , pages 1335-1339, 2016.

Yu, H. Zhang, W. Chen, and T.-Y. Liu. Do not let privacy overbill utility: Gradient embedding perturbation for private learning. In International Conference on Learning Representations , 2021.

Zhou, S. Wu, and A. Banerjee. Bypassing the ambient dimension: Private {sgd} with gradient subspace identification. In International Conference on Learning Representations , 2021.

A Additional Related Work

A closely related line of work is on Private PCA. Dwork et al. [2014] consider the simple algorithm that adds independent Gaussian noise to each entry of the covariance matrix A = ∑ n i =1 x T i x i ∈ R d × d , and then performs analysis on the noisy matrix. This method, predating the development of differential privacy Blum et al. [2005], was later analyzed under differential privacy by McSherry and Mironov [2009] and Chaudhuri et al. [2013]. This simple algorithm is versatile and several bounds are provided for the accuracy of the noisy PCA. The downside of this is that a polynomial dependence on the ambient dimension d is inherent for any instances (including 'easy" ones). While this approach has a variant that improves the accuracy of estimating the topk subspace as a function of the additive gap σ 2 k -σ 2 k +1 (Appendix B.8.4), it does not prevent the polynomial dependency on the dimension d even for very easy instances.

Techniques from dimensionality reduction have been applied by Hardt and Roth [2012] and Arora et al. [2018] to privately compute a low-rank approximation to the input matrix X . Similarly, Hardt and Roth [2013] and Hardt and Price [2014] utilize the power iteration method with noise injected at each step to compute low-rank approximations to X . Despite their effectiveness, these methods, relying on noise addition, require sample complexity to grow polynomially with the ambient dimension to achieve meaningful guarantees.

Another approach, employed by Blocki et al. [2012] and Sheffet [2019], involves approximating the covariance matrix A using dimensionality reduction. They show that the dimensionality reduction step itself provides a privacy guarantee (whereas the aforementioned results did not exploit this and relied on noise added at a later stage).

Chaudhuri et al. [2012], Kapralov and Talwar [2013], Wei et al. [2016] apply variants of the exponential mechanism McSherry and Talwar [2007] to privately select a low-rank approximation to the covariance matrix A . This method is nontrivial to implement and analyse, but it ultimately requires the sample complexity to grow polynomially in the ambient dimension.

Gonem and Gilad-Bachrach [2018] exploit smooth sensitivity Nissim et al. [2007] to release a lowrank approximation to the matrix A . This allows adding less noise than worst case sensitivity, under an eigenvalue gap assumption. However, the sample complexity n is polynomial in the dimension d .

Another related area involves estimating the parameters of unbounded Gaussians Kamath et al. [2022], Ashtiani and Liaw [2022], Kothari et al. [2022], Tsfadia et al. [2022]. Notably, Kamath et al. [2022] used the subspace learning algorithm of Singhal and Steinke [2021] to efficiently learn the covariance matrix.

Arecent popular trend in DP learning is to utilize a few public examples to enhance accuracy. This has led to methods for private ML which project the sensitive gradients onto a subspace estimated from the public gradients. By using a small amount of i.i.d. public data, Zhou et al. [2021] demonstrate that this approach can improve the accuracy of differentially private stochastic gradient descent in high-privacy regimes and achieve a dimension-independent error rate. Similarly, Yu et al. [2021] proposed GEP, a method that utilizes public data to identify the most useful information carried by gradients, and then splits and clips them separately. These works underscore the importance of identifying the subspace of gradients in private ML.

B Preliminaries

B.1 Notations

We use calligraphic letters to denote sets and distributions, uppercase for matrices and datasets, boldface for random variables, and lowercase for vectors, values and functions. For n ∈ N , let [ n ] = { 1 , 2 , . . . , n } . Throughout this paper, we use i ∈ [ n ] as a row index, and j ∈ [ d ] as a column index (unless otherwise mentioned).

For a matrix X = ( x j i ) i ∈ [ n ] ,j ∈ [ d ] , we denote by x i the i th row of X and by x j the j th column of X . A column vector x ∈ R n is written as ( x 1 , . . . , x n ) or x = x 1 ...n , and a row vector y ∈ R d is written as ( y 1 , . . . , y d ) or y 1 ...d . In this work we consider mechanisms who receive an n × d matrix X as input, which is treated as the dataset X = ( x 1 , . . . , x n ) where the rows of X are the elements (and therefore, we sometimes write X ∈ ( R d ) n instead of X ∈ R n × d to emphasize it).

For a vector x ∈ R d we define ∥ x ∥ 2 = √∑ n i =1 x 2 i (the ℓ 2 norm of x ), and for a subset S ⊆ [ d ] we define x S = ( x i ) i ∈S , and in case x is a row vector we write x S . Given two vectors x = ( x 1 , . . . , x n ) , y = ( y 1 , . . . , y n ) , we define ⟨ x, y ⟩ = ∑ n i =1 x i y i (the inner-product of x and y ). We denote by S d the set of d -dimensional unit vectors, that is, S d = { v ∈ R d : ∥ v ∥ 2 = 1 } . For a matrix X = ( x j i ) i ∈ [ n ] ,j ∈ [ d ] we let ∥ X ∥ = max v ∈S d ∥ Xv ∥ 2 (the spectral norm of X ) and let ∥ X ∥ F = √ ∑ i ∈ [ n ] ,j ∈ [ d ] ( x j i ) 2 (the Forbenius norm of X ). For a matrix X = ( x j i ) i ∈ [ n ] ,j ∈ [ d ] ∈ {-1 , 1 } n × d and b ∈ {-1 , 1 } , we define the b -marked columns of X as the subset J b X ⊆ [ d ] defined by J b X = { j ∈ [ d ] : x j i = b for all i ∈ [ n ] } .

For d ∈ N we denote by P d the set of all d × d permutation matrices. For a permutation matrix P ∈ P d and i ∈ [ d ] we denote by P ( i ) the index j ∈ [ d ] for which e i P = e j (where e i and e j are the corresponding one-hot row vectors), and for I ⊆ [ d ] we denote P ( I ) = { P ( i ): i ∈ I} .

For d, k ∈ N we denote by W d,k the set of all d × d projection matrices of rank k . For a matrix A ∈ R n × d we denote by Span ( A ) the columns subspace of A (and therefore the rows subspace of A is Span ( A T ) ).

B.2 Distributions and Random Variables

Given a distribution D , we write x ∼ D to denote that x is sampled according to D . For a set S , we write x ←S to denote that x is sampled from the uniform distribution over S .

B.3 Singular Value Decomposition (SVD)

For a matrix X ∈ R n × d , the singular value decomposition of X is defined by X = U Σ V T , where U ∈ R n × n and V ∈ R d × d are unitary matrices. The matrix Σ ∈ R n × d is a diagonal matrix with non-negative entries σ 1 ≥ . . . ≥ σ min { n,d } ≥ 0 along the diagonal, called the singular values of X .

The SVD of X can also be written in the form X = ∑ min { n,d } i =1 σ i u i v T i , where u i and v i are the i th columns of U and V (respectively). It holds that ∥ X ∥ 2 F = ∑ i σ 2 i and ∥ X ∥ = σ 1 . We define the topk rows subspace of X as the subspace spawned by the first k columns of V .

Fact B.1 (Min-Max principle for singular values) . For every matrix X ∈ R n × d and i ∈ [min { n, d } ] it holds that

B.4 Concentration Bounds

Fact B.2 (Montgomery-Smith [1990]) . Let x = ( x 1 , . . . , x n ) where the x i 's are i.i.d. random variables with Pr[ x i = 1] = Pr[ x i = -1] = 1 / 2 , and let v ∈ R n . Then

Fact B.3 (Bernstein's Inequality for sampling without replacement (Bardenet and Maillard [2015], Proposition 1.4)) . Let x 1 , . . . , x m be a random sample drawn without replacement from { w 1 , . . . , w n } for n ≥ m . Let a = min i ∈ [ n ] w i , b = max i ∈ [ n ] w i , µ = 1 n ∑ n i =1 w i and σ 2 = 1 n ∑ n i =1 ( w i -µ ) 2 . Then for every t ≥ 0 ,

The following proposition is used for proving Lemma 2.1.

Proposition B.4. Let X = ( x 1 , . . . , x n ) ∈ ( S d ) n with singular values σ 1 ≥ . . . ≥ σ min { n,d } ≥ 0 . Let X ′ = ( x ′ 1 , . . . , x ′ m ) be the random matrix that is generated by taking a uniformly random

m -size subset of the rows of X . Let v ∈ S d and a 2 = ∥ Xv ∥ 2 2 = ∑ n i =1 ⟨ x i , v ⟩ 2 . Assuming that m ≥ 2 n ln(1 /β ) 9 a 2 , it holds that

Proof. The random variable ∥ X ′ v ∥ 2 2 is distributed as the sum of a uniformly random m -size subset of ( ⟨ x 1 , v ⟩ 2 , . . . , ⟨ x n , v ⟩ 2 ) (each element is bounded in [0 , 1] ). In the notations of Fact B.3, it holds that µ = 1 n ∑ n i =1 ⟨ x i , v ⟩ 2 = a 2 /n , and

where the last inequality holds since ⟨ x i , v ⟩ 2 ≤ 1 .

Let t = √ 2 m n a 2 ln(1 /β ) . By Fact B.3,

where the penultimate inequality holds by the assumption on m .

B.4.1 Hypergeometric Distributions

Definition B.5. For n ∈ N , m ∈ [ n ] and w ∈ {-n, . . . , n } , define the Hypergeometric probability distribution HG n,m,w as the output of the following process: Take a vector v ∈ {-1 , 1 } n with ∑ n i =1 v i = w , choose a uniformly random subset I ⊆ [ n ] of size m , and output ∑ i ∈I v i . Fact B.6 (Scala [2009], Equations 10 and 14) . If x ∼ HG n,m,w then

where µ = E[ x ] = m · w n .

B.4.2 Sub-Exponential Distributions

Definition B.7 (Sub-Exponential Random Variable and Norm) . We say that a random variable x ∈ R is sub-exponential if there exists t > 0 such that E [ e | x | /t ] ≤ 2 . The sub-exponential norm of x , denoted ∥ x ∥ ψ 1 , is

Fact B.8 (Bernstein's inequality (Theorem 2.8.1 in Vershynin [2018])) . Let x 1 , . . . , x n be independent, mean zero, sub-exponential random variables. Then

□

B.5 Nets

Definition B.9 ( γ -Net) . Let T be a subspace of R d . Consider a subset K ⊂ T and let γ > 0 . A subset N ⊆ K is called an γ -net of K if every point in K is within distance γ of some point of N , i.e.,

Fact B.10 (Extension of Corollary 4.2.13 in Vershynin [2018]) . If E is a subspace with dim( E ) = k , then there exists an γ -net of size (3 /γ ) k to the unit sphere in E (i.e., to E ∩ S d ).

B.6 Projections

Recall that W d,k denotes the set of all d × d rankk projection matrices. For a matrix A ∈ R d , we denote that Π A the projection matrix onto the subspace spawned by the columns of A (in case A is a unitary matrix, Π A = AA T ).

Fact B.11 (Theorem 1 and Lemma 1 in Cai and Zhang [2018]) . Let X,Y,Z ∈ R n × d such that X = Y + Z . Let [ U U ⊥ ]Σ[ V V ⊥ ] T be the SVD of Y , and let [ ̂ U ̂ U ⊥ ] ̂ Σ[ ̂ V ̂ V ⊥ ] T be the SVD of X , where U, V, ̂ U, ̂ V denote the first k columns of [ U U ⊥ ] , [ V V ⊥ ] , [ ̂ U ̂ U ⊥ ] , [ ̂ V ̂ V ⊥ ] (respectively). Let Z 12 = Π U Z Π V ⊥ and Z 21 = Π U ⊥ Z Π V . In addition, let z ij = ∥ Z ij ∥ , let α = σ min ( U T XV ) (i.e., the smallest singular value larger than 0 ), and β = ∥ ∥ U T ⊥ XV ⊥ ∥ ∥ . If α 2 > β 2 +min { z 2 12 , z 2 21 } , then

Proposition B.12. Let X,Y,Z,V, ̂ V as in Fact B.11 such that Y has rankk and Span ( Y T ) is orthogonal to Span ( Z T ) . If σ k ( Y ) 2 ≥ 2 ∥ Z ∥ 2 , then

Proof. Note that Span ( V ) = Span ( Y T ) . Therefore ZV = 0 , which implies that Z 21 = Π U ⊥ Z Π V = 0 . Compute

where the last inequalities in the above two equations hold since for any matrix A and projection matrices Π 1 , Π 2 it holds that ∥ Π 1 A Π 2 ∥ ≤ ∥ A ∥ and ∥ Π 1 A Π 2 ∥ F ≤ ∥ A ∥ F . The proof now immediately follows by applying Fact B.11. □

and

Proposition B.13. Let P ∈ R n × d be a rankk matrix and let P ′ ∈ R n × d such that ∥ P -P ′ ∥ F ≤ α . Let Π be the projection matrix onto Span ( P T ) , and let Π ′ be the projection onto the topk rows subspace of P ′ . If σ k ( P ) ≥ 2 α , then

Proof. Let E = P ′ -P , and divide E into E = E P + E ¯ P where the rows of E P belong to the rows subspace of P and the rows of E ¯ P are orthogonal to it. Let Y = P + E P , so we can write P ′ = Y + E ¯ P . Note that ∥ E ¯ P ∥ F ≤ α , and σ k ( Y ) ≥ σ k ( P ) -∥ E P ∥ ≥ σ k ( P ) -α . The proof now follows by applying Proposition B.12(2) on P ′ , Y, E ¯ P . □

Fact B.14 (Implied by Corollary 4.6 in Singhal and Steinke [2021]) . Let Π , Π 1 , . . . , Π t ∈ W d,k s.t. for all j ∈ [ t ] , ∥ Π -Π j ∥ ≤ α . Let p 1 , . . . , p q be i.i.d. random vectors in R d from N ( ⃗ 0 , I d × d ) . Then

Fact B.15 (Implied by the proof of Lemma 4.9 in Singhal and Steinke [2021]) . Let Π ∈ W d,k and let p 1 , . . . , p q be i.i.d. random vectors in R d from N ( ⃗ 0 , I d × d ) . Let P be the d × q matrix whose columns are Π p 1 , . . . , Π p q . If q ≥ c · k for some large enough constant c , then w.p. 0 . 95 it hold that σ k ( P ) ≥ Ω( √ k ) (and in particular, Span ( P ) = Span (Π) ).

Proposition B.16. For any Π , ˜ Π ∈ W d,k and X ∈ ( S d ) n , it holds that

Proof. Compute

□

Proposition B.17. Let v 1 , . . . , v k ∈ S d with max i,j |⟨ v i , v j ⟩| ≤ α ≤ 1 20 . Let u 1 , . . . , u k be the result of the Gram-Schmidt process applied on v 1 , . . . , v k . Then for every i ∈ [ k ] , there exists λ i -1 ∈ R with | λ i -1 | ≤ α (1 + 4 α ) and w i ∈ R d with ∥ w i ∥ 2 ≤ 2 α 2 such that

Proof. We prove it by induction on i . The case i = 1 holds since u 1 = v 1 . Assume it holds for i , and we prove it for i +1 . Define λ i = -⟨ u i , v i +1 ⟩ and w i +1 = λ i ( λ i -1 v i -1 + w i ) . Note that

and that ∥ w ∥ 2 ≤ α 2 (1 + 4 α ) 2 +2 α 3 ≤ 2 α 2 (recall that α ≤ 1 20 ). The proof now follows since

□

B.7 Algorithms

Let M be a randomized algorithm that uses m random coins. For r ∈ { 0 , 1 } m we denote by M r the (deterministic) algorithm M after fixing its random coins to r . Given an oracle-aided algorithm A and algorithm B , we denote by A B the algorithm A with oracle access to B .

B.8 Differential Privacy

Definition B.18 (Differential privacy [Dwork et al., 2006b,a]) . A randomized mechanism M : X n → Y is ( ε, δ ) -differentially private (in short, ( ε, δ ) -DP) if for every neighboring databases X = ( x 1 , . . . , x n ) , X ′ = ( x ′ 1 , . . . , x ′ n ) ∈ X n (i.e., differ by exactly one entry), and every set of outputs T ⊆ Y , it holds that

B.8.1 Basic Facts

Fact B.19 (Post-Processing) . If M : X n →Y is ( ε, δ ) -DP then for every randomized F : Y → Z , the mechanism F ◦ M : X n →Z is ( ε, δ ) -DP.

Post-processing holds when applying the function on the output of the DP mechanism. In this work we sometimes need to apply the mechanism on the output of a function. While this process does not preserve DP in general, it does so assuming the function is neighboring-preserving .

Definition B.20 (Neighboring-Preserving Algorithm) . We say that a randomized algorithm G : X n → Y m is neighboring-preserving if for every neighboring X,X ′ ∈ X n , the outputs G ( X ) , G ( X ′ ) ∈ Y m are neighboring w.p. 1 .

Fact B.21. If G : X n → Y m is neighboring-preserving and M : Y m → Z is ( ε, δ ) -DP, then M ◦ G : X n →Z is ( ε, δ ) -DP.

B.8.2 Zero-Concentrated Differential Privacy (zCDP)

Our empirical evaluation (Section 4) is performed in the zCDP model of Bun and Steinke [2016], defined below.

Definition B.22 (Rényi Divergence (Rényi [1961])) . Let y and y ′ be random variables over Y . For α ∈ (1 , ∞ ) , the Rényi divergence of order α between y and y ′ is defined by

where P ( · ) and P ′ ( · ) are the probability mass/density functions of y and y ′ , respectively.

Definition B.23 (zCDP Indistinguishability) . We say that two random variable y , y ′ over a domain Y are ρ -indistinguishable (denote by y ≈ ρ y ′ ), iff for every α ∈ (1 , ∞ ) it holds that

We say that y , y ′ are ( ρ, δ ) -indistinguishable (denote by y ≈ ρ,δ y ′ ), iff there exist events E,E ′ ⊆ X with Pr[ y ∈ E ] , Pr[ y ′ ∈ E ′ ] ≥ 1 -δ such that y | E ≈ ρ y | E ′ .

Definition B.24 ( ( ρ, δ ) -zCDP Bun and Steinke [2016]) . A mechanism M is δ -approximate ρ -zCDP (in short, ( ρ, δ ) -zCDP), if for any neighboring databases X,X ′ it holds that M ( X ) ≈ ρ,δ M ( X ′ ) .

The Gaussian Mechanism

Fact B.25 (The Gaussian Mechanism Dwork et al. [2006a], Bun and Steinke [2016]) . Let x , x ′ ∈ R d be vectors with ∥ x -x ′ ∥ 2 ≤ λ . For ρ > 0 , σ = λ √ 2 ρ and Z ∼ N ( 0 , σ 2 · I d × d ) it holds that x + Z ≈ ρ x ′ + Z .

B.8.3 FriendlyCore Averaging

We use the following DP averaging algorithm that given the diameter ξ of a ball that contain most of the points, it can estimate their average.

Fact B.26 (Tsfadia et al. [2022]) . Let λ ≥ 1 and δ ≤ ε, β ≤ 1 . There exists an ( ε, δ ) -DP algorithm FC _ Average that gets as input a dataset S = ( x 1 , . . . , x n ) ∈ ( R d ) n and a parameter ξ > 0 and satisfies the following utility guarantee: If n ≥ O ( log(1 /δ ) ε + √ d log(1 /δ ) log(1 /β ) λε ) and ∃ S ′ ⊆ S

′ ′

where µ = 1 | S ′ | ∑ x ∈ S ′ x . Furthermore, the running time of FC _ Average ( S, · ) is ˜ O ( dn log( n/δ )) (See Appendix B in Tsfadia et al. [2022]).

Fact B.26 is not explicitly stated in Tsfadia et al. [2022] since they only analyzed the utility guarantee of their averaging in the zCDP model. Yet, it can be achieved using similar steps. First, we need to consider a 'friendly" DP variant of their FriendlyAvg algorithm (Algorithm 3.3 in Tsfadia et al. [2022]), and as Tsfadia et al. [2022] noted, we can do it such that the probability of failure is low whenever

n

O

log(1

ε

, and the additive error (given success) decreases in a rate of

√

ξ d log(1 /δ ) log(1 /β ) nε , where ξ is the diameter of the points. Fact B.26 immediately obtained by combining FriendlyAvg with their paradigm for DP (Theorem 4.11 in Tsfadia et al. [2022] applied with α = 0 . 2 ).

B.8.4 Subspace Recovery Algorithm of Dwork et al. [2014]

We next describe the subspace recovery algorithm of Dwork et al. [2014] that strongly takes advantage of a large additive gap σ 2 k -σ 2 k +1 for decreasing the noise that is required for privacy. This algorithm is only used in our empirical evaluation (Section 4).

Algorithm B.27 (Algorithm 2 in Dwork et al. [2014]) .

Input: A dataset X = ( x 1 , . . . , x n ) ∈ ( S d ) n .

DP parameters:

ε, δ .

Rank parameter: k .

Operation:

Compute a projection matrix Π to the topk rows subspace of X , and compute the singular values σ k , σ k +1 .

Compute g = σ 2 k -σ 2 k +1 + Lap (2 /ε ) .

Compute W = Π + E , where E is a d × d symmetric matrix where the upper triangle is i.i.d. samples from N ( 0 , ( ∆ ε,δ g -2 log(1 /δ ) /ε -2 ) 2 ) , where ∆ ε,δ = 1+ √ 2 log(1 /δ ) ε .

Output a projection matrix ˜ Π to the top k eigenvectors of W .

Note that when the additive gap σ 2 k -σ 2 k +1 is large, the algorithm will add smaller noise per coordinate in Step 3.

Fact B.28 (Theorem 11 in Dwork et al. [2014]) . Algorithm B.27 is (2 ε, 2 δ ) -DP.

The privacy analysis is done by the following steps. First, by the Laplace mechanism, Step 2 is ε -DP. Second, by tail bound on the Laplace distribution, the probability that g -2 log(1 /δ ) /ε ≤ σ 2 k -σ 2 k +1 is at least 1 -δ . Furthermore, they show that if σ 2 k -σ 2 k +1 ≥ α then the Forbenius-norm sensitivity of the matrix Π is at most 2 α -2 . So conditioned on the above 1 -δ probability event, the Forbenius-norm sensitivity of Π is at most 2 g -2 log(1 /δ ) /ε -2 , and therefore the Gaussian mechanism step (Item 3) guarantees ( ε, δ ) -DP, and by composition the entire process is (2 ε, 2 δ ) -DP.

In order to consider a zCDP version of Algorithm B.27, we replace the Laplace noise Lap (2 /ε ) with a Gaussian noise N (0 , σ 2 ) for σ = √ 2 /ρ . Given this change, now it holds that g -σ √ 2 ln(1 /δ ) ≤ σ 2 k -σ 2 k +1 w.p. at least 1 -δ (by tail bound on Gaussian distribution). Finally, in Step 3 we replace ∆ ε,δ (the required standard deviation for ( ε, δ ) -DP) to 1 / √ 2 ρ (what is required for ρ -zCDP). This results with the following (2 ρ, δ ) -zCDP algorithm:

(

/δ

)

)

Algorithm B.29 (zCDP version of Algorithm 2 in Dwork et al. [2014]) .

Input: A dataset X = ( x 1 , . . . , x n ) ∈ ( S d ) n .

zCDP parameters: ρ, δ .

Rank parameter: k .

Operation:

Compute a projection matrix Π to the topk rows subspace of X , and compute the singular values σ k , σ k +1 .

Compute g = σ 2 k -σ 2 k +1 + N (0 , 2 /ρ ) .

Compute W = Π + E , where E is a d × d symmetric matrix where the upper triangle is i.i.d. samples from N ( 0 , ( √ 1 / (2 ρ ) g -2 √ ln(1 /δ ) /ρ -2 ) 2 ) .

Output a projection matrix ˜ Π to the top k eigenvectors of W .

B.8.5 Lower Bounding Tools from Peter et al. [2024]

Peter et al. [2024] showed that for d = ˜ Θ( n 2 ) , the distribution D ( n, d ) below induces a fingerprinting codebook for n users, each codeword is of length d .

Definition B.30 (FPC hard distribution D ( n, d ) Peter et al. [2024]) . Let ρ be the distribution that outputs p = ( e t -1) / ( e t + 1) ∈ [ -1 , 1] for t ← [ -ln(5 n ) , ln(5 n )] . For n, d ∈ N , let D ( n, d ) be the distribution that chooses independently p 1 , . . . , p d ∼ ρ , and outputs a codebook ( x 1 , . . . , x n ) ∈ ( {-1 , 1 } d ) n where for each i ∈ [ n ] and j ∈ [ d ] , x j i is drawn independently over {-1 , 1 } with expectation p j .

Framework for Lower Bounds Consider a mechanism M : X n →W that satisfies some weak accuracy guarantee. Peter et al. [2024] showed that the task of proving a lower bound on n is reduced to the following task: Transform an FPC codebook X ∈ {-1 , 1 } n 0 × d 0 into hard instances Y ∈ X n for M , and then extract from the output w ∈ W of M ( Y ) a vector q ∈ {-1 , 1 } d 0 that is strongly-correlated with X ( n 0 and d 0 are some functions of n and d and the weak accuracy guarantee of M ), where

Definition B.31 (Strongly Correlated) . We say that a random variable q = ( q 1 , . . . , q d ) ∈ {-1 , 1 } d is strongly-correlated with a matrix X ∈ {-1 , 1 } n × d , if

(recall that J b X = { j ∈ [ d ] : x j i = b for all i ∈ [ n ] } ).

Denote by G : {-1 , 1 } n 0 × d 0 × Z → X n the algorithm that generates the hard instances using a uniformly random secret z ← Z (i.e., z could be a random permutation, a sequence of random permutations, etc). Denote by F : Z ×W → {1 , 1 } d 0 the algorithm that extracts a good q using the secret z and the output w . We denote by A M , F , G ( X ) the entire process:

Definition B.32 (Algorithm A M , F , G ) . Let Z , W be domains, and let n 0 , d 0 , n, d ∈ N . Let ( M , F , G ) be a triplet of randomized algorithms of types G : {-1 , 1 } n 0 × d 0 ×Z → X n , M : X n →W , and F : Z × W → [ -1 , 1] d 0 , each uses m random coins. Define A M , F , G : {-1 , 1 } n 0 × d 0 → [ -1 , 1] d 0 as the randomized algorithm that on inputs X ∈ {-1 , 1 } n 0 × d 0 , samples z ← Z , Y ∼ G ( X,z ) and w ∼ M ( Y ) , and outputs q ∼ F ( z, w ) .

Definition B.33 ( β -Leaking) . Let M , F , G be randomized algorithms as in Definition B.32, each uses m random coins, and let D ( n 0 , d 0 ) be the distribution from Definition B.30. We say that the triplet ( M , F , G ) is β -leaking if

where recall that M r denotes the algorithm M when fixing its random coins to r ( F r ′ , G r ′′ are similarly defined).

LemmaB.34 (Framework for Lower Bounds Peter et al. [2024]) . Let β ∈ (0 , 1] , n 0 , n, d 0 , d ∈ N . Let M : X n →W be an algorithm such that there exists two algorithms G : {-1 , 1 } n 0 × d 0 ×Z → X n and F : Z × W → [ -1 , 1] d 0 such that the triplet ( M , F , G ) is β -leaking (Definition B.33). If M is ( 1 , β 4 n 0 ) -DP and G ( · , z ) is neighboring-preserving (Definition B.20) for every z ∈ Z , then n 0 ≥ Ω ( √ d 0 log 1 . 5 ( d 0 /β ) ) .

Padding-And-Permuting (PAP) FPC The main technical tool of Peter et al. [2024] for generating hard instance is to sample a random fingerprinting codebook from D ( n, d 0 ) , append many 1 -marked and ( -1) -marked columns, and randomly permute all the columns.

Definition B.35 ( PAP n,d 0 ,ℓ ) . Let ℓ, n, d 0 ∈ N , and let d = d 0 + 2 ℓ . We define PAP n,d 0 ,ℓ : {-1 , 1 } n × d 0 ×P d →{-1 , 1 } n × d as the function that given X ∈ {-1 , 1 } n × d 0 and a permutation matrix P ∈ P d as inputs, outputs X ′ = X ′′ · P (i.e., permutes the columns of X ′′ according to P ), where X ′′ is the {-1 , 1 } n × d matrix after appending ℓ 1 -marked and ℓ ( -1) -marked columns to X (where recall that a b -marked is a column with all entries equal to b ).

Note that for every n, d 0 , ℓ ∈ N and P ∈ P d , the function PAP n,d 0 ,ℓ ( · , P ) is neighboring-preserving (Definition B.20).

Definition B.36 (Strongly Agrees) . We say that a vector q = ( q 1 , . . . , q d ) strongly-agrees with a matrix X ∈ {-1 , 1 } n × d , if

The following lemma capture the main technical property of the PAP technique.

Lemma B.37 (Peter et al. [2024]) . Let ℓ, n, d 0 ∈ N such that d = d 0 +2 ℓ . Let M : {-1 , 1 } n × d → [ -1 , 1] d be an mechanism that uses m random coins, P ← P d (a random variable) and for X ∈ {-1 , 1 } n × d 0 let Y X = PAP ( X, P ) . Then for any distribution D over {-1 , 1 } n × d 0 :

C Upper Bounds

In this section we prove our upper bounds for subspace estimation. In Appendix C.2 we prove Theorem 1.6, and in Appendix C.3 we prove Theorem 1.7. Both algorithm share a similar structure that is defined next in Appendix C.1.

C.1 Base Algorithms

Similarly to Singhal and Steinke [2021], our algorithms will follow the sample and aggregate approach of Nissim et al. [2007]. That is, we partition the rows into t subsets, compute (non-privately) the topk projection matrix of each subset, and then privately aggregate the projections. This is Algorithm C.1 that uses oracle access to an aggregation algorithm. Unlike Singhal and Steinke [2021] who assumed that the rows are i.i.d. Gaussian samples, here we take a random partition and show that with large enough probability over the randomness of the partition, the projection matrices are indeed close to each other. We consider two types of aggregations: The first type, called Algorithm C.2, simply treats the matrices as vectors of dimension d 2 and computes a DP-average of them using FriendlyCore averaging (Fact B.26). The second type, called Algorithm C.3, is more similar to the aggregation done by Singhal and Steinke [2021]. That is, sample reference points p 1 , . . . , p q and then aggregate the kd dimensional points { (Π j p 1 , . . . , Π j p q ) } t j =1 . The difference from Singhal and Steinke [2021] is that we use FriendlyCore averaging (Fact B.26) which simplifies the construction.

Algorithm C.1 (Algorithm EstSubspace ) .

Input: A dataset X = ( x 1 , . . . , x n ) ∈ ( S d ) n .

Parameters:

k, t .

Oracle: A DP algorithm Agg for aggregating projection matrices.

Operation:

Randomly split X into t subsets, each contains (at least) m = ⌊ n/t ⌋ rows. Let X 1 , . . . , X t be the resulting subsets.

For each j ∈ [ t ] : Compute the projection matrix Π j of the topk rows subset of X j .

Output ˜ Π ∼ Agg (Π 1 , . . . , Π t ) .

Algorithm C.2 (Algorithm Naive _ Agg ) .

Input: A dataset ⃗ Π = (Π 1 , . . . , Π t ) ∈ ( W d,k ) t .

Privacy parameters:

ε, δ ≤ 1 .

Utility parameter:

ξ ∈ [0 , 1] .

Operation:

Compute ̂ Π ∼ FC _ Average ε,δ ( ⃗ Π , ξ ) (i.e., each Π j is treated like a vector in R d 2 ).

Output ˜ Π = argmin Π ∈W d,k ∥ ∥ ∥ Π -̂ Π ∥ ∥ ∥ F .

Algorithm C.3 (Algorithm SS _ Agg ) .

Input: A dataset (Π 1 , . . . , Π t ) ∈ ( W d,k )

t .

Utility Parameters:

k, q, ξ .

Privacy parameters:

ε, δ ≤ 1 .

Operation:

Sample p 1 , . . . , p q ∼ N ( ⃗ 0 , I d × d ) (i.i.d. samples from a standard spherical Gaussian).

For j ∈ [ t ] , compute y j = (Π j p 1 , . . . , Π j p q ) ∈ R qd , and let Y = ( y 1 , . . . , y t ) .

Compute z = ( z 1 , . . . , z qd ) ∼ FC _ Average ε,δ ( Y, ξ ) ( z ∈ R qd ).

Let ˜ P be the q × d matrix whose i th row (for i ∈ [ q ] ) is ( z d ( i -1)+1 , . . . , z di ) (which estimates the projection of p i onto the topk rows subspace of X ).

Output the projection matrix ˜ Π of the topk rows subspace of ˜ P .

C.1.1 Running Time

We analyze the running time of EstSubspace SS _ Agg k,q,ξ k,t . Denote by T ( n, d, k ) the running time of computing a projection matrix to the topk row subspace of an n × d matrix. The running time of Step 2 in EstSubspace is t · T ( n/t, d, k ) . The running time of SS _ Agg is O ( dqtk ) on Step 2, O ( dqt log t ) on Step 3 (Fact B.26), and T ( q, d, k ) on Step 5. Overall it is t · T ( n/t, d, k )+ T ( q, d, k )+ O ( dqt (log t + k )) . For both our weak and strong estimators (described next) we use n/t = ˜ Θ( k ) and q = ˜ O ( k ) , and therefore we obtain that the total running time is n m · T ( m,d,k ) + ˜ O ( dkn ) for m = n/t = ˜ Θ( k ) .

C.1.2 Key Property

In order to claim that the (non-private) projection matrices are close to each other, we use the following lemma which states that with high enough probability over a random subset, the topk projection matrix in the subset is close to the topk projection matrix of the entire matrix.

Lemma C.4 (Restatement of Lemma 2.1) . Let X = ( x 1 , . . . , x n ) ∈ ( S d ) n with singular values σ 1 ≥ . . . ≥ σ min { n,d } ≥ 0 and σ 2 k ≥ 0 . 01 n/k . Let X ′ ∈ ( S d ) m be a uniformly random m -size

subset of the rows of X (without replacement). Let Π and Π ′ be the projection matrices to the topk rows subspace of X and X ′ , respectively. Then the following holds for γ 1 = σ k +1 σ k and

Proof. We will prove each part of the lemma by applying Proposition B.12. In the following, let X = ∑ n i =1 σ i u i v T i be the SVD of X , and note that we can write X ′ = ( x i 1 , . . . , x i m ) where { i 1 , . . . , i m } is a random subset of [ n ] (without replacement). Let Y = ( y 1 , . . . , y n ) = ∑ k i =1 σ i u i v T i and let Z = ( z 1 , . . . , z n ) = ∑ min { n,d } i = k +1 σ i u i v T i , and note that Span { y 1 , . . . , y n } = Span { v 1 , . . . , v k } is orthogonal to Span { z 1 , . . . , z n } = Span { v k +1 , . . . , v min { n,d } } . Furthermore, define the random matrices Y ′ = ( y i 1 , . . . , y i m ) and Z ′ = ( z i 1 , . . . , z i m ) and note that X ′ = Y ′ + Z ′ .

First, by Proposition B.4 (applied on v 1 , . . . , v k ) and the assumptions on m,σ k , it holds by the union bound that

In the following we assume that the event in Equation (1) occurs. We first prove Item 1. Note that ∥ Z ′ ∥ ≤ ∥ Z ∥ ≤ σ k +1 ≤ γ 1 σ k and that σ k ( Y ′ ) 2 ≥ m 2 n σ 2 k ≥ 2 γ 2 1 σ 2 k ≥ 2 ∥ Z ′ ∥ 2 (the second inequality holds since m ≥ 4 γ 2 1 n ). By applying Proposition B.12(1) on X ′ , Y ′ , Z ′ we conclude that

We next focus on proving Item 2. Note that ∥ Z ∥ 2 F = ∑ min { n,d } i = k +1 σ 2 i = γ 2 2 σ 2 k , and that E [ ∥ Z ′ ∥ 2 F ] = m n ∥ Z ∥ 2 F = m n · γ 2 2 σ 2 k . Therefore by Markov's inequality

In the following we assume that the event in Equation (2) occurs. Note that σ k ( Y ′ ) 2 ≥ m 2 n σ 2 k ≥ 2 m βn γ 2 2 σ 2 k ≥ 2 ∥ Z ′ ∥ 2 (the second inequality holds since β ≥ 4 γ 2 2 ). By applying Proposition B.12(2) on X ′ , Y ′ , Z ′ we conclude that

C.2 Weak Estimator

In this section, we prove Theorem 1.6, stated below.

Theorem C.5 (Restatement of Theorem 1.6) . Let n, k, d ∈ N , λ > 0 , ε, δ ∈ (0 , 1] where k ≤ min { n, d } . There exists an ( k, λ, β = 0 . 9 , γ max = Ω(min { 1 λ , 1 } )) -weak subspace estimator M : ( S d ) n × [0 , 1] → R d × d with

such that M ( · , γ ) is ( ε, δ ) -DP for every γ ∈ [0 , 1] .

Theorem C.5 is an immediate corollary of the following Lemmas C.6 and C.7.

Lemma C.6. Let t = c 1 · ( log(1 /δ ) ε + d √ log(1 /δ ) log(20) λε ) (where c 1 is the hidden constant in Fact B.26). Then for any n ≥ 800 k ln(25 k ) · t , the mechanism M : ( S d ) n × [0 , 1] →W d,k defined by M ( X,γ ) := EstSubspace Naive _ Agg ε,δ,ξ =60 γ k,t ( X ) is an ( k, λ, β = 0 . 9 , γ max = 1 20 ) -weak-subspaceestimator.

Proof. Let X ∈ ( S d ) n with √ ∑ min { n,d } i = k +1 σ i ( X ) 2 σ k ( X ) 2 ≤ γ ≤ γ max and let Π ∈ W d,k be the projection of the topk rows subspace of X . Consider a random execution of M ( X ) . Let { Π j } t j =1 , ̂ Π be (random variables of) the values of { Π j } t j =1 , ̂ Π in the execution, and let ˜ Π be the output. By Lemma C.4(2) (recall that γ max ≤ 1 20 ) and the union bound,

Let a j = 1 {∥ Π -Π j ∥ ≤ 60 γ } (indicator random variable) and let a = ∑ t j =1 a j . By Equation (3) it holds that E[ a ] ≥ 0 . 99 t , and recall that a ≤ t . It follows that

In the following we assume that the event a ≥ 0 . 8 t occurs. Let J = { j ∈ [ t ] : a j = 1 } . Note that our choice of t satisfies

where c ′ denotes the constant from Fact B.26. Therefore we conclude by Fact B.26 (FriendlyCore averaging) that

The proof of the lemma now follows by Proposition B.16 since ∥ ∥ ∥ Π -˜ Π ∥ ∥ ∥ F ≤ 2 ∥ ∥ ∥ Π -̂ Π ∥ ∥ ∥ F . □

Lemma C.7. Let c 1 , c 2 , c 3 be the constants from Facts B.14 to B.15 (respectively), and let c be a large enough constant. Let t = c ·   log(1 /δ ) ε + ( √ k + √ log dk log(1 /δ ) λε ) √ kd log(1 /δ ) λε   ,

η = c 1 · ( √ k + √ ln( qt ) ) and q = c 2 · k . Then for any n ≥ 800 k ln(25 k ) · t , the mechanism M : ( S d ) n × [0 , 1] → W d,k defined by M ( X,γ ) := EstSubspace SS _ Agg ε,δ,k,q,ξ =120 η √ kγ k,t ( X ) is an ( k, λ, β = 0 . 9 , γ max = Θ(min { 1 λ , 1 } )) -weak-subspace-estimator.

Proof. Let X ∈ ( S d ) n with √ ∑ min { n,d } i = k +1 σ i ( X ) 2 σ k ( X ) 2 ≤ γ ≤ γ max and let Π ∈ W d,k be the projection of the topk rows subspace of X . Consider a random execution of M ( X ) . Let { Π j } t j =1 , { p i } q i =1 , { y j } t j =1 , z , ˜ P be (random variables of) the values of { Π j } t j =1 , { p i } q i =1 , { y j } t j =1 , z, ˜ P in the execution, and let ˜ Π be the output. As in the proof of Lemma C.6, let a = ∑ t j =1 a j where a j = 1 {∥ Π -Π j ∥ ≤ 60 γ } . Then Equation (4) imples that

In the following we assume that the event a ≥ 0 . 8 t occurs. Let J = { j ∈ [ t ] : a j = 1 } . By Fact B.14 and the definition of η we obtain that

In the following we assume that the above event occurs. This yields that

Furthermore, by the definition of t and η (note that η depends on log( t ) ), it holds that

where c ′ denotes the constant from Fact B.26. Therefore we obtain by Fact B.26 (FriendlyCore averaging) that

where P is the q × d matrix whose rows are Π p 1 , . . . , Π p q . By Fact B.15 we have that

and in the following we assume that the above event occurs (which in particular implies that Span ( P T ) = Span (Π) ). Finally, since 2 α ≤ σ k ( P ) by assumption (and assuming γ max ≤ 6 c 3 λ ), we conclude by Proposition B.13 that

We therefore conclude the proof of the lemma by Proposition B.16.

C.3 Strong Estimator

In this section, we prove Theorem 1.7, stated below.

Theorem C.8 (Restatement of Theorem 1.7) . Let n, k, d ∈ N , λ > 0 , such that k ≤ min { n, d } . There exists an ( k, λ, β = 0 . 8 , γ max ) -weak subspace estimator M : ( S d ) n × [0 , 1] → R d × d with

and

such that M ( · , γ ) is ( ε, δ ) -DP for every γ ∈ [0 , 1] .

LemmaC.9. Let c 1 , c 2 , c 3 be the constants from Facts B.14 to B.15 (respectively), and let c be a large enough constant. Let t = c · ( log(1 /δ ) ε + ( k +log ( dk log(1 /δ ) λε )) dk log(1 /δ ) λ 2 ε 2 ) , η = c 1 · ( √ k + √ ln( qt ) ) and q = c 2 · k . Then for any n ≥ 800 k ln(25 k ) · t , the mechanism M : ( S d ) n × [0 , 1] →W d,k defined by M ( X,γ ) := EstSubspace SS _ Agg ε,δ,k,q,ξ =8 √ 2 tkηγ k,t ( X ) is an ( k, λ, β = 0 . 8 , γ max = min { 1 2 t , 6 c 3 λ } ) -strong-subspace-estimator.

Proof. Fix X ∈ ( S d ) n with σ k +1 ( X ) σ k ( X ) ≤ γ ≤ γ max and let Π ∈ W d,k be the projection of the topk rows subspace of X . Consider a random execution of M ( X ) . Let { Π j } t j =1 , { p i } q i =1 , { y j } t j =1 , z , ˜ P

□

be (random variables of) the values of { Π j } t j =1 , { p i } q i =1 , { y j } t j =1 , z, ˜ P in the execution, and let ˜ Π be the output. By Lemma C.4(1) (recall that γ max ≤ 1 2 t ) and the union bound,

Let a j = 1 {∥ Π -Π j ∥ ≤ 4 √ 2 tγ } (indicator random variable) and let a = ∑ t j =1 a j . As in Equation (4), the above yields that

In the following we assume that the event a ≥ 0 . 8 t occurs. Let J = { j ∈ [ t ] : a j = 1 } . By Fact B.14 and the definition of η we obtain that

In the following we assume that the above event occurs. This yields that

Furthermore, by the definition of t and η , it holds that

where c ′ denotes the constant from Fact B.26. Therefore we obtain by Fact B.26 (FriendlyCore averaging) that

where P is the q × d matrix whose rows are Π p 1 , . . . , Π p q . By Fact B.15 we have that

and in the following we assume that the above event occurs (which in particular implies that Span ( P T ) = Span (Π) ). Finally, since 2 α ≤ σ k ( P ) by assumption (and using γ max = 6 c 3 λ ), we conclude by Proposition B.13 that

We therefore conclude the proof of the lemma by Proposition B.16. □

D Lower Bounds

In this section, we prove our lower bounds. In Appendix D.1 we prove Theorem 1.8 (lower bound for weak estimators) and in Appendix D.2 we prove Theorem 1.9 (lower bound for strong estimators). Both lower bounds rely on the framework of Peter et al. [2024], described in Appendix B.8.5.

Throughout this section, recall that for d, k ∈ N we denote by W d,k the set of all d × d projection matrices of rank k , and denote by P d the set of all d × d permutation matrices.

D.1 Weak Estimators

Theorem D.1 (Restatement of Theorem 1.8) . Let n, k, d ∈ N , λ ≥ 1 , β ∈ (0 , 1] such that d ≥ ck and λ 2 ≤ d ck log k for large enough constant c > 0 , and n is a multiple of k . If M : ( S d ) n × [0 , 1] →W d,k is a ( k, λ, β, γ max = 1 10 6 λ 2 ) -weak subspace estimator and M ( · , γ ) is ( 1 , β 5 nk ) -DP for every γ ∈ [0 , 1] , then n ≥ Ω ( √ kd/λ log 1 . 5 ( dk λβ ) ) .

Theorem 1.8 is an immediate corollary of Lemma B.34 (Framework for lower bounds) and the following Lemma D.2.

Lemma D.2. Let n, k, d, λ, β and M as in Theorem D.1. Let α = 1 5 · 10 6 λ 2 k , n 0 = n/k , ℓ = 2 · ⌈ 1 4 (1 -α ) d ⌉ , d 0 = d -2 ℓ , X = S d , Z = [ k ] × ( P d ) k and W = W d,k . Let G : {-1 , 1 } n 0 × d 0 × Z → X n be Algorithm D.3, and let F : Z × W → [ -1 , 1] d 0 be Algorithm D.4. Then the triplet ( M , F , G ) is 0 . 8 β k -leaking (Definition B.33).

Note that by Lemmas D.2 and B.34, we obtain that n 0 ≥ Ω ( √ d 0 log 1 . 5 ( d 0 /β ) ) , but since n 0 = n/k and d 0 = Θ( αd ) = Θ ( d λ 2 k ) , the proof of Theorem D.1 follows. We next prove Lemma D.2.

Algorithm D.3 (Algorithm G ) .

Parameters: n 0 , n, d 0 , d, ℓ ∈ N .

Inputs: z = ( s, ( P 1 , . . . , P k )) for s ∈ [ k ] and P 1 , . . . , P k ∈ P d , and a matrix X ∈ {-1 , 1 } n 0 × d 0 . Operation:

Sample A = ( A 1 , . . . , A k ) ∼ D ( n 0 , d 0 ) k , and set A s = X .

For t ∈ [ k ] , compute B t = PAP n 0 ,d 0 ,ℓ ( A t , P t ) ∈ {-1 , 1 } n 0 × d (Definition B.35), and let B ∈ {-1 , 1 } n × d be the vertical concatenation of B 1 , . . . , B k .

Output Y = 1 √ d B ∈ ( S d ) n .

Algorithm D.4 (Algorithm F ) .

Parameters: n 0 , n, d 0 , d, ℓ ∈ N .

Inputs: z = ( s, ( P 1 , . . . , P k )) for s ∈ [ k ] and P 1 , . . . , P k ∈ P d , and a rankk projection matrix ˜ Π ∈ W (which is the output of M ( Y, γ = 1 1000 λ ) ).

Operation:

Compute a vector u = ( u 1 , . . . , u d ) ∈ Span ( ˜ Π · P T s ) that maximizes min { ∑ d 0 + ℓ/ 2 j = d 0 +1 sign ( u j ) , -∑ d 0 +3 ℓ/ 2 j = d 0 + ℓ +1 sign ( u j ) } .

Output q = sign ( u ) 1 ,...,d 0 ∈ {-1 , 1 } d 0 .

D.1.1 Proving Lemma D.2

In the following, we define random variables X ∼ D ( n 0 , d 0 ) (Definition B.30) and z = ( s , ( P 1 , . . . , P k )) ← Z , and consider a random execution of A M , F , G ( X ) = F ( z , M ( G ( X , z ))) . Let B 1 , . . . , B k , B , Y be the values of B 1 , . . . , B k ∈ {-1 , 1 } n 0 × d , B ∈ {-1 , 1 } n × d and Y ∈ ( S d ) n in the execution of G , and let Y 1 = 1 √ d B 1 , . . . , Y k = 1 √ d B k (note that Y is a vertical concatenation of Y 1 , . . . , Y k ). Let u be the value of u ∈ S d in the execution of F . For b ∈ {-1 , 1 } let F b be the set of b -marked columns of B s · P T s (note that F 1 includes d 0 +1 , . . . , d 0 + ℓ and F -1 includes d 0 + ℓ +1 , . . . , d ). Let H 1 = { d 0 +1 , . . . , d 0 + ℓ/ 2 } ⊆ F 1 and H -1 = { d 0 + ℓ +1 , . . . , d 0 +3 ℓ/ 2 } ⊆ F -1 , and let H = H 1 ∪ H -1 . For t ∈ [ k ] , define

The following claim holds under our assumption that λ 2 ≤ d ck log k for large enough constant c .

Claim D.5. Let γ = 1 1000 λ . It holds that

Proof. Recall that d = d 0 +2 ℓ and ℓ ≥ 1 2 (1 -α ) d for α = 1 5 · 10 6 λ 2 k . First, note that

where the last inequality holds since ℓ ≈ d/ 2 ≥ c · k/ 2 for large enough constant c > 0 . Furthermore, note that for every s, t ∈ [ k ] , 2 ℓ · ⟨ v s , v t ⟩ = ∑ j : sign ( v j s )=1 sign ( v j t ) -∑ j : sign ( v j s )= -1 sign ( v j t ) where each sum has Hypergeometric distributions HG 2 ℓ, 0 ,ℓ (Definition B.5). Therefore by Fact B.6 and the union bound, it holds that

(i.e., v 1 , . . . , v k are almost orthogonal).

In the following, we assume that the events in Equations (8) and (9) occur. Using the Gram-Schmidt process on v 1 , . . . , v k , we obtain orthogonal basis u 1 , . . . , u k to Span { v 1 , . . . , v k } such that for every t ∈ [ k ] , u t = v t + λ t -1 v t -1 + w t , where | λ t -1 | ≤ O (√ log k d ) and ∥ w t ∥ 2 ≤ O ( log k d ) (holds by Proposition B.17). Recall that Y is a vertical concatenation of 1 √ d B 1 , . . . , 1 √ d B k and for every t ∈ [ k ] , the rows of B t are all in

Therefore, we obtain that

where the last inequality holds whenever α ≥ Θ(log k/d ) , which holds by the assumption on λ . We therefore obtain that σ 2 1 ( Y ) , . . . , σ 2 k ( Y ) ≥ (1 -4 α ) · n 0 which yields that ∑ n i = k +1 σ 2 i ( Y ) ≤ n -k (1 -4 α ) · n 0 = 4 αn . Hence

where the second inequality holds since α = 1 5 · 10 6 λ 2 k .

The following claim holds under our assumption that d ≥ ck for large enough constant c > 0 . Claim D.6. It holds that

where 'strongly-agrees" is according to Definition B.36.

Proof. In the following we assume that the 0 . 9 probability event in Claim D.5 occurs. Since M is ( k, λ, β, γ max = 1 1000 λ ) -subspace estimator, it follows from Equation (11) that w.p. β , the output ˜ Π of M ( Y ) satisfy

□

where we denote by Π the projection matrix onto Span { v 1 , . . . , v k } (defined in Equation (7)). In the following, we assume that the event in (12) occurs. This yields that there must exists t ∈ [ k ] such that

Since s (part of z ) is chosen at random and does not change the distribution of Y (the input of the mechanism), w.p. 1 /k the above holds for t = s , i.e.,

In the following we assume that the 1 /k -probability event in Equation (13) occurs.

Recall that B s = √ d · Y s , and let ˜ Π s = ˜ Π · P T s and B ′ s = B s · P T s . It follows that

In the following, define

Note that each row i of B ′ s can be written as v + ξ i where ξ i ∈ {-1 , 0 , 1 } d 0 × (0 , . . . , 0 ︸ ︷︷ ︸ 2 ℓ ) . This yields that

Now, since

we conclude that for ˜ v T = ˜ Π t v T ∈ Span ( ˜ Π t ) :

We next define ( I , η ) -good vectors.

Definition D.7. We say that a vector w ∈ {-1 , 1 } d is ( I , η ) -good iff for (1 -η ) fraction of the indices j ∈ I it holds that sign ( w j ) = sign ( v j ) (for the v defined in Equation (15) ).

We use the following trivial fact:

Observation D.8. If w is not ( I , η ) -good, then ∥ w -v ∥ 2 2 ≥ η |I| .

Since for both b ∈ {-1 , 1 } , |H b | ≥ 1 4 (1 -α ) d ≥ d/ 5 and ∥ v -˜ v ∥ 2 2 ≤ d 500 , Observation D.8 implies that ˜ v is ( H 1 , η ) -good and ( H -1 , η ) -good for η = 1 100 . Therefore, the vector u (computed in F ) is also ( H 1 , η ) -good and ( H -1 , η ) -good.

In the following, we use similar arguments to Dwork et al. [2014] for claiming that because u is good on half of the padding location, then it should also be good on the rest of the marked locations.

From the point of view of the algorithm M (which does not know P s ), the locations in H b are indistinguishable from those in F b \ H b . Therefore, for any point that is not ( F b , 3 η ) -good, the probability (taken over the random choice of P s ) that it is ( H b , 2 η ) -good is at most exp( -Ω( η 2 d )) . Now let N be an ( η √ d ) -net of the √ d -sphere in Span ( ˜ Π s ) (Definition B.9). Taking a union bound over a exp( O ( k log(1 /η ))) = exp( O ( k )) points in N (Fact B.10), and recall that d ≥ ck for large enough constant c , we conclude that except with probability exp( O ( k )) · exp( -Ω( η 2 d )) ≤ 0 . 01 , any given vector in N that is ( H b , 2 η ) -good is also ( F b \ H b , 4 η ) -good. Since u is ( H b , η ) -good, then its nearest net point u ′ is ( H b , 2 η ) -good. Thus u ′ is ( F b \ H b , 4 η ) -good which implies that u is ( F b \ H b , 5 η ) -good except w.p. exp( -Ω( η 2 d )) ≤ 0 . 01 . But by definition of v , it perfectly agrees with the marked columns of B s · P T s . Since 5 γ < 0 . 1 the above implies that sign ( u ) F b \H b strongly-agrees (Definition B.36) with the matrix ( B s · P T s ) F b \H b which implies that sign ( u ) [ d ] \H b strongly-agrees with the matrix ( B s · P T s ) [ d ] \H b , as required.

□

We now ready to prove the final claim that concludes the proof of Lemma D.2.

Proof. In the following we assume that the event from the statement of Claim D.6 occurs, and let H = H 1 ∪ H -1 . Define the permutation matrix P ′ ∈ P d 0 + ℓ that is obtained by removing the rows H and the columns P s ( H ) from P s (i.e., P ′ is the permutation induced by P s between [ d ] \ H and [ d ] \ P s ( H ) ). Similarly, define the permutation matrix P ′ ∈ P ℓ that is obtained by removing the rows H = [ d ] \ H and the columns P s ( H ) from P s (i.e., P ′ is the permutation induced by P s between H and P s ( H ) ). Note that P ′ is distributed uniformly over P d 0 + ℓ for any choice of P ′ . In the following, let ˜ Π ′ = ˜ Π [ d ] \H , q ′ = sign ( u ) [ d ] \H · P ′ , and B ′ = B [ d ] \H s . By Claim D.6 it holds that

But note that B ′ = PAP n 0 ,d 0 ,ℓ/ 2 ( X , P ′ ) and also note that q ′ is just a function of ˜ Π ′ and P ′ (i.e., independent of P ′ ) since it equals to sign ( w ) P ′ ([ d ] \H ) where w is the vector in Span ( ˜ Π ) that maximizes min { ∑ j ∈ P ′ ( H 1 ) sign ( w j ) , -∑ j ∈ P ′ ( H -1 ) sign ( w j ) } . Furthermore, note that ( q ′ · ( P ′ ) T ) 1 ,...,d 0 = q , where q is the final output of A M , F , G ( X ) = F ( z , M ( G ( X , z ))) . Thus by Lemma B.37 and Equation (17) we conclude that

D.2 Strong Estimators

Theorem D.10 (Restatement of Theorem 1.9) . Let n, k, d ∈ N , λ ≥ 1 , β ∈ (0 , 1] such that d ≥ ck and λ 2 ≤ d c log k for large enough constant c > 0 , and n is a multiple of k . If M : ( S d ) n × [0 , 1] → R d × d is an ( k, λ, β, γ max = 1 10 6 λ 2 ) -strong subspace estimator and M ( · , γ ) is ( 1 , β 5 nk ) -DP for every γ ∈ [0 , 1] , then n ≥ Ω ( k √ d/λ log 1 . 5 ( dk λβ ) √ log( dk ) log(2 n/k ) ) .

We prove Theorem D.10 using a similar technical lemma to Lemma D.2, but now since M is a strong subspace estimator, we can use α = ˜ Θ ( 1 λ 2 ) (rather than Θ ( 1 λ 2 k ) as in Lemma D.2).

Lemma D.11. There exists large enough constant c > 0 such that the following holds: Let n, k, d, λ, β and M as in Theorem D.10, let α = 1 c · log( dk ) log(2 n/k ) · λ 2 , n 0 = n/k , ℓ = 2 · ⌈ 1 4 (1 -α ) d ⌉ , d 0 = d -2 ℓ , X = S d , Z = [ k ] × ( P d ) k and W = W d,k . Let G : {-1 , 1 } n 0 × d 0 × Z → X n be Algorithm D.3, and let F : Z × W → [ -1 , 1] d 0 be Algorithm D.4. Then the triplet ( M , F , G ) is 0 . 8 β k -leaking (Definition B.33).

By Lemmas D.11 and B.34, it holds that n 0 ≥ Ω ( √ d 0 log 1 . 5 ( d 0 /β ) ) . The proof of Theorem D.10 now follows since n 0 = n/k and d 0 ≈ αd for the α defined in the lemma.

D.2.1 Proving Lemma D.11

As in the proof of Lemma D.2, we define random variables X ∼ D ( n 0 , d 0 ) (Definition B.30) and z = ( s , ( P 1 , . . . , P k )) ←Z , and consider a random execution of A M , F , G ( X ) = F ( z , M ( G ( X , z ))) . Let { A t } , { B t } , Y be the values of { A t } , { B t } , Y ∈ ( S d ) n in the execution of G , and recall that Y is a vertical concatenation of Y 1 , . . . , Y k where Y t = 1 √ d B t .

The only difference from proving Lemma D.2 is to prove a different version of Claim D.5 that only considers the gap between σ k and σ k +1 (which will meet the requirements of the strong estimator M ). Namely, is suffices to prove the following claim:

Claim D.12. It holds that for γ = 1 1000 λ .

Proof of Claim D.12. As in Equation (7), for t ∈ [ k ] we define

Let E = Span { v 1 , . . . , v k } . Note that by construction, { v t } , E are independent of { A t } . Similarly to Equation (9) it holds that

This yields (using similar steps as in the proof of Claim D.5) that w.p. 0 . 98 , dim( E ) = k and every unit vector u ∈ E has ∥ Yu ∥ 2 2 ≥ (1 -4 α ) n k . This in particular implies that σ 2 k ( Y ) ≥ (1 -4 α ) n k .

Our goal is to prove that w.p. 0 . 99 it also holds that σ 2 k +1 ( Y ) ≤ ˜ O ( α ) n k . Let ¯ E be the orthogonal subspace to E . Our goal is reduced to showing that there exists a constant c such that

Given that the event in Equation (20) holds we conclude that σ 2 k +1 ( Y ) ≤ α · c log( k/α ) log(2 n/k ) · n k and hence

where the last inequality holds by taking

Note that for every u ∈ ¯ E ∩ S d is holds that

where the last equality holds since u is orthogonal to v 1 , . . . , v k . Therefore, we can prove Equation (20) by proving that there exists a constant c ′ such that

Given that Equation (21) holds, we prove the claim using a net argument. By taking an exp( d ln(3 k/α )) -size √ α/k -net of S d (Fact B.10), Equation (20) follows by Equation (21) and the union bound over all the net points, which concludes the proof of the claim.

In the following we focus on proving Equation (21). Fix a columns vector u ∈ S d . Recall that A t ∼ D ( n 0 , d 0 ) (Definition B.30), and A t is located in d 0 random columns out of d , which are the columns J = P t ([ d 0 ]) . Let a t,i ∈ {± 1 √ d } d 0 be the i th row of A t . By Definition B.30, the coordinates of a t,i are i.i.d. Bernoulli distribution over {-1 , 1 } , each takes 1 w.p. 1 / 2 and -1 w.p. 1 / 2 . Therefore by Fact B.2 it holds that

where the first inequality holds by Fact B.2, the second one holds by Jensen's inequality (and since the function f ( x ) = e -1 /x is concave), and the last one hold since E [ ∥ u J ∥ 2 2 ] = d 0 d ≤ α . By the union bound over the n/k rows of A t we obtain that

or equivalently

In the following, let b t = ∥ Y t u ∥ 2 2 , and define b ′ t = b t -E[ b t ] , and µ = 2 αn ln(2 n/k ) dk . First, note that

In addition, it holds that

Namely, b ′ t is a Sub-Exponential random variable (Definition B.7) with ∥ b ′ t ∥ ψ 1 ≤ 8 µ . Since { b ′ t } are independent, each has zero mean, we obtain by Fact B.8 that

We now take ξ = c · µd ln( k/α ) ≥ cα · log( k/α ) log(2 n/k ) · n k for large enough constant c . Recall that by our assumption on d it holds that ξ ≥ 4 kµ . Hence

where the second inequality holds assuming that c is large enough. This concludes the proof of Equation (21) and therefore the proof of the lemma. □

E Broader Impact

In this work we develop algorithms that maintain the differential privacy of the input points. When the points represent individuals, our work helps to maintain the privacy of those individuals.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Upper bounds, lower bounds, and experiments are provided in Appendices C and D and Section 4 (respectively).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations in Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Full proofs are provided in Appendices C and D.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: This information is disclosed in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The Python code is provided in the supplementary material along with a README file.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the details are given in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We mentioned in Section 4 that we perform 30 repetitions for generating each graph point which represents the trimmed average of values between the 0 . 1 and 0 . 9 quantiles.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We mentioned in Section 4 that all experiments were tested on a MacBook Pro Laptop with 8-core Apple M1 CPU with 16GB RAM.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We only perform experiments on synthetic datasets that do not cause any harm. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We mentioned in Section 4 that we use the Python implementation of Tsfadia et al. [2022] that is publicly available in https://media.icml.cc/Conferences/ ICML2022/supplementary/tsfadia22a-supp.zip and we cite them as required.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper tackles the challenge of high costs in private data analysis due to the curse of dimensionality, despite many datasets having an underlying low-dimensional structure. It builds on prior work by introducing measures based on multiplicative singular-value gaps to quantify how "easy" a dataset is for private subspace estimation. The authors provide new bounds and a practical algorithm that can estimate subspaces with a number of points independent of the dimension, showing improved performance in high-dimensional settings compared to previous approaches.

STRENGTHS:
1.The problem of the paper is well-motivated and the authors give both upper bounds and lower bounds. 
2.They also provide experimental results.
3. The writing is clear.
4. The literature review is relatively comprehensive.

WEAKNESSES:
There are some gaps between their upper bounds and lower bounds.

QUESTIONS:
1. Can your approaches work on real-world datasets?
2. Can you further improve the gap between upper bounds and lower bounds?

LIMITATIONS:
There are some gaps between their upper bounds and lower bounds.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of private subspace learning without assuming that the dataset follows a Gaussian distribution. They propose new measure on the hardness of subspace learning problem based on the ratio between the $k^{th}$ and the remaining eigenvalues, or between the $k^{th}$ and the $(k+1)^{th}$ eigenvalues. They derive upper and lower bounds on the utility guarantees for private subspace estimation for datasets that meet these criteria, showing that the bounds can be dimension-independent under suitable conditions. However, these conditions depend on some additional parameters, e.g. $\\lambda$, that might need to be carefully selected. Also, the paper presents an implementable algorithm that achieves the derived upper bounds.\n\nSTRENGTHS:\n- The paper contributed to remove a restrictive assumption in previous works. Previous research on private subspace estimation has derived dimension-independent sample complexity for Gaussian-distributed data with large multiplicative eigen-gaps. This paper removes the Gaussianity assumption, and proves that dimension-independent sample complexity can still be achieved for easier datasets.\n- The proposed measure of \"easiness\" of the datasets for subspace estimation, based on multiplicative eigen-gap, is simple and intuitive. Their derived bounds on sample complexity require prior knowledge of this measure, however, they show that the bounds are hardly affected when this measure is agnostic. \n- The paper has practical relevance due to its realistic assumptions and the inclusion of an implementable algorithm for private subspace estimation.\n\nWEAKNESSES:\nThe description of some parameters is vague, which hinders the understanding of the significance of the sample complexity bounds. For example, while $\\lambda$ is an important parameter that appears in the sample complexity bounds, it is not defined in Definition 1.3 or in the text above it.\n\nQUESTIONS:\nCan you explain the parameter $\\lambda$? Is it a parameter that can be freely selected, or is it intrinsically determined by the property of the dataset or the algorithm?\n\nLIMITATIONS:\nI don't see any significant limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the problem of differentially private dimension reduction without dependence on the ambient dimension d. While this is known to be impossible generally, this work gives (distinct) necessary and sufficient conditions on gaps between the kth singular value of the data matrix and its subsequent singular values for privately identifying a k-dimensional subspace that captures the data with minimal distortion. The theoretical results are empirically verified, demonstrating that for datasets that are very close to a k-dimensional subspace, for small k, that the algorithms demonstrating sample upper-bounds do outperform existing, general DP subspace estimation techniques.\n\nSTRENGTHS:\nThis work makes significant progress on an interesting problem, and the contributions could potentially have applications to notably improving efficiency of the broadly useful DP-SGD algorithm (in cases where gradients approximately lie in low-dimensional subspaces).\n\nWEAKNESSES:\nThe presentation of this work could be improved. The reader's first introduction to what gamma-easiness means and how it eliminates dimension dependence comes in a footnote in page 2. I think that given the centrality of easy instances to this work, it would be good to dedicate a few sentences early on to give the reader a more precise understanding of what gamma-easiness means generally and how we should think of how gamma will relate to other parameters of a problem.\n\nI also found the lower-bounds overview exceptionally hard to follow. These results are very technical and make use of prior work on fingerprinting codes and DP, so it does seem challenging to give an overview that is both sufficiently high level for the page limits and also illuminating, but I think it should be possible to make the overview more modular, and in that way easier to follow. \n\nTypos/Suggested edits:\n\nSection 1.5.2\n\n“combination between generating” -> “combination of generating”\n\n“However, their result strongly rely” -> “However, their results strongly rely”\n\n“top-singular vector” -> “top-singular vectors”\n\n1.5.3\n\n“which have the property that we seek for” -> “which have the property we seek”\n\n1.7\n\n“Additional related work appear at Appendix A. Notations, …” -> “Additional related work appears in Appendix A. Notation, …”\n\n2\n\n“that arbitrary partition” -> “that arbitrary partitions”\n\n“expected Forbenius”\n\n“The first one simply treat each matrix” -> “treats”\n\n“who privately estimate” -> “which privately estimates”\n\n“asymptotical” -> “asymptotic”\n\nFootnote 4 on page 6 defines the Frobenius norm, but it’s already been referred to in Definition 1.2, so could be good to move it earlier, also Frobenius is mispelled in the footnote.\n\n3\n\nParagraph 4 of Section 3, second line, I think sigma_i should be sigma_i^2\n\n4\n\n“that are approximately lie” -> “that approximately lie”\n\n5\n\n“Yet, it still left open to close the gap” -> “Yet, closing the gap is still left open”\n\n“Forbenius”\n\n“via a private subspace estimation” -> “via private subspace estimation”\n\nAppendix B\n\nProof of proposition B.12 “$Pi_1, Pi_2$ is holds” -> “it holds”\n\nAppendix C.1\n\n“that uses an oracle access” -> “that uses oracle access”\n\n“we obtain that the projection matrices” -> “the projection matrices”\n\n“simply treat the matrices … and compute” -> “simply treats … and computes”\n\nAlgorithm C.1\n\n“Randomly split X intro t” -> “Randomly split X into t”\n\nQUESTIONS:\nThe experiments all involve very small choices of k. Does this reflect an appropriate choice of constant dimension for the intended applications, computational limits, or performance limitations?\n\nLIMITATIONS:\nYes, the authors have addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-14"}

comments: SUMMARY:
The paper studies private PCA under the assumption that the singular values of the data matrix shows multiplicative decay. It is a problem that was initiated by Steinke-Singhal and has not seen much improvement since that work. This works provides a more thorough study of this problem.

STRENGTHS:
PCA is one of the most important problem in modern data analysis and ML. Unfortunately, with privacy constraints, it requires a lot of data samples to perform PCA with any reasonable accuracy. This has led researchers to look at instances where PCA can be easy. Most of these works rely on the assumption that the singular values of the data matrix has a particular behavior. This work extends on those line of work. 

It provides nice upper bound on various assumptions regarding the multiplicative decay of the singular values. The upper bound result follows the framework presented in Steinke-Singhal. The lower bound to me are the more interesting one, which is based on recent ideas of Peter et al.

WEAKNESSES:
The lower and upper bound do not match. There is a large gap between the two. One major focus in the area of PCA is to get a bound under these assumptions such that when $k \to d$, we recover the optimal rate. For example, this was the motivation behind Mangroubi-Vishnoi. Unfortunately, under that assumption, the upper bound is way worse due to its dependence on $k^2 \sqrt{d}$ and $k^3d$, respectively. Due to that reason, I believe that the lower bound is tight.

QUESTIONS:
Do the authors perceive a way to improve their upper bound to get better dependence on $k$?

LIMITATIONS:
The quadratic and cubic dependence on $k$.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-17