Title: Private Geometric Median
Venue: NeurIPS 2024
Accepted: yes
Keywords: Differential Privacy

Differentially Private Convex Optimization

Geometric Median
Abstract: In this paper, we study differentially private (DP) algorithms for computing the geometric median (GM) of a dataset: Given $n$ points, $x_1,\dots,x_n$ in $\mathbb{R}^d$, the goal is to find a point $\theta$ that minimizes the sum of the Euclidean distances to these points, i.e., $\sum_{i=1}^{n} \lVert|\theta - x_i\rVert_2$. Off-the-shelf methods, such as DP-GD, require strong a priori knowledge locating the data within a ball of radius $R$, and the excess risk of the algorithm depends linearly on $R$. In this paper, we ask: can we design an efficient and private algorithm with an excess error guarantee that scales with the (unknown) radius containing the majority of the datapoints? Our main contribution is a pair of polynomial-time DP algorithms for the task of private GM with an excess error guarantee that scales with the effective diameter of the datapoints. Additionally, we propose an inefficient algorithm based on the inverse smooth sensitivity mechanism, which satisfies the more restrictive notion of pure DP. We complement our results with a lower bound and demonstrate the optimality of our polynomial-time algorithms in terms of sample complexity.

Private Geometric Median

Abstract

In this paper, we study differentially private (DP) algorithms for computing the geometric median (GM) of a dataset: Given n points, x 1 , . . . , x n in R d , the goal is to find a point θ that minimizes the sum of the Euclidean distances to these points, i.e., ∑ n i =1 ∥ θ -x i ∥ 2 . Off-the-shelf methods, such as DP-GD, require strong a priori knowledge locating the data within a ball of radius R , and the excess risk of the algorithm depends linearly on R . In this paper, we ask: can we design an efficient and private algorithm with an excess error guarantee that scales with the (unknown) radius containing the majority of the datapoints? Our main contribution is a pair of polynomial-time DP algorithms for the task of private GM with an excess error guarantee that scales with the effective diameter of the datapoints. Additionally, we propose an inefficient algorithm based on the inverse smooth sensitivity mechanism, which satisfies the more restrictive notion of pure DP. We complement our results with a lower bound and demonstrate the optimality of our polynomial-time algorithms in terms of sample complexity.

1 Introduction

Differentially private (DP) convex optimization is a fundamental task where we approximately minimize a data-dependent convex loss function while limiting what can be learned about individual data points. The predominant algorithm for DP convex optimization is DP (stochastic) gradient descent, or DP-(S)GD, for short [SCS13; BST14]. Given a dataset X ( n ) which contains private information, and a loss function F ( θ ; X ( n ) ) , DP-(S)GD starts with an initial value θ 0 ∈ R d and iteratively updates it using θ t +1 =Π Θ ( θ t -η · ( ∇ θ t F ( θ t ; X ( n ) )+ ξ t ) ) where η > 0 is the step size, ξ t is noise to ensure DP, Θ ⊆ R d is a closed convex feasible set, and Π Θ is the Euclidean projection operator. In the most general setting of Lipschitz convex functions, the excess error depends linearly on the radius of the set Θ , and this linear dependence is necessary in the worst-case [BST14]. This linear dependence is problematic because we can think of the diameter of the set Θ as capturing a measure of the uncertainty we have about the location of the minimizer, and we want our algorithm to perform well even with a high degree of uncertainty. This linear dependence can be improved under certain unrealistically strong assumptions, such as strong convexity, but it is unclear whether we can improve the dependence on the radius under weaker, more natural conditions. In this paper, as a step towards answering this question, we identify a simple, optimization task-computing the geometric median -where we can exponentially improve the dependence on the radius.

We study private algorithms for computing the geometric median (GM) of a dataset: We are given a set of n data points X ( n ) ≜ ( x 1 , . . . , x n ) ∈ ( R d ) n , where x i represents the private information of

one individual, and we are interested in approximately solving the following optimization problem:

The geometric median generalizes the standard one-dimensional median. The geometric median is a useful tool for robust estimation and aggregation, because it is less sensitive to outliers than the mean of the data, i.e., it is a nontrivial estimator even when ≤ 49% of the input data is arbitrarily corrupted. These properties make GM a popular tool for designing robust versions of distributed optimization methods [CSX17; WLCG20; FGGPS22; AHJSDT22; PKH22; EFGH23], boosting the confidence of weakly concentrated estimators [Min15], clustering [BMM03], etc.

Baseline for Private GM. Since the geometric median is the minimizer of a Lipschitz convex loss function, we can privately approximate it using the standard approach of DP-(S)GD. In particular, if we know a priori that all the data points lie in a known ball of radius R (without loss of generality this ball is centered at the origin, i.e., ∥ x i ∥ 2 ≤ R for every i ∈ [ n ] ), then DP-(S)GD guarantees ( ε, δ ) -DP with the following excess error [BST14]:

As discussed in the beginning of this section, this guarantee has a significant drawback: the excess error of the algorithm depends linearly on the radius R of the a priori bound on the data. This bound could be very loose; it does not scale with the data. Can we do better? What quantity should the excess error guarantee scale with?

It is known that the GM is inside the convex hull of the datapoints. However, this convex hull can have a very large diameter due to a small number of outliers , while most of the datapoints live in a ball with a small diameter. A key property of GM is robustness to outliers, so we want our accuracy guarantee to also be robust to some outliers. Specifically, if ≥ 51% of the points lie in a ball of diameter ∆ ≪ R then the geometric median is O (∆) far from that ball (see Lemma C.6 for a more precise statement). Thus, we aim to design a DP algorithm whose error is proportional to the actual scale of the majority of the data, rather than the a priori worst-case bound. However, the algorithm designer does not have a priori knowledge of the location or diameter of a ball that contains most of the data; the algorithm must discover this information from the data. This prompts the following question: Can we design an efficient and private algorithm with an excess error guarantee that scales with the radius that contains majority of the datapoints? Our results provide a positive answer.

1.1 Contributions

Our main contribution is a pair of polynomial-time DP algorithms for approximating the geometric median with an excess error guarantee that scales with the effective diameter of the datapoints. Also, the sample complexity and the runtime of our algorithms depend logarithmically on the a priori bound R . Both of our algorithms achieve the same excess error bounds up to logarithmic factors, but have incomparable running times. We also give a simple numerical experiment on synthetic data as a proof of concept that our algorithm improves over DP-(S)GD, as presdicted by the theory. In

terms of optimality, we show the our proposed algorithms is optimal in terms of sample complexity. Furthermore, we propose an algorithm based on the inverse smooth sensitivity mechanism for the private geometric median problem that satisfies the more restrictive notion of pure DP. Below, we give an overview of these algorithms and the techniques involved.

Polynomial-Time Algorithms. Both of our algorithms for the private geometric median problem are two-phase algorithms: in the first phase, which we refer to as warm-up , the algorithm shrinks the feasible set to a ball whose diameter is proportional to what we call the quantile radius in time that depends logarithmically on R . The second phase, which we call fine-tuning , uses the output of the warm-up algorithm to further improve the error.

First, we formalize the notion of the quantile radius as the radius of the smallest ball containing sufficiently many points.

Definition 1.1 (Quantile Radius) . Fix a dataset X ( n ) = ( x 1 , . . . , x n ) ∈ ( R d ) n and θ ∈ R d . For every γ ∈ [0 , 1] , define ∆ γn ( θ ) ≜ min { ∆ : | i ∈ [ n ] : ∥ x i -θ ∥ ≤ ∆ | ≥ γn } .

To motivate the idea behind our algorithms, assume the algorithm designer knew a ball, with center θ 0 and radius ˆ ∆ such that ∥ θ 0 -θ ⋆ ∥ ≤ O ( ˆ ∆) and ˆ ∆ = ˜ O (∆ 4 n/ 5 ( θ ⋆ )) . Then, running DP-(S)GD over this ball would give excess error O (∆ 4 n/ 5 ( θ ⋆ ) √ d/ε ) . This guarantee is particularly interesting as the excess error scales with the quantile radius and not the largest possible norm of any point. Also, by definition of the quantile radius and the geometric median loss function, we have that F ( θ ⋆ ; X ( n ) ) ≥ (1 -γ ) n ∆ γn ( θ ⋆ ) . This inequality shows that an algorithm whose excess error depends on ∆ γn ( θ ⋆ ) has a multiplicative guarantee rather than the standard additive guarantee for DP-(S)GD. This type of guarantee is particularly desirable for the geometric median since an algorithm with a multiplicative guarantee will be scale free and be adaptive to the niceness of the dataset. However, since we do not know such a pair θ 0 and ˆ ∆ a priori, the objective of the warm-up algorithm is to privately find these quantities.

The warm-up algorithm is based on the following structural result: given a point θ that satisfies ∥ θ -θ ⋆ ∥ ≳ ∆ 3 n/ 4 ( θ ⋆ ) , we have F ( θ ; X ( n ) ) -F ( θ ⋆ ; X ( n ) ) ≳ ∥ θ -θ ⋆ ∥ . (See Lemma 2.6 for a formal statement.) This result implies that, even though F ( θ ; X ( n ) ) is not a strongly convex function, we have a growth condition such that the excess error increases with the distance to the global minimizer, at least when the excess error is large enough. (In contrast, strong convexity would imply quadratic growth F ( θ ; X ( n ) ) -F ( θ ⋆ ; X ( n ) ) ≳ ∥ θ -θ ⋆ ∥ 2 , rather than linear growth.) Intuitively, this growth condition allows us to take larger step sizes and make progress faster, consuming less of the privacy budget. However, since this growth condition only holds for θ that is more than ∆ 3 n/ 4 ( θ ⋆ ) away from the minimizer, which is a data-dependent property, we first need to develop a private algorithm to estimate ∆ 3 n/ 4 ( θ ⋆ ) in order to make use of this property. In Section 2.1, we develop an efficient algorithm, RadiusFinder , for this task, which is inspired by [NSV16]. Our procedure assumes that we know some potentially very small lower bound r ≤ ∆ 3 n/ 4 ( θ ⋆ ) , which is necessary by the impossibility results in [BNSV15]. Since the sample complexity of this procedure depends only on log(1 /r ) , we can choose this parameter to be very small. In Section 2.1, we show how to eliminate this assumption at the cost of a small additive error. With high probability, RadiusFinder (see Theorem 2.4) outputs ˆ ∆ such that ∆ 3 n/ 4 ( θ ⋆ ) ≤ ˆ ∆ ≤ O (∆ 4 n/ 5 ( θ ⋆ )) . Having obtained ˆ ∆ , the second step of the warm-up algorithm is finding a good initialization point. In Section 2.2, we propose Localization , based on DP-GD with geometrically decaying step sizes , to perform this task. Due to the growth condition we show that DP-GD makes a fast progress towards some point that is within O (∆ 4 n/ 5 ( θ ⋆ )) from the optimizer: in log( R ) iterations, with high probability, it outputs θ 0 such that θ ⋆ is in the ball of radius O ( ˆ ∆) = O (∆ 4 n/ 5 ( θ ⋆ )) centered at θ 0 .

DP Cutting Plane Method for Private GM. The main drawback of using DP-SGD for the finetuning stage is that its run-time can be large when n ≫ d . To address this, we design the second fine-tuning algorithm, LocDPCuttingPlane , based on private variant of the cutting plane method that has faster running time when n is large. There are two challenges in the analysis: by using the noisy gradients, we cannot argue that the optimal point always lives in the intersection of the cutting planes, which is a crucial part of the standard analysis. The second challenge is that the cutting plane method is not a descent method in the sense that the loss function is not decreasing with the iteration, and we need to privately select an iterate with small loss. The challenge for developing the private variant here is that the loss F ( θ ; X ( n ) ) has sensitivity proportional to R , so running the exponential

mechanism in the natural way incurs loss proportional to R . We address both of these challenges and develop an algorithm whose excess error is proportional to ∆ 4 n/ 5 ( θ ⋆ ) .

Pure DP algorithm for Private Geometric Median Problem. In Section 4, we propose a pure ( ε, 0) -DP algorithm for the geometric median problem, albeit a computationally inefficient one. Our algorithm is based on the inverse smooth sensitivity mechanism of [AD20]. At a high level, the algorithm outputs θ ∈ R d with a probability proportional to exp( -ε · len( X ( n ) , θ ) / 2) where len( X ( n ) , θ ) is the minimum number of data points from X ( n ) that needs to be modified to obtain a dataset ˜ X ( n ) such that the geometric median of ˜ X ( n ) be equal θ . Our analysis shows that the proposed mechanism outputs ˆ θ = GM( ˜ X ( n ) ) such that ˜ X ( n ) and X ( n ) differ in at most k ⋆ = O ( d log( R ) /ε ) with a high probability. Then, by a careful sensitivity analysis, we show ∥ ˆ θ -θ ⋆ ∥ can be upper bounded by the F ( θ ⋆ ; X ( n ) ) . Using this result we provide an algorithm with a multiplicative guarantee. Moreover, we show ∥ ˆ θ -θ ⋆ ∥ is upper bounded O (∆ γn ( θ ⋆ )) for some γ ∈ (1 / 2 , 1] .

Lower bound on the Sample Complexity. We show every ( ε, δ ) -DP algorithm requires ˜ Ω( √ d/ε ) samples so that it satisfies E ˆ θ ∼A n ( X ( n ) ) [ F ( ˆ θ ; X ( n ) )] ≤ (1 + α ) min θ ∈ R d F ( θ, X ( n ) ) for a constant α . This result shows that the sample complexity of our polynomial-time algorithms is nearly optimal.

A summary of the results is provided in Table 1, comparing the proposed algorithms in terms of privacy, utility, runtime, and sample complexity. As discussed earlier, algorithms with error adaptive to the quantile radius can achieve a nearly multiplicative guarantee. The utility column in Table 1 compares the algorithms based on the achievable α mul and α add in order to F ( A n ( X ( n ) ); X ( n ) ) ≤ (1 + α mul ) F ( θ ⋆ ; X ( n ) ) + α add with a high probability.

1.2 Related Work

DP convex optimization is a well-studied problem [CMS11; KST12; BST14; ACGMMTZ16; STU17; FKT20]. There has been significant interest in developing new algorithms that offer improved guarantees compared to DP-(S)GD for specific problem classes or by leveraging additional information. For instance, [LUZ20; SSTT21; ABGMU22; BMS22] demonstrate that for linear models the dependency of the excess error on the dimension can be improved, [GHST24; ABL23] study the impact of the second-order information on the convergence, [KDRT21; AGMRSSSTT22; GHNOSTTW23] explore the impact of public data, etc. The current paper addresses a drawback of DP-(S)GD, namely, the linear dependence of the excess error on the distance from the initializer to the optimal point in non-strongly convex settings.

Another related line of work to our warm-up strategy is private averaging of [NSV16; NS18; CKMST21; TCKMS22]. The advantage of the algorithm proposed in this work is its simplicity while being optimal in terms of sample complexity: we exploit a structural property of the geometric median and show that running DPGD with the geometrically decaying stepsizes can yield a suitable initialization point without the need for preprocessing steps such as filtering [CKMST21; TCKMS22], coordinate-wise discretization [NSV16], hashing [NS18], etc. The proposed quantile radius can be seen as a robust notion of radius proposed in [BHI02].

In one dimension (i.e., d = 1 ), private versions of the median are well studied [DNPR10; BNS13; BNSV15; DNRR15; BDRS18; ALMM19; KLMNS20; ASSU23; CLNSS23]. In particular, these works improve the dependence on the a priori bound R to log ∗ R , rather than log R in our results.

1.3 Notation

Let d ∈ N . For a vector x ∈ R d , ∥ x ∥ denotes the ℓ 2 norm of x . We use the following notation for the ball of radius R : B d ( a, R ) = { x ∈ R d : ∥ x -a ∥ ≤ R } . Also, B ∞ d ( a, R ) denotes { x ∈ R d : ∥ x -a ∥ ∞ ≤ R } . We refer to B d (0 , R ) = B d ( R ) , similarly, it holds for B ∞ d (0 , R ) = B ∞ d ( R ) . Let ⟨· , ·⟩ denote the standard inner product in R d . For a convex and closed subset Θ ⊆ R d , let Π Θ : R d → Θ be the Euclidean projection operator, given by Π Θ ( x ) = arg min y ∈ Θ ∥ y -x ∥ 2 . For a (measurable) space R , M 1 ( R ) denotes the set of all probability measures on R . Let Z be the data space and let Θ ⊆ R d be the parameter space. Let f : Θ ×Z → R be a loss function. We say f is L-Lipschitz iff there exists L ∈ R such that ∀ z ∈ Z , ∀ w,v ∈ Θ : | f ( w,z ) -f ( v, z ) | ≤ L ∥ w -v ∥ .

1.4 Notions of DP

Definition 1.2. Let ε > 0 and δ ∈ [0 , 1) . A randomized mechanism A n : Z n → M 1 (Θ) is ( ε, δ ) -DP, iff, for every neighbouring dataset (i.e., replacement) X ∈ Z n and X ′ ∈ Z n , and for every measurable subset M ⊆ Θ , it holds P θ ∼A n ( X ) ( θ ∈ M ) ≤ e ε · P θ ∼A n ( X ′ ) ( θ ∈ M ) + δ .

For some of our privacy analysis, we use concentrated differential privacy [DR16; BS16], as it provides a simpler composition theorem - the privacy parameter ρ adds up when we compose.

Definition 1.3 ([BS16, Def. 1.1]) . A randomized mechanism A : Z n →M 1 ( R ) is ρ -zCDP, iff, for every neighbouring dataset (i.e., replacement) X ∈ Z n and X ′ ∈ Z n , and for every α ∈ (1 , ∞ ) , it holds D α ( A n ( X ) ∥A n ( X ′ )) ≤ ρα, where D α ( A n ( X ) ∥A n ( X ′ )) is the α -Renyi divergence between A n ( X ) and A n ( X ′ ) .

We should think of ρ ≈ ε 2 : to attain ( ε, δ ) -DP, it suffices to set ρ = ε 2 4 log(1 /δ )+4 ε [BS16, Lem. 3.5].

Lemma 1.4 ([BS16, Prop. 1.3]) . Assume we have a randomized mechanism A : Z → M 1 ( R ) that satisfies ρ -zCDP, then for every δ > 0 , A is ( ρ +2 √ ρ log(1 /δ ) , δ ) -DP.

2 Private Localization

In this section, we present the proposed algorithm for the warm-up stage; it has two steps: Private Estimation of Quantile Radius and Private Localization .

2.1 Step 1: Private Estimation of Quantile Radius

Algorithm 1 describes our private algorithm RadiusFinder for quantile radius estimation.

Algorithm 1 RadiusFinder n

1: Inputs: data set X ( n ) ∈ ( B d ( R )) n , fraction γ ∈ (1 / 2 , 1] , privacy budget ρ -zCDP, failure probability β , discretization error 0 < r < R .

2: m = ⌈ γn ⌉ .

3: For every ν ≥ 0 and i ∈ [ n ] , let

4: For every ν ≥ 0 , define

̸

8: Output ˆ ∆ = 2 ˆ i r if ˆ i = Fail ; else Output Fail .

Remark 2.1 . The runtime of RadiusFinder is Θ(( n 2 + n log( n )) log( ⌈ R/r ⌉ )) : First, we need to compute the pairwise distances which take n 2 time. Then, for a fixed ν , we can compute N ( ν ) using the pairwise distances in time Θ( n 2 ) . To compute N ( ν ) , we need to sort { N i ( ν ) } i ∈ [ n ] , in Θ( n log( n )) time, and pick top m . Finally, we need to repeat this for each ν ∈ [ r, . . . , 2 ⌈ log ( 2 R r ) ⌉ r ] . ◁

Notice that Algorithm 1 uses the datapoints as centers for computing the number of the datapoints in a given distance. The privacy proof of Algorithm 1 is based on the following lemma.

Lemma 2.2. Fix n ∈ N . For every dataset X ( n ) , for every 1 / 2 ≤ γ ≤ 1 and for every fixed ν , the query N ( ν ) ≜ 1 m max { i 1 ,...,i m }⊆ [ n ] { N i 1 ( ν ) + · · · + N i m ( ν ) } , has a sensitivity upper-bounded by 3 where m = ⌈ γn ⌉ and N i ( ν ) ≜ | X ( n ) ∩ B d ( x i , ν ) | . Here B d ( x, ν ) := { y ∈ R d : ∥ y -x ∥ ≤ ν } .

▷ Number of datapoints within distance of ν from x i

The objective of Algorithm 1 is to privately approximate ∆ γn ( θ ⋆ ) . Nonetheless, Algorithm 1 relies on computing the pairwise distances between datapoints. The following lemma elucidates why computing these pairwise distances serves as an effective proxy for computing ∆ γn ( θ ⋆ ) .

Lemma 2.3. Fix n ∈ N , 1 ≤ m ≤ n , γ 1 , γ 2 ∈ (1 / 2 , 1] such that γ 2 ≥ γ 1 , and dataset X ( n ) . For every ν ≥ 0 , define N ( ν ) ≜ 1 m max { i 1 ,...,i m }⊆ [ n ] { N i 1 ( ν ) + · · · + N i m ( ν ) } , where N i ( ν ) ≜ | X ( n ) ∩B d ( x i , ν ) | . Let θ ⋆ = GM( X ( n ) ) . For every ˆ ν such that N (ˆ ν ) ≥ ⌈ γ 1 n ⌉ and N (ˆ ν/ 2) < ⌈ γ 2 n ⌉ , we have

Using these two lemmas, in the next theorem, we present the privacy and utility guarantees of Algorithm 1. As we are interested in finding the smallest radius, we use the standard AboveThreshold from [DNRRV09; DR+14] as a subroutine in Algorithm 1. The algorithmic description of AboveThreshold is provided in Appendix B for completeness.

Theorem 2.4. Let RadiusFinder n denote Algorithm 1. Fix d ∈ N , R > 0 , r > 0 , β ∈ (0 , 1] , and ρ > 0 . Then, for every n ∈ N and every dataset X ( n ) ∈ ( B d ( R )) n the output of RadiusFinder n satisfies ρ -zCDP. Also, the output of RadiusFinder n satisfies the following utility guarantees:

Assume that the data points satisfies N ( r ) < m . Let ˜ γ ≜ min { γ + 1 n 36 √ 2 ρ log ( 2( ⌈ log ( 2 R r )⌉ +1) /β ) , 1 } , then, given n > 18 (1 -γ ) √ 2 ρ log(4 /β ) , we have

Let ˜ γ ≜ min { γ + 1 n 36 √ 2 ρ log ( 2( ⌈ log ( 2 R r )⌉ +1) /β ) , 1 } . Given n > 18 (1 -γ ) √ 2 ρ log(4 /β ) , we have

Remark 2.5 . A sufficient condition for N ( r ) < m in Item 2 is that max i ∈ [ n ] | X ( n ) ∩ B d ( x i , r ) | < m = ⌈ γn ⌉ . Intuitively, this means that no data point should have a significant portion of other data points within a ball of radius r centered on it. ◁

2.2 Step 2: Fast Localization

In the second step of the warm-up phase, we develop a fast algorithm for finding a good initialization point using the private estimate of the quantile radius. The main structural result that we use for the algorithm design is stated in the next lemma.

Lemma 2.6. Fix n ∈ N , X ( n ) ∈ ( R d ) n and θ 1 , θ 0 ∈ R d . For every γ ∈ [0 , 1] , define ∆ γn ( θ 0 ) ≜ min { r ≥ 0 : | i ∈ [ n ] : ∥ x i -θ 0 ∥ ≤ r | ≥ γn } . Assume there exists ζ ≥ 0 such that F ( θ 1 ; X ( n ) ) -F ( θ 0 ; X ( n ) ) ≤ ζn . Then, for every γ ∈ (1 / 2 , 1] , we have

To gain some intuition behind Lemma 2.6, let us instantiate θ 0 = θ ⋆ . This result implies that for a θ ∈ R d such that ∥ θ -θ ⋆ ∥ ≳ ∆ γn ( θ ⋆ ) , the loss function of the geometric median satisfies F ( θ ; X ( n ) ) -F ( θ ⋆ ; X ( n ) ) ≳ ∥ θ -θ ⋆ ∥ . Using this result, we propose Algorithm 2 for finding a good initialization. The next theorem states the privacy and utility guarantees of Algorithm 2.

Theorem 2.7. Let Localization n denote Algorithm 2. Fix d ∈ N , R > 0 , r > 0 , ρ > 0 , and β ∈ (0 , 1) . Then for every dataset X ( n ) ∈ ( B d ( R )) n the outputs of Localization n satisfies ρ -zCDP. Moreover, let ( ˆ θ, ˆ ∆) = Localization n ( X ( n ) , ρ, r, β ) and define random set Θ loc = { θ ∈ B d ( R ) : ∥ ∥ θ -ˆ θ ∥ ∥ ≤ 25 ˆ ∆ } . Then, given

Algorithm 2 Localization n

1: Inputs: dataset X ( n ) ∈ ( B d ( R )) n , privacy parameters ρ -zCDP, discretization error r , failure probability β

2: γ = 3 / 4

4: if ˆ ∆ = Fail then

5: Output Fail and Halt.

6: k wu = 1 log(2) log ( R/ ˆ ∆ )

7: θ 0 = 0 ∈ R d , T wu = 500 , rad 0 = R

8: for t ∈ { 0 , . . . , k wu -1 } do

9: Θ t = { θ ∈ B d ( R ) : ∥ θ -θ t ∥ ≤ rad t }

12: rad t +1 = 1 2 rad t +12 ˆ ∆

13: Output θ k wu and ˆ ∆ .

we have P ( θ ⋆ ∈ Θ loc and ∆ 0 . 75 n ( θ ⋆ ) ≤ 4 ˆ ∆ and { ˆ ∆ ≤ 4∆ 0 . 8 n ( θ ⋆ ) or ˆ ∆ = r }) ≥ 1 -2 β . Also, assuming that the datapoints satisfies max i ∈ [ n ] | X ( n ) ∩ B d ( x i , r ) | < 3 n/ 4 , we have

3 Private Fine-tuning

In Section 2, we developed an algorithm for the warm-up stage. The output of the warm-up stage is θ 0 and radius ˆ ∆ such that ∥ θ 0 -θ ⋆ ∥ ≤ O ( ˆ ∆) and ˆ ∆ = ˜ O (∆ 4 n/ 5 ( θ ⋆ )) as formalized in Theorem 2.7. In this section, we build upon the output of the warm-up algorithm to develop two polynomial-time algorithms for the fine-tuning stage.

3.1 Fine-tuning Using DPGD

Our first algorithm is based on DP-GD [BST14]. The main ideas behind Algorithm 3 is as follows: 1) from the utility guarantee of the warm-up phase in Theorem 2.7, the distance of the initialization and θ ⋆ only depends on ˆ ∆ , i.e., it does not depend on R , 2) By definition of the quantile radius in Definition 1.1 and Equation (1), we have that F ( θ ⋆ ; X ( n ) ) ≥ (1 -γ ) n ∆ γn ( θ ⋆ ) , 3) in the case that the data satisfies some regularity conditions, we have ˆ ∆ ≤ 4∆ 0 . 8 n ( θ ⋆ ) from Theorem 2.7. The next theorem summarizes the utility and privacy guarantees of this algorithm.

Algorithm 3 LocDPGD n

1: Inputs: dataset X ( n ) ∈ ( B d ( R )) n , privacy parameters ρ -zCDP, discretization error r , failure probability β .

( ) ▷ Algorithm 2

2: θ 0 , ˆ ∆ = Localization n X ( n ) , ρ 2 , r, β 2

3: Θ 0 = { θ ∈ B d ( R ) : ∥ θ -θ 0 ∥ ≤ 25 ˆ ∆ }

4: η ft = 50 ˆ ∆ √ d 6 ρn 2 and T ft = n 2 ρ 256 d

5: ˆ θ = DPGD ( θ 0 , X ( n ) , ρ 2 , Θ 0 , η ft , T ft )

6: Output ˆ θ

Theorem 3.1. Let LocalizedDPGD n denote Algorithm 3. For every d ∈ N , R > 0 , r > 0 , ρ > 0 , and β ∈ (0 , 1] , A = { LocalizedDPGD n } n ≥ 1 satisfies the following: for every n ∈ N and every

▷ Algorithm 6

dataset X ( n ) ∈ ( B d ( R )) n the output of LocalizedDPGD n satisfies ρ -zCDP. Also, given

we have

Moreover, given that the datapoints satisfies max i ∈ [ n ] | X ( n ) ∩ B d ( x i , r ) | < 3 n/ 4 , we have

where ˆ θ is the output of Algorithm 3.

3.2 Fine-tuning Using Noisy Cutting Plane Method

In this section, we present the second fine-tuning algorithm: LocDPCuttingPlane of Algorithm 4. This algorithm is based on the well-known cutting plane method [New65; Lev65; Nes98].

Algorithm 4 LocDPCuttingPlane n

1: Inputs: dataset X ( n ) ∈ ( B d ( R )) n , privacy parameters ( ε, δ ) -DP, discretization error r , failure probability β 2: ρ = ε 2 16 log(2 /δ )+8 ε

3:

4:

0

θ

,

Θ

(

n

Localization

∆ =

ˆ

=

X

θ

(

,

n

)

6:

d

0

for

0

θ

{

t

(

R

) :

θ

∈ B

∥

-

∈ {

ft

0

, . . . , k

1

-

7:

}

θ

t

t

Centre

=

(Θ

)

(

exp

ε

448 ˆ ∆

(

(

n

do

ρ

2

, r, min

}

25 ˆ ∆

{

β

3

,

δ

2

}

)

▷ Algorithm 2

▷ See Assumption 1 for definition of τ

▷ See Assumption 1

for

t

∈ {

0

, . . . , k ft

-

1

}

10:

Define Probability Measure:

11: Output θ ˆ t where ˆ t ∼ π

Similar to non-private cutting plane method, LocDPCuttingPlane is not a descent algorithm. As a result, we need to devise a mechanism for selecting an iterate with minimal loss. In the next lemma, we provide a bespoke analysis of the exponential mechanism with the score function F ( θ ; X ( n ) ) defined in Equation (1). Note that the sensitivity of F ( θ ; X ( n ) ) is R . However, the next result demonstrates that through a novel analysis of the sensitivity of F ( θ ; X ( n ) ) , the noise scale due to privacy can be significantly reduced. Proof can be found in Appendix E.

Lemma 3.2. Let ε ∈ R , k ∈ N , and d ∈ N be constants. Let Θ ⊆ R d be a set with a bounded diameter of diam . Let X ( n ) ∈ ( R d ) n be a dataset and θ ⋆ ∈ GM( X ( n ) ) . Let { θ 1 , . . . , θ k } ⊆ Θ be k fixed vectors. Also, assume that θ ⋆ ∈ Θ . Let ∆ be such that 3∆ 3 n/ 4 ( θ ⋆ ) + 2 diam ≤ ∆ . Consider the following probability measure over { 1 , . . . , k } :

Let ˆ i ∼ π ( · ; X ( n ) ) and OPT ≜ min i ∈ [ k ] { F ( θ i ; X ( n ) ) -F ( θ ⋆ ; X ( n ) ) } . Then, for every β ∈ (0 , 1] , we have

π

(

t

)

∝

-

F

θ

t

;

X

)

) )

Let ˜ X ( n ) be a dataset of size n that differs in one sample from X ( n ) . Then, for every i ∈ [ k ] , we have

The next theorem provides the privacy guarantee of Algorithm 4. The privacy analysis differs from the rest of the algorithms in the paper. This deviation arises from the fact that for analyzing the privacy guarantee of Line 10 of Algorithm 4, we use Lemma 3.2. Notice that the guarantee in Lemma 3.2 holds provided that Θ 0 , defined in Line 4 of Algorithm 4, satisfies θ ⋆ ∈ Θ 0 . Ergo, the privacy guarantee of Algorithm 4 only satisfies approximate-DP .

Theorem 3.3. Let LocDPCuttingPlane n denote Algorithm 4. Fix d ∈ N , R > 0 , r > 0 , ε > 0 , δ ∈ (0 , 1] , and β ∈ (0 , 1] . Then, for every n ∈ N and every dataset X ( n ) ∈ ( B d ( R )) n the output of LocDPCuttingPlane n satisfies ( ε, δ ) -DP.

We also make the following assumption about the performance of Centre subroutine in Algorithm 4.

Assumption 1. There exists some τ ∈ (0 , 1] such that for all t ∈ { 0 , . . . , k ft -1 } , the subroutine of Centre in Algorithm 4 satisfies vol (Θ t +1 ) ≤ (1 -τ ) vol (Θ t ) . Furthermore, the time for calling the routine Centre is T c .

Using the John Ellipsoid [Joh14] as the Centre makes τ a dimension independent constant and T c = ˜ O ( d 1+ ω ) (by [LSW15]). Now we are ready to state the utility guarantee of Algorithm 4.

Theorem 3.4. Let LocDPCuttingPlane n denote Algorithm 4. For every d ∈ N , R > 0 , r > 0 , ε > 0 , δ ∈ (0 , 1] , and β ∈ (0 , 1] , A = { LocDPCuttingPlane n } n ≥ 1 satisfies the following: for every n ∈ N and every dataset X ( n ) ∈ ( B d ( R )) n , given

where ρ = ε 2 16 log(2 /δ )+8 ε , we have the following: Let κ ≜ n √ ρ √ d + √ d and α = O (√ d log( κ ) τρ · log ( d log( κ ) τβ ) ) . Then,

Moreover, assuming that the datapoints satisfies max i ∈ [ n ] | X ( n ) ∩ B d ( x i , r ) | < 3 n/ 4 , we have

where ˆ θ is the output of Algorithm 4.

4 Pure-DP Algorithm for Geometric Median

̸

In this section, we propose an algorithm based on the assumption that we have an access to an oracle that outputs an exact GM ( X ( n ) ) . Before presenting the algorithm, we need a definition: For two sequences of a = ( a 1 , . . . , a n ) ∈ ( R d ) n and b = ( b 1 , . . . , b n ) ∈ ( R d ) n , we define the hamming distance as d H ( a , b ) = ∑ n i =1 1 [ a i = b i ] . The proposed algorithm is shown in Algorithm 5, and its utility and privacy guarantees are presented in the following theorem.

Theorem 4.1. Let SInvS n denote the algorithm in Algorithm 5. Fix d ∈ N , R > 0 , r > 0 , and ε > 0 . Then, for every n ∈ N and every dataset X ( n ) ∈ ( B d ( R )) n the output of SInvS n satisfies ε -DP. Also, for every β ∈ (0 , 1) and for every n > 2 k ⋆ ≜ 2 ⌊ 2 ε (log(1 /β ) + d log( R/r )) ⌋ , with probability at least 1 -β , we have:

The value of the cost function satisfies

Algorithm 5 SInvS n

1: Input: dataset X ( n ) ∈ ( B d ( R )) n , privacy parameter ε -DP, discretization error r .

2: For every y ∈ B d ( R )

4: Output ˆ θ ∼ π

2. In terms of distance,

The proof of Theorem 4.1 is provided in Appendix F. The proof is based on showing that the output ˆ θ = GM ( ˜ X ( n ) ) is such that ˜ X ( n ) and X ( n ) differ in at most k ⋆ = O ( d log( R ) /ε ) datapoints with a high probability. Then, we use the properties of the geometric median to show that the sensitivity of GMto changing k < n/ 2 points can be bounded by the value of the optimal loss at θ ⋆ = GM( X ( n ) ) .

5 Lower Bound on the Sample Complexity

In this section we prove a lower bound on the sample complexity of any ( ε, δ ) -DP algorithm for the task of private geometric median with a multiplicative error.

Theorem 5.1. Let ε 0 , α 0 , d 0 be universal constants. Then, for every ε ≤ ε 0 , α ≤ α 0 , and d ≥ d 0 and every ( ε, δ ) -DP algorithm A n : ( R d ) n →M 1 ( R d ) (with δ = ˜ O ( √ d/n ) ) such that for every dataset X ( n ) ∈ ( R d ) n its output satisfies E ˆ θ ∼A n ( X ( n ) ) [ F ( ˆ θ ; X ( n ) )] ≤ (1 + α ) min θ ∈B ∞ d (1) F ( θ ; X ( n ) ) we require n = ˜ Ω ( √ d ε ) .

This result, whose proof can be found in Appendix G, shows that the sample complexity of the proposed polynomial time algorithms is tight in terms of the dependence on ε and d .

6 Numerical Example

In this section, we numerically compare LocDPGD n (Algorithm 3) and DPGD on a synthetic dataset. The dataset consists of two subsets: one tightly clustered at a random location on B d ( R ) , and the other uniformly distributed over B d ( R ) . We plot F ( θ ; X ( n ) ) /F ( θ ⋆ ; X ( n ) ) for both algorithms as R varies. The results show that LocDPGD n 's performance degrades more gracefully than DP-GD with increasing R . See Appendix H for experimental details and more results.

7 Conclusion and Limitations

In this paper, we presented three private algorithms for the geometric median task, ensuring an excess error guarantee that scales with the effective data scale. Our results open up many directions: we believe our warm-up algorithm has broader applications, and finding other problems where it can be used as a subroutine is interesting. Another direction is to characterize the optimal run-time: is it possible to develop a linear time algorithm, i.e. ˜ Θ( nd ) , with an optimal excess error?

10 3

10 4

10 5

10 6

10 7

10 8

10 9

10 10

R

10 0

10 1

10 2

10 3

10 4

10 5

10 6

F

(

;

X

(

n

)

)/

F

(

;

X

(

n

)

)

d

=200,

n

=3000,

=2,

=1/

n

Ours

Acknowledgments

The authors would like to thank Jad Silbak, Eliad Tsfadia, and Mohammad Yaghini for helpful discussions.

References

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and the introduction completely summarize our findings.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Section 7, we discussed two limitations of our work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: In our problem setup, we completely discussed all the assumptions. Also, a complete proof of every claim is presented in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In Appendix H, we discussed all the details behind our implementation. Also, we release the code. Since the dataset considered is synthetic, there is no concern regarding the dataset.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have released the code along with a Colab notebook.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In our experiments, we compare our proposed algorithm with a well-known baseline. We implemented our algorithm from scratch. Also, all the details are included in the code and Appendix H.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We have included the error bars in the plots.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer:[Yes]

Justification: Our results can be produced using public Google Colab.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer:[NA]

Justification: This question is not applicable to our paper.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work does not have any societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: It is not applicable to our work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: Not applicable to our work.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: It is not applicable to our work.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: It is not applicable to our work.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: It is not applicable to our work.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

A Preliminaries

A.1 Gradient of the Geometric Loss

A.2 DP Gradient Descent (DPGD)

In this section, we provide the algorithmic description of DPGD and its privacy and utility analysis for completeness.

Algorithm 6 DPGD

1: Inputs: initialization point θ 1 ∈ R d , dataset X ( n ) ∈ ( R d ) ( n ) , privacy budget ρ , feasible set Θ , stepsize η , number of iterations T . 2: σ 2 = T 2 ρn 2

where ξ t ∼ N (0 , σ 2 I d ) .

4: Output 1 T ∑ T t =1 θ t

Lemma A.1. Let Θ ⊆ R d be a closed and convex set with a finite diameter diam . Let ℓ : Θ ×Z → R be a loss function such that for every z ∈ Z , ℓ ( · , z ) is convex and L -Lipschitz. Let X ( n ) = ( z 1 , . . . , z n ) ∈ Z n and ˆ L n ( θ ) = 1 n ∑ i ∈ [ n ] ℓ ( θ, z i ) . Consider DP-Gradient descent algorithm θ t +1 = Π Θ ( θ t -η ( ∇ ˆ L n ( θ t ) + ξ t )) , where ξ t ∼ N (0 , σ 2 I d ) . Then, for every T ∈ N , by setting η = diam √ d 12 L 2 ρn 2 , and σ 2 = L 2 T 2 ρn 2 , we have the following: { θ t } t ∈ [ T ] satisfies ρ -zCDP. Also, for every β > 0 , given Td ≥ log(4 /β ) and 1 ≤ √ 56 log(2 /β ) , with probability at least 1 -β , we have

Proof. The privacy proof is based on the zCDP analysis of the Gaussian mechanism and the composition property of zCDP [BS16].

Let g t ≜ ∇ ˆ L n ( θ t ) + ξ t and θ ⋆ ∈ arg min θ ∈ Θ ˆ L n ( θ ) . Note that we can replace ∇ ˆ L n ( θ t ) by any subgradient at θ t . By the convexity of ℓ and the first-order convexity condition we can write

Then, by the contraction property of the projection, we can write

Here, we have used for every a, b ∈ R d , ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 +2 ∥ b ∥ 2 , and ∥ ∥ ∥ ∇ ˆ L n ( θ ) ∥ ∥ ∥ ≤ L for every θ . Therefore, we conclude that

̸

Define the following random variable for every t ∈ [ T ]

Also, define the following filtration

which is the sigma-field generated by θ 0 , . . . , θ t .

Lemma A.2. { Y t } t ∈ [ T ] is a martingale difference sequence adapted to {F t } t ∈ [ T ] .

Proof. Notice that ∇ ˆ L n ( θ t ) , θ t , and θ ⋆ are F t -measurable. Therefore, we can write

By definition ξ t is independent of the history up to time t . Therefore, E [ ξ t |F t ] = 0 since E [ ξ t ] = 0 which gives

Therefore, E [ Y t |F t ] = 0 . Moreover, by Cuachy-Schwartz inequality and the boundedness of Θ we can write

Therefore, { Y t } t ∈ [ T ] is a martingale difference sequence as was to be shown.

Using Equation (4) and by the definition of Y t in Equation (5), we can write

Summing it from 0 to T -1 gives

Analyzing (A) in Equation (9)

Notice that ∑ t ∈ [ T ] ∥ ξ t ∥ 2 d = σ 2 ∥ Y ∥ 2 . Therefore, for every β ∈ (0 , 1) provided that Td ≥ log(4 /β ) , with probability at least 1 -β/ 2 , we have

Analyzing (B) in Equation (9)

Lemma A.3 (Shamir [Sha11]) . Let m ∈ N . Let { Z m } m ∈ [ M ] be a martingale difference sequence adapted to a filtration {F m } m ∈ [ M ] , and suppose there are constants b > 1 and c > 0 such that for any m and any α > 0 , it holds that

Then for any β > 0 , it holds with probability at least 1 -β that

We can rephrase Equation (5) as

Notice that condition on F t , ⟨ ξ t , θ ⋆ -θ t ⟩|F t ∼ N (0 , σ 2 ∥ θ t -θ ⋆ ∥ 2 ) . Therefore,

Note that the bound holds for every t ∈ [ T ] . Therefore, using Lemma A.3, with probability at least 1 -β/ 2 , we have

From Equation (9), Equation (10), and Equation (11), we have with probability at least 1 -β

provided that Td ≥ log(4 /β ) . Let

Using these parameters, we obtain that

where the last step is by assuming that 1 ≤ √ 56 log(2 /β ) .

B Above Threshold Algorithm

Algorithm 7 AboveThreshold

1: Inputs: Queries { f 0 , . . . , f k -1 } , Privacy Budget ρ -zCDP, Threshold T .

C Technical Lemma

Lemma C.1. Let σ > 0 . Let Y be a random variable with the distribution N (0 , σ 2 ) . Then, for every β ∈ (0 , 1] , we have P ( | Y | > σ √ 2 log(2 /β ) ) ≤ β .

Lemma C.2 (Laurent and Massart [LM00]) . Let m ∈ N . Consider random vector Y ∼ N (0 , I m ) . Then, for every t ≥ 0 ,

Corollary C.3. Let β ∈ (0 , 1) , m ∈ N , and m ≥ log 2 β . Consider Y ∼ N (0 , I m ) , then

Lemma C.4. Let n ∈ N and n ≥ 4 . Let X ( n ) ∈ ( R d ) n and ˜ X ( n ) ∈ ( R d ) n be two datasets that differ in one sample. Let θ ⋆ ∈ GM( X ( n ) ) and θ ⊛ ∈ GM( ˜ X ( n ) ) . Let ∆ 3 n/ 4 ( θ ⋆ ) be the radius of the ball around θ ⋆ that contains at least 3 n/ 4 of X ( n ) . Then,

Proof. The proof is by contrapositive. In particular, we show that for every θ ∈ R d such that ∥ θ -θ ⋆ ∥ > 3 2 ∆ 3 n/ 4 ( θ ⋆ ) , we have, θ / ∈ GM( ˜ X ( n ) ) . Let I = { i ∈ [ n ] : x i ∈ B d ( θ ⋆ , ∆ 3 n/ 4 ( θ ⋆ )) and x i ∈ ˜ X ( n ) } . Using the variational representation of ∥·∥ 2 , we can write

where the last step follows from Cauchy-Schwarz inequality. Then, we can write

Therefore ∥ θ ⊛ -θ ⋆ ∥ ≤ 3 / 2∆ 3 n/ 4 ( θ ⋆ ) .

Lemma C.5. For every n ∈ N and for every X ( n ) = ( x 1 , . . . , x n ) , we have GM ( X ( n ) ) lies in the convex hull of { x 1 , . . . , x n } .

Lemma C.6. Let ( x 1 , . . . , x n ) ∈ ( R d ) n be a dataset and θ ⋆ = GM(( x 1 , . . . , x n )) . Let B ⊆ [ n ] such that |B| < n/ 2 . Then, for every θ , we have

It is a simple modification of [CLMPS16, Lemma. 24].

Proof.

D Proof of Section 2

Proof of Lemma 2.2. The proof follows closely [NSV16, Lemma 4.5]. Let X and X ′ are two neighboring datasets of size n that differ in the first sample. For a fixed ν and i ∈ [ n ] , if i = 1 , N i ( ν ) can change by one. Also, in the worst-case the new datapoint can be close to the rest of the datapoints. Therefore, the sensitivity is bounded by 1 m (( m -1) + n ) ≤ 1 + n m ≤ 1 + 1 γ ≤ 3 where the last step follows from γ ≥ 1 / 2 .

̸

Proof of Lemma 2.3. Let ˆ ν be such that N (ˆ ν ) ≥ ⌈ γ 1 n ⌉ , by definition of N ( · ) it means that there exists a datapoint x i such that the ball of radius ˆ ν around it contains at least ⌈ γ 1 n ⌉ datapoints. Let B = { j ∈ [ n ] : ∥ x i -x j ∥ > ˆ ν } . By the described argument, we have |B| ≤ (1 -γ 1 ) n . Then, we invoke Lemma C.6 with the described B and θ = x i to obtain

The first step follows because function h : R → R , h ( z ) = 2 n -2 z n -2 z is increasing for z < n/ 2 . In the next step, we use the triangle inequality to write

where ∆ · ( · ) is defined in Definition 1.1.

Next, we turn into proving the upperbound on ˆ ν . By assumption N (ˆ ν/ 2) < ⌈ γ 2 n ⌉ . For the sake of contradiction, assume that ˆ ν > 4∆ γ 2 n ( θ ⋆ ) . Then, consider the set G = { i ∈ [ n ] : ∥ θ ⋆ -x i ∥ ≤ ∆ γ 2 n ( θ ⋆ ) } . By definition, |G| ≥ ⌈ γ 2 n ⌉ . Consider an arbitrary subset of G with the size of ⌈ γ 2 n ⌉ . The main observation, which follows from the triangle inequality, is that a ball of radius 2∆ γ 2 n ( θ ⋆ ) around every point in G contains at least ⌈ γ 2 n ⌉ datapoint. Therefore, N (ˆ ν/ 2) ≥ N (2∆ γ 2 n ( θ ⋆ )) ≥ ⌈ γ 2 n ⌉ which contradicts with the assumption that N (ˆ ν/ 2) < ⌈ γ 2 n ⌉ .

Proof of Theorem 2.4. The privacy proof simply follows from the privacy analysis in [DR+14, Sec. 3.6]. We focus here on the utility guarantees.

Part 1: Let k = ⌈ log ( 2 R r ) ⌉ . It is simple to see that

where the last step follows from the assumption that max x i ,x j ∈ X ( n ) ∥ x i -x j ∥ ≤ 2 R , which gives us N (2 k r ) = n by the definition of N ( · ) . By a simple tail bound on the Laplace distribution, we have P ( | ξ thresh | ≥ 6 √ 2 ρ log(4 /β ) ) ≤ β/ 4 and P ( | ξ k | ≥ 12 √ 2 ρ log(4 /β ) ) ≤ β/ 4 . Therefore, given n > 1 1 -γ 18 √ 2 ρ log(4 /β ) , P ( n -m ≤ ξ thresh -ξ k ) ≤ β/ 2 .

Part 2: Lemma C.6 implies that for every ν such that N ( ν ) ≥ ⌈ γn ⌉ , we have, ∆ γn ( θ ⋆ ) · 2 γ -1 4 γ -1 ≤ ν . Therefore, we can write

̸

̸

Under the event that ˆ ∆ = Fail , there exists i ∈ { 0 , . . . , k -1 } , such that

By a simple tail bound and union bound, we have

where k = ⌈ log ( 2 R r ) ⌉ . We further upperbound the first term in Equation (15) as follows

̸

̸

We claim that the first term in this equation is zero. Recall that m = ⌈ γn ⌉ . Under the event B c , ξ thresh -ξ i ≥ -18 √ 2 ρ log(2( k +1) /β ) and as a result, m + 18 √ 2 ρ log(2( k +1) /β ) + ξ thresh -ξ i ≥ m . Therefore, it shows that the probability of the first term is zero. Also, as showed above, P ( B ) ≤ β/ 2 . Therefore, P ( N ( ˆ ∆ ) < ⌈ γn ⌉ ) ≤ P ( B )+ P ( ˆ ∆ = Fail ) . Combining it with P ( ˆ ∆ = Fail ) ≤ β/ 2 concludes the proof.

Part 3: Assume that N ( r ) < m . Let k = ⌈ log ( 2 R r ) ⌉ . Let ˜ γ = γ + 1 n 18 √ 2 ρ log(2( k +1) /β ) . In Part 2, we showed that given n > 18 √ 2 ρ log(4 /β ) , we have

We only focus on the upperbound. From Lemma 2.3, we have

We can write

where the last step follows from the union bound. For the first term, we have

where ˆ i = log ( ˆ ∆ / 2 r ) -1 . The last step follows from the following observation: under the event that ˆ ∆ / ∈ { r, Fail } , during the execution of Algorithm 7, both N ( ˆ ∆) and N ( ˆ ∆ / 2) are compared to the noisy threshold. Using the tail bounds in Equation (16), we have under the event B c , with probability at least 1 -β/ 2 ,

This shows that we have

In the next step, we bound P ( ˆ ∆ = r ) . Notice that

Using simple tail bound, we have P ( ξ thresh -ξ 1 ≤ -18 √ 2 ρ log ( 2 β · ⌈ log ( 2 R r )⌉ )) ≤ β/ 2 which shows that P ( ˆ ∆ = r ) ≤ β/ 2 since we assume that N ( r ) < m . Therefore, combining all the pieces together, we proved

Part 4: In Part 2, we showed that given n > 18 (1 -γ ) √ 2 ρ log(4 /β ) , we have

Consider the following event E = { ˆ ∆ ≤ 4∆ ˜ γn ( θ ⋆ ) or ˆ ∆ = r } . We have

̸

̸

Here, the last step follows from Equation (17). Notice that in Equation (18), we analyzed the probability of the first term and we showed that it is as most β/ 2 . We also have that P ( ˆ ∆ = Fail ) ≤ β/ 2 from Part 1. Therefore, P ( E c ) ≤ β . Combining it with Equation (19) concludes the proof.

Proof of Lemma 2.6. Let I = { i ∈ [ n ] : ∥ θ 0 -x i ∥ ≤ ∆ γn ( θ 0 ) } . For every i ∈ I , we have ∥ x i -θ 0 ∥ ≤ ∆ γn ( θ 0 ) . Using the triangle inequality, for every i ∈ I , we can write

The last equation is equivalent to

Then, for every i / ∈ I , by an application of the triangle inequality

Then, by adding both sides of Equation (20) and Equation (21), we have

This equation can be represented as

Let γ ′ n = |I| . We know that γ ′ ≥ γ . Using this representation we can write

For a fixed a, b > 0 define h ( x ) ≜ a +2 xb 2 x -1 . For x > 1 / 2 , d h ( x ) d x = -2( a + b ) (2 x -1) 2 . This shows that h ( x ) is decreasing for x > 1 / 2 . Therefore using this observation

as was to be shown.

̸

Proof of Theorem 2.7. The privacy proof is straightforward. Algorithm 2 uses the data in Line 3 and Line 11. Based on the privacy budget allocation and the composition properties of zCDP, we can show that the output satisfies ρ -zCDP.

For the claim regarding utility, in the first step, consider the recursion in Line 12 of Algorithm 2, i.e., rad t +1 = 1 2 rad t +12 ˆ ∆ initialized at rad 0 = R . It can be easily shown that rad m = 1 2 m rad 0 + 12 ˆ ∆ ∑ m -1 i =0 (1 / 2) i for m ≥ 1 . In particular, let k wu = 1 log(2) log ( R/ ˆ ∆ ) , then, we obtain that rad k wu ≤ 25 ˆ ∆ .

Let γ = 3 / 4 and

where the last step follows because n ≥ Ω ( 1 √ ρ log(( ⌈ log( R/r ) ⌉ +1) /β ) ) . Then, define the following event

In the next step, we analyze the probability that θ ⋆ ∈ Θ loc. We claim that

We prove this by induction. In particular, we claim that for every m ∈ { 0 , . . . , k wu } we have P ( θ ⋆ ∈ Θ m ∣ ∣ G 1 ) ≥ (1 -β/ (2 k wu )) m . Note that we Θ loc = Θ k wu .

For the base case, by the assumption that the datapoints are in B d ( R ) , we have P ( θ ⋆ ∈ Θ 0 |G 1 ) = P ( θ ⋆ ∈ Θ 0 ) = 1 since Θ 0 is trivially independent of every random variable. Then, we show the claim for m ∈ { 1 , . . . , k wu } assuming the claim holds for m -1 . We can write

We claim that

To show this lets instantiate Lemma 2.6 with θ 0 = θ ⋆ and γ = 3 / 4 to obtain that for every θ ∈ R d ,

Notice that conditioned on G 1 , we have 3∆ γn ( θ ⋆ ) ≤ 12 ˆ ∆ . This shows that 2 ( F ( θ m ; X ( n ) ) -F ( θ ⋆ ; X ( n ) ) ) n ≤ 1 2 rad m -1 implies that ∥ θ ⋆ -θ m ∥ ≤ rad m conditioned on G 1 by the definition of rad m in Line 12. In the next step, we invoke Lemma A.1. Conditioned on θ ⋆ ∈ Θ m -1 and G 1 , with probability at least 1 -β 2 k wu , we have

Notice k wu ≤ 1 log(2) log( R/r ) a.s. By setting T wu = 128 and the bound on the sample size, we have F ( θ m ; X ( n ) ) -F ( θ ⋆ ; X ( n ) ) ≤ rad m -1 2 . Also, notice that the randomness in DPGD is independent of history. Therefore,

Therefore, combining Equations (23) and (24), we obtain

as was to be shown. From Theorem 2.4, given n ≥ Ω ( 1 √ ρ log(( ⌈ log( R/r ) ⌉ +1) /β ) ) , we have P ( G 1 ) ≥ 1 -β.

Therefore,

This proves the first claim.

Regarding the second claim, define the following event

Notice that in the proof of P ( θ ⋆ ∈ Θ loc ) we only used the fact that with a high probability ∆ γn ( θ ⋆ ) 1 4 ≤ ˆ ∆ . Since G 2 ⊆ G 1 , we can write

where the last step follows from Part 3 of Theorem 2.4 which states that P ( G 2 ) ≥ 1 -5 β/ 4 .

E Proof of Section 3

Proof of Theorem 3.1. For the privacy proof, notice that Algorithm 3 uses the training set in Line 2 and Line 5. By the privacy budget allocation and the composition properties of zCDP in [BS16], it is immediate to see that the output satisfies ρ -zCDP.

Next, we prove the utility properties. Define the following event

Also, by the non-negativity of ∥·∥ 2 , we have

Using this inequality, for every θ , we can write

Under the event that θ ⋆ ∈ Θ 0 , we can invoke Lemma A.1 to write

where it follows because the internal randomness of DPGD is independent of the randomness in Localization step.

By the definition of event G 1 , either ˆ ∆ = r or ˆ ∆ ≤ 4∆ 0 . 8 n ( θ ⋆ ) . Note that if ˆ ∆ ≤ 4∆ 0 . 8 n ( θ ⋆ ) , we can use Equation (26) to provide a multiplicative guarantee. Therefore, conditioned on the event G 1 , we have

The first statement then follows because, from Theorem 2.7, we have P ( G 1 ) ≥ 1 -β .

For the second statement, under the condition that max i ∈ [ n ] | X ( n ) ∩B d ( x i , r ) | < 3 n/ 4 , we can define the following high probability event:

The argument then proceeds in the same way as the argument for the first claim.

Proof of Lemma 3.2. We can write π ( · ; X ( n ) )

It follows because F ( θ ⋆ ; X ( n ) ) is a constant independent of i . Then, the first claim follows from the standard utility analysis of the exponential mechanism in [MT07].

In the next step, we provide the proof for the second claim. To this end, because of Equation (28), we analyze the sensitivity of F ( θ i ; X ( n ) ) -F ( θ ⋆ ; X ( n ) ) for every i ∈ [ k ] . Note that θ ⋆ is a data dependent quantity. Let ˜ X ( n ) be a dataset that differ in one sample from X ( n ) . Also, let θ ⊛ ∈ GM( ˜ X ( n ) ) and assume, without loss of generality, that ˜ X ( n ) = ( x 1 , . . . , x ′ n ) and X ( n ) = ( x 1 , . . . , x n ) . For a fixed θ ∈ { θ 1 , . . . , θ k } , we can write

Here, the last two steps follow from the triangle inequality. Note that θ ⊛ is the geometric median of ˜ X n . Therefore by the first-order optimally condition, we have

Using the first-order convexity condition applied to the function h ( θ ) = ∥ θ -x ∥ for a fixed x , we can write

where the second step follows from Equation (30) and the last step follows because ∥∇ ( ∥ θ ⊛ -x n +1 ∥ ) ∥ ≤ 1 . Therefore, using Equation (29) and Equation (31), we have

[ F ( θ ; X ( n ) ) -F ( θ ⋆ ; X ( n ) ) ] -[ F ( θ ; ˜ X ( n ) ) -F ( θ ⊛ ; ˜ X ( n ) ) ] ≤ 2 ∥ θ -θ ⋆ ∥ +2 ∥ ∥ θ ⋆ -θ ⊛ ∥ ∥ . In the next step of the proof, we invoke Lemma C.4 to upperbound the sensitivity as follows 2 ∥ θ -θ ⋆ ∥ +2 ∥ ∥ θ ⋆ -θ ⊛ ∥ ∥ ≤ 2 diam +3∆ 3 n/ 4 ( θ ⋆ ) ≤ ∆ , where the last step follows because ∥ θ -θ ⋆ ∥ ≤ diam by the assumption. Notice that the sensitivity analysis in the reverse direction is also the same. Therefore, the second claim follows from the standard analysis of the privacy of the exponential mechanism.

Proof of Theorem 3.3. The privacy proof of Algorithm 4 is relatively non-standard. Let A 1 ( X ( n ) ) = ( ˆ ∆ , { θ i } i ∈{ 0 ,...,k ft -1 } ) . Also, let A 2 ( X ( n ) ; ( ˆ ∆ , { θ i } i ∈{ 0 ,...,k ft -1 } )) = ˆ t . In particular, A 1 ( X ( n ) ) can be viewed as the first part of Algorithm 4 before Line 10. Also, A 2 ( · ; · ) denotes the exponential mechanism in Line 10 of Algorithm 4. Using the conversion between zCDP and DP, the privacy budget allocation, and the composition properties of zCDP, we have that A 1 ( · ) satisfies ( ε/ 2 , δ/ 2) -DP.

Define the following event

Let µ be a measure on M 1 ( R × ( R d ) k ft ) that satisfies the following: for every dataset X ( n ) , P ( A 1 ( X ( n ) ) ∈ · ) ≪ µ ( · ) . Let P 1 denote the density. Since A 1 satisfies approximate-DP, we assume for every z ∈ R × ( R d ) k ft , we have P 1 ( z ; X ( n ) ) ≤ exp( ε/ 2) P 1 ( z ; ˜ X ( n ) ) + δ/ 2 .

To prove the requirement of privacy, let S ⊆ R × ( R d ) k ft ×{ 0 , . . . , k ft -1 } . Also, let ˜ X ( n ) be a dataset of size n that differs in one sample from X ( n ) . Then, we can write

Notice that under the event that ( ˆ Θ , θ 0 ) ∈ G , we can invoke Lemma 3.2 to reason about the privacy properties of A 2 . Under the event G , we can see that 3∆ 3 n/ 4 ( θ ⋆ ) + 2 diam (Θ 0 ) ≤ 112 ˆ ∆ . Therefore, by Lemma 3.2, we have

Moreover, by Theorem 2.7, we have P ( G c ) ≤ δ/ 2 . Therefore,

Then, we use the fact that A 1 ( X ( n ) ) satisfies ( ε/ 2 , δ/ 2 ):

It concludes the proof.

Proof of Theorem 3.4. Define the following event

By Theorem 2.7, and the assumption on the minimum number of samples, we have P ( G ) ≥ 1 -β .

By applying the standard tail bound on the norm of a Gaussian random vector (as outlined in Corollary C.3), we have

Also, we can write

where the last step follows because P ( G c 1 ) ≤ β . Conditioned on G 1 , for all t ∈ { 0 , . . . , k ft -1 } , we have ∥ θ t -θ ⋆ ∥ ≤ 50 ˆ ∆ since θ ⋆ ∈ Θ 0 , θ t ∈ Θ 0 , and the diameter of Θ 0 is 50 ˆ ∆ . Also, notice that ξ dir ,t ⊥ ⊥ ( θ t , Θ 0 ) . Using these observations, conditioned on the event G 1 , using the standard tail bound on Gaussian random variable (as outlined in Lemma C.1), we can write

Therefore, we conclude

To prove the claim regarding the suboptimality gap, we consider two cases:

There exists t ∈ { 0 , . . . , k ft -1 } such that θ ⋆ ∈ Θ t and θ ⋆ / ∈ Θ t +1 ,

θ ⋆ ∈ Θ k ft .

Note that these two events are mutually exclusive and their union covers all the space. In what follows, we show that in both cases there exists t ∈ { 0 , . . . , k ft -1 } such that F ( θ t ; X ( n ) ) has a small excess loss.

For the first case, suppose t be such that θ ⋆ ∈ Θ t and θ ⋆ / ∈ Θ t +1 . Therefore, we can write

Notice that using the first-order convexity condition, we have F ( θ t ; X ( n ) ) -F ( θ ⋆ ; X ( n ) ) ≤ 〈 ∇ F ( θ t ; X ( n ) ) , θ t -θ ⋆ 〉 . Therefore, by Equation (35), we have

Under the events G 1 and G n, 2 , defined in Equations (32) and (34), we have

For the second case, i.e., θ ⋆ ∈ Θ k ft , we have the following geometric fact [Nes98]: there exists t ∈ { 0 , . . . , k ft -1 } such that the distance of θ ⋆ and the separating hyperplane at time t satisfies

Here ν is a constant such that ν d ≥ exp( -τk ft )(25 ˆ ∆) d . The values of ν and k ft will be determined later. Using the first order convexity, we can write

where the second step follows from the well-known inequality ∥ a + b ∥ ≤ 2 ∥ a ∥ +2 ∥ b ∥ for every a, b ∈ R d . Then, the last step follows because for every θ ∈ R d , ∥ ∥ ∇ F ( θ ; X ( n ) ∥ ∥ ≤ n . Therefore, under the events G 1 , G n, 1 , and G n, 2 , defined in Equations (32) to (34), we have the following bound on the suboptimality gap

Recall that ν satisfies ν d ≥ exp( -τk ft )(25 ˆ ∆) d . It can be easily seen that by setting

under the events G 1 , G n, 1 , and G n, 2 , we can further upperbound Equation (39) as follows

Therefore, from Equations (37) and (40), under the event G 1 ∩G n, 1 ∩G n, 2 , for both cases we showed that there exists t such that

By the non-negativity of ∥·∥ 2 , we have

Therefore, we conclude that for both cases there exists t such that

Let us define OPT ≜ min t ∈{ 0 ,...,k ft -1 } { F ( θ t ; X ( n ) ) -F ( θ ⋆ ; X ( n ) ) } . In the next step of the proof, we show that the exponential mechanism in Line 10 with high probability can identify an iterate whose suboptimality gap is close to OPT. Using the properties of the exponential mechanism in Line 10 as outlied in Lemma 3.2, we have with probability at least 1 -β/ 3 over the randomness of the exponential mechanism

Notice that under the event G 1 and using Equation (42), we have

Moreover, under the event G 1 ∩ G n, 1 ∩ G n, 2 , we provided an upperbound on OPT in Equation (43). Combining Equation (43), Equation (44), and Equation (45), proves the first claim.

For the second statement, under the condition that max i ∈ [ n ] | X ( n ) ∩B d ( x i , r ) | < 3 n/ 4 , we can define the following high probability event:

The argument then proceeds in the same way as the argument for the first claim.

F Proof of Section 4

Proof of Lemma 4.2. For i ∈ [ n -k ] , we can write

Also for every j ∈ [ k ] , we have

Summing both sides of these inequalities, we obtain

Since ∑ n -k i =1 ∥ θ k -x i ∥ + ∑ k j =1 ∥ θ k -y j ∥ -[ ∑ n -k i =1 ∥ θ 0 -x i ∥ + ∑ k j =1 ∥ θ 0 -y j ∥ ] ≤ 0 by the assumption that θ k = GM( x 1 , . . . , x n -k , y 1 , . . . , y k ) , we obtain

Proof of Theorem 4.1. We claim that for every neighbouring datasets X ∈ ( B d ( R )) n and X ′ ∈ ( B d ( R )) n and for every y ∈ B d ( R ) , we have

This follows from the fact that for every ˜ X , we have d H ( X , ˜ X ) ≤ d H ( X ′ , ˜ X ) + 1 . Then, the proof of privacy follows from the privacy proof of the exponential mechanism [MT07].

Next, we present the utility proof. Let k ∈ N be a constant that determined later. Define the following two sets:

We can use the following simple facts: ∫ y ∈ A 1 dy ≤ ∫ y ∈B d ( R ) dy = V 1 R d where V 1 is the volume of the ball of radius one in R d . For A 2 notice that, for all y ∈ B d (GM( X ) , r ) , we have len r ( X , y ) = 0 . Thus, ∫ y ∈ A 2 dy ≥ V 1 r d . Putting these two pieces together,

where the last step follows from the fact that P ˆ θ ∼ π ( ˆ θ ∈ A 2 ) ≤ 1 . Therefore, we obtain that for every β ∈ (0 , 1) with probability at least 1 -β we have

Then, where ˆ θ ∼ π .

Under the above event, let ˆ θ ∈ B d ( R ) be such that len r ( X , ˆ θ ) ≤ k ⋆ . This is equivalent to the following: there exists z ∈ B d ( ˆ θ, r ) and ˜ X ∈ ( R d ) n such that z = GM( ˜ X ) and d H ( X , ˜ X ) ≤ k ⋆ . Using this observation, we can write

Suboptimality Gap: Let θ ⋆ ∈ GM( X ) , then

Define I ⊆ [ n ] be the indices of the points that X and ˜ X differs. We know that |I| ≤ k ⋆ . Then, we can write

By triangle inequality, we can write

For i ∈ I , let ( ˜ X ) i = x ′ i where ( ˜ X ) i denote the i -th data point in ˜ X . Since GM( ˜ X ) is a geometric median of ˜ X , by the first-order optimality condition

To control A 2 , notice that ∥ θ -x i ∥ is a convex function in θ for every x i . By the first-order convexity condition, for every θ 1 and θ 2 , we have ∥ θ 1 -x i ∥-∥ θ 2 -x i ∥ ≤ ⟨∇ ( ∥ θ 1 -x i ∥ ) , θ 1 -θ 2 ⟩ . Therefore, we can write

Then, by Equation (52),

Finally notice that by Equation (3), for every x ′ i , ∥ ∥ ∥ ∇ θ (∥ ∥ ∥ GM( ˜ X ) -x ′ i ∥ ∥ ∥ )∥ ∥ ∥ ≤ 1 . Therefore, by Cauchy-Schwarz inequality

By Equations (51) and (54), we obtain

Then, we invoke Lemma 4.2 which states that ∥ ∥ ∥ GM( ˜ X ) -GM( X ) ∥ ∥ ∥ ≤ 2 n -2 k ⋆ · F (GM( X ); X ) . Putting all the pieces together,

as was to be shown.

as was to be shown.

𝜃 ⋆

Δ𝛾𝑛(𝜃 ⋆

)

𝜃

𝜃 - 𝜃 ⋆

𝑥𝑖

Distance to θ ⋆ : From Equation (49), we know that ∥ ∥ ∥ ˆ θ -GM( X ) ∥ ∥ ∥ ≤ ∥ ∥ ∥ GM( ˜ X ) -GM( X ) ∥ ∥ ∥ + r where ˜ X is a dataset of size n such that d H ( X , ˜ X ) ≤ k ⋆ . The proof is based on characterizing the worst case distance between the geometric median of two datasets that differ in at most k ⋆ points.

For the dataset X ( n ) , recall that θ ⋆ = GM ( X ( n ) ) . Also, recall the definition of ∆ γn ( θ ⋆ ) from Definition 1.1. Let θ ∈ R d be such that ∥ θ -θ ⋆ ∥ > ∆ γn ( θ ⋆ ) . Define m = | ˜ X ∩ B d ( θ ⋆ , ∆ γn ( θ ⋆ )) | . By the variational representation of ∥·∥ , we can write

where the last step follows from Cauchy-Schwarz inequality. Then, we claim that for every x ∈ ˜ X ∩ B d ( θ ⋆ , ∆ γn ( θ ⋆ )) , we have

To gain the intuition behind it see Figure 1. Therefore, from Equation (56),

We are interested on characterizing the condition under which ∥ ∥ ∥ ∇ F ( θ ; ˜ X ) ∥ ∥ ∥ > 0 . A sufficient condition is that given n < 2 m

This shows that the distance of GM( ˜ X ) and θ ⋆ has to satisfy

The function h ( x ) = 1 √ 2 x -x 2 is decreasing in the range of x ∈ (0 , 1] . Also, notice that m = | ˜ X ∩ B d ( θ ⋆ , ∆ γn ( θ ⋆ )) | ≥ γn -k ⋆ . Therefore,

G Proof of Section 5

Proof of Theorem 5.1. The proof is based on the reduction provided in Lemma G.1 and the lowerbound on the sample complexity of the mean estimation of Gaussian distribution with known covariance matrix in [KLSU19, Thm. 6.5].

Lemma G.1. Let ε ≤ 49 × 10 -5 , α ≤ 49 × 10 -5 , δ ≤ 10 -4 , and d ≥ 22500 be constants. Let A n be an arbitrary ( ε, δ ) -DP algorithm such that for every dataset X ( n ) , its output satisfies

Let µ ∈ B ∞ d (1) and let X ( n ) = ( X 1 , . . . , X n ) ∼ N ( µ, I d ) ⊗ n . Let ˆ θ ∼ A n ( X ( n ) ) . Then, with probability at least 2 / 3 over X ( n ) and the internal randomness of A n , we have

Proof. The proof consists of several steps:

Step 1: Bound on the Empirical Error. Let A n be an arbitrary ( ε, δ ) -DP algorithm such that for every dataset X ( n ) , its output satisfies

Let µ ∈ B ∞ d ( R ) and X ( n ) = ( X 1 , . . . , X n ) ∼ N ( µ, I d ) ⊗ n . The utility guarantee of the algorithm implies that

To further upperbound the last step, we can use Jensen's inequality to write

Therefore, in-expectation over X ( n ) ∼ N ( µ, I d ) ⊗ n and the internal randomness of A n , we have

Step 2: Relating Empirical Error to Population Error. Let ( X 0 , X 1 , . . . , X n ) ∼ N ( µ, I d ) ⊗ ( n +1) . With an abuse of notation, let θ = A n (( X 1 , . . . , X n )) , and, for every i ∈ [ n ] , let θ ( i ) = A n (( X 1 , . . . , X i -1 , X 0 , X i +1 , . . . , X n )) . Let T be a constant that will be determined later. We can write

Consider the first term in Equation (59). Since A n satisfies ( ε, δ ) -DP,

Therefore, the first term can be upperbounded as

In the next step, we upperbound the the second term in Equation (59). Notice that

where the first step follows from the triangle inequality and the last step follows because µ and θ ( i ) are in B ∞ d ( R ) . Using this, we can write

√

where the last step follows from the change of variable u = t -(2 R +1) d . In the next step, we use the concentration bounds for the norm of multivariate Guassian random variable. Using Lemma C.2, we can write

Let T = 2(2 R +1) √ d . Then, using standard bounds on the complementary error function [Ksc17], we can write

In the last step, we claim that E [ ∥ θ -X 0 ∥ ] = E [∥ ∥ θ ( i ) -X i ∥ ∥ ] for every i ∈ [ n ] . It is because θ ( i ) d = θ, X i d = X 0 , and θ ( i ) ⊥ ⊥ X i . Ergo, combining and summing over i ∈ [ n ] , we obtain

This bound implies that

This equation can be rephrased as follows

where

Step 3: Relating Population Error to Distance In Step 2, we showed that in-expectation over ( X 0 , . . . , X n ) ∼ N ( µ, I d ) ⊗ ( n +1) and ˆ θ ∼ A n ( X ( n ) ) where X ( n ) = ( X 1 , . . . , X n ) , we have

For notiational convenience, let h : R d → R be h ( θ ) ≜ E X ∼N ( µ, I d ) [ ∥ θ -X ∥ ] . Equation (67) can be written as

Since µ is the minimizer of h ( θ ) , for every θ ∈ R d we have that h ( θ ) ≥ h ( µ ) . Therefore, h ( ˆ θ ) -h ( µ ) is a non-negative random variable. We can invoke Markov's inequality to write

In the next step, we provide a deterministic argument: For every θ ∈ R d such that h ( θ ) -h ( µ ) ≤ 3 β √ d , we provide an upperbound on ∥ θ -µ ∥ . By subtracting µ , we can write

Define the following events

Using Corollary C.3 and simple concentration bound for Gaussian random variable we have that P ( E 1 ∩ E 2 ) ≥ 1 -γ . Let E = E 1 ∩ E 2 . By dropping the positive term, we can write

Using Cauchy-Schwarz inequality, E [ ∥ Z ∥ · 1 [ E c ]] ≤ √ P ( E c ) √ E [ ∥ Z ∥ 2 ] = √ P ( E c ) √ d ≤ √ γ √ d . In the next step, we analyze the first term.

The value of γ will be determined later. Let d be large enough such that

Then, we can write

Notice that we assumed that h ( θ ) -h ( µ ) ≤ 3 β √ d . Therefore, we have

Simple calculations show that this bound implies that

We would like to set the parameters such that √ 1 . 1 d 1 + 3 β + √ γ √ 0 . 9 ) 2 -1 ) = 0 . 1 √ d . We can easily see that it implies 3 β + √ γ = 0 . 0045 . For example, we can pick β = 0 . 0014 and γ = 9 × 10 -8 Recall that

.

For instance, by setting ε ≤ 49 × 10 -5 , α ≤ 49 × 10 -5 , δ ≤ 2 3 × 10 -4 and d ≥ 2 , we obtain β ≤ 0 . 1 . Finally, we need to set d such that Equation (73) holds. We can see that d ≥ 7050 satisfies this condition.

H Details of the Numerical Experiment

Our goal in the experiments is to evaluate the impact of increasing the radius of the initial feasible set, i.e. R , on the performance of our proposed algorithm and compare it with DPGD. Also, we want to show that our method without any additional hyperparmeter tuning can achieve a good excess error.

Data Generation. Let n denote the number of samples. We assume that 0 . 9 n of the data is distributed as follows: let µ ∈ R d be a uniformly random vector within S d -1 (50) . We then sample 0 . 9 n of the data points from N ( µ, (0 . 01) 2 · I d ) . The remaining 0 . 1 n of the points are sampled uniformly at random from B d (100) .

Hyperparameters. We set the discretization parameter to r = 0 . 05 in Algorithm 2 and failure probability to 5% . Additionally, we repeat each algorithm 10 times and report the mean. For the other hyperparameters, we used exactly the same hyperparameters as stated in Algorithm 3. For DPGD, we use the hyperparameters in Lemma A.1, and in particular, we choose T such that √ 2 √ T = 16 √ d n √ ρ .

10 3

10 4

10 5

10 6

10 7

10 8

10 9

10 10

R

10 0

10 1

10 2

10 3

10 4

10 5

10 6

F

(

;

X

(

n

)

)/

F

(

;

X

(

n

)

)

d

=200,

n

=3000,

=2,

=1/

n

Ours

10 3

10 4

10 5

10 6

10 7

10 8

10 9

10 10

R

10 0

10 1

10 2

10 3

10 4

10 5

F

(

;

X

(

n

)

)/

F

(

;

X

(

n

)

)

d

=200,

n

=3000,

=3,

=1/

n

Ours

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes algorithms that compute the geometric median with differential privacy. Composed of two parts, the algorithm firstly finds the quantile radius that covers enough points. Secondly, it fine-tunes the geometric median. The authors suggest two methods for the second part: LocDPGD and LocDP cutting plane methods.\n\nAdditionally, the paper shows a pure DP algorithm for computing the geometric median and a lower bound on the sample complexity required to compute the geometric median with differential privacy.\n\nSTRENGTHS:\nThis paper studies a well-motivated problem. As mentioned in the paper, a line of works has studied computing private versions of the medians, but little is known about the world when the dimension is larger than one. \n\nThis problem is difficult because the runtime of applying vanilla DP-GD is highly affected by outliers. This paper uses two steps: first, it estimates the quantile radius and an initial point that is not greatly affected by outliers, and then it fine-tunes the computation of the result. This idea is intuitive but requires some good techniques. For example, the way they implement the DP cutting plane method by adding Gaussian noise to the gradient that \"cuts\" the set seems novel and interesting to me.\n\nThis paper is also very complete in the sense that it shows results in pure DP setting and lower bound in sample complexity. \n\nFrom the writing perspective, this paper is well-organized and straightforward to read.\n\nWEAKNESSES:\nIt is a bit difficult to understand the dependency between parameters in Thm 3.1 and Thm 3.4. It would be very helpful if the authors could make Thm 2.7, 3.1, and 3.4 more readable and easier to compare.\n\nQUESTIONS:\nI am wondering if the authors can explain a bit more about the comparison between LocDPGD and LocDP cutting-plane. For example, you could compare the time complexity and explain under what circumstances one is better than the other. \n\nI am also wondering what part is the bottleneck if someone wants to improve this to a linear run-time.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a pair of polynomial-time DP algorithms for computing the geometric median of a dataset. The excess error guarantees of the algorithm scale with the effective radius. \n\nThe algorithm includes two parts. The first part shrinks the feasible set to a ball whose diameter is proportional to the quantile radius of the optima, which is defined as the radius of the smallest ball containing sufficiently many points. First, it gives a private estimation of the quantile radius. Then, it locates a good initializer based on the estimated quantile radius. AboveThreshold and DP-GD are implemented in the procedure to ensure the approximated DP. \n\nThe second part fine-tunes the output of the first part. The paper provides two fine-tuning methods. The first one utilizes DP-GD. The second one modifies the cutting plane method. It adds Gaussian noise to achieve DP. Using a novel sensitivity analysis of the loss function, the authors show that the noise can be scaled down.\n\nThis paper also provides the lower bound to demonstrate the optimality of the sample complexity of their algorithm.\n\nSTRENGTHS:\n1. Quality of result:\nThis paper studies the DP convex optimization task. DP-GD method is well-established in this field. The authors point out that its excess error depends linearly on the radius R of the feasible set. Previous work shows the strongly convex assumption can remove this deficiency, but it is too strong and unrealistic. \nThis paper improves the dependence on the radius for the geometric median task under weak and natural assumptions. The main contribution of this work is a pair of polynomial-time DP algorithms (with different run-time) for geometric median estimation with an excess error linearly depending on the effective radius r of the dataset, instead of the worst-case radius R. This significantly improves the error bound when r << R. This condition is common since the dataset may have an enormous R due to a small number of outliers.\nOther contributions include a pure DP algorithm based on the inverse smooth sensitivity mechanism (yet it is computationally inefficient), and the demonstration of the sample lower bound, which verifies the optimality of the algorithm.\n\n2. Novel techniques: The algorithm's good error bound is based on a structural result stated by the paper: Given a point \\theta, if its distance to the optima \\theta^* is larger than the quantile radius \\Delta_{3n/4}(\\theta^*}, it has a linear growth in its loss value: F\\left(\\theta ; \\mathbf{X}^{(n)}\\right)-F\\left(\\theta^{\\star} ; \\mathbf{X}^{(n)}\\right) \\gtrsim\\left|\\theta-\\theta^{\\star}\\right|. This simulates the effect of strongly-convexity, which has a quadratic growth. This implies that one can take a larger step size when performing the DP-GD. It will move fast towards a sub-optimal point. To utilize this property, the authors build a private estimation algorithm to find this radius quantile. Then, the algorithm fine-tunes the sub-optimal point to find a good estimation of the geometric mean. This paper states a modified DP-SGD algorithm to do this task. To reach a better run-time under certain conditions, this paper also provides a private cutting plane method by adding local noise.\n\n3. Technically soundness: This paper is sound technical with solid proof and analysis. It has an informative technical overview. It also includes a numerical experiment to strengthen their result. It compares their algorithm with DPGD in solving the geometric mean task. Under large R, their algorithm shows significant improvement upon DPGD.\n\n4. Reproducibility: The quantile radius technique in the warm-up algorithm is of independent interest. It may be utilized to solve other DP convex optimization tasks. The authors also state the open problem of developing a linear time algorithm with the optimal excess error.\n\nWEAKNESSES:\nMore words can be added in Section 1.1 to introduce the intuition of their solution (e.g., how to solve the challenge in line 122?). A comparison of run-time and excess risk with methods from previous work may also be helpful.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

comments: SUMMARY:
This paper considers the differentially private geometric median problem. The goal of the geometric median problem is given n data points, find a point to minimize the sum of Euclidean distances from this point to all data points. 

The previous DP-GD algorithm requires prior knowledge of the radius R of the data set and also has the excess risk linearly depending on this radius R. 

This paper provides three algorithms for the differentially private geometric median. They provide LocDPSGD and LocCuttingPlane algorithms which achieve approximate DP and the excess risk bound depends on the optimal cost instead of the radius R. Finally, they provide an exponential time algorithm that achieves pure DP.

STRENGTHS:
1 This paper considers the differentially private geometric median problem, which is an important problem since the geometric median is widely used in many algorithms and requires privacy for many real-world applications. 

2 This work provides interesting techniques and algorithms to achieve the excess risk bound depending on the optimal cost instead of the radius of data points R. They developed a private estimation of quantile radius and private localization for the warm-up stage such that the excess risk does not depend on the radius R.

WEAKNESSES:
-

QUESTIONS:
-

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies DP algorithms for geometric median. The paper starts with DP-SGD on geometric median as the baseline. The error scales linearly in R, the radius of a ball containing all data points. This can be a very loose bound since the geometric mean is known to be robust to outliers and one single outlier can make this radius R to be arbitrarily large. \n\nThe algorithms proposed in this paper has both multiplicative error and additive error. The idea is to replace the estimate of R by the quantile radius -- the value \\Delta such that a ball of radius \\Delta has at least a certain fraction of the points. This comes with a cost of a multiplicative error. \n\nThe algorithm has two phases, with the first phase to estimate the quantile radius in a private manner and the second phase to find the geometric median. The first phase uses techniques from [NSV16]. For the second phase the paper reports several ideas: 1) just apply DP-SGD using the quantile radius; 2) uses a cutting plane method (non-gradient descent method) and 3) for pure DP uses inverse smooth sensitivity of [AD20] -- this one is not efficient in terms of running time.\n\nIn summary I think this paper did an OK job but the ideas are in general incremental. I would give a positive rating but I am not extremely enthusiastic (or lacks excitement).\n\nSTRENGTHS:\nThe paper has merits of developing DP algorithms for geometric medians. The algorithms have both multiplicative error and additive errors. Adding these algorithms to the repository of DP algorithms for this problem is nice.\n\nWEAKNESSES:\nThe idea of using \"robust\" notion of radius, essentially the quantile radius, is not new and has been a standard in many prior work on clustering algorithms -- search clustering with outliers. It will be good to include citation and discussion on that.  \n\nThe writing can be improved. There are many places where clarification and definition will be helpful to readers. Some examples are mentioned below.\n\nOn DP-SGD, may you explain what is the definition of \"excess error\" (line 23)? Is this the same as excess risk (line 45)?\n\nEquation (2), what is A_n? the DP -SGD algorithm? It is never defined.\n\nLine 63, effective diameter of the data points -- it will be good to explain what effective diameter means.\n\nTable 1, define lower case r? Again r does not appear until much later in the paper.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies differentially private algorithms for computing geometric median (GM) of a dataset.  Previous methods such as DP-SGD requires knowing in advance that all data points live in a ball of radius R (contribution bounding), and the resulting utility guarantee also has a dependency on that R, which can get very big even though the majority of the points are concentrated around the GM. The paper proposes three algorithms, two for approximate DP and one for pure DP. For approximate DP, we run a pre-step for finding a quantile radius which covers the majority of the points and then find a good initialization point using this radius. From there we can either run DP-SGD or using an DP-adapted cutting plane algorithm. For pure-DP the authors describe a discretized inverse-sensitivity mechanism.\n\nSTRENGTHS:\nThe problem solved in this paper is a very interesting problem, that is important in practice, and the solution provided in this paper provides the first performance guarantee without the dependency on the spread of dataset that can be achieved with an implementable, polynomial-time algorithm, which is impressive.\n\nWhile developing the algorithms the paper made many observations that are insightful and could be worth knowing for the community. I think the way it relates the quantile radius around the GM to the average of top m quantile radii of all points in the dataset is clever, and so is the relationship between distance from GM and median cost expressed in terms of quantile radius. \n\nAlthough there are certain complicated aspects of algorithms, the overall quality of writing is good.\n\nWEAKNESSES:\nI found the pure-DP algorithm not as interesting as the approximate one for a couple of reasons: (1) it requires an oracle finding exact GM, which is not known, so I'm wondering what happens if we are given an approximation but then the definition of $len_r(X, y)$ is messed up; (2) the paper does not talk about how to compute $len_r(X, y)$; which I think is non-trivial? (3) is the purpose of introducing $r$ mostly to discretize the original inverse sensitivity mechanism? what is the benefit of using $r$ compared to $r \\to 0$?\n\nNumerical setting is relatively simple but I don't really mind that since the greatest merit of this paper is in the theory.\n\nQUESTIONS:\nIt seems that the authors have used $\\rho-zCDP$ and $(\\epsilon, \\delta)$-DP interchangeably but without providing preliminaries on the connection between the two? It's better to unify.\n\nFor others, see the weakness part.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}