Title: Exactly Minimax-Optimal Locally Differentially Private Sampling
Venue: NeurIPS 2024
Accepted: yes
Keywords: Local differential privacy

Sampling

Privacy-utility trade-off
Abstract: The sampling problem under local differential privacy has recently been studied with potential applications to generative models, but a fundamental analysis of its privacy-utility trade-off (PUT) remains incomplete. In this work, we define the fundamental PUT of private sampling in the minimax sense, using the $f$-divergence between original and sampling distributions as the utility measure. We characterize the exact PUT for both finite and continuous data spaces under some mild conditions on the data distributions, and  propose sampling mechanisms that are universally optimal for all $f$-divergences. Our numerical experiments demonstrate the superiority of our mechanisms over baselines, in terms of theoretical utilities for finite data space and of empirical utilities for continuous data space.

Exactly Minimax-Optimal Locally Differentially Private Sampling

Abstract

The sampling problem under local differential privacy has recently been studied with potential applications to generative models, but a fundamental analysis of its privacy-utility trade-off (PUT) remains incomplete. In this work, we define the fundamental PUT of private sampling in the minimax sense, using the f -divergence between original and sampling distributions as the utility measure. We characterize the exact PUT for both finite and continuous data spaces under some mild conditions on the data distributions, and propose sampling mechanisms that are universally optimal for all f -divergences. Our numerical experiments demonstrate the superiority of our mechanisms over baselines, in terms of theoretical utilities for finite data space and of empirical utilities for continuous data space.

1 Introduction

Privacy leakage is a pressing concern in the realm of machine learning (ML), spurring extensive research into privacy protection techniques [1-4]. Among these, local differential privacy (LDP) [5] stands out as a standard model and has been deployed in industry, e.g., by Google [6, 7], Apple [8], Microsoft [9]. In the LDP framework, individual clients randomize their data on their own devices and send it to a potentially untrusted aggregator for analysis, thus preventing the user data from being inferred. However, this perturbation inherently diminishes data utility. Consequently, the central challenge in privacy mechanism design lies in optimizing utility while preserving the desired level of privacy protection. This goal involves characterizing the optimal balance between privacy parameter and utility, referred to as the privacy-utility trade-off (PUT). The analysis of the PUT and the proposal of privacy mechanisms have been actively conducted for various settings of statistical inference and machine learning [10-27].

Most research in this field focuses on scenarios where each client has only a single data point. However, there are increasingly more applications where each client has a large local dataset with multiple data records. One can formulate the privacy requirement in these cases by assuming that clients have datasets of the same size, generated independently from an underlying distribution [2834]. This probabilistic assumption, however, restricts practical flexibility. The work [35] explored a scenario where clients have large datasets that may vary in size and seek to privately release another dataset that closely resembles their original dataset. In this scenario, local datasets can be represented by an empirical distribution, allowing each client to be seen as holding a probability distribution and

2

0

2

-

2

0

2

Original dist.

2

0

2

Sampling dist.

[35]

2

0

2

Sampling dist.

(ours)

-

-

-

generating a private sample from it. This setup, which is called private sampling , is the main focus of this paper.

Private sampling has recently found applications in the private fine-tuning of large language models [36]. Additionally, private sampling is connected to the challenge of learning private generative models, a topic often explored in the central DP model [37-44]. While there exist studies on private generative models within the local model [45-48], all these works assume a single data point per client. Very recently, the work [49] considered a setup where each client holds a probability distribution, but in a different context of query estimation.

The private sampling mechanism in [35] can be described as follows. Initially, given a probability distribution P representing a local dataset, the mechanism assumes a fixed reference distribution. It then constructs what is termed the 'relative mollifier', a closed ball centered around the reference distribution with a radius equivalent to half of the privacy budget, within the space of probability distributions. Subsequently, the mechanism computes the projection of P onto the relative mollifier, utilizing the Kullback-Leibler (KL) divergence. This projected distribution serves as the sampling distribution for generating a sample (see Section 2.3 for more details). However, this mechanism has a notable shortcoming: the sampling distribution is only locally optimal within the relative mollifier. This, in turn, implies that the optimality of the sampling distribution depends on the choice of the reference distribution. A more fundamentally intriguing goal would be to formulate and characterize the PUT without such an ambiguity in the choice of reference distribution.

In this paper, we establish the optimality of locally private sampling in the minimax sense, and identify optimal private samplers. Our primary contributions are summarized as follows:

The fundamental PUT of private sampling is rigorously defined in terms of minimax utility, which is commonly used in the literature of private estimation [10, 11, 13, 30]. We impose some minimal assumptions on client's distributions as in [38] (which studies sampling under central DP, a weaker privacy model than the local model [50]). For utility measure, we use the f -divergence [51, 52] between the original and the sampling distributions, that includes KL divergence, total variation distance, squared Hellinger distance, and χ 2 -divergence as special cases.

Wecharacterize the exact PUT for both finite and continuous data spaces, and present optimal sampling mechanisms achieving the PUT. Surprisingly, our mechanisms are universally optimal under any choice of f -divergence for utility measure.

We numerically demonstrate that our proposed mechanism outperforms the baseline method presented in [35]. Specifically, for finite data spaces, we derive a closed-form expression for the utility of both our mechanism and the baseline, allowing for an exact comparison of their utilities. In the case of continuous data spaces, a closed-form expression for the baseline is not available, so we use empirical utility for the comparison. Figure 1 illustrates our proposed mechanism outputs a distribution closer to the original, than the baseline.

All codes for experiments and figures are attached as a supplementary material, and can be found at the online repository 1 . The instructions to reproduce the results in the paper are in Appendix H.

2 Problem Formulation

2.1 Notations and preliminaries

Notations. For a sample space X , let P ( X ) be the set of all probability distributions on X . For each n ∈ N , let C ( R n ) be the set of all continuous probability distributions on R n . For each positive integer k ∈ N , let [ k ] := { 1 , 2 , · · · , k } . For a subset A ⊂ X , ✶ A : X → { 0 , 1 } denotes the indicator function, defined as ✶ A ( x ) = 1 for x ∈ A and ✶ A ( x ) = 0 for x / ∈ A . Also, for s 1 ≤ s 2 and x ∈ R , let clip( x ; s 1 , s 2 ) := max { s 1 , min { s 2 , x }} . We refer to Appendix A for the rigorous measure-theoretic assumptions underlying the paper.

f -divergence. For a convex function f : (0 , ∞ ) → R satisfying f (1) = 0 and two probability distributions P, Q ∈ P ( X ) on the same sample space X , let D f ( P ∥ Q ) denote the f -divergence [51, 52]. The general definition of f -divergence is given in Appendix B. For X = R n and P, Q ∈ C ( R n ) with P ≪ Q (that is, P ( A ) = 0 whenever Q ( A ) = 0 ), it is defined as

where p, q are pdfs of P, Q , respectively, and we define f (0) = lim x → 0 + f ( x ) ∈ ( -∞ , ∞ ] . For finite X , we can replace the integral with the sum and replace p, q with P, Q . Several well-known distance measures between distributions are examples of f -divergence with different convex functions. For instance, KL divergence (relative entropy), total variation distance, squared Hellinger distance, and χ 2 -divergence are f -divergences with f ( x ) = x log x , f ( x ) = | x -1 | / 2 , f ( x ) = (1 - √ x ) 2 , and f ( x ) = x 2 -1 , respectively. Two important properties of general f -divergence are keys for this work. First, D f ( P ∥ Q ) ≥ 0 , and equality holds if P = Q . Furthermore, we have

where equality holds if P and Q are mutually singular (that is, they have disjoint supports). For a more comprehensive list of such f -divergences and their properties, we refer the readers to [52].

We denote the KL divergence and the total variation distance as D KL ( P ∥ Q ) and D TV ( P, Q ) , respectively. We note that the total variation distance is in fact a metric on P ( X ) .

2.2 System model

Suppose a client has access to a distribution P ∈ P ( X ) over a sample space X , and wants to produce a sample in X which looks like being drawn from P and to send it to a data curator. We assume that there are some constraints on the possible data distribution P , so that P is restricted to be in some subset ˜ P ⊂ P ( X ) , and both the client and the curator know X and ˜ P . However, it is required that a sampled element does not leak the privacy about the original distribution P . For this purpose, the client and the curator agree a private sampling mechanism Q , which is a conditional distribution from ˜ P to X . After that, the client produces a sample following the distribution Q ( ·| P ) . To guarantee the privacy protection, we impose Q to satisfy the local differential privacy (LDP) [10, 35].

Definition 2.1. Let ϵ > 0 . A private sampling mechanism Q is said to satisfy ϵ -LDP , or Q is an ϵ -LDP mechanism, if for any P, P ′ ∈ ˜ P and A ⊂ X , we have

For convenience, for each P ∈ ˜ P , let Q ( P ) ∈ P ( X ) denote the distribution of X given P through Q , that is Q ( P )( A ) = Q ( A | P ) for each A ⊂ X . In this way, we equivalently see Q as a function Q : ˜ P → P ( X ) . Let Q X , ˜ P ,ϵ denote the set of all ϵ -LDP mechanisms Q : ˜ P → P ( X ) .

As the utility loss of the private sampling, we use the f -divergence between the original distribution and the sampling distribution, D f ( P ∥ Q ( P )) . Since the sampling procedure can be performed across many clients who may have different data distributions, we measure the utility loss of Q by the worst-case f -divergence ,

Given X , ˜ P , ϵ , and f , our goal is to find the smallest possible worst-case f -divergence,

and to find a mechanism Q ∈ Q X , ˜ P ,ϵ achieving it. We say that Q ∈ Q X , ˜ P ,ϵ is optimal for ( X , ˜ P , ϵ ) under D f if R f ( Q ) = R ( X , ˜ P , ϵ, f ) .

2.3 Related work

The most closely related work to our work is [35]. The system models are the same as this paper, except the formulation of PUT. They first fix a reference probability distribution Q 0 ∈ P ( X ) , and only consider mechanisms Q satisfying e -ϵ/ 2 Q 0 ( A ) ≤ Q ( A | P ) ≤ e ϵ/ 2 Q 0 ( A ) for all P ∈ ˜ P and A ⊂ X . In other words, let M ϵ,Q 0 = { Q ∈ P ( X ) : e -ϵ/ 2 Q 0 ( A ) ≤ Q ( A ) ≤ e ϵ/ 2 Q 0 ( A ) , ∀ A ⊂ X} . Then, they only consider Q such that Q ( P ) ∈ M ϵ,Q 0 . Note that this guarantees ϵ -LDP. For each P ∈ ˜ P , they sought to find Q ∈ M ϵ,Q 0 which minimizes D KL ( P ∥ Q ) , and set this Q to be Q ( P ) . First, they claimed to find a closed-form expression of such a minimizer Q for finite X , given by

where r P > 0 is a constant depending on P ensuring ∑ x ∈X Q ( x ) = 1 . Second, they presented an algorithm, called Mollified Boosted Density Estimation (MBDE), to approximate the optimal solution for continuous X . However, the utility varies over the choice of reference distribution Q 0 , and they left the question of choosing a best Q 0 to achieve the best performance in both practice and theory. Moreover, we found that the closed-form in (6) is incomplete, because for some ( P, Q 0 ) , there may be no r P > 0 such that the RHS of (6) does not sum to one. As an example, when P, Q 0 are point masses at different points, then we can easily see that the sum of (6) is e -ϵ/ 2 for any r P > 0 .

3 Main Results

3.1 Optimal private sampling over finite space

First, we consider the finite case, where X = [ k ] for some k ∈ N . A natural setup for ˜ P is that ˜ P = P ([ k ]) , i.e. there is no restriction on the client distribution P ∈ P ([ k ]) . In this case, we completely characterize the optimal worst-case f -divergence R ( X , ˜ P , ϵ, f ) and find an optimal private sampling mechanism. Surprisingly, for each k ∈ N and ϵ > 0 , we found a single mechanism which is universally optimal for every f -divergence.

Theorem 3.1. For each k ∈ N , ϵ > 0 , and an f -divergence D f , we have

Moreover, the mechanism Q ∗ k,ϵ constructed as below satisfies ϵ -LDP and is optimal for ( X = [ k ] , ˜ P = P ([ k ]) , ϵ ) under any D f :

where r P > 0 is a constant depending on P so that ∑ k x =1 Q ∗ k,ϵ ( x | P ) = 1 . Furthermore, r P can be chosen such that 1 ≤ r P ≤ ( e ϵ + k -1) /e ϵ .

By definition, we have Q ∗ k,ϵ ( x | P ) ≥ 1 e ϵ + k -1 for all x ∈ X . This also implies that Q ∗ k,ϵ ( x | P ) = 1 -∑ x ′ ∈X\{ x } Q ∗ k,ϵ ( x ′ | P ) ≤ 1 -k -1 e ϵ + k -1 = e ϵ e ϵ + k -1 . Hence, 1 e ϵ + k -1 ≤ Q ∗ k,ϵ ( x | P ) ≤ e ϵ e ϵ + k -1 . This clearly implies that Q ∗ k,ϵ satisfies ϵ -LDP.

Behaviors of the optimal mechanism. Let us observe some behaviors of the proposed mechanism with respect to the system parameters, whose formal proofs are in Appendix E. We visualize how the mechanism Q ∗ k,ϵ works for different ϵ in Figure 2. Here, we write R to mean R ([ k ] , P ([ k ]) , ϵ, f )

for simplicity. If f (0) = ∞ , then R = ∞ , which means that R f ( Q ) = ∞ for any ϵ -LDP sampling mechanism Q . (Such a phenomenon happens for general ( X , ˜ P ) , whenever ˜ P contains two mutually singular distributions) Hence, from now on in this paragraph, we assume f (0) < ∞ . We can observe that R is decreasing in ϵ and increasing in k . For a fixed k , we have R→ 0 as ϵ →∞ , which makes sense since ϵ →∞ corresponds to the non-private case. Also, as ϵ → 0 , we have Q ∗ k,ϵ ( x | P ) → 1 /k for every P ∈ P ([ k ]) and x ∈ [ k ] , that is, Q ∗ k,ϵ ( P ) tends to the uniform distribution over [ k ] for every P ∈ P ([ k ]) . This fact can be also observed by Figure 2.

Remarks about the constant r P . The value of r P may not be unique, but the mechanism Q ∗ k,ϵ does not depend on the choice of r P . To see this, let us fix P , and let g r ( x ) = max ( 1 r P ( x ) , 1 e ϵ + k -1 ) . Suppose that ∑ k x =1 g r ( x ) = ∑ k x =1 g r ′ ( x ) = 1 for r ≤ r ′ . Since g r ( x ) ≥ g r ′ ( x ) for each x ∈ [ k ] , the equality ∑ k x =1 g r ( x ) = ∑ k x =1 g r ′ ( x ) implies that g r ( x ) = g r ′ ( x ) for all x ∈ [ k ] . Hence Q ∗ k,ϵ is uniquely determined. Since r ↦→ ∑ k x =1 g r ( x ) is non-increasing and continuous, we can use the bisection method to find r P . The 'Furthermore' part of the theorem statement precisely means that for r = 1 and r = ( e ϵ + k -1) /e ϵ ,

1

2

3

4

5

Data value

x

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

Probability

Original pmf

P

(

x

)

Q

∗

k,/epsilon1

(

x

|

P

),

/epsilon1

= 1

.

0

Q

∗

k,/epsilon1

(

x

|

P

),

/epsilon1

= 0

.

5

Q

∗

k,/epsilon1

(

x

|

P

),

/epsilon1

= 0

.

1

Figure 2: A visualization of the mechanism Q ∗ k,ϵ the value of ∑ k x =1 g r ( x ) is at least and at most 1 , respectively, so that we can perform the bisection method with these two initial endpoints to find r such that ∑ k x =1 g r ( x ) = 1 .

Comparison with the previous work [35]. The expression of the optimal mechanism in (8) is similar to (6), the expression of the KL divergence projection onto M ϵ,Q 0 derived by Husain et al. [35]. Since 1 e ϵ + k -1 ≤ Q ∗ k,ϵ ( x | P ) ≤ e ϵ e ϵ + k -1 , we can alternatively write Q ∗ k,ϵ ( x | P ) = clip ( 1 r P P ( x ); 1 e ϵ + k -1 , e ϵ e ϵ + k -1 ) . Hence, our optimal mechanism can be viewed as an instance of a generalized version of (6), where Q 0 is a positive measure, not necessarily a probability measure summing to one, given by Q 0 ( x ) = e ϵ/ 2 e ϵ + k -1 . A natural question is whether Q ∗ k,ϵ ( P ) is a projection of P onto M ϵ,Q 0 , that is Q ∗ k,ϵ ( P ) is a minimizer of D KL ( P ∥ Q ) among Q ∈ M ϵ,Q 0 , where M ϵ,Q 0 is similarly defined as in Section 2.3. As we shall discuss in Section 3.3, this statement is true -quite surprisingly- even when we replace D KL with any other f -divergences. However, our analysis is more involved, as we need to show the optimality of the proposed mechanism over any other possible mechanisms , including minimizers with respect to other choices of Q 0 . Also, in Section 5, we compare the worst-case f -divergence of our optimal mechanism with that of the mechanism proposed in [35] which restricts Q 0 to be a probability distribution.

3.2 Optimal private sampling over continuous space

Next, we consider the continuous case, where X = R n for some n ∈ N . Some of the natural setups for ˜ P are (i) ˜ P = P ( R n ) , or (ii) ˜ P = C ( R n ) . We can also think some restrictive but still reasonable setups, such as the setups where (iii) ˜ P is the set of empirical distributions supported on some non-empty open subset of R n , or where (iv) ˜ P is the set of continuous distributions on [ -1 , 1] n having smooth pdf and zero mean. However, we show that for a general class of ˜ P including these four cases, any ϵ -LDP sampling mechanisms have the worst-case f -divergence equal to the maximum value M f of the f -divergence defined in (2). In the following proposition, X may be a general sample space, not necessarily R n or finite space. The proof is in Appendix D.

Proposition 3.2. Suppose that ˜ P contains infinitely many distributions which are pairwise mutually singular. Then, for any ϵ > 0 , for any ϵ -LDP mechanism Q ∈ Q X , ˜ P ,ϵ , and for any f -divergence, we have R f ( Q ) = M f , where M f is define at (2) .

Hence, we need to consider sufficiently regular but practical setups for ˜ P .

Our setup. In this subsection, when P, Q ∈ C ( R n ) , the corresponding small letters p, q denote their pdfs. In this paper, we consider the case that

Without loss of generality, we may assume the following normalization condition on c 1 , c 2 , h, ϵ :

for some pre-known h : X → [0 , ∞ ) such that ∫ R n h ( x ) dx < ∞ and c 2 > c 1 ≥ 0 . Some of the sampling tasks and generative models in literature [53, 35] assume that the set of possible data distributions ˜ P satisfies ˜ P ⊂ ˜ P c 1 ,c 2 ,h for some c 1 , c 2 , h satisfying the aforementioned condition, hence (9) is a moderate assumption. One example is the sampling from a Gaussian mixture, which is one of the canonical sampling tasks in literature [53, 35]. Suppose that ˜ P consists of Gaussian mixtures, where each Gaussian has mean within a unit ball centered at the origin and has unit covariance. That is, ˜ P = { ∑ k i =1 λ i N ( µ i , I n ) : k ∈ N , λ i ≥ 0 , ∑ k i =1 λ i = 1 , ∥ µ i ∥ ≤ 1 } , where I n is the identity matrix of size n × n . In this case, we can observe that ˜ P ⊂ ˜ P 0 , 1 ,h for h ( x ) = (2 π ) -n/ 2 exp( -(max(0 , ∥ x ∥ -1)) 2 / 2) , and it can be easily shown that ∫ R n h ( x ) < ∞ .

The reason is as follows. First, if any one of three inequalities ∫ R n h ( x ) dx > 0 , c 1 ∫ R n h ( x ) dx < 1 , and c 2 ∫ R n h ( x ) dx > 1 is not satisfied, then ˜ P c 1 ,c 2 ,h is either an empty set or a singleton that consists of a distribution having pdf c 1 h ( x ) or c 2 h ( x ) , which makes the problem trivial. Hence we impose all of the three inequalities. Then, we can normalize c 1 , c 2 , h , to make ∫ R n h ( x ) dx = 1 and c 1 < 1 < c 2 . Furthermore, if c 2 ≤ e ϵ c 1 , then for any P 1 , P 2 ∈ ˜ P c 1 ,c 2 ,h , we have p 1 ( x ) /p 2 ( x ) ≤ c 2 /c 1 ≤ e ϵ , hence we can easily observe that the mechanism Q defined as Q ( P ) = P for all P ∈ ˜ P c 1 ,c 2 ,h satisfies ϵ -LDP and R f ( Q ) = 0 , hence the problem also becomes trivial, giving R ( R n , ˜ P c 1 ,c 2 ,h , ϵ, f ) = 0 . Hence, we may assume (10).

Minimax utility and optimal mechanism. For the aforementioned setup, we can completely characterize R ( X , ˜ P , ϵ, f ) and find a mechanism which is universally optimal for every f -divergence. The formula is similar to the discrete case, with a carefully chosen clipping bound.

Theorem 3.3. For each c 2 > c 1 ≥ 0 , ϵ > 0 , and h : R n → [0 , ∞ ) satisfying the normalization condition (10) , let us define the following constants determined by c 1 , c 2 , ϵ :

Then, we have

For each P ∈ ˜ P , Q ∗ c 1 ,c 2 ,h,ϵ ( P ) =: Q is defined as a continuous distribution with pdf

Moreover, the mechanism Q ∗ c 1 ,c 2 ,h,ϵ constructed as below satisfies ϵ -LDP and is optimal for ( X = R n , ˜ P = ˜ P c 1 ,c 2 ,h , ϵ ) under any D f :

where r P > 0 is a constant depending on P so that ∫ R n q ( x ) dx = 1 . Furthermore, r P can be chosen such that r 1 < r P ≤ r 2 .

It is also clear that Q ∗ c 1 ,c 2 ,h,ϵ satisfies ϵ -LDP. Also, we note that c 1 < b < 1 < be ϵ < c 2 and 0 ≤ r 1 < 1 < r 2 , which is shown during the proof of Theorem 3.3 in Appendix C.

In practical scenario, it may be hard to expect ˜ P = ˜ P c 1 ,c 2 ,h exactly, and we may only know ˜ P ⊂ ˜ P c 1 ,c 2 ,h for some c 1 , c 2 , h satisfying aforementioned conditions. In such case, we still propose to use Q ∗ c 1 ,c 2 ,h,ϵ , and in Section 5, we numerically show that this proposed mechanism is better than previously proposed mechanism [35] in terms of the worst-case f -divergence.

Behavior of the optimal mechanism. We also observe some behaviors of the proposed mechanism with respect to the system parameters, whose formal proofs are in Appendix E. Again, we write R to mean R ( R n , ˜ P c 1 ,c 2 ,h , ϵ, f ) for simplicity. For a fixed ( c 1 , c 2 ) , R is decreasing in ϵ . If c 1 = 0 (which implies r 1 = 0 ) and f (0) = ∞ , then R = ∞ , which means R f ( Q ) = ∞ for any ϵ -LDP sampling mechanism Q . For the behavior at ϵ →∞ for a fixed ( c 1 , c 2 ) , if c 1 > 0 , then for sufficiently large ϵ , we have c 1 e ϵ ≥ c 2 , so we fall in the aforementioned trivial case that R = 0 . If c 1 = 0 and f (0) < ∞ , then as ϵ →∞ , we have R→ 0 , which again corresponds to the non-private case.

Remarks on the constant r P . By the same reason as in the finite space case, the value of r P may not be unique, but Q ∗ c 1 ,c 2 ,h,ϵ does not depend on the choice of r P , and r P can be found by the bisection method with a numerical integration of (13). Note that the continuity of r ↦→ ∫ g r ( x ) dx, g r ( x ) = clip ( 1 r p ( x ); bh ( x ) , be ϵ h ( x ) ) , follows from the dominated convergence theorem [54] since we assume ∫ R n h ( x ) dx < ∞ . The meaning of 'Furthermore' part is also similar to the finite space case, that is, the value of ∫ g r ( x ) dx at r = r 1 and r = r 2 is at least and at most 1 , respectively, so that we can perform the bisection method with initial endpoints ( r 1 , r 2 ) (When r = 0 , we define g r ( x ) = be ϵ h ( x ) whenever p ( x ) > 0 and g r ( x ) = bh ( x ) whenever p ( x ) = 0 ). A corner case is that when r 1 = 0 , the continuity of r ↦→ ∫ g r ( x ) dx does not suffice to guarantee the existence of strictly positive r such that ∫ g r ( x ) dx = 1 . However, in the proof, we actually show that ∫ g r 1 ( x ) dx = 1 implies ∫ g r ( x ) dx = 1 for every r ∈ ( r 1 , r 2 ] , which especially implies that even when r 1 = 0 , there is a strictly positive r such that ∫ g r ( x ) dx = 1 . This is the reason that we state the strict inequality r 1 < r P .

3.3 Proof sketch of the theorems

The full proofs of the main theorems, Theorems 3.1 and 3.3, are presented in Appendix C. In Appendix C, we present a generalized theorem which includes Theorems 3.1 and 3.3 as special cases, where X can be a general sample space and ˜ P is similarly defined as in the continuous space case. The key idea for proofs and proposed mechanisms is to focus on the behavior of Q ( P ) when P is in an extreme case in ˜ P . In finite space, point masses are extreme cases, and for continuous space with ˜ P = ˜ P c 1 ,c 2 ,h , the cases that p ( x ) ∈ { c 1 h ( x ) , c 2 h ( x ) } for all x ∈ X are the extreme cases. As implied by the proof, the worst-case f -divergence of the proposed optimal mechanism is attained when P is in the aforementioned extreme cases. Such an approach using extreme case is a frequently used technique in PUT analysis [55-58].

Our proof consists of two parts, the achievability part and the converse part. The achievability part is to show that the worst-case f -divergence R f ( Q ∗ ) of our proposed mechanism Q ∗ is upper-bounded by the RHS of (7) or (12). The converse part is to show that R f ( Q ) of any ϵ -LDP mechanism Q is lower-bounded by the RHS of (7) or (12). From now, we briefly describe the proof idea of each part. Here, we omit the subscripts ( k, ϵ ) or ( c 1 , c 2 , h, ϵ ) for notational convenience.

Achievability part. Let M = { Q ∗ ( P ) : P ∈ ˜ P} . For finite space, M consists of all distributions Q ∈ P ([ k ]) such that 1 e ϵ + k -1 ≤ Q ( x ) ≤ e ϵ e ϵ + k -1 for every x ∈ [ k ] . For continuous space, M consists of all continuous distributions Q ∈ C ( R n ) whose pdf q satisfies bh ( x ) ≤ q ( x ) ≤ be ϵ h ( x ) for every x ∈ R n .

We construct a mechanism Q † such that R f ( Q † ) is upper-bounded by the RHS of (7) or (12). The construction is as follows. First, we set a reference distribution µ ∈ P ( X ) and a constant γ ∈ [0 , 1] according to a certain rule specified in Appendix C. Then, for each given P , we generate a private sample by sampling from the original P with probability γ , and sampling from the reference distribution µ with probability 1 -γ . In other words, we have Q † ( P ) = γP +(1 -γ ) µ . Our choice of µ and γ makes Q † ( P ) ∈ M for every P ∈ ˜ P , which especially implies that Q † also satisfies ϵ -LDP. Furthermore, we can find a bound on the ratio of the pmf or pdf for original distribution to that for sampling distribution. Then, invoking [59, Theorem 2.1], which bounds f -divergences given bounds on the ratio between pmf or pdfs, we show that R f ( Q † ) is upper-bounded by the RHS of (7) or (12).

Next, we demonstrate a non-trivial generalization of the main result of [35] that Q ∗ ( P ) is the f -divergence projection of P onto M for P ∈ ˜ P and for every f -divergence.

Proposition 3.4. Assuming the setups of ( X , ˜ P , ϵ ) in either Theorem 3.1 or 3.3, let Q ∗ denote the proposed mechanism Q ∗ k,ϵ or Q ∗ c 1 ,c 2 ,h,ϵ . Also, let M be as described above. Then, for every P ∈ ˜ P and every f -divergence D f , we have D f ( P ∥ Q ∗ ( P )) = inf Q ∈M D f ( P ∥ Q ) .

Notice that this proposition differs from [35] in that it holds for all general f -divergences (as opposed to only KL divergence) and general sample spaces, be it discrete or continuous. This result immediately yields D f ( P ∥ Q ∗ ( P )) ≤ D f ( P ∥ ∥ Q † ( P ) ) , and hence R f ( Q ∗ ) ≤ R f ( Q † ) ≤ ( RHS of (7) or (12) ) . We remark that combining with the converse part (to be described below), it implies that the Q † is also optimal in our minimax sense. Nevertheless, Q ∗ outperforms Q † in that D f ( P ∥ Q ∗ ( P )) ≤ D f ( P ∥ ∥ Q † ( P ) ) for every P ∈ ˜ P .

Remark 3.5 . For the finite case, µ is the uniform distribution over [ k ] and γ is taken in such a way that Q † satisfies ϵ -LDP tightly. In this case, an alternative way to implement Q † is as follows. For each given P , first we sample from P to get a raw sample and then apply the k -ary randomized response [60] to it.

Converse part. For each extreme P , let A P be the 'high probability set', defined as A P = { x ∈ X : P ( x ) = 1 } for finite space and A P = { x ∈ X : p ( x ) = c 2 h ( x ) } for continuous space. Then, using the data processing inequality of f -divergence [61], we take a lower bound of D f ( P ∥ Q ( P )) by the f -divergence between the distributions of ✶ A P ( X ) for X ∼ P and that for X ∼ Q ( P ) . Such a lower bound becomes a decreasing function of Q ( A P | P ) in a certain range. Then, we seek to find an upper bound on inf Q ( A P | P ) over extreme P , which gives a lower bound on R f ( Q ) . This involves a novel combinatorial argument. We perform a 'packing' of u copies of X by t subsets A 1 , · · · , A t with A i = A P i for some extreme P i and appropriately chosen t and u . Then, we decompose the RHS of u = ∑ u i =1 Q ( X| P i ) by an appropriate partition of X involving A i 's and use the definition of ϵ -LDP to find an upper bound on inf Q ( A P | P ) .

4 Discussions on the Proposed Mechanism

4.1 Effect of the r P approximation error

In practice, it may not be possible to find the exact value of r P such that the sum or integration of the RHS of (8) or (13) is 1 . For the case of continuous space as in Theorem 3.3, one way to implement the proposed mechanism Q ∗ c 1 ,c 2 ,h,ϵ in practice is as follows. First, we fix parameters δ 1 ∈ [0 , 1) and δ 2 ≥ 0 that quantify error tolerance. For a given P , we define g r ( x ) = clip ( 1 r p ( x ); bh ( x ) , be ϵ h ( x ) ) and find r P > 0 such that ∫ R n g r P ( x ) dx ∈ [1 -δ 1 , 1 + δ 2 ] ; we delineate a numerical algorithm for this task in Section 3.2 based on the bisection method and a numerical integration method. Then, we get a private sample by sampling from the distribution with pdf ˆ q ( x ) = g r P ( x ) / ∫ R n g r P ( x ) dx . For the finite case as in Theorem 3.1, we can implement in the same way, except replacing the integral with the sum.

It is important to note that ˆ q ( x ) ∈ [ bh ( x ) 1+ δ 2 , be ϵ h ( x ) 1 -δ 1 ] , indicating that the resulting Q ∗ k,ϵ and Q ∗ c 1 ,c 2 ,h,ϵ satisfy ( ϵ +log 1+ δ 2 1 -δ 1 ) -LDP as opposed to ϵ -LDP. Thus, the above implementation yields ϵ -LDP if it is used to implement Q ∗ k,ϵ ′ or Q ∗ c 1 ,c 2 ,h,ϵ ′ , with ϵ ′ = ϵ -log 1+ δ 2 1 -δ 1 and sufficiently small δ 1 , δ 2 such that ϵ ′ > 0 .

4.2 Continuity of the proposed mechanism

In some practical scenarios, the client may not have full access to their distribution P . One example is that the client can only access to samples from P . In such case, the client may first estimate the true distribution, and then perturb the estimated distribution through the optimal mechanism. The question is how the perturbation using the estimated distribution deviates from that using the true distribution. To answer this, we show that the proposed mechanism satisfies a pointwise Lipschitz property with respect to the total variation distance, and the Lipschitz constant is closely related to the factor r P we introduce in Theorems 3.1 and 3.3.

Proposition 4.1. Assuming the setups of ( X , ˜ P , ϵ ) in either Theorem 3.1 or 3.3, let Q ∗ denote the proposed mechanism Q ∗ k,ϵ or Q ∗ c 1 ,c 2 ,h,ϵ . Then, for any P, P ′ ∈ ˜ P , we have

where r P > 0 is as in Theorem 3.1 or 3.3.

This guarantees that for each given true P and given δ > 0 , whenever the approximated P ′ satisfies D TV ( P, P ′ ) ≤ δr P / 2 , the perturbed distribution Q ∗ ( P ′ ) satisfies D TV ( Q ∗ ( P ) , Q ∗ ( P ′ )) ≤ δ . In theoretical perspective, this proposition implies that Q ∗ is continuous when ˜ P and P ( X ) are endowed with the metric topology from the total variation distance. The proof of Proposition 4.1 is in Appendix D.

5 Numerical Results

In this section, we numerically compare the worst-case f -divergence of our proposed mechanism with that of the previously proposed sampling mechanism. To the best of our knowledge, the only work about the private sampling under LDP is [35], hence we set the baseline as the mechanism proposed in [35]. In all the cases, we perform the comparison across three canonical f -divergences: KL divergence, total variation distance, and squared Hellinger distance, as well as across five values of ϵ : 0.1, 0.5, 1, 2, and 5.

5.1 Comparison for finite data space

In this subsection, we compare the mechanisms in the finite space, X = [ k ] and ˜ P = P ([ k ]) . As mentioned in Sections 2.3 and 3.1, the baseline mechanism has a hyper-parameter, a reference probability distribution Q 0 ∈ P ( X ) . We set the baseline as a generalized f -divergence projection onto the relative mollifier. That is, for each given f -divergence, we set the baseline to satisfy Q ( P ) ∈ arg min Q ∈M ϵ,Q 0 D f ( P ∥ Q ) , where M ϵ,Q 0 is defined in Section 2.3. As expected by symmetry, for any f , choosing Q 0 to be the uniform distribution minimizes the worst-case f -divergence R f ( Q ) among all choices of Q 0 for the baseline. Also, even though we do not obtain the closed-form expression of Q ( x | P ) for the baseline, we obtain the value of R f ( Q ) when Q 0 is the uniform distribution. The proof of this fact, together with the precise value of R f ( Q ) for uniform Q 0 , is in Appendix F.1. Hence, we always set Q 0 to be the uniform distribution in the result about the baseline. Since we have the precise values of R f ( Q ) for both our proposed mechanism and the baseline, we plot such values of R f ( Q ) in Figure 3. For simplicity, we only provide the plot for k = 10 . More plots for some other k 's can be found in Appendix G. As shown by the figure, the proposed mechanism has lower worst-case f -divergence than the baseline for all choices of f -divergences and ϵ in the experiment, with significant gap in medium privacy regime ϵ ∈ [0 . 5 , 2] .

0.1

0.5

1.0

2.0

5.0

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

Worst

f

-divergence

KL

0.1

0.5

1.0

2.0

5.0

Privacy budget

/epsilon1

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

TV

0.1

0.5

1.0

2.0

5.0

0

.

0

0

.

2

0

.

4

0

.

6

Sq. Hel

Proposed

Baseline

(Left: KL divergence, Center: Total variation distance, Right: Squared Hellinger distance)

5.2 Comparison for 1D Gaussian mixture

In this subsection, we conduct an experiment to compare the mechanisms when the client distributions are Gaussian mixtures over a real line X = R , which is an instance of a continuous space case. We consider the case that each client has a Gaussian mixture distribution in R , where each Gaussian has a mean bounded by 1 and has a unit variance. To avoid arbitrarily large number of Gaussian distributions to be mixed, we set an upper bound K of the number of Gaussian distributions to be mixed per client. Also, to make the numerical integration tractable, we truncate the domain of the distributions to lie inside an interval [ -4 , 4] . Unlike the finite space case, there is no known closedform expression of the worst-case f -divergence for the mechanism in [35]. The set of Gaussian mixtures is not exactly of the form ˜ P = ˜ P c 1 ,c 2 ,h , hence our proposed mechanism also does not have a known closed-form expression of the worst-case f -divergence. Hence, instead, we compare the mechanisms by an empirical worst-case f -divergence.

For an experiment, we randomly construct N Gaussian mixture distributions P 1 , P 2 , · · · , P N ∈ ˜ P , where each P j is generated independently according to some rules specified in Appendix F.2. After that, we plot the value of the empirical maximum f -divergence max j ∈ [ N ] D f ( P j ∥ Q ( P j )) for the baseline and our proposed Q . For the baseline mechanism, we use MBDE with the same hyperparameter setup as [35, Section 5], except a slight modification of the reference distribution to consider the truncation of the domain. The implementation details are provided in Appendix F.2.

In Figure 4, we present the result for N = 100 and K = 10 . We can see that the proposed mechanism has much lower worst-case f -divergence than the baseline for all choices of f -divergences and ϵ .

0.1

0.5

1.0

2.0

5.0

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

Worst

f

-divergence

KL

0.1

0.5

1.0

2.0

5.0

Privacy budget

/epsilon1

0

.

0

0

.

1

0

.

2

0

.

3

TV

0.1

0.5

1.0

2.0

5.0

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

Sq. Hel

Proposed

Baseline

(Left: KL divergence, Center: Total variation distance, Right: Squared Hellinger distance)

6 Conclusion

In this paper, we characterized the optimal privacy-utility trade-off for the private sampling under LDP and found the optimal private sampling mechanism in terms of the minimax f -divergence between original and sampling distributions, for both finite and continuous data spaces. Compared to the previous work [35] based on relative mollifier with arbitrarily chosen reference distribution, our work characterizes PUT without dependency on external information other than the original distribution, and it is shown that the mechanism we found is universally optimal under any f -divergence.

For future works, there may be other ˜ P and other measures of utility more appropriate to practical scenarios, which are not handled in this paper. For example, f -divergence may be an inappropriate utility loss because it only depends on σ -algebra structure and does not consider additional information about geometry of X , such as underlying metric on X . Using utility measures involving the geometry, such as Wasserstein distance [62, 63, 49], may be more appropriate for some scenarios. Also, we can consider the Bayesian approach instead of the worst-case approach. Furthermore, we only consider the task of releasing a single sample per client in this paper. We may also consider the case of releasing multiple samples per client, rather than a single sample.

The limitations and broader impacts of this work are in Appendices I and J, respectively.

Acknowledgments and Disclosure of Funding

We thank the anonymous reviewers for their valuable discussions and comments, particularly regarding the identification of an alternative optimal mechanism, the advantage of the proposed mechanism over the alternative, and the effect of the approximation error on r P .

This work was supported in part by the National Research Foundation of Korea (NRF) Grant funded by the Korea Government (MSIT) under Grant 2022R1A2C2092151 and in part by Institute of Information & Communications Technology Planning & Evaluation (IITP) under 6G · Cloud Research and Education Open Hub (IITP-2024-RS-2024-00428780) grant funded by the Korea government (MSIT).

References

Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable Extraction of Training Data from (Production) Language Models. arXiv preprint arXiv:2311.17035 , November 2023.

Xiaodong Wu, Ran Duan, and Jianbing Ni. Unveiling security, privacy, and ethical concerns of ChatGPT. Journal of Information and Intelligence , 2(2):102-115, March 2024. ISSN 29497159. doi: 10.1016/j.jiixd. 2023.10.007.

Biwei Yan, Kun Li, Minghui Xu, Yueyan Dong, Yue Zhang, Zhaochun Ren, and Xiuzhen Cheng. On Protecting the Data Privacy of Large Language Models (LLMs): A Survey. arXiv preprint arXiv:2403.05156 , March 2024.

Zhangheng Li, Junyuan Hong, Bo Li, and Zhangyang Wang. Shake to Leak: Fine-tuning Diffusion Models Can Amplify the Generative Privacy Risk. In 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) , pages 18-32, April 2024. doi: 10.1109/SaTML59370.2024.00010.

Shiva Prasad Kasiviswanathan, Homin K. Lee, Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. What Can We Learn Privately? SIAM Journal on Computing , 40(3):793-826, January 2011. ISSN 0097-5397. doi: 10.1137/090756090.

Úlfar Erlingsson, Vasyl Pihur, and Aleksandra Korolova. Rappor: Randomized aggregatable privacypreserving ordinal response. In Proceedings of the 2014 ACM SIGSAC conference on computer and communications security , pages 1054-1067. ACM, 2014.

Andrea Bittau, Úlfar Erlingsson, Petros Maniatis, Ilya Mironov, Ananth Raghunathan, David Lie, Mitch Rudominer, Ushasree Kode, Julien Tinnes, and Bernhard Seefeld. Prochlo: Strong privacy for analytics in the crowd. In Proceedings of the 26th Symposium on Operating Systems Principles , SOSP '17, page 441-459. Association for Computing Machinery, 2017. ISBN 9781450350853.

Differential privacy team Apple. Learning with privacy at scale. Technical report, Apple , 2017.

Bolin Ding, Janardhan Kulkarni, and Sergey Yekhanin. Collecting Telemetry Data Privately. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

John C. Duchi, Michael I. Jordan, and Martin J. Wainwright. Local Privacy and Statistical Minimax Rates. In 2013 IEEE 54th Annual Symposium on Foundations of Computer Science , pages 429-438, October 2013. doi: 10.1109/FOCS.2013.53.

Min Ye and Alexander Barg. Optimal Schemes for Discrete Distribution Estimation Under Locally Differential Privacy. IEEE Transactions on Information Theory , 64(8):5662-5676, August 2018. ISSN 0018-9448, 1557-9654. doi: 10.1109/TIT.2018.2809790.

Abhishek Bhowmick, John Duchi, Julien Freudiger, Gaurav Kapoor, and Ryan Rogers. Protection Against Reconstruction and Its Applications in Private Federated Learning. arXiv preprint arXiv:1812.00984 , June 2019.

Hilal Asi, Vitaly Feldman, and Kunal Talwar. Optimal Algorithms for Mean Estimation under Local Differential Privacy. In Proceedings of the 39th International Conference on Machine Learning , pages 1046-1056. PMLR, June 2022.

Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, and Adam Sealfon. On computing pairwise statistics with local differential privacy. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 27129-27146. Curran Associates, Inc., 2023.

Bonwoo Lee, Jeongyoun Ahn, and Cheolwoo Park. Minimax risks and optimal procedures for estimation under functional local differential privacy. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 57964-57975. Curran Associates, Inc., 2023.

Hilal Asi, Vitaly Feldman, Jelani Nelson, Huy Nguyen, and Kunal Talwar. Fast optimal locally private mean estimation via random projections. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 16271-16282. Curran Associates, Inc., 2023.

Berivan Isik, Wei-Ning Chen, Ayfer Ozgur, Tsachy Weissman, and Albert No. Exact optimality of communication-privacy-utility tradeoffs in distributed mean estimation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 37761-37785. Curran Associates, Inc., 2023.

Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Bounds. In 2014 IEEE 55th Annual Symposium on Foundations of Computer Science , pages 464-473, October 2014. doi: 10.1109/FOCS.2014.56.

Stacey Truex, Ling Liu, Ka-Ho Chow, Mehmet Emre Gursoy, and Wenqi Wei. LDP-Fed: Federated learning with local differential privacy. In Proceedings of the Third ACM International Workshop on Edge Systems, Analytics and Networking , EdgeSys '20, pages 61-66, New York, NY, USA, May 2020. Association for Computing Machinery. ISBN 978-1-4503-7132-2. doi: 10.1145/3378679.3394533.

Badih Ghazi, Pritish Kamath, Ravi Kumar, Ethan Leeman, Pasin Manurangsi, Avinash V. Varadarajan, and Chiyuan Zhang. Regression with Label Differential Privacy. arXiv preprint arXiv:2212.06074 , October 2023.

Vitaly Feldman, Ilya Mironov, Kunal Talwar, and Abhradeep Thakurta. Privacy Amplification by Iteration. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS) , pages 521-532, October 2018. doi: 10.1109/FOCS.2018.00056.

Mengchu Li, Tom Berrett, and Yi Yu. Network change point localisation under local differential privacy. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 15013-15026. Curran Associates, Inc., 2022.

Shuzhen Chen, Dongxiao Yu, Yifei Zou, Jiguo Yu, and Xiuzhen Cheng. Decentralized Wireless Federated Learning With Differential Privacy. IEEE Transactions on Industrial Informatics , 18(9):6273-6282, September 2022. ISSN 1941-0050. doi: 10.1109/TII.2022.3145010.

Youssef Allouah, Rachid Guerraoui, Nirupam Gupta, Rafael Pinot, and John Stephan. On the PrivacyRobustness-Utility Trilemma in Distributed Learning. In Proceedings of the 40th International Conference on Machine Learning , pages 569-626. PMLR, July 2023.

Chuan Guo, Kamalika Chaudhuri, Pierre Stock, and Michael Rabbat. Privacy-Aware Compression for Federated Learning Through Numerical Mechanism Design. In Proceedings of the 40th International Conference on Machine Learning , pages 11888-11904. PMLR, July 2023.

Yi Liu, Qirui Hu, Lei Ding, and Linglong Kong. Online Local Differential Private Quantile Inference via Self-normalization. In Proceedings of the 40th International Conference on Machine Learning , pages 21698-21714. PMLR, July 2023.

Jin Sima, Changlong Wu, Olgica Milenkovic, and Wojciech Szpankowski. Online Distribution Learning with Local Privacy Constraints. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , pages 460-468. PMLR, April 2024.

Antonious M. Girgis, Deepesh Data, and Suhas Diggavi. Distributed User-Level Private Mean Estimation. In 2022 IEEE International Symposium on Information Theory (ISIT) , pages 2196-2201, June 2022. doi: 10.1109/ISIT50566.2022.9834713.

Ruiquan Huang, Huanyu Zhang, Luca Melis, Milan Shen, Meisam Hejazinia, and Jing Yang. Federated Linear Contextual Bandits with User-level Differential Privacy. In Proceedings of the 40th International Conference on Machine Learning , pages 14060-14095. PMLR, July 2023.

Jayadev Acharya, Yuhan Liu, and Ziteng Sun. Discrete Distribution Estimation under User-level Local Differential Privacy. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , pages 8561-8585. PMLR, April 2023.

Raef Bassily and Ziteng Sun. User-level Private Stochastic Convex Optimization with Optimal Rates. In Proceedings of the 40th International Conference on Machine Learning , pages 1838-1851. PMLR, July 2023.

Yulian Mao, Qingqing Ye, Haibo Hu, Qi Wang, and Kai Huang. PrivShape: Extracting Shapes in Time Series under User-Level Local Differential Privacy. arXiv preprint arXiv:2404.03873 , April 2024.

Kangkang Sun, Jun Wu, Ali Kashif Bashir, Jianhua Li, Hansong Xu, Qianqian Pan, and Yasser D. Al-Otaibi. Personalized Privacy-Preserving Distributed Artificial Intelligence for Digital-Twin-Driven Vehicle Road Cooperation. IEEE Internet of Things Journal , pages 1-1, 2024. ISSN 2327-4662. doi: 10.1109/JIOT.2024.3389656.

Alexander Kent, Thomas B. Berrett, and Yi Yu. Rate Optimality and Phase Transition for User-Level Local Differential Privacy. arXiv preprint arXiv:2405.11923 , May 2024.

Hisham Husain, Borja Balle, Zac Cranko, and Richard Nock. Local Differential Privacy for Sampling. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , pages 3404-3413. PMLR, June 2020.

James Flemings, Meisam Razaviyayn, and Murali Annavaram. Differentially Private Next-Token Prediction of Large Language Models. arXiv preprint arXiv:2403.15638 , April 2024.

Sofya Raskhodnikova, Satchit Sivakumar, Adam Smith, and Marika Swanberg. Differentially private sampling from distributions. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 28983-28994. Curran Associates, Inc., 2021.

Badih Ghazi, Xiao Hu, Ravi Kumar, and Pasin Manurangsi. On differentially private sampling from gaussian and product distributions. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 77783-77809. Curran Associates, Inc., 2023.

Hamid Ebadi, Thibaud Antignac, and David Sands. Sampling and partitioning for differential privacy. In 2016 14th Annual Conference on Privacy, Security and Trust (PST) , pages 664-673, February 2016. doi: 10.1109/PST.2016.7906954.

Nazmiye Ceren Abay, Yan Zhou, Murat Kantarcioglu, Bhavani Thuraisingham, and Latanya Sweeney. Privacy Preserving Synthetic Data Release Using Deep Learning. In Michele Berlingerio, Francesco Bonchi, Thomas Gärtner, Neil Hurley, and Georgiana Ifrim, editors, Machine Learning and Knowledge Discovery in Databases , Lecture Notes in Computer Science, pages 510-526, Cham, 2019. Springer International Publishing. ISBN 978-3-030-10925-7. doi: 10.1007/978-3-030-10925-7_31.

Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially Private Generative Adversarial Network. arXiv preprint arXiv:1802.06739 , February 2018.

Bangzhou Xin, Wei Yang, Yangyang Geng, Sheng Chen, Shaowei Wang, and Liusheng Huang. Private FL-GAN: Differential Privacy Synthetic Data Generation Based on Federated Learning. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 2927-2931, May 2020. doi: 10.1109/ICASSP40776.2020.9054559.

Reihaneh Torkzadehmahani, Peter Kairouz, and Benedict Paten. DP-CGAN: Differentially Private Synthetic Data and Label Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 0-0, 2019.

Yi Liu, Jialiang Peng, James J.Q. Yu, and Yi Wu. PPGAN: Privacy-Preserving Generative Adversarial Network. In 2019 IEEE 25th International Conference on Parallel and Distributed Systems (ICPADS) , pages 985-989, February 2019. doi: 10.1109/ICPADS47876.2019.00150.

Teddy Cunningham, Konstantin Klemmer, Hongkai Wen, and Hakan Ferhatosmanoglu. GeoPointGAN: Synthetic Spatial Data with Local Label Differential Privacy. arXiv preprint arXiv:2205.08886 , May 2022.

Hua Zhang, Kaixuan Li, Teng Huang, Xin Zhang, Wenmin Li, Zhengping Jin, Fei Gao, and Minghui Gao. Publishing locally private high-dimensional synthetic data efficiently. Information Sciences , 633:343-356, July 2023. ISSN 0020-0255. doi: 10.1016/j.ins.2023.03.014.

Hisaichi Shibata, Shouhei Hanaoka, Yang Cao, Masatoshi Yoshikawa, Tomomi Takenaga, Yukihiro Nomura, Naoto Hayashi, and Osamu Abe. Local Differential Privacy Image Generation Using FlowBased Deep Generative Models. Applied Sciences , 13(18):10132, January 2023. ISSN 2076-3417. doi: 10.3390/app131810132.

Hansle Gwon, Imjin Ahn, Yunha Kim, Hee Jun Kang, Hyeram Seo, Heejung Choi, Ha Na Cho, Minkyoung Kim, JiYe Han, Gaeun Kee, Seohyun Park, Kye Hwa Lee, Tae Joon Jun, and Young-Hak Kim. LDP-GAN : Generative adversarial networks with local differential privacy for patient medical records synthesis. Computers in Biology and Medicine , 168:107738, January 2024. ISSN 0010-4825. doi: 10.1016/j. compbiomed.2023.107738.

Jacob Imola, Amrita Roy Chowdhury, and Kamalika Chaudhuri. Metric Differential Privacy at the User-Level. arXiv preprint arXiv:2405.02665 , May 2024.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating Noise to Sensitivity in Private Data Analysis. In Shai Halevi and Tal Rabin, editors, Theory of Cryptography , Lecture Notes in Computer Science, pages 265-284, Berlin, Heidelberg, 2006. Springer. ISBN 978-3-540-32732-5. doi: 10.1007/11681878_14.

Igal Sason. On f -Divergences: Integral Representations, Local Behavior, and Inequalities. Entropy , 20(5): 383, May 2018. ISSN 1099-4300. doi: 10.3390/e20050383.

Igal Sason and Sergio Verdú. f -Divergence Inequalities. IEEE Transactions on Information Theory , 62 (11):5973-6006, January 2016. ISSN 1557-9654. doi: 10.1109/TIT.2016.2603151.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11): 139-144, October 2020. ISSN 0001-0782. doi: 10.1145/3422622.

Elias M. Stein and Rami Shakarchi. Real Analysis: Measure Theory, Integration, and Hilbert Spaces . Princeton University Press, 2005. ISBN 978-0-691-11386-9. doi: 10.2307/j.ctvd58v18.

Peter Kairouz, Sewoong Oh, and Pramod Viswanath. Extremal Mechanisms for Local Differential Privacy. Journal of Machine Learning Research , 17(17):1-51, 2016. ISSN 1533-7928.

Naoise Holohan, Douglas J. Leith, and Oliver Mason. Extreme points of the local differential privacy polytope. Linear Algebra and its Applications , 534:78-96, December 2017. ISSN 0024-3795. doi: 10.1016/j.laa.2017.08.011.

Ankit Pensia, Amir R. Asadi, Varun Jog, and Po-Ling Loh. Simple Binary Hypothesis Testing under Local Differential Privacy and Communication Constraints. arXiv preprint arXiv:2301.03566 , December 2023.

Seung-Hyun Nam, Vincent Y. F. Tan, and Si-Hyeon Lee. Optimal Private Discrete Distribution Estimation With 1-bit Communication. IEEE Transactions on Information Forensics and Security , 19:6514-6528, 2024. ISSN 1556-6021. doi: 10.1109/TIFS.2024.3419721.

Andrew Rukhin. Information-type divergence when the likelihood ratios are bounded. Applicationes Mathematicae , 4(24):415-423, 1997. ISSN 1233-7234.

Stanley L Warner. Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American Statistical Association , 60(309):63-69, 1965.

F. Liese and I. Vajda. On Divergences and Informations in Statistics and Information Theory. IEEE Transactions on Information Theory , 52(10):4394-4412, October 2006. ISSN 1557-9654. doi: 10.1109/ TIT.2006.881731.

Cédric Villani. Optimal Transport: Old and New . Number 338 in Grundlehren Der Mathematischen Wissenschaften. Springer, Berlin, 2009. ISBN 978-3-540-71049-3.

Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. arXiv preprint arXiv:1701.07875 , December 2017.

Stephen P. Boyd and Lieven Vandenberghe. Convex Optimization . Cambridge University Press, Cambridge, UK ; New York, 2004. ISBN 978-0-521-83378-3.

R. Tyrrell Rockafellar. Convex Analysis . Princeton University Press, 1970. ISBN 978-0-691-01586-6.

Nicholas Metropolis, Arianna W. Rosenbluth, Marshall N. Rosenbluth, Augusta H. Teller, and Edward Teller. Equation of State Calculations by Fast Computing Machines. The Journal of Chemical Physics , 21 (6):1087-1092, June 1953. ISSN 0021-9606. doi: 10.1063/1.1699114.

Christian P. Robert. The Metropolis-Hastings algorithm. arXiv preprint arXiv:1504.01896 , January 2016.

Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The Composition Theorem for Differential Privacy. In Proceedings of the 32nd International Conference on Machine Learning , pages 1376-1385. PMLR, June 2015.

Ryan M Rogers, Aaron Roth, Jonathan Ullman, and Salil Vadhan. Privacy Odometers and Filters: Payas-you-Go Composition. In Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016.

A Assumptions about the Measure Theory

The appendices assume the familiarity with the basic measure theory and real analysis. We refer the standard textbooks about the measure theory and real analysis, e.g. [54].

Throughout the main paper and the appendices, we assume the followings:

For each sample space X , a σ -algebra on X is implicitly given. Unless mentioned otherwise,

-the Borel σ -algebra is given for X = R n .

-the discrete σ -algebra is given for finite X , and

A 'subset' of X always means a 'measurable subset', and similarly A ⊂ X always means A is a measurable subset.

The 'continuous" distribution precisely means the 'absolutely continuous" distribution (with respect to the Lebesgue measure).

In the appendices, we also introduce the following notations:

For finite X , # denotes the counting measure.

For X = R n , m denotes the Lebesgue measure.

B General Definition and More Properties of f -divergences

In this appendix, we review the general definition and additional properties about the f -divergences which are important in our analysis. Let a convex function f : (0 , ∞ ) → R with f (1) = 0 be given. For P, Q ∈ P ( X ) , not necessarily P ≪ Q , we first take a dominating measure µ on X such that P, Q ≪ µ (e.g., µ = P + Q ), and let p = dP/dµ , q = dQ/dµ . The f -divergence D f ( P ∥ Q ) is defined as

We note that (15) is invariant under the choice of the dominating measure µ .

Since it is possible that p ( x ) = 0 or q ( x ) = 0 , we need to define what is the value of the expression qf ( p/q ) for p = 0 or q = 0 [51, 52]. It is well-known that any convex function f : (0 , 1) → R is continuous, and the function

is also convex on (0 , 1) . Furthermore, any convex function f : (0 , 1) → R with f (1) = 0 has a limit lim x → 0+ f ( x ) in R ∪ { + ∞} . Hence, we have the continuous extension f : [0 , ∞ ) → R ∪ { + ∞} by setting f (0) = lim x → 0+ f ( x ) , which is proper convex and continuous. By the same way, we have the continuous extension f ⋆ : [0 , ∞ ) → R ∪{ + ∞} . Using these extensions, we define 0 f (0 / 0) = 0 , qf (0 /q ) = qf (0) for q > 0 , and 0 f ( p/ 0) = pf ⋆ (0) for p > 0 . Especially, if f ⋆ (0) = ∞ , then D f ( P ∥ Q ) = ∞ whenever P ≪ Q does not hold. (This is the case for KL divergence and χ 2 divergence for examples) Similarly, if f (0) = ∞ , then D f ( P ∥ Q ) = ∞ whenever Q ≪ P does not hold. Also, the maximum value of the f -divergence M f presented in (2) can be written as M f = f (0) + f ⋆ (0) .

The following additional properties of f -divergences are important in our analysis. [61]

Theorem B.1. Any f -divergence is jointly convex, that is for any P 1 , P 2 , Q 1 , Q 2 ∈ P ( X ) and 0 ≤ λ ≤ 1 , we have D f ( λP 1 +(1 -λ ) P 2 ∥ λQ 1 +(1 -λ ) Q 2 ) ≤ λD f ( P 1 ∥ Q 1 ) + (1 -λ ) D f ( P 2 ∥ Q 2 ) .

Theorem B.2 (Data-Processing Inequality) . Let M be a conditional distribution (Markov kernel) from X to Y . For given P 1 , P 2 ∈ P ( X ) , let Q 1 , Q 2 ∈ P ( Y ) be the push-forward measure of P 1 , P 2 through M , respectively. Then for any f -divergence, we have D f ( P 1 ∥ P 2 ) ≥ D f ( Q 1 ∥ Q 2 ) .

Also, we present several equivalent expressions for the total variation distance [51, 52], which are used in Appendix D. In below expressions, the assumption is that P, Q ≪ µ for some dominating

measure µ with p = dP/dµ and q = dQ/dµ .

We introduce one more notation. For λ 1 , λ 2 ∈ [0 , 1] , let D B f ( λ 1 ∥ λ 2 ) denotes the f -divergence between Bernoulli distributions with Pr(1) = λ 1 and λ 2 , respectively. That is,

We should note the following facts:

By joint convexity of the f -divergence and continuity of f , D B f ( λ 1 ∥ λ 2 ) is continuous and jointly convex in ( λ 1 , λ 2 ) . (But D B f ( λ 1 ∥ λ 2 ) may be extended real-valued)

For a fixed λ 1 , D B f ( λ 1 ∥ λ 2 ) attains a global minimum 0 at λ 2 = λ 1 . Together with convexity, we derive that D B f ( λ 1 ∥ λ 2 ) is decreasing in λ 2 ∈ [0 , λ 1 ] and increasing in λ 2 ∈ [ λ 1 , 1] , respectively.

C Generalized Main Theorem and Proof

In this appendix, we present the formal proofs of the main theorems, Theorems 3.1 and 3.3. As mentioned in Section 3.3, we first state the generalized theorem with its proof, and later we show how this generalized theorem includes the main theorems as special cases.

C.1 Statement of the generalized main theorem

First, let us define the general setup we consider. Let a (general) sample space X be given. For a positive measure µ on X such that µ ( X ) < ∞ and c 2 > c 1 ≥ 0 , let us define

We generally consider the case that ˜ P = ˜ P c 1 ,c 2 ,µ for some c 1 , c 2 , µ . For example, for the setup of Section 3.2, we have ˜ P c 1 ,c 2 ,h = ˜ P c 1 ,c 2 ,µ , where µ is a positive measure with µ ≪ m and dµ/dm = h . For the setup of Section 3.1, we have P ([ k ]) = ˜ P 0 , 1 , # . By the same reason as in Section 3.2, we may impose the following normalization condition on c 1 , c 2 , µ, ϵ :

Note that µ ( X ) = 1 means that µ is a probability measure, that is µ ∈ P ( X ) . Also, note that for X = [ k ] , we can write in normalized form as P ([ k ]) = ˜ P 0 ,k,µ k , where µ k = 1 k # is the uniform distribution on [ k ] .

First, let us define the proposed mechanism, together with the related constants as the same as Theorem 3.3. For the ease of proof, we introduce an additional constant α over Theorem 3.3.

Definition C.1. Let c 1 , c 2 , µ, ϵ satisfying the normalization condition (22) be given, and let ˜ P = ˜ P c 1 ,c 2 ,µ . First, define the following constants determined by c 1 , c 2 , ϵ :

Also, define a mechanism Q ∗ c 1 ,c 2 ,µ,ϵ ∈ Q X , ˜ P ,ϵ as follows:

For each P ∈ ˜ P , Q ∗ c 1 ,c 2 ,µ,ϵ ( P ) =: Q is defined as a probability measure such that Q ≪ µ and

where r P > 0 is a constant depending on P so that ∫ dQ dµ dµ ( x ) = 1 . Furthermore, let M c 1 ,c 2 ,µ,ϵ = ˜ P b,be ϵ ,µ , so that Q ∗ c 1 ,c 2 ,µ,ϵ ( P ) ∈ M c 1 ,c 2 ,µ,ϵ for every P ∈ ˜ P .

We should note that 1 = c 2 α + c 1 (1 -α ) = be ϵ α + b (1 -α ) . Also, Q ∗ c 1 ,c 2 ,µ,ϵ clearly satisfies ϵ -LDP.

By the same reason as in Sections 3.1 and 3.2, the values of r P may not be unique, but the mechanism Q ∗ c 1 ,c 2 ,µ,ϵ does not depend on the choice of r P . Furthermore, we show that the value of ∫ clip ( 1 r dP dµ ( x ); b, be ϵ ) dµ ( x ) at r = r 1 and r = r 2 are at least and at most 1 , respectively.

Proposition C.2. Let c 1 , c 2 , µ, ϵ, ˜ P , α, b, r 1 , r 2 be as in Definition C.1. Then, for any P ∈ ˜ P , we have

where, in the case of r 1 = 0 , we define

Furthermore, if ∫ clip ( 1 r 1 dP dµ ( x ); b, be ϵ ) dµ ( x ) = 1 , then ∫ clip ( 1 r dP dµ ( x ); b, be ϵ ) dµ ( x ) = 1 for every r ∈ [ r 1 , r 2 ] .

This proposition, together with the fact that r ↦→ ∫ clip ( 1 r dP dµ ( x ); b, be ϵ ) dµ ( x ) is continuous and monotone decreasing, implies that r P can be chosen such that r 1 < r P ≤ r 2 , as stated in Theorem 3.3. (Again, the continuity is from µ ( X ) < ∞ and the dominated convergence theorem)

Also, we should remark that 0 < α < 1 and c 1 < b < 1 < be ϵ < c 2 . The first one easily follows from c 1 < 1 < c 2 and the definition of α . For the second one, first we have 1 < αe ϵ +1 -α < e ϵ , as αe ϵ +1 -α is a propoer convex combination of 1 and ϵ . This directly implies that b < 1 < be ϵ . Next, by calculations, we can observe that

which implies c 1 < b , and

Now, we show that under a mild 'decomposability' condition, the proposed mechanism Q ∗ c 1 ,c 2 ,µ,ϵ is universally optimal under any f -divergences for ( X , ˜ P = ˜ P c 1 ,c 2 ,µ , ϵ ) . After that, we show that such a mild condition holds for the setups of both of the main theorems, which finishes the proofs of the main theorems.

which implies be ϵ < c 2 . Especially, these inequalities imply that 0 ≤ r 1 < 1 < r 2 .

First, let us state the 'decomposability' condition. This condition is a formal definition of the concept of 'packing of u copies of X by t subsets A 1 , · · · , A t ', which is briefly mentioned in Section 3.3,

We say that µ ∈ P ( X ) is α -decomposable if for any δ > 0 , there exists t, u ∈ N , t > u , such that α ≤ u/t < α + δ , and µ is ( α, t, u ) -decomposable.

Definition C.3. Let α ∈ (0 , 1) and t, u ∈ N , t > u . We say that a probability measure µ ∈ P ( X ) is ( α, t, u ) -decomposable if there exist t subsets A 1 , A 2 , · · · , A t ⊂ X such that µ ( A i ) = α for all i ∈ [ t ] , and for every x ∈ X , we have | { i ∈ [ t ] : x ∈ A i } | ≤ u .

We remark that ( α, t, 1) -deomposability means that there are t disjoint subsets B 1 , B 2 , · · · , B t such that µ ( B i ) = α for each i ∈ [ t ] . Also, if α is a rational number with α = u/t , u, t ∈ N , and µ is ( α, t, u ) -decomposable, then µ is α -decomposable.

Then, we state the generalized theorem.

Theorem C.4. Let c 1 , c 2 , µ, ϵ, ˜ P , α, b, r 1 , r 2 be as in Definition C.1. If µ is α -decomposable, then for any f -divergences D f , we have

and furthermore, the mechanism Q ∗ c 1 ,c 2 ,µ,ϵ as in Definition C.1 is optimal for ( X , ˜ P , ϵ ) under any f -divergences D f .

As guided in Section 3.3, the proof of this theorem is broken into two parts, the achievability part and the converse part.

Proposition C.5 (Achievability part) . Let c 1 , c 2 , µ, ϵ, ˜ P , α, b, r 1 , r 2 be as in Definition C.1. Then, for any f -divergences D f , we have

(Here, we do not need to assume that µ is α -decomposable)

Proposition C.6 (Converse part) . Let c 1 , c 2 , µ, ϵ, ˜ P , α, b, r 1 , r 2 be as in Definition C.1, and suppose that µ is α -decomposable. Then for any f -divergences D f and for any ϵ -LDP mechanism Q ∈ Q X , ˜ P ,ϵ , we have

The remaining of this appendix is organized as follows. We first present the proofs of Propositions C.5 and C.6 in Appendices C.2 and C.3, in which we grant Proposition C.2 and some intermediate lemmas. After that, in Appendix C.4, we show that Theorem C.4 contains main theorems, Theorems 3.1 and 3.3, as special cases. Finally, Appendix C.5 presents the proof of Proposition C.2, and Appendices C.6 and C.7 prove intermediate lemmas.

C.2 Proof of achievability part (Proposition C.5)

As mentioned in Section 3.3, the proof of the achievability part consists of two steps: first presenting an alternative mechanism and then proving that Q ∗ c 1 ,c 2 ,µ,ϵ performs the f -divergence projection for any general f -divergence.

C.2.1 An alternative mechanism

Let Q † c 1 ,c 2 ,µ,ϵ ∈ Q X , ˜ P ,ϵ be a mechanism defined as follows:

where

Since c 2 > e ϵ c 1 , it follows that 0 < γ < 1 . Also, a direct computation shows that

Notice that since c 1 ≤ dP dµ ≤ c 2 and d Q † c 1 ,c 2 ,µ,ϵ ( P ) dµ = γ dP dµ +(1 -γ ) , we have

Hence, Q † c 1 ,c 2 ,µ,ϵ ( P ) ∈ M c 1 ,c 2 ,µ,ϵ for every P ∈ ˜ P , implying that Q † c 1 ,c 2 ,µ,ϵ also satisfies ϵ -LDP. Now, we show that

To this end, we need to show that for each P ∈ ˜ P , we have

Fix P ∈ ˜ P . Let p = dP/dµ and q = d Q † c 1 ,c 2 ,µ,ϵ ( P ) /dµ . First, we claim that

for µ -almost every x ∈ X . We have

which is increasing in p ( x ) ≥ 0 . Since c 1 ≤ p ( x ) ≤ c 2 , we have

for µ -almost every x ∈ X . From (37), (38), and Definition C.1, we conclude that

which proves the claim. For the remaining of the proof, we need the following lemma.

Lemma C.7 ([59], Theorem 2.1) . Let P, Q ∈ P ( X ) . Suppose that P, Q ≪ µ for some reference measure µ on X , and there exist r 1 , r 2 ∈ R with 0 ≤ r 1 < 1 < r 2 such that the densities p = dP dµ , q = dQ dµ satisfy q ( x ) > 0 and r 1 ≤ p ( x ) q ( x ) ≤ r 2 for µ -almost every x ∈ X . Then for any f -divergence D f , we have

This lemma directly implies (42), and consequently (41).

C.2.2 f -divergence projection of the proposed mechanism

Next, we prove that Q ∗ c 1 ,c 2 ,µ,ϵ ( P ) is the projection of P onto M c 1 ,c 2 ,µ,ϵ for every f -divergence . Proposition 3.4 can be stated in a more general way as follows.

Proposition C.8. For any P ∈ ˜ P and any f -divergences D f , we have

This proposition implies that

for every P ∈ ˜ P , and hence which completes the proof of the achievability part.

Next, we prove Proposition C.8. Fix P ∈ ˜ P and p = dP/dµ . If f (0) = ∞ and µ ( { x : p ( x ) = 0 } ) > 0 , then D f ( P ∥ Q ) = ∞ for every Q ∈ M c 1 ,c 2 ,µ,ϵ , thus the proposition holds trivially. Consequently, we assume either f (0) < ∞ or µ ( { x : p ( x ) = 0 } ) = 0 .

The optimization problem inf Q ∈M c 1 ,c 2 ,µ,ϵ D f ( P ∥ Q ) can be cast as the following

where q = dQ/dµ . Our goal is to show that q ∗ = d Q ∗ c 1 ,c 2 ,µ,ϵ ( P ) /dµ is an optimal solution of the above optimization problem.

Motivation for the optimality proof. To provide the motivation for the proof of the optimality of q ∗ for the above optimization problem, we first consider the following analogous finite-dimensional optimization problem

for convex function f : (0 , ∞ ) → R and p i > 0 . The convexity of f ⋆ ( x ) = xf (1 /x ) (see Appendix B) implies that the above is a convex optimization problem. For simplicity, we assume that f ⋆ is differentiable. However, note that this assumption is only for simplifying the motivation for the proof, and the result holds for general f or f ⋆ .

We formulate the Lagrangian for the above optimization as

with dual variables ϕ, ψ ∈ [0 , ∞ ) n and ν ∈ R .

The Karush-Kuhn-Tucker (KKT) condition yields:

Now, suppose that there is a feasible point q ∗ ∈ (0 , ∞ ) n satisfying (57), (58), and (59) such that q ∗ = clip( p i /r ; b, be ϵ ) for some r > 0 . We show that q ∗ satisfies the KKT condition for some feasible dual variables ( ϕ, ψ, ν ) .

For ( q ∗ , ϕ, ψ, ν ) to satisfy the KKT condition, the following should hold:

If b < p i /r < be ϵ , then we have q ∗ i = p i /r . From (61) and (64), we must have ϕ i = ψ i = 0 and ν = ( f ⋆ ) ′ (1 /r ) .

If p i /r ≤ b , then q ∗ i = b . Then, ψ i = 0 and ϕ i = ( f ⋆ ) ′ ( b/p i ) -ν .

If p i /r ≥ be ϵ , then q ∗ i = be ϵ . Then ϕ i = 0 and ψ i = ν -( f ⋆ ) ′ ( be ϵ /p i ) .

Now, since f ⋆ is convex, ( f ⋆ ) ′ is monotonically increasing. Hence, the following should be satisfied:

If p i /r ≤ b , then ( f ⋆ ) ′ ( b/p i ) -( f ⋆ ) ′ (1 /r ) ≥ 0 , and

If p i /r ≥ be ϵ , then ( f ⋆ ) ′ (1 /r ) -( f ⋆ ) ′ ( be ϵ /p i ) ≥ 0 .

It can thus be verified that ( q ∗ , ϕ, ψ, ν ) satisfies the KKT condition with

Optimality proof. Next, we present a proof for the optimality of q ∗ for the optimization problem

Here, note that we do not assume f is differentiable. To this goal, we first review the following basic facts about a general convex function f : (0 , ∞ ) → R , which may not be differentiable [64, 65]:

The left derivative f ′ -( x ) := lim h → 0 -f ( x + h ) -f ( x ) h and the right derivative f ′ + ( x ) := lim h → 0+ f ( x + h ) -f ( x ) h exist and finite for every x ∈ (0 , ∞ ) , regardless of whether f is differentiable or not.

For every 0 < x < y , we have f ′ -( x ) ≤ f ′ + ( x ) ≤ f ′ -( y ) ≤ f ′ + ( y ) .

For every x, y (0 , ) and any g [ f ′ ( x ) , f ′ ( x )] , we have

By continuous extension, this holds for y = 0 also. That is, f (0) ≥ f ( x ) -gx for every x ∈ (0 , ∞ ) and g ∈ [ f ′ -( x ) , f ′ + ( x )] .

̸

Let q : X → (0 , ∞ ) be any feasible function satisfying (66) - (68). Recall that f ⋆ ( x ) := xf (1 /x ) is convex, and we can express q ( x ) f ( p ( x ) /q ( x )) = p ( x ) f ⋆ ( q ( x ) /p ( x )) whenever p ( x ) = 0 . Also, whenever p ( x ) = 0 , we bound f ⋆ ( q ( x ) /p ( x )) by the linear approximation of f ⋆ at q ∗ ( x ) /p ( x ) using (69), as follows:

̸

Hence, we have where

Also, whenever p ( x ) = 0 , we have q ( x ) f ( p ( x ) /q ( x )) = q ( x ) p (0) . Hence, we set ζ ( x ) = p (0) and ξ ( x ) = 0 when p ( x ) = 0 , so that

holds for all x ∈ X . We note that ζ ( x ) and ξ ( x ) does not depend on the choice of q ( x ) . Also, the equality holds if q ( x ) = q ∗ ( x ) .

Next, as an analogous to ν in the above motivation for the proof, let ν = ( f ⋆ ) ′ + (1 /r P ) . Since ∫ q ( x ) dµ ( x ) = 1 , we can write

Now, we define the following sets which form a partition of X :

For each case of x ∈ L, M, U , we have q ∗ ( x ) = b, 1 r P p ( x ) , be ϵ , respectively. We observe the following:

If x ∈ M , then ζ ( x ) = ν clearly.

̸

If x ∈ U , then q ∗ ( x ) /p ( x ) < 1 /r P . Since ( f ∗ ) ′ + is monotone increasing, we have ζ ( x ) ≤ ν .

Finally, if p ( x ) = 0 (which implies x ∈ L also), then from the definition f ⋆ ( t ) = tf (1 /t ) , we have ( f ⋆ ) ′ + ( t ) = f (1 /t ) -1 t f ′ -(1 /t ) . From (69) with y = 0 , we have ν = ( f ⋆ ) ′ + (1 /r P ) = f ( r P ) -r P f ′ -( r P ) ≤ f (0) = ζ ( x ) . Hence, ζ ( x ) ≥ ν for every x ∈ L .

If x ∈ L and p ( x ) = 0 , then q ∗ ( x ) /p ( x ) > 1 /r P . Again, since ( f ∗ ) ′ + is monotone increasing, we have ζ ( x ) ≥ ν .

Therefore, since b ≤ q ( x ) ≤ be ϵ , we can write

≥ -✶ -✶ The last expression does not depend on the choice of q ( x ) . Also, we observe that all inequalities become equality if q ( x ) = q ∗ ( x ) . This completes the proof for the optimality of q ∗ , and hence completes the proof of the achievability part.

C.3 Proof of converse part (Proposition C.6)

Let Q ∈ Q X , ˜ P ,ϵ be given. Let A = { A ⊂ X : µ ( A ) = α } . For each A ∈ A , let p A ( x ) = { c 2 if x ∈ A c 1 if x ∈ X\ A . Since c 2 α + c 1 (1 -α ) = 1 , we have ∫ p A ( x ) dµ ( x ) = 1 . Hence for each A , we can define a probability measure P A ˜ by dP A = p A . Also, note that P A ( A ) = c 2 α .

∈ A ∈ P dµ

For each A ∈ A , let β A = Q ( A | P A ) . Then, the push-forward measures of P A and Q ( P A ) by the indicator function ✶ A are Bernoulli distributions with Pr(1) = c 2 α and β A , respectively. By the data processing inequality (Theorem B.2), we have

The main lemma to proceed is the following.

Lemma C.9. Let X be a sample space. Let t, u ∈ N , t > u . Let A 1 , A 2 , · · · , A t ⊂ X be subsets such that for each x ∈ X , we have | { i ∈ [ t ] : x ∈ A i } | ≤ u . Then for any t probability measures Q 1 , · · · , Q t ∈ P ( X ) satisfying that Q i ( A ) ≤ e ϵ Q j ( A ) for all i, j ∈ [ t ] and A ⊂ X , we have

Now, by the assumption that µ is α -decomposable, there exist sequences { t j } ∞ j =1 , { u j } ∞ j =1 ⊂ N of positive integers such that t j > u j , α ≤ u j /t j , lim j →∞ u j /t j = α , and for each j ∈ N , there exist t j subsets A j, 1 , A j, 2 , · · · , A j,t j ⊂ X such that µ ( A j,i ) = α for all i ∈ [ t j ] , and for every x ∈ X , we have ∣ ∣ { i ∈ [ t j ] : x ∈ A j,t j }∣ ∣ ≤ u j . By applying Lemma C.9 to Q i = Q ( P A j,i ) , we obtain min i ∈ [ t j ] β A i,j ≤ ( u j /t j ) e ϵ ( u j /t j ) e ϵ +1 -( u j /t j ) . This implies that inf A ∈A β A ≤ ( u j /t j ) e ϵ ( u j /t j ) e ϵ +1 -( u j /t j ) for all j ∈ N , and by taking the limit j →∞ , we have inf A ∈A β A ≤ αe ϵ αe ϵ +1 -α = be ϵ α .

Since be ϵ < c 2 , we have be ϵ α < c 2 α . By continuity of D B f and the fact that D B f ( λ 1 ∥ λ 2 ) is decreasing in λ 2 ∈ [0 , λ 1 ] , we have

It follows that

Furthermore, we have

and we can derive be ϵ α = 1 -r 1 r 2 -r 1 and b (1 -α ) = r 2 -1 r 2 -r 1 , as follows. From (30), (31) and the definition of r 1 , r 2 , we have

Also, from above and 1 = be ϵ α + b (1 -α ) , we have

This ends the proof of the converse part.

From this, we have

C.4 Deduction to main theorems

In this subsection, we show that Theorem C.4 contains main theorems, Theorems 3.1 and 3.3, as special cases.

C.4.1 Deduction to Theorem 3.1

Recall that in this setup, X = [ k ] , ˜ P = P ([ k ]) = ˜ P 0 ,k,µ k , where µ k is the uniform distribution on [ k ] . That is, ( c 1 , c 2 ) = (0 , k ) . The values of the constants α, b, r 1 , r 2 are

We can easily observe that µ k is ( α, k, 1) -decomposable, because the sets { i } , i ∈ [ k ] , are disjoint and µ k ( { i } ) = 1 k . It follows that µ k is α -decomposable. Hence, we can apply Theorem C.4. By a direct calculation of R ( X , ˜ P , ϵ, f ) = 1 -r 1 r 2 -r 1 f ( r 2 ) + r 2 -1 r 2 -r 1 f ( r 1 ) , we can derive the formula of R ([ k ] , P ([ k ]) , ϵ, f ) presented in Theorem 3.1. It remains to show that the mechanism Q ∗ k,ϵ presented in Theorem 3.1 is the same as Q ∗ 0 ,k,µ k ,ϵ , and show the claim about the range of r P . Recall that

and we claim that r can be chosen such that 1 r ( e ϵ + k 1) /e ϵ .

P ≤ P ≤ -

First, as explained in Section 3.1, we can alternatively write

Since P ( x ) = 1 k dP dµ k ( x ) for each x ∈ [ k ] and P ∈ P ([ k ]) , (102) can be written as

hence,

Hence, Q ∗ k,ϵ = Q ∗ 0 ,k,µ k ,ϵ .

Second, to prove the claim about the range of r P , let us fix P ∈ P ([ k ]) . We need to show that

The left inequality can be easily derived by

For the right inequality, we recall that b = k e ϵ + k -1 . Since P ( x ) ≤ 1 , we have e ϵ e ϵ + k -1 P ( x ) ≤ e ϵ e ϵ + k -1 . Hence, again by using P ( x ) = 1 k dP dµ k ( x ) , we have

The desired inequality follows from Proposition C.2.

C.4.2 Deduction to Theorem 3.3

Recall that in this setup, X = R n , ˜ P = ˜ P c 1 ,c 2 ,h = ˜ P c 1 ,c 2 ,µ , where µ ≪ m and dµ/dm = h . Note that the normalization condition (10) about ( c 1 , c 2 , h, ϵ ) implies the normalization condition (22) about ( c 1 , c 2 , µ, ϵ ) . It can be directly observed that Q ∗ c 1 ,c 2 ,h,ϵ = Q ∗ c 1 ,c 2 ,µ,ϵ , because for each P ∈ ˜ P with corresponding pdf p ( x ) , the chain rule of the Radon-Nikodym derivative shows that p ( x ) = dP dm = dP dµ ( x ) dµ dm ( x ) = dP dµ ( x ) h ( x ) . Hence, once we show that µ is α -decomposable, Theorem C.4 and Proposition C.2 directly contain Theorem 3.3 as a special case. Thus, it remains to show µ is α -decomposable.

In fact, we prove a stronger statement that: µ is ( α, t, u ) -decomposable for any t, u ∈ N such that t > u and α ≤ u/t . Then, since the set of rational numbers is dense in R , this also implies that µ is α -decomposable.

To prove this, let us first introduce the following lemma.

Lemma C.10. Let α ∈ (0 , 1) and t, u ∈ N , t > u , α ≤ u/t . If µ ∈ P ( X ) is ( α/u, t, 1) -decomposable, then µ is also ( α, t, u ) -decomposable.

Proof. In this proof, assume that the sum and subtraction operations performed in subscripts are modulo t operations, with the identification that 0 = t .

By ( α/u, t, 1) -decomposability, there are t disjoint subsets B 1 , B 2 , · · · , B t such that µ ( B i ) = α/u for each i ∈ [ t ] . Using this, for each i ∈ [ t ] , define A i as A i = ∪ u -1 j =0 B i + j . As B i 's are disjoint, we have µ ( A i ) = ∑ u -1 j =0 µ ( B i + j ) = u × ( α/u ) = α for all i ∈ [ t ] . Also, for each x ∈ B i , x is contained in exactly u sets among A 1 , · · · , A t , which are A i , A i -1 , · · · , A i -u +1 . Furthermore, if x / ∈ B i for all i ∈ [ t ] , then x is contained in none of A i . Hence | { i ∈ [ t ] : x ∈ A i } | ≤ u for all x ∈ X . Thus µ is ( α, t, u ) -decomposable.

By this lemma, it suffices to show that for any t ∈ N such that t ≥ 2 and α ≤ 1 /t , µ is ( α, t, 1) -decomposable. As µ ≪ m , the map s ∈ R ↦→ µ (( -∞ , s ] × R n -1 ) is continuous, and as s →-∞ and ∞ , we have µ (( -∞ , s ] → 0 and 1 , respectively. Hence by the intermediate value theorem, for each i ∈ [ t ] , there exists s i ∈ R such that µ (( -∞ , s ] × R n -1 ) = αi . Then, setting A 1 = ( -∞ , s 1 ] × R n -1 and A i = ( s i -1 , s i ] × R n -1 for i ≥ 2 gives the desired A i 's in the definition of decomposability.

In conclusion, µ is α -decomposable, and hence Theorem 3.3 can be deduced from Theorem C.4.

and thus

C.5 Proof of Proposition C.2

Let P ∈ ˜ P be given. Let p = dP dµ , and again assume that c 1 ≤ p ( x ) ≤ c 2 for all x ∈ X . Similar to the proof of the achievability part, for each r > 0 , let us define the following sets which form a partition of X :

For r = 0 , we let L 0 = { x ∈ X : p ( x ) = 0 } , U 0 = { x ∈ X : p ( x ) > 0 } , M 0 = ∅ . Also, let q r ( x ) = clip ( 1 r p ( x ); b, be ϵ ) . Then, q r ( x ) = b, 1 r p ( x ) , be ϵ for x ∈ L r , M r , U r , respectively.

Suppose first that c 1 = 0 . Then r 1 = 0 , α = 1 c 2 , and b = c 2 e ϵ + c 2 -1 . Observe that

First, we show that ∫ q r 1 ( x ) dµ ( x ) ≥ 1 . We divide the case of c 1 = 0 and c 1 > 0 .

hence µ ( U 0 ) ≥ 1 /c 2 . It follows that

Next, suppose that c 1 > 0 . Then r 1 > 0 . From p ( x ) ≥ c 1 , we have 1 r 1 p ( x ) ≥ c 1 r 1 = b . Hence L r 1 = ∅ . Thus

Let S 1 = ∫ M r 1 p ( x ) dµ ( x ) and T 1 = µ ( U r 1 ) , so that

As c 1 ≤ p ( x ) ≤ c 2 , we have and

From these, we can get

As c 1 < c 1 e ϵ < c 2 , we can express c 1 e ϵ as a convex combination of c 1 and c 2 , as

Hence, by taking c 2 -c 1 e ϵ c 2 -c 1 [ equation (130) ] + c 1 e ϵ -c 1 c 2 -c 1 [ equation (131) ] , we get

Similarly, we show that ∫ q r 2 ( x ) dµ ( x ) ≤ 1 . from p ( x ) ≤ c 2 , we have 1 r 2 p ( x ) ≤ c 2 r 2 = be ϵ . Hence U r 2 = ∅ . Thus

Thus we have ∫ q r 1 ( x ) dµ ( x ) ≥ 1 .

Similar as above, let S 2 = µ ( L r 2 ) and T 2 = ∫ M r 2 p ( x ) dµ ( x ) , so that

and, we have and

hence

As c 1 ≤ c 2 e -ϵ ≤ c 2 , we have and by the same reason, we have

Thus we have ∫ q r 2 ( x ) dµ ( x ) ≤ 1 .

Finally, we show that ∫ q r 1 ( x ) dµ ( x ) = 1 implies ∫ q r ( x ) dµ ( x ) = 1 for all r ∈ [ r 1 , r 2 ] . Let us assume ∫ q r 1 ( x ) dµ ( x ) = 1 . We first claim that: p ( x ) = c 2 for µ -a.e. x ∈ U r 1 , and p ( x ) = c 1 for µ -a.e. x ∈ X\ U r 1 . Again, we divide the case of c 1 = 0 and c 1 > 0 .

Next, suppose that c 1 > 0 . Again, by tracking the proof of ∫ q r 1 ( x ) dµ ( x ) ≥ 1 , We can observe that the equality ∫ q r 1 ( x ) dµ ( x ) = 1 is equivalent to any of the following statements:

Suppose first that c 1 = 0 . By tracking the proof of ∫ q r 1 ( x ) dµ ( x ) ≥ 1 , We can observe that the equality ∫ q r 1 ( x ) dµ ( x ) = 1 holds if and only if µ ( U 0 ) = 1 /c 2 , if and only if p ( x ) = c 2 for µ -a.e. x ∈ U 0 . By definition of U 0 , we have p ( x ) = 0 = c 1 for all x ∈ X\ U 0 . Hence we get the claim.

Both S 1 + c 1 T 1 = c 1 and S 1 + c 2 T 1 = 1 holds.

Both ∫ M r 1 p ( x ) dµ ( x ) = c 1 µ ( M r 1 ) and ∫ U r 1 p ( x ) dµ ( x ) = c 2 µ ( U r 1 ) holds.

p ( x ) = c 1 for µ -a.e. x ∈ M r 1 and p ( x ) = c 2 for µ -a.e. x ∈ U r 1 .

Since L r 1 = ∅ , we also get the claim.

WLOG, assume that p ( x ) = c 2 for all x ∈ U r 1 , and p ( x ) = c 1 for all x ∈ X\ U r 1 . Then, for every r ∈ ( r 1 , r 2 ] , we have the following:

For each x ∈ U r 1 , we have 1 r p ( x ) ≥ c 2 r 2 = be ϵ , hence q r ( x ) = be ϵ .

For each x U

Also, we have 1 = ∫ p ( x ) dµ ( x ) = c 2 µ ( U r 1 ) + c 1 (1 -µ ( U r 1 )) , hence µ ( U r 1 ) = 1 -c 1 c 2 -c 1 = α . It follows that for every r ∈ ( r 1 , r 2 ] , we have ∫ q r ( x ) dµ ( x ) = be ϵ µ ( U r 1 ) + b (1 -µ ( U r 1 )) = be ϵ α + b (1 -α ) = 1 . This concludes the proof.

C.6 Proof of Lemma C.7

Although the proof can be found in [59], we present the proof here for the completeness.

If r 1 = 0 and f (0) = ∞ , then the RHS of the inequality we want to show is ∞ , thus it becomes trivial. Hence, we may assume that either r 1 > 0 or f (0) < ∞ .

By the assumption, p ( x ) /q ( x ) is the convex combination of r 1 and r 2 for µ -a.e. x ∈ X , as follows.

Hence, by the convexity of f and ∫ p ( x ) dµ ( x ) = ∫ q ( x ) dµ ( x ) = 1 , we have

C.7 Proof of Lemma C.9

First, we claim that for each i ∈ [ t -u ] , we can construct a partition { B i, 1 , B i, 2 , · · · , B i,u } of A u + i into u (measurable) sets, such that for each j ∈ [ u ] , the sets A j , B 1 ,j , B 2 ,j , · · · , B t -u,j are disjoint. The construction is in the inductive way as follows: Given i ∈ [ t -u ] , suppose that we have constructed

such partitions of A u +1 , · · · , A u + i -1 such that for each j ∈ [ u ] , A j , B 1 ,j , B 2 ,j , · · · , B i -1 ,j are disjoint. Let C i,j = A j ∪ ( ∪ i -1 k =1 B k,j ) . Then for each x ∈ A u + i , at least one of C i, 1 , C i, 2 , · · · , C i,u does not contain x , because

where the last line is from that since x ∈ A u + i , at most u -1 sets among A 1 , A 2 , · · · , A u + i -1 can contain x . Hence, setting

gives the partition { B i, 1 , B i, 2 , · · · , B i,u } of A u + i , such that for each j ∈ [ u ] , C i,j and B i,j are disjoint. This implies that A j , B 1 ,j , B 2 ,j , · · · , B i,j are disjoint.

Now, let ℓ = min i ∈ [ t ] Q i ( A i ) . Using these partitions, we can now show that

Hence

D Proofs of Remaining Propositions

We present the proofs of remaining propositions in the paper, Propositions 3.2 and 4.1.

D.1 Proof of Proposition 3.2

By the assumption, there are subsets { A i } ∞ i =1 of X which are pairwise disjoint and P i ( A i ) = 1 for all i ∈ N . Let us pick P 0 ∈ ˜ P , and let Q 0 = Q ( P 0 ) . Since A i 's are disjoint, we have ∑ ∞ i =1 Q 0 ( A i ) = Q 0 ( ⋃ ∞ i =1 A i ) ≤ 1 < ∞ . Hence, we have lim i →∞ Q 0 ( A i ) = 0 . Also, by definition of ϵ -LDP, for any P ∈ ˜ P and i ∈ N , we have Q ( P )( A i ) ≤ e ϵ Q 0 ( A i ) . Especially, this implies Q ( P i )( A i ) ≤ e ϵ Q 0 ( A i ) , and thus lim i →∞ Q ( P i )( A i ) = 0 .

Now, similar to the converse proof in Section C.3, let β i = Q ( P i )( A i ) . Then, the push-forward measures of P i and Q ( P i ) by the indicator function ✶ A are Bernoulli distributions with Pr(1) = 1 and β i , respectively. By the data processing inequality (Theorem B.2), we have

Since lim i →∞ β i = 0 , by continuity of D B f , we have

where the last equality is because two Bernoulli distributions with Pr(1) = 1 and Pr(1) = 0 , respectively, are mutually singular. Hence, we must have R f ( Q ) = M f .

D.2 Proof of Proposition 4.1

For generality, we prove that the statement of Proposition 4.1 holds in the general setup in Definition C.1. That is, for the setup in Definition C.1, the mechansim Q ∗ = Q ∗ c 1 ,c 2 ,µ,ϵ satisfies

for all P, P ′ ∈ ˜ P . As the mechanisms Q ∗ k,ϵ and Q ∗ c 1 ,c 2 ,h,ϵ in the paper are special cases of Q ∗ c 1 ,c 2 ,µ,ϵ , a proof in this general setup induces Proposition 4.1.

Now, let us assume the setup in Definition C.1 Let P, P ′ ∈ ˜ P be given. WLOG, assume that r P ≥ r P ′ . Let p = dP/dµ , p ′ = dP ′ /dµ , and q ( x ) = clip ( 1 r P p ( x ); b, be ϵ ) , q ′ ( x ) = clip ( 1 r P ′ p ′ ( x ); b, be ϵ ) , so that q = d Q ∗ ( P ) /dµ and q ′ = d Q ∗ ( P ′ ) /dµ . For simplicity, we denote clip( x ) := clip( x ; b, be ϵ ) .

We first note the fact that clip( x ) is monotone increasing and 1-Lipschitz in x . From this and the equivalent expressions of the total variation distance in Appendix B, we have

This ends the proof.

E Behaviors of Proposed Mechanisms

In this appendix, we present the formal proofs for the behaviors of the proposed mechanisms presented in Sections 3.1 and 3.2.

We first observe that the formula of the optimal worst-case f -divergence in general case

is the y -coordinate value at x = 1 of the line segment joining ( r 1 , f ( r 1 )) and ( r 2 , f ( r 2 )) . As f is convex, this formula is increasing in r 2 and decreasing in r 1 , provided that r 1 < 1 < r 2 . Now, let us present the proofs.

If f (0) = ∞ and ˜ P contains two mutually singular distributions, then R ( X , ˜ P , ϵ, f ) = ∞ .

Proof. Let P 1 , P 2 ∈ ˜ P be mutually singular distributions with disjoint supports A 1 , A 2 ⊂ X respectively (That is, P 1 ( A 1 ) = P 2 ( A 2 ) = 1 and A 1 ∩ A 2 = ∅ ). Suppose that Q is an ϵ -LDP sampling mechanism for ( X , ˜ P ) such that R f ( Q ) < ∞ . Since f (0) = ∞ , D f ( P ∥ Q ) < ∞ , implies Q ≪ P . Hence, as D f ( P i ∥ Q ( P i )) < ∞ and P i ( A c i ) = 0 , we have Q ( A c i | P i ) = 0 for each i = 1 , 2 . As Q satisfies ϵ -LDP, we have Q ( A c 1 | P 2 ) ≤ Q ( A c 1 | P 1 ) = 0 , Q ( A c 1 | P 2 ) = 0 . Now, since A 1 ∩ A 2 = ∅ , we have A c 1 ∪ A c 2 = X . But by the union bound, 1 = Q ( X| P 2 ) ≤ Q ( A c 1 | P 2 ) + Q ( A c 2 | P 2 ) = 0 , which is a contradiction. Hence, R f ( Q ) = ∞ for every ϵ -LDP sampling mechanism Q .

From now, assume f (0) < ∞ .

Let us first prove the behaviors for the case of finite X in Section 3.1.

R ([ k ] , P ([ k ]) , ϵ, f ) is decreasing in ϵ and increasing in k .

Proof. Recall from Appendix C.4.1 that the case of X = [ k ] , ˜ P = P ([ k ]) can be fit into the general case with

Here, r 2 is decreasing in ϵ and increasing in k . As (174) is increasing in r 2 , we get the desired claim.

For a fixed k , we have R ([ k ] , P ([ k ]) , ϵ, f ) → 0 as ϵ →∞ .

Proof. As ϵ → ∞ , we have r 2 → 1 . As f is continuous, f (1) = 0 , and f (0) < ∞ , we obtain from (174) that

For a fixed k , as ϵ → 0 , we have Q ∗ k,ϵ ( x | P ) → 1 /k for every P ∈ P ([ k ]) and x ∈ [ k ] .

Proof. We know that 1 e ϵ + k -1 ≤ Q ∗ k,ϵ ( x | P ) ≤ e ϵ e ϵ + k -1 . As ϵ → 0 , both of 1 e ϵ + k -1 and e ϵ e ϵ + k -1 converges to 1 k , hence we get the desired claim.

Next, let us prove the behaviors for the continuous case in Section 3.2.

If c 1 = 0 and f (0) = , then ( R n , ˜ c ,c ,h , ϵ, f ) = .

∞ R P 1 2 ∞ r 2 -r 1 ∞ R P ∞

For a fixed ( c 1 , c 2 ) , ( R n , ˜ c ,c ,h , ϵ, f ) is decreasing in ϵ

Proof. We can observe that r 1 is increasing in ϵ and r 2 is decreasing in ϵ . Since (174) is increasing in r and decreasing in r , we get the desired claim.

R P 1 2 . 2 1

For a fixed ( c 1 , c 2 ) with c 1 = 0 , as ϵ →∞ , we have R ( R n , ˜ P c 1 ,c 2 ,h , ϵ, f ) → 0 .

Proof. We can observe that r 1 = 0 and r 2 → 1 . Hence, by the same argument as (177), we have the desired claim.

F Detailed Explanation of Setups in Numerical Results

We present details about the setups in producing numerical results in Section 5, and generating Figure 1 in Section 1.

In this appendix, for each µ ∈ R n , N µ, Σ [ x ] = 1 (2 π ) n/ 2 √ det(Σ) exp ( -( x -µ ) T Σ -1 ( x -µ ) 2 ) is the pdf of the n -dimensional Gaussian distribution with mean µ and covariance Σ . Note that we denote N ( µ, Σ) to refer the Gaussian distribution itself with mean µ and covariance matrix Σ .

F.1 Explanation for numerical result for finite data space

In this appendix, we show that among the choices of Q 0 in the baseline for X = [ k ] , choosing Q 0 to be the uniform distribution minimizes R f ( Q ) , and present the precise value of R f ( Q ) for the baseline with uniform Q 0 . From now, let us fix k, ϵ, f , and we denote Q Q 0 to mean the baseline with the reference distribution Q 0 . Also, let δ x ∈ P ([ k ]) be the point mass at x ∈ [ k ] . Recall that

and we set the baseline to satisfy that

First, if Q 0 ( x ) = 0 for some x ∈ [ k ] , then Q Q 0 ( x | P ) ≤ e ϵ/ 2 Q 0 ( x ) = 0 , Q Q 0 ( x | P ) = 0 for every P ∈ P ([ k ]) . Especially, Q Q 0 ( x | δ x ) = 0 , and this implies that Q Q 0 ( δ x ) and δ x are mutually singular. Hence,

We note that for every x ∈ [ k ] and P ∈ P ([ k ]) , we have we have Q Q 0 ( x | P ) ≤ e ϵ/ 2 Q 0 ( x ) , and furthermore concluding that R f ( Q Q 0 ) = M f . Hence, to minimize R f ( Q Q 0 ) , it suffices to set Q 0 ( x ) > 0 for all x ∈ [ k ] . Hence from now, we only consider such Q 0 .

Letting we have

Note that B ( t ) is increasing in t . Now, for any x ∈ [ k ] , we have

We can observe that the last term can be written in the form of the optimal worst-case f -divergence (174) with r 1 = 0 and r 2 = 1 / Q Q 0 ( x | δ x ) . In other words, let

Then we have R f ( Q Q 0 ) ≥ R (0 , 1 / Q Q 0 ( x | δ x )) .

As noted in Appendix E, R ( r 1 , r 2 ) is increasing in r 2 for 0 ≤ r 1 < 1 < r 2 . Since Q Q 0 ( x | δ x ) ≤ B ( Q 0 ( x )) , we have

Since this should hold for all x ∈ [ k ] , we have

Since min x ∈ [ k ] Q 0 ( x ) ≤ 1 /k for all Q 0 ∈ P ([ k ]) , and B ( t ) is increasing in t , we have

Now, let µ k be the uniform distribution over [ k ] . We will show that

which suffices to prove that Q 0 = µ k minimizes R f ( Q Q 0 ) .

We observe that M ϵ,µ k is a convex set in P ([ k ]) . Since D f ( P ∥ Q ) is jointly convex in ( P, Q ) , we obtain that D f ( P ∥ Q µ k ( P )) = min Q ∈M ϵ,µ k D f ( P ∥ Q ) is convex in P by [64, Section 3.2.5]. Hence, the maximum of D f ( P ∥ Q µ k ( P )) occurs when P is one of the extreme points of P ([ k ]) , that is, the point masses δ x . By the same arguments as in (187)-(189), we have D f ( δ x ∥ Q ) = R (0 , 1 /Q ( x )) , and hence

Also, by the same arguments as in (181)-(184), we have

Now, we will show that sup Q ∈M ϵ,µ k Q ( x ) = B (1 /k ) for any x ∈ [ k ] . To show this, we will prove that there is a distribution Q ∈ P ([ k ]) such that Q ( x ) = B (1 /k ) and Q ( y ) = 1 -B (1 /k ) k -1 for all y ∈ [ k ] \{ x } , and this Q is contained in M ϵ,µ k . It suffices to show the followings:

We note that whenever 0 ≤ t ≤ 1 , we have

From these, we can easily observe that B (1 /k ) ≤ min { 1 , 1 k e ϵ/ 2 } , and

Also,

This shows that sup Q ∈M ϵ,µ k Q ( x ) = B (1 /k ) for any x ∈ [ k ] . Thus,

This ends the proof that Q 0 = µ k minimizes R f ( Q Q 0 ) , and R f ( Q µ k ) = R (0 , 1 /B (1 /k )) .

F.2 Explanation for the experiment in 1D Gaussian mixture

The precise description of the set of possible client distributions in our experimental setup is as follows:

Each of P j ∈ ˜ P is generated by choosing k, λ i , and µ i in (208) as follows: (i) First, choose the number of Gaussian distributions k by first sample ˜ k from the Poisson distribution with mean k 0 (which is chosen beforehand), and let k = min( ˜ k + 1 , K ) , and (ii) after that, sample each of µ 1 , · · · , µ k independently from the uniform distribution on [ -1 , 1] , and sample ( λ 1 , · · · , λ k ) from the uniform distribution on P ([ k ]) .

The implementation of our proposed mechanism is as follows. We can observe that

where Φ is the cdf of the 1D standard Gaussian distribution, and for each x ∈ [ -4 , 4] , we have

Hence, we have ˜ P ⊂ ˜ P 0 , 1 , ˜ h = ˜ P 0 ,c 2 ,h , where

and c 2 = ∫ ˜ h ( x ) dx , h ( x ) = ˜ h ( x ) /c 2 . When implementing our proposed mechanism, we use the bisection method to find a constant r P . We predetermine the error tolerances δ 1 , δ 2 ≥ 0 , δ 1 < 1 , and we terminate the bisection method to find r P if the integration of (13) lies in the interval [1 -δ 1 , 1+ δ 2 ] . As mentioned in Section 4.1, to consider the error tolerances, we actually implement Q ∗ 0 ,c 2 ,h,ϵ ′ ( P ) , with ϵ ′ = ϵ -log 1+ δ 2 1 -δ 1 .

In the experiment in the paper, we use k 0 = 2 and δ 1 = δ 2 = 10 -5 .

For the baseline, we use the same hyperparameter setups as in [35, Section 5, Paragraph 'Architectures"], except a slight modification in a reference distribution Q 0 . In [35], they set the standard Gaussian as the reference distribution, Q 0 = N (0 , 1) . To consider the truncated domain [ -4 , 4] , we set Q 0 as the truncation of the standard Gaussian, that is, Q 0 has a pdf

The baseline has many other hyperparameters not specified in [35], such as the proposal distribution used for the Metropolis-Hasting algorithm [66, 67], initial model parameters for the weak learner, etc. We noticed that the code for [35] is available at https://github.com/karokaram/ PrivatedBoostedDensities/tree/master and hence we made every effort to faithfully reproduce the baseline with exactly the same hyperparameters, including those not mentioned in the paper [35]. However, subtle variations may arise due to differences in the programming languages employed; we used Python, while [35] used Julia.

F.3 Explanation for Figure 1

For k ∈ N and σ > 0 , the Gaussian ring distribution with k modes and a component-wise variance σ 2 is the mixture of k Gaussian distributions in R 2 with equal weights, where each Gaussian distribution has the covariance σ 2 I 2 and equally spaced mean in the unit circle, and one of the Gaussian distribution has mean (1 , 0) . That is, it has a density 1 k ∑ k i =1 N µ i ,σ 2 I 2 [ x ] , with µ i = (cos 2 πi k , sin 2 πi k ) . In Figure 1, the left image is the pdf of the Gaussian ring distribution with k = 3 and σ 2 = 0 . 5 . It is used as a client's original distribution.

For our proposed mechanism, similar to Section 3.2, we use the setup that ˜ P consists of Gaussian mixtures, where each Gaussian has mean within a unit ball centered at the origin and has covariance 0 . 5 I 2 . that is, ˜ P = { ∑ k i =1 λ i N ( x i , σ 2 I 2 ) : k ∈ N , λ i ≥ 0 , ∑ k i =1 λ i = 1 , ∥ x i ∥ ≤ 1 } , with σ 2 = 0 . 5 . We can also observe that ˜ P ⊂ ˜ P 0 , 1 , ˜ h for ˜ h ( x ) = 1 2 πσ 2 exp ( -[max(0 , ∥ x ∥-1)] 2 2 σ ) .

Same as Appendix F.2, we have ˜ P 0 , 1 , ˜ h = ˜ P 0 ,c 2 ,h , where c 2 = ∫ ˜ h ( x ) dx , h ( x ) = ˜ h ( x ) /c 2 . Hence, we use Q ∗ 0 ,c 2 ,h,ϵ . The implementation of Q ∗ c 1 ,c 2 ,h,ϵ ( P ) is the same as the description in Appendix F.2 with δ = 10 -5 . For the baseline, we also use MBDE with the same hyperparameter setup as [35, Section 5, Paragraph 'Architectures"], with the (untruncated) 2D standard Gaussian as a reference distribution, Q 0 = N (0 , I 2 ) .

G Additional Numerical Results for Finite Space

In this appendix, we present more numerical results for the finite space case explained in Section 5.1 for other k . We present the result for k = 5 , 20 , and 100 in Figures 5-7. As shown by the figures, the proposed mechanism always has the smaller worst-case f -divergence compared to the baseline, as the optimality is proved for the proposed mechanism. However, the performance gap between two mechanisms decreases as ϵ becomes smaller and k becomes larger.

0.1

0.5

1.0

2.0

5.0

0

.

0

0

.

5

1

.

0

1

.

5

Worst

f

-divergence

KL

0.1

0.5

1.0

2.0

5.0

Privacy budget

/epsilon1

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

TV

0.1

0.5

1.0

2.0

5.0

0

.

0

0

.

2

0

.

4

Sq. Hel

Proposed

Baseline

0.1

0.5

1.0

2.0

5.0

0

1

2

3

Worst

f

-divergence

KL

0.1

0.5

1.0

2.0

5.0

Privacy budget

/epsilon1

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

TV

0.1

0.5

1.0

2.0

5.0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

Sq. Hel

Proposed

Baseline

0.1

0.5

1.0

2.0

5.0

0

1

2

3

4

Worst

f

-divergence

KL

0.1

0.5

1.0

2.0

5.0

Privacy budget

/epsilon1

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

TV

0.1

0.5

1.0

2.0

5.0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

Sq. Hel

Proposed

Baseline

H Instructions for Reproducing Results

In this appendix, we provide instructions for reproducing the experiments and figures in the paper. For a detailed document about the code, refer to the provided file README.md . For the tasks consuming a large time, we also specify the running times of such tasks. We do not specify the running times of the short tasks consuming less than 5 seconds. All experiments are performed on our simulation PC with the following specifications:

OS: Ubuntu 22.04.1

CPU: Intel(R) Core(TM) i9-9900X

Memory: 64GB

There are a few remarks:

Although we expect that the codes can be run and reproduce our results on sufficiently recent versions of Python libraries, we provide the information about the anaconda environment used in the experiment in environment.yaml for the completeness.

The provided codes contain some lines to make the figures use T E X fonts. Running such lines requires the L A T E X to be installed in the experimental environment. We can remove the following lines to disable using T E X:

H.1 Instruction for producing Figure 1

Figure 1 can be obtained by running the code plot_GaussRing.py , without any program arguments. We measure the running time for initializing the mechanism and and calculating the sampling distribution for the baseline (MBDE [35]) and our proposed mechanism. The measured running times in our environment are as follows: (unit: second)

Initializing the mechanism

-Baseline: 0.80

-Proposed: 1.22

Calculating the sampling distribution

Proposed: 664.18

H.2 Instruction for producing Figure 2

Figure 2 can be obtained by running the code visualize_finiteSpace.py , without any program arguments.

H.3 Instruction for producing results for finite space

The results about the theoretical worst-case f -divergence for finite space, Figures 3,5,6,7, can be obtained by running the code plot_finite.py with a program argument --k to specify the value of k . For example, in the command line, the aforementioned four figures can be generated by the following commands, respectively:

H.4 Instruction for producing results for 1D Gaussian mixture

The experiment of 1D Gaussian mixture in Section 5.2 consists of the following two codes:

The results in the paper, Figure 4, can be obtained as follows:

These can be run in any order or in parallel. Check that all of the five result files data_1DGaussMix_eps{eps}.npy corresponding to five values of ϵ are generated. In our environment, running all of the five commands in parallel consumes 3h 13m 35s.

Then, run the code plot_1DGaussMix.py without any program arguments.

I Limitations

Our main contribution lies in proposing a minimax-optimal mechanism, and we present several experimental results based on synthetic data to support the superiority of our mechanism. However, since we have not conducted experiments based on real datasets, the analysis of performance in real-world scenarios is insufficient. Also, our PUT formulation in the minimax sense provides optimal utility in the worst case, which may result in reduced average utility when prior information is given. Finally, our implementation of the proposed mechanism requires a large amount of running time due to numerical integration, which makes experiments in multidimensional spaces challenging.

J Broader Impacts

Our proposed mechanism can be utilized for privacy protection in the field of generative models, which has recently received significant attention. A major deterrent to the adoption of privacy protection algorithms in real-world scenarios is the potential performance degradation. Our PUToptimal mechanism, that minimizes the loss of utility given the privacy budget, can alleviate such concerns.

We should note that an LDP mechanism provides a certain level of privacy protection but cannot guarantee complete privacy protection without completely sacrificing utility. Additionally, clients typically provide multiple data points through various channels, and when these data are combined, it can lead to greater privacy leakage [68, 69].

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly stated the main contributions and scope of our paper in the abstract and introduction (Section 1).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We stated the limitations of our work in Appendix I.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All theorems, propositions, and lemmas in the paper include all the necessary assumptions, and are accompanied with either the proofs in the appendices or relevant references.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provided instructions to reproduce all the figures and experimental results in Appendix H.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We plan to make the code attached in the supplementary material publicly open after the review period. Also, the instructions to reproduce the results are fully specified in Appendix H.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We presented enough details to implement our proposed mechanism, and we attached the code for implementing both the proposed mechanism and the baseline in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: As the main contribution is theoretically characterizing the worst-case utility, we focused on extracting the worst-case utility in the experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provided the information about the computer resources and running times for the experiments in Appendix H.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discussed both the positive and negative societal impacts in Appendix J.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not recognize any risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: All of the codes only use the basic Python libraries like numpy, scipy, torch, and so on. No other external assets are used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: As mentioned in Appendix H, we documented the details of our codes in the file README.md , which is provided by the NeurIPS Code and Data Submission Guidelines.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nConsider the following setting: A public domain $\\mathcal{X}$, a family of distributions $\\mathcal{P}$ over $\\mathcal{X}$, and a mechanism $M$ which give a distribution $P \\in \\mathcal{P}$ releases an element $x \\in \\mathcal{X}$, such that $M$ is $\\epsilon$-pure local DP with respect to the input, that is - the output will have similar distribution for any two distribution inputs $P, P' \\in \\mathcal{P}$. Can we expect the output distribution - that is, the distribution of output elements $x$ given an input distribution $P$ - to be accurate in any way, where accuracy is measured by some $f$-divergence between the distribution of outputs and the input distribution.\n\nAt first, this seems like a hopeless task, since the mechanism has to maintain privacy even against distributions that share no support. Surprisingly, as this paper proves, there is some hope and things are not that bleak.\n\nThis setting was first considered by [1], who proposed optimal mechanisms for finite and continuous domains, with respect to some given reference distribution, for KL-divergence. This paper removes the dependence on a given reference distribution, and instead provides an intuitive algorithm that is optimal for the finite case and does not depend on the choice of $f$. It extends these results to the continuous case, for a family of distributions that are point-wise bounded from above and below by some reference distribution up to scale. Finally, it provides numerical evaluations for the continuous case, which demonstrate the improved utility.\n\n\n[1] Hisham Husain, Borja Balle, Zac Cranko, and Richard Nock. Local Differential Privacy for Sampling. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, pages 3404–3413. PMLR, June 2020.\n\nSTRENGTHS:\nThis paper is clearly written. Its contribution lays both in its conceptual framing as well as in its theoretical results.\n\nWEAKNESSES:\nThough many of the ideas and proofs in this work are novel, the results can be viewed as a generalization of previously known results as mentioned in the summary.\n\nThough the results achieved in this paper are optimal, they are still quite weak. This is not surprising. As discussed in the summary, the very fact that we can learn anything in this setting is already surprising, given that (in the finite domain setting) the output given two different singleton distributions must be indistinguishable. As a result, the output distribution is heavily skewed towered the uniform distribution.\n\nThe main motivation for this setting is somewhat unclear. Since each user has access to a single distribution, we want to maintain privacy w.r.t. the user's distribution, and so we release a single element, the output is useful only under some assumption on the relation between the users' distributions.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of sampling from a user's distribution under local differential privacy (LDP). Namely, a client has a distribution P in some set of distributions, where the set is known to a data curator. The client wants to perturb P to get Q, such that a sample from Q sent to the data curator satisfies LDP with respect to P (i.e., for any P, P' the probability of sampling a certain outcome differs by at most $e^\\epsilon$ multiplicatively), but also minimizes a given f-divergence between P and Q. The authors give lower bounds on the f-divergence achievable for a given LDP guarantee in two settings, one where the set of distributions is over a discrete support, and another where the set of distributions is over a continuous support, but is lower and upper bounded by some constants times a base measure. They also give an explicit perturbation in these settings that exactly matches the lower bound, i.e. is an optimal solution to the problem. They also discuss why in the continuous setting the lower/upper bound assumption is necessary to get meaningful bounds (in particular, to avoid two disjoint point distributions being in the support), show how to extend their results to a setting where clients only have samples from their distribution, and perform experiments showing that their perturbation achieves better divergence bounds than the previous work \"Local Differential Privacy for Sampling\".\n\nSTRENGTHS:\n* The paper gives matching upper and lower bounds for a very general form (i.e. for arbitrary f-divergence, rather than a specific divergence) of a natural and important problem of generating private samples with LDP\n* The assumptions for the settings where the authors derive optimal results are well-justified - in the continuous setting, the authors demonstrate that their assumption (10) is somewhat necessary to derive any meaningful results, and the authors discuss how to extend to a more practical setting of clients having samples rather than distributions\n* Numerical results support the theoretical results and make them easier to contextualize, specifically they demonstrate significant improvements in the bounds over the past work\n\nWEAKNESSES:\n* While the proofs of Theorems 3.1 and 3.3 in the appendix are far from trivial, given knowledge of the randomized response mechanism, it's not clear to me that the optimal bounds are as surprising as the authors suggest. See Questions for more detail. \n* The introduction suggests one of the major improvements of the paper over [35] is avoiding the optimality depending on the choice of reference distribution. However, it seems that once we relax reference distribution to measure as the authors do, in both settings the authors consider, there is a natural choice of reference measure to apply the approach of [35] to. It also seems like the results could be retrieved just shifting the lower and upper bounds on the probabilities that [35] uses. While the authors justify the assumptions in the continuous setting that lead to an easy choice of reference distribution, and a lot of work goes into proving optimality of the authors' suggested mechanisms, it's seems both Theorems 3.1 and 3.3 are achievable via a slight variation of the technique in [35], which suggests limited originality/significance for the algorithmic techniques in the paper.\n\nQUESTIONS:\n* In the discrete setting, we could sample from P and then apply a randomized response to the sample. This seems to achieve the same f-divergence bound (7) in the extreme case where P is a point distribution. This mechanism might be worse for non-extreme distributions, but for the minimax f-divergence this might now matter. Is this mechanism also minimax optimal? If so, in the continuous case is the following mechanism minimax optimal? We sample from P, with some probability p keep the sample, otherwise sample from the distribution with pdf proportional to the base measure h. p can be chosen to be the largest value that preserves LDP.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}

comments: SUMMARY:
This paper addresses the problem of sampling under local differential privacy (LDP) requirements, focusing on the privacy-utility trade-off (PUT). It defines the fundamental PUT of private sampling in the minimax sense, using f-divergence as the utility measure. The authors characterize the exact PUT for both finite and continuous data spaces and propose sampling mechanisms that are universally optimal for all f-divergences. The numerical experiments demonstrate the superiority of these mechanisms over existing baselines, showcasing their empirical utilities.

STRENGTHS:
1. The paper presents a novel approach to analyzing the privacy-utility trade-off (PUT) in locally differentially private (LDP) sampling, utilizing f-divergence as the utility measure.
2. The authors offer optimal mechanisms for both finite and continuous data spaces, given certain assumptions. Figure 2 provides a clear visualization that enhances understanding of the proposed mechanisms.

WEAKNESSES:
The constant $r_P$ does not have a closed form, requiring calculation through approximate methods. This introduces potential approximation errors that could degrade the privacy or utility guarantees. See question 1.

QUESTIONS:
1. In practice, would the approximation error of calculating $r_P$ degrade the $\epsilon$-LDP guarantee to $(\epsilon,\delta)$-LDP with $\delta>0$?
2. Is the mechanism $Q^*_{k,\epsilon}$ uniquely optimal?
3. Is there a typo on line 127: there may be no $r_P>0$ such that (6) **does not** sum to one?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the issue of protecting probability distributions with local differential privacy when they are sampled. This problem has been recently studied in view of potential applications to generative models. In particular, the paper studies mechanisms that, given a distribution, produce a \"sampler\", i.e., a sampling mechanism that can be seen as an obfuscated version of the original distribution and which is used to produce samples in replacement of the original distribution.  The paper investigates the privacy-utility trade-off of the proposed method, where utility is measured in terms of f-divergence between the original distribution and the sampling distribution.\n\nSTRENGTHS:\n+ The proposed method is shown to be *universally optimal*, i.e., the produced sampling mechanism minimizes any f-divergence  wrt the original distribution\n\n+ For finite data spaces, the paper presents a closed-form expression for the utility of both the proposed mechanism and the baseline method proposed in [35], allowing for an exact comparison of the utilities.\n\nWEAKNESSES:\n- The paper should justify better, in the introduction, why inferring a distribution presents a privacy risk. In general, in the DP literature, we want to protect an individual data point, not the distribution that can be inferred from the data collection.\n\nQUESTIONS:\nCan you please explain why knowing a distribution presents a privacy risk?  In general, the problem in privacy consists in protecting the individual data point, while allowing statistics to be derived, including the inference of the distribution from which the data points are drawn.\n\nLIMITATIONS:\nThe limitations are discussed. In particular, the paper points out that there are other notions of utility (distances between distributions) that could be considered.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-28"}