Title: Unified Mechanism-Specific Amplification by Subsampling and Group Privacy Amplification
Venue: NeurIPS 2024
Accepted: yes
Keywords: Differential Privacy

Privacy-Preserving Machine Learning

Amplification

Subsampling

Accounting
Abstract: Amplification by subsampling is one of the main primitives in machine learning with differential privacy (DP): Training a model on random batches instead of complete datasets results in stronger privacy. This is traditionally formalized via mechanism-agnostic subsampling guarantees that express the privacy parameters of a subsampled mechanism as a function of the original mechanism's privacy parameters. We propose the first general framework for deriving mechanism-specific guarantees, which leverage additional information beyond these parameters to more tightly characterize the subsampled mechanism's privacy. Such guarantees are of particular importance for privacy accounting, i.e., tracking privacy over multiple iterations. Overall, our framework based on conditional optimal transport lets us derive existing and novel guarantees for approximate DP, accounting with Renyi DP, and accounting with dominating pairs in a unified, principled manner. As an application, we analyze how subsampling affects the privacy of groups of multiple users. Our tight mechanism-specific bounds outperform tight mechanism-agnostic bounds and classic group privacy results.

Unified Mechanism-Specific Amplification by Subsampling and Group Privacy Amplification

Abstract

Amplification by subsampling is one of the main primitives in machine learning with differential privacy (DP): Training a model on random batches instead of complete datasets results in stronger privacy. This is traditionally formalized via mechanism-agnostic subsampling guarantees that express the privacy parameters of a subsampled mechanism as a function of the original mechanism's privacy parameters. We propose the first general framework for deriving mechanismspecific guarantees, which leverage additional information beyond these parameters to more tightly characterize the subsampled mechanism's privacy. Such guarantees are of particular importance for privacy accounting , i.e., tracking privacy over multiple iterations. Overall, our framework based on conditional optimal transport lets us derive existing and novel guarantees for approximate DP, accounting with Rényi DP, and accounting with dominating pairs in a unified, principled manner. As an application, we analyze how subsampling affects the privacy of groups of multiple users. Our tight mechanism-specific bounds outperform tight mechanismagnostic bounds and classic group privacy results.

1 Introduction

Composability and amplification by subsampling are two key properties of Differential Privacy (DP) [1-3] that make it possible to provide provable privacy guarantees for machine learning algorithms that iteratively learn from batches of data.

Composability means that privacy gracefully deteriorates when iteratively applying a differentially private mechanism to a dataset [2]. Kairouz et al. [4], Murtagh and Vadhan [5] ultimately derived tight composition theorems that optimally characterize the privacy parameters ( ε ′ , δ ′ ) of a composed mechanism as a function of the component mechanisms' parameters ( ε, δ ) . However, later work demonstrated that these guarantees are only tight in a mechanism-agnostic sense [6]. Stronger mechanism-specific composition guarantees can often be obtained by using additional information beyond the fact that the mechanisms satisfies some notion of DP. Methods for tracking privacy over multiple iterations (e.g. [6-12]) are refererred to as privacy accountants .

Amplification by subsampling means that privacy can be strenghtened by applying a differentially private mechanism to randomly sampled batches of a dataset [13, 14]. Similar to Kairouz et al. [4], Balle et al. [15] ultimately proposed a framework for deriving tight subsampling theorems that optimally characterize the privacy parameters ( ε ′ , δ ′ ) of a subsampled mechanism as a function of the underlying base mechanism 's parameters ( ε, δ ) .

However, their framework for deriving subsampling theorems has two key limitations. Firstly, as we shall demonstrate, the resultant guarantees are generally only tight in a mechanism-agnostic

sense: Given ( ε, δ ) , one can construct some worst-case ( ε, δ ) -DP mechanism that is ( ε ′ , δ ′ ) -DP under subsampling. The specific mechanism at hand may however be significantly more private. Secondly, their framework does not explain how to derive subsampling guarantees for privacy accountants. In fact, there is so far no unified framework for deriving subsampling guarantees for privacy accountants.

To address these two limitations, we propose a novel framework for unified mechanism-specific amplification by subsampling analysis, using conditional optimal transport. Our proposed framework (1) lets us derive mechanism-specific subsampling guarantees, which can be stronger than mechanismagnostic guarantees, (2) lets us recover mechanism-agnostic guarantees via a pessimistic upper bound - essentially subsuming the approach from [15] - and (3) lets us derive guarantees for approximate differential privacy [2], moments accounting [6] (i.e., Rényi differential privacy [7]), and accounting with dominating pairs [10] (e.g., numerical accounting [16, 8, 17-21]) in a unified, principled manner.

As a practical application, we consider the problem of tightly analyzing group privacy under subsampling (see Fig. 1). Assume a scenario where K individuals may independently decide to contribute to a dataset. Further assume that this dataset is processed by randomly sampling a batch and applying an ( ε, δ ) -DP base mechanism . A special case of this setting is discussed in a recent technical note [22], which assumes that the K individuals collaboratively agree to either contribute or not contribute all their data and that the base mechanism is Gaussian. In the general setting we consider, the best known method for providing privacy guarantees for the entire group is to (1) use existing bounds to show that the subsampled mechanism guarantees ( ε ′ , δ ′ ) -DP for individuals and (2) use the group privacy property[23] to show that this implies ( K · ε ′ , K · e K · ε ′ · δ ′ ) -DP for the group.

� � �

𝑥𝑥1

𝑥𝑥2

𝑥𝑥

𝑁𝑁

𝑥𝑥𝑥1

𝑥𝑥𝑥2

Base mechanism

𝐵𝐵

Subsampling scheme

𝑆𝑆

𝑥𝑥1

𝑥𝑥2

𝑥𝑥6

𝑥𝑥2

𝑥𝑥10

𝑥𝑥6

𝑥𝑥5

𝑥𝑥5

𝑥𝑥9

𝑥𝑥𝑥1

𝑥𝑥7

𝑥𝑥10

𝑥𝑥9

𝑥𝑥𝑥2

𝑥𝑥9

𝑥𝑥𝑥1

𝑥𝑥𝑥2

𝑥𝑥17

𝑥𝑥25

𝑥𝑥𝑥1

𝑥𝑥𝑥2

Pr = (1 - 𝑟𝑟)

2

Pr = 2𝑟𝑟(1 -𝑟𝑟)

Pr = 𝑟𝑟

2

�

�

�

�

�

�

�

�

�

𝑥𝑥𝑥3

Our proposed framework lets us derive stronger, tight group privacy amplification guarantees. By analyzing group privacy and subsampling jointly, these guarantees accurately capture that it is unlikely for a large fraction of the group's data to simultaneously appear in a batch. Our framework further lets us derive dominating pairs [10], which enables tight tracking of group privacy under composition. Overall, our main contributions are that we

demonstrate for the first time that there is a qualitative difference between mechanismspecific and mechanism-agnostic tightness in privacy amplification,

propose a general framework for deriving mechanism-specific amplification by subsampling guarantees, subsuming prior work on mechanism-agnostic amplification,

develop the first unified method for deriving subsampling guarantees for privacy accounting,

and derive the first tight subsampling guarantees for general group privacy.

As part of our group privacy analysis, we also significantly generalize recent results derived for subsampled matrix mechanisms [24], which is of independent interest. Experimental evaluation demonstrates that our tight mechanism-specific guarantees outperform both tight mechanism-agnostic bounds and post-hoc use of the group privacy property.

2 Background and preliminaries

Problem setting. We consider the same setting as Balle et al. [15], but with an additional privacy accounting and group privacy perspective. We assume a space of datasets X , a space of batches Y that can be constructed from these datasets, and an output space Z . For the sake of exposition, we assume that X is the powerset P ( A ) of some finite discrete set A , that Y = { y ⊆ x | x ∈ X } , and that Z = R D . We further consider a random subsampling scheme S : X → Y that generates batches from datasets, and a random base mechanism B : Y → R D that maps batches to outputs. We write s x ( y ) for the pmf of S ( X ) and b y ( z ) for the pdf of B ( y ) . In Appendix D, we introduce a more general setting that admits arbitrary spaces, for which we derive all our results. Our goal is to

provide formal privacy guarantees for the subsampled mechanism M = B ◦ S , which takes a dataset, subsamples a batch from it, and then applies the random base mechanism to this batch.

Dataset and batch relations. Specifically, we want to prove privacy under neighboring relations ≃ X ⊆ X 2 between datasets. For example, the insertion/removal relation x ≃ ± , X x ′ implies that x ′ = x ∪ { a } or x ′ = x \ { a } for some a . The substitution relation x ≃ ∆ , X x ′ implies that x ′ = x \ { a } ∪ { a ′ } for some a, a ′ . In addition, we assume that Y is equipped with a batch neighboring relation ≃ Y ⊆ Y 2 . The batch relation ≃ Y can be distinct from the dataset relation ≃ X .

Subsampling schemes. While we consider arbitrary subsampling schemes S : X → Y , we shall later apply our framework to three particularly common ones: Subsampling without replacement and with replacement sample sets and multisets of fixed size | y | = q uniformly at random. Poisson subsampling includes each element of x with independent probability ('rate') r ∈ [0 , 1] .

Approximate differential privacy. A mechanism M : X → R D is privacy-preserving under symmetric neighboring relation ≃ X when the distributions of M ( x ) , M ( x ′ ) with densities m x , m ′ x are almost indistinguishable for all x ≃ X x ′ . Approximate differential privacy (ADP) [2] quantifies this via hockey stick divergences [25]:

Definition 2.1. For ε ≥ 0 , a mechanism M : X → R D is ( ε, δ ) -DP under symmetric relation ≃ X iff ∀ x ≃ X x ′ : H e ε ( m x || m x ′ ) ≤ δ with H α ( m x || m x ′ ) = ∫ R D max { m x ( z ) /m x ′ ( z ) -α, 0 }· m x ′ ( z ) d z.

Note that this divergence-based definition of approximate differential privacy is equivalent to requiring for all datasets x ≃ X x ′ and all events S ⊆ R D that Pr[ M ( x ) ∈ S ] ≤ e ε · Pr[ M ( x ′ ) ∈ S ] + δ .

Rényi differential privacy. Privacy accounting was first popularized by methods that used moment bounds instead of ( ε, δ ) pairs to better track privacy under composition [6, 26, 27]. These methods were later developed into a moments-based notion of privacy - Rényi differential privacy (RDP) [7]:

Definition 2.2. For α ≥ 1 , a mechanism M : X → R D is ( α, ρ ) -RDP under symmetric relation ≃ X iff ∀ x ≃ X x ′ : log(Λ α ( m x || m x ′ )) / ( α -1) ≤ ρ with Λ α ( m x || m x ′ ) = ∫ R D m x ( z ) α · m x ′ ( z ) 1 -α d z.

Note that Λ α is not the Rényi divergence, but its α th moment, i.e., a scaled and exponentiated Rényi divergence. We use this notation to eliminate exponential terms that arise in amplification for RDP (see [28-30]). If a mechanism is ( α, ρ ) -RDP, its T -fold self-composition is ( α, T · ρ ) -RDP. These RDP parameters can then be converted to ADP parameters ( ε, δ ) , albeit in a lossy manner [7, 31, 32].

Dominating pairs. Later work proposed other numerical [16, 8, 17-21] or analytical accounting techniques [9, 33, 28, 34, 35]. Zhu et al. [10] developed a unified view on these works by introducing the notion of dominating pairs :

Definition 2.3. Apair of distributions ( P, Q ) with densities ( p, q ) is a dominating pair for mechanism M under neighboring relation ≃ X , if H α ( m x || m x ′ ) ≤ H α ( p || q ) for all x ≃ X x ′ and all α ≥ 0 .

Given a dominating pair ( P, Q ) , one can use various numerical or analytic techniques, such as convolution of privacy loss distributions [16] or central limit theorems of tradeoff functions [9], to track the privacy of mechanism M under composition (see Fig. 2 in [10]).

Group privacy. The group privacy property is the graceful decay of privacy when considering multiple user's data. This is normally formalized via the notion of induced distance (see [7, 15, 23]):

Definition 2.4. The distance d X ( x, x ′ ) induced by relation ≃ X is the length of the shortest sequence ( x 1 , . . . , x K -1 ) ∈ X K -1 such that x ≃ X x 1 , ∀ k : x k ≃ X x k +1 , and x K -1 ≃ X x ′ .

Example 2.5 . Let ≃ X be the insertion/removal relation ≃ ± . Then the induced distance d X between x = { 1 , 2 } and x ′ = { 2 , 3 } is 2 , because { 1 , 2 } ≃ ± { 1 , 2 , 3 } ≃ ± { 2 , 3 } .

Proposition 2.6 (Vadhan [23]) . If mechanism M : X → R D is ( ε, δ ) -DP under relation ≃ X , then it is ( K · ε, K · e K · ε · δ ) -DP under group relation { ( x, x ′ ) ∈ X 2 | d X ( x, x ′ ) = K } .

That is, for sufficiently small ε the ADP parameters deteriorate appproximately linearly with induced distance. As baselines for our experiments, we use even tighter bounds (see Appendix C.1.4).

3 Unified mechanism-specific amplification by subsampling

Our goal is to develop a general procedure for (tightly) deriving ADP parameters ( ε ′ , δ ′ ) , RDP parameters ( α ′ , ρ ′ ) , or dominating pairs ( P ′ , Q ′ ) for the subsampled mechanism M = B ◦ S with

subsampling scheme S and base mechanism B . Based on Definitions 2.1 to 2.3, this requires that we evaluate or bound the hockey stick divergence H α or the scaled and exponentiated Rényi divergence Λ α between the distributions of M ( x ) and M ( x ′ ) with x ≃ X x ′ . To discuss both simultaneously, let us write Ψ α for either H α or Λ α and assume that α ≥ 0 or α ≥ 1 , respectively.

There are two challenges: Firstly, the distribution of M ( x ) is high-dimensional mixture distribution with one component per possible batch y ∈ Y and weights given by batch probabilitites s x ( y ) , i.e., m x ( z ) = ∑ y ∈ Y b y ( z ) · s x ( y ) . Secondly, there may be no simple analytic formula for the component densities. For instance, evaluating the density of perturbed model gradients in noisy SGD [36] requires backpropagation. Both challenges make it hard to evaluate or bound Ψ α ( m x || m x ′ ) .

Auseful property that will help us in addressing the first problem of having a large number of mixture components is the joint convexity of Ψ α in the space of probability density functions [37, 38]:

Lemma 3.1. Consider arbitrary densities f (1) 1 , f (1) 2 , f (2) 1 , f (2) 2 : R D → R + and weight w ∈ [0 , 1] . Then, Ψ α ( wf (1) 1 +(1 -w ) f (1) 2 || wf (2) 1 +(1 -w ) f (2) 2 ) ≤ w Ψ α ( f (1) 1 || f (2) 1 ) +(1 -w )Ψ α ( f (1) 2 || f (2) 2 ) .

̸

In our case, the f s can be base mechanism densities b y with different y ∈ Y , and the weights can be given by the subsampling distribution. We could thus upper-bound the mixture divergence Ψ α ( m x || m x ′ ) in terms of component divergences - if the mixtures had identical weights. This is generally not the case, since subsampling mass functions s x ( · ) and s x ′ ( · ) depend on datasets x = x ′ .

To still leverage the joint convexity of Ψ α ( m x || m x ′ ) , we want to rewrite m x and m x ′ as mixtures with identical weights. This is exactly what is offered by couplings between probability mass functions. Intuitively, a coupling between two pmfs p 1 , p 2 : Y → [0 , 1] specifies for every y 1 , y 2 ∈ Y how much probability should be transported from y 1 to y 2 to transform p 1 into p 2 . Couplings can be formalized and generalized to multiple pmfs as follows (for a more thorough introduction, see [39]):

Definition 3.2. A coupling between N pmfs p 1 , . . . , p N : Y → [0 , 1] on batch space Y is a joint pmf γ : Y N → [0 , 1] where the n th marginal is p n , i.e., p n ( y ∗ n ) = ∑ y ∈ Y N [ y n = y ∗ n ] · γ ( y ) .

✶ Given a valid coupling γ between subsampling mass functions s x and s x ′ , we can use marginalization to rewrite m x ( z ) as ∑ y ∈ Y 2 b y 1 ( z ) γ ( y ) . Similarly, we can rewrite m x ′ ( z ) as ∑ y ∈ Y 2 b y 2 ( z ) γ ( y ) . Now, both m x and m x ′ are mixtures with identical weights and one component per pair of batches in Y 2 . We can thus recursively apply Lemma 3.1 to show (full proof in Appendix E):

Theorem 3.3. Consider a subsampled mechanism M = B ◦ S , and an arbitrary coupling γ between subsampling mass functions s x ( · ) and s x ′ ( · ) . Then,

with cost function c α : Y × Y → R + defined by c α ( y (1) , y (2) ) = Ψ α ( b y (1) || b y (2) ) .

We write y (1) instead of y 1 to simplify later notations. While every coupling γ yields a valid upper bound, the guarantees can be tightened by finding a coupling γ ∗ that minimizes the r.h.s. of Eq. (1). We thus have an optimal transport problem , where the cost c α of transporting probability from batch y (1) to batch y (2) depends on the divergence Ψ α of base mechanism densities b y (1) and b y (2) .

Unfortunately, experimental evaluation in Appendix B.1.4 shows that the resultant guarantees can be much weaker than those from prior work - even with an optimal coupling γ ∗ . This is because Theorem 3.3 results from recursively applying the joint convexity property (Lemma 3.1) to Ψ α ( m x || m x ′ ) . Each recursive step splits each mixture into two smaller mixtures and further upper-bounds the divergence that is achieved by our specific subsampled mechanism M on our specific pair of datasets x, x ′ . Upon fully decomposing the overall divergence into divergences between single-mixture components, this sequence of bounds is in fact larger than even the divergence Ψ α ( ˜ m ˜ x || ˜ m ˜ x ′ ) achieved by a worst-case subsampled mechanism ˜ M on a worst-case pair of datasets ˜ x, ˜ x ′ . To overcome this limitation, we propose to limit the recursion depth in order to obtain a tighter upper bound that matches our specific subsampled mechanism M on a worst-case pair of datasets ˆ x, ˆ x ′ .

Limiting the recursion depth to which Lemma 3.1 is applied means upper-bounding Ψ α ( m x || m x ′ ) in terms of mixture divergences that have not been fully decomposed into their individual components. Specifically, we propose to do so by defining an optimal transport problem between multiple subsampling mass functions conditioned on different events (proof in Appendix E):

Theorem 3.4. Consider a subsampled mechanism M = B ◦ S . Further consider two disjoint partitionings ⋃ I i =1 A i = Y and ⋃ J j =1 E j = Y of batch space Y such that all A i and E j have non-zero probability under the distribution of S x and S x ′ , respectively. Let γ be an arbitrary coupling between conditional mass functions s x ( · | A 1 ) , . . . , s x ( · | A I ) , s x ′ ( · | E 1 ) , . . . , s x ′ ( · | E J ) . Then,

with cost function c α : Y I × Y J → R + defined by

In other words: We now have an optimal transport problem between I + J probability mass functions coupled by γ . The transport cost c α is a divergence between two mixtures. The components of the first mixture are base mechanisms densities given batches y (1) i ∈ Y from batch tuple y (1) ∈ Y I . The weights of the first mixture are probabilities of events A i . The second mixture is defined analogously.

Note that we can recover Theorem 3.3 by conditioning on a single event, i.e., A 1 = Y , E 1 = Y . As we shall demonstrate, a more fine-grained partitioning lets us obtain tighter bounds that match the divergence Ψ α ( m ˜ x || m ˜ x ′ ) of our specific subsampled mechanism M on a worst-case pair of datasets ˆ x, ˆ x ′ . In the extreme case of defining event per possible batch, Theorem 3.4 holds with equality.

Overall, Theorem 3.4 reduces the broad problem of bounding mixture divergences to the canonical problem of optimal transport between conditional distributions. But before we can apply it, we need to address two open problems: Evaluating the cost function and designing optimal couplings.

Cost function bound. Not having an analytic expression for every base mechanism density b y ( z ) may make it intractable to evaluate cost function c α . We thus propose to bound it via an approach that is inherent to differential privacy: Considering worst-case inputs (proof in Appendix E).

Proposition 3.5. Consider y (1) ∈ Y I , y (2) ∈ Y J , and cost function c defined in Eq. (2) . Let d Y be the distance induced by ≃ Y (see Definition 2.4). Then, c α ( y (1) , y (2) ) ≤ ˆ c α ( y (1) , y (2) ) , with

subject to ∀ k, l ∈ { 1 , 2 } , ∀ t, u : d Y (ˆ y ( k ) t , ˆ y ( l ) u ) ≤ d Y ( y ( k ) t , y ( l ) u ) and ˆ y (1) ∈ Y I , ˆ y (2) ∈ Y J .

Put differently: We construct two new mixtures with components that are adversarially chosen to maximize divergence while retaining the pairwise distances between batches in y (1) and y (2) . Note that, for the special case of Ψ α = H α and I = J = 1 , this bound corresponds to the 'group privacy profile' in [15]. As we shall demonstrate in the next sections, this bound ˆ c can often be evaluated using high-level information about the base mechanism B : Y → R D , such as global sensitivity.

Sufficient optimality condition. While every coupling γ yields a valid upper bound in Theorem 3.4, this bound can be tightened by designing an optimal coupling γ ∗ . To inform this design, we generalize the notion of distance-compatible couplings from [15] to an arbitrary number of distributions:

Definition 3.6. A coupling γ between mass functions p 1 , . . . , p N on batch space Y is d Y -compatible when γ ( y ) > 0 only if ∀ u > 1 : d Y ( y 1 , y u ) = d Y ( { y 1 } , supp( p u )) and ∀ u > t > 1 : d Y ( y u , y t ) = d Y (supp( p t ) , supp( p u )) , where supp( p u ) is the support of s u and the distance between two sets is the minimum distance of their elements.

Essentially, a d Y -compatible coupling only assigns probability to a tuple of batches y when all pairs y i , y j have the smallest possible distance to y 1 and each other while still being in the support of their distributions. In Appendix F, we prove that d Y -compatibility is sufficient for optimality. We further show that the optimal value has a canonical form whenever a d Y -compatible couplings exists.

Summary. To summarize, we propose the following three-step procedure for deriving subsampling guarantees: (1) Define two partitions of the batch space into I and J events. (2) Define a simultaneous coupling between the corresponding I + J conditional distributions to obtain a bound in terms of divergences between small mixtures with I and J components. (3) Bound these mixture divergences by considering I + J worst-case mixture components under pairwise batch distance constraints.

3.1 Tight mechanism-specific group privacy amplification

As a concrete example, and to illustrate the difference between mechanism-specific and -agnostic (see Section 3.2) bounds, let us consider the group privacy setting from Fig. 1. We have a group of K users that can independently decide to contribute or not contribute their data, and the resulting dataset is Poisson subsampled. To provide formal privacy guarantees to the group, we need to prove that the distributions of M ( x ) , M ( x ′ ) are almost indistinguishable for x, x ′ with induced distance d ± , X ( x, x ′ ) ≤ K . Let x ≃ K + ,K -, X x ′ be the relation that implies that K + records are inserted and K -records are removed to construct x ′ from x . We can then show (full proof in Appendix M):

Theorem 3.7. Let M = B ◦ S , where S is Poisson subsampling with rate r . Let ≃ Y be the insertion/removal batch relation ≃ ± , Y . Then, for all x ≃ K + ,K -, X x ′ , Ψ α ( m x || m x ′ ) is l.e.q.

subject to constraints y ∈ Y K -+ K + +2 , as well as ∀ l ∈ { 1 , 2 } , ∀ t, u : d Y ( y ( l ) t , y ( l ) u ) ≤ | t -u | , and ∀ t, u : d Y ( y (1) t , y (2) u ) ≤ ( t -1) + ( u -1) .

Proof sketch. We let A i and E j be the events that S ( x ) contains i -1 deleted elements and S ( x ′ ) contains j -1 inserted elements, respectively. Our coupling defines the following generative process: We sample y (1) 1 from s x ( · | A 1 ) and let y (2) 1 ← y (1) 1 . We then iteratively generate the other y (1) i by sampling a permutation in which we add the K -deleted elements to y (1) 1 . We then generate the other y (2) j by sampling a permutation in which we add the K + inserted elements to y (2) 1 . The result then follows from the cost function bound in Proposition 3.5 and d Y -compatibility of the coupling. Batches y (1) u and y (1) t and have a distance bounded by | u -t | , because one can be obtained from the other by removing/inserting | u -t | elements. The constraints for y (2) u and y (2) t are analogous. Batches y (1) u and y (2) t have a distance bounded by ( t -i ) + ( u -1) because we need to remove t -1 elements and insert u -1 elements to construct one from the other.

Next, we can solve the constrained optimization problem in Theorem 3.7 - i.e., determine worst-case components - to obtain mechanism-specific guarantees. For instance (proof in Appendix M.2.1):

Theorem 3.8. Let M = B ◦ S , where S is Poisson subsampling with rate r , and B is the Gaussian mechanism h + V with h : Y → R D and V ∼ N ( 0 , σ 2 I D ) . Define the ℓ 2 -sensitivity L 2 = max y ≃ ± , Y y ′ || f ( y ) -f ( y ′ ) || 2 . Then for all x ≃ K + ,K -, X x ′ , Ψ α ( m x || m x ′ ) is l.e.q.

with univariate normal densities f (1) i = N ( · | ( i -1) , σ / L 2 ) , f (2) j = N ( · | -( j -1) , σ / L 2 ) .

Note that this bound is mechanism-specific in that it explictly depends on B being a Gaussian mechanism with standard deviation σ and sensitivity L 2 . The bound can be numerically evaluated to arbitrary precision using standard techniques from privacy accounting literature (see Appendix M.4). In Appendix M.2 we derive similar guarantees for Laplace and randomized response mechanisms.

Tightness. These bounds are tight in a mechanism-specific sense: One cannot derive stronger ADP or RDP guarantees without additional information about the datasets x, x ′ ∈ X or the underlying function h (proofs in Appendix M.3).

Asymptotic bounds. Our focus is on tight bounds that can be explicitly computed. However, some early works on RDP accounting (e.g.[6, 28]) also provided asymptotic versions of their bounds, e.g., as a function of divergence parameter α . For completeness, we use Theorem 3.8 to generalize the asymptotic bounds of Abadi et al. [6] to the group privacy setting in Appendix N.5.

Other contributions. Our solutions to the optimization problem in Theorem 3.7 are of independent interest: The special case of Ψ α = H α , K -= 0 or K + = 0 , and Gaussian mechanisms was

𝑆𝑆

(

𝑋𝑋

)|

𝐴𝐴1

𝑆𝑆

(

𝑋𝑋

)|

𝐴𝐴2

𝑆𝑆

(

𝑋𝑋𝑋

)|

𝐸𝐸1

𝑆𝑆

(

𝑋𝑋𝑋

)|

𝐸𝐸2

𝑆𝑆

(

𝑋𝑋

)|

𝐴𝐴1

,

,

𝑆𝑆

(

𝑋𝑋

)|

𝐴𝐴

𝐼𝐼

,

𝑆𝑆

(

𝑋𝑋𝑋

)|

𝐸𝐸

𝐽𝐽

OT

OT

OT

� � �

� � �

𝑆𝑆

(

𝑋𝑋𝑋

)

𝑆𝑆

(

𝑋𝑋𝑋

)|

𝐸𝐸1

,

𝑆𝑆

(

𝑋𝑋

)

𝑆𝑆

(

𝑋𝑋𝑋

)

𝑆𝑆

(

𝑋𝑋

)

𝑆𝑆

(

𝑋𝑋𝑋

)

OT

𝑆𝑆

(

𝑋𝑋𝑋

)

𝑆𝑆

(

𝑋𝑋

)

JOINTCONVEXITY

𝑆𝑆

(

𝑋𝑋

)|

𝐴𝐴1

𝑆𝑆

(

𝑋𝑋

)|

𝐴𝐴2

𝑆𝑆

(

𝑋𝑋

)

JOINTCONVEXITY

(a)

(b)

(c)

(d)

derived to analyze matrix mechanisms in [24]. We significantly generalize it via an entirely different proof strategy that admits Ψ α ∈ { H α , Λ α } , arbitrary K + , K -∈ N 0 , and non-Gaussian mechanisms (see Appendix O).

3.2 Tight mechanism-agnostic group privacy amplification

To further illustrate the difference between mechanism-specific and -agnostic bounds, let us apply the framework of Balle et al. [15] to the same setting: For K -= 0 or K + = 0 and ε ≥ 0 , their ansatz shows that the subsampled mechanism M = B ◦ S is ( ε ′ , δ ′ ) -DP with ε ′ = log(1 + (1 -Binom(0 | K,r )) · ( e ε -1)) and δ ′ = ∑ K k =1 Binom( k | K,r ) · δ k , with group privacy parameters δ k = max y,y ′ H exp( ε ) ( b y || b y ′ ) s.t. d Y ( y, y ′ ) ≤ k (see Appendix N).

Tightness. This bound is tight in a mechanism-agnostic sense, i.e., for every ε ≥ 0 , one can construct some worst-case mechanism that exactly attains the bound (see Appendix N.2).

Mechanism-specific vs mechanism-agnostic tightness. An apparent advantage of this result is that it expresses ( ε ′ , δ ′ ) as a simple analytic formula of base mechanism DP parameters ε, δ 1 , . . . , δ K . However, this simplicity comes at a cost: As we show in Appendix N.3, this guarantee implicitly upper-bounds the tight mixture divergence bound from Theorem 3.7 in terms of its component divergences using joint convexity. 2 As we experimentally demonstrate in Section 4, this relaxation leads to weaker privacy guarantees for group size K ≥ 2 . This gap has gone unnoticed in earlier work, because Theorem 3.8 happens to be identical to the bound of Balle et al. [15] for the special case of K = 1 (see proof of Proposition 30 in [10]). By studying the more complex group privacy setting, we demonstrate for the first time that there is a qualitative difference between mechanism-agnostic and mechanism-specific tightness in privacy amplification .

3.3 From mechanism-specific to mechanism-agnostic guarantees

The observed relation between tight mechanism-specific and mechanism-agnostic bounds does in fact extend beyond group privacy (see Fig. 2): As we demonstrate in Appendices G and H, mechanismagnostic subsampling guarantees from prior work can equivalently be derived by (1) conditioning on at most 4 events indicating the presence of inserted / deleted / substituted elements, (2) defining a simultaneous coupling, and (3) using joint convexity to upper-bound the resultant mechanism-specific guarantee by component divergences. This includes the ADP guarantees of Balle et al. [15] (and thus prior work [14, 43, 44]) and the RDP guarantees from [28-30, 40].

Subsumption of [15]. Going even further, we show in Appendix G.2 that our approach subsumes the entire framework of Balle et al. [15]: Any mechanism-agnostic guarantee derived via their ansatz can equivalently be derived by defining a (potentially suboptimal) simultaneous coupling between four subsampling distributions and upper-bounding the resultant guarantee via (advanced) joint convexity.

Contribution. Importantly, our contribution is not in the bounds themselves, but in the identification of this implicitly underlying pattern that unifies prior work. We further generalize this pattern through our proposed framework to enable tight analysis for more challenging scenarios like group privacy.

Novel RDP accounting bounds. In Appendix I, we use this observation to derive a variety of novel mechanism-agnostic RDP bounds for different combinations of ≃ X , ≃ Y , and subsampling schemes, such as insertion/removal and subsampling without replacement. We further derive a simple but tight mechanism-specific guarantee for subsampled randomized response under substitution (see Theorem I.3) that outperforms the best known mechanism-agnostic bound (see Section 4).

3.4 From mechanism-specific guarantees to dominating pairs.

Because the mechanism-specific guarantees derived via our framework do not obfuscate the underlying distributions, they can be used to identify dominating pairs. These dominating pairs can then be combined with arbitrary (numerical) accountants to track privacy under composition. For example, we can immediately read off from Theorem 3.8 that the two Gaussian mixtures are a dominating pair for the 'insertK + -removeK -' relation. In Appendix J, we describe a procedure for constructing dominating pairs when the bound is a weighted sum of multiple mixture divergences. We can thus determine dominating pairs for any bound derived via optimal transport. As we demonstrate in Appendix K, the dominating pairs for subsampled mechanisms derived by Zhu et al. [10] (special cases of which appear in [8, 17-19, 11]) can equivalently be proven via our framework.

Subsampling with replacement. As a novel contribution, we derive for the first time dominating pairs for subsampling with replacement (see Theorem L.5), which were posited but not proven in [8] (see discussion in Appendix L). This is enabled by our solution to the problem in Theorem 3.7. As we experimentally show in Appendix B.1.5, these bounds can be much stronger than those derived via the framework of Balle et al. [15]. These results thus demonstrate that there is a qualitative difference between mechanism-specific and mechanism-agnostic tightness even for group size K = 1 .

3.5 Limitations and future work

Limitations. While our proposed framework can be applied to arbitrary subsampling distributions, conditioning on a finite set of events may be too restrictive in certain settings (e.g., continuous batch spaces). Also, while we found conditioning on the number of modified elements to be sufficient for all considered scenarios, an automated procedure for selecting events would be desirable. Maximal couplings fulfill this purpose in [15], but only yield pairs of distributions.

Future work. A natural direction is applying our ansatz to novel settings other than group privacy. In particular, it could potentially be used to provide epoch-level guarantees for correlated subsampling (e.g., batching via shuffling), similar to [38]. We present a preliminary result for 2 -fold non-adaptive composition in Appendix P. Future work could also use our solutions to the problem in Theorem 3.7 to generalize the matrix mechanism analysis from [24] to substitutions and non-Gaussian distributions.

4 Experimental evaluation

The purpose of the following experiments is to verify that there can be a benefit to using mechanismspecific over mechanism-agnostic subsampling bounds, and that a joint analysis of subsampling and group privacy can offer stronger guarantees than post-hoc application of the generic group privacy property. In all figures, 'specific' refers to our proposed mechanism-specific bounds, 'agnostic' refers to (tight) mechanism-agnostic bounds, and 'post-hoc' refers to applying the generic group privacy property to tight mechanism-specific bounds derived for group size 1 . For all experiments, we assume ℓ p sensitivities of 1 . Further details on the experimental setup are provided in Appendix C. An implementation will be made available at https://cs.cit.tum.de/daml/group-amplification.

10

1

10

2

10

3

10

4

RDP

α

10

-

6

10

-

4

10

-

2

RDP

ρ

(

α

)

θ

0.9

0.75

0.6

Agnostic

Specific

0

1

2

3

4

5

6

ADP

ε

0

.

0

0

.

2

0

.

4

0

.

6

ADP

δ

(

ε

)

Group size

16

8

4

2

1

Agnostic

Specific

4.1 Mechanism-agnostic and mechanism-specific guarantees

Randomized response and RDP. One potential source of looseness in mechanism-agnostic bounds is that they bound mixture divergences in terms of component divergences that can be summarized by a single privacy parameter. As an example, let us consider the best known guarantee for substitution, subsampling without replacement, and RDP from [28]. As discussed by the authors, it is only tight up to factors that are constant in α . In Fig. 3 we compare it to our tight mechanism-specific bound for randomized response (see Theorem I.3) with batch-to-dataset ratio q / N = 0 . 001 and true response probability θ ∈ { 0 . 6 , 0 . 75 , 0 . 9 } . In Appendix B.1.1 we consider other ratios. The tight guarantee eliminates the constant factors and thus achieves much smaller ρ for a wide range of α ∈ (1 , 10 4 ] .

Group privacy and ADP. Another potential source of looseness in mechanism-agnostic bounds is that they stem from a binary partitioning of the batch space (recall Fig. 2), which may not be sufficient when there are multiple possible levels of privacy leakage (e.g. number of sampled group elements). We demonstrate this in Fig. 4, by comparing the tight mechanism-agnostic group privacy bound derived via the framework from [15] to our tight mechanism-specific bound for Laplace mechanisms (see Theorem M.2), ADP, scale λ = 1 , subsampling rate r = 0 . 2 , and varying group size. In Appendix B.1.2 we repeat the experiment with other mechanisms and parameter values, and also consider RDP. As discussed in Section 3.2, the mechanism-agnostic bound is identical to the mechanisms-specific bound for group size K = 1 . For group sizes K ≥ 2 , however, the fine-grained partitioning underlying the mechanism-specific bound yields stronger privacy guarantees. These results confirm that we need to distinguish between mechanism-agnostic and mechanism-specific tightness when analyzing complex subsampling settings.

4.2 Post-hoc and mechanism-specific group privacy analysis

Single-iteration group privacy. Our next goal is to demonstrate the benefit of analyzing group privacy and subsampling jointly. In Fig. 5, we evaluate our tight guarantee for Gaussian mechanisms (see Theorem 3.8) with σ = 2 , rate r = 0 . 2 , and varying group size. As a baseline, we evaluate the same tight guarantee with K + + K -= 1 and apply the generic group privacy property (see Appendix C.1.4) in a post-hoc manner. As can be seen, our tight analysis can lead to much stronger privacy guarantees. However, we interestingly observe in Appendix B.2.1 that the generic group privacy properties of ADP and RDP can serve as increasingly tight upper bounds when considering more private base mechanisms (e.g., σ = 5 ) and much smaller subsampling rates (e.g., r = 10 -3 ).

Composed group privacy. Nevertheless, even the gaps in tightness for very private mechanisms may quickly accumulate when repeatedly applying these mechanisms to a dataset. In Fig. 6, we use the dominating pairs derived via our framework to conduct tight PLD accounting [16, 8] with 'connect the dots' [20] quantization using Google's dp_accounting library [45]. For σ = 5 , r = 10 -3 , and fixed ε = 2 (for other pararameters and mechanisms, see Appendix B.2.3), the post-hoc analysis quickly diverges with increasing group size and number of iterations. For example, with group size 16 and privacy budget δ = 10 -6 , the post-hoc analysis allows less than 100 iterations of DP-SGD training [36, 6], whereas our tight analysis enables training for over 1000 iterations. In Appendix B.2.5 we train an image classification model on MNIST with PLD accounting to demonstrate that this increased number of training iterations can translate to much higher model utility.

0

1

2

3

4

5

6

ADP

ε

0

.

0

0

.

2

0

.

4

0

.

6

ADP

δ

(

ε

)

Group size

16

8

4

2

1

Post-hoc

Specific

0

250

500

750

1000

Iteration

t

10

-

22

10

-

16

10

-

10

10

-

4

ADP

δ

(

ε

)

Group size

16

8

4

2

1

Post-hoc

Specific

5 Related Work

Amplification by subsampling for ADP. Using subsampling to strenghten ( ε, δ ) -DP guarantees has a long history [13, 14] particularly in privacy-preserving machine learning [46, 47, 6]. For a thorough introduction to subsampling and its interaction with composition, we refer the reader to [48]. Balle et al. [15] ultimately proposed a framework for deriving tight mechanism-agnostic subsampling guarantees. Our work subsumes theirs and enables the derivation of mechanism-specific guarantees.

Amplification for privacy accountants. Abadi et al. [6]'s seminal work on moments accounting already considered subsampling. Similiar results were derived for the more general notion of RDP [7] in [28-30]. More recent works on accounting generally include a discussion of either Poisson subsampling or subsampling without replacement [17-19, 21, 11]. Subsampling with replacement is discussed in [8], albeit without complete proofs, which we provide in Appendix L. We demonstrate that amplification guarantees for the central notions of RDP [7] and dominating pairs [10] can - just like guarantees for ADP - be derived via optimal transport. Our novel approach not only unifies prior, but also enables a principled analysis of challenging scenarios like group privacy.

Group privacy amplification. A special case of group privacy amplification (which we use as an example to demonstrate the utility of our general framework) with hockey stick divergences, Gaussian mechanisms, and groups that collaboratively agree to contribute all their data was posited in [24] and proven in a recent follow-up note [22]. However, their result (and proof strategy) does not cover Rényi divergences, other mechanisms (Laplace, randomized response), and the standard notion of group privacy under which group members are not forced to collaborate (see, e.g., [3, 7, 9, 15, 23]).

Hierarchical randomized smoothing. Randomized smoothing [49-51] is a technique for constructing provably robust models via DP achieved through input perturbations (e.g. [52-68]). Similar to subsampling, Scholten et al. [69] proposed to only apply perturbations to randomly sampled parts of an input. We repurpose their technique of treating subsampling indicators as observable random variables in order to construct dominating pairs from weighted sums of divergences (see Appendix J.1).

Unified amplification for f -DP. Wang et al. [70] proved a form of joint concavity for the tradeoff functions underlying f -DP accounting [9, 34]. This enables them to analyze mixtures induced by random initialization and shuffling in a unified manner. However, they do not consider amplification by subsampling, and explicitly state that they need to address this problem in future work. Note that dominating pairs, which can be derived via our proposed framework, can be used in f-DP accounting due to duality of privacy profiles and tradeoff functions [9, 10].

6 Conclusion

The main purpose of this work is to provide a unified, principled framework for mechanism-specific subsampling analysis and subsampling analysis for privacy accountants. To this end, we proposed a three-step procedure based on optimal transport between conditional subsampling distributions. Beyond recovering known guarantees for Rényi DP and dominating pairs, this procedure lets us derive novel results that were previously only available for approximate DP, such as non-standard combinations of subsampling schemes and neighboring relations, or subsampling with replacement. We then applied this procedure to the problem of analyzing group privacy under subsampling. Our experimental evaluation demonstrates that our mechanism-specific group privacy amplification bounds are not only tight, but can also significantly outperform tight mechanism-agnostic bounds and traditional group privacy results - even under composition. On a higher level, these bounds represent a novel contribution to a larger body of work (e.g., [38, 71, 72]) that demonstrates the benefit of analyzing multiple properties of differential privacy jointly instead of independently.

References

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography , pages 265-284. Springer, 2006.

Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Advances in Cryptology EUROCRYPT , pages 486-503, 2006.

Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science , 9(3-4):211-407, 2014.

Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The composition theorem for differential privacy. In International conference on machine learning , pages 1376-1385, 2015.

Jack Murtagh and Salil Vadhan. The complexity of computing the optimal composition of differential privacy. In Theory of Cryptography Conference , pages 157-175, 2015.

Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the SIGSAC conference on computer and communications security , pages 308-318, 2016.

Ilya Mironov. Rényi differential privacy. In IEEE 30th computer security foundations symposium (CSF) , pages 263-275, 2017.

Antti Koskela, Joonas Jälkö, and Antti Honkela. Computing tight differential privacy guarantees using fft. In International Conference on Artificial Intelligence and Statistics , pages 2560-2569, 2020.

Jinshuo Dong, Aaron Roth, and Weijie J Su. Gaussian differential privacy. Journal of the Royal Statistical Society Series B: Statistical Methodology , 84(1):3-37, 2022.

Yuqing Zhu, Jinshuo Dong, and Yu-Xiang Wang. Optimal accounting of differential privacy via characteristic function. In International Conference on Artificial Intelligence and Statistics , pages 4782-4817, 2022.

Wael Alghamdi, Juan Felipe Gómez, Shahab Asoodeh, Flávio P. Calmon, Oliver Kosut, and Lalitha Sankar. The saddle-point method in differential privacy. In International Conference on Machine Learning , 2023.

Jiachen Tianhao Wang, Saeed Mahloujifar, Tong Wu, Ruoxi Jia, and Prateek Mittal. A randomized approach to tight privacy accounting. Advances in Neural Information Processing Systems , 36, 2023.

Shiva Prasad Kasiviswanathan, Homin K Lee, Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. What can we learn privately? SIAM Journal on Computing , 40(3):793-826, 2011.

Ninghui Li, Wahbeh Qardaji, and Dong Su. On sampling, anonymization, and differential privacy or, k-anonymization meets differential privacy. In Proceedings of the 7th ACM Symposium on Information, Computer and Communications Security , pages 32-33, 2012.

Borja Balle, Gilles Barthe, and Marco Gaboardi. Privacy amplification by subsampling: Tight analyses via couplings and divergences. Advances in neural information processing systems , 31, 2018.

Sebastian Meiser and Esfandiar Mohammadi. Tight on budget? tight bounds for r-fold approximate differential privacy. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security , pages 247-264, 2018.

Antti Koskela and Antti Honkela. Computing differential privacy guarantees for heterogeneous compositions using fft. In International Conference on Learning Representations , 2021.

Antti Koskela, Joonas Jälkö, Lukas Prediger, and Antti Honkela. Tight differential privacy for discrete-valued mechanisms and for the subsampled gaussian mechanism using fft. In International Conference on Artificial Intelligence and Statistics , pages 3358-3366, 2021.

Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. Numerical composition of differential privacy. Advances in Neural Information Processing Systems , 34:11631-11642, 2021.

Vadym Doroshenko, Badih Ghazi, Pritish Kamath, Ravi Kumar, and Pasin Manurangsi. Connect the dots: Tighter discrete approximations of privacy loss distributions. Proceedings on Privacy Enhancing Technologies , 2022.

Badih Ghazi, Pritish Kamath, Ravi Kumar, and Pasin Manurangsi. Faster privacy accounting via evolving discretization. In International Conference on Machine Learning , pages 7470-7483, 2022.

Arun Ganesh. Tight group-level dp guarantees for dp-sgd with sampling via mixture of gaussians mechanisms. arXiv preprint arXiv:2401.10294 , 2024.

Salil Vadhan. The complexity of differential privacy. Tutorials on the Foundations of Cryptography , pages 347-450, 2017.

Christopher A. Choquette-Choo, Arun Ganesh, Thomas Steinke, and Abhradeep Guha Thakurta. Privacy amplification for matrix mechanisms. In International Conference on Learning Representations , 2024.

Gilles Barthe and Federico Olmedo. Beyond differential privacy: Composition theorems and relational logic for f-divergences between probabilistic programs. In Fedor V. Fomin, R¯ usin , š Freivalds, Marta Kwiatkowska, and David Peleg, editors, Automata, Languages, and Programming , pages 49-60, 2013.

Cynthia Dwork and Guy N Rothblum. Concentrated differential privacy. arXiv preprint arXiv:1603.01887 , 2016.

Mark Bun and Thomas Steinke. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Theory of Cryptography Conference , pages 635-658, 2016.

Yu-Xiang Wang, Borja Balle, and Shiva Prasad Kasiviswanathan. Subsampled rényi differential privacy and analytical moments accountant. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1226-1235, 2019.

Ilya Mironov, Kunal Talwar, and Li Zhang. Rényi differential privacy of the sampled gaussian mechanism. arXiv preprint arXiv:1908.10530 , 2019.

Yuqing Zhu and Yu-Xiang Wang. Poisson subsampled rényi differential privacy. In International Conference on Machine Learning , pages 7634-7642, 2019.

Borja Balle, Gilles Barthe, Marco Gaboardi, Justin Hsu, and Tetsuya Sato. Hypothesis testing interpretations and renyi differential privacy. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108, pages 2496-2506, 2020.

Shahab Asoodeh, Jiachun Liao, Flavio P Calmon, Oliver Kosut, and Lalitha Sankar. A better bound gives a hundred rounds: Enhanced privacy guarantees via f-divergences. In IEEE International Symposium on Information Theory (ISIT) , 2020.

David M. Sommer, Sebastian Meiser, and Esfandiar Mohammadi. Privacy loss classes: The central limit theorem in differential privacy. Proceedings on Privacy Enhancing Technologies , 2019(2):245-269, 2019.

Zhiqi Bu, Jinshuo Dong, Qi Long, and Weijie J Su. Deep learning with gaussian differential privacy. Harvard data science review , 2020(23):10-1162, 2020.

Hua Wang, Sheng Gao, Huanyu Zhang, Milan Shen, and Weijie J Su. Analytical composition of differential privacy via the edgeworth accountant. arXiv preprint arXiv:2206.04236 , 2022.

Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with differentially private updates. In IEEE global conference on signal and information processing , pages 245-248, 2013.

Borja Balle, Gilles Barthe, and Marco Gaboardi. Privacy profiles and amplification by subsampling. Journal of Privacy and Confidentiality , 10(1), 2020.

Jiayuan Ye and Reza Shokri. Differentially private learning needs hidden state (or much faster convergence). Advances in Neural Information Processing Systems , 35:703-715, 2022.

Cédric Villani. Couplings and changes of variables , pages 5-20. 2009. ISBN 978-3-540-710509.

Ameya Daigavane, Gagan Madan, Aditya Sinha, Abhradeep Guha Thakurta, Gaurav Aggarwal, and Prateek Jain. Node-level differentially private graph neural networks. In ICLR 2022 Workshop on PAIR^2Struct: Privacy, Accountability, Interpretability, Robustness, Reasoning on Structured Data , 2022.

Morgane Ayle, Jan Schuchardt, Lukas Gosch, Daniel Zügner, and Stephan Günnemann. Training differentially private graph neural networks with random walk sampling. In Workshop on Trustworthy and Socially Responsible Machine Learning, NeurIPS , 2022.

Zihang Xiang, Tianhao Wang, and Di Wang. Preserving Node-level Privacy in Graph Neural Networks. In IEEE Symposium on Security and Privacy (SP) , pages 4714-4732, 2024.

Mark Bun, Kobbi Nissim, Uri Stemmer, and Salil Vadhan. Differentially private release and learning of threshold functions. In 2015 IEEE 56th Annual Symposium on Foundations of Computer Science , pages 634-649, 2015.

Jonathan Ullman. Cs7880: Rigorous approaches to data privacy. https://www.khoury. northeastern.edu/home/jullman/cs7880s17/HW1sol.pdf , 2017. Accessed May 21, 2024.

Google Differential Privacy Team. Privacy loss distributions. https://raw. githubusercontent.com/google/differential-privacy/main/common_docs/ Privacy_Loss_Distributions.pdf , 2024. Accessed May 22, 2024.

Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In IEEE 55th annual symposium on foundations of computer science , pages 464-473, 2014.

Yu-Xiang Wang, Stephen Fienberg, and Alex Smola. Privacy for free: Posterior sampling and stochastic gradient monte carlo. In International Conference on Machine Learning , pages 2493-2502, 2015.

Thomas Steinke. Composition of differential privacy & privacy amplification by subsampling. arXiv preprint arXiv:2210.00597 , 2022.

Mathias Lécuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In IEEE Symposium on Security and Privacy , pages 656-672, 2019.

Bai Li, Changyou Chen, Wenlin Wang, and Lawrence Carin. Certified adversarial robustness with additive noise. Advances in neural information processing systems , 2019.

Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning , 2019.

Alexander Levine and Soheil Feizi. Robustness certificates for sparse adversarial attacks by randomized ablation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 4585-4593, 2020.

Aleksandar Bojchevski, Johannes Klicpera, and Stephan Günnemann. Efficient robustness certificates for discrete data: Sparsity-aware randomized smoothing for graphs, images and more. In International Conference on Machine Learning , 2020.

Marc Fischer, Maximilian Baader, and Martin Vechev. Certified defense to image transformations via randomized smoothing. Advances in Neural information processing systems , 2020.

Yihan Wu, Aleksandar Bojchevski, Aleksei Kuvshinov, and Stephan Günnemann. Completing the picture: Randomized smoothing suffers from curse of dimensionality for a large family of distributions. In International Conference on Artificial Intelligence and Statistics , 2021.

Aounon Kumar and Tom Goldstein. Center smoothing: Certified robustness for networks with structured outputs. Advances in Neural Information Processing Systems , 2021.

Jan Schuchardt and Stephan Günnemann. Invariance-aware randomized smoothing certificates. In Advances in Neural Information Processing Systems , 2022.

Motasem Alfarra, Adel Bibi, Naeemullah Khan, Philip HS Torr, and Bernard Ghanem. DeformRS: Certifying input deformations with randomized smoothing. In Proceedings of the AAAI Conference on Artificial Intelligence , number 6, pages 6001-6009, 2022.

Yan Scholten, Jan Schuchardt, Simon Geisler, Aleksandar Bojchevski, and Stephan Günnemann. Randomized message-interception smoothing: Gray-box certificates for graph neural networks. In Advances in Neural Information Processing Systems , 2022.

Nikita Murarev and Aleksandr Petiushko. Certified robustness via randomized smoothing over multiplicative parameters of input transformations. In International Joint Conference on Artificial Intelligence , 2022.

Peter Súkeník, Aleksei Kuvshinov, and Stephan Günnemann. Intriguing properties of inputdependent randomized smoothing. In International conference on machine learning , 2022.

Mikhail Pautov, Olesya Kuznetsova, Nurislam Tursynbek, Aleksandr Petiushko, and Ivan Oseledets. Smoothed embeddings for certified few-shot learning. Advances in Neural Information Processing Systems , 2022.

Jan Schuchardt, Tom Wollschläger, Aleksandar Bojchevski, and Stephan Günnemann. Localized randomized smoothing for collective robustness certification. In International Conference on Learning Representations , 2023.

Taras Rumezhak, Francisco Girbal Eiras, Philip HS Torr, and Adel Bibi. Rancer: Non-axis aligned anisotropic certification with randomized smoothing. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 4672-4680, 2023.

Aman Saxena, Tom Wollschläger, Nicola Franco, Jeanette Miriam Lorenz, and Stephan Günnemann. Randomized smoothing-inspired quantum encoding schemes with formal robustness guarantees. In Quantum Techniques in Machine Learning , 2023.

Samuel Pfrommer, Brendon G. Anderson, and Somayeh Sojoudi. Projected randomized smoothing for certified adversarial robustness. Transactions on Machine Learning Research , 2023. ISSN 2835-8856.

Jan Schuchardt, Yan Scholten, and Stephan Günnemann. Provable adversarial robustness for group equivariant tasks: Graphs, point clouds, molecules, and more. In Advances in Neural Information Processing Systems , 2023.

Zhuoqun Huang, Neil Marchant, Keane Lucas, Lujo Bauer, Olya Ohrimenko, and Benjamin I. P. Rubinstein. RS-Del: Edit distance robustness certificates for sequence classifiers via randomized deletion. In Advances in Neural Information Processing Systems , NeurIPS, 2023.

Yan Scholten, Jan Schuchardt, Aleksandar Bojchevski, and Stephan Günnemann. Hierarchical randomized smoothing. Advances in Neural Information Processing Systems , 36, 2023.

Chendi Wang, Buxin Su, Jiayuan Ye, Reza Shokri, and Weijie J Su. Unified enhancement of privacy bounds for mixture mechanisms via $f$-differential privacy. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Jason Altschuler and Kunal Talwar. Privacy of noisy stochastic gradient descent: More iterations without more privacy loss. Advances in Neural Information Processing Systems , 35, 2022.

Antonious Girgis, Deepesh Data, and Suhas Diggavi. Rényi differential privacy of the subsampled shuffle model in distributed learning. In Advances in Neural Information Processing Systems , volume 34, pages 29181-29192, 2021.

Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User-friendly differential privacy library in PyTorch. arXiv preprint arXiv:2109.12298 , 2021.

Borja Balle and Yu-Xiang Wang. Improving the gaussian mechanism for differential privacy: Analytical calibration and optimal denoising. In International Conference on Machine Learning , pages 394-403, 2018.

Úlfar Erlingsson, Vitaly Feldman, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Abhradeep Thakurta. Amplification by shuffling: From local to central differential privacy via anonymity. In Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 2468-2479, 2019.

Albert Cheu, Adam Smith, Jonathan Ullman, David Zeber, and Maxim Zhilyaev. Distributed differential privacy via shuffling. In Advances in Cryptology-EUROCRYPT , pages 375-403, 2019.

A Table of contents

B Additional experiments

B.1 Mechanism-agnostic and mechanism-specific bounds

B.1.1 Randomized response and RDP

In Fig. 7, we repeat the experiment from Fig. 3 with other ratios of batch and dataset sizes. That is, we compare our tight mechanism-specific RDP guarantee from Theorem I.3 to the best known mechanism-agnostic bound from [28] to illustrate that mechanism-agnostic bounds may loose privacy by bounding mixture divergences in terms of component divergences - even outside the group privacy setting. The tight guarantee eliminates constant factors and thus achieves much smaller ρ for a wide range of α ∈ (1 , 10 4 ] , especially for small ratios.

q / N

= 0

.

01

10

1

10

2

10

3

10

4

RDP

α

10

-

4

10

-

3

10

-

2

10

-

1

RDP

ρ

(

α

)

θ

0.9

0.75

0.6

Agnostic

Specific

q / N

= 0

.

1

10

1

10

2

10

3

10

4

RDP

α

10

-

2

10

-

1

RDP

ρ

(

α

)

q / N

= 0

.

2

10

1

10

2

10

3

10

4

α

10

-

2

10

-

1

10

0

RDP

ρ

(

α

)

q / N

= 0

.

5

10

1

10

2

10

3

10

4

α

10

-

1

10

0

RDP

ρ

(

α

)

RDP

RDP

B.1.2 Group privacy and ADP

In Figs. 8 and 9 we repeat the experiment from Fig. 4 for other subsampling rates r , standard deviations σ , and also with Laplace mechanisms. That is, we compare our tight mechanism-specific group privacy guarantees to the tight mechanism-agnostic bounds derived via the framework of [15]. The results demonstrate that mechanism-specific tightness is a stronger property that can result in better privacy guarantees - although the difference in significantly more pronounced for Laplace mechanisms and/or larger group sizes.

In Fig. 10 we repeat the experiment with randomized response mechanisms. Here, both approaches yield identical guarantees. This verifies that randomized reseponse is indeed the worst-case mechanisms for which the mechanism-agnostic bound is optimal (see [15] and Appendix N.2).

Gaussian mechanism

r

= 0

.

1

, σ

= 1

0

2

4

6

8

10

ADP

ε

0

.

0

0

.

2

0

.

4

ADP

δ

(

ε

)

Group size

16

8

4

2

1

Agnostic

Specific

r

= 0

.

1

, σ

= 2

0

1

2

3

4

5

ADP

ε

0

.

0

0

.

1

0

.

2

0

.

3

ADP

δ

(

ε

)

r

= 0

.

1

, σ

= 5

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

ADP

ε

0

.

00

0

.

05

0

.

10

ADP

δ

(

ε

)

r

= 0

.

2

, σ

= 2

0

1

2

3

4

5

6

ADP

ε

0

.

0

0

.

2

0

.

4

ADP

δ

(

ε

)

r

= 0

.

2

, σ

= 5

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

ADP

ε

0

.

0

0

.

1

0

.

2

ADP

δ

(

ε

)

r

= 0

.

5

, σ

= 1

0

2

4

6

8

10

ADP

ε

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ADP

δ

(

ε

)

r

= 0

.

5

, σ

= 2

0

2

4

6

8

10

ADP

ε

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

ADP

δ

(

ε

)

r

= 0

.

5

, σ

= 5

0

1

2

3

4

5

6

ADP

ε

0

.

0

0

.

2

0

.

4

ADP

δ

(

ε

)

Laplace mechanism

r

= 0

.

1

, λ

= 0

.

5

0

2

4

6

8

10

ADP

ε

0

.

0

0

.

2

0

.

4

0

.

6

ADP

δ

(

ε

)

Group size

16

8

4

2

1

Agnostic

Specific

r

= 0

.

1

, λ

= 1

0

1

2

3

4

ADP

ε

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

ADP

δ

(

ε

)

r

= 0

.

1

, λ

= 2

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

ADP

ε

0

.

0

0

.

1

0

.

2

0

.

3

ADP

δ

(

ε

)

r

= 0

.

2

, λ

= 0

.

5

0

2

4

6

8

10

ADP

ε

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

ADP

δ

(

ε

)

r

= 0

.

2

, λ

= 2

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

ADP

ε

0

.

0

0

.

2

0

.

4

ADP

δ

(

ε

)

r

= 0

.

5

, λ

= 0

.

5

0

2

4

6

8

10

ADP

ε

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ADP

δ

(

ε

)

r

= 0

.

5

, λ

= 1

0

2

4

6

8

10

ADP

ε

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ADP

δ

(

ε

)

r

= 0

.

5

, λ

= 2

0

1

2

3

4

5

6

ADP

ε

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

ADP

δ

(

ε

)

Randomized response mechanism

r

= 0

.

1

, θ

= 0

.

7

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

ADP

ε

0

.

0

0

.

1

0

.

2

0

.

3

ADP

δ

(

ε

)

Group size

16

8

4

2

1

Agnostic

Specific

r

= 0

.

1

, θ

= 0

.

8

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

ADP

ε

0

.

0

0

.

2

0

.

4

ADP

δ

(

ε

)

r

= 0

.

1

, θ

= 2

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

ADP

ε

0

.

0

0

.

2

0

.

4

0

.

6

ADP

δ

(

ε

)

r

= 0

.

2

, θ

= 0

.

7

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

ADP

ε

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

ADP

δ

(

ε

)

r

= 0

.

2

, θ

= 0

.

8

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

ADP

ε

0

.

0

0

.

2

0

.

4

0

.

6

ADP

δ

(

ε

)

r

= 0

.

5

, θ

= 0

.

7

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

ADP

ε

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

ADP

δ

(

ε

)

r

= 0

.

5

, θ

= 0

.

8

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

ADP

ε

0

.

0

0

.

2

0

.

4

0

.

6

ADP

δ

(

ε

)

r

= 0

.

5

, θ

= 0

.

9

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

ADP

ε

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

ADP

δ

(

ε

)

B.1.3 Group privacy and RDP

In Fig. 11 we repeat our comparison of mechanism-agnostic (Theorem N.5) and tight mechanismspecific group privacy amplification (Theorem 3.8) with RDP instead of ADP. We observe that the tight guarantee delays the phase transition from a high- to a low-privacy regime that was already observed in [30]. In other words, we have small ρ for larger α , which is beneficial for conversion to ADP after composition (see conversion formulae in [7, 31]).

r

= 0

.

1

, σ

= 1

1

.

1

×

10

0

2

×

10

0

4

×

10

0

RDP

α

10

-

1

10

1

10

3

10

5

RDP

ρ

(

α

)

Group size

8

4

2

1

Agnostic

Specific

r

= 0

.

001

, σ

= 1

10

1

1

.

1

×

10

0

4

×

10

0

RDP

α

10

-

4

10

-

1

10

2

10

5

RDP

ρ

(

α

)

r

= 0

.

1

, σ

= 2

10

1

1

.

1

×

10

0

4

×

10

0

RDP

α

10

-

2

10

0

10

2

10

4

RDP

ρ

(

α

)

r

= 0

.

001

, σ

= 2

10

1

RDP

α

10

-

5

10

-

2

10

1

10

4

RDP

ρ

(

α

)

r

= 0

.

001

, σ

= 5

10

1

10

2

10

3

10

4

RDP

α

10

-

3

10

-

1

10

1

10

3

RDP

ρ

(

α

)

r

= 0

.

001

, σ

= 5

10

1

10

2

10

3

10

4

RDP

α

10

-

6

10

-

3

10

0

10

3

RDP

ρ

(

α

)

B.1.4 Benefit of conditioning

Next, we demonstrate the benefit of coupling multiple conditional distributions over coupling just two subsampling distributions in deriving amplification guarantees. Specifically, we evaluate Proposition H.7, which can be derived via Theorem 3.3, i.e., optimal transport without conditioning. Note that, unlike with the guarantees of Wang et al. [28], this bound depends on both the dataset size and the batch size, not just their ratio. We make the following observations: For large α Proposition H.7 converges to the baseline. Furthermore, this approach never outperforms the baseline for group size 1 . Neither does it outperform the baseline for ratios q / L ∈ { 0 . 01 , 0 . 001 } . But, for q / L = 0 . 1 , there is a sweet spot of alphas between 10 1 and 10 2 in which it offers stronger guarantees. This further reinforces our claim that there is a benefit to treating group privacy and amplification jointly. Furthermore, we observe that in the case of q = 1 , Proposition H.7 outperforms the baseline for large alpha, since it can capture that one cannot possibly include more than one substituted element in a singleton batch.

Overall, we can conclude that there is a benefit to jointly analyzing group privacy and subsampling in this manner, but that optimal transport without conditioning is not sufficient for tightly analyzing such complicated scenarios.

N

= 10

5

, q

= 10

4

10

1

10

2

10

3

10

4

RDP

α

10

-

2

10

0

10

2

RDP

ρ

(

α

)

Group size

4

2

1

No conditioning

Post-hoc

N

= 10

5

, q

= 10

2

10

1

10

2

10

3

10

4

RDP

α

10

-

5

10

-

2

10

1

10

4

RDP

ρ

(

α

)

Group size

4

2

1

No conditioning

Post-hoc

N

= 10

4

, q

= 10

3

10

1

10

2

10

3

10

4

RDP

α

10

-

2

10

0

10

2

RDP

ρ

(

α

)

Group size

4

2

1

No conditioning

Post-hoc

N

= 10

4

, q

= 10

1

10

1

10

2

10

3

10

4

RDP

α

10

-

5

10

-

2

10

1

10

4

RDP

ρ

(

α

)

Group size

4

2

1

No conditioning

Post-hoc

N

= 10

3

, q

= 10

2

10

1

10

2

10

3

10

4

RDP

α

10

-

2

10

0

10

2

RDP

ρ

(

α

)

Group size

4

2

1

No conditioning

Post-hoc

N

= 10

3

, q

= 10

0

10

1

10

2

10

3

10

4

RDP

α

10

-

5

10

-

2

10

1

10

4

RDP

ρ

(

α

)

Group size

4

2

1

No conditioning

Post-hoc

B.1.5 Subsampling with replacement

The previous experiments demonstrated that mechanism-specific analysis can improve upon non-tight mechanism-agnostic and - in the group privacy setting - tight mechanism-agnostic analysis. In the following, we demonstrate that mechanism-specific bounds can improve upon tight mechanism-agnostic bounds even for single-element relations, i.e., group size 1. To this end, we consider subsampling with replacement for Gaussian mechanisms under the substitution relation, comparing Theorem L.5 to Theorem 10 of Balle et al. [15].

Fig. 13 shows the resultant privacy profiles for standard deviation σ = 1 , dataset size N = 100 , and batch size q = 8 . For ε ≥ 2 , the mechanism-specific bound is more than an order of magnitude smaller. Intuitively, this gap can be explained similarly to the gap in the group privacy setting: The single substituted element can be sampled 0 , 1 , 2 , or up to q times, with each case causing different levels of privacy leakage. Mechanism-agnostic bounds rely on a binary partitioning of the event space, which is not sufficient for capturing this granular behavior (recall Fig. 2.

Note that due to the relaxations of distance constraints we performed in deriving Theorem L.5, the mechanism-specific bound is not tight. A tight bound might lead to an even larger gap. Thus, this experiments further reinforces that there is a qualitative difference between mechanism-specific and mechanism-agnostic tightness.

10

0

10

1

ADP

ε

10

-

9

10

-

7

10

-

5

10

-

3

ADP

δ

(

ε

)

Agnostic

Specific

B.2 Post-hoc and tight group privacy

B.2.1 Single-iteration ADP

In Figs. 14 to 15 we repeat the experiment from Fig. 5 for other subsampling rates r , standard deviations σ , and mechanisms. That is, we compare our tight mechanism-specific guarantees to post-hoc use of the group privacy property. For all mechanisms, the tight mechanism-specific analysis yields stronger privacy guarantees than the baseline - particularly for large group sizes. However, we interestingly see that with increasing base mechanisms noise level and decreasing subsampling rate, the post-hoc bound converges towards the tight bound.

Gaussian mechanism

r

= 0

.

1

, σ

= 1

0

2

4

6

8

10

ADP

ε

0

.

0

0

.

2

0

.

4

0

.

6

ADP

δ

(

ε

)

Group size

16

8

4

2

1

Post-hoc

Specific

r

= 0

.

1

, σ

= 2

0

1

2

3

4

5

ADP

ε

0

.

0

0

.

1

0

.

2

0

.

3

ADP

δ

(

ε

)

r

= 0

.

1

, σ

= 5

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

ADP

ε

0

.

00

0

.

05

0

.

10

ADP

δ

(

ε

)

r

= 0

.

2

, σ

= 1

0

2

4

6

8

10

ADP

ε

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ADP

δ

(

ε

)

r

= 0

.

2

, σ

= 5

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

ADP

ε

0

.

0

0

.

1

0

.

2

ADP

δ

(

ε

)

r

= 0

.

5

, σ

= 1

0

2

4

6

8

10

ADP

ε

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ADP

δ

(

ε

)

r

= 0

.

5

, σ

= 2

0

2

4

6

8

10

ADP

ε

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ADP

δ

(

ε

)

r

= 0

.

5

, σ

= 5

0

1

2

3

4

5

6

ADP

ε

0

.

0

0

.

2

0

.

4

0

.

6

ADP

δ

(

ε

)

Laplace mechanism

r

= 0

.

1

, λ

= 0

.

5

0

2

4

6

8

10

ADP

ε

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ADP

δ

(

ε

)

Group size

16

8

4

2

1

Post-hoc

Specific

r

= 0

.

1

, λ

= 1

0

1

2

3

4

ADP

ε

0

.

0

0

.

2

0

.

4

0

.

6

ADP

δ

(

ε

)

r

= 0

.

1

, λ

= 2

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

ADP

ε

0

.

0

0

.

1

0

.

2

0

.

3

ADP

δ

(

ε

)

r

= 0

.

2

, λ

= 0

.

5

0

2

4

6

8

10

ADP

ε

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ADP

δ

(

ε

)

r

= 0

.

2

, λ

= 2

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

ADP

ε

0

.

0

0

.

2

0

.

4

0

.

6

ADP

δ

(

ε

)

r

= 0

.

5

, λ

= 0

.

5

0

2

4

6

8

10

ADP

ε

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ADP

δ

(

ε

)

r

= 0

.

5

, λ

= 1

0

2

4

6

8

10

ADP

ε

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ADP

δ

(

ε

)

r

= 0

.

5

, λ

= 2

0

1

2

3

4

5

6

ADP

ε

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ADP

δ

(

ε

)

Randomized response mechanism

r

= 0

.

1

, θ

= 0

.

7

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

ADP

ε

0

.

0

0

.

2

0

.

4

0

.

6

ADP

δ

(

ε

)

Group size

16

8

4

2

1

Post-hoc

Specific

r

= 0

.

1

, θ

= 0

.

8

0

1

2

3

4

5

6

ADP

ε

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ADP

δ

(

ε

)

r

= 0

.

1

, θ

= 2

0

1

2

3

4

5

6

ADP

ε

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ADP

δ

(

ε

)

r

= 0

.

2

, θ

= 0

.

7

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

ADP

ε

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ADP

δ

(

ε

)

r

= 0

.

2

, θ

= 0

.

8

0

1

2

3

4

ADP

ε

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ADP

δ

(

ε

)

r

= 0

.

5

, θ

= 0

.

7

0

1

2

3

4

5

6

ADP

ε

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ADP

δ

(

ε

)

r

= 0

.

5

, θ

= 0

.

8

0

2

4

6

8

10

ADP

ε

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ADP

δ

(

ε

)

r

= 0

.

5

, θ

= 0

.

9

0

2

4

6

8

10

ADP

ε

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ADP

δ

(

ε

)

B.2.2 Single-iteration RDP

In Fig. 17 we repeat our comparison of tight mechanism-specific group privacy amplification and post-hoc group privacy for RDP instead of ADP. The tight analysis yields stronger guarantees and delays the phase transition from a high- to a low-privacy regime. However, as with ADP, the post-hoc bound can be a good upper bound for small subsampling rates and very private base mechanisms.

r

= 0

.

1

, σ

= 1

1

.

1

×

10

0

2

×

10

0

4

×

10

0

RDP

α

10

-

1

10

1

10

3

10

5

RDP

ρ

(

α

)

Group size

8

4

2

1

Post-hoc

Specific

r

= 0

.

001

, σ

= 1

10

1

1

.

1

×

10

0

4

×

10

0

RDP

α

10

-

4

10

-

1

10

2

10

5

RDP

ρ

(

α

)

r

= 0

.

1

, σ

= 2

10

1

1

.

1

×

10

0

4

×

10

0

RDP

α

10

-

2

10

0

10

2

10

4

RDP

ρ

(

α

)

r

= 0

.

001

, σ

= 2

10

1

RDP

α

10

-

5

10

-

2

10

1

10

4

RDP

ρ

(

α

)

r

= 0

.

001

, σ

= 5

10

1

10

2

10

3

10

4

RDP

α

10

-

3

10

-

1

10

1

10

3

RDP

ρ

(

α

)

r

= 0

.

001

, σ

= 5

10

1

10

2

10

3

10

4

RDP

α

10

-

6

10

-

3

10

0

10

3

RDP

ρ

(

α

)

B.2.3 PLD Accounting

In Figs. 18 to 21 we repeat our comparison of tight mechanism-specific and post-hoc group privacy amplification guarantees under composition from. We consider different combinations of subsampling rate r , standard deviation σ , privacy parameter ε , as well as Laplace mechanisms. Even in high privacy scenarios, where the mechanism-specific and post-hoc analysis yield similar results on a single-iteration level, the mechanism-specific guarantees are significantly stronger under composition. Specifically, the post-hoc analysis diverges to much larger δ after some number of iterations. In settings with moderate privacy (e.g. σ = 1 ) or large group sizes (e.g. 16 ), the baseline diverges after less than 100 iterations.

Gaussian mechanism ( r = 0 . 001 )

ε

= 0

.

5

, σ

= 1

0

250

500

750

1000

Iteration

t

10

-

11

10

-

8

10

-

5

10

-

2

ADP

δ

(

ε

)

Group size

16

8

4

2

1

Post-hoc

Specific

ε

= 0

.

5

, σ

= 5

0

250

500

750

1000

Iteration

t

10

-

22

10

-

16

10

-

10

10

-

4

ADP

δ

(

ε

)

ε

= 1

, σ

= 1

0

250

500

750

1000

Iteration

t

10

-

13

10

-

9

10

-

5

10

-

1

ADP

δ

(

ε

)

ε

= 1

, σ

= 5

0

250

500

750

1000

Iteration

t

10

-

22

10

-

16

10

-

10

10

-

4

ADP

δ

(

ε

)

ε

= 2

, σ

= 1

0

250

500

750

1000

Iteration

t

10

-

17

10

-

12

10

-

7

10

-

2

ADP

δ

(

ε

)

ε

= 2

, σ

= 5

0

250

500

750

1000

Iteration

t

10

-

22

10

-

16

10

-

10

10

-

4

ADP

δ

(

ε

)

Gaussian mechanism ( r = 0 . 01 )

ε

= 0

.

5

, σ

= 1

0

200

400

600

800

1000

Iteration

t

10

-

5

10

-

3

10

-

1

ADP

δ

(

ε

)

Group size

16

8

4

2

1

Post-hoc

Specific

ε

= 0

.

5

, σ

= 5

0

250

500

750

1000

Iteration

t

10

-

21

10

-

15

10

-

9

10

-

3

ADP

δ

(

ε

)

ε

= 1

, σ

= 1

0

200

400

600

800

1000

Iteration

t

10

-

7

10

-

5

10

-

3

10

-

1

ADP

δ

(

ε

)

ε

= 1

, σ

= 5

0

250

500

750

1000

Iteration

t

10

-

21

10

-

15

10

-

9

10

-

3

ADP

δ

(

ε

)

ε

= 2

, σ

= 1

0

250

500

750

1000

Iteration

t

10

-

10

10

-

7

10

-

4

10

-

1

ADP

δ

(

ε

)

ε

= 2

, σ

= 5

0

250

500

750

1000

Iteration

t

10

-

21

10

-

15

10

-

9

10

-

3

ADP

δ

(

ε

)

Laplace mechanism ( r = 0 . 001 )

ε

= 0

.

5

, λ

= 1

0

250

500

750

1000

Iteration

t

10

-

15

10

-

11

10

-

7

10

-

3

ADP

δ

(

ε

)

Group size

16

8

4

2

1

Post-hoc

Specific

ε

= 0

.

5

, λ

= 5

0

250

500

750

1000

Iteration

t

10

-

14

10

-

10

10

-

6

10

-

2

ADP

δ

(

ε

)

ε

= 1

, λ

= 1

0

250

500

750

1000

Iteration

t

10

-

15

10

-

11

10

-

7

10

-

3

ADP

δ

(

ε

)

ε

= 1

, λ

= 5

0

250

500

750

1000

Iteration

t

10

-

14

10

-

10

10

-

6

10

-

2

ADP

δ

(

ε

)

ε

= 2

, λ

= 1

0

250

500

750

1000

Iteration

t

10

-

15

10

-

11

10

-

7

10

-

3

ADP

δ

(

ε

)

ε

= 2

, λ

= 5

0

250

500

750

1000

Iteration

t

10

-

14

10

-

10

10

-

6

10

-

2

ADP

δ

(

ε

)

Laplace mechanism ( r = 0 . 01 )

ε

= 0

.

5

, λ

= 1

0

250

500

750

1000

Iteration

t

10

-

15

10

-

11

10

-

7

10

-

3

ADP

δ

(

ε

)

Group size

16

8

4

2

1

Post-hoc

Specific

ε

= 0

.

5

, λ

= 5

0

250

500

750

1000

Iteration

t

10

-

14

10

-

10

10

-

6

10

-

2

ADP

δ

(

ε

)

ε

= 1

, λ

= 1

0

250

500

750

1000

Iteration

t

10

-

15

10

-

11

10

-

7

10

-

3

ADP

δ

(

ε

)

ε

= 1

, λ

= 5

0

250

500

750

1000

Iteration

t

10

-

14

10

-

10

10

-

6

10

-

2

ADP

δ

(

ε

)

ε

= 2

, λ

= 1

0

250

500

750

1000

Iteration

t

10

-

15

10

-

11

10

-

7

10

-

3

ADP

δ

(

ε

)

ε

= 2

, λ

= 5

0

250

500

750

1000

Iteration

t

10

-

14

10

-

10

10

-

6

10

-

2

ADP

δ

(

ε

)

B.2.4 RDP accounting

Finally, we compare tight group privacy amplification to the post-hoc approach for RDP accounting. We begin with the Gaussian mechanism and subsampling rate r = 0 . 001 in Fig. 22. Our bound offers stronger group privacy guarantees for σ = 1 . 0 , but both results are almost identical for very private base mechanisms with σ = 5 . 0 . We suspect that this is because the phase transition from high to low privacy (recall Fig. 17), which the tight guarantee delays, gets shifted to very high α regions that are not useful for conversion to ( ε, δ ) -DP.

Our observations for randomized response mechanisms (see Fig. 23 are also consistent with earlier results: For moderate subsampling rates r = 0 . 1 , our guarantees are stronger, particularly for group size 8 and large numbers of iterations. But, when decreasing the subsampling rate to 0 . 001 , both methods are almost identical. Nevertheless, group privacy amplification can demonstrably improve upon a direct combination of independently derived group privacy and amplification guarantees.

Note that these observations are mostly of theoretic interest. In practice, one would use PLD accounting, which tightly characterizes the composed mechanism's privacy leakage, instead of the looser RDP accounting. As shown in Figs. 18 to 21, the tight mechanisms-specific analysis drastically outperforms the post-hoc analysis for PLD accounting.

Gaussian mechanism

r

= 0

.

001

, σ

= 1

10

0

10

1

10

2

10

3

10

4

10

5

Iteration

t

10

0

10

1

ADP

ε

Group size

8

4

2

1

Post-hoc

Specific

r

= 0

.

001

, σ

= 5

10

0

10

1

10

2

10

3

10

4

10

5

Iteration

t

10

-

1

10

0

ADP

ε

Randomized response mechanism

r

= 0

.

1

, θ

= 0

.

6

10

0

10

1

10

2

10

3

10

4

10

5

t

10

0

10

2

10

4

ADP

ε

Group size

8

4

2

1

Post-hoc

Specific

r

= 0

.

001

, σ

= 0

.

6

10

0

10

1

10

2

10

3

10

4

10

5

t

10

-

3

10

-

2

10

-

1

10

0

ADP

ε

Iteration

Iteration

B.2.5 Model utility

The previous results demonstrated that handling group privacy via a tight mechanism-specific analysis allows for a larger number of compositions, i.e., iterations at a given privacy budget than post-hoc use of the group privacy property. In the following, we demonstrate that this increased number of iterations can in fact result in increased utility for group-private machine learning models.

We train a convolutional neural network (2 convolution layers with kernel sizes 3 and 32 / 64 channels, followed by two linear layers with hidden dimension 128 ) for image classification on MNIST ( 55000 training, 5000 validation, 10000 test samples). We set the gradient clipping norm of DP-SGD [6] to C = 10 -4 , the Gaussian noise standard deviation to 0 . 6 · C , and the subsampling rate to r = 64 / 55000 . The optimizer is ADAM with learning rate 1 e -3 . If the privacy budget is not used up earlier, training is terminated after 8 epochs, with ⌈ 55000 / 64 ⌉ iterations per epoch.

Even with a large privacy budget of ε = 8 and δ = 1 e -5 , training with the post-hoc privacy analysis needs to terminate after less than 200 iterations. With the mechanism-specific analysis, δ does not even exceed 1 e -7 after 8 epochs, i.e., the model could potentially be trained for even more iterations. The resultant validation accuracy's are 79 . 6% and 91 . 2% , respectively. This showcases the superior privacy-utility trade-off that can be achieved by analyzing subsampling and group privacy jointly.

0

2000

4000

6000

Iteration t

10

-

10

10

-

7

10

-

4

ADP

δ

(

ε

= 8)

Post-hoc

Specific

Budget

(a) Privacy

0

2

4

6

8

Epoch

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Val. accuracy

Post-hoc

Specific

(b) Utility

C Experimental setup

C.1 Evaluation of privacy guarantees

In all experiments, we assume that all mechanisms have underlying sensitivity 1 w.r.t. ℓ ∞ (randomized response), ℓ 1 (Laplace), or ℓ 2 (Gaussian) output norms. We thus only specify noise parameters σ or λ , rather than the ratio of σ or λ and the sensitivities.

C.1.1 Single-iteration approximate differential privacy

Privacy parameters. We evaluate all guarantees for 121 equidistant values of ε in [0 , 4] and 121 logspace-equidistant values in 10 -3 , 10 1 , i.e., { 10 x | x ∈ {-3 , -3 + 4 120 , . . . , 1 }} . We clip values of δ ( ε ) that are larger than 1 to [0 , 1] . For our baselines, we enforce that δ ( ε ) is monotonically decreasing by taking a running minimum from larger to smaller ε (this may improve but never worsens the baselines).

Mechanism-agnostic group privacy baseline. As our mechanism-agnostic group privacy baselines, we use Proposition N.1. We take the maximum over the two cases ( K + = K,K -= 0) and ( K + = 0 , K -= K ) , where K is the group size. For Gaussian and Laplace mechanisms, we evaluate group privacy profiles δ k ( ε ) by determining δ ( ε ) for the same mechanism with sensitivity k . For randomized response, we let δ k = δ 1 . This is referred to to as 'white-box' group privacy in [15]). We evaluate this baseline analytically.

Post-hoc group privacy baseline. For our post-hoc baseline, we combine the tight Poisson subsampling guarantee for insertion/removal from [15] with the group privacy property of approximate differential privacy (see Appendix C.1.4 below). We evaluate this baseline analytically.

Tight mechanism-specific group privacy. In all ADP figures 'specific' refers to our tight group privacy guarantees for Gaussian (Theorem 3.8), Laplace (Theorem M.2), or randomized response mechanisms (Theorem M.3). We take the maximum over all K + , K -∈ N 0 with K + + K -= K , where K is the group size. To evaluate the bounds for Gaussian and Laplace mechanisms, we pessimistically invert the privacy loss of dominating pairs P, Q via binary search (see details in Appendix M.4), as implemented in the dp_accounting library [45]. We set the precision to 10 -6 , i.e., find the global optimum over multiples of 10 -6 . The bound for the randomized response mechanism is evaluated analytically in O (1) .

C.1.2 Single-iteration Rényi differential privacy

Privacy parameters . We evaluate all guarantees for α ∈ { 2 , 3 , . . . , 1000 } , as well as 121 logspaceequidistant values in [0 , 10 4 ] rounded to the next smallest integer (without 0 or 1 ), and 121 logspaceequidistance values in (1 , 10] . For our baselines, we enforce that ρ ( α ) is monotonically increasing by taking a running minimum from smaller to larger α (this may improve but never worsens the baselines).

Mechanism-agnostic group privacy baseline. As our mechanism-agnostic group privacy baselines, we use Proposition N.1. We take the maximum over the two cases ( K + = K,K -= 0) and ( K + = 0 , K -= K ) , where K is the group size. For integer α , we use the binomial expansion in Eq. (33) (without factor 2 ). For Gaussian and Laplace mechanisms, we evaluate group privacy profiles ζ k ( α ) by determining ζ ( α ) for the same mechanism with sensitivity k . For randomized response, we let ζ k = ζ 1 . For continuous α , we apply tanh -sinh -quadrature with 50 digits of decimal precision to Eq. (33)

Post-hoc group privacy baseline. For our post-hoc baselines, we combine either tight Poisson subsampling guarantee for insertion/removal from [30] or the subsampling without replacement guarantee for insertion/removal from [28] with the group privacy property of approximate differential privacy (see Appendix C.1.4 below). For Poisson subsampling and integer α , we use the binomial expansion from [30] (without factor 2 ). For continuous α , we apply tanh -sinh -quadrature with 50 digits of decimal precision. For subsampling without replacement, we use the improved selfconsistency bound (Theorem 27 from [28]), which we evaluate via tanh -sinh -quadrature with 50 digits of decimal precision due to the intractability of the nested binomial expansions for larger α .

Tight mechanism-specific group privacy. In all RDP figures 'specific' refers to our tight group privacy guarantees for Gaussian (Theorem 3.8), Laplace (Theorem M.2), or randomized response

mechanisms (Theorem M.3). We take the maximum over all K + , K -∈ N 0 with K + + K -= K , where K is the group size. To evaluate the bounds for Gaussian and Laplace mechanisms, we use tanh -sinh -quadrature with 50 digits of decimal precision (see discussion in Appendix M.4). The bound for the randomized response mechanism is evaluated analytically in O (1) .

C.1.3 Privacy accounting

RDP accounting. For RDP accounting, we simply evaluate the single-iteration guarantees as described in Appendix C.1.2. We then multiply the ρ ( α ) with the number of iterations, apply the improved RDP-to-ADP formula from [31], and take the minimum over all obtained values. For the post-hoc baseline, we apply the group privacy property before composition (which is equivalent to applying it after composition, due to additivity of composition).

PLD accounting. For PLD accounting with dominating pairs, we use the implementation of the dp_accounting library [45]. To quantize the privacy loss distribution, we use 'connect the dots' [20] with pessimistic estimates, a discretization interval size of 10 -3 , and truncation of e -50 of the probability mass. During composition, we truncate 10 -15 of the tail mass. For the post-hoc baseline, we apply the group privacy property after composition.

C.1.4 Post-hoc group privacy

ADP. For RDP, we use the following result from the proof of Lemma 2.2 in [23], which provides tigher guarantees than Lemma 2.2 itself: Let M be ( ε, δ ) -DP under neighboring relation ≃ X . Then, M is ( ε ′ , δ ′ ) -DP with ε ′ = ε · K and δ ′ = ∑ K -1 e k · ε · δ .

RDP. For RDP, we use the following result from Corollary 4 of [7], which provides tighter guarantees than the upper bound in their Proposition 2. Let D α be log(Λ α ) / ( α -1) . This D α fulfills the following triangle inequality:

We recursively apply this bound log 2 ( K ) times when evaluating the group privacy of our baselines for groups of size K .

C.2 Computational resources

We conduct all experiments on a set of Xeon E5-2630 v4 CPUs @ 2 . 2 GHz .

We use one worker and job per subsampling theorem, base mechanism noise level, subsampling rate, and combination of group privacy parameters K -, K + . Per job, we allocate 2 CPU cores, 4 GB and 10 minutes of runtime (in practice, most jobs were completed in a few seconds). In total, we ran 13060 such jobs for ADP, 9731 for RDP, and 984 for RDP accounting. We estimate that over the course of the full research project twice as many jobs were executed.

C.3 Assets and licenses

To perform high-precision quadrature for RDP guarantees, we use the tanh -sinh quadrature implementation from the mpmath library (version 1 . 3 . 0 . ), which is available under the BSD3 -Clause license at https://github.com/mpmath/mpmath .

For PLD accounting and evaluation of ADP guarantees via bisection, we use and extend the dp_accounting library [45] (commit 0b109e959470c43e9f177d5411603b70a56cdc7a ), which is available under Apache2 . 0 license at https://github.com/google/ differential-privacy

For conversion from RDP to ADP guarantees, we use the get_privacy_spent method implemented in the Opacus library [73] (version 1 . 4 . 1 ), which is available under Apache2 . 0 license at https: //github.com/pytorch/opacus .

An implementation will be made available at https://cs.cit.tum.de/daml/group-amplification.

D General setting and definitions

In the following, we generalize some of the definitions introduced in Sections 2 and 3 to their measure-theoretic equivalents. The purpose of this generalization is to handle both continuous and discrete spaces, base mechanisms, and subsampling schemes, without having to make constant case distinctions.

We use these more general definitions throughout the remaining appendix sections. The theoretic results presented in Section 3 follow as special cases.

D.1 Spaces

Unlike before, we assume that dataset space X is some arbitrary space, whose elements do not have to be sets. Instead, datasets x ∈ X can also be graphs, sequences or any other data collection. We further assume that the batch space is a measurable space ( Y , Y ) with σ -algebra Y . For example, Y can be composed of subsets, subgraphs, or subsequences. We also assume the output space to be a measurable space ( Z , Z ) . Finally, we assume the existence of some measure λ on the output space, such as the Lebesgue measure for continuous Z or the counting measure # for discrete Z .

Whenever we consider Poisson subsampling, we assume that X ⊆ P ( A ) , Y = { y ⊆ x | x ∈ X } , Y = P ( Y ) , where A is some discrete, finite set and P ( · ) is the powerset.

Whenever we consider subsampling without replacement with batch size q , we assume that X ⊆ { x ∈ P ( A ) | | x | > q } , Y = { y ⊆ x | x ∈ X , | y | = q } , Y = P ( Y ) , where A is some discrete, finite set and P ( · ) is the powerset.

D.2 Neighboring relations

For set-valued datasets and batches, the insertion/removal relation and the substitution relation are formally defined as follows:

̸

Definition D.1. Sets x, x ′ ∈ X are related by the insertion/removal relation ( x ≃ ± x ′ ) if x = x ′ and there is some a such that x ′ = x ∪ { a } or x ′ = x \ { a } .

Definition D.2. Sets x, x ′ ∈ X are related by the substitution relation ( x ≃ ∆ x ′ ) if there is some a ∈ x and a ′ / ∈ x such that x ′ = x \ { a } ∪ { a ′ } .

In our general setting, we also allow non-symmetric neighboring relations.

D.3 Mechanisms and subsampling schemes

As before, the term 'mechanism' refers to random functions that map to the output space ( Z , Z ) . Formally, a random function M : X → Z is a family of random variables indexed by elements of X .

Definition D.3. A random function M : X → Z is a function M : X × Ω → Z , where (Ω , F , P ) is some probability space and all M ( x, · ) : ω ↦→ M ( x, ω ) are measurable.

Wewrite P M x : Z → [0 , 1] for the distribution of random variable M ( x ) . Wefurther assume that each P M x is absolutely continuous w.r.t. the aforementioned output measure λ , i.e., ∀ x ∈ X : P M x ≪ λ , and write m x : Z → R + for the corresponding Radon-Nikodym derivative d P M x / d λ . For example, when the output space ( Z , Z ) is continuous and λ is the Lebesgue measure, then m x is the density.

Similarly, we define our base mechanism to be a random function B : Y → Z and write P B y : Z → [0 , 1] for the distribution of base mechanism outputs given a batch y ∈ Y . We assume that each P B y is absolutely continuous w.r.t. output measure λ and write b y : Z → R + for d P B y / d λ .

Finally, we define our subsampling scheme to be a random function S : X → Z and write P S x : Y → [0 , 1] for the distribution of batches given a dataset x ∈ X . We generally do not require P S x to be absolutely continuous w.r.t. some other measure. When P S x is absolutely continuous w.r.t. counting measure # , we write s x : Y → [0 , 1] for the corresponding probability mass function d P S x / d# .

In particular, Poisson subsampling and subsampling without replacement are defined as follows:

Definition D.4. Poisson subsampling with rate r ∈ [0 , 1] has probability mass function s x ( y ) = r | y | (1 -r ) | x |-| y | for batches y ⊆ x .

Definition D.5. Subsampling without replacement with batch size q has probability mass function s x ( y ) = ( | x | q ) -1 for batches y ⊆ x with | y | = q .

As before, our goal is to provide privacy guarantees for subsampled mechanisms M = B ◦ S . Similar to our discussion in Section 2, its distribution P M x given a dataset x ∈ X is a mixture 3 with

There is one component per batch y from batch space Y , and the weights depend on subsampling distribution P S x .

D.4 Differential privacy notions

Since we no longer require the output space to be continuous, we also need to generalize the definitions of approximate differential privacy, Rényi differential privacy, dominating pairs, and the divergences underlying their definition. For this, recall that λ is our assumed measure on output space ( Z , Z ) .

Definition D.6. For ε ≥ 0 , a mechanism M : X → Z is ( ε, δ ) -DP under relation ≃ X if ∀ x ≃ X x ′ : H exp( ε ) ( m x || m x ′ ) ≤ δ and H exp( ε ) ( m x ′ || m x ) ≤ δ with hockey stick divergence

Definition D.7. A pair of distributions ( P, Q ) with p = d P / d λ and q = d Q/ d λ is a dominating pair for mechanism M under neighboring relation ≃ X , if H α ( m x || m x ′ ) ≤ H α ( p || q ) for all x ≃ X x ′ and all α ≥ 0 .

Definition D.8. For α ≥ 1 , a mechanism M : X → R D is ( α, ρ ) -RDP under neighboring relation ≃ X if ∀ x ≃ X x ′ : log(Λ α ( m x || m x ′ )) / ( α -1) ≤ ρ and log(Λ α ( m x ′ || m x )) / ( α -1) ≤ ρ with

Importantly, note that Λ α is not the Rényi divergence as used in [7]. It is its α th moment, i.e., a scaled and exponentiated Rényi divergence. This is why a logarithm and quotient appears in Definition D.8. We use this definition, so that we can simultaneously discuss ADP and RDP without notational clutter.

D.5 Joint convexity

The joint convexity of Ψ α ∈ { H α , Λ α ) is not limited to densitites, but also applies to other nonnegative Radon-Nikodym derivatives of probability distributions [37, 38]:

Lemma D.9. Consider arbitrary Radon-Nikodym derivatives f (1) 1 , f (1) 2 , f (2) 1 , f (2) 2 : Z → R + and weight w ∈ [0 , 1] . Then,

D.6 Couplings

As with the discrete, finite-support subsampling distributions we considered in Section 3, the key tool we use for analyzing amplification are couplings. However, we no longer assume the subsampling scheme to always have a mass function. We thus use a more general notion of couplings between distributions instead of couplings between mass functions:

Definition D.10. A coupling between probability measures P 1 , . . . , P N on space ( Y , Y ) is a probability measure Γ on product space ( Y , Y ) N , where the n th marginal is P n , i.e., Γ ◦ π -1 n = P n with projection π n ( y ) = y n .

Here, ◦ is the composition operator and π -1 n is the preimage (not necessarily the inverse) of the projection function. As before, when considering a coupling between two distributions P 1 , P 2 , the value Γ( T, R ) specifies for all events T, R ∈ Y how much probability should be transported from P 1 ( T ) to P 2 ( R ) to transform P 1 into P 2 .

E Proof of optimal transport bounds

E.1 Proof of Theorem 3.3

In the following, we show a more general statement for the general setting introduced in Appendix D. Theorem 3.3 immediately follows in the special case where batch space Y is finite and discrete, and the subsampling distribution has a probability mass function s x .

Theorem E.1. Consider a subsampled mechanism M = B ◦ S , and an arbitrary coupling Γ between subsampling distributions P S x and P S x ′ . Then

with cost function c α ( y (1) , y (2) ) = Ψ α ( b y (1) || b y (2) ) .

Proof. Recall that m x and m x ′ are mixtures with m x ( z ) = ∫ b y ( z ) dP S x ( y ) and m x ′ ( z ) = ∫ b y ( z ) dP S x ′ ( y ) . Since Γ is a coupling between P S x and P S x ′ , we can use the projection π n ( y ) = y n and change of variables to rewrite these mixtures as

Since m x ( z ) and m x ′ ( z ) are now expectations w.r.t. the same measure, we can use the joint convexity of Ψ α (Lemma D.9) to show Ψ α ( m x , m x ′ ) ≤ ∫ Y 2 Ψ α ( b y 1 || b y 2 ) dΓ( y ) .

E.2 Proof of Theorem 3.4

As before, we prove a more general statement from which Theorem 3.4 immediately follows. For this, recall that Y is the σ -algebra of batch space ( Y , Y ) , and that P ( T | R ) = P ( T ∩ R ) / P ( R ) .

Theorem E.2. Consider a subsampled mechanism M = B ◦ S . Further consider two disjoint partitionings ⋃ I i =1 A i = Y and ⋃ J j =1 E j = Y such that all A i , E j are in Y and have non-zero measure under S x and S x ′ , respectively. Let Γ be an arbitrary coupling between P S x ( · | A 1 ) , . . . , P S x ( · | A I ) , P S x ′ ( · | E 1 ) , . . . , P S x ′ ( · | E J ) . Then,

with cost function c : Y I × Y J → R + defined by

Proof. Using the law of total expectation, linearity of integration, and change of variables with projection π n ( y ) = y n shows that

and

Since m x ( z ) and m x ′ ( z ) are now expectations w.r.t. the same measure, we can use the joint convexity of Ψ α (Lemma D.9) to show

A change of indexing via y (1) i = y i and y (2) j = y ( j + I ) concludes our proof.

E.3 Proof of Proposition 3.5

Proposition 3.5. Consider y (1) ∈ Y I , y (2) ∈ Y J , and cost function c defined in Eq. (2) . Let d Y be the distance induced by ≃ Y (see Definition 2.4). Then, c α ( y (1) , y (2) ) ≤ ˆ c α ( y (1) , y (2) ) , with

subject to ∀ k, l ∈ { 1 , 2 } , ∀ t, u : d Y (ˆ y ( k ) t , ˆ y ( l ) u ) ≤ d Y ( y ( k ) t , y ( l ) u ) and ˆ y (1) ∈ Y I , ˆ y (2) ∈ Y J .

Proof. The original tuples of batches y (1) ∈ Y I and y (2) ∈ Y J constitute a feasible solution to the maximization problem, since they fulfill the constraints with equality, i.e., ∀ k, l, t, u : d Y ( y ( k ) t , y ( l ) u ) = d Y ( y ( k ) t , y ( l ) u ) . The value of any feasible solution to a maximization problem is l.e.q. its optimal value.

F Distance-compatible couplings of multiple distributions

d = 2

d = 1

d = 1

𝑦1

𝑦2

supp(𝑠1)

supp(𝑠3)

compatible

coupling

d = 2

d > 1

d > 1

𝑦1

𝑦2

supp(𝑠1)

supp(𝑠3)

incompatible

coupling

In the following, we generalize the notion of distance-compatible couplings proposed in [15] from two distributions to an arbitrary number of distributions. This provides a sufficient optimality condition for Theorem E.2.

Note that we use a more general, measure-theoretic definition of distance-compatibility (see Definition F.4). Definition 3.6 is a special case for subsampling schemes with finite, discrete support.

As discussed in Section 3, a d Y -compatible coupling between (conditional) subsampling distributions only assigns probability to tuples of batches y when all pairs y i , y j have the smallest possible distance to y 1 and to each other, while still being in the support of their respective distributions.

To avoid having to make additional assumptions about the topology of batch space ( Y , Y ) , we will assume that all subsampling schemes have densities and reason about the support of these densities.

Definition F.1. The support of a function p : Y → R + is supp( p ) = { y ∈ Y | p ( y ) > 0 } .

Based on this notion of support, we can define a notion of distance between a batch y ∈ Y and the support of a density:

Definition F.2. Consider a distance d Y induced by a neighboring relation ≃ Y . The distance between an element y ∈ Y and the support of a function p : Y → R + is defined as d Y ( y, supp( p )) = min y ′ ∈ supp( p ) d Y ( y, y ′ ) .

Furthermore, we can define a notion of distance between the support of two different densities:

Definition F.3. Consider a distance d Y induced by a neighboring relation ≃ Y . The distance between the support of functions p 1 , p 2 : Y → R + is defined as

Based on these definitions, we can now formally define distance-compatible couplings between multiple distributions.

Definition F.4. Consider a coupling Γ between probability measures P 1 , . . . , P N on measurable space ( Y , Y ) with symmetric neighboring relation ≃ Y and induced distance d Y . Assume that ∀ n : P n ≪ ν n for some measures ν 1 , . . . , ν N and define p n = d P n / d ν n . Further assume that Γ ≪ ∏ N n =1 ν n with product measure ∏ N n =1 ν n , and define γ = d Γ / d ∏ ν n . Then, Γ is a d Y -compatible coupling if

That is, Γ only assigns probability to a tuple of batches y if all y t and y u are as close as possible to y 1 and as close as possible to each other, while still being in the support of their corresponding densities. Note that our choice of focusing on y 1 is arbitrary, and d Y -compatibility could also be defined for any other reference index n ∈ { 1 , . . . , N } .

We shall now prove that d Y -compatibility is a sufficient optimality condition for our optimal transport problem. For this proof, we will use the following lemma, which immediately follows from Definitions F.2 and F.3:

Lemma F.5. Consider a distance d Y induced by a relation ≃ Y and two functions p 1 , p 2 : Y → R + . Then, for all y 1 ∈ supp( p 1 ) , y 2 ∈ supp( p 2 ) ,

Theorem F.6. Consider a subsampled mechanism M = B ◦ S . Further consider two finite partitions A 1 , . . . , A I ∈ Y and E 1 , . . . , E J ∈ Y of Y such that all A i and E j have non-zero measure under S x and S x ′ , respectively. Let d Y be the distance induced by a symmetric neighboring relation ≃ Y . Let Γ ∗ be a d Y -compatible coupling between P S x ( · | A 1 ) , . . . , P S x ( · | A I ) , P S x ′ ( · | E 1 ) , . . . , P S x ′ ( · | E J ) , which have Radon-Nikodym derivatives s 1 , . . . , s I + J . Then, for all α > 1 ,

where G is the set of valid couplings between the I + J measures, and ˆ c α : Y I × Y J → R + is the cost function upper bound defined in Proposition 3.5.

Proof. Consider an arbitrary, not necessarily d Y -compatible coupling Γ . By definition of ˆ c and symmetry of ≃ , we have

with original cost function c α : Y I × Y J → R + defined in Theorem E.2.

We can now use Lemma F.5 to tighten the constraints of the optimization problem inside the integrand and thus lower-bound its optimal value for all y ∈ supp(Γ) :

We notice that the lower bound only depends on y 1 and shall thus refer to it as κ α ( y 1 ) . Further note that any y / ∈ supp(Γ) does not contribute to the integral. Since Γ is a valid coupling, we can marginalize out all variables except y 1 via projection π 1 ( y ) = y 1 to show

By construction of κ α , this holds with equality whenever Γ ∗ is a d Y -compatible coupling.

Finally, the exact derivations we used for Theorem F.6 can also be used to show that the optimal value of our transport problem has a simple, canonical form whenever a d Y -compatible coupling exists:

Corollary F.7. Consider a subsampled mechanism M = B ◦ S . Further consider two finite partitions A 1 , . . . , A I ∈ Y and E 1 , . . . , E J ∈ Y of Y such that all A i and E j have non-zero measure under S x and S x ′ , respectively. Let d Y be the distance induced by a symmetric neighboring relation ≃ Y . Assume that a d Y -compatible coupling between P S x ( · | A 1 ) , . . . , P S x ( · | A I ) , P S x ′ ( · | E 1 ) , . . . , P S x ′ ( · | E J ) with Radon-Nikodym derivatives s 1 , . . . , s I + J exists. Then, for all α > 1 ,

where G is the space of valid couplings between the I + J measures,

and c α : Y I × Y J → R + is the original cost function defined in Theorem E.2.

Thus, we can focus on constructing distance-compatible couplings when trying to derive existing or novel amplification by subsampling guarantees. Note that these result also generalize to asymmetric neighboring relations ≃ Y . We just wanted to avoid further complicating the indexing. Future work may want to generalize these results to a more general, topological notion of support that does not rely on the existence of subsampling densities.

G Recovering known mechanism-agnostic ADP guarantees

In the following, we first provide an overview of the framework for deriving mechanism-agnostic ADP guarantees proposed by Balle et al. [15]. We then show that the same guarantees can be derived via optimal transport between multiple subsampling distributions. Finally, we use observations made during the proof to explain why the mechanism-agnostic guarantees derived via this approach can be suboptimal.

G.1 Overview of Balle et al. framework

The framework from [15] uses four steps to derive tight mechanism-agnostic guarantees for subsampled mechanisms: (1) Partitioning the subsampling distributions via maximal couplings, (2) applying advanced joint convexity, (3) applying joint convexity, and (4) defining two couplings involving two distributions.

Maximal couplings are a construction that makes it possible to partition the subsampling probability mass functions as

with probability mass functions p x , q x , q x ′ : Y → [0 , 1] chosen such that q x and q x ′ have disjoint support and w ∈ [0 , 1] is as small as possible.

Note that, when considering typical subsampling schemes (Poisson, without replacement, with replacement) and single-element neighboring relations (insertion/removal, substitution), this partition is simply equivalent to

where A 1 and E 1 are the event that the inserted/removed or substituted element is not sampled, and A 2 , E 2 are their complements (see Appendix B in [15]).

Using this construction, the densitities m x and m x ′ of subsampled mechanisms M = B ◦ S can be rewritten as

Next, one can rewrite the divergence H α ( m x || m x ′ ) via the following property:

Proposition G.1 (Advanced joint convexity [15]) . Let m x , m ′ x : Z → [0 , 1] be probability mass functions satisfying m x ( z ) = (1 -w ) f ( z ) + wg ( z ) and m x ′ ( z ) = (1 -w ) f ( z ) + wg ′ ( z ) for some w ∈ [0 , 1] , f, g, g ′ : Z → [0 , 1] . Given α ≥ 1 , let α ′ = 1 + w ( α -1) and β = α ′ / α . Then the following holds:

Applying advanced joint convexity, followed by joint convexity to Eqs. (12) and (13) shows that

Finally, one can construct a coupling γ : Y 2 → [0 , 1] between q x and p x , as well as a coupling γ ′ : Y 2 → [0 , 1] between q x and q x ′ . One can then invoke a special case of Theorem 3.3 with Ψ α = H α and the cost function upper bound from Proposition 3.5 to show

Since we are only considering pairs of batches, we have

with induced distance d Y from Definition 2.4. This specific cost function bound is referred to as 'group privacy profile' in [15].

G.2 Subsumption of Balle et al. framework

Next, we show that we can always obtain the same guarantee by defining a (potentially suboptimal) coupling between four subsampling distributions, and then pessimistically upper-bounding the guarantee we would obtain through Theorem 3.4:

Theorem G.2. Consider a subsampled mechanism M = B ◦ S and some x, x ′ ∈ X . Assume that subsampling pmfs s x , s x ′ : Y → [0 , 1] satisfy s x ( z ) = (1 -w ) p x ( z ) + wq x ( z ) and s x ′ ( z ) = (1 -w ) p x ( z ) + wq x ′ ( z ) for some w ∈ [0 , 1] and p x , q x , q x ′ : Y → [0 , 1] . Let γ : Y 2 → [0 , 1] be an arbitrary coupling of q x , p x , and γ ′ : Y 2 → [0 , 1] be an arbitrary coupling of q x , p x ′ . Then, there is a coupling ˜ γ : Y 4 → [0 , 1] of p x , q x , p x , q x ′ such that for all α ≥ 1

with cost function upper bound ˆ c α defined in Proposition 3.5, α ′ = 1 + w ( α -1) , and β = α ′ / α .

Proof. The main idea is to invoke Theorem 3.4 with a specifically crafted coupling, and then apply advanced joint convexity and joint convexity.

Specifically, we define a coupling ˜ γ : Y → [0 , 1] that corresponds to the following generative process: We first sample y (1) 2 and y (2) 2 from the coupling γ ′ (recall that a coupling is a joint mass function). This gives us two elements from the support of q x and q x ′ , respectively. Then, we sample y (1) 1 from γ conditioned on y (1) 2 . This gives us an element from the support of p x . Finally, we let y (2) 1 ← y (1) 1 . Formally, this coupling can be defined as

Validity of coupling. Before proceeding, we need to verify that this is a valid coupling, i.e., its four marginals are p x , q x , p x , q x ′ . For any y (1) 1 ∈ Y , we have

where the first inequality is due to the indicator function, the second equality follows from marginalizing γ ′ , and the last equality follows from marginalizing γ .

The proof for any y (2) 1 ∈ Y is analogous.

For any y (1) 2 ∈ Y , we similarly have

First inequality. Now that we have a valid coupling, we can use the same joint-convexity argument as in our proof of Theorem 3.4, combined with our cost function bound ˆ c α to show

with each of the | Y 2+2 | optimization problems being constrained by ∀ k, l, t, u : d Y (ˆ y ( k ) t , ˆ y ( l ) u ) ≤ d Y ( y ( k ) t , y ( l ) u ) and ˆ y (1) ∈ Y 2 , ˆ y (2) ∈ Y 2 . This corresponds to the first equality in our Theorem.

Second inequality. Due to construction of our coupling, we always have d Y ( y (1) 1 , y (2) 1 ) = 0 , i.e., y (1) 1 = y (2) 1 . We can thus use advanced joint convexity, joint convexity, and linearity of summation to obtain a looser bound via

with each of the 2 · | Y 2+2 | optimization problems being constrained by ∀ k, l, t, u : d Y (ˆ y ( k ) t , ˆ y ( l ) u ) ≤ d Y ( y ( k ) t , y ( l ) u ) and ˆ y (1) ∈ Y 2 , ˆ y (2) ∈ Y 2 .

Next, we can further loosen this bound by dropping all constraints involving y (2) 1 and y (2) 2 in the first optimization problem. We can also drop all constraints involving y (1) 1 and y (2) 1 in the second optimization problem. Thus, by definition of the group privacy profile, we have

Note that the first cost function term does not depend on y (2) 1 nor y (2) 2 , and the second cost function term does not depend on y (1) 1 nor y (2) 1 . We can thus marginalize out these variables (recall Eq. (16) and Eq. (20)) to conclude our proof.

and for any y (2) 2 we have

The same argument also applies to the general problem setting defined in Appendix D: Given two couplings Γ , Γ ′ , we can define a product coupling ˜ Γ ∝ Γ · Γ ′ , invoke Theorem 3.4, and then apply (advanced) joint convexity to pessimistically upper-bound the guarantee that would be obtained through our proposed framework.

G.3 Deficiencies of mechanism-agnostic ADP bounds.

Following our discussion, we can identify three potential sources for looseness in this approach for deriving mechanism-agnostic bounds. Firstly, it only uses a binary partitioning of subsampling pmfs s x and s x ′ . The resultant mechanism-specific guarantee only depends on divergences between two-component mixtures, which may not be sufficient to tightly bound the overall divergence in complicated scenarios like group privacy. Secondly, it neglects the pairwise distances of y (1) 1 and y (2) 2 , resulting in potentially very large values of the cost function. Thirdly, the bound might be further loosened by applying joint convexity once more.

H Recovering known mechanism-agnostic RDP guarantees

In the following, we demonstrate that existing amplification by subsampling guarantees for Rényi-DP can be derived by instantiating our proposed framework (see Fig. 2). Specifically, we demonstrate that these guarantees can be derived via the procedure discussed in Section 3.3 and shown in Figs. 2b and 2c: (1) Conditioning on at most 4 events indicating the presence of inserted / deleted / substituted elements, (2) defining a simultaneous coupling, and (3) using joint convexity to upper-bound the resultant mechanism-specific guarantee by component divergences.

These guarantees are mechanism-agnostic in the sense that they express the subsampled mechanism's privacy parameters ( α, ρ ) as a function of the base mechanism's privacy parameters. We derive guarantees for the general measure-theoretic problem setting introduced in Appendix D, where the base mechanism can be either discrete or continuous.

The reader may notice that parts of the proofs in Appendices H.1 and H.2 are very similar to those in [28, 30], safe for the discussion of couplings and d Y -compatibility. That is precisely the point: There is an optimal transport problem that implicitly underlies results from prior work, which we have identified and can now generalize to more challenging scenarios like group privacy amplification.

For this section, recall that ∆ α is not the Rényi divergence, but its α th moment, i.e., a scaled and exponentiated Rényi divergence (see Definition D.8).

H.1 Subsampling without replacement and substitution

For subsampling without replacement and substitution relation ≃ ∆ we first show a more general result Theorem H.1. We then demonstrate that it can be upper-bounded via joint convexity of exponentiated Rényi divergence ∆ α to recover the guarantee from [28].

Theorem H.1. Let M = B ◦ S be a subsampled mechanism, where S is subsampling without replacement with batch size q . Let ≃ Y be the substitution relation ≃ ∆ , Y . Then, for α > 1 and all x ≃ ∆ , X x ′ of size N ,

subject to d ∆ , Y ( y (1) 1 , y (1) 2 ) ≤ 1 , d ∆ , Y ( y (1) 1 , y (2) 2 ) ≤ 1 , d ∆ , Y ( y (1) 2 , y (2) 2 ) ≤ 1 , y (1) 1 = y (2) 1 , and with w = q / N .

Proof. Consider arbitrary x ≃ ∆ , X x ′ . By definition of ≃ ∆ , there must be some a ∈ x , a ′ ∈ x ′ such that x ′ = x \ { a } ∪ { a ′ } . We thus define both A 1 and E 1 from Theorem E.2 to be the event that neither a nor a ′ is sampled, i.e., A 1 = E 1 = { y ∈ Y | y ∩ { a, a ′ } = ∅ } . We further define A 2 and E 2 to be the event that a or a ′ is sampled, i.e., A 2 = A 1 and E 2 = E 1 .

By definition of subsampling without replacement, we have

which corresponds to the weights (1 -w ) and w , respectively. We further have

and

Coupling. We now define a coupling γ : Y 2+2 → R + that corresponds to the following generative process: We first generate y (1) 1 by sampling a batch that does not contain a uniformly at random from

x. We then let y (2) 1 ← y (1) 1 Finally, we pick a random element ˜ a of y (1) 1 and replace it with a and a ′ to generate y (1) 2 and y (2) 2 , respectively. More formally:

Condition H.2. A tuple y (1) ∈ Y 2 , y (2) ∈ Y 2 fulfills this condition when y (2) 1 = y (1) 1 and ∃ ˜ a ∈ y (1) 1 : ( y (1) 2 = y (1) 1 \ { ˜ a } ∪ { a } ∧ y (2) 2 = y (1) 1 \ { ˜ a } ∪ { a ′ } ) .

Validity. We now show that this constitutes a valid coupling. Consider y (1) 1 . If and only if s x ( y (1) 1 | A 1 ) > 0 , there are exactly q combinations of y (1) 2 , y (2) 1 , y (2) 2 for which γ ( y ) is non-zero. Thus,

The proof for y (2) 1 is analogous.

Next, consider y (1) 2 . If and only if s x ( y (1) 2 | A 1 ) > 0 , there are exactly | x | -q elements that could have been replaced by a to generate y (1) 2 from y (1) 2 . Specifically, these elements are all elements that do not appear in y (1) 2 . We thus have

The proof for y (2) 2 is analogous.

Compatibility. Finally, we show that γ is a d Y -compatible coupling (see Appendix F). Whenever γ ( y ) > 0 , then

Similarly, the pairwise distances between all y t , y u with u > t > 1 are identical to the distance of their respective supports: The batches have a distance of 1 because one can transform one into another using a single substitution. The supports have a distance of 1 because one can transition from one to another using a single substitution.

The result then immediately follows from Corollary F.7.

Next, we can derive the upper bound from [28]. For this derivation, we will use the following Lemma, which is proven in Appendix B of [6]:

Lemma H.3. Consider two probability measures P, Q on output measure space ( Z , Z , λ ) . Define p = d P / d λ and q = d Q/ d λ . Then

The following proof essentially follows that of [28], but skips their Appendix B.2, since we have already successfully decomposed mixtures m x and m x ′ into small terms that only involve base mechanism densities.

Proposition H.4 (Wang et al. [28]) . Let M = B ◦ S be a subsampled mechanism, where S is subsampling without replacement with batch size q . Let ≃ Y be the substitution relation ≃ ∆ , Y . Then, for α > 1 and all x ≃ ∆ , X x ′ of size N , ∆ α ( m x || m x ′ ) is l.e.q.

with w = q / N .

Proof. Using the constraint y (1) 1 = y (2) 1 in Theorem H.1, we can rewrite its objective as

Using Lemma H.3, we can upper-bound its optimal value via

where each of the α +1 optimization problems is independently constrained by d Y ( y (1) 1 , y (1) 2 ) ≤ 1 , d Y ( y (1) 1 , y (2) 2 ) ≤ 1 , and d Y ( y (1) 2 , y (2) 2 ) ≤ 1 .

Next, we bound the optimal value of each of the α + 1 optimization problems. Using the joint convexity of x, y ↦→ x l · y 1 -l , which implies convexity in the second component, shows that

where all optimization problems are independent, with each one being independently constrained by d Y ( y (1) 1 , y (1) 2 ) ≤ 1 , d Y ( y (1) 1 , y (2) 2 ) ≤ 1 , and d Y ( y (1) 2 , y (2) 2 ) ≤ 1 . Note that, for the last inequality, we replaced a y (2) 2 in the second maximization with a y (1) 1 , which essentially adds a degree of freedom and thus leads to an upper bound.

In [28], the optimal value of the final problem is referred to as the ternary-| χ | l -divergence of b 2 , b y (2) 2 , and b y (1) 1 . As shown in their Lemma 19, it can be upper bounded via 2 · max y ≃ ∆ , Y y ′ ∆ l ( b y || b y ′ ) , which concludes our proof.

Additional terms. Note that [28] derive three additional bounds (see their Lemma 17, Lemma 19, and Theorem 27) on the the ternary-| χ | l -divergence. This introduces additional terms, but does not change the fact that their result is lower-bounded by Theorem H.1. We use their full theorem as a baseline in our experiments.

H.2 Poisson subsampling and insertion/removal

Next, we show that the Poisson subsampling guarantees in [30, 7] follow from another optimal transport problem. For our proof, we will use the following Lemma, which corresponds to the 'novel alternative decomposition' in Appendix A.1 of [30].

Lemma H.5. Consider K +1 ∈ N distributions P, Q 1 , . . . , Q K on output measure space ( Z , Z , λ ) , and define p = d P / d λ , q k = d Q k / d λ . Further consider some w 1 , . . . , w K ∈ [0 , 1] with ∑ K k =1 w k = 1 . Then,

Proof. Based on the definition of ∆ α , we have

The result then follows from joint convexity of ∆ α .

Note that the proof of this lemma is very similar to the proof strategy we used in deriving Proposition H.4 using Lemma H.3: We add 0 = c -c with some c to the numerator (which leads to the binomial expansion in Lemma H.3) and then apply joint convexity to obtain an upper bound.

Proposition H.6 (Zhu and Wang [30]) . Let M = B ◦ S be a subsampled mechanism, where S is Poisson subsampling with rate r . Let ≃ Y be the insertion/removal relation ≃ ± , Y . Then, for α > 1 and all x ≃ ± , X x ′ , ∆ α ( m x || m x ′ ) is l.e.q.

Proof. Since we are concerned with insertion/removal, we need to consider two cases:

Case 1: Removal. In this case, there is some a ∈ x such that x ′ = x \ { a } . We let A 1 be the event that a is not sampled, i.e. A 1 = { y ∈ Y | a / ∈ y } , and let A 2 = A 1 . We let E 1 = Y , i.e., do not condition on any particular event.

By definition of Poisson subsampling, we have P S x ( A 1 ) = 1 -r , P S x ( A 2 ) = r , and P S x ′ ( E 1 ) = 1 . We further have

and

Note that s x ( y | A 1 ) = s x ′ ( y | E 1 ) .

Coupling. We now define a coupling γ : Y 2+1 → R + that corresponds to the following generative process: We first generate y (1) 1 by sampling a batch that does not contain a via Poisson subsampling from x \ { a } . We then let y (2) 1 ← y (1) 1 . Finally, we deterministically insert a to generate y (1) 2 . This can be formally defined via

Validity. We can verify the validity of this coupling as follows: For every y (1) 1 with s x ( y (1) 1 | A 1 ) > 0 , there is exactly one combination y (1) 2 , y (2) 1 for which γ ( y ) > 0 , namely y (2) 1 = y (1) 1 and y (1) 2 = y (1) 1 ∪ { a } . We thus have ∑ y (1) 2 ,y (2) 1 ∈ Y 2 γ ( y (1) 1 , y (1) 2 , y (2) 1 ) = s x ( y (1) 1 | A 1 ) . The proof for y (2) 1 is analogous. For every y (1) 2 , we have exactly one combination of y (1) 1 , y (2) 1 for which γ ( y ) > 0 , namely y (1) 1 = y (2) 1 = y (1) 2 \ { a } . We thus have

Compatibility. Finally, it can be easily shown that γ is a d Y -compatible coupling (see Appendix F). Whenever γ ( y ) > 0 , then

Similarly, the pairwise distance between y (1) 2 and y (2) 1 is always 1 . This is identical to the distance of their supports, because one can always transition between them via a single insertion/removal.

It thus follows from Corollary F.7 that

Because y (1) 1 = y (2) 1 and because d ± , Y ( y (1) 1 , y (1) 2 ) ≤ 1 is equivalent to y (1) 1 ≃ ± , Y y (1) 2 , this bound can be restated as

Finally, one can use the definition of ∆ α and binomial expansion to show

Note that this is smaller than the term in Proposition H.6 by a factor of 2 .

Case 2: Insertion In this case, there is some a ∈ x ′ such that x = x ′ \ { a } . Very similar to before, we let A 1 = Y , i.e., we do not condition on any particular event. We further let E 1 be the event that a is not sampled, i.e. E 1 = { y ∈ Y | a / ∈ y } , and let E 2 = E 1 . We notice that we have exactly the same conditional distributions as in the first case. We can thus define exactly the same coupling (up to changes in indexing) to show that

Next, applying Lemma H.5 and using the definition of ∆ α shows that

This corresponds exactly to Eq. 6 of the 'novel alternative decomposition' in Appendix A.1 of [30]. One can then through the remaining steps in their Appendix A to show that this term is at most two times larger than the one we derived in the deletion case.

As already mentioned at the beginning of this section, the proof is very similar to that in [30], except for the discussion of couplings and d Y -compatibility. We emphasize that the crucial aspect, which is not discussed in any prior work on Rényi-DP amplification, is that there is an implicit, underlying optimal transport problem between conditional subsampling distributions. Now that we have identified this connection, we can apply this more general optimal transport principle to a much broader range of amplification by subsampling scenarios for Rényi-DP.

Further tightening. The factor 2 in Proposition H.6 can be eliminated for distributions with particular symmetries (see Theorem 5 in [29]) or bounded Pearson-Vajda χ l -pseudo-divergence (see Theorem 8 in [30]). We thus do not include the factor 2 when using this amplification guarantee as a baseline in our experiments.

H.3 Graph subsampling without replacement and node modification

Daigavane et al. [40] consider a setting that differes from the usual insertional/removal into datasets: Node-level privacy for graphs. There, the dataset space is the set of all directed, attributed graphs X = ⋃ N,D ∈ N R N × D × { 0 , 1 } N × N , which are composed of a continuous feature matrix and a discrete adjacency matrix. Two graphs x, x ′ are related by dataset relation ≃ X if x ′ can be constructed by inserting a node (including new edges) into x , or removing a node (including its edges) from x .

To analyze this problem setting, they perform a preprocessing step (see their Algorithm 2) to represent each graph by a set of subgraphs, with each subgraph corresponding to a node in the graph. Via this construction, they return to the traditional setting where X ⊆ P ( A ) for some set A , and the modification of a node's features and edges corresponds to the substitution of K elements, 4 i.e.

where K depends on the maximum considered graph degree.

Irrespective of the graph neural network specific details, this discussion suggests that group privacy and differentially private learning for graphs are related.

We shall now demonstrate that their result for subsampling without replacement can be derived from Theorem E.1, i.e. optimal transport without conditioning.

Proposition H.7 (Daigavane et al. [40]) . Let M = B ◦ S be a subsampled mechanism, where S is subsampling without replacement with batch size q . Let ≃ Y be the substitution relation ≃ ∆ , Y . Let

≃ K ∆ , X be the K-fold substitution relation defined in Eq. (22) . Then, for α > 1 and all x ≃ K ∆ , X x ′ of size N ,

with w k = HyperGeom( k | N,K,q ) .

Proof. Consider arbitrary x ≃ ∆ , X x ′ . By definition of ≃ ∆ , there must be some g ∈ x \ x ′ , g ′ ∈ x ′ \ x such that x ′ = x \ g ∪ g ′ and | g | = | g ′ | = K .

We condition on the events A 1 = Y and E 1 = Y , meaning

Coupling. We now define a coupling via γ : Y 1+1 → R + that define the following generative process: We first define y (1) by sampling a batch of size q from x without replacement. We then remove all elements that are in g and insert an equal number of elements, which are chosen uniformly at random from g ′ . This coupling can be formally defined as

Condition H.8. A tuple y (1) ∈ Y , y (2) ∈ Y fulfills this condition when there exists a ˜ g ⊆ g such that y (2) = y (1) \ g ∪ ˜ g and | y (2) ∩ ˜ g | = | y (1) ∩ g | .

Validity. We now show that this constitutes a valid coupling. Consider any y (1) with s x ( y (1) | A 1 ) > 0 and | y (1) ∩ g | = k . There are exactly ( K k ) different y (2) for which γ ( y ) is non-zero, and each one has the same probability. Thus,

The other case is analogous.

Compatibility. Evidently, this is a d Y -compatible coupling, because (a) whenever y (1) contains k elements from g , it has a distance of k from s x ′ (y | E 1 ) and (b) whenever y (1) contains k elements from g , we generate y (2) by substituting exactly k elements.

It thus follows from Corollary F.7 that

with

The result then follows from the definition of induced distance, the definition of the K-fold substitution relation ≃ K ∆ , X and the fact that | y (1) ∩ g | is a random variable with distribution HyperGeom( N,K,q ) .

Note that [40] instantiate this result with Gaussian mechanisms with sensitivity C , for which max y ≃ k ∆ , Y y ′ ∆ α ( b y || b y ′ ) = exp( α · ( α -1) · k 2 C 2 σ 2 ) .

In Appendix B.1.4, we experimentally demonstrate that this result can improve upon the baseline of combining [28] with the traditional group privacy property, but is not sufficiently tight to consistently outperform it across a wide range of parameters. This emphasizes the benefit of considering optimal transport between proper conditional distributions when deriving amplification guarantees.

I Novel RDP guarantees

Now that we have a framework that enables generic subsampling analysis for Rényi differential privacy, we can derive a variety of novel guarantees that were previously only available for approximate differential privacy.

In the following, we first demonstrate that RDP guarantees for a dataset relation ≃ X can be derived from a base mechanism that is only known to be RDP under a different batch relation ≃ Y , i.e., 'hybrid neighboring relations' [15]. We then show that we can derive RDP guarantees for nonstandard combinations of subsampling schemes and neighboring relations, such as subsampling without replacement and insertion/removal. Finally, we derive a simple, tight mechanism-specific subsampling guarantee for randomized response. We use this guarantee in Section 4 to demonstrate the limitations of using mechanism-agnostic RDP bounds instead of mechanism-specific RDP bounds.

For this section, recall again that Λ α is not the Rényi divergence, but its α th moment, i.e., a scaled and exponentiated Rényi divergence (see Definition D.8).

I.1 Hybrid neighboring relations

Hybrid neighboring relations have thus far not been discussed in the context of Rényi-DP. Analyzing such scenarios may be particularly useful when the dataset space X and the batch space ( Y , Y ) are different from each other, e.g., when mapping from large text corpora to short sequences of token embeddings. Note that hybrid relations may also be useful for noisy stochastic gradient descent with fixed batch sizes (see end of Appendix I.2).

As an example, we consider the following scenario: Subsampling without replacement, with substitution relation ≃ ∆ , X for datasets and insertion/removal relation ≃ ± , Y for batches. The proof is largely identical to that of Theorem H.1, but uses the fact that a substitution can be represented by an insertion, followed by a removal.

Theorem I.1. Let M = B ◦ S be a subsampled mechanism, where S is subsampling without replacement with batch size q . Let ≃ Y be the insertion/removal relation ≃ ± , Y and ≃ X be the substitution relation ≃ ∆ , X . Then, for α > 1 and all x ≃ ∆ , X x ′ of size N ,

with y ∈ Y 2+2 subject to y (1) 1 = y (2) 1 , d ± , Y ( y (1) 1 , y (1) 2 ) ≤ 2 , d ± , Y ( y (1) 1 , y (2) 2 ) ≤ 2 , d ± , Y ( y (1) 2 , y (2) 2 ) ≤ 2 , and with w = q / N .

Proof. Consider arbitrary x ≃ ∆ , X x ′ . By definition of ≃ ∆ , there must be some a ∈ x , a ′ ∈ x ′ such that x ′ = x \ { a } ∪ { a ′ } . We thus define both A 1 and E 1 from Theorem E.2 to be the event that neither a nor a ′ is sampled, i.e., A 1 = E 1 = { y ∈ Y | y ∩ { a, a ′ } = ∅ } . We further define A 2 and E 2 to be the event that a or a ′ is sampled, i.e., A 2 = A 1 and E 2 = E 1 .

By definition of subsampling without replacement, we have

which corresponds to the weights (1 -w ) and w , respectively.

Coupling. We now define a coupling via γ : Y 4 → R + :

Condition H.2. A tuple y (1) ∈ Y 2 , y (2) ∈ Y 2 fulfills this condition when y (2) 1 = y (1) 1 and ∃ ˜ a ∈ y (1) 1 : ( y (1) 2 = y (1) 1 \ { ˜ a } ∪ { a } ∧ y (2) 2 = y (1) 1 \ { ˜ a } ∪ { a ′ } ) .

As explained before, γ defines the following generative process: We first generate y (1) 1 by sampling a batch that does not contain a uniformly at random from x. We then let y (2) 1 ← y (1) 1 Finally, we pick a random element ˜ a of y (1) 1 and replace it with a and a ′ to generate y (1) 2 and y (2) 2 , respectively. Note that each of these substitution corresponds to exactly one insertion and one removal.

Validity. The validity of this coupling has already been demonstrated in our proof of Theorem H.1, since validity of a coupling does not depend on batch neighboring relation ≃ Y .

Compatibility. We can thus focus on showing that γ is a d Y -compatible coupling (see Appendix F). Whenever γ ( y ) > 0 , then

We see that the pairwise distances between all y t , y u with u > t > 1 are identical to the distance of their respective supports: The batches have a distance of 2 because one can transform one into another using a single substitution, i.e. an insertion and a removal. The supports also have a distance of 2 because one can transition from one to another using a single substitution.

The result then immediately follows from Corollary F.7.

If we wanted to express this result in terms of the Rényi-DP parameters of the base mechanism, we could go through the same derivations as in our proof of Proposition H.4 to show that

with w = q / N . The inner term can, for instance, be evaluated using the traditional group privacy property from [31].

I.2 Subsampling without replacement under insertion/removal

To demonstrate that our framework is not limited to analyzing the usually considered combination of subsampling without replacement and substitution or Poisson subsampling and insertion/removal, we consider the following non-standard combination: Subsampling without replacement and insertion/removal.

We show that we can derive a guarantee that is qualitatively similar to that of [30], while preserving a fixed batch size. Such results could be useful when implementing noisy stochastic gradient descent in deep learning frameworks with static computation graphs, where the variable batch sizes resulting from Poisson subsampling may be problematic. However, note that this guarantee only guarantees privacy for spaces of datasets whose elements are larger than some Q ∈ N with Q > q , where q is the batch size.

For the following proof, we condition on the same events as in our proof of Proposition H.6, but need to construct a different d Y -compatible coupling, since we have a different subsampling distribution.

Theorem I.2. Assume a dataset space and batch space defined by X ⊆ { x ∈ P ( A ) | | x | > Q } , Y = { y ⊆ x | x ∈ X , | y | = q } , Y = P ( Y ) and finite set A , with Q,q ∈ N and Q > q . Let M = B ◦ S be a subsampled mechanism, where S is subsampling without replacement with batch size q r . Let ≃ Y be the insertion/removal relation ≃ ± , Y . Then, for α > 1 and all x ≃ ± , X x ′ ,

subject to N > Q and with w = q / N .

Proof. Like in our proof of Proposition H.6, we need to distinguish between insertion and removal.

Case 1: Removal. In this case, there is some a ∈ x such that x ′ = x \ { a } . We define N = | x ′ | , which fulfills N > Q by definition of dataset space X .

We let A 1 be the event that a is not sampled, i.e. A 1 = { y ∈ Y | a / ∈ y } , and let A 2 = A 1 . We let E 1 = Y , i.e., do not condition on any particular event.

By definition of subsampling without replacement, we have P S x ( A 1 ) = 1 -qN , P S x ( A 2 ) = q / N , and P S x ′ ( E 1 ) = 1 . We further have

and

.

Note that s x ( y | A 1 ) = s x ′ ( y | E 1 ) and that all three conditional distribution are subsampling without replacement from sets of size N , not of size N +1 or N -1 . Further note that, for event A 1 , we only need to sample q -1 elements, since one element is always fixed to be a .

Coupling. We now define a coupling via γ : Y 2+1 → R + :

Simply put, γ defines the following generative process: We first generate y (1) 1 by sampling a batch that does not contain a via subsampling without replacement from x \ { a } . We then let y (2) 1 ← y (1) 1 . We then randomly replace one of the q batch elements with a to generate y (1) 2 .

Validity. We can verify the validity of this coupling as follows: For every y (1) 1 with s x ( y (1) 1 | A 1 ) > 0 , there are exactly q combination y (1) 2 , y (2) 1 for which γ ( y ) > 0 . We thus have ∑ y (1) 2 ,y (2) 1 ∈ Y 2 γ ( y (1) 1 , y (1) 2 , y (2) 1 ) = s x ( y (1) 1 | A 1 ) . The proof for y (2) 1 is analogous.

For every y (1) 2 , there are exactly N -( q -1) combinations of y (1) 1 , y (2) 1 for which γ ( y ) > 0 . This is because a must have replaced one of the N -( q -1) elements of x ′ that are not in y (2) 1 .

We thus have

Compatibility. Finally, it can be easily shown that γ is a d Y -compatible coupling. Whenever γ ( y ) > 0 , then

Similarly, the pairwise distance between y (1) 2 and y (2) 1 is always 2 . This is identical to the distance of their supports, because one can always transition between them via a substitution, i.e., an insertion and a removal.

It thus follows from Corollary F.7 that

As in our proof of Proposition H.6 one can use the definition of Λ α and binomial expansion to finally show that

with w = q / N . Note that this is smaller than the term in Proposition H.6 by a factor of 2 .

Case 2: Insertion In this case, there is some a ∈ x ′ such that x = x ′ \ { a } . We can thus define the same events and the same coupling in a symmetric manner to show

subject to d Y ( y, y ′ ) ≤ 2 . with w = q / N . Again, this corresponds exactly to Eq. 6 of the 'novel alternative decomposition' in Appendix A.1 of [30]. One can then through the remaining steps in their Appendix A to show that this term is at most two times larger than the one we derived in the deletion case.

Finally, since this bound depends on N , and we want the guarantee to hold for all x ≃ ± , X x ′ , we need to determine the N that maximizes the bound.

Hybrid Relations. Note that we could have also gone through the above derivations with batch substitution relation ≃ ∆ , Y . Then, each substitution would correspond to an actual substitution, instead of a pair of insertion and deletion, and we would obtain

with w = q / N .

I.3 Tight mechanism-specific subsampling without replacement for randomized response

As mentioned earlier, the following result is meant as a simple example to demonstrate the benefit of using mechanism-specific over mechanism-agnostic RDP bounds in Section 4. For the following discussion, recall Theorem H.1, which we proved in the previous section.

Theorem H.1. Let M = B ◦ S be a subsampled mechanism, where S is subsampling without replacement with batch size q . Let ≃ Y be the substitution relation ≃ ∆ , Y . Then, for α > 1 and all x ≃ ∆ , X x ′ of size N ,

subject to d ∆ , Y ( y (1) 1 , y (1) 2 ) ≤ 1 , d ∆ , Y ( y (1) 1 , y (2) 2 ) ≤ 1 , d ∆ , Y ( y (1) 2 , y (2) 2 ) ≤ 1 , y (1) 1 = y (2) 1 , and with w = q / N .

We shall now solve Theorem H.1 for randomized response and prove the tightness of the resultant guarantee.

Theorem I.3. Let B be the randomized response mechanism | h -(1 -V ) | with h : Y →{ 0 , 1 } , V ∼ Bern( θ ) , and true response probability θ ∈ [0 , 1] . Let S be subsampling without replacement with batch size q . Then, for all x ≃ ∆ , X x ′ of size N , Λ α ( m x || m x ′ ) is l.e.q.

subject to τ ∈ { θ, 1 -θ } and with w = q / N .

Proof. Because we have no further information, we must consider the worst possible underlying function h : Y →{ 0 , 1 } .

Due to the last constraint in Theorem H.1, we must have h ( y (1) 1 ) = h ( y (2) 1 ) . Due to symmetry in Λ α (we are summing over z ∈ { 0 , 1 } ), we can assume w.l.o.g. that h ( y (1) 1 ) = h ( y (2) 1 ) = 1 and thus b y (1) 1 ( z ) = b y (1) 1 ( z ) = Bern( z | θ ) .

Because the batches y (1) 2 and y (1) 2 can differ from y (1) 1 , we can choose the corresponding function values h ( y (1) 2 ) and h ( y (2) 2 ) arbitrarily. To make Λ α greater than 1 , the two mixtures must be different from each other. Thus, we must choose either h ( y (1) 2 ) = 1 and h ( y (2) 2 ) = 0 , or h ( y (1) 2 ) = 0 and h ( y (2) 2 ) = 1 . This corresponds to b y (1) 2 ( z ) = Bern( z | τ ) and b y (2) 2 ( z ) = Bern( z | 1 -τ ) with τ ∈ { θ, 1 -θ } . Maximizing over both options yields our result.

Note that this guarantee can be evaluated in O (1) : We need to iterate over two possible values of τ , and evaluating the corresponding Λ α requires summing over two values z ∈ { 0 , 1 } .

Next, we prove the mechanism-specific tightness of this result, meaning it is not possible to derive stronger guarantees without additional information about dataset space X and the function h underlying the randomized response mechanism.

Theorem I.4. Let S be subsampling without replacement with arbitrary batch size q ∈ N and θ ∈ [0 , 1] be some true response probability. There exists a dataset space X and a batch space ( Y , Y ) fulfilling the constraints in Definition D.5, as well a pair of datasets x ≃ ∆ , X of size N , and a function h : Y →{ 0 , 1 } , such that the corresponding subsampled randomized response mechanism M = B ◦ S fulfills

subject to τ ∈ { θ, 1 -θ } and with w = q / N .

Proof. Let X = { x ⊆ N | | x | > q } . Consider an arbitrary x ∈ X and select arbitrary elements a ∈ x , a ′ ∈ N \ x . Define x ′ = x \ { a } ∪ { a ′ } .

Assume w.l.o.g. that the divergence is maximized by τ = θ . We now construct an indicator function h : Y →{ 0 , 1 } for a and a ′ that leads to the largest possible divergence.

By construction of h , the corresponding base mechanism pmf is

Under the distribution of S ( x ) , the first case occurs with probability 1 -w and the second case occurs with probability w . Under the distribution of S ( x ) , the first case occurs with probability 1 -w and the second case occurs with probability w . We thus have

which exactly attains the desired divergence when τ = θ is the optimal value.

J From mechanism-specific guarantees to dominating pairs

Our proposed optimal transport approach lets us derive mechanism-specific guarantees, which upperbound the hockey stick divergence between the distribution of M ( x ) and M ( x ′ ) with x ≃ X x ′ via a weighted sum 5 of mixture divergences, i.e.,

Here, the w ( x,x ′ ) α,k are weights with ∑ K k =1 w ( x,x ′ ) α,k = 1 , which depend on the chosen coupling γ and are indexed by α ≥ 0 , 1 ≤ k ≤ K , and x, x ′ ∈ X . The p ( x,x ′ ) α,k and p ( x,x ′ ) α,K are densities of mixture distributions P ( x,x ′ ) α,k and q ( x,x ′ ) α,K on the output space R D , 6 which are also indexed by α ≥ 0 , 1 ≤ k ≤ K , and x, x ′ ∈ X .

Our goal is to construct dominating pairs from such bounds, so that we can then use them for privacy accounting. For this discussion, recall the definition of dominating pairs:

Definition 2.3. Apair of distributions ( P, Q ) with densities ( p, q ) is a dominating pair for mechanism M under neighboring relation ≃ X , if H α ( m x || m x ′ ) ≤ H α ( p || q ) for all x ≃ X x ′ and all α ≥ 0 .

If the densities on the r.h.s. of Eq. (23) are constant in α , k , x , and x ′ , i.e.., p ( x,x ′ ) α,k = ˜ p and q ( x,x ′ ) α,k = ˜ q with some densitities ˜ p , ˜ q , then we can immediately identify that the corresponding distributions ˜ P and ˜ Q are a dominating pair. For instance, in Section 3.4 we could immediately determine that the two Gaussian mixtures in Theorem 3.8 are a dominating pair for Poisson subsampling and the 'insertK + -removeK -' relation.

However, this is generally not the case. In the following, we discuss a three-step procedure that let us (1) reduce the weighted sum of divergences in Eq. (23) to a single divergence (2) resolve non-constancy in the dataset pairs ( x, x ′ ) , and (3) resolve non-constancy in the divergence order α .

Note that, depending on the considered setting, it may be possible to skip one or multiple of these steps (like in our group privacy example).

J.1 Step 1: Eliminating weighted sums

Consider some fixed order α and fixed datasets x ≃ X x ′ . Let us thus omit the corresponding indexes. In this step, we construct a pair of distributions ˜ P , ˜ Q with densitities ˜ p and ˜ q such that

To achieve this, we notice that there is no requirement for dominating pairs to be distributions on the same space R D . Taking inspiration from hierarchical randomized smoothing [69], we thus construct ˜ p and ˜ q that are mixtures of the p k and q k and additionally release their component indices:

Lemma J.1. Consider arbitrary densitities p 1 , . . . , p K , q 1 , . . . , q K : R D → R + and arbitrary weights w 1 , . . . , w K ∈ [0 , 1] with ∑ K k =1 w k = 1 . Further let r : { 1 , . . . , K } → [0 , 1] be the probability mass function of Categorical( w 1 , . . . , w K ) . Define ˜ p, ˜ q : R D ×{ 1 , . . . , K } → R + as ˜ p ( z, k ) = p k ( z ) · r ( k ) and ˜ q ( z, k ) = q k ( z ) · r ( k ) . Then, H α (˜ p || ˜ q ) = ∑ K k =1 w k · H α ( p k || q k ) .

Proof. By the definition of hockey stick divergence and ˜ p, ˜ q , we have

Note that one could equivalently construct continuous distributions ˜ P and ˜ Q by defining the categorical distribution as a transformation of a uniform distribution. Further note that the distribution of privacy loss random variable log(˜ p ( Z ) / ˜ q ) with Z ∼ ˜ P is simply a weighted sum of the components' privacy loss distributions, which can be easily shown via law of total probability. This makes this construction amenable to numerical privacy accounting.

J.2 Step 2: Resolving non-constancy in dataset pairs

After applying the construction from the previous step, we have bounds of the form H α ( m x || m x ′ ) ≤ H α (˜ p ( x,x ′ ) α || ˜ q ( x,x ′ ) α ) . If these bounds are not constant in the datasets x, x ′ ∈ X , we cannot simply read off a dominating pair.

However, the ultimate goal behind identifying dominating pairs is to use them in a privacy accounting method to algorithmically derive guarantees for all possible pair x ≃ X x ′ under composition. Providing guarantees for all possible pair does not require that we derive guarantees for all pairs simultaneously. To formalize this, recall that the neighboring relation ≃ X is a set of tuples from X 2 .

Lemma J.2. Consider any α ≥ 0 , mechanism M and L different neighboring relations ≃ (1) , . . . , ≃ ( L ) ⊆ X 2 . Assume that there are L constants c (1) , . . . , c ( L ) such that H α ( m x || m x ′ ) ≤ c ( l ) for all l ∈ { 1 , . . . , L } and all x ≃ ( l ) x ′ . If the neighboring relations partition ≃ X , i.e., ≃ X = ⋃ L l =1 ≃ ( l ) , then

Proof. Consider any x ≃ X x ′ . Because the neighboring relations partition ≃ X , there must be some l ∗ such that x ≃ ( l ∗ ) x ′ and thus H α ( m x || m x ′ ) ≤ c ( l ∗ ) ≤ max l c ( l ) .

For the purposes of privacy accounting, it is thus sufficient to determine some sufficiently fine-grained partition ≃ (1) , . . . , ≃ ( L ) such that for all l ∈ { 1 , . . . , L } and all α ≥ 0

where ˆ p l α and ˆ q ( l ) α are families of densities indexed by partition index l and divergence order α . One can then perform privacy accounting independently for each of the relations.

This goal can always be achieved by having one atomic relation per possible pair of neighboring elements. In practice, however, much more coarse-grained partitions may be sufficient. Zhu et al. [10] applied this ansatz to the insertion/removal relation, by deriving dominating pairs for insertion and removal separately. In our work, we partition the group insertion/removal relation for groups of size K into K +1 different 'insertK + -removeK -' relations with K + + K -= K .

J.3 Step 3: Resolving non-constancy in divergence order

Consider some fixed partition index l ∈ { 1 , . . . , L } , which we omit in the following. We are now left with a bound of the form ∀ x ≃ x ′ : H α ( m x || m x ′ ) ≤ H α (ˆ p α || ˆ q α ) . If the families of densitities ˆ p α , ˆ q α is constant in α , i.e.

with some ˇ p, ˇ q , then the corresponding distributions ˇ P, ˇ Q are dominating pairs. Importantly, the numeric value of the bound can vary with α . We just want the numeric value to be identical to the divergence H α of two specific distributions for all α .

If this is not the case, there are two options to construct a dominating pair. The first option uses a characterization of dominating pairs as a function of privacy profiles. It was applied by Zhu et al. [10] to determine dominating pairs for subsampling without replacement and the substitution relation. The second option is enabled by our novel optimal transport and constrained optimization perspective on subsampling.

J.3.1 Option 1: Convex conjugation of privacy profiles

The following result from [10] establishes a correspondence between privacy profiles and dominating pairs, as well as providing a formula for constructing dominating pairs from privacy profiles:

Proposition J.3 (Zhu et al. [10]) . For a given f : R + → R , there exists ˇ P, ˇ Q such that ∀ α ≥ 0 : H ( α ) = H α ( P, Q ) if and only if f ∈ F where

Moreoever, one can explicitly construct such P and Q : P has CDF 1 + f ∗ ( x -1) in [0 , 1) and Q = Uniform([0 , 1]) , where f ∗ is the convex conjugate of f .

Next, we show how to construct a function f ( α ) ∈ F from our families of densities ˆ q α , ˆ q α indexed by α that allows us to determine dominating pairs.

Lemma J.4. Consider two families of densities ˆ p α , ˆ q α indexed by α ≥ 0 such that ∀ x ≃ x ′ : H α ( m x || m x ′ ) ≤ H α (ˆ p α || ˆ q α ) . Define the function f : R + → R with

Then f is a valid privacy profile, i.e., f ∈ F with H defined in Eq. (24) .

Proof. For every β ≥ 0 , let f β ( α ) = H α (ˆ p β || ˆ q β ) be the privacy profile associated with a specific pair from the considered families. All these functions are elements of F as per the first part of Proposition J.3. We thus know that each f β is convex, decreasing, and has value 1 at α = 0 . Naturally, their maximum is also convex, decreasing, and has value 1 at α = 0 . We further know that, for all β ≥ 0 , f β ( x ) ≥ max { 1 -x, 0 } . Thus, we have f ( x ) ≥ f β ( x ) ≥ (1 -x ) for any β ≥ 0 .

We can then invoke the second part of Proposition J.3 to define our dominating pair.

As mentioned earlier, Zhu et al. [10] applied this principle to subsampling without replacement by taking a maximum over two different pairs of distributions. Lemma J.4 generalizes this principle to larger families.

J.3.2 Option 2: Relaxation of cost function constraints

Anovel solution is enabled by our proposed framework for deriving mechanism-specific subsampling guarantees: Recall that the densitities ˆ p α and ˆ q α are the result of identifying worst-case mixture components under pairwise batch distance constraints (Proposition 3.5). In Appendix L, we demonstrate that it is possible to relax some of these constraints to obtain a single pair of densitities that bounds H α ( m x || m x ′ for all α and pairs of datasets x ≃ x ′ . Specifically, we apply this principle to the substitution relation and subsampling without replacement, as well as subsampling with replacement. Due to the relaxation, these dominating pairs are not necessarily tight. They may however be easier to use in practice than convex conjugation, which requires solving an optimization problem.

K Recovering known dominating pairs

In this section, we demonstrate that known dominating pairs for the standard combinations of Poisson subsampling under insertion/removal and subsampling without replacement under substitution, as well as subsampling without replacement under insertion/removal, can also be derived via our proposed optimal transport framework.

As before, our contribution are not the guarantees themselves. Our contribution is identifying that, just like mechanism-agnostic guarantees for ADP and RDP, these guarantees can be derived by defining an (optimal) coupling and then identifying worst-case mixture components under pairwise batch distance constraints.

K.1 Poisson subsampling and insertion/removal

As discussed in Appendix J, it is beneficial to partition the insertion removal ≃ ± , X into two relations: The insertion relation ≃ + , X , where x ≃ + , X x ′ implies that there is some a / ∈ x such that x ′ = x ∪{ a } and the removal relation ≃ -, X , where x ≃ -, X implies that there is some a ∈ x such that x ′ = x \{ a } .

While we could derive dominating pairs for both relations from scratch, we can use the following result from [10] to reduce our workload:

Lemma K.1. If distributions ( P, Q ) are a dominating pair for mechanism M under the insertion relation ≃ + , then ( Q,P ) are a dominating pair under the removal relation ≃ -.

Next, we use our proposed framework to derive the following guarantee in a very natural manner that does not require reasoning about tradeoff functions and optimal testing rules (c.f. proof of Lemma 29 in [10]).

Proposition K.2 (Zhu et al. [10]) . Consider a subsampled mechanism M = B ◦ S , where S is Poisson subsampling with subsampling rate r ∈ [0 , 1] . assume that ( P, Q ) are a dominating pair for base mechanism B under the batch insertion relation ≃ + , Y . Then, ( P, (1 -r ) P + rQ ) are a dominating pair for M under the dataset insertion relation ≃ + , X and ((1 -r ) P + rQ,Q ) are a dominating pair for M under the dataset removal relation ≃ -, X .

Proof. We begin by deriving the dominating pair for the removal relation ≃ -, X . Consider an arbitrary pair x ≃ -x ′ and an arbitrary α ≥ 0 . Recall from our proof of the mechanism-agnostic Poisson subsampling guarantee Proposition H.6 that we can construct a d Y -compatible coupling to show that

The last constraint is because there is no sequence of deletions that will result in an insertion. It can be ignored, meaning our optimization problem is equivalent to

Using the definition of hockey stick divergence, 7 this can be restated as

We can now distinguish two cases, based on the value of α ' = α -(1 -r ) / r .

Case 1: Assume that α ′ < 0 . Then, the integrand is always non-negative, and the objective function evaluates to a constant r (1 -a ′ ) . We can thus choose y, y ′ arbitrarily, because

where p and q are the densities of dominating pair ( P, Q ) .

Case 2: Assume that α ′ ≥ 0 . Because ( P, Q ) is a dominating pair for the batch insertion relation ≃ + , Y , we know from Lemma K.1 that ( Q,P ) is a dominating pair for the batch removal relation ≃ -, Y . Thus

This shows that ((1 -r ) P + rQ,Q ) is a dominating pair for the removal relation ≃ -, X . It then follows from Lemma K.1 that ( Q, (1 -r ) P + rQ ) is a dominating pair for the insertion relation ≃ + , X .

Note that case 1 (but not case 2) of Theorem 11 in [10] states that ((1 -r ) Q + rP, P ) were a dominating pair for the removal relation. This does however appear to be a typographical error, since the authors use the same symmetry argument (Lemma K.1) for their proof.

K.2 Subsampling without replacement and insertion/removal

The proof for the following result is virtually identical to the previous one. They only differ in the coupling which we need to construct, which again highlights the generality and usefulness of our proposed framework.

Proposition K.3 (Zhu et al. [10]) . Assume a dataset space and batch space defined by X ⊆ { x ∈ P ( A ) | | x | ∈ { N,N -1 } , Y = { y ⊆ x | x ∈ X , | y | = q } , and finite set A , with q < N . Consider a subsampled mechanism M = B ◦ S , where S is subsampling without replacement with batch size q , and define batch-to-dataset ratio w = q / N . assume that ( P, Q ) are a dominating pair for base mechanism B under the batch substitution relation ≃ ∆ , Y . Then, ( P, (1 -w ) P + wQ ) are a dominating pair for M under the dataset insertion relation ≃ + , X and ((1 -w ) P + wQ,Q ) are a dominating pair for M under the dataset removal relation ≃ -, X .

Proof. As in the previous proof, we begin with removal relation ≃ -, X . Consider an arbitrary pair x ≃ -x ′ and an arbitrary α ≥ 0 . Recall from our novel proof of the mechanism-agnostic RDP guarantee for subsampling without replacement and insertion/removal (see Theorem I.2) that we can construct a d Y -compatible coupling to show that

subject to y ∈ Y 2+1 , y (1) 1 = y (2) 1 , d Y ( y (1) 2 , y (1) 1 ) ≤ 1 , d Y ( y (1) 1 , y (1) 2 ) ≤ 1 .

By definition of the induced distance, this optimization problem is equivalent to

We can now go through exactly the same steps as in our derivation of Proposition K.2, replacing every occurrence of ' ≃ -' with ' ≃ ′′ ∆ , to conclude our proof.

K.3 Subsampling without replacement and substitution

In the case of subsampling without replacement and substitution (as well as Poisson subsampling and insertion/removal without partitioning of the neighboring relation) we do not need to provide a

new proof. This is because Zhu et al. [10] prove this result by invoking a tight mechanism-agnostic subsampling guarantee derived by Balle et al. [15] and then applying advanced joint convexity in reverse order (see proof of Proposition 30 in [10]).

As we discussed in Appendix G.2 and formalized in Theorem G.2, any guarantee that is derived via the framework from [15] can be equivalently derived by defining a (potentially suboptimal) coupling between multiple conditional subsampling distributions. Thus, we know that the guarantee could have equivalently been derived via our proposed framework

Proposition K.4 (Zhu et al. [10]) . Consider a subsampled mechanism M = B ◦ S , where S is subsampling without replacement with batch size q ∈ N . Assume that ( P, Q ) are a dominating pair for base mechanism B under the batch substitution relation ≃ ∆ , Y . Consider arbitrary x ≃ ∆ , X x ′ of size N and define batch-to-dataset ratio w = q / N . Then

Note that this result does not immediately specify a dominating pair. However, as shown in [10], one can construct a dominating pair via convex conjugation (recall Appendix J.3.1 and Proposition J.3).

L Novel results for dominating pairs

In the following, we formally derive dominating pairs for subsampling with replacement under the substitution relation. We also derive an alternative dominating pair for subsampling without replacement under substitution which does not require convex conjugation (c.f. Appendix K.3). These results are novel in three respects.

Novel contribution 1 - Formal guarantees. Using the dominating pairs we shall shortly derive for privacy accounting was already proposed in [8]. However, the authors did not provide a formal proof for why they should lead to valid privacy guarantees. They only proved that if two pairs of multivariate Gaussians with colinear means were to be a dominating pair 8 , then one could use a change of variable and marginalization to reduce them to a univariate dominating pair (see Appendix B.1 in [8]). They did not prove why this assumption should hold. A formal proof of this assumption is now enabled by the analysis we conducted for solving the group privacy optimization problem in Theorem M.6 (see Appendix O).

Novel contribution 2 - Other base mechanisms. The generality of our solution to the group privacy optimization problem in Theorem M.6 lets us not only derive dominating pairs for Gaussian mechanisms, but also for Laplace mechanisms and randomized response.

Novel contribution 3 - Resolving non-constancy via constraint relaxation. Perhaps most importantly, our proposed framework offers a novel perspective on an issue encountered when analyzing subsampling without replacement: While one can derive a tight mechanism-specific guarantee, this guarantee depends on different mixture distributions for α < 1 and α ≥ 1 (see Proposition K.4 [10] and our more general discussion in Appendix J.3). Thus, one cannot simply read off a dominating pair from this mechanism-specific guarantee, as we did with our group privacy bounds. However, as we shall demonstrate, this issue no longer appears when we relax some of the batch distance constraints in our proposed cost function bound from Proposition 3.5. This suggests that this could be a more general approach for addressing this non-constancy issue. While the resultant dominating pairs are not necessarily tight, they may be easier to use in practice than the convex conjugation construction from [10].

L.1 Subsampling without replacement and substitution

We begin with subsampling without replacement to demonstrate the potential benefit of relaxing batch distance constraints to determine tractable dominating pairs.

Theorem L.1. Let M = B ◦ S , where S is subsampling without replacement with batch size q , and B is the Gaussian mechanism h + V with h : Y → R D and V ∼ N ( 0 , σ 2 I D ) . Define the ℓ 2 -sensitivity L 2 = max y ≃ ∆ , Y y ′ || f ( y ) -f ( y ′ ) || 2 , where ≃ ∆ is the substitution relation. Consider an arbitrary dataset size N ∈ N and define batch-to-dataset ratio w = q N . Then, for any pair x ≃ ∆ , X x ′ of size N ,

Proof. Recall from our proof of the mechanism-agnostic subsampling without replacement guarantee Proposition H.4 that we can construct a d Y -compatible coupling to show that

Next, we relax the constraint between the two components that contain a substituted element: d Y ( y (2) 1 , y (2) 1 ) ≤ 2 .

We now observe that this optimization problem is identical to the optimization problem from our group privacy bound (Theorem 3.7) with one insertion ( K + = 1 ), one deletion ( K -= 1) , and subsampling rate r = w . The result thus follows immediately from our mechanism-specific group privacy amplification bound for Gaussian mechanism (Theorem 3.8).

We can generalize this result to other mechanisms by using group privacy amplification bounds for Laplace mechanisms (Theorem M.2) and randomized response mechanisms (Theorem M.2).

We see that the upper bound in Eq. (25) depends on the same pair of mixture distributions for all α ≥ 0 . Thus, this pair of mixture distributions is a dominating pair for subsampling without replacement under substitution.

L.2 Subsampling with replacement and substitution

Next, we provide the first formal derivation of dominating pairs for subsampling with replacement.

In subsampling with replacement, a single element may appear in a batch multiple times. To formalize this, we assume a dataset space X ⊆ P ( A ) that is composed of sets (not multisets) of elements from an underlying set A . We further assume a batch space Y ⊆ P multi ( A ) that is composed of multisets of elements from A . Given some y ∈ Y , we write ξ y ( a ) for the number of times that a appears in y . We further define the support of batch y ∈ Y as supp( y ) = { a ∈ Y | ξ y ( a ) ≥ 0 } .

Definition L.2. Subsampling with replacement with batch size q has probability mass function s x ( y ) = ( | x | + q -1 q ) -1 for batches y ∈ Y with supp( y ) ⊆ x and | y | = q .

Given this subsampling scheme, we can use optimal transport to derive the following constrained optimization problem, which we shall then relax and solve:

Theorem L.3. Let M = B ◦ S , where S is subsampling without replacement with batch size q , and B is the Gaussian mechanism h + V with h : Y → R D and V ∼ N ( 0 , σ 2 I D ) . Define the ℓ 2 -sensitivity L 2 = max y ≃ ∆ , Y y ′ || f ( y ) -f ( y ′ ) || 2 , where ≃ ∆ is the substitution relation. Consider an arbitrary dataset size N ∈ N and define batch-to-dataset ratio w = q N . Then, for any pair x ≃ ∆ , X x ′ of size N ,

subject to y ∈ Y ( q +1)+( q +1) , ∀ l, t, u : d Y ( y ( l ) t , y ( l ) u ) ≤ | t -u | , ∀ t, u : d Y ( y (1) t , y (2) u ) ≤ max { t, u } , and with weights w i = ( 1 N ) i · ( 1 -1 N ) q -i .

Proof. By definition of the substitution relation ≃ ∆ there is some a ∈ x with a / ∈ x ′ and some a ′ ∈ x ′ with a ′ / ∈ x such that x ′ = x \ { a } ∪ { a ′ } .

To simplify indexing, we define zero-based indexed events A 0 , . . . , A q and E 0 , . . . , E q . We define A i to be the event that a is sampled i times, i.e., ξ y ( a ) = i . We define E j to be the event that a ′ is sampled j times, i.e., ξ y ( a ′ ) = j .

By definition subsampling with replacement, we have P S x ( A i ) = w i , and P S x ( E j ) = w j with weights w i = ( 1 N ) i · ( 1 -1 N ) q -i .

For dataset size N , we have

and

̸

Note that s x ( y | A 0 ) = s x ′ ( y | E 0 ) . Further note that s x ( y | A q ) = ✶ [supp( y ) = { a } ∧ | y | = q ] and s x ′ ( y | E q ) = ✶ [supp( y ) = { a ′ } ∧ | y | = q ] .

̸

Coupling. We now define a coupling γ : Y ( q +1)+( q +1) → R + that corresponds to the following generative process: We first generate y (1) q by constructing a batch of size q whose elements are all a . Beginning at i ← q , we then iteratively generate y (1) i -1 from y (1) i -1 by replacing one instance of a with an element ˜ a ∈ x \ a that is sampled uniformly at random. Finally, we construct the y (2) j by replacing every occurence of a in y (1) j with a ′ . More formally,

Condition L.4. A tuple y (1) ∈ Y q +1 , y ( q +1) ∈ Y K + +1 fulfills this condition when ξ y (1) q ( a ) = ξ y (2) q ( a ′ ) = q and ∀ i, ∃ ˜ α ∈ x \ { a } : y (1) i -1 = y (1) i -1 \ { a } ∪ { ˜ a } and ∀ i, ∀ ˜ a / ∈ { a, a ′ } ) : ξ y (1) q (˜ a ) = ξ y (2) q (˜ a ) and ξ y (1) q ( a ) = ξ y (2) q ( a ′ ) .

̸

Validity. To verify the validity of this coupling, consider an arbitrary i and an arbitrary y such that γ ( y (1) , y (2) ) > 0 . We know that there are ( q -i )! ∏ ˜ a = a 1 ξ y (˜ a )! sequences of y (1) q , y (1) q -1 , . . . , y (1) i that could have generated y (1) i . We further know that there are ( 1 N -1 ) i possible sequences of y (1) i -1 , . . . , y (1) 0 that can be generated from y (1) i . All y (2) j are deterministically determined by y (1) j . We thus have

̸

The proof for the y (2) j is analogous.

Compatibility. Finally, it can be easily shown that γ is a d Y -compatible coupling (recall Appendix F). Whenever γ ( y ) > 0 , then

because we generate the y (1) i from y (1) 0 by substituting exactly i elements, and construct the y (2) j by substituting exactly j elements. Similarly, we have for the pairwise distances that do not involve y (1) 0

The last equality holds for t ≤ u because to construct y (2) u from y (1) t we need to substitute all occurrences of a with a ′ and then substitute u -t additional elements with a ′ It also holds for t > u , because then we need to replace u occurences of a with a ′ and then replace another t -u occurences with values other than a ′ .

Finally, we can relax and solve Theorem L.3 for specific mechanisms, such as the Gaussian mechanism:

̸

Theorem L.5. Let M = B ◦ S , where S is subsampling with replacement with batch size q , and B is the Gaussian mechanism h + V with h : Y → R D and V ∼ N ( 0 , σ 2 I D ) . Define the ℓ 2 -sensitivity L 2 = max y ≃ ∆ , Y y ′ || f ( y ) -f ( y ′ ) || 2 , where ≃ ∆ is the substitution relation. Consider an arbitrary dataset size N ∈ N . Then, for any pair x ≃ ∆ , X x ′ of size N ,

with univariate normal densities f (1) i = N ( · | ( i -1) , σ / L 2 ) , f (2) j = N ( · | -( j -1) , σ / L 2 ) , and weights w i = ( 1 N ) q · ( 1 -1 N ) q -i .

Proof. We first relax the optimization problem in Theorem L.3 by replacing the last constraint ∀ t, u : d Y ( y (1) t , y (2) u ) ≤ max { t, u } with a new constraint ∀ t, u : d Y ( y (1) t , y (2) u ) ≤ t + u .

This leaves us with

subject to y ∈ Y ( q +1)+( q +1) , ∀ l, t, u : d Y ( y ( l ) t , y ( l ) u ) ≤ | t -u | , ∀ t, u : d Y ( y (1) t , y (2) u ) ≤ t + u , and with weights w i = ( 1 N ) i · ( 1 -1 N ) q -i .

We now observe that this optimization problem is identical to the optimization problem from our group privacy bound (Theorem 3.7) with q insertions ( K + = q ), q deletions ( K -= q ) , and a different set of weights. The result thus follows immediately from our solution to the group privacy amplification problem from Theorem 3.7, which did not make any specific assumptions about the weights.

We see that the upper bound in Eq. (25) depends on the same pair of mixture distributions for all α ≥ 0 . Thus, this pair of mixture distributions is a dominating pair for subsampling without replacement under substitution.

M Tight mechanism-specific group privacy amplification

In the following, we first prove our generic group privacy amplification guarantee for Poisson subsampling and insertion/removal, which gives us a constrained optimization problem whose optimal value bounds Ψ α ( m x || m x ′ . We then solve this constrained optimization problem for Gaussian, Laplace, and randomized response mechanisms. After that, we prove the mechanism-specific tightness for the resultant guarantees. Finally, we discuss how to evaluate these guarantees numerically.

M.1 Proof of Theorem 3.7

Theorem 3.7. Let M = B ◦ S , where S is Poisson subsampling with rate r . Let ≃ Y be the insertion/removal batch relation ≃ ± , Y . Then, for all x ≃ K + ,K -, X x ′ , Ψ α ( m x || m x ′ ) is l.e.q.

subject to constraints y ∈ Y K -+ K + +2 , as well as ∀ l ∈ { 1 , 2 } , ∀ t, u : d Y ( y ( l ) t , y ( l ) u ) ≤ | t -u | , and ∀ t, u : d Y ( y (1) t , y (2) u ) ≤ ( t -1) + ( u -1) .

Proof. By definition of the 'insert K + , remove K -' relation ≃ K + ,K -, X there is some inserted set g + ⊆ x ′ of size K + with g + ∩ x = ∅ and some removed set g -⊆ x of size K -with g -∩ x ′ = ∅ such that x ′ = x \ g -∪ g + .

To simplify indexing, we define zero-based indexed events A 0 , . . . , A K -and E 0 , . . . , E K + . We define A i to be the event that i removed elements are sampled, i.e. | y ∩ g -| = i . We define E j to be the event that j inserted elements are sampled, i.e. | y ∩ g + | = j .

By definition of Poisson subsampling, we have P S x ( A i ) = Binom( i | K -, r ) , and P S x ( E j ) = Binom( j | K + , r ) . We further have

and

Note that s x ′ ( y | E 0 ) = s x ( y | A 0 ) .

Coupling. We now define a coupling γ : Y ( K + +1)+( K -+1) → R + that corresponds to the following generative process: We first generate y (1) 0 by sampling a batch that does not contain any inserted or removed elements via Poisson subsampling from x \ g -. We then let y (2) 0 ← y (1) 0 . Finally, we sample uniformly at random an order in which we include removed elements from g -and inserted elements from g + to iteratively construct batches from ( A i ) i> 1 and ( E j ) j> 1 , respectively. More formally,

Condition M.1. A tuple y (1) ∈ Y K -+1 , y (2) ∈ Y K + +1 fulfills this condition when y (1) 0 = y (2) 0 , and ∀ i : ∃ a -∈ g -\ y (1) i : y (1) i +1 = y (1) i ∪ { a -} , and ∀ j : ∃ a + ∈ g + \ y (2) j : y (2) j +1 = y (2) j ∪ { a -} .

Validity. We can verify the validity of this coupling as follows:

For A 0 and every y with s x ( y | A 0 ) > 0 , there are exactly K + ! · K -! combinations with y (1) 0 = y for which γ ( y ) > 0 . We thus have ∑ y ∈ Y K ++ K -+2 ✶ [ y (1) 0 = y ] γ ( v ) = s x ( y | A 0 ) . The proof for E 0 is analogous.

Next, consider an arbitrary A i with 0 < i ≤ K -. For every y with s x ( y | A 0 ) > 0 , there are exactly i ! · ( K --i )! · K + ! combinations with y (1) i = y for which γ ( y ) > 0 , because there are (a) i ! orders in which the removed elements leading up to i could have been sampled (b) ( K --i !) permutations for the remaining removed elements (c) K + ! permutations for the inserted elements that are only relevant for y (2) . We thus have

For the last step, we have used that y (1) 0 contains 0 of the removed elements from g -, whereas y contains i of the removed elements.

The proof for E j with 0 < j ≤ K + is analogous.

Compatibility. Finally, it can be easily shown that γ is a d Y -compatible coupling (recall Appendix F). Whenever γ ( y ) > 0 , then

because we generate the y (1) i from y (1) 0 by inserting exactly i elements, and construct the y (2) j by inserting exactly j elements. Similarly, we have for the pairwise distances that do not involve y (1) 0

The last equality holds because to construct y (2) u from y (1) t we need to remove t elements and insert u elements.

The result then follows from Corollary F.7, P S x ( A i ) = Binom( i | K -, r ) , P S x ′ ( E j ) = Binom( j | K + , r ) , and reverting back to one-based indexing.

M.2 Instantiations

Next, we can solve the derived optimization problem. In this section, we only provide proof sketches in which we reduce the optimization problem over batches to an optimization problem over the means of multiple Gaussian, Laplace, or Bernoulli random variables. Because solving these problems requires some further exposition, we then refer the reader to Appendix O.

M.2.1 Gaussian mechanism guarantee

Theorem 3.8. Let M = B ◦ S , where S is Poisson subsampling with rate r , and B is the Gaussian mechanism h + V with h : Y → R D and V ∼ N ( 0 , σ 2 I D ) . Define the ℓ 2 -sensitivity L 2 =

max y ≃ ± , Y y ′ || f ( y ) -f ( y ′ ) || 2 . Then for all x ≃ K + ,K -, X x ′ , Ψ α ( m x || m x ′ ) is l.e.q.

with univariate normal densities f (1) i = N ( · | ( i -1) , σ / L 2 ) , f (2) j = N ( · | -( j -1) , σ / L 2 ) .

Proof sketch. Recall from Theorem 3.7 that we need to solve the optimization problem

subject to y ∈ Y ( K + +1)+( K -+1) , ∀ l, t, u : d Y ( y ( l ) t , y ( l ) u ) ≤ | t -u | , ∀ t, u : d Y ( y (1) t , y (2) u ) ≤ ( t -1) + ( u -1) , and with w (1) i = Binom( i -1 | K -, r ) , w (2) j = Binom( j -1 | K + , r ) .

Let µ (1) i = h ( y (1) i ) and µ (2) j = h ( y (2) j ) . Since we do not have any additional information about h beyond its ℓ 2 -sensitivity, we have to make the worst-case assumption that the µ (1) i , µ (2) j are arbitrary vectors constrained by ∀ l, t, u : || µ ( l ) t -µ ( l ) u || 2 ≤ L 2 | t -u | , ∀ t, u : || µ (1) t -µ (2) u || 2 ≤ L 2 (( t -1) + ( u -1)) ,

Thus, the optimization problem is equivalent to

subject to ∀ l, t, u : || µ ( l ) t -µ ( l ) u || 2 ≤ | t -u | , ∀ t, u : || µ (1) t -µ (2) u || 2 ≤ ( t -1) + ( u -1) .

In Appendix O, we rigorously prove that the maximum is attained by co-linear, equidistant means that fulfill these distance constraints exactly. Thus, we can perform a coordinate transformation such that µ 1 = 0 and µ (1) i = ( i -1) e 1 and µ (2) j = -( j -1) e 1 with first-component indicator vector e 1 ∈ R D . Since the likelihood ratio in Ψ α is Gaussian with zero mean in all but the first dimension, we can marginalize all other dimensions out to obtain our one-dimensional result (cf. Appendix O).

M.2.2 Laplace mechanism guarantee

Theorem M.2. Let M = B ◦ S , where S is Poisson subsampling with rate r , and B is the Laplacian mechanism h + V with h : Y → R D and V ∼ Lap( 0 , σ 2 I D ) , with location 0 ∈ R D and diagonal scale matrix λ I D ∈ R D × D + , which adds independent Laplacian noise to each dimension. Define the ℓ 1 -sensitivity L 1 = max y ≃ ± , Y y ′ || f ( y ) -f ( y ′ ) || 1 . Then, for all x ≃ K + ,K -, X x ′ , Ψ α ( m x || m x ′ ) is l.e.q.

with univariate Laplace densities f (1) i = Lap( · | ( i -1) , λ / L 1 ) , f (2) j = Lap( · | -( j -1) , λ / L 1 ) .

Proof sketch. Recall from Theorem 3.7 that we need to solve the optimization problem

subject to y ∈ Y K + + K -, ∀ l, t, u : d Y ( y ( l ) t , y ( l ) u ) ≤ | t -u | , ∀ t, u : d Y ( y (1) t , y (2) u ) ≤ ( t -1) +( u -1) , and with w (1) i = Binom( i -1 | K -, r ) , w (2) j = Binom( j -1 | K + , r ) .

Let µ (1) i = h ( y (1) i ) and µ (2) j = h ( y (2) j ) . Since we do not have any additional information about h beyond its ℓ 1 -sensitivity, we have to make the worst-case assumption that the µ (1) i , µ (2) j are arbitrary vectors constrained by ∀ l, t, u : || µ ( l ) t -µ ( l ) u || 1 ≤ L 1 | t -u | , ∀ t, u : || µ (1) t -µ (2) u || 1 ≤ L 1 (( t -1) + ( u -1)) ,

Thus, the optimization problem is equivalent to.

In Appendix O, we rigorously show that the maximum is attained by collinear, equidistant vectors along a single coordinate axis that leave no slack on the pairwise distance constraints. This then allows us to marginalize out all remaining dimensions to obtain our guarantee in terms of univariate Laplace densitities (see Appendix O).

M.2.3 Randomized response mechanism guarantee

Theorem M.3. Let B be the randomized response mechanism | h -(1 -V ) | with h : Y → { 0 , 1 } , V ∼ Bernoulli( θ ) , and true response probability θ ∈ [0 , 1] . Let M = B ◦ S be the corresponding subsampled mechanism, where S is Poisson subsampling with rate r . Finally, let ≃ Y be the insertion/removal relation ≃ ± , Y . Then, for all x ≃ K + ,K -, X x ′ , Ψ α ( m x || m x ′ ) is l.e.q.

Proof sketch. Recall from Theorem 3.7 that we need to solve the optimization problem

subject to y ∈ Y K + + K -, ∀ l, t, u : d Y ( y ( l ) t , y ( l ) u ) ≤ | t -u | , ∀ t, u : d Y ( y (1) t , y (2) u ) ≤ ( t -1) +( u -1) , and with w (1) i = Binom( i -1 | K -, r ) , w (2) j = Binom( j -1 | K + , r ) .

Let µ (1) i = h ( y (1) i ) and µ (2) j = h ( y (2) j ) . Since we do not have any additional information about h beyond it mapping to { 0 , 1 } , we have to make the worst-case assumption that the µ (1) i , µ (2) j are only constrained by µ (1) 2 = µ (2) 1 .

Thus, the optimization problem is equivalent to.

subject to µ (1) 1 = µ (2) 1

In Appendix O, we rigorously show that the maximum is attained whenever all µ (1) i with i > 1 are simultaneously set to either 0 or 1 , and all µ (2) j with j > 1 are set to the opposite value.

Note that this guarantee can be evaluated in O (1) : We need to iterate over two possible values of τ , and evaluating the corresponding Ψ α requires summing over two values z ∈ { 0 , 1 } .

M.3 Tightness

Next, we prove tightness by constructing datasets and underlying functions h such that the corresponding base mechanism B exactly attains the different bounds under subsampling scheme S .

M.3.1 Gaussian mechanism tightness

Theorem M.4. Let S be Poisson subsampling with rate r ∈ [0 , 1] and θ ∈ [0 , 1] be some true response probability. There exists a dataset space X and a batch space ( Y , Y ) fulfilling the constraints in Definition D.4, as well a pair of datasets x ≃ K + ,K -, X x ′ , and a function h : Y → R D with ℓ 2 -sensitivity L 2 , such that the corresponding subsampled Gaussian mechanism M = B ◦ S fulfills

with univariate normal densities f (1) i = N ( · | ( i -1) , σ / L 2 ) , f (2) j = N ( · | -( j -1) , σ / L 2 ) .

Proof. Let X = P ( N ) . Consider an arbitrary x ∈ X and select arbitrary deleted elements g ⊆ x , with | g | = K -and arbitrary inserted elements g ⊆ X \ x . Define x ′ = x \ g -∪ g + .

We now construct a counting function for h that leads to the largest possible divergence under the sensitivity constraint. We define function h as follows:

with first-component indicator vector e 1 ∈ R D .

By construction, P M x is a mixture distributions with K +1 components, each corresponding to a size of a subset of g -that is included in batch y . These cases each have probability Binom( i -1 | K -, r ) under subsampling distributions P S x . Each component has distribution N ( · | ( i -1) , e 1 L 2 , σ 2 I D ) .

Analogously, P m x ′ is the other desired mixture of Gaussians.

As in our previous proof, we can now notice that the likelihood ratio in Ψ α is constant in all but the first dimension. We can thus marginalize out the remaining dimensions to obtain our result.

M.3.2 Laplace mechanism tightness

Theorem M.5. Let S be Poisson subsampling with rate r ∈ [0 , 1] and θ ∈ [0 , 1] be some true response probability. There exists a dataset space X and a batch space ( Y , Y ) fulfilling the constraints in Definition D.4, as well a pair of datasets x ≃ K + ,K -, X x ′ , and a function h : Y → R D with ℓ 1 -sensitivity L 1 , such that the corresponding subsampled Laplace mechanism M = B ◦ S fulfills

with univariate Laplace densities f (1) i = Lap( · | ( i -1) , λ / L 1 ) , f (2) j = Lap( · | -( j -1) , λ / L 1 ) .

Proof. The proof is identical to that of the tightness guarantee from Theorem M.4. We can construct exactly the same counting function that indicates the number of inserted or removed element that appear in a batch sampled from S x and S x ′ , respectively.

As in our previous proof, we can now notice that the likelihood ratio in Ψ α is constant in all but the first dimension. We can thus marginalize out the remaining dimensions to obtain our result.

M.3.3 Randomized response mechanism tightness

Theorem M.6. Let S be Poisson subsampling with rate r ∈ [0 , 1] and θ ∈ [0 , 1] be some true response probability. There exists a dataset space X and a batch space ( Y , Y ) fulfilling the constraints in Definition D.4, as well a pair of datasets x ≃ K + ,K -, X x ′ , and a function h : Y → 0 , 1 , such that the corresponding subsampled randomized response mechanism M = B ◦ S fulfills

Proof. The following proof is largely identical to our randomized response tightness proof for subsampling without replacement from Appendix I.3.

Let X = P ( N ) . Consider an arbitrary x ∈ X and select arbitrary deleted elements g ⊆ x , with | g | = K -and arbitrary inserted elements g ⊆ X \ x . Define x ′ = x \ g -∪ g + .

Assume w.l.o.g. that the divergence is maximized by τ = θ .

We now construct an indicator function h : Y →{ 0 , 1 } for a and a ′ that leads to the largest possible divergence.

̸

By construction of h , the corresponding base mechanism pmf is

̸

Under the distribution of S ( x ) , the first case occurs with probability (1 -r ) K -and the second case occurs with probability 1 -(1 -r ) K -. Under the distribution of S ( x ′ ) , the first case occurs with probability (1 -r ) K + and the second case occurs with probability 1 -(1 -r ) K + . We thus have

which exactly attains the desired divergence when τ = θ is the optimal value.

M.4 Numerical evaluation

Evidently, we do not have a closed-form analytical expression for our tight mechanism-specific group privacy bounds. However, we can evaluate them to arbitrary precision using standard techniques from privacy accounting literature.

M.4.1 Gaussian mechanism

ADP. To evaluate the guarantee from Theorem 3.8, we can use Lemma 5 from [10], which generalizes an alternative characterization of privacy profiles from [74] to dominating pairs. Let P, Q be the dominating pair of two univariate Gaussian mixtures. Define privacy loss random variables

Because the privacy loss between the two Gaussian mixtures is monotonically increasing in z [8], one can perform a change of variables via a binary search for a z ∗ such that log( p ( z ∗ ) q ( z ∗ ) ) ≈ log( α ) . By picking one of the two search boundaries, one can either over- or under-approximate the hockey stick divergence (see, e.g., [8, 24]).

RDP via quadrature. To evaluate the guarantee for RDP, we can simply use numerical quadrature. This can be done efficiently because we only need to integrate over univariate Gaussians. This approach was already proposed and used in Abadi et al. [6]'s work on moments accounting.

RDP via expansion. For the special case of K -= K and K + = 0 , one can also use multinomial expansion (similar to prior work on RDP subsampling from [28-30]): We have

Using quadratic expansion, we have

Since only the first factor depends on z , our problem reduces to computing the divergence

for different l k . This can be done in closed form, as shown in [7].

M.4.2 Laplace mechanism

ADP. Because the privacy loss is constant on ( -∞ , -K + ) , monotonically increasing on [ -K + , K -] and constant on ( K -, ∞ ) , we can again use the same bisection method.

RDP. As with the Gaussian mechanism, we can evaluate the bound via univariate numerical quadrature. Because the privacy loss is non-smooth at the component means {-K + , -K + + 1 , . . . , 0 , 1 , . . . , K -} , we partition R at these means and integrate over each interval separately.

M.4.3 Randomized response mechanism

The guarantee for randomized smoothing can be evaluated exactly in O (1) . We just need to iterate over the two options τ ∈ { 0 , 1 } , and for each one evaluate the divergence on space { 0 , 1 } , which only requires evaluating two fractions and two sums.

N Tight mechanism-agnostic group privacy amplification

In this section, we apply the framework from [15], which we summarized in Appendix G.1, to the group privacy setting. We then show the tightness of the resultant guarantees, using the same proof strategy as in Section 5 of [15]. We then demonstrate that it is directly related to our tight mechanism-specific guarantee through joint convexity. Finally, we derive a qualitatively similar guarantee for RDP.

For this discussion, we focus on the special case where all of the K group members collaboratively agree to insert their data ( K + = K,K -= 0) or delete their data ( K + = 0 , K -= K ) , so that the partition induced by the maximal coupling remains interpretable. We define the corresponding neighboring relation as

In our experiments, we only evaluate the baseline for ( K + = K,K -= 0) and ( K + = 0 , K -= K ) , while we evaluate our method for all K + + K -= K and take the maximum. This evaluation favors the baseline.

N.1 ADP guarantee

Proposition N.1. Let M = B ◦ S , where S is Poisson subsampling with rate r . Let ≃ Y be the insertion/removal batch relation ≃ ± , Y . Then, for all x ≃ K ± , X x ′ and all ε ≥ 0

with ε ′ = log(1 + (1 -(1 -r ) K ) · ( e ε -1)) and group privacy parameters

Proof. Case 1: Deletion. We first consider the case where K elements are deleted, i.e., there is some deleted set g ⊆ x of size K with g -∩ x ′ = ∅ such that x ′ = x \ g .

The partition induced by the maximal coupling 9 is

where A 2 = { y ∈ Y | y ∩ g = ∅ is the event that no element of g is sampled, A 1 is its complement, and w = (1 -(1 -r ) K ) = P S x ( A 1 ) . We use these indices for A 2 and A 1 because advanced joint convexity will later reverse their order.

Applying advanced joint convexity (see Proposition G.1) and joint convexity shows that

Next, we can bound this mixture divergence by constructing a coupling invoking Theorem 3.3 with the special case of Ψ α = H exp( ε ) . For this, notice that

and

Coupling. We define a coupling γ : Y 2 → [0 , 1] that corresponds to the following generative process: We first sample y (2) from s x ( y | A 2 ) . We then sample from a truncated binomial distribution how many elements k from group g should be included. Given this k , we sample uniformly at random a ˜ g ⊆ g from all sizek subsets of g and let y (1) ← y (2) ∪ ˜ g . Formally this is defined by

Condition N.2. A tuple y (1) ∈ Y , y (2) ∈ Y fulfills this condition when there exists a ˜ g ⊆ g with | ˜ g | ≥ 1 such that y (1) = y (2) ∪ ˜ g .

Validity. Next, we verify the validity of this coupling. For every y (1) , there is exactly one y (2) such that γ ( y (1) , y (2) ) > 0 , namely y (2) = y (1) \ g ′ . Thus,

For every y (2) , there are exactly ( K k ) batches y (1) such that γ ( y (1) , y (2) ) > 0 and | y (1) ∩ g = k | . Thus,

Now that we have proven the validity of the coupling, we can apply the optimal transport bound (Theorem 3.3) to Eq. (28) in order to prove

Case 2: Insertion. In this case, the partition induced by the maximal coupling is

which is identical to the previous partition, up to symmetry. The proof is identical, except for changes in indexing.

Taking the maximum over both guarantees yields the result.

N.2 Tightness of ADP guarantee

Proposition N.3. Let S be Poisson subsampling with rate r . Let ≃ Y be the insertion/removal batch relation ≃ ± , Y . Then, for all x ≃ K ± , X x ′ and all ε ≥ 0 , there exists a worst-case base mechanism B

such that the corresponding subsampled mechanism M = B ◦ S fulfills

with ε ′ = log(1 + (1 -(1 -r ) K ) · ( e ε -1)) and group privacy parameters

Proof. To show tightness of the bound, we notice that the bound ∑ K k =1 Binom( k | K,r ) · δ k is identical to the bound for subsampling with replacement from [15], except for the numeric value of the weights in the weighted sum. We can thus use exactly the same proof strategy.

Assume w.l.o.g. that we are in the insertion case, i.e., there is some g of size K with g ∩ x = ∅ and x ′ = x ∪ g . Consider an arbitrary ε and define an arbitrary true response probability θ ∈ [0 , 1] . Further define the randomized membership base mechanism

B ( y ) = | ✶ [ | y ∩ g | > 0] -(1 -V ) | with V ∼ Bern( θ ) . It is easy to verify [15] that δ k = ψ θ ( ε ) = max { θ -e ε (1 -p ) , 0 } . We thus have for the r.h.s. of Eq. (29)

Because Poisson subsampling is a 'natural subsampling'[15] scheme that only yields elements from the dataset it is applied to, it follows from Lemma 12 of [15] that also

N.3 Relation to tight mechanism-specific bound

In the following, we show that this mechanism-agnostic guarantee implicitly upper bounds our tight mechanism-specific guarantee via joint convexity. Note that this is qualitatively different from our discussion in Appendix G. There we showed, that the mechanism-agnostic guarantees can be derived from mechanism-specific bounds that only use a binary partitioning of the batch space (unlike the mechanism-specific guarante considered here). We show this result w.l.o.g. for K -= 0 and K + = K .

Theorem N.4. Let B : Y → Z be an arbtirary base mechanism. Let y (1) ∈ Y 1 and , y (2) ∈ Y K +1 be arbitrary tuples of batches that fulfill ∀ l, t, u : d Y ( y ( l ) t , y ( l ) u ) ≤ | t -u | , and ∀ t, u : d Y ( y (1) t , y (2) u ) ≤ ( t -1) + ( u -1) . Then, for α ≥ 1

with w = (1 -(1 -r ) K ) , α ′ = 1 + w ( α -1) and group privacy parameters

Proof. We can apply joint convexity to show

The result then immediately follows from advanced joint convexity (see Proposition G.1) and joint convexity.

N.4 RDP guarantee

For RDP, we can obtain a qualitatively similar guarantee by simply applying the known mechanismagnostic RDP subsampling guarantee for Poisson subsampling from [29, 30] to each of the K divergences (although we could also derive this RDP guarantee from scratch via optimal transport) in Eq. (33) of the previous derivation.

Theorem N.5. Let M = B ◦ S , where S is Poisson subsampling with rate r . Let ≃ Y be the insertion/removal batch relation ≃ ± , Y . Then, for all x ≃ K ± , X x ′ and all α > 0

with w = (1 -(1 -r ) K ) and group privacy parameters

The factor 2 in Theorem N.5 can be eliminated for distributions with particular symmetries (see Theorem 5 in [29]) or bounded Pearson-Vajda χ l -pseudo-divergence (see Theorem 8 in [30]). We thus do not include the factor 2 when using this amplification guarantee as a baseline in our experiments.

N.5 Asymptotic RDP guarantees

As mentioned in Section 3.1, our focus is on tight bounds that can be explicitly computed. However, analyzing the asymptotic behavior of these bounds can provide a potentially useful high-level picture of their behavior. As discussed in the previous section, and as can be seen from Eq. (33), Rényi divergence in the group privacy setting is bounded by a weighted sum of Rényi divergences between a single distribution and a mixture of two distributions. For the special case of additive Gaussian mechanisms with global sensitivity L , this bound is equivalent (see Appendix A of [30]) to

Asymptotic bounds on Rényi divergences with one or two components have been derived in prior work on privacy accounting [6, 28, 48]. We can apply these asymptotic bounds to each summand. For instance (see Lemma 5 in [6]):

Proposition N.6. Abadi et al. [6] Let σ ≥ 1 and q ≤ 1 16 σ . Then, for any positive integer α ≤ σ 2 ln ( 1 qσ ) -1 ,

Applying Proposition N.6 to Eq. (34) yields the following asymptotic bound for RDP group privacy:

Theorem N.7. Let M = B ◦ S , where S is Poisson subsampling with rate r and B is an additive gaussian mechanism with global sensitivity L under the insertion/removal relation ≃ ± . Consider arbitrary datasets x ≃ K ± , X x ′ . Define weight w = (1 -(1 -r ) K ) . If σ ≥ k · L and w ≤ 1 16 σ , then it holds for any positive integer α ≤ σ 2 k 2 · L 2 ln ( 1 wσ ) -1 that

Alternatively, one could apply the asymptotic bound from Theorem 38 from [48] to each summand. If we were to instead consider group privacy under dataset-level substitutions, where each summand would include a divergence between two mixtures with two components, we could instead use the asymptotic bounds from Appendix C of [28].

O Worst-case mixture components

O.1 Gaussian and Laplacian mixtures

We outline a self-contained and extendable proving strategy which we use to find dominating pairs of Gaussian and Laplacian mixtures given divergences of the form Ψ α ( P || Q ) = ∫ R D f ( P ( x ) , -Q ( x )) d x , where f is (not necessarily strictly) convex and increasing in both arguments. Our two main examples of the hockey stick divergence Ψ α = H α and (scaled and exponentiated) Rényi divergence Ψ α = Λ α are special cases with f ( x, y ) = max { x + αy, 0 } and f ( x, y ) = x α ( -y ) 1 -α , respectively. Throughout the subsection, we use M , N to denote the sets of means of the two mixtures P , Q . We start with two general lemmata before treating the Gaussian and Laplacian mixture cases in particular. Together, they provide a constructive toolset to connect any mixture pair with means ( M,N ) to a dominating mixture pair with means ( M ∗ , N ∗ ) via a path of geometric transformations ( M,N ) ↦-→ ( M (1) , N (1) ) ↦-→ ··· ↦- → ( M ∗ , N ∗ ) : concretely, these are

mirroring the means of the two mixtures onto opposite sides of a hyperplane, and

pushing such hyperplane-separated means further away along the hyperplane normal.

In the first part of this section, we prove that the divergence can only stay equal or grow under any such transformation. Afterwards, we construct an explicit path of transformations that maps any Gaussian, as well as any Laplacian mixture onto a dominating pair which is feasible under the pairwise distance constraints discussed in Appendix M.2.

Lemma O.1. Given p ∈ { 1 , 2 } , consider two mixtures of the form P M ( x ) = ∑ K k =0 w k ρ ( ∥ x -µ k ∥ p ) , Q N ( x ) = ∑ K κ =0 ω κ ρ ( ∥ x -ν κ ∥ p ) with means in M := { µ 0 , . . . , µ K } ⊂ R D , N := { ν 0 , . . . , ν K } ⊂ R D and a decreasing ρ : R + 0 → [0 , 1] . Consider all hyperplanes which contain zero and are normal to L 2 -unit vectors ˆ n ∈ H p , where

̸

and define the lower half-space R -ˆ n = { x ∈ R D | ( x T ˆ n ) < 0 } and upper half-space R + ˆ n = { x ∈ R D | ( x T ˆ n ) > 0 } .

Consider the map

which mirror-reflects the lower into the upper half-space, and its image sets M ′ ˆ n = R D \ R -ˆ n and N ′ -ˆ n ⊂ R D \ R + ˆ n .

The reflected pair of mixtures has equal or greater divergence:

Remark O.2 . Lemma O.1 applies to Laplacian and Gaussian mixtures with p = 1 and p = 2 , respectively.

Proof. We will need the following lemma.

Lemma O.3. Given p ∈ { 1 , 2 } , a hyperplane normal to ˆ n ∈ H p , and points x 1 , x 2 ∈ R D , we have

Corollary O.4. If the sets M , N are feasible under pairwise p -norm distance constraints, then so are M ′ ˆ n , N ′ -ˆ n .

Proof of Lemma O.3. If sign ( n T x 1 ) = sign ( n T x 2 ) , then ( · ) ′ ˆ n acts uniformly on both vectors and the condition clearly holds. Else, assume w.l.o.g. that x 2 ∈ R -ˆ n . We start with the case p = 1 .

First, assume ˆ n ∈ { σ ˆ e i | 0 ≤ i ≤ K,σ 1 ∈ { + , -}} . Since ( · ) ′ ˆ n now acts only on vector component i , we can write ∥ ( x 1 ) ′ ˆ n -( x 2 ) ′ ˆ n ∥ 1 -∥ x 1 -x 2 ∥ 1 = || ˆ e T i x 1 | - | ˆ e T i x 2 || - | ˆ e T i x 1 -ˆ e T i x 2 | ≤ 0 by the inverse triangle inequality.

̸

Now assume instead that ˆ n ∈ { 1 √ 2 ( σ 1 ˆ e i + σ 2 ˆ e j ) | 1 ≤ i = j ≤ D,σ 1 , σ 2 ∈ { +1 , -1 }} . Now, ( · ) ′ ˆ n acts only on vector components i and j . Thus, ∥ ( x 1 ) ′ ˆ n -( x 2 ) ′ ˆ n ∥ 1 - ∥ x 1 -x 2 ∥ 1 = ∥ π i,j [( x 1 ) ′ ˆ n -( x 2 ) ′ ˆ n ] ∥ 1 - ∥ π ( i,j ) [ x 1 -x 2 ] ∥ 1 where π ( i,j ) denotes projection onto the subspace spanned by the basis vectors ˆ e i , ˆ e j . It is useful to write

The argument of the max function admits two cases. Either, σ T ( π i,j ˆ n ) = 0 , in which case | σ T π i,j [( x 1 ) ′ ˆ n -( x 2 ) ′ ˆ n ] | = | σ T π i,j [ x 1 -x 2 ] | . In the other case, π i,j σ = ± 2 √ 2 π i,j ˆ n . Then, by the inverse triangle inequality,

For completeness, we also prove p = 2 . By invariance of the scalar product under mirror reflection, ∥ x ′ 1 -x ′ 2 ∥ 2 = x T 1 x 1 + x ′ T 2 x ′ 2 -2 x T 1 x ′ 2 = x T 1 x 1 + x T 2 x 2 -2 x T 1 x 2 + 2( n T x 2 )( n T x 1 ) = ∥ x 1 -x 2 ∥ 2 +2( n T x 2 )( n T x 1 ) ≤ ∥ x 1 -x 2 ∥ 2 .

We recall the notation introduced at the start of the section, Ψ α ( P || Q ) = ∫ R D f ( P ( x ) , -Q ( x )) d x , where we assume an integrand f that is convex in both arguments. The statement of Lemma O.1 will follow from the next lemma, which informally states that at any point x in the upper half-space, the mirror-reflection of means contained in M -ˆ n causes an increase of the integrand f ( P M ( x ) , -Q N ( x )) at x which dominates the corresponding decrease at the mirror image point x ′ in the lower half-space.

Lemma O.5. Given a hyperplane normal to ˆ n ∈ H p as defined in Lemma O.1 as well as any point x ∈ R + ˆ n along with its mirror image x ′ = x -2 ( x T ˆ n ) ˆ n ∈ R -ˆ n , the change of f satisfies

Proof of Lemma O.5. We introduce the following notation,

thus decomposing the densities into a part that stays fixed and a part that gets mirror-reflected. Using Lemma O.3 and the monotonicity requirement on ρ , we can directly verify that

Using invariance of distances under simultaneous mirroring of both vectors, we rewrite the expression of the lemma as

The statement of the lemma follows from convexity of f in both arguments by Jensen's inequality.

We now proceed to prove Lemma O.1. The difference of divergences can be rewritten as

Apart from Lemma O.5, we used the fact that mirror reflection has a unit absolute Jacobian determinant, and that the hyperplane, a null set, does not contribute to the integral.

Lemma O.1 shows that we can construct a dominating mixture by mirroring all means onto opposite sides of a suitable hyperplane. This is exactly the condition under which the follow-up transform discussed in the next lemma yields equal or greater divergence. We slightly change perspective and consider the means of both mixtures as a joint vector. Given two sets of means M = { µ 0 , . . . , µ K } , N = { ν 0 , . . . , ν K } we define (in arbitrary ordering), µ = ( µ 0 , . . . , µ K , ν 0 , . . . , ν K ) ∈ F ⊂ R D × ( K + K ) where F denotes the region feasible under the distance constraints. With this implicit mapping ( M,N ) ↦→ µ between sets and vectors of means, we can treat the divergence as a function Ψ α : µ ↦→ ∫ R D ˜ f ( x , µ ) d x with integrand ˜ f ( x , µ ) := f ( P M ( x ) , -Q ( x )) . Now we can state:

Lemma O.6. For p ∈ { 1 , 2 } , let ˆ n ∈ H p be a normal vector, M , N with corresponding vector µ as above be two sets of means for which M ∩ R -ˆ n and N ∩ R --ˆ n are empty (i.e., the hyperplane separates the two sets), and let Ψ α denote either Λ α or H α .

Then, the normal directional derivatives of the divergence with respect to the means µ l and ν ι are nonnegative for all l ∈ { 1 , . . . , K } , ι ∈ { 1 , . . . , K} ,

In the case Ψ α = H α , the above expressions are instead defined in the sense of weak derivatives.

The lemma is quite intuitive: once both sets of means are already separated by a hyperplane, pushing them apart in the normal direction, thereby increasing the margin to the hyperplane, never decreases the divergence. As we will only ever use the directional derivatives within line integrals, the weak sense in which the lemma holds for Ψ α = H α is sufficient for our purposes.

Proof of Lemma O.6. We first check that for all x ∈ R D and µ ∈ F , the derivative ∂ µ ˜ f (possibly in the weak sense) exists and is dominated by an integrable function. In the case of Gaussian or Laplacian mixtures and Ψ α = Λ α , existence of the derivative is clear. For Gaussian mixtures, the integrability condition holds since ∂ µ ˜ f = O (exp( -∥ x ∥ 2 2 )) , for Laplacian mixtures ∂ µ ˜ f = O (exp( -∥ x ∥ 1 )) since the privacy loss factor in the integrand eventually becomes constant. In the case Ψ α = H α , ˜ f has a weak derivative ∂ µ ˜ f = 1 f ≥ 0 ∂ µ ( P -αQ ) , which agrees with its derivative except where the latter is not defined (i.e., on the null set of means and x at the boundary of exact DP). Integrability

follows again from ∂ µ ˜ f = O (exp( -∥ x ∥ 1 )) . In the case of Ψ α = Λ α ,

+ ′ -

Consider x ∈ R and its mirror image x ∈ R . By Lemma O.3 and the assumption that the hyperplane separates M and N , P M ( x + µ l ) Q N ( x + µ l ) = ρ ( ∥ x + µ l ∥ p ) ρ ( ∥ x ∥ p ) P M ( x ) ρ ( ∥ x + µ l ∥ p ) ρ ( ∥ x ∥ p ) Q N ( x ) = P M ( x ) Q N ( x ) ≥ P M ( x ′ ) Q N ( x ′ ) = P M ( x ′ + µ l ) Q N ( x ′ + µ l ) . Moreover, also by Lemma O.3, ρ ( ∥ x ∥ p ) = ρ ( ∥ x ′ ∥ p ) , and therefore d ( µ l ) ˆ n ρ ( ∥ x ∥ p ) = d ( µ l ) -ˆ n ρ ( ∥ x ′ ∥ p ) = -d ( µ l ) ˆ n ρ ( ∥ x ′ ∥ p ) .

Similarly, we observe how in

the negative signs of -ˆ n and (1 -α ) cancel and apply analogous reasoning thereafter.

The statement for Ψ α = Λ α For Ψ α = H α , we find now follows in analogy to the proof of Lemma O.1 via Lemma O.5.

and can proceed by analogous reasoning from here on.

We are now equipped to construct the sequence of transformations taking two arbitrary sets M,N of Gaussian or Laplacian mixture means to the means M ∗ , N ∗ of a dominating pair. We start with the Gaussian case.

O.1.1 Dominating pair of Gaussian mixtures

Theorem O.7. Let Ψ α = H α or Ψ α = Λ α . Any pair of Gaussian mixtures with means satisfying distance constraints of the form

Proof. We start by proving the following lemma.

Lemma O.8. Any pair of Gaussian mixtures with means at a fixed set of radii,

and satisfying the constraints of Theorem O.7 is dominated by a feasible pair with means at equal radii that are collinear on diametral half-lines through zero, i.e.,

Proof of Lemma O.8. Without loss of generality, we can pick as the collinearity direction from the lemma ˆ e = e 1 , the first canonical basis vector, since the divergence is rotation invariant. Pick any orthogonal direction, say, the second basis vector e 2 . By Lemma O.1, we can mirror the two mixtures onto opposite sides of the hyperplanes with normal vectors ˆ e 1 and ˆ e 2 such that ˆ e T 1 µ k ≥ 0 ≥ ˆ e T 2 µ k for all k and ˆ e T 1 ν κ ≥ 0 ≥ ˆ e T 2 ν κ for all κ .

Now, consider a hyperplane with normal vector ˆ n ( θ ) = ˆ e 1 cos( θ ) + ˆ e 2 sin( θ ) , θ ∈ [0 , π/ 2] . Intuitively, we let this hyperplane undergo a full rotation and 'scoop up' all the means which are not in the hyperplane normal to ˆ e 2 , giving a new set of means which are. Formally, we find that by the above sign condition, there is an angle θ k ∈ [0 , π/ 2] for every k such that sign (ˆ n ( θ ) T ˆ µ k ) = sign ( θ k -θ ) . Namely, using π 12 to denote the projection onto the subspace spanned by vectors ˆ e 1 and ˆ e 2 , π 12 µ k = ∥ π 12 µ k ∥ 2 (sin( θ k )ˆ e 1 -cos( θ k )ˆ e 2 )) . Consider the paths γ k : [0 , π/ 2] → R D , where

and a sign-flipped construction with angles θ κ and curves ζ κ for the means ν κ . For the derivatives along the curves, we find γ ′ k ( θ ) = ∥ π 12 µ k ∥ 2 ˆ n ( θ ) and ζ ′ κ ( θ ) = -∥ π 12 ζ κ ∥ 2 n ( θ ) . Also, by construction, ˆ n ( θ ) T ˆ γ k ( θ ) ≥ 0 ≥ ˆ n ( θ ) T ˆ ζ κ ( θ ) . Hence, the prerequisites for Lemma O.6 are fulfilled at every θ ∈ [0 , π/ 2] . By evaluating the path integral along these curves and invoking Lemma O.6, we find that the divergence along the path M,N ↦→ ˜ M, ˜ N cannot decrease:

Clearly, the paths also preserve the radius of each mean. The final set of means ˜ M = { γ 1 ( π/ 2) , . . . , γ K ( π 2 ) } , ˜ N = { ζ 1 ( π/ 2) , . . . , ζ K ( π 2 ) } is contained in the hyperplane normal to ˆ e 2 . We can now simply repeat this procedure with all basis vectors orthogonal to ˆ e 1 to find the set from the lemma. Since it is collinear with the same radii as M,N , and the means M,N are feasible, the new set is also feasible by the Cauchy-Schwarz inequality.

Since we can map any set of Gaussian mixture means onto one with the same radii that is collinear without decreasing the divergence, we can from now on study the problem restricted to a single radial dimension, using

We can now state the next lemma which implies Theorem O.7.

Theorem O.9. (0 , 1 , . . . , K, 0 , . . . , K ) is a global maximizer of Ψ α ∣ ∣ ∣ ∣ R ˆ e under the constraints r 0 = ρ 0 = 0 , | r i -r j | ≤ | i -j | ∀ i, j ∈ { 0 , . . . , K } , | ρ ι -ρ τ | ≤ | ι -τ | ∀ ι, τ ∈ { 0 , . . . , K} .

Proof of Theorem O.9. To avoid clutter, we constrain ourselves to the case with only means µ k and a single ν 0 = 0 , since the proof involving several ν κ is completely analogous. Consider the line integral of the gradient

between any feasible ( r 0 , . . . , r K ) and the point (0 , . . . , K ) along the connecting path

First, observe that r k ≤ k ∀ k ∈ { 0 , . . . , K } anywhere in the feasible region, since r k = ∑ k -1 j =0 ( r j +1 -r j ) ≤ ∑ k -1 j =0 | r j +1 -r j | ≤ ∑ k -1 j =0 1 = k . This means that v is componentwise-

∣

∣

∣

∣

nonnegative and vanishes only if ( r 0 , . . . , r K ) = (0 , . . . , K ) . By Lemma O.6, the gradient ∇ Ψ α

is componentwise-nonnegative anywhere along γ r 0 ,...,r K . We may thus conclude that

This concludes the proof of Theorem O.7.

O.1.2 Dominating pair of Laplacian mixtures

We also find an analogous result for Laplacian mixtures.

Theorem O.10. Let Ψ α = H α or Ψ α = Λ α . Any pair of Laplacian mixtures with means in M = { µ 0 , . . . , µ K } , N = { ν 0 , . . . , ν K } ⊂ R D satisfying pairwise L 1 distance constraints

is dominated by a pair for M ∗ , N ∗ of the form

where 1 ≤ i ≤ D such that ˆ e i is the i -th canonical basis vector of R D .

Proof.

Lemma O.11. There is a dominating pair M ∗ , N ∗ located in diametral quadrants of R D : given the component-wise sign function,

Proof. This is an immediate consequence of considering the canonical basis vectors as normal vectors in Lemma O.1.

Lemma O.12. For µ k ∈ M , ν κ ∈ N , define the offset vectors δµ k := µ k -µ k -1 , δν κ := ν κ -ν κ -1 for 1 ≤ k ≤ K , 1 ≤ κ ≤ K . A dominating pair M ∗ , N ∗ has offset vectors located on diametral simplices of the 1-sphere: ∃ σ ∈ { +1 , -1 } D : δµ ∗ k ∈ S σ , δν ∗ κ ∈ S -σ , with S σ := { x ∈ R D | σ T x = 1 } , ∀ k ∈ { 1 , . . . , K } ∀ κ ∈ { 1 , . . . , K} .

R

+

0

ˆ

e

Proof. For any pair M,N of means not satisfying this condition, we will construct a new pair that does and has at least equal divergence. By Lemma O.11, we can assume without generality loss that ∃ σ ( D ) ∈ { +1 , -1 } D : sign ( µ k ) = σ ( D ) = -sign ( ν κ ) ∀ µ k ∈ M, ν κ ∈ N , where we put a superscript ( D ) above σ for later reasons. At a later stage of the proof, we will invoke that shifting the means along the basis vector directions with signs prescribed by σ ( D ) are positive almost anywhere as a consequence of Lemma O.6.

It is useful to recast the optimization constraints in terms of the offset vectors: ∀ k ∈ { 1 , . . . , K } ∀ κ ∈ { 1 , . . . , K} ,

where the constraints between non-ascending and non-neighboring index pairs are implied by the symmetry and triangle inequality of the norm. For 0 ≤ k ≤ K , and 0 ≤ κ ≤ K , set µ (0) k := µ k , ν (0) κ := ν κ , and define (recursively for 1 ≤ i ≤ D ) the paths

By construction,

We can easily check that δµ ( i ) k ∈ S σ ( i ) k , δν ( i ) κ ∈ S σ ( i ) κ , for 1 ≤ k ≤ K, 1 ≤ κ ≤ K .

We can moreover verify that, based on the definitions of σ ( i ) k and σ ( i ) κ , the feasibility conditions ∥ δµ ( i -1) k ∥ 1 ≤ 1 and ∥ δν ( i -1) κ ∥ 1 ≤ 1 imply ∥ δµ ( i ) k ∥ 1 = max σ ∈{-1 , +1 } D ( σ T δµ ( i ) k ) ≤ 1 and ∥ δν ( i ) κ ∥ 1 = max σ ∈{-1 , +1 } D ( σ T δν ( i ) κ ) ≤ 1 . Since, furthermore, µ ( i ) 0 = µ 0 = 0 , ν ( i ) 0 = ν 0 = 0 for 1 ≤ i ≤ D , and M,N are feasible by assumption, all pairs of M ( i ) := { µ ( i ) 0 , . . . , µ ( i ) K } , N ( i ) := { ν ( i ) 0 , . . . , ν ( i ) K } are also feasible by induction.

Finally, the above simplex and feasibility properties together imply that ∀ k ∈ { 1 , . . . , K } , ∀ κ ∈ { 1 , . . . K} , sign ( ˆ e T i δµ ( j ) k ) = ( σ ( D ) ) T ˆ e i and sign ( ˆ e T i δν ( j ) κ ) = -( σ ( D ) ) T ˆ e i if j ≥ i . But this means in particular that we can identify σ ( D ) k = -σ ( D ) κ = σ ( D ) for 1 ≤ k ≤ K and 1 ≤ κ ≤ K .

Putting everything together, we conclude that the pair M ( D ) , N ( D ) obtained from M,N by concatenation of all paths is feasible and has offset vectors on diametral simplices, δµ ( D ) k ∈ S σ ( D ) , δν ( D ) κ ∈ S -σ ( D ) ∀ k ∈ { 1 , . . . , K } , ∀ κ ∈ { 1 , . . . K} . What is left to show is that M ( D ) , N ( D ) has at least the same divergence as M,N . To this end, define the product curves

connecting the endpoints ( M ( i -1) , N ( i -1) ) and ( M ( i ) , N ( i ) ) , and the image sets C ( i ) := ( ˜ M ( i ) × ˜ N ( i ) )([0 , 1]) ⊂ R ( K + K ) × D . The divergence difference is a sum of line integrals:

At the start of the proof, we made the assumption without generality loss that ∃ σ ( D ) ∈ { +1 , -1 } D : sign ( µ k ) = σ ( D ) = -sign ( ν κ ) ∀ µ k ∈ M, ν κ ∈ N . Therefore, the directional derivatives d ( µ k ) ˆ n i , d ( ν κ ) -ˆ n i along the curve directions ± ˆ n i = ± ( ( σ ( D ) ) T ˆ e i ) ˆ e i are nonnegative. The determinant factors ∑ k l =1 ( 1 -( σ ( i ) l ) T δµ ( i -1) l ) and ( 1 -( σ ( i ) ι ) T δν ( i -1) ι ) are nonnegative due to the feasibility of M ( i ) , N ( i ) ∀ i ∈ { 0 , . . . , D } .

Lemma O.13. There is a dominating pair M ∗ , N ∗ with i ∈ { 1 , . . . , D } and σ ∈ {-, + } such that δµ k = σ ˆ e i , δν κ = -σ ˆ e i ∀ k ∈ { 1 , . . . , K } ∀ κ ∈ { 1 , . . . , K} .

Proof. By Lemma O.12, we may assume without generality loss that ∃ σ ∈ { +1 , -1 } D : δµ k ∈ S σ , δν κ ∈ S -σ , with S σ := { x ∈ R D | σ T x = 1 } , ∀ k ∈ { 1 , . . . , K } ∀ κ ∈ { 1 , . . . , K} .

̸

The following curves will be useful. For 1 ≤ i = j ≤ D , l ∈ { 1 , . . . , D } define

̸

By construction, γ ( ijl ) k (0) = µ k , ζ ( ijι ) κ (0) = ν κ , γ ( ijl ) 0 (1) = 0 , ζ ( ijl ) 0 (1) = 0 , γ ( ijl ) k (1) -γ ( ijl ) k -1 (1) = δµ k for 1 ≤ k = l ≤ K , ζ ( ijι ) κ (1) -ζ ( ijι ) κ -1 (1) = δν κ for 1 ≤ κ = ι ≤ K ,

̸

and

In particular, this implies that the new pairs of sets ( M ( l ) := { γ ( ijl ) 0 (1) , . . . , γ ( ijl ) K (1) } , N ), ( M,N ( ι ) := { ζ ( ijι ) 0 (1) , . . . , ζ ( ijι ) K (1) } ) are feasible.

̸

Assume that ∃ l ∈ { 1 , . . . , K } , m ∈ { 1 , . . . , K } , i ∈ { 1 , . . . , D } , j ∈ { 1 , . . . , D } , i = j : ( δµ T l ˆ e i ) = 0 ∧ ( δµ T m ˆ e j ) } = 0 . Note that we include the possibility of l = m . We can then assume without generality loss (by Lemma O.1) that the conditions of Lemma O.6 hold for either ˆ n + = 1 √ 2 ( ˆ ( σ T ˆ e j ) e j -( σ T ˆ e i )ˆ e i ) or ˆ n -= 1 √ 2 (( σ T ˆ e i )ˆ e i -( σ T ˆ e j )ˆ e j ) . In the case of ˆ n + , consider the product curve

̸

̸

and the image set C ( l ) := ( ˜ M ( l ) × N )([0 , 1]) . The divergence difference has the form

By construction, the directional derivatives d ( µ k ) ˆ n + Ψ α ( ˜ M ( k ) ( t ) || N ) are nonnegative. Likewise, the determinant term √ 2( σ T ˆ e i )(ˆ e T i δµ l ) is nonnegative by the simplex condition together with feasibility of ( M,N ) .

If the separability condition is fulfilled for ˆ n -instead, we can repeat the argument with indices i ↔ j swapped and index m instead of l .

̸

̸

Theorem O.10 immediately follows from Lemma O.13 by induction.

O.2 Reduction of the divergence to a univariate integral

In the previous subsections, we showed that Ψ α ( N || M ) is maximized by collinear and equidistant sets of means. This allows to evaluate Ψ α ( N || M ) through a one-dimensional integral via marginalization. Recall (Theorem O.9). Note furthermore that in both cases, constraining the first element of either N or M into the origin incurs no generality loss due to the translation invariance of Ψ α .

Being able to constrain the problem to a single dimension, we now show

Lemma O.14. In the collinear case, we may evaluate Ψ α as the one-dimensional divergence Ψ 1D α between two mixtures ∑ K k =0 w k N ( r k , σ 2 ) , ∑ K κ =0 ω κ N ( -ρ κ , σ 2 ) of univariate Gaussians centered around the means r k and -ρ κ ,

Proof. Without loss of generality, we may assume ˆ e to be the indicator vector of the first component due to rotation invariance of Ψ α . Using bracketed superscripts (0) to indicate the first vector component, and (1 :) to indicate the rest, we can use Fubini's theorem to write

The proof is analogous for the Laplacian mechanism.

O.3 Randomized Response Proofs

The following two results (first for the Rényi divergence, second for the hockey stick divergence) show that our (group) privacy guarantees for randomized response are given by the maximum of two terms that can be evaluated in constant time.

Theorem O.15.

arg max τ ∈ [0 , 1] ( K ++1)+( K -+1) 1 ∑ z =0      K + +1 ∑ i =1 w (1) i (1 -| z -τ (1) i | )   α ·   K -+1 ∑ j =1 w (2) j (1 -| z -τ (2) j | )   1 -α    subject to τ (1) i ∈ { θ, 1 -θ } , ∀ i, τ (1) j ∈ { θ, 1 -θ } , ∀ j, τ (1) 1 = τ (2) 1 , with w (1) i = Binomial( i -1 | K -, r ) , w (2) j = Binomial( j -1 | K + , r ) , is given by arg max { Ψ( m ˜ τ (1) ∥ m ˜ τ (2) ) , Ψ( m ˆ τ (1) ∥ m ˆ τ (2) ) } , where ˜ τ (1) 1 = ˜ τ (2) 1 =: τ ∈ { θ, 1 -θ } ( symmetric ) , ˜ τ (1) 2 ,..., ( K + +1) = 1 -τ, ˜ τ (2) 2 ,..., ( K -+1) = τ and ˆ τ (1) 1 = ˆ τ (2) 1 =: τ ∈ { θ, 1 -θ } ( symmetric ) , ˆ τ (1) 2 ,..., ( K + +1) = τ, ˆ τ (2) 2 ,..., ( K -+1) = 1 -τ

Proof.

The strategy is to show that

(a) Any mixture pair ( m τ (1) , m τ (2) ) where m τ (1) has greater mass in (1 -τ ) , i.e., ∑ i : τ (1) i =1 -τ w (1) i ≥ ∑ j : τ (2) j =1 -τ w (2) j , is dominated by ( m ˜ τ (1) , m ˜ τ (2) )

(b) Any mixture pair ( m τ (1) , m τ (2) ) where m τ (2) has greater mass in (1 -τ ) , i.e., ∑ i : τ (1) i =1 -τ w (1) i ≤ ∑ j : τ (2) j =1 -τ w (2) j , is dominated by ( m ˆ τ (1) , m ˆ τ (2) )

Notation:

where f strictly convex, increasing, and g strictly convex, decreasing.

̸

Trivial case: θ = 1 2 , thus assume θ = 1 2 for the rest of the proof.

0

Assume w.l.o.g. that τ 0 = max { θ, 1 -θ } , else swap roles of 0 ↔ 1 in the argument.

Decompose x : x ( z ) = x 0 ( z ) + δx ( z ) , where

x

z

(

z

) =

-|

δx

(1)

1

w

) = (

τ

-

w

(1)

τ

(

z

w

(1

-

Similarly, and by construction,

(1)

1

(1

-

)(1

-

-|

τ

z

)

|

-

) +

τ

(1)

1

i.e., 'part that stays fixed'; see Fig. 26

τ

)

,

,

-

|

.

w

(1

z

-|

i.e., 'part that relocates'; see Fig. 26

)

,

|

We can assume w.l.o.g. τ 0 = max { θ, 1 -θ } = ⇒ δx (1) > δx (0) ( ⋆⋆ ) (else, swap 1 ↔ 0 )

̸

Then, θ = 1 2 , hence we can assume w (1) τ > w (1) 1 (else, m ′ ≡ m in the first place).

Ψ

(

α

⋆

)

(

m

= [

f

τ

(

(1)

′

x

0

τ

′

, m

(0) +

(2)

)

δx

τ

τ

(1)

(2)

α

Ψ

(

m

, m

)

-

(1))

-

f

(

x

0

(0) +

δx

(0))]

Then, using that f is strictly convex, we have

Where f ′ ( x ) = x α -1 , and thus f ′ ( x ) g ( y ) = ( α -1) ( x y ) α -1

𝜃𝜃

𝜏𝜏′

0

0

1

1

𝛿𝛿𝑥𝑥

𝛿𝛿 �

𝑥𝑥

(here:

𝜏𝜏

)

(here:

𝜏𝜏

)

𝜏𝜏

1

1

=

𝜏𝜏

1

2

𝜏𝜏

1

-𝜃𝜃

𝜃𝜃

1

-𝜃𝜃

Proof of (a):

Given: Pair of mixtures ( m τ (1) , m τ (2) ) such that w (1) 1 -τ := ∑ i : τ (1) i =1 -τ w (1) i ≥ ∑ j : τ (2) j =1 -τ w (2) j =:

w (2) 1 -τ .

Further define w (1) τ = 1 -w (1) 1 -τ , w (2) τ = 1 -w (2) 1 -τ .

Define a new mixture via

g

(

y

(0)))

-

[

f

(

x

0

(1) +

δx

(1))

-

f

(

x

0

(1) +

δx

(0))]

g

(

y

(1))

̃

𝜃𝜃

𝜏𝜏

0

0

1

1

𝛿𝛿 �

𝑦𝑦

𝛿𝛿𝑦𝑦

′

(here:

𝜏𝜏

)

𝜏𝜏′

1

-𝜃𝜃

𝜃𝜃

1

-𝜃𝜃

(here:

𝜏𝜏

)

The last inequality follows from the following facts. Since we assume w (1) τ > w (1) 1 (else, m ≡ m ′ in the first place), we can bound the second term by:

Moreover, we have that the first term is bounded by

(strict > if w (1) 1 -τ > w (2) 1 -τ ). The conclusion follows from ( i ) , ( ii ) , and since WLOG τ > (1 -τ ) .

We have shown so far:

To complete the proof of ( a ) , we show now that Ψ α ( m ˜ τ (1) , m ˜ τ (2) ) > Ψ α ( m τ ′ (1) , m τ ′ (2) ) . Idea : Apply symmetric argument to means τ (2) .

where w (1) τ ′ , w (2) τ ′ are defined like w (1) τ , w (2) τ but for new means τ ′ .

Decompose y ′ : y ′ ( z ) = y ′ 0 ( z ) + δy ′ ( z ) , where

We can assume w.l.o.g. that τ = max { θ, 1 -θ } , so δy ′ (0) > δy ′ (1) .

And using the strict convexity of f we have

Proof of ( b ) is fully analogous.

This proves the theorem statement, since condition (a) or (b) is fulfilled for any mixture pair, and hence any mixture pair is dominated by the argmax of the two mixture pairs stated in the theorem.

Theorem O.16.

arg max τ ∈ [0 , 1] ( K ++1)+( K -+1) 1 ∑ z =0     K + +1 ∑ i =1 w (1) i (1 -| z -τ (1) i | )   -e ε   K -+1 ∑ j =1 w (2) j (1 -| z -τ (2) j | )     subject to τ (1) i ∈ { θ, 1 -θ } , ∀ i, τ (1) j ∈ { θ, 1 -θ } , ∀ j, τ (1) 1 = τ (2) 1 , with

is given by where

and

Proof sketch.

The strategy is to show that

(a) Any mixture pair ( m τ (1) , m τ (2) ) where m τ (1) has greater mass in (1 -τ ) , i.e., ∑ i : τ (1) i =1 -τ w (1) i ≥ ∑ j : τ (2) j =1 -τ w (2) j , is dominated by ( m ˜ τ (1) , m ˜ τ (2) )

(b) Any mixture pair ( m τ (1) , m τ (2) ) where m τ (2) has greater mass in (1 -τ ) , i.e., ∑ i : τ (1) i =1 -τ w (1) i ≤ ∑ j : τ (2) j =1 -τ w (2) j , is dominated by ( m ˆ τ (1) , m ˆ τ (2) )

Notation:

where f = [ · ] + is convex and differentiable anywhere except at 0.

̸

Trivial case: θ = 1 2 , thus assume θ = 1 2 for the rest of the proof.

Proof of (a):

Given: Pair of mixtures ( m τ (1) , m τ (2) ) such that w (1) 1 -τ := ∑ i : τ (1) i =1 -τ w (1) i ≥ ∑ j : τ (2) j =1 -τ w (2) j =:

Further define w (1) τ = 1 -w (1) 1 -τ , w (2) τ = 1 -w (2) 1 -τ .

Define a new mixture via

Assume w.l.o.g. that τ 0 = max { θ, 1 -θ } , else swap roles of 0 ↔ 1 in the argument.

Decompose x : x ( z ) = x 0 ( z ) + δx ( z ) , where

Similarly, and by construction,

We can assume w.l.o.g. τ 0 = max { θ, 1 -θ } = ⇒ δx (1) > δx (0) ( ⋆⋆ ) (else, swap 1 ↔ 0 )

̸

Then, θ = 1 2 , hence we can assume w (1) τ > w (1) 1 (else, m ′ ≡ m in the first place).

̸

Assume for now that x 0 (0) + δx (0) -e ε y (0) = 0 = x 0 (1) + δx (0) -e ε y (1) .

̸

The second-to-last inequality follows from w (1) τ > w (1) 1 (else, m ≡ m ′ in the first place). The last inequality follows from w (1) 1 -τ > w (2) 1 -τ and the prior assumption that, WLOG, τ > 1 -τ . If x 0 (0) + δx (0) -e ε y (0) > x 0 (1) + δx (0) -e ε y (1) and x 0 (1) + δx (0) -e ε y (1) = 0 , then we can arrive at the same conclusion by using any subderivative of f in [0 , 1] at x 0 (1) + δx (0) -e ε y (1) instead.

We have shown so far:

Continue in analogy to Theorem O.15.

P Towards epoch-level subsampling analysis

Standard composition theorems assume that the outputs of composed mechanisms are conditionally independent, meaning each output only depends on the previous outputs and the dataset (see, e.g., the proof of Proposition 1 in [7] and the proof of Theorem 27 in [10]). For this reason, one typically considers subsampling schemes like Poisson subsampling or subsampling without replacement, which yield independent batches in each iteration.

However, there may be scenarios where batches are not independent, such as when creating batches by permuting a dataset and splitting it into equal sized chunks. Such correlated subsampling schemes are appealing because one can, for instance, limit privacy leakage by ensuring that any element appears in only one iteration per epoch when training a model. A downside of these corellated subsampling schemes is that they may brake the conditional independence assumption of standard compositions theorems.

There already exist approaches to analyzing compositions of correlated mechanisms, such as probabilistically upper-bounding a mechanism's privacy leakage with uncorellated mechanisms [75, 24]. Furthermore, permutation-based batching has already been discussed in the context of convergence guarantees for noisy SGD [38]. Nevertheless, this problem setting provides a great opportunity to demonstrate that the utility of our conditional optimal transport framework extends beyond the subsampling schemes that are typically discussed in amplification-by-subsampling literature.

P.1 Problem setting

For this example, we consider a non-adaptive composition of two mechanisms, where two batches are created by permuting a dataset and splitting it in half. We assume that our dataset space is composed of size 2 N subsets of some finite, discrete set A , i.e., X ⊆ { x ∈ P ( A ) | | x | = 2 N } , and equipped with substitution relation ≃ ∆ . We further define a per-iteration batch space Y that is composed of size N subsets, i.e., Y = { y ⊆ x | x ∈ X ∧ | y | = N } , which is also equipped with substitution relation ≃ ∆ . Finally, we assume a base mechanism B : Y → Z with conditional density b y : Z → R + that maps from this batch space to some output space.

To apply our framework to epoch-level subsampling distributions, we define the composed batch space Y , which consists of equal-sized partitions of datasets in X , i.e.,

On this composed batch space, we can now define our epoch-level subsampling scheme:

Definition P.1. The permute-and-partition subsampling scheme is the subsampling scheme S : X → ˆ Y with

This definition follows from the fact that permuting-and-partitioning is equivalent to sampling a batch of size N without replacement for the first batch and using the remaining N elements for the second batch. We further consider the non-adaptively composed base mechanism ˆ B : ˆ Y → Z 2 with ˆ B ( y 1 ,y 2 ) = ( B y 1 , B y 2 ) and resultant joint density ˆ b ( y 1 ,y 2 ) = b y 1 · b y 2 .

P.2 Optimal transport without conditioning

As a baseline, we use Theorem 3.3, i.e., optimal transport without conditioning, to provide a RényiDP bound for the composed, subsampled mechanism. This guarantee captures the intuition that by permuting-and-partitioning, we only leak an elements' private information once. Again, note we prove this result for non-adaptive composition.

Theorem P.2. Assume a dataset space X ⊆ { x ∈ P ( A ) | | x | = 2 N } , batch space Y = { y ⊆ x | x ∈ X ∧ | y | = N } , and a base mechanism B : Y → Z that is ( α, ε ) -Rényi-DP w.r.t. single-element substitution relation ≃ ∆ , Y . Let S : X → ˆ Y be the permute-and-partition subsampling scheme defined in Definition P .1. Let ˆ B be the composed base mechanism with ˆ B ( y 1 ,y 2 ) = ( B y 1 , B y 2 ) . Then the subsampled, composed mechanism ˆ M = ˆ B ◦ S is also ( α, ε ) -DP w.r.t. single-element substitution relation ≃ ∆ , X .

Proof. Consider an arbitrary pair of datasets x ≃ ∆ , X . By definition of the substitution relation, there must be some a ∈ x and some a ′ ∈ x ′ such that x ′ = x \ { a } ∪ { a ′ } .

We can now define the following simple coupling between S x and S x ′ :

This coupling generates a pair of batch sequences by first applying permute-and-partition to original dataset x and then replacing a with a ′ in the one batch it appears in. Because we only couple pairs that are identical in one batch and differ by a substitution in the other batch, we have per Theorem 3.3

subject to ( y (1) 1 ≃ ∆ , Y y (2) 1 ) ∧ ( y (1) 2 = y (2) 2 ) ∨ ( y (1) 2 ≃ ∆ , Y y (2) 2 ) ∧ ( y (1) 1 = y (2) 1 ) . Either way, one of the factors is cancelled out when computing the likelihood ratio in the definition of Rényi-divergence (see Eq. (7)). We thus have

subject to y (1) 1 ≃ ∆ , Y y (2) 1 . The result then follows from the definition of Rényi-DP (see Definition 2.2).

P.3 Optimal transport with conditioning

Next, we use Theorem 3.4, i.e., optimal transport between conditional subsampling distributions, to derive a stronger guarantee. Again, note we prove this result for non-adaptive composition. Note that this result is similar in spirit to amplification by shuffling [76, 75], but considers central differential privacy of mechanisms operating on batches, instead of locally differentially private mechanisms that operate on individual elements. Again, note that we do not claim to be the first to consider the permute-and-partition scheme for composed mechanisms (see, e.g., [38]). It is just a nice showcase for the versatility of our framework in analyzing different subsampling schemes.

For the following result and proof we will use the following indexing convention: y (1) i,k is a batch associated with the first mixture ˆ m x , event A i , and the k th iteration. Similarly, y (2) j,k is a batch associated with the second mixture ˆ m x ′ , event E j , and the k th iteration.

Theorem P.3. Assume a dataset space X ⊆ { x ∈ P ( A ) | | x | = 2 N } , and corresponding batch space Y = { y ⊆ x | x ∈ X ∧ | y | = N } , equipped with single-element substitution relation ≃ ∆ , Y . Let S : X → ˆ Y be the permute-and-partition subsampling scheme defined in Definition P.1, and let B : Y → Z be some base mechanism. Let ˆ B be the composed base mechanism with ˆ B ( y 1 ,y 2 ) = ( B y 1 , B y 2 ) , and ˆ M = ˆ B ◦ S be the subsampled, coposed mechanism. Then, for all x ≃ ∆ , X x ′ ,

subject to y (1) 1 , 1 ≃ ∆ , Y y (2) 1 , 1 .

Proof. By definition of the substitution relation, there must be some a ∈ x and some a ′ ∈ x ′ such that x ′ = x \ { a } ∪ { a ′ } .

For the original subsampling scheme S x we let A 1 be the event that a appears in the first batch and A 2 be the event that a appears in the second batch. For the modified subsampling scheme S x ′ we let E 1 be the event that a ′ appears in the first batch and E 2 be the event that a ′ appears in the second batch. We have P S x ( A 1 ) = P S x ( A 2 ) = P S ′ x ( E 1 ) = P S ′ x ( E 2 ) = 1 2 .

One can sample from the distributions conditioned on A 1 or E 1 by first sampling uniformly at random N elements from the 2 · N -1 elements that are not a or a ′ , and then using the remaining elements as the first batch. Similarly, one can sample from the distributions conditioned on A 2 or E 2 by first sampling uniformly at random N elements from the 2 · N -1 elements that are not a or a ′ , and then using the remaining elements as the second batch.

We thus have

and

Next, we can define a coupling via

Condition P.4. Batch tuples y (1) 1 ∈ A 1 , y (1) 2 ∈ A 2 , y (2) 1 ∈ E 1 , y (2) 2 ∈ E 2 fulfill Condition P.4 when y (1) 1 , 1 = y (1) 2 , 2 , and y (1) 1 , 2 = y (1) 2 , 1 , and y (1) 1 , 2 = y (2) 1 , 2 , and y (1) 2 , 1 = y (2) 2 , 1 , and y (2) 1 , 1 = y (2) 1 , 1 \ { a } ∪ { a ′ } , and y (2) 2 , 2 = y (2) 2 , 2 \ { a } ∪ { a ′ } .

In short: We sample uniformly at random a batch tuple ( y (1) 1 , 1 , y (1) 1 , 2 ) from A 1 . We then generate a batch tuple from A 2 by permuting the A 1 tuple. We then generate a batch tuple from E 2 by replacing a with a ′ in the A 2 tuple. We finally generate a batch tuple from E 1 by permuting the E 2 tuple.

Because for each possible value of a batch tuple there is only one combination of the other three batch tuples such that γ ( y ) > 0 , and in this case γ ( y ) = ( 2 N -1 N ) -1 , this is a valid coupling.

The result then follows from Theorem 3.4 and the constraints imposed by Condition P.4.

Note that this result could be generalized to K iterations: We can upper-bound the subsampled, composed mechanism's divergence by adversarially choosing a K -fold partition of x , constructing a mixture with K (not K ! ) components, with each component corresponding to a appearing in a specific batch, and finally constructing a modified mixture by replacing every occurence of a with a ′ .

P.4 Experimental Evaluation

Finally, we can compare our epoch-level amplification guarantees obtained via optimal transport with and without conditioning. As an additional baseline, we use 2 -fold composition of subsampling without replacement with batch size 2 . Note that, with the baseline, a modified element may appear in 0 , 1 or 2 of the iterations. For the sake of this simple example, we consider output space Z = R , and let base mechanism b be the Gaussian mechanism f + N (0 , σ ) with f : Y →{ 0 , 1 } and univariate standard deviation σ ∈ { 0 . 5 , 1 . 0 , 2 . 0 , 5 . 0 } . By this construction we do not need to reason about f 's sensitivity in determining the worst-case mixture components.

We make the following observation: For small σ , both epoch-level guarantees are almost identical and outperform the without replacement guarantee. With increasing σ , subsampling without replacement outperforms Theorem P.2 for some ranges of α . Theorem P.2, which is derived via conditional optimal transport, however yields smaller ε .

The main takeaway of this example should be that (a) there is a benefit to using conditional optimal transport to bound privacy in terms of mixtures and (b) conditional optimal transport can be used to reason about schemes other than the typically considered Poisson subsampling, subsampling without replacement, and subsampling with replacement.

q / N

= 0

.

01

10

1

10

2

RDP order

α

10

1

10

2

RDP

ε

(

α

)

Without replacement

Theorem P.2

Theorem P.3

q / N

= 0

.

1

10

1

10

2

RDP order

α

10

0

10

1

10

2

RDP

ε

(

α

)

Without replacement

Theorem P.2

Theorem P.3

q / N

= 0

.

2

10

1

10

2

RDP order

α

10

-

1

10

0

10

1

RDP

ε

(

α

)

Without replacement

Theorem P.2

Theorem P.3

q / N

= 0

.

5

10

1

10

2

RDP order

α

10

-

2

10

-

1

10

0

RDP

ε

(

α

)

Without replacement

Theorem P.2

Theorem P.3

Q Broader impact

Our work contributes towards provably protecting users, and specifically groups of users, from the negative societal impact of privacy leakage. However, differential privacy may have negatively impact other aspects of trustworthy machine learning, such as fairness or robustness. Also, training for differentially private machine learning is often performed with 1 < ε ≤ 10 . This is useful for relative comparisons between models, but does not impose any meaningful constraints on absolute privacy leakage (the probability of any event may increase by a factor of up to ≈ 22000 ) and may thus be used to falsely advertise privacy.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We verify the generality of our proposed framework by formally deriving various known and novel subsampling guarantees Appendices G to I and K to M. We verify the benefit of using mechanism-specific guarantees and analyzing group privacy and subsampling jointly through our experimental evaluation in Section 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss limitations in Section 3.5. Our main contribution is a framework for proving theoretic results, as such many of the points listed below do not apply.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Yes, we provide formal proofs for all theoretical results in Appendix E through Appendix P. All theorems and lemmata state the underlying assumptions.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We fully describe our experimental setup in Appendix C. We further provide an implementation with the supplementary material, which includes the needed environment, code, configuration files, and plotting scripts.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide an implementation that includes the needed environment, code, configuration files, and plotting scripts for reproducing our experimental results at https://cs.cit.tum.de/daml/group-amplification.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We fully describe our experimental setup in Appendix C. Note that these experiments only require numerical evaluation of different divergences, meaning there are no involved data splits or optimizers.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Our experiments only require numerical evaluation of different divergences. There are no typical sources of randomness like data splits or weight initializations that could be visualized with error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes, we report the type and number of CPUs, as well as the number of workers and allocated runtime Appendix C.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research in this paper conforms, in every respect, with the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Yes, we discuss both positive and negative broader impact in Appendix Q, which we reference in our 'Limitations, broader impact, and future work' section Section 3.5.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not propose any new data or models that could be released.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We list the libraries that we use and/or extend to conduct our numerical experiments in Appendix C.3, including version numbers and licenses. We cite the original papers where applicable.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The study proposes a framework utilizing optimal transport theory to derive mechanism-specific and mechanism-agnostic guarantees for subsampling mechanisms in differential privacy. It focuses on leveraging conditional optimal transport to establish tight bounds on privacy parameters such as α and β, crucial for various privacy scenarios including Rényi differential privacy and moments accounting. The experiments evaluate privacy guarantees under different mechanisms including Gaussian, Laplace, and randomized response, with specific settings for noise parameters and group sizes.

STRENGTHS:
- Defining partitions into events based on dataset manipulation scenarios.
        - Creating couplings between conditional distributions to ensure compatibility.
        - Bounding mixture divergences using worst-case mixture components.
        - Employing advanced joint convexity techniques to tighten divergence bounds.
        - Applying distance-compatible couplings to enhance robustness of privacy guarantees.

WEAKNESSES:
This manuscript is too long, which are 100 pages.  
Lack of theoretical analysis in the main text

QUESTIONS:
Will the approach be adopted to LDP?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a principled approach to analyzing group-privacy amplification through sub-sampling by generalizing the coupling arguments of Balle et al., 2018. This generalization extends the analysis from $1$-neighboring datasets to $K$-neighboring datasets. The core idea is to define a coupling between partitions of batches rather than the batches themselves, ensuring that the divergence of the output distribution, conditional on the partitioning, is bounded individually by the worst-case divergence. By carefully selecting the partitioning and the optimal coupling specific to the sub-sampling mechanism, the authors demonstrate that improved Rényi and Approximate DP bounds for group privacy can be achieved, surpassing the bounds obtained by applying privacy amplification by subsampling and group-privacy theorems separately. The paper shows that the combined analysis of grouping and subsampling can be tailored to a specific base mechanism, leading to mechanism-specific privacy amplification bounds that are superior to mechanism-agnostic bounds. Although the DP bounds presented are not in closed form, they can be numerically approximated to arbitrary precision using standard techniques from the literature. The paper also provides a comprehensive evaluation of how privacy is amplified for different group sizes under Gaussian, randomized response, and Laplace base mechanisms.\n\nSTRENGTHS:\n- The paper presents an in-depth analysis of privacy amplification due to subsampling under groupings. This topic has not been extensively studied before, making it an interesting paper to read.\n\n- The paper demonstrates that there is a significant interplay between subsampling and grouping concerning privacy, which is not captured in standard analyses. This finding is an important contribution to the privacy community.\n\n- The paper argues that by considering the underlying base mechanism, the amplification bounds due to subsampling can be refined beyond the tight mechanism-agnostic bounds established by Balle et al., 2018. While the results indicate this improvement for group privacy, the paper also claims it applies more generally. This could be a significant finding (although I am not entirely convinced of its validity).\n\nWEAKNESSES:\n- The authors argue that tailoring privacy amplification through subsampling to a specific mechanism can yield better bounds than the tight amplification bounds established by Balle et al., 2018. However, it appears that the paper only demonstrates privacy amplification under subsampling for Approximate-DP with groups of size 2 or larger. Does this imply that for the standard setting with no grouping (i.e., group size of 1), there is no additional subsampling amplification achievable by tailoring the analysis to the mechanism?\n\n - I remain unconvinced that tailoring the analysis to a specific mechanism can result in tighter amplification-due-to-subsampling bounds for a group size of 1. Could the authors provide an example of two mechanisms, $\\mathcal{M}_1$ and $\\mathcal{M}_2$, that both satisfy $(\\epsilon, \\delta)$-DP tightly, but for some subsampling mechanism $S$, mechanism $\\mathcal{M}_1 \\circ S$ is $(\\epsilon', \\delta')$-DP while $\\mathcal{M}_2 \\circ S$ cannot achieve $(\\epsilon', \\delta')$-DP? \n\n- The figures in the paper (Figure 4 to Figure 10, Figure 13 to 15, Figure 17 to 20) do not include a baseline comparison with the separate application of tight amplification by subsampling and group privacy as described in lines 58-64. Without this comparison, it is unclear the gap between the bounds in this paper and the standard approach.\n\n- Definition 2.2 of Rényi divergence is misleading; a concave function $f(\\bullet) = \\frac{1}{\\alpha - 1} \\log(\\bullet)$ needs to be applied to your expression for $R_\\alpha(m_x\\Vert m_{x'})$ to obtain the Rényi divergence. The paper's expression for $R_\\alpha(m_x\\Vert m_{x'})$ is typically referred to as the $\\alpha$th moment of the privacy loss random variable. Although the moment $R_\\alpha$ may be jointly convex (a property heavily used in the paper), the Rényi divergence is not jointly convex due to the mapping $f$ being concave. Additionally, showing that $R_\\alpha$ shrinks by a factor of $p$ on subsampling does not mean that the Rényi divergence shrinks by a factor of $p$ due to the concavity of $f$. \n\n- The paper does not provide analytical bounds to help understand the asymptotics of the subsampling amplification results proved. Without the asymptotics, it's hard to verify claims of tightness.\n\n### Minor Concerns:\n\n- The privacy amplification results are not presented in a closed-form solution, making them difficult to operationalize.\n\n- The subsampling bounds for Rényi DP have not been compared with other existing works, such as [1,2].\n\n- The placement of figures does not match the respective appendices where they are discussed, making referencing the figures very challenging.\n\nReferences:\n\n[1] Steinke, Thomas. \"Composition of differential privacy & privacy amplification by subsampling.\" arXiv preprint arXiv:2210.00597 (2022).\n\n[2] Zhu, Yuqing, and Yu-Xiang Wang. \"Poisson subsampled Rényi differential privacy.\" International Conference on Machine Learning. PMLR, 2019.\n\nQUESTIONS:\nI don't have any specific questions. I encourage authors to explain or address the problems I described in the weakness section. Based on how convincing the answers are, I'm open to adjusting my score.\n\nLIMITATIONS:\nAuthors discuss some of the limitations of the paper. Following are some additional limitations that need addressing.\n\n- The ansatz is unwieldy for finding general use.\n- Closed-form asymptotics for the presented bounds are missing.\n- It's not clear if mechanism-specific subsampling bounds can be strictly better than mechanism-agnostic subsampling bounds for group size=1.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
The authors propose a general framework for deriving mechanism-specific differential privacy guarantees for amplification by subsampling. The current methods are generally only tight in a mechanism-agnostic sense, but may possibly be significantly more private. The authors propose a new framework using conditional optimal transport, which allows stronger mechanism-specific subsampling guarantees, while still recovering the mechanism-agnostic guarantees. They also derive guarantees for privacy accountants in a unified manner. As an application, they derive tight guarantees for group privacy, where previously only a weak bound is known.

STRENGTHS:
- The problem being studied is important and relevant for practical applications.
- The framework derived in this paper seems general and broadly applicable.

WEAKNESSES:
- The authors only demonstrate improvements in the privacy bounds using their framework for group privacy. It would be interesting to see if their general framework could give improvements in other practical areas. It would also be good to demonstrate utility improvement in practice when doing DP training with these improved bounds.

QUESTIONS:
- Is there any intuition about the types of scenarios where this new framework gives improvements over existing mechanism-agnostic bounds?

LIMITATIONS:
- See weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

comments: SUMMARY:
The authors propose a framework for privacy accounting of amplification by subsampling. An existing principle for this problem is to consider couplings between the output distribution of the mechanism on neighboring datasets and apply joint convexity of the privacy measure at hand. The primary contribution is to select the optimal coupling and minimize the resulting privacy bound by viewing it as an optimal transport problem.

STRENGTHS:
The paper provides a novel technical framework for accounting of subsampling amplification. The authors are able to reproduce important closure properties of dominating pairs using their framework (e.g. Proposition K.3), suggesting their framework may be used to analyze mechanisms that previously evaded accurate accounting (e.g. shuffling).

WEAKNESSES:
The main technical result Theorem 3.4 did not feel clearly motivated in terms of Theorem 3.3 and the experiments in Appendix B. It was not clear to me from the justification provided on p.4 lines 171-174 why Theorem 3.4 is able to improve substantively upon the preceding result.

QUESTIONS:
Small editorial suggestions:
- It is difficult to follow the quantifier constraints in some places e.g. p.5 l.193
- p.4 l.178 Missing $y^{(1)}$ and $y^{(2)}$ under sum
- p.36 l.852 should this be $x'$ not $x$?

LIMITATIONS:
While the paper leverages their framework to provide new tighter bounds for the group privacy setting, the framework is only used in the other privacy accounting settings to reproduce known results (from e.g. Zhu et al 2022 or Balle et al 2018). A future direction for this work, which the authors do mention, is to apply the framework to address the practical gaps in privacy accounting for machine learning.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

comments: SUMMARY:
This work proposed a novel analysis of mechanism-specific amplification via subsampling. The authors decompose the subsampled mechanism into two parts: batch subsampling + mechanism. Then the analysis decompose the probability density into sums of pdf of every batch. The authors then provide upper bound to the divergence and specifically applied to group privacy under poisson distribution. The empirical analysis shows that the proposed analysis is able to significantly improves upon prior mechanism-agnostic group privacy bounds.

STRENGTHS:
- The paper proposed novel analysis for group privacy. The application to poisson subsampling + gaussian mechanism seems solid.
- Solid theoretical guarantees.
- Simulation results look promising.

WEAKNESSES:
- Theorem 3.4 + proposition 3.5 themselves seem very costly to compute. I appreciate the authors for providing example of poisson subsampling. Still I'm wondering whether, e.g. proposition 3.5, is easy to solve for non Poisson distribution. 
- Similar to the previous point, a small weakness for Theorem 3.7 is it's unclear how hard it is to solve for mechanisms such as Laplace. If it does not have a closed form (or closed form hard to derive), it might lead to un-tight upper bound or costly numerical simulation in practice. (Edit: I see the Laplace analysis in the appendix)
- The theoretical guarantees consist of a lot of notation. Might be helpful for the authors to add small interpretation of the bounds in e.g. Eq  2 and Eq 4 for better readibility.
- Could the authors add group size = 1 in the experiment for completeness?

QUESTIONS:
See above.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14